From 0944d9bb78f1d07b48b54a21d1c4e381b71ac69b Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 14:51:29 +0200 Subject: [PATCH] docs(plans): re-cast Phase 4-7 to stateless scraper-APIs, flatten subplans --- plans/backend/00-umbrella-plan.md | 124 ++++--- plans/backend/00b-pipeline-diagrams.md | 110 ++++++ plans/backend/02-rename-backend.md | 2 +- plans/backend/03c-crawl-billing.md | 2 +- plans/backend/04-capabilities.md | 89 +++++ plans/backend/04a-connector-category.md | 167 --------- plans/backend/04b-source-discovery.md | 120 ------ plans/backend/05-access.md | 85 +++++ plans/backend/05-pipelines-model.md | 321 ---------------- plans/backend/06-ongoing-automation.md | 25 ++ plans/backend/06-pipelines-exec.md | 350 ------------------ plans/backend/07-orchestration.md | 73 ++++ plans/backend/07-upload-pipeline-kb.md | 195 ---------- .../backend/revamp phases 4-7/00-overview.md | 125 ------- .../00b-pipeline-diagrams.md | 202 ---------- .../revamp phases 4-7/04a-capabilities.md | 198 ---------- plans/backend/revamp phases 4-7/04b-access.md | 136 ------- .../backend/revamp phases 4-7/05a-timeline.md | 125 ------- .../revamp phases 4-7/05b-intelligence.md | 191 ---------- .../backend/revamp phases 4-7/06-triggers.md | 131 ------- .../revamp phases 4-7/07-orchestration.md | 126 ------- 21 files changed, 447 insertions(+), 2450 deletions(-) create mode 100644 plans/backend/00b-pipeline-diagrams.md create mode 100644 plans/backend/04-capabilities.md delete mode 100644 plans/backend/04a-connector-category.md delete mode 100644 plans/backend/04b-source-discovery.md create mode 100644 plans/backend/05-access.md delete mode 100644 plans/backend/05-pipelines-model.md create mode 100644 plans/backend/06-ongoing-automation.md delete mode 100644 plans/backend/06-pipelines-exec.md create mode 100644 plans/backend/07-orchestration.md delete mode 100644 plans/backend/07-upload-pipeline-kb.md delete mode 100644 plans/backend/revamp phases 4-7/00-overview.md delete mode 100644 plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md delete mode 100644 plans/backend/revamp phases 4-7/04a-capabilities.md delete mode 100644 plans/backend/revamp phases 4-7/04b-access.md delete mode 100644 plans/backend/revamp phases 4-7/05a-timeline.md delete mode 100644 plans/backend/revamp phases 4-7/05b-intelligence.md delete mode 100644 plans/backend/revamp phases 4-7/06-triggers.md delete mode 100644 plans/backend/revamp phases 4-7/07-orchestration.md diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index cd0373052..300cd9e68 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -2,16 +2,30 @@ > Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`). -This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities & Access**, **Phase 5 Intelligence & Timeline**, **Phase 6 Triggers**, **Phase 7 Orchestration** (canonical subplans in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was **superseded on 2026-06-30** — see the Architecture correction below. The earlier "Phase 5′ automation-enhancements" framing is also retired (folded into Phases 4/5/6 of the revamp). +This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities** ([`04-capabilities.md`](04-capabilities.md)), **Phase 5 Access** ([`05-access.md`](05-access.md)), **Phase 6 Ongoing-Automation** ([`06-ongoing-automation.md`](06-ongoing-automation.md), design deferred), **Phase 7 Orchestration** ([`07-orchestration.md`](07-orchestration.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful **Tracker + Timeline + Triggers** design was superseded on **2026-07-01** (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below. > SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases. ## Positioning -"NotebookLM for Competitive Intelligence" — each WorkSpace acts as a workspace for setting up competitive-intelligence-optimised notebooks. +Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return **cleaned, AI-ready data** for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time. + +## ⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native + +> **This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6).** The canonical Phase 4–7 subplans (`04`–`07`, below) are re-cast to four phases: +> +> - **The product = scraper capabilities**, not a CI engine. `web.*` + `maps.*` typed verbs return cleaned, AI-ready data, exposed identically through **REST + API key**, an **MCP server**, and **chat**, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable). +> - **Stateless.** No Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`. Memory = the **chat history**; the agent reasons over prior tool outputs in context to report "what changed." +> - **Direct calls.** Capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling. Async results reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table). +> - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6). +> - **KB stays input-only**; crawled data is never indexed. +> +> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** (deferred) · **07 Orchestration** (`intelligence_agent`). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`. ## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted +> **Historical record.** The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the **2026-07-01 pivot** above. + > **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model: > > - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages. @@ -20,7 +34,7 @@ This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shi > - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots. > - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source. > -> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical revamp in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md): +> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical subplans `04`–`07` (this framing was itself superseded by the 2026-07-01 pivot above): > - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.* > - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine. > - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path). @@ -28,27 +42,23 @@ This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shi > > **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors. -## Target architecture (corrected + revamped) +## Target architecture (scraper-APIs-first, stateless) -> Full stateless/stateful end-to-end flow diagrams live in [`revamp phases 4-7/00b-pipeline-diagrams.md`](revamp%20phases%204-7/00b-pipeline-diagrams.md). Summary below. +> Full end-to-end flow diagrams live in [`00b-pipeline-diagrams.md`](00b-pipeline-diagrams.md). Summary below. ```mermaid flowchart TD - ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (P8)"]] - REG["Capability registry (P4a): web.scrape · web.discover (MVP) · <platform>.* scrapers (later) — typed verbs, bill at executor"] + ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]] + REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"] ACQ --> REG - REG --> DOORS["Access doors (P4b): chat tools · REST+API key · MCP — generated from the registry"] - DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · Tracker crafting"] + REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"] + DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · 'what changed' from chat history"] KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT - AGENT -->|one-shot / Product A| ANS["plain-language answer (nothing persists)"] - AGENT -->|standing concern / Product B| TR["Tracker (P5b): locked schema · refresh hot loop"] - REG --> TR - TR --> TL[("Timeline (P5a): tracked_entities · entity_current_state · entity_changes — delta moat")] - TRIG["Triggers (P6): manual · agent · external-cron · CI automation action"] --> TR - AUTO["Automations (existing): schedule selector + AutomationRun (recurrence + audit; NO delivery)"] --> TRIG - TL --> DELIVER["alerts on material change → app/notifications (Zero-synced)"] - APIKEY["Platform API key (future)"] --> DOORS + AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"] + AGENT -.->|standing need| ONG["Ongoing-Automation (P6, DEFERRED): persistent chat re-invokes verbs; memory = chat history"] + ONG -.-> REG + APIKEY["Platform API key"] --> DOORS ```
@@ -78,14 +88,14 @@ flowchart TD - Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps. - Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`. -- **Capabilities, not connectors, are the core of Phase 4.** The crawler + search become typed **verbs** (MVP `web.scrape` + `web.discover`; per-platform scrapers like `maps.*` are later, uncommitted drop-ins) in a single **capability registry** that generates the chat/REST/MCP doors identically (revamp `04a`/`04b`). A capability is *call → data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into revamp `04b` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system. **~~pipeline-eligibility~~ removed.** -- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent (or a Tracker binding) can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set. -- **`Tracker` + Timeline are the standing-concern primitive** (replaces "pipeline"). A Tracker binds a capability + input and a locked, versioned `definition` (`field_schema`/`identity_rule`/`materiality`); its `refresh(tracker)` hot loop writes a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`). **Store deltas, no change → no row.** The Timeline — not the KB, not `automation_runs` — is the only new durable CI state (revamp `05a`/`05b`). -- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via `WebCrawlCreditService` (03c) — so chat, automation/recurring, REST, MCP, and external-cron all meter **uniformly**. (Closes the confirmed gap where automation-run crawls billed nothing; see 03c + the Architecture correction.) +- **Capabilities, not connectors, are the core of the product.** The crawler + search + Maps actor become typed **verbs** (`web.scrape`, `web.discover`, `maps.search/place/reviews`) in a single **capability registry** that generates the REST/MCP/chat doors identically (`04`/`05`). A capability is *call → cleaned data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into `05` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system. +- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set. +- **Stateless — the chat history is the memory.** No Timeline, no `Tracker`, no diff/materiality engine, no stored `entity_changes`. "What changed" is the agent reasoning over prior tool outputs already in context (`07`). Standing needs are served by a **persistent ongoing chat** that periodically re-invokes verbs (`06`, design deferred). +- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via the billing service (`WebCrawlCreditService`, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter **uniformly**. `maps.*` registers a new per-place / per-page unit. - Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP. - MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point). -- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines. The standing-concern need is met by the **`Tracker` + Timeline** (revamp `05`), and **recurrence + audit reuse the existing automations** (schedule selector + `AutomationRun`) via a CI **action** (revamp `06`) — **no new scheduler, no new run table**. **Delivery does NOT reuse automations** (no such path; `automation_runs.output` is never written) — alerts wire to the separate `app/notifications/` system. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper). -- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by reading the **Timeline** (`query_timeline`, revamp `05a`), not by re-deriving from run logs. Recurring execution audit rides `AutomationRun` (already persisted + queryable). No separate pipeline-run context tool. +- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a **persistent ongoing chat** re-invoking verbs (`06`, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper). +- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by **reasoning over the chat history** (prior tool outputs already in context, `07`), not by re-deriving from run logs or a stored timeline. - **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**. - Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that. @@ -124,52 +134,50 @@ flowchart TD The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans: -> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (revamp `04a`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture correction; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then revamp `04a` Capabilities → `04b` Access → `05` Intelligence & Timeline), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized. +> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (`04`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then `04` Capabilities → `05` Access → `07` Orchestration), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized. - **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran). -- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability/per-Tracker override is left as a no-op seam (revamp `04a`/`05`), not built. -- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture correction + revamp — see log): the charge moves to the **capability executor** (revamp `04a`), so **every** caller meters uniformly — chat, **automation/recurring** (the confirmed gap: `run_agent_task` establishes no turn accumulator today, so `scrape_webpage` bills nothing under automations), REST, MCP, and external-cron — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type). +- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability override is left as a no-op seam (`04`), not built. +- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture pivot + revamp — see log): the charge moves to the **capability executor** (`04`), so **every** caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type). - **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier. - **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account. - **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier. -### Phase 4 — Capabilities & Access (backend) [`subplans: revamp/04a–04b`] +### Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`] -> **Canonical subplans:** [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md). Together they ship **Product A** (stateless utility) — revenue day one. **Build `04a` first** — every later phase calls the registry. +> **Canonical subplan:** [`04-capabilities.md`](04-capabilities.md). **Build first** — every later phase calls the registry. -- **`revamp/04a-capabilities.md`** — Turn Acquisition (crawler + search + a future Maps actor) into a **small set of typed, callable verbs** over a single **capability registry**. MVP verbs = **only `web.scrape(urls[])`** (array-friendly, inline-or-job) **+ `web.discover(query, top_k)`** (SearXNG/Linkup/Baidu). **Platform-specific scrapers** (Maps/LinkedIn/Amazon/… — *examples, none committed*) are a family of **individual endpoints** added later: each is just another verb → agent tool + dev API-key REST endpoint, so `04a` only ships **contract stubs** for them, not executors. Uniform result **envelope** (`completed | pending + job_id`); slow calls use a **thin job record** + existing Celery (not a resurrected `pipeline_runs`). Each verb declares a **`billing_unit`**; charging is **delegated to the billing service** (03c is the first provider) and happens **at the executor** so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary. -- **`revamp/04b-access.md`** — Expose the registry through **three generated doors** (order: chat → REST+API-key → MCP), each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → envelope`). **Natural language is the only human-facing surface** (users never name verbs); an **intent router** classifies one-shot (Product A) vs standing-concern (Product B). Slow chat verbs reuse the shipped `deliverable_wait` poll-until-terminal + live-card path. This is where the **BYO-`MCP_CONNECTOR` routing-gap fix** (from the old `04a`) lands (consume-user-MCP). +- Turn Acquisition (crawler + search + Maps actor) into a **small set of typed, callable verbs** over a single **capability registry** — `web.scrape(urls[])`, `web.discover(query, top_k)`, `maps.search/place/reviews`. Each verb is a **direct async call that returns cleaned, AI-ready data** (no job store, no envelope, no polling). `maps.*` returns typed structured objects, not markdown. Each declares a **`billing_unit`**; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary. **Locked model (MVP):** -- **Capabilities replace connectors as data sources.** `web.scrape` / `web.discover` (MVP) — and later per-platform scraper verbs — are *call → data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule. -- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `revamp/04b`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**. +- **Capabilities replace connectors as data sources.** `web.*` / `maps.*` are *call → cleaned data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule. +- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `05`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**. - **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP. - **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook). - Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.) -- The deferred "coming soon" platforms (LinkedIn, Amazon, Google, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as **Phase 8 native verbs / API endpoints**, not Type-1 KB connectors. +- The **Google Maps actor is net-new**, proprietary, built as a separate effort; `maps.*` are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors. - Frontend connector/capability UI restructure is DEFERRED (frontend umbrella). -### Phase 5 — Intelligence & Timeline (backend) [`subplans: revamp/05a–05b`] +### Phase 5 — Access (backend) [`subplan: 05-access.md`] -> **Canonical subplans:** [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md). Ships the **Product B engine**. **Depends on `04a`** (the verbs the loop calls). **Build `05a` (tables) before `05b` (the loop that writes them).** This *replaces* the old Pipelines Phases 5–7 — the durable CI state, and **the moat**. +> **Canonical subplan:** [`05-access.md`](05-access.md). **Build after `04`.** Together `04 + 05` ship the scraper-API product — revenue day one. -- **`revamp/05a-timeline.md`** — The **3-store delta Timeline**: `tracked_entities` (stable identity, written once), `entity_current_state` (latest values + `content_hash` + `last_checked_at`, overwritten each run), `entity_changes` (append-only material deltas: `{field:{from,to}}`, `materiality`, `decided_by`, `source_ref`). **Store deltas, no change → no row** (storage ∝ rate of change). Explicitly **not the KB, not `automation_runs`, not a resurrected `pipeline_runs`.** New CI-owned Apache-2 tables + Alembic migration; write/read API for the `05b` loop and the read side. **This is where "diffable run history" actually lives** (superseding the old "structured `automation_runs.output`" idea). -- **`revamp/05b-intelligence.md`** — The `Tracker` primitive + the **hot loop** `refresh(tracker)`: crawl (call the bound verb) → **content-hash pre-check** (identical → stamp `last_checked_at`, stop, no LLM) → **fill** raw data to the locked `field_schema` (structured output; extras → `notable_signals`) → **diff** (code) → **judge — the materiality split** (deterministic numeric/clear rules in code for free; agent only on ambiguous) → **append** a change + overwrite current state, else only `last_checked_at`. Setup = an **agent-designed, human-locked, versioned schema** flow (sample-fetch → draft `definition` → review/lock). **No new run table:** recurring audit/idempotency ride `AutomationRun`, manual rides the chat job record; **billing idempotency is per-capability-call** + the hash pre-check. Optional **CI context folder** (user files uploaded in a CI chat go into the KB as normal, routed to a folder, and can feed the judge). New Apache-2 package `app/intelligence/`. +- Expose the registry through **three generated doors** — **REST + API key** (public day one), an **MCP server** (fast-follow), and **chat tools** — each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data`). The executor returns directly; there is **no envelope, no `deliverable_wait` polling**. Async results (e.g. the `06` ongoing mode) reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table). +- **Natural language is the human surface** (raw verbs only on REST/MCP); the chat door classifies **one-shot** vs **keep-watching** and hands keep-watching to `06`. Two MCP directions kept distinct: **serve** our verbs as an MCP server (new door) vs **consume** the user's BYO `MCP_CONNECTOR` (the old-`04a` routing-gap fix lands here). -### Phase 6 — Triggers (backend) [`subplan: revamp/06-triggers.md`] +### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`] -> **Canonical subplan:** [`revamp phases 4-7/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) — the thinnest phase. **Depends on `05b`** (`refresh(tracker)`). Decides **when** a Tracker refreshes; Intelligence never knows which trigger fired. +> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **⚠️ design deferred.** The periodic mechanism is designed separately, after `04`/`05`/`07`. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel). -- Adapters: **manual** (user "refresh now" via chat tool / REST), **agent** (the agent calls `refresh` as a tool), **external cron** (`POST /v1/trackers/{id}/refresh`, zero infra on us), and the **CI automation action** — recurrence reusing the **existing automations**: its hardened **schedule selector** (no new scheduler) + **`AutomationRun`** (no new run table). **Alert delivery is separate** — persist deltas to the Timeline, then emit an in-app notification via `app/notifications/` (automations have no delivery path; `run.output` is unwritten). Guard concurrency with a **per-Tracker lock** (the automation PENDING-gate is non-atomic). A single `refresh_tracker` action, **not** a new automation shape. -- **Decoupling preserved:** CI core (`refresh` + Timeline) runs with **zero** automations dependency (manual/agent/external-cron); automations is the **optional** in-app recurrence+alert adapter. This is the correct home for the old "wire a run-now trigger" item. +- Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls. -### Phase 7 — Orchestration (backend) [`subplan: revamp/07-orchestration.md`] +### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`] -> **Canonical subplan:** [`revamp phases 4-7/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — the human-facing brain. **Build last** (atop `04`–`06`). We plug into the shipped multi-agent runtime; we don't rebuild it. +> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Build last** (atop `04`/`05`, and `06` for the ongoing mode). We plug into the shipped multi-agent runtime; we don't rebuild it. -- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs standing concern, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), **Tracker crafting** (the `05b` schema-design flow in chat), and **decision-grounded answering** (read the Timeline via `query_timeline`, not chat history). -- Toolset: registry-backed capability verbs (shared with `research`) + Tracker/Timeline tools (`craft_tracker`/`lock_tracker`/`refresh_tracker`/`query_timeline`/`list_trackers`); slow verbs reuse `deliverable_wait`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing). +- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store). +- Toolset: registry-backed capability verbs (shared with `research`) + a deferred `start_watch` handoff to `06`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`). ### (Future) Platform API-key + public MCP @@ -177,7 +185,7 @@ Platform **API-key** access to the capability verbs + a public **MCP server** ex ### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`] -NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Once the hardened fetch core (Phases 3a–3e) is solid, **platform scrapers** — a family of **individual scraping endpoints** (Google Maps/Local, LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **all examples, none committed**) — layer **on top** of it. **Each is simply another capability verb** (`revamp/04a`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha), callable by the chat agent, by automations, and by the future platform API key. Billed per call via `WebCrawlCreditService`. +NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the **committed MVP actor** (a `maps.*` namespace, Phase 4). Beyond it, more **platform scrapers** — a family of **individual scraping endpoints** (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **examples, none committed**) — layer **on top** of the hardened fetch core once it's solid. **Each is simply another capability verb** (`04`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha). Billed per call via `WebCrawlCreditService`. - **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream. - The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling. @@ -189,7 +197,7 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta - Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename. - Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs. - Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types. -- CI + automations + positioning UI (**replaces the superseded "Pipelines UI"**): the CI experience (Tracker crafting/review-&-lock, Timeline/change views, "refresh now"), automation create/list/run-history, `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards, "NotebookLM for Competitive Intelligence" copy. +- Scraper-API + chat + positioning UI (**replaces the superseded "Pipelines UI"**): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards. ## Open items to confirm during subplanning @@ -197,11 +205,12 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta - ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with. - ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`). - ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP. -- **Revamp cross-phase forks (from `revamp/00-overview.md`, decide during subplanning):** (1) schema **review-&-lock** UX pre-frontend — pure-chat confirmation for MVP? (`05b`); (2) Timeline ORM home — `app/db.py` vs a dedicated `app/timeline/` (`05a`); (3) recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback) (`06`); (4) is `web.discover` metered or free (`04a`); (5) MCP-serve auth depth — OAuth 2.1 vs bearer (`04b`). +- **Revamp open items (decide during subplanning):** (1) the **Ongoing-Automation mechanism** (`06`, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large `web.scrape` / `maps.reviews` responses are bounded/streamed (`04`); (3) is `web.discover` metered or free (`04`); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (`05`); (5) public REST rate-limiting / abuse posture (`05`). ## Resolved decisions log -- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. +- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`intelligence_agent`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below. +- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. - **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths). - ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type. - Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu). @@ -239,17 +248,10 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta | 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) | | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default | | 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff | -| 4 | `revamp phases 4-7/04a-capabilities.md` | **CANONICAL** · drafted — capability registry + MVP verbs `web.scrape`/`web.discover` (per-platform scrapers = later drop-ins); bill at executor. **← build next** | -| 4 | `revamp phases 4-7/04b-access.md` | **CANONICAL** · drafted — generated chat/REST/MCP doors + intent router + BYO-MCP routing fix | -| 5 | `revamp phases 4-7/05a-timeline.md` | **CANONICAL** · drafted — 3-store delta Timeline (the moat) | -| 5 | `revamp phases 4-7/05b-intelligence.md` | **CANONICAL** · drafted — `Tracker` + refresh hot loop + materiality split | -| 6 | `revamp phases 4-7/06-triggers.md` | **CANONICAL** · drafted — refresh clock; recurrence = CI action on automations | -| 7 | `revamp phases 4-7/07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent | -| — | `revamp phases 4-7/00-overview.md` · `00b-pipeline-diagrams.md` | revamp map + reconciliation + end-to-end flow diagrams | -| 4 | `04a-connector-category.md` *(flat)* | ↪ **REDIRECT** — demoted to backward-compat hygiene; superseded by `revamp/04a`+`04b` | -| 4 | `04b-source-discovery.md` *(flat)* | ↪ **REDIRECT** — absorbed as the `web.discover` verb in `revamp/04a` | -| 5 | `05-pipelines-model.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05a`+`05b` (Tracker/Timeline) | -| 6 | `06-pipelines-exec.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05b` (hot loop) + `revamp/06` (triggers) | -| 7 | `07-upload-pipeline-kb.md` *(flat)* | ❌ **SUPERSEDED** — uploads populate the input-only KB (no pipeline wrapper) | +| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) | +| 4 | `04-capabilities.md` | **CANONICAL** · drafted — capability registry + verbs `web.scrape`/`web.discover`/`maps.*` → cleaned data; bill at executor. **← build next** | +| 5 | `05-access.md` | **CANONICAL** · drafted — generated REST/MCP/chat doors + intent router + BYO-MCP routing fix | +| 6 | `06-ongoing-automation.md` | **CANONICAL** · **design deferred** — chat-native "keep watching" (periodic re-invocation) | +| 7 | `07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent; "what changed" from chat history | Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases"). diff --git a/plans/backend/00b-pipeline-diagrams.md b/plans/backend/00b-pipeline-diagrams.md new file mode 100644 index 000000000..d2ffb8be7 --- /dev/null +++ b/plans/backend/00b-pipeline-diagrams.md @@ -0,0 +1,110 @@ +# Pipeline diagrams — end-to-end (scraper-APIs-first, stateless) + +> Visual companion to `00-umbrella-plan.md`. +> Phase refs: `04` Capabilities · `05` Access · `06` Ongoing-Automation (design deferred) · `07` Orchestration. + +## 1. The shape — scraper APIs + a chat agent (no state) + +```mermaid +flowchart LR + subgraph FIXED["FIXED · Phases 1-3 (shipped)"] + ACQ["Acquisition
crawler · proxy · stealth · captcha"] + MET["Metering · 03c"] + ID["Identity / API keys"] + end + + subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] + CAP["04 Capabilities
web.* · maps.*
cleaned, AI-ready output"] + ACC["05 Access
REST · MCP · chat doors"] + ORC["07 Orchestration
intelligence_agent"] + ONG["06 Ongoing-Automation
(keep-watching · DEFERRED)"] + end + + ACQ --> CAP + MET --> CAP + CAP --> ACC + ID --> ACC + ACC --> ORC + ORC -. "standing need" .-> ONG + ONG -. "re-invokes verbs" .-> CAP + + classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7; + classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee; + classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; + class CAP,ACC,ORC a; + class ACQ,MET,ID b; + class ONG d; +``` + +Memory for "what changed" = the chat history. + +## 2. Doors — one registry, three surfaces (generated) + +```mermaid +flowchart TD + REG["04 Capability registry
web.scrape · web.discover · maps.search/place/reviews"] + REG --> R["REST + API keys
(public · OSS self-host · revenue)"] + REG --> M["MCP server
(external agents · fast-follow)"] + REG --> C["Chat tools
(in-app agent, 07)"] + R --> EX["same executor · direct return · 03c billing"] + M --> EX + C --> EX +``` + +## 3. One-shot scrape (the core path) — direct call, no polling + +```mermaid +sequenceDiagram + autonumber + participant U as User / Dev + participant D as Door 05 (REST · MCP · chat) + participant EX as Verb executor 04 + participant ACQ as Acquisition / Maps actor + participant BILL as Billing (03c) + + U->>D: intent (chat) OR raw verb call (REST/MCP) + D->>D: validate input_schema · authn/authz · meter-gate + D->>EX: call executor (same for all doors) + EX->>ACQ: crawl_url / maps / search + ACQ-->>EX: cleaned, AI-ready data + EX->>BILL: charge billing_unit per success + EX-->>D: results (returned directly) + D-->>U: plain-language answer (chat) / JSON (REST/MCP) + Note over U,BILL: Stateless — nothing persists. +``` + +## 4. Intent fork (in the agent, 07) + +```mermaid +flowchart TD + U(["User speaks in natural language"]) --> R{"Intent router
(intelligence_agent prompt, 07)"} + R -->|"find / compare / what is / right now"| A["ONE-SHOT
compose verbs, answer (section 3)"] + R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING
hand to Ongoing-Automation (06)"] + R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] + Q -->|once| A + Q -->|keep watching| B + B -.->|DESIGN DEFERRED| X["06 mechanism TBD
(periodicity · delivery · context limits)"] +``` + +## 5. "What changed" — chat history is the memory + +```mermaid +flowchart LR + P["Prior tool outputs
(already in chat context)"] --> AG["intelligence_agent"] + NEW["Fresh verb call (04)"] --> AG + AG --> ANS["'Here's what's new since last time'
(reasoned from chat history)"] +``` + +## 6. Ongoing-Automation (06) — intent only, mechanism deferred + +```mermaid +flowchart LR + W["'Keep watching X'"] --> LOOP["Persistent ongoing chat
periodically re-invokes verbs (04)"] + LOOP --> OUT["results delivered into the session
(write-then-sync / stream)"] + OUT --> MEM["chat history = memory
agent reports what's new"] + LOOP -.->|OPEN| Q["periodicity driver? · delivery channel? · context-window limit?"] + classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; + class W,LOOP,OUT,MEM d; +``` + +> Section 6 is a **placeholder** — the periodic mechanism is designed separately (see `06-ongoing-automation.md`). diff --git a/plans/backend/02-rename-backend.md b/plans/backend/02-rename-backend.md index c2ec94869..cc56df3f0 100644 --- a/plans/backend/02-rename-backend.md +++ b/plans/backend/02-rename-backend.md @@ -133,7 +133,7 @@ These do not move with a symbol rename; each is decided here. - Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella. - Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred. -- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04a-connector-category.md](04a-connector-category.md)). +- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04-capabilities.md](04-capabilities.md)). - Enum VALUE migration and observability-key rename — deliberately deferred announced changes. --- diff --git a/plans/backend/03c-crawl-billing.md b/plans/backend/03c-crawl-billing.md index 131ffc64a..ae3073934 100644 --- a/plans/backend/03c-crawl-billing.md +++ b/plans/backend/03c-crawl-billing.md @@ -2,7 +2,7 @@ > **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)). -> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** (revamp [`04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, **automation/recurring**, REST, MCP, external-cron. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context. +> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** ([`04-capabilities.md`](04-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, ongoing automation, REST, MCP. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context. > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`. > Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`). diff --git a/plans/backend/04-capabilities.md b/plans/backend/04-capabilities.md new file mode 100644 index 000000000..0e47fe155 --- /dev/null +++ b/plans/backend/04-capabilities.md @@ -0,0 +1,89 @@ +# Phase 4 — Capabilities (the scraper APIs) + +> Build first; every other phase calls this registry. Sibling: `05-access.md` (the doors). +> Reuses Phases 3a/3b/3c (shipped): `WebCrawlerConnector.crawl_url -> CrawlOutcome`, the proxy/stealth/ +> captcha tiers, and the `WebCrawlCreditService` billing seam. Locate code by symbol/grep. + +## Objective + +Expose the Acquisition engine as a small set of typed, stateless scraper verbs (`web.*`, `maps.*`) that +return cleaned, AI-ready structured data. One capability = one function you call and get data back from. + +## Verb set (MVP) + +Two namespaces: `web.*` (generic crawler) and `maps.*` (Google Maps actor). Future platforms slot in as +their own namespace. + +| Verb | Input → Output (cleaned) | Executes over | Billing unit | +|------|--------------------------|---------------|--------------| +| `web.scrape(urls[])` | `[{url, status, content, metadata}]` | loop `crawl_url` | per success | +| `web.discover(query, top_k)` | `[{url, title, snippet, provider}]` | search providers | per search / free | +| `maps.search(query, location)` | `[place]` | Maps actor | per place | +| `maps.place(place_id\|url)` | `place` | Maps actor | 1 | +| `maps.reviews(place)` | `[review]` (paged) | Maps actor | per page | + +`maps.*` returns typed structured objects (`{name, rating, review_count, hours, price_level, …}`). +`web.scrape` takes a URL array. + +## Execution + +Each verb is a direct async call that returns its cleaned result. Slow verbs (large `web.scrape` arrays, +`maps.reviews` paging) are bounded/streamed at the endpoint. + +## Registry + +One entry per verb, and the single source of truth the doors (`05`) and the agent (`07`) read from: + +``` +Capability { + name # dotted, e.g. "web.scrape", "maps.search" + input_schema # Pydantic + output_schema # Pydantic (cleaned, AI-ready) + executor # async fn; wraps Acquisition / Maps actor; returns data directly + billing_unit # how the billing service charges this call +} +``` + +Adding a verb once lights it up on every door. + +## Billing + +A verb declares a `billing_unit`; charging is delegated to the billing service (`03c` first provider). + +- `web.scrape` → per `SUCCESS` (`web_crawl` unit). +- `maps.*` → new per-place / per-page unit registered with the billing service. +- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`). +- `web.discover` → per-search unit (or free). + +## Location + +New Apache-2 package `app/capabilities/` (registry, schemas, executors) — open-source and self-hostable. +Executors import the proprietary Acquisition engine and Maps extractor (`app/proprietary/…`). + +## Work items + +1. Registry package: `Capability` dataclass + registry, imported by `05` and `07`. +2. `web.scrape` executor: loop `crawl_url` over the URL array → per-URL cleaned rows; `web_crawl` unit. +3. `web.discover` executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit. +4. `maps.*` contracts: input/output schemas + executor stubs against the Maps actor. +5. Output normalization: each executor returns AI-ready structured output. +6. Billing seam: honor `billing_unit` via the billing service. + +## Tests + +- `web.scrape([a,b,c])` returns one cleaned row per URL inline; partial failures don't fail the batch. +- Each `SUCCESS` bills one `web_crawl` unit; `EMPTY`/`FAILED` free; disabling the flag no-ops. +- A door and the agent hit the same executor for a verb. +- `web.discover` returns `{url,title,snippet,provider}`; self-disables when no provider key is set. +- `maps.*` returns typed structured objects against fixtures. + +## Out of scope + +- Doors → `05`. Keep-watching mode → `06`. Agent → `07`. +- Google Maps actor build (incl. sourcing legality) — net-new, separate effort; `maps.*` are contracts. +- Additional platform namespaces; recursive site crawl; a generic `web.extract` verb. + +## Open questions + +- `web.discover` metered vs free. +- How large `web.scrape` / `maps.reviews` responses are bounded/streamed. diff --git a/plans/backend/04a-connector-category.md b/plans/backend/04a-connector-category.md deleted file mode 100644 index 7ca37c2cc..000000000 --- a/plans/backend/04a-connector-category.md +++ /dev/null @@ -1,167 +0,0 @@ -# Phase 4a — Connector taxonomy (Type-1/Type-2) + availability gating + MCP routing fix - -> ## ↪ REDIRECT / DEMOTED (2026-06-30) — superseded by the revamp -> **The canonical Phase 4 is now [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md)** (typed capability verbs + generated doors — see `00-umbrella-plan.md`). This file's connector-taxonomy work is **demoted to backward-compat hygiene**: it is **not** the core of Phase 4. Read the revamp first. -> -> What survives from this doc, and where it goes: -> - **The one genuinely useful piece — the `MCP_CONNECTOR` (BYO MCP) routing-gap fix — moves to `revamp/04b`** (the "consume-user-MCP" work). That's the actionable carry-over. -> - **Keeping legacy branded natives `MIGRATING` (off) + existing indexed KB docs searchable, and Obsidian/Circleback `DISABLED`** — still true; a small hygiene task, no longer a headline phase. -> - **`is_pipeline_eligible` is dropped** (no pipelines); **Type-1 = file/KB-input only** (Drive/Dropbox/OneDrive + uploads → input-only KB); **`WEBCRAWLER_CONNECTOR` is RETIRED** (native `scrape_webpage` tool, now a `web.scrape` capability verb). -> -> The static-registry design below is retained for reference, but do not treat "connector two-type restructure" as the Phase 4 deliverable — the deliverable is the **capability registry** in `revamp/04a`. - -> Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`. -> Sibling: `04b-source-discovery.md` (web-search repurposing). Precondition: Phases 1–2 (rename) live. - -> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename shifted them. - -## Objective - -Give connectors a first-class **Type-1 (Data Source) / Type-2 (MCP Tool)** taxonomy and an **availability** state, then gate creation, indexing, chat-subagent build, and (forward-compat) pipeline-eligibility off it — without a DB migration. Also fix the long-standing bug where generic `MCP_CONNECTOR` tools are discovered but dropped. - -The taxonomy is **fully determined by `connector_type`** (every Notion row is the same category), so it is modeled as a **static code registry**, not a per-row column (resolved decision — avoids a migration + backfill drift across ~20 connector-create sites). `is_indexable` is **kept** (it is an orthogonal per-row capability that still gates the real `/index` + periodic machinery — e.g. Notion is `is_indexable=True` yet is a Type-2 tool). - -## Locked model (MVP) - -| Bucket | `category` | `availability` | Members | Behaviour | -|--------|-----------|----------------|---------|-----------| -| Functional data sources | `DATA_SOURCE` | `AVAILABLE` | `WEBCRAWLER_CONNECTOR`, `GOOGLE_DRIVE_CONNECTOR`, `COMPOSIO_GOOGLE_DRIVE_CONNECTOR`, `ONEDRIVE_CONNECTOR`, `DROPBOX_CONNECTOR` | create OK, `/index`+periodic OK, **pipeline-eligible** | -| Functional tool | `MCP_TOOL` | `AVAILABLE` | `MCP_CONNECTOR` (generic BYO MCP server) | create OK, acts in chat, **no pipelines** | -| Deprecated branded (indexers) | `MCP_TOOL` | `MIGRATING` | `NOTION_CONNECTOR`, `GITHUB_CONNECTOR`, `CONFLUENCE_CONNECTOR`, `BOOKSTACK_CONNECTOR`, `ELASTICSEARCH_CONNECTOR` | block new create, **disable `/index`+periodic**; disable subagent **only for Notion/Confluence** (GitHub/BookStack/Elasticsearch have no subagent); keep existing rows + already-indexed KB docs searchable | -| Deprecated branded (act-only) | `MCP_TOOL` | `MIGRATING` | `SLACK_CONNECTOR`, `TEAMS_CONNECTOR`, `LINEAR_CONNECTOR`, `JIRA_CONNECTOR`, `CLICKUP_CONNECTOR`, `AIRTABLE_CONNECTOR`, `DISCORD_CONNECTOR`, `GOOGLE_GMAIL_CONNECTOR`, `GOOGLE_CALENDAR_CONNECTOR`, `LUMA_CONNECTOR`, `COMPOSIO_GMAIL_CONNECTOR`, `COMPOSIO_GOOGLE_CALENDAR_CONNECTOR` | block new create, **disable subagent** (no KB data to keep) | -| Disabled | `DATA_SOURCE` | `DISABLED` | `OBSIDIAN_CONNECTOR`, `CIRCLEBACK_CONNECTOR` | block new create + subagent; not MCP-bound (distinct from MIGRATING). Obsidian is self-hosted-only today | -| Removed → `04b` | n/a | n/a | `SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API` | enum values dropped + repurposed in `04b`. **04a treats them as `HIDDEN`** (excluded from taxonomy) so the registry is total until `04b` removes them | - -**"Migrating" UX:** existing `MIGRATING` rows remain in the DB and (for the indexers) their already-indexed documents stay searchable via the always-on `knowledge_base` subagent; only the connector's own management (re-index, periodic, live subagent) is turned off, with a "Migrating to MCP soon" status surfaced by the API. - -## Current state (cited) - -### The enum + model - -- `class SearchSourceConnectorType(StrEnum)` — `db.py:85–117` (all **30** values, incl. the 5 search APIs `:86–90`, `MCP_CONNECTOR:113`, Composio variants `:115–117`). The registry's totality test must cover all 30. -- `class SearchSourceConnector` — `db.py:1829`. Columns: `connector_type` (`:1867`), `is_indexable` (`:1868`, `Boolean default False`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`). -- Schema: `schemas/search_source_connector.py` — `is_indexable: bool` (`:16`), validator "periodic only if indexable" (`:42–45`), `Update.is_indexable` (`:66`). - -### Where indexability is currently set (per connector, scattered) - -Create routes hardcode `is_indexable` per type: `True` for Notion (`notion_add_connector_route.py:409`), Confluence (`:376`), Google Drive (`:420`), Dropbox (`:373`), OneDrive (`:380`); `False` for Slack/Teams/Discord/Gmail/Calendar/Linear/Jira/ClickUp/Airtable/Luma (their `*_add_connector_route.py`), MCP (`search_source_connectors_routes.py:2696`). Composio uses `INDEXABLE_TOOLKITS` (`composio_routes.py:289,400`, `composio_service.py:99`). `oauth_connector_base.py:59,598` defaults `True`. **This scatter is exactly why category lives in a registry, not duplicated at each create site.** - -### The `/index` dispatch (the indexable set, today) - -`index_connector_content()` — `search_source_connectors_routes.py:717` — is an `if/elif` chain on `connector_type` that dispatches to a Celery task for: Notion (`:850`), GitHub (`:861`), Confluence (`:872`), BookStack (`:885`), Google Drive (`:898`), OneDrive (`:948`), Dropbox (`:995`), Elasticsearch (`~:1050`), WebCrawler (`:1058`). **Movers** (Notion/GitHub/Confluence/BookStack/Elasticsearch) lose this branch; the file-based + WebCrawler branches stay. - -### The chat subagent maps + the routing gap - -- `multi_agent_chat/constants.py` (the chat-package root `constants.py`, **not** under `subagents/`) — `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (`:5–24`, connector_type → subagent name; **note GitHub/BookStack/Elasticsearch have NO entry** — they are index-only, no chat subagent) and `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`:26–44`, subagent → required tokens; `deliverables`/`knowledge_base` require `frozenset()` = always built). The required tokens are a **mix of connector types and doc types** (e.g. `notion`→`NOTION_CONNECTOR`, but `dropbox`→`DROPBOX_FILE`, `google_drive`→`GOOGLE_DRIVE_FILE`, `onedrive`→`ONEDRIVE_FILE`). -- Subagent exclusion: `subagents/registry.py` `get_subagents_to_exclude(available_connectors)` (`:136–152`) excludes a builder in **two** cases: (a) it is **absent from** `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`required_tokens is None`, `:145–147`) → excluded; (b) it has non-empty required tokens that don't intersect the available set (`:150–151`) → excluded. An **empty** `frozenset()` (deliverables/knowledge_base) is always kept (`:148–149`). `build_subagents(..., exclude=...)` (`:182–220`) additionally hard-skips `memory`/`research` (`:195`) and the names in `exclude`, then calls each builder with `mcp_tools=mcp.get(name)` (`:209`). `SUBAGENT_BUILDERS_BY_NAME` (`:92–112`) has **no `mcp` builder** today. **Consequence for §5:** adding an `mcp` builder *without* a `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP["mcp"]` entry would hit case (a) and exclude it forever — so §5's map entry is mandatory, not optional. -- **CRITICAL — `available_connectors` is NOT raw connector types.** `factory.py:102–105` computes `connector_types = get_available_connectors(...)` then `available_connectors = map_connectors_to_searchable_types(connector_types)`. This **shared, already-mapped** list of searchable-type tokens feeds *multiple* consumers: `get_subagents_to_exclude` (`factory.py:253`, `main_agent/middleware/stack.py:202`), the on-demand KB-search tool (`subagents/builtins/knowledge_base/tools/search_knowledge_base.py:_search_types:53–63` does `types.update(available_connectors)` at `:61–62` — this is the post-`main`-merge replacement for the now-deleted `shared/middleware/knowledge_search.py`, which used to do the same in `_resolve_search_types`), `web_search`'s live-connector filter (`shared/tools/web_search.py`), and the `deliverables` `report` tool. **Filtering this list would break legacy KB-doc searchability and web search** — see Target §4. -- `map_connectors_to_searchable_types` (`connector_searchable_types.py:65–100`) maps each configured connector type to a searchable token; **`MCP_CONNECTOR` is absent from `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`)**, so MCP connectors never produce a token in `available_connectors`. -- **The bug:** `subagents/mcp_tools/index.py` `partition_mcp_tools_by_connector` (`:55–99`) routes each MCP tool via `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS.get(connector_type)`; `MCP_CONNECTOR` has **no entry**, so it hits the `connector_agent is None` branch and is **skipped with a warning** (`:88–95`). Generic user MCP servers therefore contribute zero tools today. (The bucket key is the agent name, e.g. `"mcp"`, consumed by `build_subagents` via `mcp.get("mcp")`.) - -### Query-time searchable mapping (keeps legacy docs visible) - -`main_agent/runtime/connector_searchable_types.py` `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`, unchanged by the `main` merge) maps connector types → searchable doc-types that scope the KB-search tool (via `available_connectors` → `search_knowledge_base.py:_search_types`). The MIGRATING indexers' entries (NOTION→`NOTION_CONNECTOR`, etc.) **stay** so already-indexed docs remain searchable. (WebCrawler→`CRAWLED_URL`, file connectors→`*_FILE` stay too.) - -## Target design - -### 1. The static registry (single source of truth) - -New `app/connectors/connector_registry.py` (or `app/utils/connector_registry.py`): - -```python -class ConnectorCategory(StrEnum): - DATA_SOURCE = "DATA_SOURCE" - MCP_TOOL = "MCP_TOOL" - -class ConnectorAvailability(StrEnum): - AVAILABLE = "AVAILABLE" # usable now - MIGRATING = "MIGRATING" # branded native, "moving to MCP soon" - DISABLED = "DISABLED" # off for MVP, not MCP-bound (Obsidian/Circleback) - HIDDEN = "HIDDEN" # not a real connector in the taxonomy (search APIs, until 04b removes them) - -# connector_type -> (category, availability). Total over SearchSourceConnectorType. -CONNECTOR_REGISTRY: dict[SearchSourceConnectorType, tuple[ConnectorCategory, ConnectorAvailability]] = { ... } - -def get_category(ct) -> ConnectorCategory | None -def get_availability(ct) -> ConnectorAvailability -def is_creatable(ct) -> bool # availability == AVAILABLE -def is_pipeline_eligible(ct) -> bool # category == DATA_SOURCE and availability == AVAILABLE -def is_indexable_type(ct) -> bool # the file/web data sources that own a Celery indexer -``` - -- **Totality guard (test):** assert every `SearchSourceConnectorType` member has a registry entry, so a newly added connector can't silently fall through gating. -- Exposed on the connector read schema as **computed fields** (`category`, `availability`) so the API/frontend can label and filter without a DB column. (Promote to a column later only if Zero/SQL filtering needs it — see umbrella resolved log.) - -### 2. Create gating - -In the generic create handler `create_search_source_connector` (`search_source_connectors_routes.py:172`, `@router.post("/search-source-connectors")`) **and** the per-service add/OAuth routes (e.g. `notion_add_connector_route.py`, `luma_add_connector_route.py:51`, the Composio routes) reject when `not is_creatable(connector_type)` with a clear 4xx ("This connector is migrating to MCP and can't be added yet" / "disabled for MVP"). The existing per-type duplicate check (`:198–212`) is unaffected. Because the per-service routes each build their own connector, the gate must be applied in each (or in a shared helper they all call) — verify by grepping the `*_add_connector_route.py` set. This is the behavioural change that blocks new branded connectors; existing rows are untouched. - -### 3. Index gating - -Guard `index_connector_content` (`:717`) up front: if `not is_indexable_type(connector_type)`, return a 4xx/no-op ("indexing disabled — migrating to MCP"). This neutralizes the Notion/GitHub/Confluence/BookStack/Elasticsearch branches without deleting their (now-dead) code paths. - -**Periodic path (must also be gated).** Create-gating blocks *new* MIGRATING rows, but **existing** MIGRATING connectors with `periodic_indexing_enabled=True` already have a `next_scheduled_at` and would keep firing via the meta-scheduler Beat task that polls `next_scheduled_at` every minute: `tasks/celery_tasks/schedule_checker_task.py` (due query `next_scheduled_at <= now` `:36`, dispatch loop `:88`, `task = task_map.get(connector.connector_type)` then `.delay(...)` `:118`). The first-run trigger is `create_periodic_schedule` (`utils/periodic_scheduler.py:30`, `task.delay(...)` `:97`). **Both paths re-dispatch through the same per-type Celery tasks** (`index_crawled_urls_task`, `index_notion_pages_task`, …), so the robust single chokepoint is to **gate the Celery index tasks at entry** by `is_indexable_type(connector_type)` — this covers the manual `/index` route, the first-run trigger, and the recurring meta-scheduler pass at once (a no-op early-return for MIGRATING types). (Aside: `schedule_checker_task` already auto-disables a `LIVE_CONNECTOR_TYPES` set at `:74-77` — an existing precedent for turning periodic off per type.) The schema validator "periodic only if indexable" (`schemas/search_source_connector.py:42–45`) is unchanged and unaffected (it keys off the per-row `is_indexable` bool, not the registry). - -Note: **update** (`PUT /search-source-connectors/{id}`, `search_source_connectors_routes.py:371`) is intentionally **left allowed** for MIGRATING rows so existing users can still edit/disable them; only create + index are gated. - -### 4. Subagent gating (turns off branded chat tools) — by NAME, not by token - -> **Do NOT filter `available_connectors`.** It is a shared, already-mapped searchable-type list (see Current State). Stripping MIGRATING entries from it would (a) remove `NOTION_CONNECTOR`/etc. from the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`) → **break the very legacy-doc searchability this plan promises**, and (b) remove the live-search tokens → break `web_search` before 04b. The token list must stay intact. - -Instead, **exclude the deprecated subagents by NAME**, derived from the registry. Extend `get_subagents_to_exclude` (`registry.py:136`) so that, in addition to its current token check, it also excludes any subagent whose mapped connector type(s) are all non-`AVAILABLE`: - -- Build the reverse of `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (subagent_name → set of connector types). For each `SUBAGENT_BUILDERS_BY_NAME` entry, if it has connector type(s) and **every** one is non-`AVAILABLE` in the registry, add it to the excluded set unconditionally (regardless of tokens). -- This is the single chokepoint used by both `factory.py:253` and `stack.py:202`, and feeds `main_prompt_registry_subagent_lines(exclude)`, so the deprecated specialists disappear from both the build and the prompt. - -Net effect for MVP: - -- **Excluded (MIGRATING):** `notion`, `confluence`, `slack`, `teams`, `linear`, `jira`, `clickup`, `airtable`, `discord`, `gmail`, `calendar`, `luma`. (GitHub/BookStack/Elasticsearch have no subagent — they are handled purely by §3 index gating; their indexed docs stay searchable via the untouched token list.) -- **Kept (AVAILABLE):** `knowledge_base`, `deliverables`, `research`, `memory` (builtins), the new `mcp` subagent, **and the file-source specialists `google_drive`, `dropbox`, `onedrive`** — these map to `AVAILABLE` Type-1 connectors, so by the registry rule they are NOT deprecation-excluded; they remain token-gated as today (built only when that file connector is configured). **Decision:** keep them — a Type-1 data source can still own a chat specialist over its indexed files; the two-type split governs pipeline-eligibility + the MCP migration, not whether an AVAILABLE connector may have a subagent. (If we later want pure-ingestion file connectors with no specialist, flip them by adding a `subagent_chat` flag to the registry — out of scope here.) - -### 5. MCP_CONNECTOR routing-gap fix (makes generic Type-2 work) - -Two changes are needed — the routing map AND surfacing the token, or the subagent will still never build: - -1. **Routing map** (`multi_agent_chat/constants.py`): add `"MCP_CONNECTOR": "mcp"` to `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (so `partition_mcp_tools_by_connector` stops hitting the `connector_agent is None` skip at `mcp_tools/index.py:88–95`), and `"mcp": frozenset({"MCP_CONNECTOR"})` to `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP`. -2. **Builder**: add a generic `mcp` entry to `SUBAGENT_BUILDERS_BY_NAME` (`registry.py:92`); `build_subagents` will call it with `mcp_tools=mcp.get("mcp")` (the bucket key from step 1's map value). -3. **Surface the token** (the gap step 4's predecessor missed): `MCP_CONNECTOR` is **not** in `_CONNECTOR_TYPE_TO_SEARCHABLE`, so `available_connectors` never contains it and the `mcp` subagent's required token `{MCP_CONNECTOR}` would never intersect → it would be excluded forever. Fix by adding `"MCP_CONNECTOR": "MCP_CONNECTOR"` to `_CONNECTOR_TYPE_TO_SEARCHABLE` (`connector_searchable_types.py:22`). Side effect: `MCP_CONNECTOR` joins the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`), where it matches zero indexed docs — harmless no-op. (Alternative if the no-op is undesirable: special-case `mcp` in `get_subagents_to_exclude` to include it whenever any configured connector is an `AVAILABLE` `MCP_TOOL`, passing the raw `connector_types` alongside. The searchable-map entry is simpler and preferred.) - -### 6. Keep `is_indexable` semantics intact - -No change to the column or its validator. `is_indexable` continues to gate the real `/index`+periodic machinery for the data sources; the registry's `is_indexable_type` is a **type-level** allowlist layered on top (a row must be both an indexable type AND `is_indexable=True`). This avoids reinterpreting historical rows. - -## Work items - -1. **Registry module** + `ConnectorCategory`/`ConnectorAvailability` enums + the total `CONNECTOR_REGISTRY` map + helper predicates (incl. reverse subagent-name→types helper for §4). -2. **Schema**: add computed `category`/`availability` to the connector read schema (`schemas/search_source_connector.py`). -3. **Create gating** in the generic create handler (`:172`) + every per-service add route (reject non-`AVAILABLE`). -4. **Index gating** at the Celery index-task entry (`is_indexable_type`) — covers manual `/index`, first-run trigger, and meta-scheduler. -5. **Subagent gating by name**: extend `get_subagents_to_exclude` to also exclude subagents whose mapped connector types are all non-`AVAILABLE` (do **not** filter the shared `available_connectors` token list). -6. **MCP routing-gap fix**: `multi_agent_chat/constants.py` map entries + generic `mcp` builder in `SUBAGENT_BUILDERS_BY_NAME` + `MCP_CONNECTOR` token in `_CONNECTOR_TYPE_TO_SEARCHABLE` (so the subagent actually builds). -7. **Tests** (below). - -## Tests - -- **Registry totality**: every `SearchSourceConnectorType` has an entry (guards future additions). -- **Create gating**: creating any `MIGRATING`/`DISABLED` type → 4xx; `WEBCRAWLER`/file/`MCP_CONNECTOR` → OK; MCP multi-instance still works. -- **Index gating**: index task entry for Notion/GitHub/Confluence/BookStack/Elasticsearch → no-op; WebCrawler/GDrive/OneDrive/Dropbox → runs. Include a case for an **existing** MIGRATING row with `periodic_indexing_enabled=True` + due `next_scheduled_at` → meta-scheduler triggers a no-op (does not re-index). -- **Subagent gating by name**: with a (legacy) Notion + Slack connector configured, `get_subagents_to_exclude` excludes `notion`/`slack`; `knowledge_base`/`deliverables` always built; a configured `google_drive`/`dropbox`/`onedrive` (AVAILABLE) is **not** excluded (still token-gated). -- **No collateral damage to the token list**: gating subagents must NOT change `available_connectors` — assert the KB-search tool still resolves the `NOTION_CONNECTOR` doc type (via `_search_types`) for the legacy connector (guards against the "filter the shared list" regression). -- **MCP gap (two-part)**: (a) a configured `MCP_CONNECTOR` produces a `"mcp"` token in `map_connectors_to_searchable_types`; (b) its tools land in the `mcp` bucket and the `mcp` subagent is actually built (regression for both `mcp_tools/index.py:88–95` and the missing-token exclusion). -- **Legacy searchability**: a search space with an existing (now-MIGRATING) Notion connector + indexed docs still returns those docs via the KB-search tool (`connector_searchable_types` Notion entry unchanged). - -## Risks / trade-offs - -- **Breaks current branded chat tooling (accepted).** Deprecating act-only natives removes their live chat subagents for existing users mid-pivot. Per the umbrella's "DB migrations carry users; backend behaviour can change" posture, this is acceptable; the "migrating to MCP soon" status sets expectations. -- **Dead code paths left in place.** The Notion/GitHub/Confluence/BookStack/Elasticsearch `/index` branches + indexers remain but are gated off (not deleted) to keep the diff small and reversible; delete when the real MCP migration lands. -- **No column = not SQL/Zero-filterable.** Frontend (deferred) filters via the API's computed fields. If Zero needs server-side filtering later, promoting the registry to a denormalized column is additive. -- **Registry/`is_indexable` dual-gate.** Two truths (type-level allowlist + per-row bool) must agree; the totality test + index-gating test cover the seam. -- **Shared `available_connectors` is a footgun.** It feeds subagent-exclusion, the KB-search tool's doc-type scope, web_search, and the deliverables `report` tool simultaneously. Subagent gating is therefore done by *name* (registry-driven), never by mutating that list; the "no collateral damage" test pins this. This also removes any 04a→04b sequencing hazard (04a no longer touches the live-search tokens that 04b later retires). -- **File-source specialists kept.** `google_drive`/`dropbox`/`onedrive` stay as AVAILABLE chat specialists (decision in §4). If product wants pure-ingestion file connectors, that's an additive registry flag later. - -## Out of scope (hand-offs) - -- Search-API enum removal, key relocation, and the source-discovery endpoint → `04b`. -- Pipeline tables + actually creating pipelines from Type-1 connectors → Phases 5–7 (`is_pipeline_eligible` is defined here for them to consume). -- Real MCP migration of the branded connectors (re-point to MCP servers) → post-MVP. -- Frontend connector UI restructure → frontend umbrella. diff --git a/plans/backend/04b-source-discovery.md b/plans/backend/04b-source-discovery.md deleted file mode 100644 index 59b952ce7..000000000 --- a/plans/backend/04b-source-discovery.md +++ /dev/null @@ -1,120 +0,0 @@ -# Phase 4b — Web-search repurposing + source-discovery endpoint - -> ## ↪ REDIRECT / ABSORBED (2026-06-30) — superseded by the revamp -> **This subplan is absorbed into the canonical [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) as the `web.discover` verb.** The substance still holds — drop the 5 search `connector_type` values (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) + their paths (destructive migration), move SearXNG/Linkup/Baidu to **platform env** config, and rewire the chat `web_search` tool onto them — but the "source-discovery endpoint" is now a **capability verb** (`web.discover(query, top_k)`) in the registry, exposed identically via chat/REST/MCP (not a bespoke route), and its output feeds the agent (or a Tracker binding), **not** "a pipeline." Read `revamp/04a` (`web.discover`) as canonical; use the provider/migration detail here as implementation reference. -> -> Part of **Phase 4 — Capabilities & Access (backend)**. See `00-umbrella-plan.md`. -> Sibling (old): `04a-connector-category.md` (demoted to hygiene). Precondition: Phases 1–2 (rename) live. - -> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name; locate code by **symbol/grep**, not the absolute line numbers cited. - -## Objective - -Retire the five web-search **connector types** and re-cast the survivors as **platform-level providers** that power two things: (1) the existing chat `web_search` tool, and (2) a new **source-discovery endpoint** that, given a topic/competitor, suggests candidate URLs the user can feed into the Universal WebURL Crawler / a pipeline. - -Resolved decisions driving this: - -- **Drop all 5 search `connector_type` enum values** (`SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API`). -- **Tavily + Serper removed entirely** (no provider, no code path). -- **SearXNG / Linkup / Baidu survive as platform providers.** SearXNG is already platform/env-based; **Linkup + Baidu move from per-workspace connector `config` to platform/env config** (one app-wide key set, matching the "single provider app-wide" style from `03b`). - -## Current state (cited) - -### The 5 search types are first-class connectors today - -- Enum members `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` — `db.py:86–90` (`SERPER_API` is annotated "NOT IMPLEMENTED YET"; SearXNG is already platform-backed despite being an enum value). -- Per-workspace search methods on `ConnectorService` read keys from the connector row's `config`: `search_tavily` (`connector_service.py:481`, key `TAVILY_API_KEY:510`), `search_searxng` (`:587`), `search_baidu` (`:614`), `search_linkup` (`:1968`, key `LINKUP_API_KEY:2000`). -- Query-time routing: `connector_searchable_types.py:_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–26`) maps `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` to the web_search tool ("live search connectors"); the other connector types map to KB pre-search. - -### The web_search tool already has a platform/per-workspace split - -> **Post-`main`-merge note.** A `main` sync **consolidated web_search to a single tool**: the old research-subagent copy `subagents/builtins/research/tools/web_search.py` was **deleted**, and there is now **one** factory at `app/agents/chat/shared/tools/web_search.py`. The research subagent now imports it (`subagents/builtins/research/tools/index.py:10,24–26`). So everywhere below that said "both variants / mirror" is now **one file**. The same merge also retired the XML result blob in favor of `[n]`-citation rendering (see §3). - -`web_search` tool factory `create_web_search_tool(search_space_id, available_connectors)` — the single canonical tool at `app/agents/chat/shared/tools/web_search.py`: -- `_LIVE_SEARCH_CONNECTORS = {TAVILY_API, LINKUP_API, BAIDU_SEARCH_API}` (`:41–45`), `_LIVE_CONNECTOR_SPECS` → `ConnectorService.search_*` (`:47–51`), per-connector dispatch `_search_live_connector` (`:115–151`). The factory picks the active live connectors by intersecting `available_connectors` with `_LIVE_SEARCH_CONNECTORS` (`:163–167`). -- At call time it fans out **in parallel** to platform SearXNG (`web_search_service.is_available()` / `.search()`, `:204–214`) **plus** each active per-workspace live connector (`:216–228`), then dedupes by URL (`:242–250`). -- Platform SearXNG service: `app/services/web_search_service.py` — env-gated by `config.SEARXNG_DEFAULT_HOST`, in-process circuit breaker + Redis result cache; returns the `(result_obj, documents)` shape the tool consumes. - -### Config - -- `config.SEARXNG_DEFAULT_HOST = os.getenv("SEARXNG_DEFAULT_HOST")` — `config/__init__.py:558`. There are **no** platform env vars for Linkup/Baidu/Tavily today (their keys live in connector `config`). - -## Target design - -### 1. Platform search-provider config (env) - -Add platform env knobs alongside `SEARXNG_DEFAULT_HOST` (`config/__init__.py:~558`): - -- `LINKUP_API_KEY = os.getenv("LINKUP_API_KEY")` -- `BAIDU_SEARCH_API_KEY = os.getenv("BAIDU_SEARCH_API_KEY")` (+ the optional knobs `search_baidu` reads from connector `config` today — `BAIDU_MODEL` / `BAIDU_SEARCH_SOURCE` / `BAIDU_ENABLE_DEEP_SEARCH`, per `validators.py:507-514`; the required key is `BAIDU_API_KEY` in the per-workspace config, relocated to this platform env var — port from its current `config` reads). -- (SearXNG unchanged.) Document all three in `.env.example` as optional; each provider self-disables when its key/host is unset (mirrors `web_search_service.is_available()`). - -No Tavily/Serper env (removed). - -### 2. A platform search-providers service (consolidate the survivors) - -Introduce `app/services/web_search_service.py`-level functions (or a small `search_providers/` package) so all three survivors share one platform shape `(result_obj, documents)`: - -- `searxng.search(...)` — the existing `web_search_service.search` (unchanged). -- `linkup.search(...)` / `baidu.search(...)` — **port the bodies of `ConnectorService.search_linkup`/`search_baidu`** but source the API key from `config.*` (platform) instead of `connector.config` (per-workspace), and drop the `search_space_id`/connector lookups. -- Each is `is_available()`-gated on its env key. Tavily (`search_tavily`) is deleted. - -This removes per-workspace coupling: web search + discovery no longer depend on any connector row. - -### 3. Rewire the chat `web_search` tool (single tool) - -- Drop the `_LIVE_SEARCH_CONNECTORS` / `_LIVE_CONNECTOR_SPECS` / `_search_live_connector` mechanism in the **single** tool `shared/tools/web_search.py` (`:41–51`, `:115–151`, and the factory's live-connector intersection `:163–167`, plus the per-workspace fan-out `:216–228`). Instead, fan out to the **platform providers** that are `is_available()` (SearXNG + Linkup + Baidu), all keyless from the caller's view. -- Keep the existing parallel-gather + URL-dedupe (`:233–250`). **Note:** the `main` merge replaced the old XML result blob with `[n]`-citation rendering — the tool now builds `RenderableDocument`s (`_to_renderable_web_documents:66–112`) and returns them via `render_web_results` + `load_registry`, persisting the `citation_registry` on state (`:252–281`). Leave that render/registry path **unchanged**; only the *provider sourcing* (SearXNG+Linkup+Baidu, env-keyed) changes. -- **Call-site churn — keep the `available_connectors` parameter in the factory signature but stop using it for provider selection** (providers are now env-derived). This avoids touching every caller: `main_agent/tools/registry.py:24,39`, `subagents/builtins/research/tools/index.py:10,24–26`, and `anonymous_chat/agent.py:127` (passes `available_connectors=None` already). **`deliverables` is no longer a web_search caller** (its KB/web tooling was removed in the `main` merge) — drop it from the churn list. `search_space_id` likewise stays for logging only. (A later cleanup can remove the now-dead params.) -- Net behaviour: web search works platform-wide whenever any provider env key is set; it no longer requires the workspace to have a search connector configured (so anonymous chat gains web search too). - -### 4. Source-discovery endpoint (the new capability) - -New route (e.g. `routes/source_discovery_routes.py`, mounted under `/api/v1/workspaces/{workspace_id}/...` to match the renamed surface): `POST .../source-discovery` taking `{ query/topic, top_k }` and returning a **ranked list of candidate URLs** (url, title, snippet, provider) for the user to add to the WebURL Crawler / a pipeline. - -- Implementation = call the platform providers (§2) in parallel, dedupe by URL (reuse the tool's dedupe), and return URL-centric results (plain `{url,title,snippet,provider}` — **not** the chat tool's `[n]`-cited `render_web_results` output, which is for in-conversation citation, not URL suggestion). -- Auth: standard workspace access check (same dependency as other workspace routes). -- This is **backend-only**; the UX (a "find sources" affordance when configuring a crawler/pipeline) is deferred to the frontend umbrella. -- Optional thin reuse: the chat `web_search` `_web_search_impl` and this endpoint can share a `discover_urls(query, top_k) -> list[UrlCandidate]` core. - -### 5. Drop the 5 connector types - -- **Remove the enum members** `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` from `SearchSourceConnectorType` (`db.py:86–90`). -- **Remove their connector code paths**: `ConnectorService.search_tavily` (`:481`, delete) and the per-workspace `search_searxng` (`:587`)/`search_baidu` (`:614`)/`search_linkup` (`:1968`) (replace with the platform service §2; delete the connector-config variants that read `connector.config[...]_API_KEY`, e.g. `:510`, `:2000`). Remove the three search entries from `connector_searchable_types.py:24–26`. Remove the `*_API` entries from the connector-config validation dict (`utils/validators.py:490–514`, the 5 keys in `connector_rules`: `SERPER_API:490`/`TAVILY_API:491`/`SEARXNG_API:492-505`/`LINKUP_API:506`/`BAIDU_SEARCH_API:507-514`) and `04a`'s `HIDDEN` registry entries for these types. **Grep-guard:** no remaining references to the 5 enum names or `search_tavily` anywhere. -- **Migration (this phase is NOT migration-free):** existing connector rows of these 5 types must be handled before/with the enum change: - - Alembic migration: `DELETE FROM search_source_connectors WHERE connector_type IN ('SERPER_API','TAVILY_API','SEARXNG_API','LINKUP_API','BAIDU_SEARCH_API');` (FKs cascade; these rows only held search API keys, now relocated to env). Self-hosted operators re-add Linkup/Baidu keys via env — call out in the migration docstring + `.env.example`. - - **PG enum handling:** dropping a value from a Postgres enum type requires a type recreate. Lowest-risk option: after deleting the rows, leave the now-unused labels in the PG enum type (harmless orphans) and just remove them from the Python `StrEnum` so no new rows can use them. If a clean type is wanted, do the standard "create new enum → `ALTER COLUMN ... TYPE` with a `USING` cast → drop old" dance in the migration (heavier; only if desired). Recommend the orphan-label approach for MVP. - - Also scrub `document.document_type` / search rows that referenced `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` as a **doc type** only if such persisted docs exist (live-search results were ephemeral, not indexed — verify; expectation is none). - -## Work items - -1. **Config**: `LINKUP_API_KEY`, `BAIDU_SEARCH_API_KEY` (+ Baidu host/region) in `Config` + `.env.example`; keep `SEARXNG_DEFAULT_HOST`. -2. **Platform providers**: port Linkup/Baidu search to env-keyed platform functions sharing SearXNG's `(result_obj, documents)` shape; delete Tavily. -3. **Rewire `web_search`** (the single shared tool `shared/tools/web_search.py`) to fan out to platform providers; drop `available_connectors`/live-connector plumbing; keep dedupe + the `[n]`-citation render path. -4. **Source-discovery endpoint**: `POST .../source-discovery` → ranked URL candidates; shared `discover_urls()` core. -5. **Drop the 5 enum values** + remove `search_*` connector methods, `connector_searchable_types` search entries, validators, and `04a` HIDDEN entries. -6. **Migration**: delete the 5 connector types' rows; handle the PG enum (orphan-label approach); docstring notes the env re-keying. -7. **Tests** (below). - -## Tests - -- **Provider availability**: each of SearXNG/Linkup/Baidu self-disables when its env key/host is unset; `web_search` returns "not available" only when all are unset. -- **web_search rewire**: with `LINKUP_API_KEY` set (no search connector rows), `web_search` still returns Linkup results; results from multiple providers dedupe by URL. -- **No per-workspace dependency**: a workspace with zero connectors still gets web search + source-discovery results. -- **Source-discovery endpoint**: returns URL candidates `{url,title,snippet,provider}`; enforces workspace access; empty when no providers configured. -- **Enum removal**: `SearchSourceConnectorType` no longer has the 5 values; creating one → 422; the migration deletes existing rows; reading remaining connectors is unaffected. -- **Tavily gone**: no import/path references `search_tavily`/`TAVILY_API` remain (grep guard). - -## Risks / trade-offs - -- **Per-workspace search keys become platform-wide (accepted).** Workspaces can no longer bring their own Linkup/Baidu keys; one app-wide key set serves everyone (matches the single-provider posture). Self-hosted must move keys to env — surfaced in migration docs. -- **Destructive migration.** Deleting the 5 connector types' rows is irreversible (downgrade can recreate rows but not their secrets). Gate behind the standard backup/runbook; the data is just API-key config now living in env. -- **PG enum orphan labels.** Leaving unused enum labels is cosmetically untidy but avoids a risky type-recreate; documented as a deliberate trade-off. -- **Sequencing with 04a.** 04a marks the 5 types `HIDDEN` so the taxonomy stays total in the interim; 04b removes them. If 04b ships first, 04a's HIDDEN bucket is simply never populated — both orders are safe, but `04a → 04b` is the intended order. - -## Out of scope (hand-offs) - -- Taxonomy/gating/MCP-routing fix → `04a`. -- Using discovered URLs to actually create a pipeline/crawl → Phases 5–7 (this endpoint only *suggests* URLs). -- Source-discovery UX (the "find sources" affordance) → frontend umbrella. -- Crawl billing for any crawl the user starts from a discovered URL → already covered by `03c` (the crawl path bills regardless of how the URL was found). diff --git a/plans/backend/05-access.md b/plans/backend/05-access.md new file mode 100644 index 000000000..1da602d16 --- /dev/null +++ b/plans/backend/05-access.md @@ -0,0 +1,85 @@ +# Phase 5 — Access / Surfaces (REST · MCP · chat doors) + +> Build after `04`. Together `04 + 05` ship the scraper-API product. +> Reuses shipped infra: Identity/Tenancy, API keys, the chat agent + tool registry, the streaming layer, +> Metering (`03c`), and Rocicorp Zero reactive sync. Locate code by symbol/grep. + +## Objective + +Expose the `04` capability registry through three doors — REST + API keys, an MCP server, and chat tools — +each generated from the one registry so the I/O contract can't drift. Every door is the same thin adapter +with no business logic. + +## Adapter shape (identical on every door) + +``` +parse input → validate against verb.input_schema → authn/authz → meter-gate (03c) + → call the SAME executor (04) → serialize verb.output_schema → return cleaned data +``` + +The executor returns directly (`04`). + +## The three doors + +| Door | Who | Auth | Status | +|------|-----|------|--------| +| REST + API keys | external developers | existing API-key infra | public day one | +| MCP server | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 or bearer (at implementation) | fast-follow | +| Chat tools | the in-app agent (`07`) | existing session + workspace | seed exists (`scrape_webpage`) | + +Routes are generated, so public REST on day one costs nothing extra. + +## Chat surface + +The in-app product is the conversation: the user describes a need in plain language and the agent (`07`) +picks and fills verbs. Raw typed verbs are exposed on REST/MCP for devs and external agents. The chat +door classifies intent and routes: + +``` +"find / compare / what is / pull / right now" → ONE-SHOT → call verbs, answer now +"watch / track / notify me / keep an eye / over time" → KEEP-WATCHING → hand to the ongoing mode (06) +``` + +## MCP — two directions + +- **Serve MCP** (new door): our capabilities as a remote MCP server — Streamable-HTTP `/mcp`, stateless, + bounded/paginated outputs, read-only verbs (least privilege). +- **Consume MCP** (reuse): the BYO-`MCP_CONNECTOR` — the user's own external MCP tools inside our chat + agent. Fix its routing-map gap here. + +## Result delivery + +A chat tool call returns its data inline in the turn. Asynchronous results (e.g. the `06` ongoing mode +writing between turns) reach the client via write-then-sync / stream: the worker writes rows and the +client receives them over the existing SSE stream or a Zero-published table. + +## Work items + +1. Door generator: from the `04` registry emit (a) REST routes + models, (b) MCP tool schemas + handlers, + (c) chat tool defs + handlers. +2. REST surface: public routes + API-key auth (reuse) + `03c` meter-gate. +3. Chat tools: generalize `scrape_webpage` into the registry-backed set (direct-return). +4. Intent routing: classify one-shot vs keep-watching; hand keep-watching to `06`. +5. MCP server (fast-follow): Streamable-HTTP `/mcp`, least-privilege; auth depth at implementation. +6. Consume-MCP fix: repair the BYO-`MCP_CONNECTOR` routing-map gap. + +## Tests + +- A verb added to the registry appears on REST + MCP + chat with identical I/O. +- REST without a valid key → 401; an over-budget call → blocked by the `03c` gate; a success charges once. +- A chat scrape returns cleaned data inline in the same turn. +- "compare X and Y" → one-shot; "watch X weekly" → routed to `06`. +- A configured BYO MCP tool is reachable by the chat agent. + +## Out of scope + +- Verbs, executors, billing units → `04`. +- Keep-watching mechanism + its delivery channel → `06`. +- The CI subagent + its prompt → `07`. +- Legacy branded connectors (backward-compat hygiene, separate task). + +## Open questions + +- MCP auth depth (decide at implementation). +- Public REST rate-limiting / abuse posture. +- Async delivery channel for `06` outputs: SSE stream vs Zero-published messages table (settled with `06`). diff --git a/plans/backend/05-pipelines-model.md b/plans/backend/05-pipelines-model.md deleted file mode 100644 index 0f3075208..000000000 --- a/plans/backend/05-pipelines-model.md +++ /dev/null @@ -1,321 +0,0 @@ -# Phase 5 — Pipelines data model (tables, schemas, migration, Zero, CRUD + run routes) - -> # ❌ SUPERSEDED (2026-06-30) -> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, the CI pivot no longer uses a Pipelines concept: the KB is **input-only** (user files), the crawler/search are **platform-native tools**, connectors are **MCP**, and the recurring-fetch + run-history need is met by the existing **automations** system (`app/automations/`). There are **no `pipelines`/`pipeline_runs` tables**. This document is retained **for historical context only** — do **not** implement it. **The replacement is the canonical revamp Phase 5 — [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: the `Tracker` primitive + a 3-store delta **Timeline** (the durable CI state / moat). "Diffable run history" lives in the Timeline's `entity_changes`, **not** in `automation_runs.output`. (The earlier "Phase 5′" framing is retired.) - -> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 5). -> Precondition: Phases 1–2 (rename) live, Phase 4a (connector registry) live. Siblings ahead: `06-pipelines-exec.md` (run engine + scheduling), `07-upload-pipeline-kb.md` (uploads-as-pipeline). - -> **Implementation note (post-rename).** This phase adds **brand-new** tables, created *after* the rename, so they use the canonical names natively: table `workspaces`, column `workspace_id`, ORM class `Workspace`. Phases 1–2 are now **SHIPPED**, so the live code uses the `workspace_*` names. Existing code cited below that still shows the **pre-rename** names (`searchspaces`/`search_space_id`/`SearchSpace`) was captured pre-rename — substitute the `workspace_*` equivalent and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phase 3/4 migrations shift them). - -## Objective - -Introduce two first-class tables — `pipelines` and `pipeline_runs` — plus their enums, Pydantic schemas, one Alembic migration, the backend Zero-publication entries, and the HTTP routes (CRUD + manual-run trigger + list/get runs). **This phase is data-model + API surface only.** The actual run **engine** (invoke connector fetch → optional KB save → write run record), scheduling tick, crawl-billing wiring, and chat-agent context exposure are **Phase 6**; wiring file upload onto an "Uploads" pipeline is **Phase 7**. The manual-run endpoint here **creates a `PipelineRun` row and enqueues the Phase-6 task** (which is a stub until Phase 6 lands), so the surface is testable end-to-end without the engine. - -A **Pipeline** = a saved, runnable fetch over a Type-1 data source: it references a connector + per-pipeline config + an optional cron schedule + a KB destination (`save_to_kb` + `destination_folder_id`). A **PipelineRun** = one immutable execution record (manual / scheduled / upload), carrying status, timing, doc counts, error, an optional raw-result blob ref, and a billing idempotency field. - -## Locked model (MVP) - -| Concept | Decision | -|---------|----------| -| Tables | `pipelines` (mutable definition) + `pipeline_runs` (append-only execution record). Mirrors the `automations` / `automation_runs` split. | -| Where the ORM lives | In **`app/db.py`** alongside `Folder`/`Document`/`SearchSourceConnector`/`Log` (the other core, Zero-published, cross-referenced entities), **not** a separate package. Rationale: needs `back_populates` on `Workspace`/`User`/`SearchSourceConnector`/`Folder`, a Zero entry, and a `schemas/` Pydantic mirror — all lowest-friction in `db.py`. (The `automations` package avoided `db.py` but pays for it with string-relationships + import-time model registration; pipelines are more core, like connectors.) | -| Pipeline → connector | `connector_id` **nullable**, FK `ON DELETE CASCADE`. A connector-backed pipeline points at a Type-1 source; **`NULL` = a non-connector pipeline** (the Phase-7 "Uploads" pipeline). Eligibility (`DATA_SOURCE` + `AVAILABLE`) is enforced at create via Phase 4a's `is_pipeline_eligible` (app-level, not a DB constraint — the registry is code) **and re-checked at run time in Phase 6** (the static registry can flip a type to `MIGRATING` on deploy, stranding existing pipelines). | -| Ownership | `created_by_id` FK `ON DELETE SET NULL`, **nullable** — creator metadata only (workspace-shared, like `Folder`/`Automation`), not an owner. Billing targets the workspace owner (03c). | -| Schedule representation | `schedule_cron VARCHAR NULL` (NULL = manual-only) + `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` + `enabled BOOLEAN` + `next_scheduled_at TIMESTAMPTZ NULL` (computed by Phase 6's tick). Chosen over a bare `frequency_minutes` int to match the **automations** schedule model and reuse its cron util (`automations/triggers/builtin/schedule/cron.py`), which takes **both** a cron and a timezone (`compute_next_fire_at(cron, timezone, …)`). | -| KB destination | `save_to_kb BOOLEAN` + `destination_folder_id` FK `ON DELETE SET NULL` (nullable). `save_to_kb=true` with NULL folder ⇒ index to workspace root (`documents.folder_id` is nullable). No DB CHECK couples them (an FK `SET NULL` would violate a conditional CHECK); the pairing is validated at the API only. | -| Run status enum | `pipeline_run_status`: `pending`, `running`, `succeeded`, `failed`, `cancelled` (mirrors `automation_run_status` minus `timed_out`). | -| Run trigger enum | `pipeline_run_trigger`: `manual`, `scheduled`, `upload`. | -| Periodic coexistence | **COEXIST, do not migrate** (resolves umbrella open item §2 — see "Periodic-indexing coexistence" below). | -| Zero publication | Publish **both tables full-row** (`None`), mirroring `folders` / `search_source_connectors`. Avoids the column-list `_0_version` seam entirely; neither table has a bulky column. | -| Naming | The CI **`Pipeline`** entity (a saved, runnable fetch) is distinct from the existing **`IndexingPipelineService`** (`app/indexing_pipeline/`, the KB-ingest service). They coexist but do **not** share a code path for the MVP: Phase 6's WebURL executor reuses the crawler's own KB-write path (`index_crawled_urls`, extended with `folder_id`), **not** `IndexingPipelineService` (which is the file/Composio ingest path — and is what Phase 7's uploads flow builds on). Name routes/services `pipeline_*` (not `indexing_pipeline_*`) to keep the two legible. | - -## Current state (cited) - -### The precedent: `automations` / `automation_runs` - -The cleanest existing analogue to model against. - -- Editable parent: `Automation` — `automations/persistence/models/automation.py:24`. Pattern: `BaseModel, TimestampMixin`; `search_space_id` FK CASCADE (`:27`), `created_by_user_id` FK SET NULL (`:34`), a native enum `status` with `values_callable=lambda x: [e.value for e in x]` (`:44–54`), a JSONB `definition` (`:56`), an explicit `updated_at` with `onupdate` (`:60–66`), and `runs = relationship(..., cascade="all, delete-orphan", passive_deletes=True)` (`:76–81`). -- Append-only child: `AutomationRun` — `automations/persistence/models/run.py:20`. `automation_id` FK CASCADE (`:23`), native enum `status` default `pending` (`:37–47`), JSONB `error` (`:60`), nullable `started_at`/`finished_at` (`:62–63`). -- Enums: `automations/persistence/enums/run_status.py:8` (`RunStatus(StrEnum)` = pending/running/succeeded/failed/cancelled/timed_out). -- Migration: `alembic/versions/144_add_automation_tables.py` — the **exact template** for this phase: idempotent `CREATE TYPE ... DO $$ IF NOT EXISTS` guards (`:31–76`), `CREATE TABLE IF NOT EXISTS` with inline FKs (`:79–96`), `CREATE INDEX IF NOT EXISTS` per FK/status/created_at (`:97–111`), a **partial "due" index** for the schedule scan (`ix_automation_triggers_due`, `:144–152`), and symmetric `downgrade()` dropping indexes→tables→types (`:190–213`). -- Run routes: `automations/api/run.py` — `GET /automations/{id}/runs` (list, paginated `limit/offset`, `:13–30`) + `GET /automations/{id}/runs/{run_id}` (detail, `:33–44`). The exact shape to mirror for pipeline runs. - -### The other core tables (relationship + column conventions) - -- Base: `TimestampMixin` (gives `created_at`, `db.py:487`) + `BaseModel` (`id SERIAL PK`, `db.py:498`). Tables wanting a mutable `updated_at` declare it explicitly (see `Folder.updated_at` `db.py:1333–1339`, with `onupdate` + `index=True`). -- `SearchSourceConnector` — `db.py:1829`. `connector_type` native enum (`:1867`), `is_indexable` (`:1868`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`), `search_space_id` FK CASCADE (`:1884`), `user_id` FK CASCADE (`:1891`), `documents = relationship(...)` (`:1897`). **Pipelines FK this table** (`connector_id`). -- `Folder` — `db.py:1310`. `search_space_id` FK CASCADE (`:1321`), `created_by_id` FK SET NULL (`:1327`). **Pipelines FK this table** (`destination_folder_id`, SET NULL). -- `Document.folder_id` is nullable (root/unfiled docs exist) — confirmed by `DOCUMENT_COLS` including `folder_id` (`zero_publication.py:32`) and `Document.folder` `passive_deletes=True` (`db.py:1345`). Lets `save_to_kb` work with a NULL destination. -- `Permission` enum has `CONNECTORS_CREATE/READ/UPDATE/DELETE` (`db.py:345–348`); routes authz via `check_permission(session, auth, search_space_id, Permission.X.value, msg)` (`folders_routes.py:42–48`) with `auth = Depends(get_auth_context)` + `session = Depends(get_async_session)` (`folders_routes.py:36–37`). **Reuse the `CONNECTORS_*` permissions** for pipelines (no new permission needed for MVP). - -### Schema conventions - -- `schemas/base.py`: `IDModel` (`id:int`, `:11`) + `TimestampModel` (`created_at`, `:6`), both `ConfigDict(from_attributes=True)`. -- `schemas/search_source_connector.py`: `Base` → `Create(Base)` → `Update(BaseModel, all-Optional)` → `Read(Base, IDModel, TimestampModel)` with `model_config = ConfigDict(from_attributes=True)` and `field_validator`/`model_validator` for config consistency (`:24–56`). The shape to mirror for `schemas/pipeline.py`. - -### Zero publication mechanics - -- `zero_publication.py`: `ZERO_PUBLICATION` map (`:81–94`; line numbers post-`main`-merge, which added `automations`/`new_chat_threads` entries) is the single source of truth; `None` ⇒ publish full row (e.g. `folders`, `search_source_connectors`), a list ⇒ column subset (e.g. `AUTOMATION_RUN_COLS` `:44–53`). `apply_publication(conn)` (`:163`) reconciles via `ALTER PUBLICATION ... SET TABLE`; a migration just calls it (template: `159_publish_podcasts_to_zero.py:21–22`). `_format_table_entry` omits a table until it physically exists with all its canonical columns (`:125–152`), so the migration **must create the tables before** calling `apply_publication`. The `_0_version` allowlist (`{"documents","user","podcasts"}`, `:118`) applies **only to column-list tables** — irrelevant here since we publish full-row. - -### The meta-scheduler we coexist with (Phase-6 reuse target) - -- `tasks/celery_tasks/schedule_checker_task.py:17` — `check_periodic_schedules_task`, runs every minute, scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33–39`) and dispatches per-type Celery tasks. This is the **connector-level** periodic path and is **untouched** by Phase 5. Phase 6 adds a sibling scan over `pipelines.next_scheduled_at`. -- `utils/periodic_scheduler.py:30` — `create_periodic_schedule` (first-run trigger helper). Reference for Phase 6's pipeline scheduler. - -### Alembic head - -As of the latest `main` sync the head is **`169`** (`alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py`; chain `166→167→168→169`, sequential integer-prefixed files). By the time Phase 5 is implemented, Phase 1 (rename) and Phase 4b (search-enum drop) have each added a migration ahead of `169`. **Set `down_revision` to the then-current head** — verify with `alembic heads`; do not hardcode a number. - -## Target design - -### 1. Enums (`db.py`, near the other `StrEnum`s) - -```python -class PipelineRunStatus(StrEnum): - PENDING = "pending" - RUNNING = "running" - SUCCEEDED = "succeeded" - FAILED = "failed" - CANCELLED = "cancelled" - -class PipelineRunTrigger(StrEnum): - MANUAL = "manual" - SCHEDULED = "scheduled" - UPLOAD = "upload" -``` - -Mapped with native PG enum types `pipeline_run_status` / `pipeline_run_trigger` using the `values_callable=lambda x: [e.value for e in x]` convention (matches `Automation.status` `automation.py:46–49`, so the DB stores lowercase values, not the Python member names). - -### 2. `pipelines` table (`db.py`) - -```python -class Pipeline(BaseModel, TimestampMixin): - __tablename__ = "pipelines" - - name = Column(String(200), nullable=False, index=True) - config = Column(JSONB, nullable=False, default=dict, server_default=text("'{}'::jsonb")) # per-pipeline overrides (URL list, crawl opts, proxy override seam) - - save_to_kb = Column(Boolean, nullable=False, default=False, server_default="false") - - # Schedule (NULL cron = manual-only). next_scheduled_at is owned/written by Phase 6's tick. - schedule_cron = Column(String(120), nullable=True) - schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC") # cron util needs a tz (compute_next_fire_at) - enabled = Column(Boolean, nullable=False, default=True, server_default="true") - next_scheduled_at = Column(TIMESTAMP(timezone=True), nullable=True) - - # NULL connector_id = non-connector pipeline (Phase-7 Uploads). - connector_id = Column(Integer, ForeignKey("search_source_connectors.id", ondelete="CASCADE"), nullable=True, index=True) - destination_folder_id = Column(Integer, ForeignKey("folders.id", ondelete="SET NULL"), nullable=True, index=True) - workspace_id = Column(Integer, ForeignKey("workspaces.id", ondelete="CASCADE"), nullable=False, index=True) - # Creator metadata, NOT owner: pipelines are workspace-shared (like folders/automations), - # so member deletion must NOT nuke them — SET NULL, nullable. Billing targets the workspace - # owner (03c), not this field. Phase 6 resolves the acting user as created_by ?? workspace owner. - created_by_id = Column(UUID(as_uuid=True), ForeignKey("user.id", ondelete="SET NULL"), nullable=True, index=True) - - updated_at = Column(TIMESTAMP(timezone=True), nullable=False, - default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC), index=True) - - connector = relationship("SearchSourceConnector", back_populates="pipelines") - destination_folder = relationship("Folder") - workspace = relationship("Workspace", back_populates="pipelines") - created_by = relationship("User", back_populates="pipelines") - runs = relationship("PipelineRun", back_populates="pipeline", cascade="all, delete-orphan", passive_deletes=True) -``` - -Add the inverse `pipelines = relationship("Pipeline", back_populates=...)` on **`Workspace`**, **`User`**, and **`SearchSourceConnector`** (mirror how `SearchSpace.search_source_connectors` / `User.search_source_connectors` are declared at `db.py:1887–1894`; the `created_by`/SET-NULL side mirrors `Folder.created_by` `db.py:1344` and `Automation.created_by` `automation.py:69`). `destination_folder` is intentionally one-directional (no back_populates on `Folder`) to keep the folder model lean. - -> **Ownership decision (review).** Earlier draft used connector-style `user_id NOT NULL CASCADE`. Corrected to `created_by_id … SET NULL` because pipelines are **workspace-shared** content (closer to `Folder`/`Automation` than to the per-user `SearchSourceConnector`): removing a member must not delete the workspace's pipelines or erase run-history audit. For *connector-backed* pipelines this also avoids surprising double-cascade (the connector's own `user_id CASCADE` would already drop them on member deletion via `connector_id`); for the Phase-7 Uploads pipeline (`connector_id` NULL) it's the only thing keeping the pipeline alive after its creator leaves. - -> **FK target naming:** Phases 1–2 are **SHIPPED**, so the table is `workspaces` and the column `workspace_id` — author the FK against `workspaces` / `workspace_id`. (The earlier "if sequencing slips before the rename" caveat is now moot.) - -### 3. `pipeline_runs` table (`db.py`) - -```python -class PipelineRun(BaseModel, TimestampMixin): - __tablename__ = "pipeline_runs" - - pipeline_id = Column(Integer, ForeignKey("pipelines.id", ondelete="CASCADE"), nullable=False, index=True) - - status = Column(SQLAlchemyEnum(PipelineRunStatus, name="pipeline_run_status", - values_callable=lambda x: [e.value for e in x]), - nullable=False, default=PipelineRunStatus.PENDING, - server_default=PipelineRunStatus.PENDING.value, index=True) - trigger = Column(SQLAlchemyEnum(PipelineRunTrigger, name="pipeline_run_trigger", - values_callable=lambda x: [e.value for e in x]), nullable=False, index=True) - - # Result accounting (written by Phase 6). - documents_indexed = Column(Integer, nullable=True) - crawls_attempted = Column(Integer, nullable=True) - crawls_succeeded = Column(Integer, nullable=True) - error = Column(JSONB, nullable=True) - - # Raw fetch blob ref for save_to_kb=false runs (Phase 6 writes via file_storage). Just a key/path string. - result_blob_key = Column(String, nullable=True) - - # Crawl-billing idempotency (carry-over from 03c / umbrella Phase 6 §105): micro-USD charged for this run. - charged_micros = Column(BigInteger, nullable=True) - - started_at = Column(TIMESTAMP(timezone=True), nullable=True) - finished_at = Column(TIMESTAMP(timezone=True), nullable=True) - - pipeline = relationship("Pipeline", back_populates="runs") -``` - -These extra columns (`crawls_*`, `result_blob_key`, `charged_micros`) are added **now** so Phase 6 needs **no second migration** — they sit unused until the engine writes them. (`charged_micros` directly satisfies umbrella Phase 6 line 105's "record `charged_micros` on the `PipelineRun` for idempotency".) - -> **Out of scope (provenance):** linking individual `documents` back to the `pipeline_run` that produced them (a `documents.pipeline_run_id` column) is **not** in this phase — it would touch the existing `documents` table. Defer to Phase 6/7 if run-level provenance is wanted. - -### 4. Migration (one file, `144`-shaped) - -Chain after the then-current head. In `upgrade()`: - -1. `CREATE TYPE pipeline_run_status` + `pipeline_run_trigger` behind `DO $$ IF NOT EXISTS` guards (copy `144:31–76`). -2. `CREATE TABLE IF NOT EXISTS pipelines (...)` with inline FKs: `workspace_id → workspaces(id) ON DELETE CASCADE` (NOT NULL), `connector_id → search_source_connectors(id) ON DELETE CASCADE` (nullable), `destination_folder_id → folders(id) ON DELETE SET NULL` (nullable), `created_by_id → "user"(id) ON DELETE SET NULL` (nullable; note the quoted `"user"` table). `id SERIAL PK`, `created_at`/`updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()`, `config JSONB NOT NULL DEFAULT '{}'::jsonb`, `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'`. -3. `CREATE TABLE IF NOT EXISTS pipeline_runs (...)`. -4. Indexes (`CREATE INDEX IF NOT EXISTS`): per FK (`workspace_id`, `created_by_id`, `connector_id`, `destination_folder_id`, `pipeline_id`), plus `status`, `created_at`, `updated_at`, `name`, `trigger`. Add the **partial "due" index** for Phase 6's tick: - -```sql -CREATE INDEX IF NOT EXISTS ix_pipelines_due - ON pipelines (next_scheduled_at) - WHERE enabled = true AND next_scheduled_at IS NOT NULL; -``` - -5. `from app.zero_publication import apply_publication; apply_publication(op.get_bind())` **after** the tables exist (so `_format_table_entry` includes them). Requires the `ZERO_PUBLICATION` code change (§6) to be in the tree. - -`downgrade()`: drop indexes → `pipeline_runs` → `pipelines` → both types (symmetric to `144:190–213`). (The publication reconcile is intentionally not reversed — historical publication shapes are immutable, per `159:25–27`.) - -### 5. Pydantic schemas (`app/schemas/pipeline.py`, exported via `app/schemas/__init__.py`) - -```python -class PipelineBase(BaseModel): - name: str - connector_id: int | None = None - config: dict[str, Any] = {} - save_to_kb: bool = False - destination_folder_id: int | None = None - schedule_cron: str | None = None - schedule_timezone: str = "UTC" - enabled: bool = True - -class PipelineCreate(PipelineBase): - workspace_id: int # required on create (path/body), like FolderCreate.search_space_id - -class PipelineUpdate(BaseModel): # all-Optional, like SearchSourceConnectorUpdate - name: str | None = None - config: dict[str, Any] | None = None - save_to_kb: bool | None = None - destination_folder_id: int | None = None - schedule_cron: str | None = None - schedule_timezone: str | None = None - enabled: bool | None = None - # connector_id is immutable after create (re-create to re-point); not in Update. - -class PipelineRead(PipelineBase, IDModel, TimestampModel): - workspace_id: int - created_by_id: uuid.UUID | None = None - next_scheduled_at: datetime | None = None - updated_at: datetime - model_config = ConfigDict(from_attributes=True) - -class PipelineRunRead(IDModel, TimestampModel): - pipeline_id: int - status: PipelineRunStatus - trigger: PipelineRunTrigger - documents_indexed: int | None = None - crawls_attempted: int | None = None - crawls_succeeded: int | None = None - error: dict[str, Any] | None = None - charged_micros: int | None = None - started_at: datetime | None = None - finished_at: datetime | None = None - model_config = ConfigDict(from_attributes=True) - -class PipelineRunList(BaseModel): - items: list[PipelineRunRead] - total: int -``` - -**Validators:** - -- `schedule_cron` + `schedule_timezone` (when cron not None): validate together with the existing automations cron util `validate_cron(cron, timezone)` (`automations/triggers/builtin/schedule/cron.py:15`) so a bad expression **or** bad IANA timezone 422s at create, not at the Phase-6 tick. (`schedule_timezone` is consumed by Phase 6's `compute_next_fire_at`; default `"UTC"`.) -- The `save_to_kb` ⇄ `destination_folder_id` pairing and **connector eligibility** are validated in the **route** (they need a DB session: folder-in-workspace check + `is_pipeline_eligible(connector.connector_type)`), not in the pure schema validator — mirroring how `folders_routes` validates parent-in-search-space in the handler (`folders_routes.py:54–58`). - -### 6. Zero publication (`zero_publication.py`) - -Add to `ZERO_PUBLICATION` (`:81–94`), both **full-row**: - -```python - "pipelines": None, - "pipeline_runs": None, -``` - -Full-row (like `folders`/`search_source_connectors`) — no `_0_version` allowlist edit needed, no column-drift migrations later. (If a bulky column is ever added and must be excluded, switch that table to an explicit COLS list and handle the `_0_version` seam at `:118` then.) `verify_publication` will then expect both tables; the migration's `apply_publication` call (§4) reconciles them. - -> **Client sync is not live until the frontend lands (safe).** Publishing backend-side only makes the rows *available* to Zero; nothing syncs to a client until the (deferred) frontend Zero schema + permissions include these tables. Publishing now is harmless and matches the existing pattern (e.g. `automation_runs` was published by migration before/independent of its UI). This entry exists so the chat-agent run-history context (Phase 6) and a later Pipelines UI get push for free. - -### 7. Routes (`app/routes/pipelines_routes.py`, registered in `routes/__init__.py`) - -Follow `folders_routes` for authz (`check_permission` + `get_auth_context` + `get_async_session`) and `automations/api/run.py` for the run-list shape. Register the router in `routes/__init__.py` near `search_source_connectors_router` (`:105`). - -| Method + path | Behaviour | AuthZ | -|---------------|-----------|-------| -| `POST /pipelines` | Create. Validate: workspace access; if `connector_id`, load it, assert same workspace + `is_pipeline_eligible` (4xx otherwise); if `destination_folder_id`, assert folder in same workspace; `save_to_kb=true` allowed with NULL folder (→ root). Set `created_by_id = auth.user.id`. Persist; return `PipelineRead`. | `CONNECTORS_CREATE` | -| `GET /pipelines?workspace_id=` | List pipelines in a workspace. | `CONNECTORS_READ` | -| `GET /pipelines/{id}` | Read one. | `CONNECTORS_READ` | -| `PUT /pipelines/{id}` | Update (all-optional). Re-validate folder-in-workspace + cron if changed. | `CONNECTORS_UPDATE` | -| `DELETE /pipelines/{id}` | Delete (runs cascade). | `CONNECTORS_DELETE` | -| `POST /pipelines/{id}/run` | **Manual trigger.** Insert a `PipelineRun(status=pending, trigger=manual)`, enqueue the Phase-6 Celery task with the run id, return the run. Until Phase 6, the task is a stub (no-op/marks failed) — the row + enqueue path are what this phase delivers. (Concurrency control — block a second run while one is in flight — is deferred to Phase 6, reusing the connector indexing Redis locks `utils/indexing_locks.py`.) | `CONNECTORS_UPDATE` | -| `GET /pipelines/{id}/runs?limit=&offset=` | List runs newest-first, paginated (`limit` 1–200 default 50, `offset` ≥0) → `PipelineRunList`. Mirror `automations/api/run.py:13–30`. | `CONNECTORS_READ` | -| `GET /pipelines/{id}/runs/{run_id}` | Run detail. | `CONNECTORS_READ` | - -All handlers scope by the pipeline's `workspace_id` and 404 on cross-workspace access. A thin `PipelineService` (mirroring `automations` `RunService`) is optional; for MVP, inline session logic in the routes (as `folders_routes` does) is acceptable. - -## Periodic-indexing coexistence (resolves umbrella open item §2) - -**Decision: COEXIST for MVP — do NOT migrate connector periodic config into pipelines.** - -- The existing `SearchSourceConnector.periodic_indexing_enabled` / `next_scheduled_at` path and its meta-scheduler (`schedule_checker_task.py`) **stay fully functional and untouched** in Phases 5–6. Pipelines are **purely additive**: a new `pipelines.next_scheduled_at` scan (Phase 6) runs **alongside** the connector scan. -- **Rationale:** (a) lowest risk + smallest diff (no backfill of existing connector schedules into pipeline rows, no data migration of live schedules); (b) only the WebURL crawler is the MVP pipeline executor (Phase 6) — file sources keep using their connector-level periodic path; (c) consistent with the umbrella posture "DB migrations carry users; backend behaviour can change incrementally." -- **Known overlap (flagged for Phase 6, not solved here):** a single `WEBCRAWLER_CONNECTOR` could have BOTH connector-level periodic indexing AND a pipeline wrapping it → double crawl + **double bill**. The data model permits it; the *guard* is Phase 6's responsibility. **Recommendation for Phase 6:** when a pipeline is created/enabled over a connector, treat the **pipeline as authoritative** and set that connector's `periodic_indexing_enabled=False` (single scheduler owns each connector). Recorded here so Phase 6 implements the de-dup; Phase 5 only needs the columns to support either path. - -## Work items - -1. **Enums** `PipelineRunStatus` + `PipelineRunTrigger` in `db.py`. -2. **ORM** `Pipeline` + `PipelineRun` in `db.py`; inverse `pipelines` relationships on `Workspace`, `User`, `SearchSourceConnector`. -3. **Migration**: 2 types + 2 tables + indexes (incl. `ix_pipelines_due`) + `apply_publication`; symmetric downgrade. -4. **Schemas** `app/schemas/pipeline.py` (+ export in `schemas/__init__.py`); cron syntactic validator. -5. **Zero**: add `pipelines`/`pipeline_runs` (full-row) to `ZERO_PUBLICATION`. -6. **Routes** `pipelines_routes.py` (CRUD + `/run` + `/runs` list/detail); register in `routes/__init__.py`; reuse `CONNECTORS_*` permissions. -7. **Phase-6 task stub**: a named Celery task `run_pipeline(run_id)` in **`app/pipelines/tasks.py`** (the module Phase 6 fleshes out) that `/run` enqueues — no-op/marks the run `failed` with an "engine not implemented" error until Phase 6 fills it in (keeps the endpoint honest and testable). Register it so it's dispatchable: add `"app.pipelines.tasks"` to the Celery `include` list (`celery_app.py`). Phase 6 replaces the stub body in-place (same task name + module → `/run`'s import is stable) and adds the scheduler task + Beat entry + queue routing. -8. **Tests** (below). - -## Tests - -- **Migration round-trip**: upgrade creates both tables + both enum types + `ix_pipelines_due`; `zero_publication --verify` reports no mismatch (both tables published full-row); downgrade drops cleanly. -- **Create gating (cross-link to 04a)**: `POST /pipelines` with an `AVAILABLE` `DATA_SOURCE` connector (WebURL/GDrive/OneDrive/Dropbox) → 201; with a `MIGRATING`/`MCP_TOOL`/`DISABLED` connector → 4xx (`is_pipeline_eligible` false); with `connector_id=None` → allowed (Uploads case). **(Superseded in Phase 7: once the system-managed Uploads pipeline exists, user `POST` with `connector_id=None` is rejected 4xx and the Uploads row is auto-created on upload — see `07` §7. Update this test in Phase 7.)** -- **Folder validation**: `destination_folder_id` in another workspace → 4xx; `save_to_kb=true` with NULL folder → allowed. -- **Cron validation**: invalid `schedule_cron` → 422; NULL cron → allowed (manual-only). -- **Cascade**: deleting a connector deletes its pipelines + their runs; deleting a folder SET-NULLs `destination_folder_id` (pipeline survives); deleting a workspace removes pipelines + runs. -- **Manual run**: `POST /pipelines/{id}/run` inserts a `pending`/`manual` run and enqueues the task; `GET /pipelines/{id}/runs` returns it newest-first with correct `total`. -- **AuthZ**: cross-workspace access to any pipeline/run route → 404; missing `CONNECTORS_*` permission → 403. -- **Zero shape**: `expected_publication_shape` includes `pipelines`/`pipeline_runs` as full-row. - -## Risks / trade-offs - -- **`db.py` growth.** Adds two more models to an already-large module. Accepted for relationship/Zero/schema ergonomics + consistency with the other core tables; a later extraction to a `pipelines` package is additive. -- **Coexisting schedulers (double-bill window).** Until Phase 6 adds the de-dup guard, a connector with both its own periodic indexing and a wrapping pipeline can crawl/bill twice. Documented above; Phase 6 owns the fix. MVP exposure is small (operator-created overlap only). -- **Eligibility enforced in code, not SQL.** `is_pipeline_eligible` is a registry (04a) check at the route, so a direct DB insert could bypass it. Same trade-off 04a already accepted (no DB column for the taxonomy). -- **Pre-built unused columns.** `crawls_*` / `result_blob_key` / `charged_micros` ship empty in Phase 5. Deliberate — avoids a second migration in Phase 6. -- **`connector_id` CASCADE drops run history.** Deleting a connector erases its pipelines' audit trail. If audit retention becomes a requirement, switch to `SET NULL` + a discriminator to distinguish Uploads from orphaned (additive change). -- **`created_by_id` nullable shifts work to Phase 6.** SET-NULL preserves shared pipelines past member deletion (the goal) but means the run engine cannot assume a creator — it must resolve `created_by ?? workspace owner` for the indexing actor + billing target. Flagged in the Phase-6 handoff; harmless until then (no engine). -- **Static-registry strand.** A pipeline created over an `AVAILABLE` connector can later reference a type the registry flips to `MIGRATING` on a deploy (create-time check passed; the row persists). Phase 6 must re-check `is_pipeline_eligible` at run time and fail the run cleanly. The data model intentionally does not encode eligibility, so no migration is needed when the registry changes. - -## Out of scope (hand-offs) - -- Run **engine**, scheduling tick over `pipelines.next_scheduled_at`, crawl-billing wiring (write `charged_micros`/`crawls_*`), raw-blob persistence, and chat-agent run-history context → **Phase 6** (`06-pipelines-exec.md`). Phase 6 also owns three guards this model only *enables*: (a) **run-time eligibility re-check** (`is_pipeline_eligible` — a deploy can flip a connector type to `MIGRATING`, stranding existing pipelines → fail the run cleanly, don't crash); (b) **acting-user resolution** (`created_by_id` can be NULL after member deletion → resolve `created_by ?? workspace owner` for indexing + billing); (c) **concurrency** (Redis indexing lock per pipeline/connector). -- File upload creating/using an "Uploads" pipeline (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) + generalizing opt-in KB save → **Phase 7** (`07-upload-pipeline-kb.md`). -- Document→run provenance column → deferred (Phase 6/7 if needed). -- Frontend Pipelines UI (list/create/configure/run-history/manual run) → frontend umbrella. -- Public pay-as-you-go API over Type-1 pipelines → post-MVP (umbrella "Deferred"). diff --git a/plans/backend/06-ongoing-automation.md b/plans/backend/06-ongoing-automation.md new file mode 100644 index 000000000..efb70a32f --- /dev/null +++ b/plans/backend/06-ongoing-automation.md @@ -0,0 +1,25 @@ +# Phase 6 — Ongoing Automation (chat-native "keep watching") + +> **Design deferred — placeholder.** The mechanism is designed separately, after `04`/`05`/`07`. +> Depends on `04` (the verbs it re-invokes) and `05` (the chat surface + delivery channel). + +## Objective + +Support "keep watching": a persistent, ongoing chat where the agent periodically re-invokes scraper verbs +and drops results into the session. The agent derives what's new by reading the chat history (time-based +search over prior tool outputs in context). + +## Open design questions (resolve together) + +1. **Periodicity driver** — the existing automations schedule selector, a recurring task, or a persistent + agent loop. +2. **Delivery channel** for between-turn results — existing SSE stream vs a Zero-published messages table. +3. **Context-window limit** — how far back "what changed" can reason before summarization/compaction. +4. **Loop owner** — the `07` subagent, or a thin automation wrapper that invokes the agent. +5. **Stop / pause / cost controls** for a running watch. + +## Out of scope + +- The verbs → `04`. The chat surface + delivery → `05`. The agent playbook → `07`. + +> Next step: design the periodic mechanism here, then fill in Target design / Work items / Tests. diff --git a/plans/backend/06-pipelines-exec.md b/plans/backend/06-pipelines-exec.md deleted file mode 100644 index d847c512a..000000000 --- a/plans/backend/06-pipelines-exec.md +++ /dev/null @@ -1,350 +0,0 @@ -# Phase 6 — Pipeline execution + scheduling (run engine, crawl billing, blob, chat context) - -> # ❌ SUPERSEDED (2026-06-30) -> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no pipeline run engine or scheduler. The existing **automations** system already runs the full chat agent (with the native `web_search`/`scrape_webpage` tools) on **cron/event** triggers and persists **`automation_runs`** — which is exactly what this phase was building. Retained **for historical context only** — do **not** implement it. **Its still-relevant ideas survive in the canonical revamp:** (1) **billing crawls done outside chat** → now handled at the **capability executor** ([`revamp/04a`](revamp%20phases%204-7/04a-capabilities.md)), so recurring/automation runs meter correctly; (2) the refresh **hot loop** (crawl → diff → judge → append) → [`revamp/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md), writing the **Timeline** ([`revamp/05a`](revamp%20phases%204-7/05a-timeline.md)) — not `automation_runs.output`; (3) **recurrence + "run now"** → [`revamp/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) (reuse automations via a CI action). The `bill=False` seam and `index_crawled_urls` `folder_id`/`stats` params described below are **not needed** (no pipeline path). - -> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 6). -> Precondition: Phase 5 (`05-pipelines-model.md`) live — `pipelines` / `pipeline_runs` tables, schemas, Zero, CRUD + `/run` (currently enqueues a **stub** `run_pipeline(run_id)`). Depends on `03a-crawler-core.md` (`crawl_url` SUCCESS signal + `crawls_succeeded` counter) and `03c-crawl-billing.md` (`WebCrawlCreditService`). Sibling ahead: `07-upload-pipeline-kb.md` (uploads-as-pipeline). - -> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace` / `workspaces`. Citations below that still show `search_space_id` / `SearchSpace` / `searchspaces` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated; e.g. the `get_pipeline_runs` tool's dep key is `workspace_id`, matching the renamed deps dict). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–5 shift them). - -## Objective - -Make pipelines actually **run**. Phase 5 shipped the data model + a stub task; this phase fills in: - -1. **Run engine** — turn a `PipelineRun` row into work: resolve the pipeline + connector, re-check eligibility, acquire a lock, crawl the source (WebURL crawler = the only MVP executor), optionally save to the KB destination folder, persist a raw-result blob otherwise, write the run record (status/timing/counts/error). -2. **Crawl billing wiring** (carry-over from `03c`) — bill `crawls_succeeded` to the **workspace owner** via `WebCrawlCreditService` for **both** KB-save branches (a `save_to_kb=false` run must not crawl for free); record `charged_micros` on the run for idempotency. -3. **Scheduling** — a Celery Beat tick that fires due `pipelines.next_scheduled_at` (cron), modeled on the **automations** schedule selector (cron + `FOR UPDATE SKIP LOCKED` + self-heal). Plus the **de-dup guard** Phase 5 deferred (a pipeline over a connector disables that connector's own periodic indexing). -4. **Chat-agent context** — a read-only main-agent **tool** (`get_pipeline_runs`) exposing recent run history scoped to the active workspace (umbrella default = tool). - -This is backend-only; the Pipelines UI is deferred to the frontend umbrella. - -## Locked decisions (MVP) - -| Concept | Decision | -|---------|----------| -| MVP executor | **WebURL crawler only.** `connector.connector_type == WEBCRAWLER_CONNECTOR`. Any other Type-1 type (GDrive/OneDrive/Dropbox) → run fails cleanly with `"executor not implemented for "` (their connector-level periodic path still works; pipelines over them are a Phase-7+ concern). `connector_id IS NULL` (Phase-7 Uploads) → not run by this engine (uploads register their own run in Phase 7). | -| Run-engine shape | Mirror **automations**: thin Celery task → `run_async_celery_task` → async `execute_pipeline_run(session, run_id)` (`automations/tasks/execute_run.py:19-33`). PENDING-gated + idempotent terminal no-op (`automations/runtime/executor.py:29-30`). Status lifecycle `pending → running → succeeded/failed` with `started_at`/`finished_at` (mirrors `repository.mark_running/mark_succeeded/mark_failed`). | -| KB-save path | **Reuse `index_crawled_urls`** (the battle-tested 2-phase indexer) for `save_to_kb=true`, extended with a `folder_id` param (set on created/updated `Document`s → destination folder). `save_to_kb=true` + NULL folder → root (existing behaviour; `documents.folder_id` nullable). | -| Non-KB path | `save_to_kb=false` → a **fetch-only** crawl loop (no `Document` rows): crawl each URL, collect `{url, content, metadata}`, persist as **one JSON blob** via `file_storage` backend, store `result_blob_key` on the run. (The 2-phase indexer is `Document`-centric and cannot "not persist"; a separate small loop is cleaner than bending it.) | -| Billing ownership | **Run engine owns billing for the pipeline path** (pre-check on `len(urls)` + charge on `crawls_succeeded` + record `charged_micros`), calling the crawler with **billing suppressed** (`bill=False`). The connector `/index` + connector-periodic paths keep `03c`'s in-indexer billing (`bill=True`). This keeps `charged_micros` run-idempotency clean and bills both KB branches identically. **(03c coordination — see below.)** | -| Scheduler | New `pipeline_schedule_select` Beat task modeled on `automations/triggers/builtin/schedule/selector.py` (cron, `FOR UPDATE SKIP LOCKED`, self-heal of NULL `next_scheduled_at`), **not** the simpler connector `frequency_minutes` checker — because Phase 5 chose `schedule_cron`. Reuses the existing `croniter` util (`automations/triggers/builtin/schedule/cron.py`). | -| Schedule timezone | Cron needs a timezone (`compute_next_fire_at(cron, timezone, …)`). Phase 5's model has only `schedule_cron`. **Add `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` to `pipelines`** (small additive amendment to `05` — see "Required 05 amendment"). Matches how automations store cron **and** timezone (`schedule/selector.py:86-89`). | -| Concurrency | Per-pipeline Redis lock + (when connector-backed) the **existing connector lock** `utils/indexing_locks.py` so a pipeline run and any residual connector index can't double-crawl the same connector. | -| Chat context | A main-agent **tool** `get_pipeline_runs` (registry pattern, `main_agent/tools/registry.py:85-102`), opening its own `shielded_async_session()` (like `KnowledgeTreeMiddleware`) and scoping to the build-time `workspace_id`. Always-on middleware injection is **deferred** (token cost; tool is on-demand). | -| Code location | New `app/pipelines/` package: `engine.py` (`execute_pipeline_run`), `tasks.py` (Celery `run_pipeline` + `pipeline_schedule_select`), `scheduler.py` (the tick), `storage.py` (blob key). The Phase-5 stub `run_pipeline` is **replaced** by the real task here; the `/run` route + scheduler enqueue it. (ORM stays in `db.py` per Phase 5; only the *engine* is a package, mirroring `app/automations/`.) | - -## Current state (cited) - -### Run-engine precedent (automations) - -- **Task wrapper**: `automations/tasks/execute_run.py:19-33` — `@celery_app.task(name=..., bind=True)` → `run_async_celery_task(lambda: _impl(run_id))`; `_impl` opens `get_celery_session_maker()()` and calls `execute_run(session, run_id)`, rolling back + re-raising on failure. -- **Launch**: `automations/dispatch/launch.py:43-60` — create the child row `status=PENDING`, `session.add` + `commit` + `refresh`, then `task.apply_async(args=[run.id], time_limit=…)`. (Phase 5's `/run` already does the create+enqueue; this phase makes the task real.) -- **Executor lifecycle**: `automations/runtime/executor.py:23-75` — `load run; if run.status != PENDING: return` (idempotent terminal no-op, `:29-30`); `mark_running` + `commit` (`:46-47`); on bad snapshot → `mark_failed` + commit (`:35-44`); terminal `mark_succeeded`/`mark_failed` each `commit`. - -### The MVP executor: `webcrawler_indexer.index_crawled_urls` - -- Signature `webcrawler_indexer.py:44-53`: `(session, connector_id, search_space_id, user_id, start_date=None, end_date=None, update_last_indexed=True, on_heartbeat_callback=None) -> (int, str|None)`. -- Reads URLs from `connector.config["INITIAL_URLS"]` via `parse_webcrawler_urls` (`:118-119`). -- **Creates `Document` rows with NO `folder_id`** (`:222-239`) → today everything lands at workspace root. Updates existing docs in place. Crawls via `crawler.crawl_url(url)` (`:297`; `03a` changes this signature/return). Tracks `documents_indexed` / `documents_updated` / counts; final commit at `:432`; returns `(total_processed, error)` (`:477`). -- **No billing today** — `03c` adds the owner pre-check + `charge_credits` inside this function (`03c` §2). **This phase adds a `bill: bool = True` seam** so the pipeline path can suppress it (plus `folder_id` / `urls` params + a `stats` out-param — see "Crawler changes"; the positional return is left unchanged so the shared wrapper keeps working). -- Connector entrypoint chain (the path the pipeline engine **bypasses**, because it's notification-centric, not run-centric): `connector_tasks.py:418-454` (`index_crawled_urls_task` → `_index_crawled_urls` → `run_web_page_indexing`) → a `_run_indexing_with_notifications(...)` **call site** (e.g. the WebCrawler one at `search_source_connectors_routes.py:~2524`). The wrapper itself is **defined at `:1296`** (acquires the connector lock at `:1334`, length-unpacks the indexer return at `:1503`). - -### KB-ingest service (folder support already exists there) - -- `IndexingPipelineService` (`indexing_pipeline/indexing_pipeline_service.py:82`) already threads a destination folder via `ConnectorDocument.folder_id` (`:273-274`, `:301-302`, `:326`). **But the webcrawler indexer does not use this service** — it writes `Document`s directly. So the umbrella's "route through `IndexingPipelineService` into the destination folder" is accurate for **file/Composio** sources, **not** the crawler. For the WebURL MVP, the minimal correct move is to thread `folder_id` into `index_crawled_urls` (not refactor the crawler onto `IndexingPipelineService`). Unifying them is a post-MVP cleanup. - -### Scheduler precedent (cron) + Beat registration - -- **Selector** `automations/triggers/builtin/schedule/selector.py`: tick (`:47-65`) → self-heal NULL `next_fire_at` (`:68-98`) → claim due rows `FOR UPDATE SKIP LOCKED` + advance via `compute_next_fire_at` + set `last_fired_at` (`:101-150`) → `_start_one` launches each (`:153-183`). `_TICK_BATCH=200` cap (`:35`). -- **Cron util** `automations/triggers/builtin/schedule/cron.py`: `validate_cron(cron, timezone)` (`:15`) and `compute_next_fire_at(cron, timezone, *, after)` (`:28`, returns UTC). **Requires a timezone** → drives the 05 amendment. -- **Beat source** `automations/triggers/builtin/schedule/source.py:14-20` — a `BEAT_SCHEDULE` dict (`crontab(minute="*")`) merged into the app at `celery_app.py:323` (`**SCHEDULE_BEAT_SCHEDULE`). The app's beat dict is `celery_app.py:268-324`; the worker `include` list is `:180-198`; task routing (`index_crawled_urls → CONNECTORS_QUEUE`) is `:237-255` (`:246`). -- **Connector meta-scheduler we coexist with**: `schedule_checker_task.py:17` scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33-39`), with a Redis-lock guard `is_connector_indexing_locked` (`:93`). Untouched; the pipeline tick is a sibling. - -### Locks, storage, chat tooling - -- **Locks** `utils/indexing_locks.py`: `acquire_connector_indexing_lock(connector_id)->bool` (`:24`), `release_…` (`:37`), `is_…_locked` (`:43`), keyed `indexing:connector_lock:{id}`, TTL `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`. -- **Storage** `file_storage/service.py:39,46`: `get_storage_backend()` (from `file_storage/factory.py`) → `backend.put(key, data, content_type=…)` / `open_stream(key)` / `delete(key)`. Keys are built per-purpose (`file_storage/keys.py:11`, `build_document_file_key`). For runs we add a sibling key builder (no `document_files` row — runs aren't documents). -- **Tool registry** `main_agent/tools/registry.py:85-102` — `_MAIN_AGENT_TOOL_FACTORIES: {name: (factory, required_dep_names)}`; `build_main_agent_tools(deps, …)` resolves deps + builds (`:105-147`). Display metadata lives in `shared/tools/catalog`; the main-agent name list in `main_agent/tools/index` (per the registry docstring `:1-14`). Read-only DB pattern for a chat surface: `KnowledgeTreeMiddleware` opens `shielded_async_session()` and filters by its build-time `search_space_id` (`knowledge_tree/middleware.py:199-206`). -- **Workspace owner** (billing target): `SearchSpace.user_id` (`03c` §2.1 resolves the owner with a direct `select SearchSpace.user_id where id == search_space_id`). - -## Target design - -### 1. `execute_pipeline_run(session, run_id)` — the engine (`app/pipelines/engine.py`) - -``` -load run (pipeline + connector eager); if run is None: raise -if run.status != PENDING: return # status gate; also no-ops acks_late redelivery (executor.py:29-30) -pipeline = run.pipeline -mark_running(run): status=RUNNING, started_at=now; commit -# ^ committed BEFORE any crawl/charge → a redelivered/duplicate task sees RUNNING and returns (executor.py:46-47) - -pipeline_lock = connector_lock = False -connector = None -try: - # --- guards: fail cleanly, never crash the worker (Phase-5 hand-offs (a)/(b)) --- - if pipeline.connector_id is None: # Uploads pipeline — Phase 7, not here - return fail(run, "uploads pipeline has no fetch executor") - connector = pipeline.connector - if not is_pipeline_eligible(connector.connector_type): # 04a runtime re-check (strand guard) - return fail(run, f"connector type {connector.connector_type} no longer eligible") - if connector.connector_type != WEBCRAWLER_CONNECTOR: - return fail(run, f"executor not implemented for {connector.connector_type}") - - owner_id = workspace_owner_id(session, pipeline.workspace_id) # select Workspace.user_id (03c §2.1) - actor_user_id = str(pipeline.created_by_id or owner_id) # Phase-5 hand-off (b) - urls = parse_webcrawler_urls(pipeline.config.get("INITIAL_URLS") or connector.config.get("INITIAL_URLS")) - if not urls: - return fail(run, "no URLs to crawl") - - # --- concurrency: take both; release only what we took (Phase-5 hand-off (c)) --- - pipeline_lock = acquire_pipeline_indexing_lock(pipeline.id) - if not pipeline_lock: - return fail(run, "a run for this pipeline is already in progress") - connector_lock = acquire_connector_indexing_lock(connector.id) - if not connector_lock: - return fail(run, "connector is currently indexing; retry later") - - # --- billing pre-flight (run-level; 03c statics). len(urls) is a safe upper bound. --- - svc = WebCrawlCreditService(session) - if WebCrawlCreditService.billing_enabled() and run.charged_micros is None: - await svc.check_credits(owner_id, len(urls)) # InsufficientCreditsError → except → fail("out of crawl credit") - - # --- fetch --- - if pipeline.save_to_kb: - stats = {} # out-param sink — NON-breaking (see §3) - await index_crawled_urls(session, connector.id, pipeline.workspace_id, actor_user_id, - urls=urls, folder_id=pipeline.destination_folder_id, - bill=False, stats=stats, update_last_indexed=True) - documents_indexed = stats["documents_indexed"] - crawls_attempted = stats["crawls_attempted"] - crawls_succeeded = stats["crawls_succeeded"] - result_blob_key = None - else: - outcomes = await crawl_urls_fetch_only(urls) # list[dict] from CrawlOutcome (03a) - crawls_attempted = len(urls) - crawls_succeeded = sum(1 for o in outcomes if o["status"] == "success") - documents_indexed = 0 - result_blob_key = await persist_run_blob(run, outcomes) - - # --- charge: stamp run + add audit, THEN charge (its single commit flushes all three atomically) --- - if WebCrawlCreditService.billing_enabled() and crawls_succeeded > 0 and run.charged_micros is None: - run.charged_micros = WebCrawlCreditService.successes_to_micros(crawls_succeeded) - record_token_usage(session, usage_type="web_crawl", workspace_id=pipeline.workspace_id, - user_id=owner_id, cost_micros=run.charged_micros, - call_details={"urls": len(urls), "successes": crawls_succeeded, - "pipeline_id": pipeline.id, "run_id": run.id}, - message_id=None) # add only — does not commit (03c §2.3) - await svc.charge_credits(owner_id, crawls_succeeded) # debits + COMMITS run+audit+balance (03c §2.4) - - mark_succeeded(run): status=SUCCEEDED, finished_at=now, - documents_indexed, crawls_attempted, crawls_succeeded, result_blob_key; commit -except InsufficientCreditsError: - return fail(run, "out of crawl credit") # run is RUNNING→FAILED; no debit happened -except Exception as e: - return fail(run, {"message": str(e), "type": type(e).__name__}) -finally: - if connector_lock: release_connector_indexing_lock(connector.id) - if pipeline_lock: release_pipeline_indexing_lock(pipeline.id) -``` - -`fail(run, err)` = set `status=FAILED`, `finished_at=now`, `error={...}`, `commit`, return — so the `finally` still runs and releases only the locks actually taken. `connector.id` in `finally` is reached only when `connector_lock is True`, which implies `connector` was resolved. **Idempotency rests on the status gate**, not `charged_micros`: `mark_running` commits `RUNNING` before any crawl/charge, so an `acks_late` redelivery (`celery_app.py:228-229`) or a manual re-enqueue sees `RUNNING`/terminal and returns. `charged_micros` is the belt-and-suspenders guard reserved for a future stuck-`RUNNING` re-driver (none in MVP). - -Notes: -- **`workspace_owner_id`** = `select Workspace.user_id where id == workspace_id` (`03c` §2.1 pattern). -- **`is_pipeline_eligible`** is the 04a registry helper — re-checked here at run time (a deploy can flip a type to `MIGRATING`; the create-time check in Phase 5 doesn't protect the row forever). -- **URL source**: per-pipeline `config["INITIAL_URLS"]` overrides the connector's (the `config` JSONB "per-pipeline overrides" Phase 5 reserved); fall back to the connector's `INITIAL_URLS`. - -### 2. Celery tasks (`app/pipelines/tasks.py`) - -```python -@celery_app.task(name="run_pipeline", bind=True) -def run_pipeline(self, run_id: int) -> None: - return run_async_celery_task(lambda: _run(run_id)) # execute_run.py:19-22 shape - -async def _run(run_id: int) -> None: - async with get_celery_session_maker()() as session: - try: - await execute_pipeline_run(session, run_id) - except Exception: - logger.exception("pipeline run %d failed unexpectedly", run_id) - await session.rollback(); raise -``` - -- `run_pipeline` **replaces** the Phase-5 stub of the same name; Phase 5's `/run` route already enqueues `run_pipeline(run.id)` (no route change needed). -- **Routing/registration**: ensure `"app.pipelines.tasks"` is in the `include` list (`celery_app.py:180-198`) — Phase 5 already added it for the stub; no-op if present — and route `"run_pipeline": {"queue": CONNECTORS_QUEUE}` (it crawls — same queue as `index_crawled_urls`, `celery_app.py:246`). Give the `apply_async`/`delay` a `time_limit` (the connectors queue already has an 8h hard cap, `celery_app.py:219`). - -### 3. Crawler changes — `index_crawled_urls` (coordination with `03a`/`03c`) - -Four additive params, **all default-compatible** with the existing connector path: - -1. **`folder_id: int | None = None`** — set on each created `Document` (`:222-239`) and on the existing-doc update branch, so KB-save lands in the destination folder. Default `None` = root (today's behaviour). -2. **`urls: list[str] | None = None`** — when provided, crawl this list instead of re-reading `connector.config["INITIAL_URLS"]` (`:118-119`). **Required** for the per-pipeline `config["INITIAL_URLS"]` override (a Phase-5 `config` capability) and to keep the engine's `len(urls)` billing pre-check consistent with what's actually crawled. Default `None` = read connector config (today's behaviour). -3. **`bill: bool = True`** — gate `03c`'s in-indexer `check_credits` + `charge_credits` + `record_token_usage`. The pipeline engine calls `bill=False` and bills at the run level; the connector `/index` + periodic paths keep `bill=True`. *(Small addition to `03c`'s §2 wiring — see "Cross-plan coordination".)* -4. **`stats: dict | None = None`** — an **out-param sink**, NOT a return-shape change. When provided, the indexer populates `stats["documents_indexed"|"crawls_attempted"|"crawls_succeeded"]`. **Do NOT widen the return tuple:** the shared `_run_indexing_with_notifications` wrapper unpacks every indexer's return **by length** (`search_source_connectors_routes.py:1499-1507`: `if len(result) == 3: a,b,c = result else: a,b = result`), so a 4-tuple would raise `ValueError` for *every* connector. The 2-/3-tuple return stays exactly as-is (wrapper untouched); the engine passes a fresh `stats` dict and reads it after the call (the wrapper passes none → `None` → no-op). `03a` work-item 3 already reflects this (it exposes `crawls_succeeded` via **task metadata** only, with this `stats` sink owned by Phase 6 — not via the positional return). - -The engine **bypasses** `_run_indexing_with_notifications` (it acquires the connector lock itself at `:1334` — going through the wrapper would self-deadlock on that same lock, and the wrapper's connector-`Notification` UI is the wrong surface for a pipeline run, which has its own `PipelineRun` record). It calls `index_crawled_urls` directly with no heartbeat callback. (Consequence: pipeline runs don't emit connector-indexing notifications and aren't touched by `cleanup_stale_indexing_notifications` — fine; run status lives on `pipeline_runs`.) - -### 4. Fetch-only path + blob (`app/pipelines/engine.py` + `app/pipelines/storage.py`) - -For `save_to_kb=false`: - -```python -async def crawl_urls_fetch_only(urls): - crawler = WebCrawlerConnector() # 03a: no firecrawl ctor arg - out = [] - for url in urls: - outcome = await crawler.crawl_url(url) # 03a: CrawlOutcome(status, result, error, tier) - out.append({ - "url": url, - "status": outcome.status.value, # CrawlOutcomeStatus: success | empty | failed - "content": (outcome.result or {}).get("content", ""), - "metadata": (outcome.result or {}).get("metadata", {}), - "error": outcome.error, - }) - return out -``` - -> **03a return shape.** `03a` is **committed to the `CrawlOutcome` dataclass** (`status`/`result`/`error`/`tier`) — the tuple option was dropped precisely because this fetch-only path consumes `.status`/`.result`/`.error` as attributes. So the unpack above is the firm contract; no adaptation needed. - -```python -# storage.py -def build_pipeline_run_result_key(*, workspace_id: int, run_id: int) -> str: - return f"pipeline_runs/{workspace_id}/{run_id}/result.json" - -async def persist_run_blob(run, outcomes) -> str: - key = build_pipeline_run_result_key(workspace_id=run.pipeline.workspace_id, run_id=run.id) - data = json.dumps({"runs": outcomes}).encode() - await get_storage_backend().put(key, data, content_type="application/json") - return key # → run.result_blob_key -``` - -- One blob per run (single configured backend, like document files). `result_blob_key` (Phase 5 column) stores the key; reads resolve via `get_storage_backend()`. -- **Billing still applies** here — `crawls_succeeded` is counted from `outcomes`, so non-KB runs cost exactly the same as KB runs (umbrella line 105). This is the whole reason the run engine, not the indexer, owns pipeline billing. - -### 5. Scheduler tick (`app/pipelines/scheduler.py` + `tasks.py`) - -A near-copy of the automations selector, retargeted at `pipelines`: - -```python -@celery_app.task(name="pipeline_schedule_select") -def pipeline_schedule_select() -> None: - return run_async_celery_task(_tick) - -async def _tick(): - async with get_celery_session_maker()() as session: - now = datetime.now(UTC) - await _self_heal_null_next(session, now) # enabled + cron + next_scheduled_at IS NULL → compute - claims = await _claim_due(session, now) # enabled + next_scheduled_at <= now, FOR UPDATE SKIP LOCKED, advance - for c in claims: - await _start_run(session, c) # create PipelineRun(trigger=scheduled, status=pending) + run_pipeline.apply_async -``` - -- **`_claim_due`**: `select Pipeline where enabled AND connector_id IS NOT NULL AND schedule_cron IS NOT NULL AND next_scheduled_at IS NOT NULL AND next_scheduled_at <= now ORDER BY next_scheduled_at LIMIT 200 FOR UPDATE SKIP LOCKED` (uses Phase 5's partial `ix_pipelines_due`). The `connector_id IS NOT NULL` filter is **defense-in-depth** so a non-connector pipeline (the Phase-7 Uploads pipeline) can **never** be scheduled into perpetually-failing runs even if a cron is somehow set on it — the engine would `fail` every such run (`§1`); this stops them at the selector. (Phase 7 also guards the API; this is the belt to that suspenders.) Advance `next_scheduled_at = compute_next_fire_at(schedule_cron, schedule_timezone, after=now)`; on `InvalidCronError` → `enabled=False` (self-disable, like `selector.py:90-96,131-137`). Commit, then create the run + enqueue (mirror `_start_one` reload-after-commit, `selector.py:153-183`). -- **`_self_heal_null_next`**: backfills `next_scheduled_at` for enabled cron pipelines missing it (fresh inserts, restored rows) — so Phase 5's create can leave it NULL and the tick fills it within ≤1 min. -- **Duplicate suppression**: in `_start_run`, skip creating a scheduled run if the pipeline already has a non-terminal run (`status IN (pending, running)`) — mirrors the connector checker's in-progress guard (`schedule_checker_task.py:100-116`). Without it, a pipeline whose run outlasts its cron interval piles up `scheduled` runs that immediately `fail` on the lock (noisy, though harmless). `next_scheduled_at` is still advanced (the fire isn't retried — `catchup=False`, like the automations selector). -- **Beat entry**: add `"pipeline-schedule-select": {"task": "pipeline_schedule_select", "schedule": crontab(minute="*"), "options": {"expires": 50}}` to `celery_app.py:268-324` (default fast queue, like `check_periodic_schedules`). (`"app.pipelines.tasks"` is already in `include` from Phase 5's stub, so `pipeline_schedule_select` in the same module is auto-registered.) -- **`schedule_cron` edits** (Phase 5 `PUT`): when `schedule_cron` (or timezone) changes, **null `next_scheduled_at`** so the tick recomputes (small `PUT`-handler addition — see Required 05 amendment). NULL cron ⇒ manual-only (tick ignores it). - -### 6. De-dup guard (resolves Phase-5 deferred double-bill) - -When a pipeline that **wraps a connector** is created or enabled (Phase 5 routes), set that connector's `periodic_indexing_enabled = False` (+ `next_scheduled_at = None`) so a single scheduler owns each connector (the **pipeline is authoritative**, per Phase 5's recommendation). Two touch points: - -- Phase 6 owns the rule; implement it in the Phase-5 `POST /pipelines` and `PUT /pipelines/{id}` handlers (when `connector_id` is set and `enabled`). This is a tiny handler addition (Phase 5 explicitly handed it here). -- Belt-and-suspenders: the run engine already takes the **connector lock** during a run, so even if both scan, only one crawl proceeds at a time (the connector checker skips on a held lock, `schedule_checker_task.py:93`). - -### 7. Chat-agent run-history tool (`main_agent/tools/pipeline_runs.py`) - -A read-only tool registered in the main-agent registry (umbrella default = tool): - -```python -def create_get_pipeline_runs_tool(*, workspace_id: int) -> BaseTool: - @tool - async def get_pipeline_runs(pipeline_id: int | None = None, limit: int = 20) -> str: - """Recent pipeline run history for THIS workspace: name, status, trigger, - schedule, counts, started/finished. Read-only.""" - async with shielded_async_session() as session: # knowledge_tree/middleware.py:199 pattern - q = (select(PipelineRun, Pipeline.name, Pipeline.schedule_cron) - .join(Pipeline, PipelineRun.pipeline_id == Pipeline.id) - .where(Pipeline.workspace_id == workspace_id)) - if pipeline_id is not None: - q = q.where(Pipeline.id == pipeline_id) - rows = (await session.execute( - q.order_by(PipelineRun.created_at.desc()).limit(min(limit, 100)))).all() - return render_compact(rows) # name • status • trigger • finished_at • docs/crawls - return get_pipeline_runs -``` - -- **Registration**: add `"get_pipeline_runs": (_build_get_pipeline_runs_tool, ("workspace_id",))` to `_MAIN_AGENT_TOOL_FACTORIES` (`registry.py:85-102`) — the dep key is `workspace_id`, matching the post-rename deps dict; add the name to `main_agent/tools/index`; add display metadata to `shared/tools/catalog`. -- **Workspace scope is structural** — the tool only ever sees its build-time `workspace_id`; no cross-workspace leakage (same guarantee `KnowledgeTreeMiddleware` relies on). -- **Anonymous / no-workspace turns**: exclude the tool when there's no `workspace_id` (anonymous chat has no workspace/pipelines), the same way `KnowledgeTreeMiddleware` is CLOUD-only (`knowledge_tree/middleware.py:130-131`). The registry factory requires `workspace_id`, so just don't enable it in the anonymous tool set. -- The tool reads `pipelines`/`pipeline_runs` directly (no Zero dependency), so it works server-side immediately even though client Zero sync is dormant until the frontend lands (Phase 5 note). -- **Deferred**: an always-on `` middleware injection (à la `KnowledgeTreeMiddleware`) — adds per-turn token cost; revisit if the agent under-uses the tool. - -### 8. (Optional) blob read endpoint - -Phase 5 stores `result_blob_key` but exposes no reader. Add `GET /pipelines/{id}/runs/{run_id}/result` → stream the blob via `get_storage_backend().open_stream(run.result_blob_key)` (404 if NULL), `CONNECTORS_READ`, workspace-scoped — mirroring `file_storage/api.py:69-90`. Lets API consumers + the chat tool fetch non-KB results. (Small; include if cheap, else defer to the public-API phase.) - -## Required `05` amendment (timezone + recompute) - -Small, additive. **Items 1–3 (the `schedule_timezone` column) are already folded into `05`** (model/migration/schema/validator) — listed here for traceability. **Item 4 is Phase-6 route logic** added on top of Phase 5's handlers (Work items 5–6): - -1. *(done in `05` §2)* **Model**: `schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC")` on `Pipeline`. -2. *(done in `05` §4)* **Migration**: `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'` in `CREATE TABLE pipelines`. -3. *(done in `05` §5)* **Schema**: `schedule_timezone: str = "UTC"` on `PipelineBase` + `PipelineUpdate`; `(schedule_cron, schedule_timezone)` validated together via `validate_cron(cron, tz)` (`cron.py:15`). -4. **Routes** (Phase 6 adds to `05` §7 handlers): on `PUT` when `schedule_cron`/`schedule_timezone` changes, set `next_scheduled_at = None` (tick self-heals). On `POST`/`PUT` with `connector_id` + `enabled`, apply the §6 de-dup guard. - -(If we'd rather not touch `05`: hardcode `"UTC"` everywhere in Phase 6 and skip the column — but that loses operator-meaningful local schedules and diverges from automations. The column is the right call.) - -## Cross-plan coordination - -- **`03a`** — Phase 6 calls `WebCrawlerConnector.crawl_url(url)` directly in the fetch-only path and relies on `03a`'s `CrawlOutcome.status` (`SUCCESS`/`EMPTY`/`FAILED`, dataclass form — already locked in `03a`). `03a` work-item 3 is **already aligned**: it surfaces the `crawls_succeeded` counter via **task metadata** only (not by widening `index_crawled_urls`'s positional return — the shared `_run_indexing_with_notifications` wrapper unpacks by tuple length, `:1499-1507`). Phase 6 owns the `folder_id`/`urls`/`stats` params; `03a` owns the counter + `CrawlOutcome`. -- **`03c`** — add the **`bill: bool = True`** parameter to the in-indexer billing (so the pipeline path can suppress it). `03c`'s `WebCrawlCreditService` statics (`billing_enabled()`, `successes_to_micros(n)`, `check_credits`, `charge_credits`) are reused verbatim by the run engine. `03c`'s risk note already anticipated this: *"Phase 5's `PipelineRun` can persist `charged_micros` for stronger idempotency"* — Phase 6 is where that lands. -- **`05`** — the four amendments above; `run_pipeline` task name + `/run` enqueue already exist (stub → real); `charged_micros`/`crawls_*`/`result_blob_key` columns already exist (pre-built in Phase 5). -- **`04a`** — `is_pipeline_eligible(connector_type)` is imported for the run-time re-check. - -## Work items - -1. **`app/pipelines/` package**: `engine.py` (`execute_pipeline_run` + `mark_running/succeeded/fail` helpers + `workspace_owner_id` + `crawl_urls_fetch_only`), `tasks.py` (`run_pipeline`, `pipeline_schedule_select`), `scheduler.py` (`_tick`/`_self_heal_null_next`/`_claim_due`/`_start_run`), `storage.py` (blob key + `persist_run_blob`). -2. **Pipeline lock helpers**: add `acquire_pipeline_indexing_lock(id)` / `release_pipeline_indexing_lock(id)` to `utils/indexing_locks.py` (key `indexing:pipeline_lock:{id}`), mirroring the connector lock (`:19-40`). -3. **Crawler changes**: `index_crawled_urls` gains `folder_id` + `urls` + `bill` + a `stats` **out-param** (populates `documents_indexed`/`crawls_attempted`/`crawls_succeeded`). **No return-shape change → no caller changes** (the shared wrapper's 2-/3-tuple unpack stays valid). Coordinate with `03a`/`03c` (see "Cross-plan coordination"). -4. **Celery wiring** (`celery_app.py`): ensure `"app.pipelines.tasks"` is in `include` (added in Phase 5 for the stub); route `run_pipeline → CONNECTORS_QUEUE`; add the `pipeline-schedule-select` Beat entry. -5. **De-dup guard** in Phase-5 `POST`/`PUT /pipelines` (disable wrapped connector's periodic indexing). -6. **`05` amendment**: `schedule_timezone` column/migration/schema/validator + `next_scheduled_at` reset on cron edit. -7. **Chat tool** `get_pipeline_runs` + registry/index/catalog registration. -8. **(Optional)** `GET …/runs/{run_id}/result` blob reader. -9. **Tests** (below). - -## Tests - -- **Manual run, KB-save**: WebURL pipeline `save_to_kb=true` + `destination_folder_id` → run goes `pending→running→succeeded`, `Document`s created **in that folder**, `documents_indexed`/`crawls_*` set, `finished_at` set. -- **Manual run, non-KB**: `save_to_kb=false` → **no** `Document`s, `result_blob_key` set + blob readable, `crawls_*` set. Billing **still** charges `crawls_succeeded` (parity with KB run). -- **Billing**: enabled + sufficient → owner debited `crawls_succeeded * 1000`, `charged_micros` stamped, one `web_crawl` `TokenUsage`; **insufficient → run fails pre-crawl (FAILED), no debit**; disabled (self-hosted) → no charge, no `web_crawl` row, `charged_micros` stays NULL. -- **Idempotency / lifecycle**: re-enqueuing a `RUNNING` or terminal run → immediate no-op (`status != PENDING` gate) → **no second charge** (the primary guard; `mark_running` commits before any charge); engine never raises out of `fail()` paths (bad type / NULL connector / no URLs / ineligible / lock contention) — all land the run in `FAILED` and release locks. -- **Eligibility re-check**: pipeline over a connector whose type is now `MIGRATING` → run fails cleanly (not crash). -- **Scheduler**: a due cron pipeline → tick creates a `scheduled` run + advances `next_scheduled_at` to the next cron match (in `schedule_timezone`); NULL `next_scheduled_at` self-heals; invalid cron self-disables (`enabled=False`); manual-only (NULL cron) never fires. -- **De-dup**: creating/enabling a pipeline over a connector flips that connector's `periodic_indexing_enabled=False`; concurrent connector-checker tick skips on the held lock. -- **Concurrency**: a second run while one is in-flight → fails with "already in progress" (locks held). -- **Chat tool**: `get_pipeline_runs` returns this workspace's runs only; another workspace's runs never appear; respects `limit`. -- **Counts contract (regression-critical)**: `index_crawled_urls` keeps its existing 2-/3-tuple return (the shared `_run_indexing_with_notifications` length-unpack still works for *all* connectors); when a `stats` dict is passed it's populated with `documents_indexed`/`crawls_attempted`/`crawls_succeeded`; `bill=False` suppresses in-indexer billing; connector path (`bill=True`, no `stats`) behaves byte-for-byte as before. Add a wrapper test asserting webcrawler indexing through `_run_indexing_with_notifications` still unpacks cleanly. - -## Risks / trade-offs - -- **Billing split across two call sites** (indexer for connector path, run engine for pipeline path), unified only by `03c`'s shared statics. Accepted: it's what makes `charged_micros` run-idempotency clean and keeps both KB branches billed identically. The `bill` flag is the single seam. -- **Stuck `RUNNING` runs.** A crash *after* `mark_running` but *before* terminal leaves `status=RUNNING` forever (the `status != PENDING` gate then no-ops any redelivery — same property as automations). No double-charge (gate + `charged_micros`), but the run never completes and its locks rely on Redis TTL to free. A stale-run sweeper (à la `cleanup_stale_indexing_notifications`, `celery_app.py:279-285`) is deferred; MVP exposure is a rare worker crash. -- **Two crawl code paths** (`index_crawled_urls` for KB vs `crawl_urls_fetch_only` for non-KB). Minor duplication of the crawl loop; deliberate, since forcing the `Document`-centric 2-phase indexer into a "don't persist" mode is messier. Both share `WebCrawlerConnector.crawl_url` + the same billing math. -- **WebURL-only executor.** A pipeline over GDrive/OneDrive/Dropbox is creatable (Phase 5 lets `AVAILABLE DATA_SOURCE`) but fails at run time ("executor not implemented"). Acceptable for MVP (those keep their connector-periodic path); generalize the executor in Phase 7+. -- **De-dup guard is one-way.** Creating/enabling a pipeline disables the wrapped connector's `periodic_indexing_enabled`, but **disabling or deleting that pipeline does NOT restore it** — so a connector can end up indexed by *neither* path (data goes stale) until the user re-enables connector periodic indexing manually. MVP-acceptable (operator action); a symmetric restore-on-disable is a small follow-up. Flag in the (deferred) UI copy. -- **Blob backend assumption.** `result_blob_key` stores only the key (no backend name, unlike `document_files.storage_backend`). Fine for the single configured backend; if multi-backend ever lands, add a backend column to `pipeline_runs` (additive). -- **Non-KB blob built in memory.** `crawl_urls_fetch_only` accumulates every URL's extracted content into one JSON blob before upload — a large URL list (or huge pages) inflates worker memory. Acceptable for MVP URL counts; if it bites, stream to the backend or cap per-run URLs (a `config` knob). Also: non-KB runs have **no dedup** (every run re-crawls + re-bills all URLs) — that's inherent to "don't persist", and the user opted out of KB. -- **Lock lifetime vs run length.** The Redis locks use `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`; a crawl run longer than the TTL could let a second run start. Pre-existing property of the connector lock (the connectors queue caps at 8h); revisit the TTL if pipeline runs routinely exceed it. -- **Two minute-level Beat scans** (connector checker + pipeline tick) now run every minute. Both are indexed (`ix_pipelines_due` partial index; connector scan on its own predicate) + batch-capped (200), so cost is negligible. - -## Out of scope (hand-offs) - -- **Uploads pipeline execution** (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) → **Phase 7** (`07-upload-pipeline-kb.md`). This engine explicitly fails NULL-connector runs. -- **Non-WebURL executors** (GDrive/OneDrive/Dropbox pipelines) → Phase 7+. -- **Always-on chat context middleware** (`` injection) → deferred (tool-only for MVP). -- **Document→run provenance** (`documents.pipeline_run_id`) → deferred (touches `documents`). -- **Public pay-as-you-go API over pipelines** + **public MCP KB server** → post-MVP (umbrella "Deferred"). -- **Frontend** Pipelines UI (run-history, manual run, schedule editor) → frontend umbrella. diff --git a/plans/backend/07-orchestration.md b/plans/backend/07-orchestration.md new file mode 100644 index 000000000..09da4ba2e --- /dev/null +++ b/plans/backend/07-orchestration.md @@ -0,0 +1,73 @@ +# Phase 7 — Orchestration / Conversation (the CI-expert subagent) + +> Build last; sits atop `04`/`05` (and `06` for the ongoing mode). +> The multi-agent chat runtime (deepagents, subagent dispatch, streaming, citation middleware) is shipped. +> Net-new here = one builtin subagent + its tools + its prompt. Locate code by symbol/grep. + +## Objective + +Ship the `intelligence_agent` — a builtin CI/scraper-expert subagent that delivers the product through +plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand +standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's +new vs prior tool outputs already in context. + +## The subagent + +New builtin `subagents/builtins/intelligence_agent/`, peer to `research`/`deliverables`, packed via +`pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py` +(`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook). +The main agent delegates CI-/scraping-flavored requests to it via `description.md`. + +## The playbook (`system_prompt.md`) + +1. **Intent routing** — one-shot ("compare/find/what is") → compose verbs & answer; standing concern + ("watch/track/notify when/weekly") → hand to `06`; ambiguous → ask one question ("just once, or keep + watching?"). +2. **Verb composition** — chains `web.discover → web.scrape` and `maps.search → maps.place → maps.reviews`; + infer URLs/queries/locations from context. +3. **"What changed"** — re-invoke the verb and compare against prior tool outputs in the chat history, + summarizing what's new. + +## The toolset (`load_tools`) + +| Tool | Wraps | Billing | +|------|-------|---------| +| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04` executors (direct-return) | `03c` turn accumulator (as `scrape_webpage` does) | +| `start_watch(intent)` *(deferred)* | hands off to `06` | per re-invocation (design with `06`) | + +Capability verbs are a shared, registry-generated tool module (`04`) that `research` can also load; the +`intelligence_agent` adds the CI prompt and the `06` handoff. + +## Boundaries + +- The subagent drives the `04` verbs via tools; fetch/bypass/clean logic stays in Acquisition + `04`, + callable headless. +- Humans get the agent; REST/MCP callers call `04` verbs directly, skipping this subagent. + +## Work items + +1. Subagent scaffold: `subagents/builtins/intelligence_agent/` (`agent.py` / `tools/index.py` / + `description.md` / `system_prompt.md`), packed via `pack_subagent`. +2. CI playbook prompt: intent routing (one clarifying question), verb chains, "what changed" reasoning. +3. Shared verb tool module: registry-backed (`04`), reusable by `research`. +4. `start_watch` seam to `06` (wired when `06` is designed). +5. Router registration: `description.md` for main-agent delegation. + +## Tests + +- A CI-/scraping-flavored request routes to `intelligence_agent`; a non-CI request does not. +- "compare X and Y" composes verbs and answers in plain language, persisting nothing. +- "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`. +- A follow-up re-invokes the verb and reports new items vs prior in-context outputs. +- "watch this weekly" routes to the ongoing mode, not a one-shot answer. + +## Out of scope + +- Verbs, doors → `04`/`05`. Ongoing/periodic mechanism → `06`. +- Richer multi-step playbooks, proactive suggestions, cross-topic synthesis. +- Frontend CI surfaces → frontend umbrella. + +## Open questions + +- Whether CI one-shot scraping stays in `research` or the `intelligence_agent` owns all CI-flavored calls. +- The `start_watch` handoff shape (settled with `06`). diff --git a/plans/backend/07-upload-pipeline-kb.md b/plans/backend/07-upload-pipeline-kb.md deleted file mode 100644 index 365e1e872..000000000 --- a/plans/backend/07-upload-pipeline-kb.md +++ /dev/null @@ -1,195 +0,0 @@ -# Phase 7 — File upload as a pipeline + KB-save-secondary - -> # ❌ SUPERSEDED (2026-06-30) -> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no "Uploads pipeline" and no upload `PipelineRun`. Uploads simply populate the **input-only KB** through the existing upload routes (an upload creates a `Document` — that *is* KB membership). The umbrella's KB-input invariant ("uploads always land in the KB") is inherent and needs no pipeline wrapper. Retained **for historical context only** — do **not** implement it. **The CI-relevant successor is the "context folder" idea in [`revamp phases 4-7/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: files uploaded in a CI chat go into the KB as normal (indexed), routed to a dedicated folder, and may feed the materiality judge via KB retrieval. (Canonical Phase 7 is now **Orchestration** — [`revamp/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — unrelated to uploads.) - -> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 7) — the final backend phase. -> Precondition: Phase 5 (`05-pipelines-model.md`) live (`pipelines`/`pipeline_runs` tables, `PipelineRunTrigger.UPLOAD`, `connector_id` nullable) and Phase 6 (`06-pipelines-exec.md`) live (run engine, which **explicitly fails `connector_id IS NULL` runs**). No sibling ahead — this closes the backend umbrella. - -> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace`. Citations below that still show `search_space_id` / `SearchSpace` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–6 shift them). - -## Objective - -Close the loop on the umbrella's positioning — *"User file uploads also do a pipeline entry and register a run that files were uploaded by user; only file uploads are always saved in the knowledge base"* — by: - -1. **Uploads-as-pipeline** — give every workspace a singleton **"Uploads" pipeline** (`connector_id = NULL`, `save_to_kb = true`), lazily created, and make each upload request **register a `PipelineRun(trigger=upload)`** so uploads show up in pipeline run history (and in the Phase-6 chat `get_pipeline_runs` tool) exactly like connector fetches. -2. **KB-save-secondary (confirm + close)** — the opt-in `save_to_kb` + `destination_folder_id` machinery for *connector* pipelines already shipped in Phases 5–6; this phase records the **inverse invariant for uploads** (uploads **always** save to KB) and verifies nothing in the pivot accidentally made KB-save the *default* for connector pipelines. - -**The core design fact (drives everything):** uploads are **NOT executed by the Phase-6 run engine**. Phase 6's `execute_pipeline_run` fails any `connector_id IS NULL` run with `"uploads pipeline has no fetch executor"` (`06` §1). The existing upload routes remain the executor (they already create `Document`s + dispatch ETL); Phase 7 only **attaches an audit `PipelineRun`** to that flow and ensures the Uploads pipeline row exists. No `run_pipeline` enqueue, no engine change. - -## Locked decisions (MVP) - -| Concept | Decision | -|---------|----------| -| Uploads pipeline | **One singleton per workspace**, `connector_id = NULL`, `save_to_kb = true`, `schedule_cron = NULL` (never scheduled), `enabled = true`, `name = "Uploads"`. **Lazily get-or-created** on first upload (not at workspace creation — avoids backfilling every existing workspace). | -| Uniqueness / race-safety | A **partial unique index** `ON pipelines (workspace_id) WHERE connector_id IS NULL` makes "one Uploads pipeline per workspace" a DB invariant and lets get-or-create be race-safe via `INSERT … ON CONFLICT DO NOTHING` + re-`SELECT`. (Phase 5 already reserves `connector_id IS NULL` to mean "Uploads", so this index is exactly that semantic.) Small Phase-7 migration. | -| Run lifecycle | **Audit record, created terminal at request time.** The upload route inserts `PipelineRun(trigger=upload, status=succeeded, documents_indexed=, started_at=finished_at=now())`. Per-file ETL outcomes continue to live on `Document.status` (`pending→ready/failed`, already Zero-synced). Rationale below. | -| Why not track per-file ETL into the run | The `fileupload` path fans out **N independent per-file Celery tasks** (`task_dispatcher.py:44`) with **no batch barrier**, and accurate "which docs belong to this run" needs the **`documents.pipeline_run_id` provenance column that Phase 5 deferred** (`05` §3 "Out of scope (provenance)"). Building a barrier is out of MVP scope; the run is an *upload event* record, matching the umbrella's "register a run that files were uploaded". | -| Billing | **No crawl billing for uploads** — uploads aren't crawls. `charged_micros` / `crawls_*` stay `NULL` on upload runs. Existing ETL/embedding credit behaviour (if any) is unchanged and orthogonal (`03c` only governs `web_crawl`). | -| Folder placement | The Uploads pipeline's `destination_folder_id` stays **NULL** — it is a singleton and cannot encode per-upload folders. Folder placement remains owned by the **existing upload code** (`fileupload` → root; `folder-upload` → its created/declared folder). The run/pipeline is **descriptive**, not the folder driver (unlike connector pipelines, where the engine reads `destination_folder_id`). | -| Surfaces wired | `POST /documents/fileupload` and `POST /documents/folder-upload` (both create `Document`s). The desktop precheck route `POST /documents/folder-mtime-check` (`documents_routes.py:1533`) creates nothing and is **not** wired; `folder-unlink` / `folder-sync-finalize` are delete/sync lifecycle, also out. | -| Connector-pipeline KB default | **Unchanged from Phase 5:** `save_to_kb` defaults `False` (`05` §2 `server_default="false"`). This phase only asserts uploads set it `true`; it does **not** flip the connector default. | - -## Current state (cited) - -### The upload routes (the executors we attach to) - -- **`POST /documents/fileupload`** — `documents_routes.py:126-332`. Permission `DOCUMENTS_CREATE` (`:163-169`). Phase 1: create `Document(search_space_id, document_type=FILE, status=pending, created_by_id, …)` per file, dedup by `unique_identifier_hash` (`:217-266`), `commit` (`:278`). Phase 1.5: `store_document_file(...)` per doc (`:289-297`). Phase 2: `dispatcher.dispatch_file_processing(document_id=…)` per file (`:307-316`). Returns `document_ids` / `skipped_duplicates` (`:318-325`). **Creates no folder (root), no run record today.** `files_to_process` (`:213`) = the accepted (newly-queued) set; `skipped_duplicates` (`:214`) = already-`READY` dupes. -- **`POST /documents/folder-upload`** — `documents_routes.py:1598-1748`. Permission `DOCUMENTS_CREATE` (`:1623-1629`). Creates/reuses a root `Folder` (`:1673-1704`), writes temps, then dispatches **one** batch task `index_uploaded_folder_files_task.delay(...)` (`document_tasks.py:1403`) which has its own notification + heartbeat (`:1446-1457`) — i.e. a **real completion barrier** exists on this path (unlike `fileupload`). Returns `root_folder_id` + `file_count` (`:1743-1748`). -- **Dispatcher** — `task_dispatcher.py:12-52`: `dispatch_file_processing(...)` → `process_file_upload_with_document_task.delay(document_id=…)`, **one task per file**, fire-and-forget (no barrier). - -### Phase 5/6 hand-offs this phase consumes - -- `Pipeline.connector_id` **nullable**, `NULL = "non-connector pipeline (the Phase-7 Uploads pipeline)"` (`05` §"Pipeline → connector"; model `05` §2 `:105-106`). `PipelineRunTrigger.UPLOAD` already exists (`05` §1). `PipelineRun` has `trigger`, `status`, `documents_indexed`, `started_at`, `finished_at`, `charged_micros` (nullable) (`05` §3). -- **Phase 6 engine refuses uploads**: `if pipeline.connector_id is None: return fail(run, "uploads pipeline has no fetch executor")` (`06` §1) — confirms uploads must be recorded by the route, **not** enqueued to `run_pipeline`. -- **Chat tool** `get_pipeline_runs` (`06` §7) selects `PipelineRun` joined to `Pipeline` by `workspace_id` — so upload runs surface to the agent for free once they exist. -- **De-dup guard** (`06` §6) only touches **connector-backed** pipelines (`when connector_id is set`); the Uploads pipeline (`connector_id NULL`) is untouched by it. The **scheduler** (`06` §5 `_claim_due`) ignores the Uploads pipeline **two ways** — `schedule_cron IS NOT NULL` (NULL cron) **and** `connector_id IS NOT NULL` (the defense-in-depth filter `06` added at Phase 7's request). Both confirmed safe. - -### Document model - -- `Document` — `db.py:1348`: `document_type` (`:1352`, `DocumentType.FILE`), `unique_identifier_hash` (`:1365`), `folder_id` **nullable** (`:1389`, root/unfiled), `created_by_id` (`:1397`), plus `search_space_id`/`status`. KB membership = a `Document` row exists; uploads always create one ⇒ uploads are always "in the KB" by construction. - -## Target design - -### 1. `get_or_create_uploads_pipeline(session, workspace_id, created_by_id)` — `app/pipelines/uploads.py` - -```python -async def get_or_create_uploads_pipeline(session, *, workspace_id: int, created_by_id) -> Pipeline: - # fast path - existing = (await session.execute( - select(Pipeline).where(Pipeline.workspace_id == workspace_id, - Pipeline.connector_id.is_(None)))).scalar_one_or_none() - if existing: - return existing - # race-safe create (partial unique index ux_pipelines_uploads_per_workspace) - stmt = (pg_insert(Pipeline.__table__) - .values(workspace_id=workspace_id, connector_id=None, name="Uploads", - save_to_kb=True, enabled=True, schedule_cron=None, - created_by_id=created_by_id, config={}) - .on_conflict_do_nothing(index_elements=[Pipeline.workspace_id], - index_where=Pipeline.connector_id.is_(None))) - await session.execute(stmt) - await session.flush() - return (await session.execute( - select(Pipeline).where(Pipeline.workspace_id == workspace_id, - Pipeline.connector_id.is_(None)))).scalar_one() -``` - -- Lazy (first upload), idempotent, race-safe. `created_by_id` = the uploading user (becomes the pipeline's creator metadata; `SET NULL` on member deletion per Phase 5). -- Lives in the `app/pipelines/` package (created in Phase 6) so the upload route imports it without a route↔route dependency. -- **Implementation gotcha — core insert bypasses ORM defaults.** `Pipeline.updated_at`/`created_at` use **Python-side** `default=`/`onupdate=` callables (`05` §2 `:114-115`), which a core `pg_insert` does **not** fire. The `.values(...)` must set `created_at`/`updated_at` explicitly (e.g. `now`) — or implement get-or-create as **ORM-insert + `flush` inside a `session.begin_nested()` SAVEPOINT, catching `IntegrityError` → rollback savepoint → re-`SELECT`** (avoids the core/ORM default mismatch entirely; the SAVEPOINT keeps the route's outer transaction intact). Either is fine; the SAVEPOINT form is closer to the ORM conventions used elsewhere. - -### 2. Register an upload run (shared helper) — `app/pipelines/uploads.py` - -```python -async def record_upload_run(session, *, workspace_id, created_by_id, documents_indexed: int) -> PipelineRun: - pipeline = await get_or_create_uploads_pipeline(session, workspace_id=workspace_id, - created_by_id=created_by_id) - now = datetime.now(UTC) - run = PipelineRun(pipeline_id=pipeline.id, trigger=PipelineRunTrigger.UPLOAD, - status=PipelineRunStatus.SUCCEEDED, - documents_indexed=documents_indexed, started_at=now, finished_at=now) - session.add(run) - return run # caller commits (folds into the route's existing commit) -``` - -- **Terminal-at-creation** (locked decision): records the upload event with the accepted file count. `crawls_*` / `charged_micros` left NULL (not a crawl). -- Caller does **not** await any Celery task — the run is purely the route's own DB write, folded into the route's existing transaction (no extra round-trips, no new failure mode that could block the upload). - -### 3. Wire `POST /documents/fileupload` - -After Phase 2 dispatch (`documents_routes.py:307-316`), before building the response, add: - -```python -if files_to_process: # only when something was actually queued - await record_upload_run(session, workspace_id=workspace_id, - created_by_id=str(user.id), - documents_indexed=len(files_to_process)) - await session.commit() -``` - -- Count = `len(files_to_process)` (the newly-created/re-queued docs), **excluding** `skipped_duplicates` (already-READY no-ops). An all-duplicate upload records **no** run (nothing ingested). -- **Best-effort, non-blocking — and the try/except is mandatory, not cosmetic.** The route has an **outer `except Exception` → `session.rollback()` → HTTP 500** (`documents_routes.py:328-332`). If `record_upload_run` raised unguarded, it would hit that handler and **500 an upload whose documents are already committed + dispatched**. So the call MUST be wrapped in its **own inner `try/except`** that swallows: on failure `await session.rollback()` (discards only the run insert — documents were committed at `:278`/`:304`) + `logger.warning(...)`, then fall through to the response. A run-record write must **never** fail the upload. (Mirrors the `store_document_file` best-effort pattern at `:298-303`.) - -### 4. Wire `POST /documents/folder-upload` - -Folder-upload dispatches one batch task, so record the run at request time with the file count (`documents_routes.py` before the `return` at `:1743`): - -```python -await record_upload_run(session, workspace_id=workspace_id, - created_by_id=str(user.id), documents_indexed=len(files)) -await session.commit() # best-effort / logged, same as §3 -``` - -- Uses `len(files)` (folder-upload doesn't pre-dedup in the route; dedup happens inside the batch task). -- **Optional richer accounting (deferred):** because this path *has* a barrier (`index_uploaded_folder_files_task`), a later enhancement could create the run `pending` here and flip it `succeeded`/`failed` from inside the task (it already manages a notification lifecycle). MVP keeps it uniform with `fileupload` (terminal-at-request). Flagged, not built. - -### 5. Migration (one small file) — partial unique index - -```sql -CREATE UNIQUE INDEX IF NOT EXISTS ux_pipelines_uploads_per_workspace - ON pipelines (workspace_id) WHERE connector_id IS NULL; -``` - -- Chain after the then-current head (verify `alembic heads`). Enforces the singleton + powers `ON CONFLICT`. `downgrade()` drops it. -- **Pre-flight (safety):** if any workspace somehow already has >1 `connector_id IS NULL` pipeline (shouldn't, pre-Phase-7), the `CREATE UNIQUE INDEX` fails — the migration docstring notes a dedup step (keep the oldest, delete extras) just in case. Expectation: none exist (uploads pipelines are introduced *here*). - -### 6. KB-save-secondary verification (no new code, a guard) - -- Uploads pipeline hardcodes `save_to_kb=True` (§1) — uploads always land in the KB (a `Document` row), matching the umbrella invariant. -- Confirm Phase 5's connector-pipeline default is still `save_to_kb=False` (`05` §2) — i.e. KB-save is **opt-in for connectors, mandatory for uploads**. Add a test asserting both defaults so a future change can't silently make connector KB-save the default. - -### 7. Guard the generic Phase-5 CRUD against the system Uploads pipeline (the gap) - -The Uploads pipeline is a **system-managed** row (`connector_id IS NULL`, auto-created by §1). Phase 5's generic `/pipelines` CRUD doesn't know it's special, which opens three footguns — Phase 7 closes them: - -| Phase-5 endpoint | Problem if left generic | Phase-7 guard | -|------------------|-------------------------|---------------| -| `POST /pipelines` | Phase 5 **allows `connector_id=None`** (`05` §7 create + test). A user could hand-create a rogue/duplicate non-connector pipeline. | **Reject `connector_id is None` with 4xx** — non-connector pipelines are system-managed (auto-created on upload), not user-created. This **supersedes** Phase 5's "`connector_id=None` → allowed" create test (update it). The partial unique index (§5) is the DB backstop. | -| `POST /pipelines/{id}/run` | Manual-running the Uploads pipeline enqueues `run_pipeline`, which the Phase-6 engine **fails** (`"uploads pipeline has no fetch executor"`) → a confusing **FAILED** manual run every click. | **Reject `connector_id IS NULL` with 4xx** ("uploads are triggered by uploading files, not a manual run") — before creating the run row / enqueuing. | -| `PUT /pipelines/{id}` | Setting `schedule_cron` on the Uploads pipeline makes the scheduler eligible-by-cron → **perpetually-failing scheduled runs** (engine refuses null-connector). Editing `save_to_kb`/`destination_folder_id` is silently **ineffective** (uploads are route-driven, not engine-driven). | **Reject schedule edits (`schedule_cron`/`schedule_timezone`) on `connector_id IS NULL` pipelines with 4xx.** (`connector_id` is already immutable per Phase 5.) Optionally reject all PUTs to the Uploads pipeline; minimally block the scheduling fields. Belt-and-suspenders: Phase 6's `_claim_due` also filters `connector_id IS NOT NULL` (`06` §5). | -| `DELETE /pipelines/{id}` | Deleting the Uploads pipeline cascades its run history. | **Allowed** (MVP): it is **lazily re-created** on the next upload (§1); only past upload-run history is lost. Note in UI copy later; an optional hard block is a small follow-up. | - -Net: after Phase 7, the only non-connector pipeline that can exist is the **system Uploads pipeline**, it is never user-created / manually-run / scheduled, and the scheduler ignores it two ways (NULL cron *and* the `connector_id IS NOT NULL` filter). - -## Work items - -1. **`app/pipelines/uploads.py`**: `get_or_create_uploads_pipeline` (race-safe via `ON CONFLICT`) + `record_upload_run` (terminal upload run). -2. **Migration**: partial unique index `ux_pipelines_uploads_per_workspace ON pipelines(workspace_id) WHERE connector_id IS NULL`; docstring dedup pre-flight note; `downgrade` drops it. -3. **Wire `fileupload`** (`documents_routes.py:126-332`): call `record_upload_run(documents_indexed=len(files_to_process))` after dispatch, in an **inner** best-effort try/except (must not reach the route's outer 500 handler), only when `files_to_process` non-empty. -4. **Wire `folder-upload`** (`documents_routes.py:1598-1748`): call `record_upload_run(documents_indexed=len(files))` before the response, same inner best-effort guard. -5. **Guard the generic CRUD** (§7): `POST /pipelines` rejects `connector_id is None`; `/run` rejects `connector_id IS NULL`; `PUT` rejects schedule edits on `connector_id IS NULL`. Update Phase 5's "`connector_id=None` → allowed" create test accordingly. (Phase 6's `_claim_due` `connector_id IS NOT NULL` filter is the scheduler backstop.) -6. **Tests** (below). - -## Tests - -- **First upload creates the pipeline**: `fileupload` into a fresh workspace → exactly one `connector_id IS NULL` pipeline named "Uploads" with `save_to_kb=True`; a `PipelineRun(trigger=upload, status=succeeded, documents_indexed=N)`. -- **Subsequent uploads reuse it**: a second `fileupload` → still one Uploads pipeline, two upload runs. Concurrency: two simultaneous first-uploads (race) → still exactly one pipeline (partial-unique + `ON CONFLICT`), two runs. -- **Duplicate-only upload**: re-uploading already-READY files (`skipped_duplicates == len(files)`, `files_to_process == []`) → **no** run recorded. -- **folder-upload**: records one upload run with `documents_indexed=len(files)`; the existing batch task/folder behaviour is unchanged. -- **No crawl billing**: upload runs have `charged_micros IS NULL`, `crawls_* IS NULL`; no `web_crawl` `TokenUsage` row is written. -- **Engine still refuses uploads**: enqueuing `run_pipeline` for an Uploads pipeline's run (or a NULL-connector pipeline) → Phase-6 engine fails it cleanly (regression guard for `06` §1). Confirms uploads are never engine-executed. -- **Scheduler ignores uploads (two ways)**: the Phase-6 tick never selects the Uploads pipeline — both because `schedule_cron IS NULL` and because `_claim_due` filters `connector_id IS NOT NULL`. Add a case: even with a cron forced onto a NULL-connector row (direct DB), `_claim_due` still skips it. -- **CRUD guards (§7)**: `POST /pipelines` with `connector_id=None` → 4xx; `POST /pipelines/{uploads_id}/run` → 4xx (no FAILED run created); `PUT` setting `schedule_cron` on the Uploads pipeline → 4xx; `DELETE` of the Uploads pipeline → allowed, and the next upload re-creates it. -- **Chat tool surfaces uploads**: `get_pipeline_runs` returns the upload runs for the workspace (trigger=upload), scoped correctly. -- **KB defaults guard**: Uploads pipeline `save_to_kb=True`; a connector pipeline created without `save_to_kb` defaults `False` (§6). -- **Best-effort isolation**: a forced failure inside `record_upload_run` does not fail the upload request (documents still created + dispatched, 2xx returned). - -## Risks / trade-offs - -- **Run is an upload *event*, not per-file ETL truth.** `status=succeeded` + `documents_indexed=` is stamped at request time; individual files may still fail ETL afterward (visible on `Document.status`, not reflected back onto the run). Accepted for MVP — accurate per-file roll-up needs the deferred `documents.pipeline_run_id` provenance column. Documented; the folder-upload barrier is the natural place to add real status later (§4). -- **Singleton Uploads pipeline ⇒ no per-upload folder on the pipeline.** `destination_folder_id` stays NULL; folder placement stays in the upload code. Fine because the upload run is descriptive, not engine-driven. If product later wants "uploads to folder X" as a first-class pipeline, that's a connector-less pipeline variant (additive). -- **Partial-unique index assumes `connector_id IS NULL ⇔ Uploads`.** True for MVP (Phase 5 reserved NULL for Uploads, and Phase 6 fails NULL-connector runs). If a *second* kind of connector-less pipeline is ever introduced, replace the index predicate with an explicit `kind`/discriminator column (additive migration) — flagged so the NULL-overload doesn't silently block it. -- **Best-effort run write can be skipped on error.** If `record_upload_run` raises (e.g. transient DB error) the upload still succeeds but no run is logged — an audit gap, never a data-loss or user-facing failure. Acceptable: the `Document` rows (the actual KB content) are the source of truth; the run is supplementary history. -- **No backfill for historical uploads.** Pre-Phase-7 uploads have no upload runs (pipeline is created lazily on the next upload). Accepted — run history starts accumulating from rollout; no migration backfills synthetic runs. -- **Upload run doesn't capture the uploader.** `PipelineRun` has no user column (Phase 5), so the run records *what/when/how-many*, not *who* — the per-document uploader is still on `Document.created_by_id`, and the pipeline's `created_by` is just whoever triggered the first-ever upload. If per-run attribution is wanted, add a nullable `triggered_by_id` to `pipeline_runs` (additive Phase-5 model change); deferred. -- **`enabled=True` on a never-scheduled pipeline.** The Uploads pipeline is `enabled=True` with NULL cron — harmless (scheduler ignores it via NULL cron *and* the `connector_id IS NOT NULL` filter); `enabled` only ever gates scheduling, which uploads never use. Left `True` so a future UI can show it as an active pipeline. -- **CRUD-guard coupling to Phase 5.** Phase 7 tightens Phase 5's create contract (`connector_id=None` now rejected) and adds `/run` + `PUT` guards. This is a documented evolution (Phase 7 ships after 5/6); the Phase-5 create test is updated here, not left contradictory. - -## Out of scope (hand-offs) - -- **`documents.pipeline_run_id` provenance** (per-file → run linkage, accurate ETL roll-up, folder-upload barrier-driven status) → post-MVP (touches `documents`). -- **Engine execution of uploads** — intentionally never; uploads are route-recorded (`06` fails NULL-connector runs by design). -- **Migrating connector periodic indexing into pipelines** — still COEXIST (Phase 5 decision); not revisited here. -- **Public pay-as-you-go API / MCP KB server** → post-MVP (umbrella "Deferred"). -- **Frontend** — surfacing the Uploads pipeline + upload runs in the Pipelines UI, "uploads always saved to KB" copy → frontend umbrella. diff --git a/plans/backend/revamp phases 4-7/00-overview.md b/plans/backend/revamp phases 4-7/00-overview.md deleted file mode 100644 index b2c2578fd..000000000 --- a/plans/backend/revamp phases 4-7/00-overview.md +++ /dev/null @@ -1,125 +0,0 @@ -# Phase 4 → end revamp — Overview & phase index (CI pivot · WIP) - -> **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small, -> single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace. -> This file is the map + the reconciliation + the subplan index. **End-to-end flow diagrams (stateless & -> stateful paths) live in `00b-pipeline-diagrams.md`.** -> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing). -> The revamp is **Phase 4 → end only**. We do **not** touch 1–3. -> -> **Citation verification — 2026-06-30 (principal-engineer pass).** All load-bearing code citations in this -> revamp were re-checked against `surfsense_backend`. Confirmed accurate: the schedule selector -> (`selector.py` — `FOR UPDATE SKIP LOCKED`, `next_fire_at`, self-heal, `catchup=False`, `croniter`), the -> `AutomationRun` model/fields, `format_to_structured_document(exclude_metadata=True)` content-hashing, the -> MCP routing gap (`constants.py::CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS`), the connector enum, and the -> `MANUAL` trigger placeholder. **Corrected during this pass:** (a) automations have **no** delivery/ -> notification path and `automation_runs.output` is never written — CI alerts wire to `app/notifications/` -> (`06`/`05b`); (b) the automation PENDING-gate is **not atomic** — the per-Tracker lock is the real -> concurrency guard (`06`/`05b`); (c) folder upload uses `root_folder_id` (not `destination_folder`) and KB -> folder scoping goes through `referenced_document_ids → SearchScope.document_ids` (`05b`); (d) the billable -> predicate is `SUCCESS and outcome.result`, to be single-sourced in the executor (`04a`). - -## The two products - -``` -PRODUCT A — stateless utility Phase 04 (Capabilities + Access) - call a verb → get data → bill. Nothing persists. - -PRODUCT B — decision-grounded CI Phase 05 (Intelligence + Timeline) (+ 06 Triggers, 07 Orchestration) - a Tracker accumulates structured signal over time. The Timeline is the moat. -``` - -## The phases (this revamp) - -| Phase | Subplan(s) | Domain(s) | One line | Ships | -|-------|-----------|-----------|----------|-------| -| **04 — Capabilities & Access** | `04a-capabilities.md` · `04b-access.md` | ① + ② | typed verbs over Acquisition + the chat/REST/MCP doors | **Product A** (revenue day one) | -| **05 — Intelligence & Timeline** | `05a-timeline.md` · `05b-intelligence.md` | ④ + ③ | the Tracker, locked schema, hot loop + the 3-store delta moat | **Product B engine** | -| **06 — Triggers** | `06-triggers.md` | ⑤ | the pluggable refresh clock; recurrence+delivery = optional CI action on automations | recurrence + alerts | -| **07 — Orchestration** | `07-orchestration.md` | ⑥ | the `intelligence_agent` CI-expert subagent (intent routing, verb chains, Tracker crafting) | the human-facing CI experience | - -> **Build order = phase order.** `04a → 04b` (unblocks everything; ships Product A) → `05a → 05b` -> (state before the loop that writes it) → `06` (drive it) → `07` (front it). `04a` first because every -> other phase calls into the capability registry. - -## Domain ↔ phase map - -``` - FIXED (Phases 1–3) OUR SCOPE (Phase 04 → 07) - ┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐ - │ Acquisition │ │ 04a Capabilities typed verbs over Acquisition │ - │ proprietary/web_crawler │◄──┤ 04b Access chat · REST · MCP doors │ - │ CrawlOutcome · billing (03c) │ │ 05a Timeline 3-store delta state (moat) │ - │ │ │ 05b Intelligence Tracker · schema · hot loop │ - │ │ │ 06 Triggers pluggable refresh clock │ - │ │ │ 07 Orchestration CI-expert subagent + tools │ - └─────────────────────────────────┘ └──────────────────────────────────────────────┘ - data engine (untouched) stateless 04 → stateful 05 , driven by 06 , fronted by 07 -``` - -## Reconciliation with the old plans - -| Old plan | Fate | Where it goes | -|----------|------|---------------| -| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in `04b` (consume-user-MCP); the taxonomy work is not core | -| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in `04a` | -| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (`05b`) + Timeline tables (`05a`); no `pipeline`/`pipeline_runs` | -| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (`05b`); scheduling/runs/delivery → **reuse automations** via a CI action (`06`) — its selector + `AutomationRun`, **not** a rebuilt cron | -| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploads are routed to a dedicated **folder** and may feed the judge (`05b`) | - -**Net:** the old `05/06/07` pipeline+KB stack is replaced by the new `04`–`07`. KB indexing of *crawled* -data is out; *user uploads* still work (and gain a CI context-folder role). - -## Confirmed decisions (locked across phases) - -- **Natural language is the only human-facing surface.** Users never name verbs; the chat agent - understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot - (A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only - on the REST/MCP dev doors. -- MVP surface = the **general web crawler + search** (`web.scrape` + `web.discover`), exposed chat + REST (MCP fast-follow). **Platform-specific scrapers are a family of individual endpoints added incrementally — none (incl. Google Maps) is committed for MVP;** each is just another verb → agent tool + dev API-key REST endpoint. -- Verbs namespaced per platform (`web.*`, and `.*` per scraper, e.g. `maps.*` as an example); `web.scrape` takes a URL **array** (inline-or-job). -- **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state. -- CI is **decoupled from automations** (automations = one optional Trigger adapter). -- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity - per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check. -- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`) on - the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools. -- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, not a - new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and delivery. - CI core still runs with **zero** automations dependency (manual/agent/cron). -- **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing - idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state. -- **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first - provider; `maps.*` / `web.discover` register their own units there. -- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a **separate - effort**; `maps.*` verbs are contracts against it and don't block this design. -- **CI context files:** files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a - dedicated `Folder`, and may feed the materiality judge via KB retrieval. ("Don't index" applies only to - *crawled* data.) -- **Names:** primitive = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent). -- License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor. - -## Cross-phase open forks (need your call) - -1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? (`05b`) -2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. (`05a`) -3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). (`06`) - -*(Resolved: the "built-in scheduler tick" fork — reuse the automations schedule selector via a CI action -instead of building a tick. See `06`.)* - -> **Per-phase implementation-time questions** (refresh failure path, first-run baseline, tracker lifecycle, -> mid-run balance, concurrency lock, delivery payload shape) are intentionally **not** resolved here — they -> don't change any boundary, contract, or data shape, and are settled when each phase is built. - -## Subplan index (revamp) - -| Phase | Subplan file | Status | -|-------|--------------|--------| -| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams | -| 04 | `04a-capabilities.md` | drafted | -| 04 | `04b-access.md` | drafted | -| 05 | `05a-timeline.md` | drafted | -| 05 | `05b-intelligence.md` | drafted | -| 06 | `06-triggers.md` | drafted | -| 07 | `07-orchestration.md` | drafted | diff --git a/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md b/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md deleted file mode 100644 index 6e6e011e0..000000000 --- a/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md +++ /dev/null @@ -1,202 +0,0 @@ -# Pipeline diagrams — end-to-end (stateless & stateful paths) - -> Visual companion to `00-overview.md`. Shows the whole CI pipeline across Phases 04–07, with the -> **stateless** (Product A) and **stateful** (Product B) paths called out explicitly. Phase refs: -> `04a` Capabilities · `04b` Access · `05a` Timeline · `05b` Intelligence · `06` Triggers · `07` Orchestration. - -## 1. Domain map — the two products - -```mermaid -flowchart LR - subgraph FIXED["FIXED · Phases 1-3 (shipped, untouched)"] - ACQ["Acquisition
proprietary/web_crawler
CrawlOutcome"] - MET["Metering · 03c
WebCrawlCreditService"] - ID["Identity / Tenancy
+ API keys"] - RT["Chat runtime
deepagents · deliverable_wait"] - end - - subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] - CAP["04a Capabilities
typed verbs + registry + job store"] - ACC["04b Access
chat · REST · MCP doors"] - INT["05b Intelligence
Tracker · schema · hot loop"] - TL["05a Timeline
3-store delta state (moat)"] - TRG["06 Triggers
refresh clock"] - ORC["07 Orchestration
intelligence_agent subagent"] - end - - ACQ --> CAP - MET --> CAP - CAP --> ACC - ID --> ACC - RT --> ORC - ACC --> ORC - CAP --> INT - INT --> TL - TRG --> INT - ORC --> CAP - ORC --> INT - ORC --> TL - - classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7; - classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee; - class CAP,ACC,INT,TL,TRG,ORC a; - class ACQ,MET,ID,RT b; -``` - -- **Product A (stateless):** `04a Capabilities + 04b Access` — call a verb, get data, bill, nothing persists. -- **Product B (stateful):** `05b Intelligence + 05a Timeline` (driven by `06`, fronted by `07`) — a Tracker - accumulates structured signal; the Timeline is the moat. - -## 2. The fork — intent router (the only human-facing decision) - -```mermaid -flowchart TD - U(["User speaks in natural language"]) --> R{"Intent router
(in intelligence_agent prompt, 07)"} - R -->|"compare / find / what is / pull / right now"| A["ONE-SHOT
Product A · stateless"] - R -->|"watch / track / notify me / every week / over time"| B["STANDING CONCERN
Product B · stateful"] - R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] - Q -->|once| A - Q -->|keep watching| B - A --> P1["section 3 · Stateless path"] - B --> P2["section 4 · Stateful path"] -``` - -Raw verbs are exposed only on REST/MCP (devs/external agents). Humans never name a verb. - -## 3. Stateless path (Product A) — end to end - -```mermaid -sequenceDiagram - autonumber - participant U as User / Dev - participant D as Door 04b (chat · REST · MCP) - participant REG as Registry 04a - participant EX as Verb executor 04a - participant ACQ as Acquisition / Maps actor - participant BILL as Billing service (03c) - participant JOB as Job record (+Celery) - - U->>D: intent (chat) OR raw verb call (REST/MCP) - D->>D: validate input_schema · authn/authz · meter-gate - D->>REG: resolve verb - REG->>EX: same executor (all doors) - alt fast / small input -> inline - EX->>ACQ: crawl_url / search / place - ACQ-->>EX: data - EX->>BILL: charge billing_unit per success - EX-->>D: envelope {status: completed, results} - D-->>U: plain-language answer (chat) / JSON (REST) - else slow / large input -> job - EX->>JOB: dispatch, return {status: pending, job_id} - D-->>U: chat: deliverable_wait polls status
REST/MCP: GET /v1/jobs/{id} - JOB->>ACQ: work loop - JOB->>BILL: charge per success - JOB-->>D: status -> completed (+ results) - D-->>U: results inline / tracked card - end - Note over U,JOB: Nothing persists beyond chat history. No Tracker, no Timeline row. -``` - -## 4. Stateful path (Product B) — setup, then refresh, then read - -### 4a. Setup once — craft & lock the Tracker (05b) - -```mermaid -flowchart TD - S0(["Standing concern detected"]) --> S1["Bind capability + input
(maps.place X / web.scrape Y)"] - S1 --> S2["Sample fetch · ONE real verb call
(billed)"] - S2 --> S3["Agent drafts definition from decision + sample:
field_schema · materiality · identity_rule · notable_signals"] - S3 --> S4{"Human reviews in chat"} - S4 -->|add field X / tweak| S3 - S4 -->|looks good| S5["LOCK · versioned snapshot
status: draft -> active"] - S5 --> S6[("Tracker persisted
+ tracked_entities row (05a)")] -``` - -### 4b. The hot loop — refresh(tracker) (05b → writes 05a) - -```mermaid -flowchart TD - T(["Trigger fires refresh(tracker) · 06"]) --> C1["1 · Crawl: call bound capability (04a)"] - C1 --> C2{"2 · Content-hash
== stored hash?"} - C2 -->|identical| STOP["stamp last_checked_at · STOP
(zero LLM cost)"] - C2 -->|changed / first run| C3["3 · Fill: agent conforms raw -> locked field_schema
(extras -> notable_signals)"] - C3 --> C4["4 · Diff (code): new record vs entity_current_state -> raw deltas"] - C4 --> C5{"5 · Judge each delta"} - C5 -->|numeric / clear rule| C5a["code decides material/noise
(free, reproducible)"] - C5 -->|ambiguous / notable_signals| C5b["agent decides
(1 LLM call; may read CI context folder)"] - C5a --> C6{"material?"} - C5b --> C6 - C6 -->|yes| W["6 · Append entity_changes row
+ overwrite entity_current_state (05a)"] - C6 -->|no| N["only bump last_checked_at
(no change -> no row)"] - W --> DONE([done]) - N --> DONE - STOP --> DONE -``` - -### 4c. Read / answer — straight from the Timeline (05a) - -```mermaid -flowchart LR - Q(["'Is competitor X pulling ahead?'"]) --> A["intelligence_agent · query_timeline(tracker_id)"] - A --> TL[("entity_changes + entity_current_state")] - TL --> ANS["Answer from stored deltas
(NOT re-derived from chat history)"] -``` - -## 5. Triggers — who calls refresh(tracker) (06) - -```mermaid -flowchart LR - M["Manual
'refresh now' (chat/REST)"] --> RF["refresh(tracker)
headless unit of work · 05b"] - AG["Agent
calls refresh as a tool"] --> RF - XC["External cron
POST /v1/trackers/{id}/refresh"] --> RF - CA["CI automation ACTION
(existing automations)"] --> RF - RF --> TLW[("writes Timeline · 05a")] - - CA -. "schedule selector" .-> SEL["hardened cron selector"] - CA -. "run record + idempotency" .-> AR["AutomationRun (PENDING-gate)"] - CA -. "delivery" .-> DEL["alert / output to user"] - - classDef opt fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; - class CA,SEL,AR,DEL opt; -``` - -CI **core** (`refresh` + Timeline) has **zero** automations dependency — manual/agent/cron all work -standalone. The CI **action** is the *optional* adapter that buys recurrence + idempotency + delivery. - -## 6. The persisted state — Timeline data model (05a) - -```mermaid -erDiagram - TRACKER ||--|| TRACKED_ENTITY : "MVP 1:1 (multi-ready)" - TRACKED_ENTITY ||--|| ENTITY_CURRENT_STATE : "latest" - TRACKED_ENTITY ||--o{ ENTITY_CHANGES : "append-only log" - - TRACKER { - uuid id - text decision - json capability_binding - json definition_locked "field_schema·identity_rule·materiality (versioned)" - text status "draft|active" - } - TRACKED_ENTITY { - uuid id - uuid workspace_id - text entity_key "from identity_rule, unique per tracker" - ts first_seen_at - } - ENTITY_CURRENT_STATE { - uuid entity_id FK - json fields "conforms to field_schema" - text content_hash "powers step-2 pre-check" - ts last_checked_at - } - ENTITY_CHANGES { - uuid id - ts captured_at - json delta "{field:{from,to}}" - text materiality "material|notable" - text decided_by "code|agent (provenance)" - text source_ref - text note "why material" - } -``` diff --git a/plans/backend/revamp phases 4-7/04a-capabilities.md b/plans/backend/revamp phases 4-7/04a-capabilities.md deleted file mode 100644 index 44e3d3928..000000000 --- a/plans/backend/revamp phases 4-7/04a-capabilities.md +++ /dev/null @@ -1,198 +0,0 @@ -# Phase 4a — Capabilities (typed verbs over Acquisition) - -> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7). -> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above -> calls into it. -> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome` -> contract, the proxy provider, and the `WebCrawlCreditService` billing seam. -> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies -> them. Locate foundation code by **symbol/grep**, not the line numbers cited. - -## Objective - -Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable -verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume -**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the -KB" with "capability that returns data."** A capability is a stateless function you *call and get data -back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached. - -## Current state (cited) - -- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content, - metadata, crawler_type}` on `SUCCESS`. `CrawlOutcomeStatus.SUCCESS` documents the billable signal, but - **operationally every caller checks `status == SUCCESS and outcome.result`** (see `scrape_webpage.py` and - `webcrawler_indexer.py`) — the predicate is duplicated, not one helper. **The executor (below) should - single-source it** in a shared `is_billable(outcome)` helper (`app/proprietary/web_crawler/`). -- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` / - `successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`). -- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`). -- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the - substrate for `web.discover`. -- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope). - -## Target design - -### The verb set (MVP) — namespaced, nothing top-level - -Two namespaces: **`web.*`** (generic crawler product) and **`.*`** (per-platform scrapers). -Future platforms slot in as their own namespace (`maps.*`, `linkedin.*`, `amazon.*`, …). - -> **Scope clarification (2026-06-30).** Platform-specific scrapers are a **family of individual scraping -> endpoints**, **not** one committed integration — **no specific platform (incl. Google Maps) is committed -> for MVP**; `maps.*` below is an **illustrative example** of the pattern. Each such scraper is *just -> another capability verb*: adding one lights up (a) an **agent tool** (chat) and (b) a **dev-callable REST -> endpoint behind the platform API key** — same executor, same billing, zero new machinery. **The MVP -> builds only `web.scrape` + `web.discover` executors**; every `.*` row is a **contract stub** -> demonstrating that the registry + generated doors (`04b`) make per-platform endpoints a drop-in. - -| Verb | Input → Output | Mode | Executes over | Bills (03c) | -|------|----------------|------|---------------|-------------| -| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success | -| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* | -| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place | -| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 | -| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page | - -### `web.scrape` — one array-friendly verb (no separate "batch") - -Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model: -"give me URLs, I give you per-URL content." - -### Execution mode — a property of the **result**, not two verbs - -A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**: - -``` -{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... } -``` - -- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline → - zero polling. (single page, one place, a search) -- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until - `completed`. (batch scrape, Maps search/reviews) - -So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller. - -- **Threshold** (how many URLs / how heavy before async) is **configurable**. -- A caller may pass `async: true` to **force** a job even for small input (agents that never block). -- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the - **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it. - -### The capability registry — single source of truth - -One registry entry per verb: - -``` -Capability { - name # dotted, e.g. "web.scrape", "maps.search" - input_schema # Pydantic - output_schema # Pydantic - mode # can-complete-inline? + job-capable? - executor # the async fn (wraps Acquisition / Maps actor) - billing_unit # how Metering charges this call -} -``` - -The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool, -REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly. -Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces. - -### Billing — open to the billing service (not hardcoded per verb) - -A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the -**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite. -`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service. - -- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit). -- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet). -- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged). -- `web.discover` → register a per-search unit (or mark free) **with the billing service**. - -The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs -stay pure; pricing stays pluggable. - -> **Charge at the executor, not the turn accumulator (locked 2026-06-30).** The billable charge fires -> **inside the capability executor** (this phase), so *every* caller meters identically — chat, -> **automation/recurring**, REST, MCP, external-cron. This is deliberate: a code review confirmed the -> automation path (`run_agent_task` → `agent_task`) establishes **no chat turn accumulator**, so the -> existing `scrape_webpage` billing (which no-ops when `get_current_accumulator()` is `None`) would let -> **automation-run crawls bill nothing**. The interactive chat **turn accumulator stays only as an -> optional presentation fold** (so a chat turn still shows the crawl line on its bill) — it is **not** the -> charging mechanism. Billing idempotency is **per capability call** (+ the `05b` content-hash pre-check), -> not per run. - -### Platform scrapers (illustrated with a Maps actor — example only, not committed) - -- Each platform scraper is a **standalone endpoint**; the one shown here (a hypothetical Maps actor) is - **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). -- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw - markdown — exactly what the Intelligence/Timeline layer needs to diff reliably. -- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on - the capability. The capability never knows what decision it serves. - -### Where it lives / license boundary - -- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store). -- It **imports from** the proprietary Acquisition engine but never moves into it. -- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*` - executors — same boundary rule Phase 3 set. - -## Work items - -1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import. -2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL - result shape; declare the `web_crawl` `billing_unit`. -3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed); - declare its `billing_unit` (or free). -4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs. -5. **Uniform envelope**: shared `completed|pending` result type returned by every executor. -6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor. -7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code). - -## Tests - -- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns - `pending` + `job_id`; the job completes to `results`. -- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail - the batch. -- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED` - free; disabling the billing flag makes it a no-op. -- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb. -- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set. - -## Risks / trade-offs - -- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when - the actor lands (doesn't block `web.*`). -- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never - grows back into `pipeline_runs`. -- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact - mid-run behavior is an implementation-time call. - -## Resolved decisions - -1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level. -2. One array-friendly `web.scrape` (no separate batch verb). -3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs. -4. Build the thin job model now (Maps search / batch scrape force it). -5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`). -6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service** - (verbs declare a `billing_unit`; `03c` is the first provider, new units register there). -7. Connector → capability is a **replacement** for these data sources, not an extension. - -## Out of scope (hand-offs) - -- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`. -- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless. -- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a - **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain - doesn't block on it. -- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces - (`linkedin.*`, `amazon.*`) — deferred. -- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing - fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`. - -## Open questions (carry forward) - -- Default async threshold for `web.scrape`. -- Whether `web.discover` is metered or free. diff --git a/plans/backend/revamp phases 4-7/04b-access.md b/plans/backend/revamp phases 4-7/04b-access.md deleted file mode 100644 index 3cea58d45..000000000 --- a/plans/backend/revamp phases 4-7/04b-access.md +++ /dev/null @@ -1,136 +0,0 @@ -# Phase 4b — Access / Surfaces (chat · REST · MCP doors) - -> Part of **Phase 4 — Capabilities & Access**. Sits on top of `04a-capabilities.md` (the registry it exposes). -> **Build after** `04a`. Together, `04a + 04b` ship **Product A** (stateless utility) — revenue day one. -> **Depends on** SHIPPED infra: Identity/Tenancy, **API keys**, the chat agent + its tool registry, the -> streaming layer, and Metering (`03c`). Access *reuses* them — it builds none net-new. -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. Locate code by **symbol/grep**, not the cited lines. - -## Objective - -Expose the capability registry (`04a`) to callers, **authenticated + metered**, through three doors — -**chat tools, REST + API keys, MCP server** — all **generated from the one registry** so the I/O -contract cannot drift between surfaces. Access contains **no business logic**: every door is the same -thin adapter. - -## Current state (cited) - -- **API keys** — existing per-workspace key infra (reuse; do **not** build net-new); billed to the - workspace owner via `03c`. -- **Chat agent + tool registry** — capability tools register *into* the existing registry; the seed is - `research/tools/scrape_webpage.py` (capability executor + access door + `03c` turn-accumulator billing). -- **Slow-job pattern** — `subagents/builtins/deliverables/deliverable_wait.py`: dispatch a Celery task, - poll the row's `status` until `READY`/`FAILED` (1.5s cadence), bounded by - `SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s); `deliverables/tools/podcast.py` is the - "return now + live card tracks progress" model. -- **BYO-`MCP_CONNECTOR`** (old `04a`) — the user's own external MCP tools inside our chat agent; its - routing-map gap is the one "connector" worth fixing, and it lands **here**. - -## Target design - -### The adapter shape (identical on every door) - -``` -parse input → validate against verb.input_schema → authn/authz → meter-gate (03c) - → call the SAME executor (04a) → serialize verb.output_schema → return the uniform envelope -``` - -### The three doors (locked order: chat → REST → MCP) - -| Door | Who | Auth | Status | -|------|-----|------|--------| -| **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) | -| **REST + API keys** | external developers (Product A) | **existing API-key infra** | **public day one** | -| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — chosen at implementation | **fast-follow** | - -REST being public day one is cheap precisely because the routes are **generated**, not hand-written — -it's a go-to-market choice, not an engineering cost. - -### Natural language is THE surface (verbs are internal) — non-negotiable - -The human-facing product is **the conversation**. A user **never** names a verb, fills an `input_schema`, -or knows "Product A vs B" exists — they describe a *need* in plain language and the agent does the rest. -Raw typed verbs are exposed only on the REST/MCP doors (devs/external agents). The chat agent owns three -responsibilities per message: **understand intent** → **pick & fill verbs** (incl. chains like -`discover → scrape`, `search → place → reviews`) → **answer in plain language** (results, not envelopes). - -### The intent router (the one new orchestration rule) - -The agent classifies each request along the stateless/stateful line **from the language**: - -``` -"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless) -"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start Tracker setup (Product B, stateful → 05b) -ambiguous → ask ONE question: "just this once, or should I keep watching it for you?" -``` - -This router is the friendly seam between the two products; it lives in the chat door (its prompt lives -in the `intelligence_agent`, `07`) and is the only human-facing decision point. - -### The two MCP directions (keep distinct) - -- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new): Streamable-HTTP `/mcp`, - stateless, bounded/paginated outputs, untrusted inputs, least-privilege (read-only verbs). -- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` (old 04a): the user's own external MCP tools inside our - chat agent. The 04a routing-gap fix lands **here**, not in Capabilities. - -### Chat ↔ slow jobs — reuse the existing background-worker pattern - -Do **not** invent a chat-async mechanism. A slow verb (`web.scrape` over many URLs, `maps.search`, -`maps.reviews`) invoked from chat dispatches the **job** (the `04a` job record) and uses the -`deliverable_wait` poll-until-terminal path; the capability **job record's `status`** is what the helper -polls. Most calls finish inside the poll window → results inline; genuinely long ones surface a tracked -card. REST/MCP expose the same job via `GET /v1/jobs/{id}` (and an MCP equivalent). - -## Work items - -1. **Door generator**: from the `04a` registry, emit (a) chat tool defs + handlers, (b) REST routes + - request/response models, (c) MCP tool schemas + handlers — one adapter shape. -2. **REST surface**: public routes + API-key auth (reuse existing) + the `03c` meter-gate; `GET /v1/jobs/{id}`. -3. **Chat tools**: generalize `scrape_webpage` into the registry-backed set; wire slow verbs through - `deliverable_wait` polling the `04a` job record's `status`. -4. **Intent router**: the A-vs-B classifier (its home is the `07` subagent prompt; the seam is here). -5. **MCP server (fast-follow)**: Streamable-HTTP `/mcp`, least-privilege; auth depth chosen at implementation. -6. **Consume-MCP fix**: repair the BYO-`MCP_CONNECTOR` routing-map gap (old 04a). - -## Tests - -- **Generated parity**: a verb added to the registry appears on chat + REST (+ MCP) with identical I/O. -- **Auth + meter**: REST without a valid key → 401; an over-budget call → blocked by the `03c` gate - before execute; a success charges once. -- **Chat slow verb**: a many-URL `web.scrape` returns inline when fast; surfaces a tracked card when long; - `GET /v1/jobs/{id}` reflects the same terminal status. -- **Intent router**: "compare X and Y" → one-shot; "watch X weekly" → Tracker setup handoff; ambiguous → - exactly one clarifying question. -- **Consume-MCP**: a configured BYO MCP tool is reachable by the chat agent (routing-gap closed). - -## Risks / trade-offs - -- **Public REST day one** → needs bounded inputs + per-key quotas / abuse posture (design alongside launch). -- **MCP auth depth** deferred to implementation (OAuth 2.1 vs bearer) — don't over-build before a consumer exists. -- **Serve-vs-consume MCP confusion** — keep the two directions explicitly separate in code and docs. - -## Resolved decisions - -0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the chat - agent understands intent, picks & fills verbs, answers in plain language; an intent router classifies - one-shot (A) vs standing-concern (B), asking one question only when ambiguous. Raw verbs live only on REST/MCP. -1. Three doors, generated from the capability registry; order chat → REST → MCP. -2. REST is **public day one** (cheap; go-to-market choice). -3. API keys: **reuse existing infra**, billed to workspace owner. -4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation. -5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the `04a` job record's `status`. -6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the old-04a fix is the consume side. - -## Out of scope (hand-offs) - -- **The verbs themselves** (executors, registry, billing units) → `04a`. -- **The CI subagent + its prompt/playbook** (where the intent router actually lives) → `07`. -- **Stateful flows** (Tracker crafting, refresh, timeline reads) → `05b`/`05a`, surfaced via chat tools in `07`. -- Legacy branded connectors stay only for backward-compat (separate hygiene task). - -## Open questions (carry forward) - -- MCP auth depth (decide at implementation). -- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas). -- Whether `web.discover` is metered or free (carried from `04a`). diff --git a/plans/backend/revamp phases 4-7/05a-timeline.md b/plans/backend/revamp phases 4-7/05a-timeline.md deleted file mode 100644 index 436257cd4..000000000 --- a/plans/backend/revamp phases 4-7/05a-timeline.md +++ /dev/null @@ -1,125 +0,0 @@ -# Phase 5a — Timeline (the moat asset) - -> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes -> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them. -> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline). -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not** -> the Knowledge Base (documents/embeddings) and **not** `automation_runs`. - -## Objective - -Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** — -accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no -change, no row.** Storage grows with the *rate of change*, not the number of runs. - -## Current state (cited) - -- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is - what this phase introduces. -- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero - publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog - (this is the **fact** analog, deliberately separate). -- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document( - exclude_metadata=True)` to compute the `content_hash` stored here. - -## Target design - -### The state / process split - -Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain -today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without -running the loop**. That separation is why Timeline is its own domain. - -### The three stores - -| Store | Role | Write pattern | -|-------|------|---------------| -| `tracked_entities` | stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`) | written **once** | -| `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run | -| `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten | - -**The timeline = the change log read in order.** To reconstruct a past point: take Current state and -replay deltas backward (north-star tooling; MVP just *stores* the deltas). - -### Data model sketch (new tables) - -``` -tracked_entities - id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at - # MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready. - -entity_current_state - entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema) - · content_hash · last_checked_at · updated_at - # overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2) - -entity_changes # the append-only change log - id · entity_id (FK) · tracker_id · captured_at - · delta JSONB # { field: { from, to } } - · materiality # material | notable(=notable_signals-sourced) - · decided_by # code | agent (audit of the materiality split) - · source_ref # url / blob key the change was observed from - · note TEXT NULL # optional agent rationale (the "why material") -``` - -- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`. -- `decided_by` records whether code or the agent ruled the change material (provenance seed). - -### What it is NOT - -- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.) -- **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store. -- **Not a resurrected `pipeline_runs`.** - -### Where it lives - -- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide - at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor). -- Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend). - -## Work items - -1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration. -2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop. -3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side. -4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit. -5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred). - -## Tests - -- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged. -- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`. -- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls. -- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row. -- **Read API**: `query_timeline` returns deltas in `captured_at` order. - -## Risks / trade-offs - -- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against — - baseline semantics (silent establish vs flood the log) are an implementation-time call. -- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very - chatty entities is deferred. -- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity - guard (lean: yes) but adds a write-path dependency on `05b`'s lock. - -## Resolved decisions - -1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`. -2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change. -3. CI-owned new tables; **not** KB, **not** `automation_runs`. -4. `content_hash` on current state powers the hot-loop cheap pre-check. -5. `decided_by` on changes records the code-vs-agent materiality provenance. -6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later). - -## Out of scope (hand-offs) - -- **The writer** (hot loop, materiality, schema lock) → `05b`. -- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the - resale/data-product surface → north star (deferred). -- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella. - -## Open questions (carry forward) - -- ORM home: `app/db.py` vs a dedicated `app/timeline/` package. -- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes). -- Retention / archival policy for very high-velocity entities (deferred). diff --git a/plans/backend/revamp phases 4-7/05b-intelligence.md b/plans/backend/revamp phases 4-7/05b-intelligence.md deleted file mode 100644 index e2763627f..000000000 --- a/plans/backend/revamp phases 4-7/05b-intelligence.md +++ /dev/null @@ -1,191 +0,0 @@ -# Phase 5b — Intelligence (the decision-grounded engine) - -> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes). -> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the -> **Product B engine** (the Tracker, locked schema, hot loop, deltas). -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the -> automations subsystem. -> **Name:** the standing-concern primitive is the **Tracker**. - -## Objective - -Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent -judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the -**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the -Access→Intelligence boundary stays pure functions. - -``` -STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists -STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state -``` - -## Current state (cited) - -- **Capability executors** (`04a`) — called directly by the loop (not through a door). -- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)` - produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`). -- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`; - note the executor writes results to `step_results[n].result`, **not** `run.output`/`artifacts`, which stay - unwritten) + the automations executor's PENDING→running gate (`execute_run()` early-returns unless - `status == PENDING`; `acks_late` is global). **Caveat:** the gate is **not atomic** (no - `UPDATE … WHERE status='pending'`), so it stops terminal/post-`mark_running` redelivery but **not** - concurrent double-claim — pair it with the per-Tracker lock (`06`). Plus the chat **job record** (`04a`) + - `deliverable_wait`. **No new run table.** -- **CI context uploads** — the existing folder-upload machinery (`POST /documents/folder-upload`, param - `root_folder_id` — there is no `destination_folder`) + KB retrieval scoped via mention-pins → - `referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids` (no direct `folder_id` on search). - -## Target design - -### The primitive — `Tracker` - -A saved, decision-grounded subject that accumulates structured signal over time: - -| Field | Meaning | -|-------|---------| -| `decision` | the question being tracked toward ("is this competitor pulling ahead?") | -| `capability_binding` | which verb + input feeds it (`maps.place(X)`, `web.scrape([Y])`) | -| `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract | -| `status` | `draft` → `locked`/`active` | - -One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred -(the `05a` model stays multi-entity-ready so it's additive). - -### Setup (once) — the agent-designed schema flow (IN MVP) - -The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is -derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first): - -1. **Bind** a capability + input. -2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data. -3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types), - `materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable - entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field. -4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run. -5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how - `automations` snapshots `definition`). - -### The hot loop (per refresh) — `refresh(tracker)` - -1. **Crawl** — call the bound capability (`04a`) → raw data. -2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`, - **stop** (no LLM cost). -3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does - **not** invent fields. Unanticipated observations go into `notable_signals`. -4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas. -5. **Judge — the materiality split:** - - **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100% - reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`). - - **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new - `notable_signals` entry → one LLM call rules material/noise. -6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`. - **No change → no row.** - -**Worked example (`maps.place` refresh):** -``` -rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM) -review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM) -hours unchanged → no delta -description reworded → code: no rule → ASK AGENT → NOISE -⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick. -``` - -### Refresh execution & idempotency — ride the invoking surface (no new run table) - -`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever -surface invoked it**: - -- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`** - is the run record and the automations executor's PENDING-gate handles redelivery. Reused, not re-written — - but the gate is **not atomic** against concurrent claims, so the **per-Tracker lock (`06`)** is the actual - double-refresh guard. -- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`. -- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via - the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No - run-level `charged_micros` ledger required for MVP. - -Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed. - -### User-supplied context files (the F idea, generalized) - -When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** — -uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies -only to *crawled* data.)** The CI-specific part is **organization + use**: - -- **Routed to a dedicated folder** for that CI chat/Tracker (reuse `POST /documents/folder-upload` with - `root_folder_id`; uploads land as `Document`s with `folder_id`). -- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder via the mention-pin - path (`referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids`), since search has no direct - `folder_id` filter: - -``` -competitor price 12.00 → 9.90 + user's context file says "our price is 10.00" - → agent: competitor crossed *below our price* → MATERIAL (and explain why) -``` - -The user's private context **shapes what counts as material**. Reuses existing KB upload + folder + -retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now. - -### Where it lives / decoupling - -- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls - capability `executor`s directly. -- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is - the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler. - -## Work items - -1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`. -2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version. -3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback. -4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`). -5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table. -6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder. - -## Tests - -- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped. -- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields. -- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword → - `decided_by=agent`; sub-threshold → noise, no row. -- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state. -- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill. -- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame. - -## Risks / trade-offs - -- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an - implementation-time call (no architecture impact). -- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM. -- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version. - -## Resolved decisions - -1. `Tracker` is the standing-concern primitive; replaces "pipeline". -2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary. -3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned. -4. Single entity per Tracker for MVP. -5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. -6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. -7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic. -8. **No new run table** — refresh audit rides `AutomationRun` (recurring) or the chat job record; redelivery - is stopped by the PENDING-gate **plus a per-Tracker lock** (the gate is non-atomic); billing idempotency - is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state. -9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed), - routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled* - data only. MVP-optional seam. - -## Out of scope (hand-offs) - -- **The state tables** (the three stores, content-hash, read API) → `05a`. -- **When refresh fires + recurrence/delivery** → `06` (Triggers). -- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`. -- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the - resale/data-product stage → north star (deferred). - -## Open questions (carry forward) - -- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?). -- Versioning policy on re-lock (new version vs in-place) — lean new version. -- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?). -- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope). diff --git a/plans/backend/revamp phases 4-7/06-triggers.md b/plans/backend/revamp phases 4-7/06-triggers.md deleted file mode 100644 index b35e90d37..000000000 --- a/plans/backend/revamp phases 4-7/06-triggers.md +++ /dev/null @@ -1,131 +0,0 @@ -# Phase 6 — Triggers (the pluggable refresh clock) - -> **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes -> `refresh(tracker)`). -> **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED -> automations subsystem (its schedule selector + `AutomationRun`) plus `app/notifications/` for alert delivery. -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no -> dependency on it, and **automations is at most one optional adapter, never required**. - -## Objective - -Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** — -and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and -the engine still works. This replaces the old Phase-6 cron scheduler — and the resolution is **not** to -rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path. - -## Current state (cited) - -- **`refresh(tracker)`** — the headless unit of work exposed by `05b`. -- **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED` - claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the - `croniter` util. *(Verified: `app/automations/triggers/builtin/schedule/selector.py::_claim_due_triggers` - + `schedule/cron.py::compute_next_fire_at`.)* -- **`AutomationRun`** — the existing run record + a PENDING-gate. *(Verified with a caveat: the gate is - `execute_run()` early-returning unless `status == PENDING` + `mark_running`, and `acks_late` / - `task_reject_on_worker_lost` are set globally — so **terminal redelivery and post-`mark_running` - redelivery are no-ops**. But there is **no atomic `UPDATE … WHERE status='pending'` claim**, so two - concurrent workers can both read PENDING → double-run. The per-Tracker lock below is therefore the - primary concurrency guard, not just belt-and-suspenders.)* -- **Delivery ≠ automations (correction).** Automations have **no** push/email/notification path, and - `automation_runs.output` is **never written** by the executor (results live in `step_results[n].result`); - today "delivery" is only **pull via `GET /automations/{id}/runs/{run_id}`** + **Zero dashboard sync** - (`zero_publication.AUTOMATION_RUN_COLS`). In-app alerts must wire to the **separate `app/notifications/` - system** — `NotificationService.create_notification(...)` (DB-backed, Zero-synced, typed - `NotificationType`). So alert delivery is a **small net-new wiring**, not a free automations reuse. -- **Access routes** (`04b`) — where the external-cron and manual REST endpoints live. - -## Target design - -### The adapters - -| Adapter | Fired by | MVP? | -|---------|----------|------| -| **Manual** | user "refresh now" (chat tool / REST) | ✅ | -| **Agent** | the in-app agent calls `refresh` as a tool | ✅ | -| **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) | -| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)`, persists deltas to the Timeline, **and emits an in-app alert** via `app/notifications/` | ✅ (the in-app recurrence + alert path) | - -### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) - -The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes."** Instead of -building a bespoke tick, we **add a CI *action* to the existing automations subsystem**: - -- **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new - scheduler. **(closes old Gap B — scheduler rigor.)** -- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run - table. **(closes old Gap A — run/idempotency)** — *but* pair it with the per-Tracker lock (the gate is - non-atomic; see Current state + `05b`). -- **Delivery / alert** → **not** free from automations. Persist the material change to the **Timeline** - (`05a`), then emit an in-app alert via **`app/notifications/`** (`NotificationService.create_notification` - with a new CI notification type; Zero-synced to the UI). **(old Gap E — alert delivery — is *mostly* - closed but needs this small wiring, not zero-cost reuse.)** - -**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery -that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action -too constraining, a thin CI-specific shape is the fallback — but the action is the default.) - -### Decoupling is preserved (automations is still optional) - -CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs -via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + audit -for in-app users (alerts ride `app/notifications/`, not automations). So we honor "don't glue CI to -automations" *and* get its machinery for free. - -### Where it lives - -- The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from - `app/intelligence/`. No new scheduler/Beat task. -- The **external-cron** and **REST manual** paths are just Access-door routes (`POST - /v1/trackers/{id}/refresh`) — `04b` plumbing. - -## Work items - -1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`. -2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`. -3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that - calls `refresh(tracker)` and, on material change, emits an in-app alert via `app/notifications/` - (`NotificationService.create_notification`, new CI notification type). *(Not "automations delivery" — - that path doesn't exist; see Current state.)* -4. **Concurrency guard**: a per-Tracker in-flight lock — the **primary** guard against concurrent - double-refresh, since the automation PENDING-gate is non-atomic (also protects the manual/cron paths - that don't go through automations). - -## Tests - -- **Decoupling**: `refresh(tracker)` works via manual/agent/external-cron with automations disabled entirely. -- **Recurring path**: a scheduled CI action fires `refresh` on cron and delivers material changes; an - unchanged refresh delivers nothing. -- **Idempotency**: a redelivered scheduled run (same `AutomationRun`) does not double-refresh. -- **External cron**: `POST /v1/trackers/{id}/refresh` triggers exactly one refresh; rejects bad auth. -- **Concurrency**: a second refresh while one is in flight is skipped/queued, not run concurrently. - -## Risks / trade-offs - -- **Action vs shape**: the CI action is the default; a thin CI automation shape is the fallback if the - action proves too constraining. -- **Delivered payload shape** (agent-summarized vs raw deltas since last fire) — implementation-time call. -- **Double-guarding concurrency**: the per-Tracker lock overlaps the automation run-gate, but the lock also - protects the manual/cron paths that don't go through automations. - -## Resolved decisions - -1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled. -2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery). -3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule - selector + `AutomationRun` (Closes old Gaps A/B). **Delivery does NOT reuse automations** (no such - path; `run.output` is unwritten) — alerts go through `app/notifications/` (Gap E: small wiring). -4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**. -5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron). - -## Out of scope (hand-offs) - -- **`refresh(tracker)` internals** (the hot loop, materiality) → `05b`. -- **The Timeline reads** the delivery summarizes → `05a`. -- **The chat tool surface** for manual/agent refresh → `07` (the `intelligence_agent` toolset). - -## Open questions (carry forward) - -- CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback). -- What the delivered payload looks like (summarized material changes vs raw deltas since last fire). -- Concurrency: per-Tracker lock granularity (like the connector indexing lock). diff --git a/plans/backend/revamp phases 4-7/07-orchestration.md b/plans/backend/revamp phases 4-7/07-orchestration.md deleted file mode 100644 index f8b3cc142..000000000 --- a/plans/backend/revamp phases 4-7/07-orchestration.md +++ /dev/null @@ -1,126 +0,0 @@ -# Phase 7 — Orchestration / Conversation (the CI-expert subagent) - -> **Phase 7** of the CI-pivot revamp — the **human-facing brain** that turns "natural language is the only -> surface" (`04b`, Decision 0) into a real deliverable. **Build last** — it sits atop `04`–`06`. -> **Depends on** all prior phases: `04a` verbs, `04b` doors + `deliverable_wait`, `05a`/`05b` Tracker + -> Timeline, `06` refresh triggers. -> **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming, -> citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin -> subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it. - -## Objective - -Ship the **`intelligence_agent`** — a net-new builtin CI-expert subagent — so users get the whole product -through plain conversation: intent routing (one-shot vs standing concern), verb composition, Tracker -crafting, and decision-grounded answering. The intent router + verb-composition + Tracker-crafting are -where "user-friendly" is won or lost, so the orchestration layer is **designed, not assumed**. - -## Current state (cited) - -- **Builtin subagent pattern** — `subagents/builtins//`: `agent.py` (`build_subagent(...)` → - `pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py` - (`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md` - (the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`. -- **Tool shape** — `research/tools/scrape_webpage.py` shows the **tool = capability executor + access door** - pattern (calls `WebCrawlerConnector.crawl_url`, returns a typed dict). **Billing note:** its current - `get_current_accumulator()` fold is **interactive-chat-only** and no-ops outside a chat turn — so CI tools - do **not** copy that as the charging path. Per `04a` (locked), **the charge fires in the capability - executor**; the chat turn accumulator stays only as an optional presentation fold. This is what makes the - recurring `06 → refresh → verb → crawl` path (no chat turn) bill correctly. -- **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`). - -## Target design - -### The `intelligence_agent` subagent (CI expert) - -A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to -`research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research -a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing -happen. It owns the **CI playbook** in `system_prompt.md`: - -1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") → call - verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; ambiguous → - ask the single clarifying question. -2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place → - maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand. -3. **Tracker crafting** — the conversational schema-design flow from `05b`: sample-fetch → propose - `field_schema` + materiality + identity → user validates & locks → versioned. -4. **Decision-grounded answering** — read the Timeline (`05a`) to answer "what changed / is X pulling - ahead?" from stored deltas, not by re-deriving from chat history. - -### The toolset (what `load_tools` returns) - -| Tool | Wraps | Mode | Billing | -|------|-------|------|---------| -| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | at the `04a` executor (per-call); chat turn accumulator = optional presentation fold only | -| `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed | -| `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — | -| `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call | -| `query_timeline(tracker_id, …)` | `05a` read API | inline | — | -| `list_trackers()` | `05a`/`05b` read | inline | — | - -- **Capability verbs are a shared tool module** (generated from the `04a` registry) — `research` can load - the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt. - (`scrape_webpage` is the seed; generalize it into the registry-backed set.) -- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the - existing `deliverable_wait` poll-until-terminal + live-card path (`04b`). - -### Boundaries - -- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `05a`/`05b` via tools; the hot loop, - materiality, and Timeline writes live in `05a`/`05b`, callable headless (so REST/MCP and Triggers reach the - same logic with no agent in the loop). -- **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this - subagent entirely and call `04a` verbs directly — they *want* explicitness. - -## Work items - -1. **Subagent scaffold**: `subagents/builtins/intelligence_agent/` with `agent.py` / `tools/index.py` / - `description.md` / `system_prompt.md`, packed via `pack_subagent`. -2. **CI playbook prompt**: intent routing (A/B + one clarifying question), verb chains, crafting flow, - decision-grounded answering. -3. **Shared verb tool module**: registry-backed (`04a`) capability tools, reusable by `research` too. -4. **Tracker/Timeline tools**: `craft_tracker` / `lock_tracker` / `update_tracker` / `refresh_tracker` / - `query_timeline` / `list_trackers`. -5. **Slow-verb wiring**: route job-mode tools through `deliverable_wait` + the live card. -6. **Router registration**: `description.md` so the main agent delegates CI-flavored requests here. - -## Tests - -- **Delegation**: a CI-flavored request routes to `intelligence_agent`; a non-CI request does not. -- **One-shot**: "compare X and Y" composes verbs and answers in plain language, persisting nothing. -- **Standing concern**: "watch X weekly" runs the crafting flow → lock → (refresh via `06`). -- **Crafting**: `craft_tracker` does a real sample-fetch and proposes a schema; `lock_tracker` versions it. -- **Answering**: `query_timeline` answers "what changed?" from stored deltas, not chat history. -- **Slow verb**: a multi-URL scrape surfaces a live card and returns terminal results. - -## Risks / trade-offs - -- **Prompt quality is the product**: intent-router/composition quality is a first-class deliverable, not - plumbing — budget iteration on the prompt. -- **One-shot ownership**: whether `research` keeps generic scraping or the `intelligence_agent` owns all - CI-flavored calls (lean: shared verb tools, CI agent owns the *playbook*). -- **Pre-frontend lock UX**: "review & lock" must render in pure chat for MVP (ties to `05b`'s open Q). - -## Resolved decisions - -1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not - a runtime rebuild. -2. Tools follow the `scrape_webpage` executor+door shape, but **billing fires in the `04a` executor** (not the chat turn accumulator, which is interactive-only and would skip automation/cron runs). -3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds - Tracker/Timeline tools + the CI prompt. -4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`. -5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async. - -## Out of scope (hand-offs) - -- **The verbs, doors, engine, state, and triggers** → `04`/`05`/`06` (this phase only *orchestrates* them). -- **Richer multi-step CI playbooks** (auto competitor discovery → multi-Tracker setup), proactive "you - should watch this" suggestions, cross-Tracker synthesis → north star (deferred). -- **Frontend CI surfaces** (cards, dashboards) → frontend umbrella. - -## Open questions (carry forward) - -- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls - (lean: shared verb tools, `intelligence_agent` owns the *CI playbook*). -- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `05b`'s open Q).