From 05bbb64527d2dd72b64d48e33128db958c90d34b Mon Sep 17 00:00:00 2001 From: "DESKTOP-RTLN3BA\\$punk" Date: Tue, 30 Jun 2026 19:30:21 -0700 Subject: [PATCH] docs(plans): principal-engineer citation pass before Phase 4 Verified revamp Phase 4-7 code citations against surfsense_backend and corrected drift: - automations have NO delivery/notification path and automation_runs.output is never written; CI alerts wire to the separate app/notifications system (NotificationService.create_notification, Zero-synced) - automation PENDING-gate is non-atomic (no UPDATE...WHERE status=pending); per-Tracker lock is the primary concurrency guard, not belt-and-suspenders - folder upload uses root_folder_id (not destination_folder); KB folder scoping goes via referenced_document_ids -> SearchScope.document_ids (no folder_id search filter) - billable predicate is 'SUCCESS and outcome.result' duplicated per caller; to be single-sourced in the 04a executor Confirmed accurate: schedule selector (FOR UPDATE SKIP LOCKED/next_fire_at/self-heal/catchup=False/croniter), AutomationRun model, format_to_structured_document(exclude_metadata=True), MCP routing gap (constants.py), connector enum, MANUAL trigger placeholder. Added dated verification stamp to revamp/00-overview.md. Also reconciled 03b/03c/umbrella wording. Co-authored-by: Cursor --- plans/backend/00-umbrella-plan.md | 157 ++++++++++++------ plans/backend/03b-proxy-expansion.md | 10 +- plans/backend/03c-crawl-billing.md | 2 + plans/backend/04a-connector-category.md | 10 ++ plans/backend/04b-source-discovery.md | 7 +- plans/backend/05-pipelines-model.md | 3 + plans/backend/06-pipelines-exec.md | 3 + plans/backend/07-upload-pipeline-kb.md | 3 + .../backend/revamp phases 4-7/00-overview.md | 16 +- .../revamp phases 4-7/04a-capabilities.md | 33 +++- .../revamp phases 4-7/05b-intelligence.md | 31 ++-- .../backend/revamp phases 4-7/06-triggers.md | 46 +++-- .../revamp phases 4-7/07-orchestration.md | 13 +- 13 files changed, 244 insertions(+), 90 deletions(-) diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index e6dd99889..cd0373052 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -2,7 +2,7 @@ > Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`). -This is the high-level roadmap. It is sequenced to match the agreed order: rename first, then connector restructure, then Pipelines. +This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities & Access**, **Phase 5 Intelligence & Timeline**, **Phase 6 Triggers**, **Phase 7 Orchestration** (canonical subplans in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was **superseded on 2026-06-30** — see the Architecture correction below. The earlier "Phase 5′ automation-enhancements" framing is also retired (folded into Phases 4/5/6 of the revamp). > SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases. @@ -10,7 +10,49 @@ This is the high-level roadmap. It is sequenced to match the agreed order: renam "NotebookLM for Competitive Intelligence" — each WorkSpace acts as a workspace for setting up competitive-intelligence-optimised notebooks. -## Target architecture +## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted + +> **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model: +> +> - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages. +> - **Web search + the WebURL crawler = platform-native agent tools** (already `web_search` + `scrape_webpage` on the main agent, `main_agent/tools/index.py`), always available; later exposed to developers via a platform **API key**. They are **NOT connectors and NOT pipelines**. +> - **Connectors = MCP tools** (Type-2) **+ file/KB-input connectors** (the only Type-1 that survives). All branded natives → MCP. +> - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots. +> - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source. +> +> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical revamp in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md): +> - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.* +> - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine. +> - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path). +> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `intelligence_agent` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language. +> +> **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors. + +## Target architecture (corrected + revamped) + +> Full stateless/stateful end-to-end flow diagrams live in [`revamp phases 4-7/00b-pipeline-diagrams.md`](revamp%20phases%204-7/00b-pipeline-diagrams.md). Summary below. + +```mermaid +flowchart TD + ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (P8)"]] + REG["Capability registry (P4a): web.scrape · web.discover (MVP) · <platform>.* scrapers (later) — typed verbs, bill at executor"] + ACQ --> REG + REG --> DOORS["Access doors (P4b): chat tools · REST+API key · MCP — generated from the registry"] + DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · Tracker crafting"] + KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT + CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT + AGENT -->|one-shot / Product A| ANS["plain-language answer (nothing persists)"] + AGENT -->|standing concern / Product B| TR["Tracker (P5b): locked schema · refresh hot loop"] + REG --> TR + TR --> TL[("Timeline (P5a): tracked_entities · entity_current_state · entity_changes — delta moat")] + TRIG["Triggers (P6): manual · agent · external-cron · CI automation action"] --> TR + AUTO["Automations (existing): schedule selector + AutomationRun (recurrence + audit; NO delivery)"] --> TRIG + TL --> DELIVER["alerts on material change → app/notifications (Zero-synced)"] + APIKEY["Platform API key (future)"] --> DOORS +``` + +
+Original (superseded) Pipelines-centric architecture ```mermaid flowchart TD @@ -30,17 +72,22 @@ flowchart TD RUN -->|"read-only context"| CHAT ``` +
+ ## Decisions locked - Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps. - Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`. -- Connectors split into two types via a STATIC code registry (`connector_type` → category/availability), NOT a DB column. Type 1 = `DATA_SOURCE` (pull → pipelines/KB): WebURL crawler, Google Drive (native + Composio), OneDrive, Dropbox, file uploads, plus deferred platform connectors. (YouTube is DEFERRED — no backend connector exists today.) Type 2 = `MCP_TOOL` (act in chat): only the generic `MCP_CONNECTOR` is functional for MVP. All branded natives are deprecated (`MIGRATING`, turned off) pending a post-MVP MCP re-point — they are NOT functionally moved to MCP in this MVP. Artifacts stay in the existing `deliverables` agent system (not routed through MCP). (Full model: Phase 4 below / `04a`.) -- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as a SOURCE-DISCOVERY helper: they suggest URLs the user can add to the Universal WebURL Crawler when setting up pipelines (they are not a standalone connector type and do not index data). NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set. +- **Capabilities, not connectors, are the core of Phase 4.** The crawler + search become typed **verbs** (MVP `web.scrape` + `web.discover`; per-platform scrapers like `maps.*` are later, uncommitted drop-ins) in a single **capability registry** that generates the chat/REST/MCP doors identically (revamp `04a`/`04b`). A capability is *call → data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into revamp `04b` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system. **~~pipeline-eligibility~~ removed.** +- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent (or a Tracker binding) can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set. +- **`Tracker` + Timeline are the standing-concern primitive** (replaces "pipeline"). A Tracker binds a capability + input and a locked, versioned `definition` (`field_schema`/`identity_rule`/`materiality`); its `refresh(tracker)` hot loop writes a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`). **Store deltas, no change → no row.** The Timeline — not the KB, not `automation_runs` — is the only new durable CI state (revamp `05a`/`05b`). +- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via `WebCrawlCreditService` (03c) — so chat, automation/recurring, REST, MCP, and external-cron all meter **uniformly**. (Closes the confirmed gap where automation-run crawls billed nothing; see 03c + the Architecture correction.) - Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP. - MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point). -- `Pipeline` and `PipelineRun` are new first-class tables. A Pipeline references a connector + config + schedule + KB destination. File upload creates/uses a pipeline and registers a run; uploads always save to KB. -- The chat agent gets read-only access to pipeline run history (pipelines + their recent runs/status) as context, so it can reason about what was fetched, when, and whether runs succeeded — even for data not saved to the KB. -- Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core), public pay-as-you-go API for Type-1 connectors, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that. +- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines. The standing-concern need is met by the **`Tracker` + Timeline** (revamp `05`), and **recurrence + audit reuse the existing automations** (schedule selector + `AutomationRun`) via a CI **action** (revamp `06`) — **no new scheduler, no new run table**. **Delivery does NOT reuse automations** (no such path; `automation_runs.output` is never written) — alerts wire to the separate `app/notifications/` system. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper). +- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by reading the **Timeline** (`query_timeline`, revamp `05a`), not by re-deriving from run logs. Recurring execution audit rides `AutomationRun` (already persisted + queryable). No separate pipeline-run context tool. +- **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**. +- Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that. ## Platform connector research list (deferred build, MVP = "coming soon") @@ -77,54 +124,60 @@ flowchart TD The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans: -> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path** — Phases 4–7 (connector taxonomy, pipelines, execution) consume the `03a` `CrawlOutcome`/`FetchStrategy` contract, the `03b` proxy provider, and the `03c` `WebCrawlCreditService` billing seam. **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phases 4–7** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then proceed to Phase 4), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized. The dependency notes inside `03d`/`03e`/`03f` (each requires `03a`/`03b`) still hold; this only frees them from gating Phase 4+. +> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (revamp `04a`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture correction; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then revamp `04a` Capabilities → `04b` Access → `05` Intelligence & Timeline), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized. - **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran). -- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-pipeline override is left as a no-op seam for Phase 5/6. -- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Two surfaces: **connector/pipeline crawls** billed to the **workspace owner** via a dedicated `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`, **not** `billable_call`); **chat scrapes** fold their crawl cost into the chat turn's existing bill (turn accumulator). No DB migration (uses the existing free-form `web_crawl` usage_type). +- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability/per-Tracker override is left as a no-op seam (revamp `04a`/`05`), not built. +- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture correction + revamp — see log): the charge moves to the **capability executor** (revamp `04a`), so **every** caller meters uniformly — chat, **automation/recurring** (the confirmed gap: `run_agent_task` establishes no turn accumulator today, so `scrape_webpage` bills nothing under automations), REST, MCP, and external-cron — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type). - **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier. - **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account. - **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier. -### Phase 4 — Connector two-type restructure (backend) [`subplans: 04a–04b`] +### Phase 4 — Capabilities & Access (backend) [`subplans: revamp/04a–04b`] -Split into two independently testable subplans: +> **Canonical subplans:** [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md). Together they ship **Product A** (stateless utility) — revenue day one. **Build `04a` first** — every later phase calls the registry. -- **`04a-connector-category.md`** — Connector taxonomy + availability gating + the MCP routing-gap fix. Introduce a **static registry** keyed by `connector_type` → (`category` = `DATA_SOURCE` | `MCP_TOOL`, `availability` = `AVAILABLE` | `MIGRATING` | `DISABLED`), exposed as computed fields on the connector schema/API. Keep `is_indexable` (orthogonal capability flag — still gates the real `/index` + periodic machinery; NOT replaced). Gate creation/index/subagent-build/pipeline-eligibility off the registry. Fix the `MCP_CONNECTOR` subagent routing-map gap. **No DB migration** (registry is code). -- **`04b-source-discovery.md`** — Web-search repurposing. **Drop all 5 search-API `connector_type` enum values** (`SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API`) + their connector paths; relocate survivors to platform/env config (SearXNG already env-based; **Linkup + Baidu keys move from per-connector `config` to platform env**); rewire the chat `web_search` tool onto platform providers; add a backend **source-discovery endpoint** suggesting candidate URLs for the WebURL Crawler. **Has a (destructive) migration** to delete the 5 connector types' rows. +- **`revamp/04a-capabilities.md`** — Turn Acquisition (crawler + search + a future Maps actor) into a **small set of typed, callable verbs** over a single **capability registry**. MVP verbs = **only `web.scrape(urls[])`** (array-friendly, inline-or-job) **+ `web.discover(query, top_k)`** (SearXNG/Linkup/Baidu). **Platform-specific scrapers** (Maps/LinkedIn/Amazon/… — *examples, none committed*) are a family of **individual endpoints** added later: each is just another verb → agent tool + dev API-key REST endpoint, so `04a` only ships **contract stubs** for them, not executors. Uniform result **envelope** (`completed | pending + job_id`); slow calls use a **thin job record** + existing Celery (not a resurrected `pipeline_runs`). Each verb declares a **`billing_unit`**; charging is **delegated to the billing service** (03c is the first provider) and happens **at the executor** so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary. +- **`revamp/04b-access.md`** — Expose the registry through **three generated doors** (order: chat → REST+API-key → MCP), each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → envelope`). **Natural language is the only human-facing surface** (users never name verbs); an **intent router** classifies one-shot (Product A) vs standing-concern (Product B). Slow chat verbs reuse the shipped `deliverable_wait` poll-until-terminal + live-card path. This is where the **BYO-`MCP_CONNECTOR` routing-gap fix** (from the old `04a`) lands (consume-user-MCP). **Locked model (MVP):** -- **Functional Type-1 (Data Sources — can create pipelines + feed KB):** Universal WebURL Crawler (`WEBCRAWLER_CONNECTOR`), Google Drive (native + Composio), OneDrive, Dropbox, plus file uploads. YouTube is **deferred** (no backend connector exists today — net-new work; reserve the slot). Deferred "coming soon" platforms: LinkedIn, Amazon, Google, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor. -- **Functional Type-2 (MCP Tools — act in chat, NO pipelines):** ONLY the generic `MCP_CONNECTOR` (bring-your-own MCP server). The `MCP_CONNECTOR` routing-gap fix (04a) is what makes it work. -- **Deprecated for MVP ("migrating to MCP soon", `availability = MIGRATING`):** every branded native — current indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) AND act-only ones (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Behaviour: block new creation, disable `/index`+periodic + their chat subagents; KEEP existing rows + already-indexed KB docs **searchable** (via the always-on knowledge-base subagent). Real MCP migration is post-MVP. -- **Disabled for MVP (`availability = DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook) — distinct from MIGRATING (not MCP-bound). -- Frontend connector UI restructure is DEFERRED (frontend umbrella). +- **Capabilities replace connectors as data sources.** `web.scrape` / `web.discover` (MVP) — and later per-platform scraper verbs — are *call → data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule. +- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `revamp/04b`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**. +- **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP. +- **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook). +- Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.) +- The deferred "coming soon" platforms (LinkedIn, Amazon, Google, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as **Phase 8 native verbs / API endpoints**, not Type-1 KB connectors. +- Frontend connector/capability UI restructure is DEFERRED (frontend umbrella). -### Phase 5 — Pipelines data model [`subplan: 05-pipelines-model.md`] +### Phase 5 — Intelligence & Timeline (backend) [`subplans: revamp/05a–05b`] -- New tables: `pipelines` (workspace_id, `created_by_id` nullable SET-NULL [creator metadata, **not** owner], connector_id nullable, name, config JSONB, `schedule_cron` + `schedule_timezone` + enabled + next_scheduled_at, `save_to_kb` bool, `destination_folder_id` nullable) and `pipeline_runs` (pipeline_id, status, trigger = manual/scheduled/upload, timestamps, doc counts, `crawls_*`, `charged_micros`, error, optional raw-result blob ref). -- Models + Pydantic schemas + Alembic migration + backend Zero publication entry. -- Pipelines API routes: CRUD + manual run trigger + list runs. +> **Canonical subplans:** [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md). Ships the **Product B engine**. **Depends on `04a`** (the verbs the loop calls). **Build `05a` (tables) before `05b` (the loop that writes them).** This *replaces* the old Pipelines Phases 5–7 — the durable CI state, and **the moat**. -### Phase 6 — Pipeline execution + scheduling [`subplan: 06-pipelines-exec.md`] +- **`revamp/05a-timeline.md`** — The **3-store delta Timeline**: `tracked_entities` (stable identity, written once), `entity_current_state` (latest values + `content_hash` + `last_checked_at`, overwritten each run), `entity_changes` (append-only material deltas: `{field:{from,to}}`, `materiality`, `decided_by`, `source_ref`). **Store deltas, no change → no row** (storage ∝ rate of change). Explicitly **not the KB, not `automation_runs`, not a resurrected `pipeline_runs`.** New CI-owned Apache-2 tables + Alembic migration; write/read API for the `05b` loop and the read side. **This is where "diffable run history" actually lives** (superseding the old "structured `automation_runs.output`" idea). +- **`revamp/05b-intelligence.md`** — The `Tracker` primitive + the **hot loop** `refresh(tracker)`: crawl (call the bound verb) → **content-hash pre-check** (identical → stamp `last_checked_at`, stop, no LLM) → **fill** raw data to the locked `field_schema` (structured output; extras → `notable_signals`) → **diff** (code) → **judge — the materiality split** (deterministic numeric/clear rules in code for free; agent only on ambiguous) → **append** a change + overwrite current state, else only `last_checked_at`. Setup = an **agent-designed, human-locked, versioned schema** flow (sample-fetch → draft `definition` → review/lock). **No new run table:** recurring audit/idempotency ride `AutomationRun`, manual rides the chat job record; **billing idempotency is per-capability-call** + the hash pre-check. Optional **CI context folder** (user files uploaded in a CI chat go into the KB as normal, routed to a folder, and can feed the judge). New Apache-2 package `app/intelligence/`. -- Run engine: pipeline run -> invoke connector fetch (WebURL crawler for MVP) -> if `save_to_kb`, persist to the destination folder (the crawler writes `Document`s directly via `index_crawled_urls`, extended with a `folder_id` param; it does **not** use `IndexingPipelineService`, which is the file/Composio path) -> write `PipelineRun` record. (06 verified the crawler indexer is its own KB-write path.) -- **Crawl billing wiring (carry-over from `03c`):** `03c` meters crawls inside `webcrawler_indexer`. A pipeline run that crawls but has `save_to_kb=false` must NOT bypass billing — wire the pipeline fetch through the same `WebCrawlCreditService` (pre-check + charge on `crawls_succeeded`) regardless of the KB-save branch, ideally recording `charged_micros` on the `PipelineRun` for idempotency. Otherwise non-KB pipeline crawls are free by accident. -- Scheduling: a Celery Beat tick over `pipelines.next_scheduled_at`, modeled on the **automations** cron **selector** (`automations/triggers/builtin/schedule/selector.py` — cron + `FOR UPDATE SKIP LOCKED` + self-heal, reusing the `croniter` util), which fits `schedule_cron` better than the connector `frequency_minutes` checker. Plus the de-dup guard (a pipeline over a connector disables that connector's own periodic indexing) so the two minute-level scans never double-crawl/bill. -- When `save_to_kb` is off, persist the raw fetch result on the run (blob via `file_storage`) so it is retrievable without indexing. -- Chat agent context: expose pipeline run history to the `multi_agent_chat` agent (read-only) — via a tool (e.g. `list_pipelines` / `get_pipeline_runs`) and/or a context middleware injection (similar to `KnowledgeTreeMiddleware`). Scope strictly to the active workspace. Gives the agent awareness of recent runs, statuses, schedules, and last-fetched timestamps. +### Phase 6 — Triggers (backend) [`subplan: revamp/06-triggers.md`] -### Phase 7 — File upload as a pipeline + KB-save-secondary [`subplan: 07-upload-pipeline-kb.md`] +> **Canonical subplan:** [`revamp phases 4-7/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) — the thinnest phase. **Depends on `05b`** (`refresh(tracker)`). Decides **when** a Tracker refreshes; Intelligence never knows which trigger fired. -- Wire file upload (`documents_routes.py` `fileupload` + `folder-upload` flows) to lazily get-or-create a **singleton "Uploads" pipeline** per workspace (`connector_id = NULL`, `save_to_kb = true`) and register a `PipelineRun(trigger=upload)`; uploads always `save_to_kb = true`. -- **Key design fact:** uploads are **route-recorded, not engine-executed** — Phase 6's engine fails `connector_id IS NULL` runs, so the upload routes themselves write a terminal audit `PipelineRun` (the existing upload code stays the executor). The run is an upload-*event* record; per-file ETL outcomes stay on `Document.status` (accurate per-file roll-up needs the deferred `documents.pipeline_run_id` provenance column). -- A small migration adds a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL` (enforces the singleton + race-safe get-or-create). -- KB-save-secondary: the opt-in `save_to_kb` + destination folder for connector pipelines already shipped in Phases 5–6; Phase 7 only records the inverse invariant (uploads always KB) and guards the connector default stays `False`. +- Adapters: **manual** (user "refresh now" via chat tool / REST), **agent** (the agent calls `refresh` as a tool), **external cron** (`POST /v1/trackers/{id}/refresh`, zero infra on us), and the **CI automation action** — recurrence reusing the **existing automations**: its hardened **schedule selector** (no new scheduler) + **`AutomationRun`** (no new run table). **Alert delivery is separate** — persist deltas to the Timeline, then emit an in-app notification via `app/notifications/` (automations have no delivery path; `run.output` is unwritten). Guard concurrency with a **per-Tracker lock** (the automation PENDING-gate is non-atomic). A single `refresh_tracker` action, **not** a new automation shape. +- **Decoupling preserved:** CI core (`refresh` + Timeline) runs with **zero** automations dependency (manual/agent/external-cron); automations is the **optional** in-app recurrence+alert adapter. This is the correct home for the old "wire a run-now trigger" item. + +### Phase 7 — Orchestration (backend) [`subplan: revamp/07-orchestration.md`] + +> **Canonical subplan:** [`revamp phases 4-7/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — the human-facing brain. **Build last** (atop `04`–`06`). We plug into the shipped multi-agent runtime; we don't rebuild it. + +- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs standing concern, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), **Tracker crafting** (the `05b` schema-design flow in chat), and **decision-grounded answering** (read the Timeline via `query_timeline`, not chat history). +- Toolset: registry-backed capability verbs (shared with `research`) + Tracker/Timeline tools (`craft_tracker`/`lock_tracker`/`refresh_tracker`/`query_timeline`/`list_trackers`); slow verbs reuse `deliverable_wait`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing). + +### (Future) Platform API-key + public MCP + +Platform **API-key** access to the capability verbs + a public **MCP server** exposing the input-only KB — post-MVP (see Phase 8 + Deferred). ### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`] -NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Once the hardened fetch core (Phases 3a–3e) is solid, **platform actors** layer **on top** of it: per-platform structured extractors (Google Maps/Local, LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list" above), built in-house "Apify-style" rather than via third-party paid actors. They reuse the existing machinery end-to-end: each actor is a **Type-1 data-source connector** (Phase 4 taxonomy), runs via **Pipelines** (Phases 5–6, manual/cron), saves to the **KB** (opt-in folder), and is **billed per run** (extending `03c`/`06`). They consume the `03a` `FetchStrategy` core (proxies + `03e` hardening + `03d` captcha) under their own extractors. +NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Once the hardened fetch core (Phases 3a–3e) is solid, **platform scrapers** — a family of **individual scraping endpoints** (Google Maps/Local, LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **all examples, none committed**) — layer **on top** of it. **Each is simply another capability verb** (`revamp/04a`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha), callable by the chat agent, by automations, and by the future platform API key. Billed per call via `WebCrawlCreditService`. - **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream. - The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling. @@ -136,18 +189,21 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta - Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename. - Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs. - Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types. -- Pipelines UI + positioning: Pipelines section (list/create/configure/run-history/manual run), WebURL source-discovery UX, file-upload-as-pipeline surfacing, "coming soon" platform cards, "NotebookLM for Competitive Intelligence" copy. +- CI + automations + positioning UI (**replaces the superseded "Pipelines UI"**): the CI experience (Tracker crafting/review-&-lock, Timeline/change views, "refresh now"), automation create/list/run-history, `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards, "NotebookLM for Competitive Intelligence" copy. ## Open items to confirm during subplanning - ~~Rename transition: hard cutover vs temporary API aliases~~ RESOLVED: HARD CUTOVER (see resolved log + 02-rename-backend.md). The frontend is rebuilt against the corrected backend in its own umbrella; backend is verified via tests/OpenAPI, not the old UI. -- ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ RESOLVED (Phase 5): COEXIST — connector periodic path stays untouched; pipelines add a sibling `next_scheduled_at` scan. Phase 6 owns the de-dup guard (a pipeline over a connector disables that connector's `periodic_indexing_enabled`) to avoid double crawl/bill. See `05-pipelines-model.md`. -- ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ RESOLVED (Phase 6): **tool** — a read-only main-agent `get_pipeline_runs` tool (registry pattern), workspace-scoped via its build-time `search_space_id`, opening its own `shielded_async_session()` (like `KnowledgeTreeMiddleware`). Always-on `` middleware injection is deferred (per-turn token cost). See `06-pipelines-exec.md`. +- ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with. +- ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`). - ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP. +- **Revamp cross-phase forks (from `revamp/00-overview.md`, decide during subplanning):** (1) schema **review-&-lock** UX pre-frontend — pure-chat confirmation for MVP? (`05b`); (2) Timeline ORM home — `app/db.py` vs a dedicated `app/timeline/` (`05a`); (3) recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback) (`06`); (4) is `web.discover` metered or free (`04a`); (5) MCP-serve auth depth — OAuth 2.1 vs bearer (`04b`). ## Resolved decisions log -- Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines); not a standalone connector type. +- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. +- **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths). +- ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type. - Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu). - Obsidian + Circleback: disabled for MVP. - MCP-availability audit: BookStack, Elasticsearch, Luma all have MCP available -> eligible for Type-2 (so deprecated as `MIGRATING`, not `DISABLED`). For MVP they are turned off pending the post-MVP MCP re-point, like the other branded natives. @@ -167,9 +223,9 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta - Phase 3 test-harness subplan `03f` ADDED: **manual-only** (no CI/automated gating now) undetectability + extraction scorecard, modeled on CloakBrowser's `bin/cloaktest`. Two labeled axes (Suite S stealth + Suite E extraction) so they scale independently. Drives the **real** Scrapling tiers (browser + curl_cffi HTTP/TLS), reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. **TLS JA3/JA4 parity = informational axis, not a hard gate.** Adopt CloakBrowser bars as aspirational; record our actual free-stack numbers as the committed baseline. The scorecard is the documented evidence/trigger for flipping `03e`'s deferred paid-unblocker tier. - Licensing placement of 03e/03f code (decided during 03f impl, applying the §boundary test): the **stealth kwargs builder + geoip coherence** (`03e` bypass tuning) live **proprietary** at `app/proprietary/web_crawler/stealth.py`; the **block classifier** (passive telemetry, public markers) stays **Apache-2** at `app/utils/crawl/classifier.py` (direct analog of the captcha split: proprietary `captcha.py` logic + Apache-2 `app/utils/captcha/` config). The **03f scorecard harness** moved whole to `app/proprietary/web_crawler/testbench/` (run `python -m app.proprietary.web_crawler.testbench`) — it's the moat's measurement tool and can't be cleanly half-moved (a proprietary Suite S would back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). The `scripts/e2e_phase3_crawl_billing.py` billing e2e stays in `scripts/` (Apache-2) since it exercises billing, not the stealth moat. - Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7. -- Phase 5 pipelines data model: two new tables `pipelines` (mutable) + `pipeline_runs` (append-only), modeled on `automations`/`automation_runs`; ORM lives in `db.py` next to connectors/folders. `connector_id` nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's `is_pipeline_eligible`. Schedule = `schedule_cron` + `schedule_timezone` (default UTC) + `next_scheduled_at` (cron, matching automations). `pipeline_runs` pre-includes `charged_micros`/`crawls_*`/`result_blob_key` so Phase 6 needs no extra migration. Both tables published to Zero **full-row** (like folders/connectors). Routes reuse `CONNECTORS_*` permissions. Phase 5 ships the data model + API surface only; the `/run` endpoint enqueues a Phase-6 task stub. -- Phase 7 uploads-as-pipeline: a **singleton "Uploads" pipeline** per workspace (`connector_id NULL`, `save_to_kb=true`), lazily get-or-created (race-safe via a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL`). Each `fileupload`/`folder-upload` request writes a **terminal audit `PipelineRun(trigger=upload, status=succeeded, documents_indexed=)`** — uploads are **route-recorded, not engine-executed** (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an **inner** try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; `charged_micros` NULL). Per-file ETL truth stays on `Document.status`; accurate roll-up needs the deferred `documents.pipeline_run_id` provenance. Connector `save_to_kb` default stays `False` (opt-in for connectors, mandatory for uploads). Phase 7 also **guards Phase-5's generic CRUD** against the system Uploads pipeline: `POST /pipelines` rejects `connector_id=None` (supersedes Phase 5's permissive create), `/run` and schedule-`PUT` reject NULL-connector pipelines, and Phase 6's scheduler `_claim_due` filters `connector_id IS NOT NULL` as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See `07-upload-pipeline-kb.md`. -- Phase 6 pipeline execution: run engine mirrors **automations** (thin Celery `run_pipeline(run_id)` → `execute_pipeline_run`; PENDING-gated, idempotent terminal no-op; `pending→running→succeeded/failed` with timing/counts/error). MVP executor = **WebURL crawler only** (other types fail cleanly). `save_to_kb=true` reuses `index_crawled_urls` extended with a `folder_id` param (lands in the destination folder); `save_to_kb=false` runs a **fetch-only** loop and persists one JSON blob via `file_storage` (`result_blob_key`). **Crawl billing is owned by the run engine** for the pipeline path (pre-check on `len(urls)` + charge `crawls_succeeded` + idempotent `charged_micros`), calling the crawler with a new `bill=False` seam (the connector `/index`+periodic paths keep `03c`'s in-indexer `bill=True`) — so non-KB runs are billed identically. Scheduler = a `pipeline_schedule_select` Beat tick modeled on the automations cron **selector** (cron + `FOR UPDATE SKIP LOCKED` + self-heal, using the existing `croniter` util), plus the **de-dup guard** (a pipeline over a connector disables that connector's `periodic_indexing_enabled`). Chat context = the `get_pipeline_runs` tool. Carries a small additive `05` amendment: a `schedule_timezone` column (cron util needs a tz). See `06-pipelines-exec.md`. +- **[SUPERSEDED — Architecture correction 2026-06-30; no pipelines]** Phase 5 pipelines data model: two new tables `pipelines` (mutable) + `pipeline_runs` (append-only), modeled on `automations`/`automation_runs`; ORM lives in `db.py` next to connectors/folders. `connector_id` nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's `is_pipeline_eligible`. Schedule = `schedule_cron` + `schedule_timezone` (default UTC) + `next_scheduled_at` (cron, matching automations). `pipeline_runs` pre-includes `charged_micros`/`crawls_*`/`result_blob_key` so Phase 6 needs no extra migration. Both tables published to Zero **full-row** (like folders/connectors). Routes reuse `CONNECTORS_*` permissions. Phase 5 ships the data model + API surface only; the `/run` endpoint enqueues a Phase-6 task stub. +- **[SUPERSEDED — Architecture correction 2026-06-30; uploads just populate the input-only KB]** Phase 7 uploads-as-pipeline: a **singleton "Uploads" pipeline** per workspace (`connector_id NULL`, `save_to_kb=true`), lazily get-or-created (race-safe via a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL`). Each `fileupload`/`folder-upload` request writes a **terminal audit `PipelineRun(trigger=upload, status=succeeded, documents_indexed=)`** — uploads are **route-recorded, not engine-executed** (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an **inner** try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; `charged_micros` NULL). Per-file ETL truth stays on `Document.status`; accurate roll-up needs the deferred `documents.pipeline_run_id` provenance. Connector `save_to_kb` default stays `False` (opt-in for connectors, mandatory for uploads). Phase 7 also **guards Phase-5's generic CRUD** against the system Uploads pipeline: `POST /pipelines` rejects `connector_id=None` (supersedes Phase 5's permissive create), `/run` and schedule-`PUT` reject NULL-connector pipelines, and Phase 6's scheduler `_claim_due` filters `connector_id IS NOT NULL` as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See `07-upload-pipeline-kb.md`. +- **[SUPERSEDED — Architecture correction 2026-06-30; automations are the run engine]** Phase 6 pipeline execution: run engine mirrors **automations** (thin Celery `run_pipeline(run_id)` → `execute_pipeline_run`; PENDING-gated, idempotent terminal no-op; `pending→running→succeeded/failed` with timing/counts/error). MVP executor = **WebURL crawler only** (other types fail cleanly). `save_to_kb=true` reuses `index_crawled_urls` extended with a `folder_id` param (lands in the destination folder); `save_to_kb=false` runs a **fetch-only** loop and persists one JSON blob via `file_storage` (`result_blob_key`). **Crawl billing is owned by the run engine** for the pipeline path (pre-check on `len(urls)` + charge `crawls_succeeded` + idempotent `charged_micros`), calling the crawler with a new `bill=False` seam (the connector `/index`+periodic paths keep `03c`'s in-indexer `bill=True`) — so non-KB runs are billed identically. Scheduler = a `pipeline_schedule_select` Beat tick modeled on the automations cron **selector** (cron + `FOR UPDATE SKIP LOCKED` + self-heal, using the existing `croniter` util), plus the **de-dup guard** (a pipeline over a connector disables that connector's `periodic_indexing_enabled`). Chat context = the `get_pipeline_runs` tool. Carries a small additive `05` amendment: a `schedule_timezone` column (cron util needs a tz). See `06-pipelines-exec.md`. ## Subplan index (backend) @@ -183,10 +239,17 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta | 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) | | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default | | 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff | -| 4 | `04a-connector-category.md` | drafted | -| 4 | `04b-source-discovery.md` | drafted | -| 5 | `05-pipelines-model.md` | drafted | -| 6 | `06-pipelines-exec.md` | drafted | -| 7 | `07-upload-pipeline-kb.md` | drafted | +| 4 | `revamp phases 4-7/04a-capabilities.md` | **CANONICAL** · drafted — capability registry + MVP verbs `web.scrape`/`web.discover` (per-platform scrapers = later drop-ins); bill at executor. **← build next** | +| 4 | `revamp phases 4-7/04b-access.md` | **CANONICAL** · drafted — generated chat/REST/MCP doors + intent router + BYO-MCP routing fix | +| 5 | `revamp phases 4-7/05a-timeline.md` | **CANONICAL** · drafted — 3-store delta Timeline (the moat) | +| 5 | `revamp phases 4-7/05b-intelligence.md` | **CANONICAL** · drafted — `Tracker` + refresh hot loop + materiality split | +| 6 | `revamp phases 4-7/06-triggers.md` | **CANONICAL** · drafted — refresh clock; recurrence = CI action on automations | +| 7 | `revamp phases 4-7/07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent | +| — | `revamp phases 4-7/00-overview.md` · `00b-pipeline-diagrams.md` | revamp map + reconciliation + end-to-end flow diagrams | +| 4 | `04a-connector-category.md` *(flat)* | ↪ **REDIRECT** — demoted to backward-compat hygiene; superseded by `revamp/04a`+`04b` | +| 4 | `04b-source-discovery.md` *(flat)* | ↪ **REDIRECT** — absorbed as the `web.discover` verb in `revamp/04a` | +| 5 | `05-pipelines-model.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05a`+`05b` (Tracker/Timeline) | +| 6 | `06-pipelines-exec.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05b` (hot loop) + `revamp/06` (triggers) | +| 7 | `07-upload-pipeline-kb.md` *(flat)* | ❌ **SUPERSEDED** — uploads populate the input-only KB (no pipeline wrapper) | Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases"). diff --git a/plans/backend/03b-proxy-expansion.md b/plans/backend/03b-proxy-expansion.md index 44ce57017..64de97ac8 100644 --- a/plans/backend/03b-proxy-expansion.md +++ b/plans/backend/03b-proxy-expansion.md @@ -53,7 +53,7 @@ Both are **publicly importable**: `from scrapling.engines.toolbelt import ProxyR The user expects a **single proxy provider across the entire app**, so 03b's selection scope is simply the **global, env-configured provider** — no per-connector or per-crawl override is built now. That is both the fastest path and, with one thin seam, the most scalable later: - All resolution stays behind today's `get_active_provider()` (env-selected, process-cached). One provider, app-wide. -- "Scalable in future": if per-pipeline proxying is ever wanted (Phase 5+), it layers on as an **optional argument** to a resolver (`resolve_proxy(override=pipeline.config)`) with **zero** changes to existing call sites — but it is explicitly **not** implemented in 03b (YAGNI for a single-provider app). +- "Scalable in future": if per-capability / per-Tracker proxying is ever wanted (revamp Phase 4a/5+), it layers on as an **optional argument** to a resolver (`resolve_proxy(override=...)`) with **zero** changes to existing call sites — but it is explicitly **not** implemented in 03b (YAGNI for a single-provider app). ## Target design @@ -92,7 +92,7 @@ Because there is one global provider and rotation lives **inside** it, the crawl - All three Scrapling tiers keep calling `get_proxy_url()` (which yields the rotating value when pool-backed). The only crawler edit is the bounded `is_proxy_error` retry from §4. - **No change** to YouTube/chat-tool consumers — same zero-arg getters, same global provider. -- Future seam (NOT built): if per-pipeline proxying is ever needed, resolve a provider from `pipeline.config` and pass its `get_proxy_url` into the crawl — an additive optional arg, no change to today's call sites. +- Future seam (NOT built): if per-capability / per-Tracker proxying is ever needed, resolve a provider from the capability/Tracker config and pass its `get_proxy_url` into the crawl — an additive optional arg, no change to today's call sites. ## Config / env changes @@ -110,18 +110,18 @@ Because there is one global provider and rotation lives **inside** it, the crawl - **Backward compatibility of the getters.** The zero-arg `get_proxy_url()` contract is consumed in 4+ places; the design keeps it **completely intact** — no signature changes, no per-caller keys. Only the cached active provider's *implementation* changes when `PROXY_PROVIDER="custom"`. - **Over-rotation cost.** Rotation-retry is bounded to one extra attempt per tier so a billable crawl can't silently multiply upstream proxy usage. -- **Single provider is a deliberate constraint.** One global provider app-wide (resolved). Per-pipeline/per-connector selection is intentionally deferred behind a no-op seam (§5); not built now. +- **Single provider is a deliberate constraint.** One global provider app-wide (resolved). Per-capability/per-Tracker/per-connector selection is intentionally deferred behind a no-op seam (§5); not built now. - **Pool rotation under `to_thread`.** `ProxyRotator`'s `Lock` makes the rotating `get_proxy_url()` safe to call from the browser tiers `03a` offloads via `asyncio.to_thread`. ## Resolved decisions - **Branded vendors → NONE.** Ship `CustomProxyProvider` (BYO) only; no Webshare/BrightData/Smartproxy/etc. subclasses. A user who wants a specific vendor points `CUSTOM_PROXY_URLS` at it. -- **Selection scope → single global provider, app-wide.** No per-connector/per-crawl override is built; resolution is env-only via `Config.PROXY_PROVIDER`. A future per-pipeline override is left as a no-op seam (§5) so it stays "scalable + fast to add later." +- **Selection scope → single global provider, app-wide.** No per-connector/per-crawl override is built; resolution is env-only via `Config.PROXY_PROVIDER`. A future per-capability/per-Tracker override is left as a no-op seam (§5) so it stays "scalable + fast to add later." - **Client-side rotation → built, but only active for a pool-backed `CustomProxyProvider`.** `anonymous_proxies` (server-side rotating) and single-URL custom configs skip it automatically. Rotation lives inside the provider so it's transparent to all callers. ## Out of scope (hand-offs) -- Per-**pipeline** / per-connector proxy selection → deferred (Phases 5–7 *if ever needed*); §5 leaves a no-op seam, nothing is wired now. +- Per-**capability** / per-Tracker / per-connector proxy selection → deferred (revamp Phase 4a/5 *if ever needed*); §5 leaves a no-op seam, nothing is wired now. - Branded-vendor provider subclasses → not planned (use `CustomProxyProvider`). - **Static / sticky-session proxies (future).** A later capability will add **static proxy** support — sticky IPs held for the duration of a session — most likely paired with **authenticated/account-based scraping** to bypass logged-in platforms (the deferred platform connectors: LinkedIn, Instagram, etc.). This is a *different axis* from the rotating pool here: rotation maximizes IP diversity, whereas account bypass needs IP **stability** so a session/cookie stays bound to one IP. It is additive to this design — a new `ProxyProvider` (or a "sticky" mode/flag on `CustomProxyProvider`) registered under a new `PROXY_PROVIDER` key, with no change to the zero-arg getter contract — and stays consistent with the single-provider model (the active provider would be the static one when that workflow is selected). Build it alongside the platform connectors, not in Phase 3. - Crawl credit metering (proxy cost is absorbed into the flat `$1 / 1000 successful` price, **not** metered separately) → `03c`. diff --git a/plans/backend/03c-crawl-billing.md b/plans/backend/03c-crawl-billing.md index f115ddd91..131ffc64a 100644 --- a/plans/backend/03c-crawl-billing.md +++ b/plans/backend/03c-crawl-billing.md @@ -2,6 +2,8 @@ > **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)). +> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** (revamp [`04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, **automation/recurring**, REST, MCP, external-cron. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context. + > Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`. > Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`). diff --git a/plans/backend/04a-connector-category.md b/plans/backend/04a-connector-category.md index 8f3a7c549..7ca37c2cc 100644 --- a/plans/backend/04a-connector-category.md +++ b/plans/backend/04a-connector-category.md @@ -1,5 +1,15 @@ # Phase 4a — Connector taxonomy (Type-1/Type-2) + availability gating + MCP routing fix +> ## ↪ REDIRECT / DEMOTED (2026-06-30) — superseded by the revamp +> **The canonical Phase 4 is now [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md)** (typed capability verbs + generated doors — see `00-umbrella-plan.md`). This file's connector-taxonomy work is **demoted to backward-compat hygiene**: it is **not** the core of Phase 4. Read the revamp first. +> +> What survives from this doc, and where it goes: +> - **The one genuinely useful piece — the `MCP_CONNECTOR` (BYO MCP) routing-gap fix — moves to `revamp/04b`** (the "consume-user-MCP" work). That's the actionable carry-over. +> - **Keeping legacy branded natives `MIGRATING` (off) + existing indexed KB docs searchable, and Obsidian/Circleback `DISABLED`** — still true; a small hygiene task, no longer a headline phase. +> - **`is_pipeline_eligible` is dropped** (no pipelines); **Type-1 = file/KB-input only** (Drive/Dropbox/OneDrive + uploads → input-only KB); **`WEBCRAWLER_CONNECTOR` is RETIRED** (native `scrape_webpage` tool, now a `web.scrape` capability verb). +> +> The static-registry design below is retained for reference, but do not treat "connector two-type restructure" as the Phase 4 deliverable — the deliverable is the **capability registry** in `revamp/04a`. + > Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`. > Sibling: `04b-source-discovery.md` (web-search repurposing). Precondition: Phases 1–2 (rename) live. diff --git a/plans/backend/04b-source-discovery.md b/plans/backend/04b-source-discovery.md index 08d451cd8..59b952ce7 100644 --- a/plans/backend/04b-source-discovery.md +++ b/plans/backend/04b-source-discovery.md @@ -1,7 +1,10 @@ # Phase 4b — Web-search repurposing + source-discovery endpoint -> Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`. -> Sibling: `04a-connector-category.md`. Best sequenced **after** `04a` (taxonomy in place). Precondition: Phases 1–2 (rename) live. +> ## ↪ REDIRECT / ABSORBED (2026-06-30) — superseded by the revamp +> **This subplan is absorbed into the canonical [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) as the `web.discover` verb.** The substance still holds — drop the 5 search `connector_type` values (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) + their paths (destructive migration), move SearXNG/Linkup/Baidu to **platform env** config, and rewire the chat `web_search` tool onto them — but the "source-discovery endpoint" is now a **capability verb** (`web.discover(query, top_k)`) in the registry, exposed identically via chat/REST/MCP (not a bespoke route), and its output feeds the agent (or a Tracker binding), **not** "a pipeline." Read `revamp/04a` (`web.discover`) as canonical; use the provider/migration detail here as implementation reference. +> +> Part of **Phase 4 — Capabilities & Access (backend)**. See `00-umbrella-plan.md`. +> Sibling (old): `04a-connector-category.md` (demoted to hygiene). Precondition: Phases 1–2 (rename) live. > **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name; locate code by **symbol/grep**, not the absolute line numbers cited. diff --git a/plans/backend/05-pipelines-model.md b/plans/backend/05-pipelines-model.md index 747e5c697..0f3075208 100644 --- a/plans/backend/05-pipelines-model.md +++ b/plans/backend/05-pipelines-model.md @@ -1,5 +1,8 @@ # Phase 5 — Pipelines data model (tables, schemas, migration, Zero, CRUD + run routes) +> # ❌ SUPERSEDED (2026-06-30) +> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, the CI pivot no longer uses a Pipelines concept: the KB is **input-only** (user files), the crawler/search are **platform-native tools**, connectors are **MCP**, and the recurring-fetch + run-history need is met by the existing **automations** system (`app/automations/`). There are **no `pipelines`/`pipeline_runs` tables**. This document is retained **for historical context only** — do **not** implement it. **The replacement is the canonical revamp Phase 5 — [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: the `Tracker` primitive + a 3-store delta **Timeline** (the durable CI state / moat). "Diffable run history" lives in the Timeline's `entity_changes`, **not** in `automation_runs.output`. (The earlier "Phase 5′" framing is retired.) + > Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 5). > Precondition: Phases 1–2 (rename) live, Phase 4a (connector registry) live. Siblings ahead: `06-pipelines-exec.md` (run engine + scheduling), `07-upload-pipeline-kb.md` (uploads-as-pipeline). diff --git a/plans/backend/06-pipelines-exec.md b/plans/backend/06-pipelines-exec.md index d1944cf87..d847c512a 100644 --- a/plans/backend/06-pipelines-exec.md +++ b/plans/backend/06-pipelines-exec.md @@ -1,5 +1,8 @@ # Phase 6 — Pipeline execution + scheduling (run engine, crawl billing, blob, chat context) +> # ❌ SUPERSEDED (2026-06-30) +> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no pipeline run engine or scheduler. The existing **automations** system already runs the full chat agent (with the native `web_search`/`scrape_webpage` tools) on **cron/event** triggers and persists **`automation_runs`** — which is exactly what this phase was building. Retained **for historical context only** — do **not** implement it. **Its still-relevant ideas survive in the canonical revamp:** (1) **billing crawls done outside chat** → now handled at the **capability executor** ([`revamp/04a`](revamp%20phases%204-7/04a-capabilities.md)), so recurring/automation runs meter correctly; (2) the refresh **hot loop** (crawl → diff → judge → append) → [`revamp/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md), writing the **Timeline** ([`revamp/05a`](revamp%20phases%204-7/05a-timeline.md)) — not `automation_runs.output`; (3) **recurrence + "run now"** → [`revamp/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) (reuse automations via a CI action). The `bill=False` seam and `index_crawled_urls` `folder_id`/`stats` params described below are **not needed** (no pipeline path). + > Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 6). > Precondition: Phase 5 (`05-pipelines-model.md`) live — `pipelines` / `pipeline_runs` tables, schemas, Zero, CRUD + `/run` (currently enqueues a **stub** `run_pipeline(run_id)`). Depends on `03a-crawler-core.md` (`crawl_url` SUCCESS signal + `crawls_succeeded` counter) and `03c-crawl-billing.md` (`WebCrawlCreditService`). Sibling ahead: `07-upload-pipeline-kb.md` (uploads-as-pipeline). diff --git a/plans/backend/07-upload-pipeline-kb.md b/plans/backend/07-upload-pipeline-kb.md index 4be4f8874..365e1e872 100644 --- a/plans/backend/07-upload-pipeline-kb.md +++ b/plans/backend/07-upload-pipeline-kb.md @@ -1,5 +1,8 @@ # Phase 7 — File upload as a pipeline + KB-save-secondary +> # ❌ SUPERSEDED (2026-06-30) +> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no "Uploads pipeline" and no upload `PipelineRun`. Uploads simply populate the **input-only KB** through the existing upload routes (an upload creates a `Document` — that *is* KB membership). The umbrella's KB-input invariant ("uploads always land in the KB") is inherent and needs no pipeline wrapper. Retained **for historical context only** — do **not** implement it. **The CI-relevant successor is the "context folder" idea in [`revamp phases 4-7/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: files uploaded in a CI chat go into the KB as normal (indexed), routed to a dedicated folder, and may feed the materiality judge via KB retrieval. (Canonical Phase 7 is now **Orchestration** — [`revamp/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — unrelated to uploads.) + > Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 7) — the final backend phase. > Precondition: Phase 5 (`05-pipelines-model.md`) live (`pipelines`/`pipeline_runs` tables, `PipelineRunTrigger.UPLOAD`, `connector_id` nullable) and Phase 6 (`06-pipelines-exec.md`) live (run engine, which **explicitly fails `connector_id IS NULL` runs**). No sibling ahead — this closes the backend umbrella. diff --git a/plans/backend/revamp phases 4-7/00-overview.md b/plans/backend/revamp phases 4-7/00-overview.md index 8b6846c98..b2c2578fd 100644 --- a/plans/backend/revamp phases 4-7/00-overview.md +++ b/plans/backend/revamp phases 4-7/00-overview.md @@ -6,6 +6,18 @@ > stateful paths) live in `00b-pipeline-diagrams.md`.** > **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing). > The revamp is **Phase 4 → end only**. We do **not** touch 1–3. +> +> **Citation verification — 2026-06-30 (principal-engineer pass).** All load-bearing code citations in this +> revamp were re-checked against `surfsense_backend`. Confirmed accurate: the schedule selector +> (`selector.py` — `FOR UPDATE SKIP LOCKED`, `next_fire_at`, self-heal, `catchup=False`, `croniter`), the +> `AutomationRun` model/fields, `format_to_structured_document(exclude_metadata=True)` content-hashing, the +> MCP routing gap (`constants.py::CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS`), the connector enum, and the +> `MANUAL` trigger placeholder. **Corrected during this pass:** (a) automations have **no** delivery/ +> notification path and `automation_runs.output` is never written — CI alerts wire to `app/notifications/` +> (`06`/`05b`); (b) the automation PENDING-gate is **not atomic** — the per-Tracker lock is the real +> concurrency guard (`06`/`05b`); (c) folder upload uses `root_folder_id` (not `destination_folder`) and KB +> folder scoping goes through `referenced_document_ids → SearchScope.document_ids` (`05b`); (d) the billable +> predicate is `SUCCESS and outcome.result`, to be single-sourced in the executor (`04a`). ## The two products @@ -64,8 +76,8 @@ data is out; *user uploads* still work (and gain a CI context-folder role). understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot (A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only on the REST/MCP dev doors. -- MVP surface = Google Maps actor + general web crawler, exposed chat + REST (MCP fast-follow). -- Verbs namespaced per platform (`web.*`, `maps.*`); `web.scrape` takes a URL **array** (inline-or-job). +- MVP surface = the **general web crawler + search** (`web.scrape` + `web.discover`), exposed chat + REST (MCP fast-follow). **Platform-specific scrapers are a family of individual endpoints added incrementally — none (incl. Google Maps) is committed for MVP;** each is just another verb → agent tool + dev API-key REST endpoint. +- Verbs namespaced per platform (`web.*`, and `.*` per scraper, e.g. `maps.*` as an example); `web.scrape` takes a URL **array** (inline-or-job). - **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state. - CI is **decoupled from automations** (automations = one optional Trigger adapter). - Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity diff --git a/plans/backend/revamp phases 4-7/04a-capabilities.md b/plans/backend/revamp phases 4-7/04a-capabilities.md index 8d7a01a4b..44e3d3928 100644 --- a/plans/backend/revamp phases 4-7/04a-capabilities.md +++ b/plans/backend/revamp phases 4-7/04a-capabilities.md @@ -19,8 +19,10 @@ back from* — no `SearchSourceConnector` row to configure, no KB write, no sche ## Current state (cited) - `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content, - metadata, crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS` - (`app/proprietary/web_crawler/`). + metadata, crawler_type}` on `SUCCESS`. `CrawlOutcomeStatus.SUCCESS` documents the billable signal, but + **operationally every caller checks `status == SUCCESS and outcome.result`** (see `scrape_webpage.py` and + `webcrawler_indexer.py`) — the predicate is duplicated, not one helper. **The executor (below) should + single-source it** in a shared `is_billable(outcome)` helper (`app/proprietary/web_crawler/`). - `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` / `successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`). - Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`). @@ -32,8 +34,16 @@ back from* — no `SearchSourceConnector` row to configure, no KB write, no sche ### The verb set (MVP) — namespaced, nothing top-level -Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor). -Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …). +Two namespaces: **`web.*`** (generic crawler product) and **`.*`** (per-platform scrapers). +Future platforms slot in as their own namespace (`maps.*`, `linkedin.*`, `amazon.*`, …). + +> **Scope clarification (2026-06-30).** Platform-specific scrapers are a **family of individual scraping +> endpoints**, **not** one committed integration — **no specific platform (incl. Google Maps) is committed +> for MVP**; `maps.*` below is an **illustrative example** of the pattern. Each such scraper is *just +> another capability verb*: adding one lights up (a) an **agent tool** (chat) and (b) a **dev-callable REST +> endpoint behind the platform API key** — same executor, same billing, zero new machinery. **The MVP +> builds only `web.scrape` + `web.discover` executors**; every `.*` row is a **contract stub** +> demonstrating that the registry + generated doors (`04b`) make per-platform endpoints a drop-in. | Verb | Input → Output | Mode | Executes over | Bills (03c) | |------|----------------|------|---------------|-------------| @@ -101,9 +111,20 @@ A capability only **declares a `billing_unit`** in its registry entry; charging The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs stay pure; pricing stays pluggable. -### The Maps actor (clarification) +> **Charge at the executor, not the turn accumulator (locked 2026-06-30).** The billable charge fires +> **inside the capability executor** (this phase), so *every* caller meters identically — chat, +> **automation/recurring**, REST, MCP, external-cron. This is deliberate: a code review confirmed the +> automation path (`run_agent_task` → `agent_task`) establishes **no chat turn accumulator**, so the +> existing `scrape_webpage` billing (which no-ops when `get_current_accumulator()` is `None`) would let +> **automation-run crawls bill nothing**. The interactive chat **turn accumulator stays only as an +> optional presentation fold** (so a chat turn still shows the crawl line on its bill) — it is **not** the +> charging mechanism. Billing idempotency is **per capability call** (+ the `05b` content-hash pre-check), +> not per run. -- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). +### Platform scrapers (illustrated with a Maps actor — example only, not committed) + +- Each platform scraper is a **standalone endpoint**; the one shown here (a hypothetical Maps actor) is + **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). - Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw markdown — exactly what the Intelligence/Timeline layer needs to diff reliably. - The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on diff --git a/plans/backend/revamp phases 4-7/05b-intelligence.md b/plans/backend/revamp phases 4-7/05b-intelligence.md index 97b915d53..e2763627f 100644 --- a/plans/backend/revamp phases 4-7/05b-intelligence.md +++ b/plans/backend/revamp phases 4-7/05b-intelligence.md @@ -24,10 +24,16 @@ STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the - **Capability executors** (`04a`) — called directly by the loop (not through a door). - **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)` produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`). -- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`) - + the automations executor's PENDING→running idempotency gate (safe under Celery `acks_late`); and the - chat **job record** (`04a`) + `deliverable_wait`. **No new run table.** -- **CI context uploads** — the existing folder-upload / `destination_folder` machinery + KB retrieval. +- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`; + note the executor writes results to `step_results[n].result`, **not** `run.output`/`artifacts`, which stay + unwritten) + the automations executor's PENDING→running gate (`execute_run()` early-returns unless + `status == PENDING`; `acks_late` is global). **Caveat:** the gate is **not atomic** (no + `UPDATE … WHERE status='pending'`), so it stops terminal/post-`mark_running` redelivery but **not** + concurrent double-claim — pair it with the per-Tracker lock (`06`). Plus the chat **job record** (`04a`) + + `deliverable_wait`. **No new run table.** +- **CI context uploads** — the existing folder-upload machinery (`POST /documents/folder-upload`, param + `root_folder_id` — there is no `destination_folder`) + KB retrieval scoped via mention-pins → + `referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids` (no direct `folder_id` on search). ## Target design @@ -90,8 +96,9 @@ description reworded → code: no rule → ASK AGENT → NOISE surface invoked it**: - **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`** - is the run record and the automations executor provides the **PENDING→running idempotency gate**. This is - exactly the rigor old `06` hand-built — reused, not re-written. + is the run record and the automations executor's PENDING-gate handles redelivery. Reused, not re-written — + but the gate is **not atomic** against concurrent claims, so the **per-Tracker lock (`06`)** is the actual + double-refresh guard. - **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`. - **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No @@ -105,8 +112,11 @@ When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes i uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies only to *crawled* data.)** The CI-specific part is **organization + use**: -- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload machinery). -- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder: +- **Routed to a dedicated folder** for that CI chat/Tracker (reuse `POST /documents/folder-upload` with + `root_folder_id`; uploads land as `Document`s with `folder_id`). +- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder via the mention-pin + path (`referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids`), since search has no direct + `folder_id` filter: ``` competitor price 12.00 → 9.90 + user's context file says "our price is 10.00" @@ -158,8 +168,9 @@ retrieval machinery (nothing new). **MVP-optional** (the loop works without it); 5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. 6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. 7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic. -8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job record; - billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state. +8. **No new run table** — refresh audit rides `AutomationRun` (recurring) or the chat job record; redelivery + is stopped by the PENDING-gate **plus a per-Tracker lock** (the gate is non-atomic); billing idempotency + is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state. 9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled* data only. MVP-optional seam. diff --git a/plans/backend/revamp phases 4-7/06-triggers.md b/plans/backend/revamp phases 4-7/06-triggers.md index 00dd5008c..b35e90d37 100644 --- a/plans/backend/revamp phases 4-7/06-triggers.md +++ b/plans/backend/revamp phases 4-7/06-triggers.md @@ -3,7 +3,7 @@ > **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes > `refresh(tracker)`). > **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED -> automations subsystem (its schedule selector + `AutomationRun` + delivery). +> automations subsystem (its schedule selector + `AutomationRun`) plus `app/notifications/` for alert delivery. > **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no > dependency on it, and **automations is at most one optional adapter, never required**. @@ -19,9 +19,20 @@ rebuild a scheduler at all, but to **reuse the automations subsystem** for the i - **`refresh(tracker)`** — the headless unit of work exposed by `05b`. - **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED` claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the - `croniter` util. -- **`AutomationRun`** — the existing run record + PENDING-gate (audit + idempotency). -- **Automations output/delivery** — the existing path that carries results to the user. + `croniter` util. *(Verified: `app/automations/triggers/builtin/schedule/selector.py::_claim_due_triggers` + + `schedule/cron.py::compute_next_fire_at`.)* +- **`AutomationRun`** — the existing run record + a PENDING-gate. *(Verified with a caveat: the gate is + `execute_run()` early-returning unless `status == PENDING` + `mark_running`, and `acks_late` / + `task_reject_on_worker_lost` are set globally — so **terminal redelivery and post-`mark_running` + redelivery are no-ops**. But there is **no atomic `UPDATE … WHERE status='pending'` claim**, so two + concurrent workers can both read PENDING → double-run. The per-Tracker lock below is therefore the + primary concurrency guard, not just belt-and-suspenders.)* +- **Delivery ≠ automations (correction).** Automations have **no** push/email/notification path, and + `automation_runs.output` is **never written** by the executor (results live in `step_results[n].result`); + today "delivery" is only **pull via `GET /automations/{id}/runs/{run_id}`** + **Zero dashboard sync** + (`zero_publication.AUTOMATION_RUN_COLS`). In-app alerts must wire to the **separate `app/notifications/` + system** — `NotificationService.create_notification(...)` (DB-backed, Zero-synced, typed + `NotificationType`). So alert delivery is a **small net-new wiring**, not a free automations reuse. - **Access routes** (`04b`) — where the external-cron and manual REST endpoints live. ## Target design @@ -33,7 +44,7 @@ rebuild a scheduler at all, but to **reuse the automations subsystem** for the i | **Manual** | user "refresh now" (chat tool / REST) | ✅ | | **Agent** | the in-app agent calls `refresh` as a tool | ✅ | | **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) | -| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)` **and delivers** the material changes | ✅ (the in-app recurrence + alert path) | +| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)`, persists deltas to the Timeline, **and emits an in-app alert** via `app/notifications/` | ✅ (the in-app recurrence + alert path) | ### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) @@ -43,9 +54,12 @@ building a bespoke tick, we **add a CI *action* to the existing automations subs - **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new scheduler. **(closes old Gap B — scheduler rigor.)** - **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run - table. **(closes old Gap A — run/idempotency, see `05b`.)** -- **Delivery / alert** → the automation's existing **output/delivery** carries the material changes to the - user. **(closes old Gap E — alert delivery.)** + table. **(closes old Gap A — run/idempotency)** — *but* pair it with the per-Tracker lock (the gate is + non-atomic; see Current state + `05b`). +- **Delivery / alert** → **not** free from automations. Persist the material change to the **Timeline** + (`05a`), then emit an in-app alert via **`app/notifications/`** (`NotificationService.create_notification` + with a new CI notification type; Zero-synced to the UI). **(old Gap E — alert delivery — is *mostly* + closed but needs this small wiring, not zero-cost reuse.)** **Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action @@ -54,8 +68,9 @@ too constraining, a thin CI-specific shape is the fallback — but the action is ### Decoupling is preserved (automations is still optional) CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs -via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + delivery -+ audit for in-app users. So we honor "don't glue CI to automations" *and* get its machinery for free. +via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + audit +for in-app users (alerts ride `app/notifications/`, not automations). So we honor "don't glue CI to +automations" *and* get its machinery for free. ### Where it lives @@ -69,8 +84,12 @@ via manual / agent / external-cron. Automations is the **optional adapter** that 1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`. 2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`. 3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that - calls `refresh(tracker)` and routes the resulting material changes into automations delivery. -4. **Concurrency guard**: a per-Tracker in-flight lock (belt-and-suspenders over the automation run-gate). + calls `refresh(tracker)` and, on material change, emits an in-app alert via `app/notifications/` + (`NotificationService.create_notification`, new CI notification type). *(Not "automations delivery" — + that path doesn't exist; see Current state.)* +4. **Concurrency guard**: a per-Tracker in-flight lock — the **primary** guard against concurrent + double-refresh, since the automation PENDING-gate is non-atomic (also protects the manual/cron paths + that don't go through automations). ## Tests @@ -94,7 +113,8 @@ via manual / agent / external-cron. Automations is the **optional adapter** that 1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled. 2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery). 3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule - selector + `AutomationRun`; delivery reuses automations' output. (Closes old Gaps A/B/E.) + selector + `AutomationRun` (Closes old Gaps A/B). **Delivery does NOT reuse automations** (no such + path; `run.output` is unwritten) — alerts go through `app/notifications/` (Gap E: small wiring). 4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**. 5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron). diff --git a/plans/backend/revamp phases 4-7/07-orchestration.md b/plans/backend/revamp phases 4-7/07-orchestration.md index 13e8f4654..f8b3cc142 100644 --- a/plans/backend/revamp phases 4-7/07-orchestration.md +++ b/plans/backend/revamp phases 4-7/07-orchestration.md @@ -21,9 +21,12 @@ where "user-friendly" is won or lost, so the orchestration layer is **designed, `pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py` (`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md` (the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`. -- **Tool shape** — `research/tools/scrape_webpage.py` already demonstrates **tool = capability executor + - access door + 03c billing**: calls `WebCrawlerConnector.crawl_url`, bills via the turn accumulator - (`get_current_accumulator()` + `WebCrawlCreditService`), returns a typed dict. CI tools follow this exactly. +- **Tool shape** — `research/tools/scrape_webpage.py` shows the **tool = capability executor + access door** + pattern (calls `WebCrawlerConnector.crawl_url`, returns a typed dict). **Billing note:** its current + `get_current_accumulator()` fold is **interactive-chat-only** and no-ops outside a chat turn — so CI tools + do **not** copy that as the charging path. Per `04a` (locked), **the charge fires in the capability + executor**; the chat turn accumulator stays only as an optional presentation fold. This is what makes the + recurring `06 → refresh → verb → crawl` path (no chat turn) bill correctly. - **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`). ## Target design @@ -49,7 +52,7 @@ happen. It owns the **CI playbook** in `system_prompt.md`: | Tool | Wraps | Mode | Billing | |------|-------|------|---------| -| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | `03c` turn accumulator (as `scrape_webpage` already does) | +| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | at the `04a` executor (per-call); chat turn accumulator = optional presentation fold only | | `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed | | `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — | | `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call | @@ -103,7 +106,7 @@ happen. It owns the **CI playbook** in `system_prompt.md`: 1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not a runtime rebuild. -2. Tools follow the `scrape_webpage` shape: capability executor + access door + `03c` billing. +2. Tools follow the `scrape_webpage` executor+door shape, but **billing fires in the `04a` executor** (not the chat turn accumulator, which is interactive-only and would skip automation/cron runs). 3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds Tracker/Timeline tools + the CI prompt. 4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`.