From 43893fc1f8112fbb885488a9b7e94681906b4d6e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Thu, 2 Jul 2026 19:05:45 +0200 Subject: [PATCH] docs(plans): drop remaining intelligence_agent references --- plans/backend/00-umbrella-plan.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index 9037793dd..cfcda1489 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -38,7 +38,7 @@ Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / > - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.* > - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine. > - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path). -> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `intelligence_agent` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language. +> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `scraping` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language. > > **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors. @@ -209,8 +209,8 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta ## Resolved decisions log -- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`intelligence_agent`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below. -- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. +- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`scraping`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below. +- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`scraping`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`. - **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths). - ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type. - Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).