diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md index 300cd9e68..9037793dd 100644 --- a/plans/backend/00-umbrella-plan.md +++ b/plans/backend/00-umbrella-plan.md @@ -20,7 +20,7 @@ Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / > - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6). > - **KB stays input-only**; crawled data is never indexed. > -> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** (deferred) · **07 Orchestration** (`intelligence_agent`). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`. +> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** · **07 Orchestration** (`scraping` subagent). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`. ## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted @@ -52,11 +52,11 @@ flowchart TD REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"] ACQ --> REG REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"] - DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · 'what changed' from chat history"] + DOORS --> AGENT["scraping subagent (P7): intent routing · verb chains · 'what changed' from chat history"] KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"] - AGENT -.->|standing need| ONG["Ongoing-Automation (P6, DEFERRED): persistent chat re-invokes verbs; memory = chat history"] + AGENT -.->|standing need| ONG["Ongoing-Automation (P6): persistent chat re-invokes verbs; memory = chat history"] ONG -.-> REG APIKEY["Platform API key"] --> DOORS ``` @@ -168,16 +168,16 @@ The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). ### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`] -> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **⚠️ design deferred.** The periodic mechanism is designed separately, after `04`/`05`/`07`. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel). +> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **shipped.** A chat watch is an `Automation` bound to the current chat (`schedule` + `chat_message`); the durable checkpointer is the memory. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel). - Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls. ### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`] -> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Build last** (atop `04`/`05`, and `06` for the ongoing mode). We plug into the shipped multi-agent runtime; we don't rebuild it. +> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Shipped for the web verbs** (atop `04`/`05`/`06`). We plug into the shipped multi-agent runtime; we don't rebuild it. -- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store). -- Toolset: registry-backed capability verbs (shared with `research`) + a deferred `start_watch` handoff to `06`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`). +- Ship the builtin **`scraping`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, block on ambiguous cadence), **verb composition** (`web.discover → web.scrape`; `maps.*` chains deferred until the Maps actor exists), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store). +- Toolset: registry-backed capability verbs + `start_watch`/`stop_watch`/`refresh_watch` bound to the current chat (`06`). Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`). ### (Future) Platform API-key + public MCP @@ -249,9 +249,9 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta | 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default | | 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff | | — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) | -| 4 | `04-capabilities.md` | **CANONICAL** · drafted — capability registry + verbs `web.scrape`/`web.discover`/`maps.*` → cleaned data; bill at executor. **← build next** | -| 5 | `05-access.md` | **CANONICAL** · drafted — generated REST/MCP/chat doors + intent router + BYO-MCP routing fix | -| 6 | `06-ongoing-automation.md` | **CANONICAL** · **design deferred** — chat-native "keep watching" (periodic re-invocation) | -| 7 | `07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent; "what changed" from chat history | +| 4 | `04-capabilities.md` | **IMPLEMENTED (web verbs)** — capability registry + `web.scrape`/`web.discover` → cleaned data, bill at executor (top_k bounds, max_length truncation, per-attempt captcha metering). `maps.*` deferred (Maps actor) | +| 5 | `05-access.md` | **IMPLEMENTED (REST + chat doors)** — generated REST/API-key + chat tools from the registry. MCP door + BYO-MCP routing fix deferred (MCP server) | +| 6 | `06-ongoing-automation.md` | **IMPLEMENTED** — chat-native "keep watching": `Automation` bound to the chat (`schedule` + `chat_message`), worker-safe durable checkpointer, watch tools + REST controls | +| 7 | `07-orchestration.md` | **IMPLEMENTED (web verbs)** — `scraping` subagent; intent routing, verb composition, "what changed" from chat history | Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases"). diff --git a/plans/backend/00b-pipeline-diagrams.md b/plans/backend/00b-pipeline-diagrams.md index d2ffb8be7..b199fbf1b 100644 --- a/plans/backend/00b-pipeline-diagrams.md +++ b/plans/backend/00b-pipeline-diagrams.md @@ -16,8 +16,8 @@ flowchart LR subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] CAP["04 Capabilities
web.* · maps.*
cleaned, AI-ready output"] ACC["05 Access
REST · MCP · chat doors"] - ORC["07 Orchestration
intelligence_agent"] - ONG["06 Ongoing-Automation
(keep-watching · DEFERRED)"] + ORC["07 Orchestration
scraping subagent"] + ONG["06 Ongoing-Automation
(keep-watching)"] end ACQ --> CAP @@ -77,20 +77,19 @@ sequenceDiagram ```mermaid flowchart TD - U(["User speaks in natural language"]) --> R{"Intent router
(intelligence_agent prompt, 07)"} + U(["User speaks in natural language"]) --> R{"Intent router
(scraping prompt, 07)"} R -->|"find / compare / what is / right now"| A["ONE-SHOT
compose verbs, answer (section 3)"] - R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING
hand to Ongoing-Automation (06)"] - R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] + R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING
start_watch -> Ongoing-Automation (06)"] + R -->|ambiguous cadence| Q["Block: ask for the missing cadence / timezone"] Q -->|once| A Q -->|keep watching| B - B -.->|DESIGN DEFERRED| X["06 mechanism TBD
(periodicity · delivery · context limits)"] ``` ## 5. "What changed" — chat history is the memory ```mermaid flowchart LR - P["Prior tool outputs
(already in chat context)"] --> AG["intelligence_agent"] + P["Prior tool outputs
(already in chat context)"] --> AG["scraping subagent"] NEW["Fresh verb call (04)"] --> AG AG --> ANS["'Here's what's new since last time'
(reasoned from chat history)"] ``` diff --git a/plans/backend/06-ongoing-automation.md b/plans/backend/06-ongoing-automation.md index 40c74c6e7..8006b4a22 100644 --- a/plans/backend/06-ongoing-automation.md +++ b/plans/backend/06-ongoing-automation.md @@ -1,6 +1,6 @@ # Phase 6 — Ongoing Automation (chat-native "keep watching") -> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `intelligence_agent`). +> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `scraping` subagent). > Reuses the existing chat + automations machinery; adds no parallel engine. ## Objective @@ -28,7 +28,7 @@ chat's own checkpoint is the memory. opened them, but Celery uses a fresh loop per task (`PoolTimeout`). Dispose the checkpointer pool per task in `run_async_celery_task`, mirroring `_dispose_shared_db_engine`, so a worker can use the *durable* checkpointer (not `InMemorySaver`) that "what changed" requires. -- **`start_watch`** — an `intelligence_agent` tool that binds a watch to the *current* chat: it distills +- **`start_watch`** — a `scraping` subagent tool that binds a watch to the *current* chat: it distills the recurring question + cadence and creates the automation (`schedule` + `chat_message(thread_id = current chat)`). - **"Is this chat watched?"** — derived: an active automation with a `chat_message` action targeting the @@ -44,7 +44,7 @@ chat's own checkpoint is the memory. 2. `chat_message` action: params + factory + handler (drains `stream_new_chat`); concurrency guard. **[done]** 3. Watch service: create (bind `schedule` + `chat_message` automation to a chat) / stop (delete) / find-for-thread (is-watched) / run-now. **[done]** -4. `start_watch` tool on `intelligence_agent` (+ prompt line); binds to the current chat. **[done]** +4. `start_watch` tool on the `scraping` subagent (+ prompt line); binds to the current chat. **[done]** 5. Controls — chat tools (`stop_watch`, `refresh_watch`) + REST (`GET /automations/watches`, `POST /automations/{id}/run`; stop = `DELETE /automations/{id}`). **[done]** diff --git a/plans/backend/07-orchestration.md b/plans/backend/07-orchestration.md index 82754de18..50f3e6d9a 100644 --- a/plans/backend/07-orchestration.md +++ b/plans/backend/07-orchestration.md @@ -1,19 +1,22 @@ # Phase 7 — Orchestration / Conversation (the CI-expert subagent) -> Build last; sits atop `04`/`05` (and `06` for the ongoing mode). -> The multi-agent chat runtime (deepagents, subagent dispatch, streaming, citation middleware) is shipped. -> Net-new here = one builtin subagent + its tools + its prompt. Locate code by symbol/grep. +> **Status (2026-07-02): shipped for the web verbs.** The `scraping` subagent, its tools, prompt, and +> router blurb are live; `web.discover`/`web.scrape` compose and the "keep watching" handoff (`06`) is +> wired. `maps.*` composition is deferred until the Maps actor exists. +> Sits atop `04`/`05` (and `06` for the ongoing mode). The multi-agent chat runtime (deepagents, subagent +> dispatch, streaming, citation middleware) is shipped. Net-new here = one builtin subagent + its tools + +> its prompt. Locate code by symbol/grep. ## Objective -Ship the `intelligence_agent` — a builtin CI/scraper-expert subagent that delivers the product through +Ship the `scraping` subagent — a builtin CI/scraper-expert that delivers the product through plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's new vs prior tool outputs already in context. ## The subagent -New builtin `subagents/builtins/intelligence_agent/`, peer to `research`/`deliverables`, packed via +Builtin `subagents/builtins/scraping/`, peer to `research`/`deliverables`, packed via `pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py` (`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook). The main agent delegates CI-/scraping-flavored requests to it via `description.md`. @@ -32,11 +35,11 @@ The main agent delegates CI-/scraping-flavored requests to it via `description.m | Tool | Wraps | Billing | |------|-------|---------| -| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) | -| `start_watch(intent)` *(deferred)* | hands off to `06` | per re-invocation (design with `06`) | +| capability verbs — `web.scrape`, `web.discover` shipped; `maps.search`/`maps.place`/`maps.reviews` deferred (Maps actor) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) | +| `start_watch` / `stop_watch` / `refresh_watch` | bind a watch to the current chat via `06` | per re-invocation | Verbs reach the subagent through the chat door generator `access/chat.py::build_capability_tools` (`05`), -which turns registry verbs (`04`) into tools with owner-wallet billing. The `intelligence_agent` owns these +which turns registry verbs (`04`) into tools with owner-wallet billing. The `scraping` subagent owns these CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched. ## Boundaries @@ -47,16 +50,16 @@ CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched ## Work items -1. Subagent scaffold: `subagents/builtins/intelligence_agent/` (`agent.py` / `tools/index.py` / +1. **[done]** Subagent scaffold: `subagents/builtins/scraping/` (`agent.py` / `tools/index.py` / `description.md` / `system_prompt.md`), packed via `pack_subagent`. -2. CI playbook prompt: intent routing (one clarifying question), verb chains, "what changed" reasoning. -3. Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`). -4. `start_watch` seam to `06` (wired when `06` is designed). -5. Router registration: `description.md` for main-agent delegation. +2. **[done]** CI playbook prompt: intent routing (block on ambiguous cadence), verb chains, "what changed" reasoning. +3. **[done]** Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`). +4. **[done]** `start_watch` / `stop_watch` / `refresh_watch` seam to `06`, bound to the current chat. +5. **[done]** Router registration: `description.md` for main-agent delegation. ## Tests -- A CI-/scraping-flavored request routes to `intelligence_agent`; a non-CI request does not. +- A CI-/scraping-flavored request routes to `scraping`; a non-CI request does not. - "compare X and Y" composes verbs and answers in plain language, persisting nothing. - "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`. - A follow-up re-invokes the verb and reports new items vs prior in-context outputs. @@ -70,4 +73,4 @@ CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched ## Open questions -- The `start_watch` handoff shape (settled with `06`). +- None outstanding for the web verbs. `maps.*` composition reopens when the Maps actor lands.