SurfSense/plans/backend/07-orchestration.md

# Phase 7 — Orchestration / Conversation (the CI-expert subagent)

> **Status (2026-07-02): shipped for the web verbs.** The `scraping` subagent, its tools, prompt, and
> router blurb are live; `web.discover`/`web.scrape` compose and the "keep watching" handoff (`06`) is
> wired. `maps.*` composition is deferred until the Maps actor exists.
> Sits atop `04`/`05` (and `06` for the ongoing mode). The multi-agent chat runtime (deepagents, subagent
> dispatch, streaming, citation middleware) is shipped. Net-new here = one builtin subagent + its tools +
> its prompt. Locate code by symbol/grep.

## Objective

Ship the `scraping` subagent — a builtin CI/scraper-expert that delivers the product through
plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand
standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's
new vs prior tool outputs already in context.

## The subagent

Builtin `subagents/builtins/scraping/`, peer to `research`/`deliverables`, packed via
`pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py`
(`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook).
The main agent delegates CI-/scraping-flavored requests to it via `description.md`.

## The playbook (`system_prompt.md`)

1. **Intent routing** — one-shot ("compare/find/what is") → compose verbs & answer; standing concern
   ("watch/track/notify when/weekly") → hand to `06`; ambiguous → ask one question ("just once, or keep
   watching?").
2. **Verb composition** — chains `web.discover → web.scrape` and `maps.search → maps.place → maps.reviews`;
   infer URLs/queries/locations from context.
3. **"What changed"** — re-invoke the verb and compare against prior tool outputs in the chat history,
   summarizing what's new.

## The toolset (`load_tools`)

| Tool | Wraps | Billing |
|------|-------|---------|
| capability verbs — `web.scrape`, `web.discover` shipped; `maps.search`/`maps.place`/`maps.reviews` deferred (Maps actor) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) |
| `start_watch` / `stop_watch` / `refresh_watch` | bind a watch to the current chat via `06` | per re-invocation |

Verbs reach the subagent through the chat door generator `access/chat.py::build_capability_tools` (`05`),
which turns registry verbs (`04`) into tools with owner-wallet billing. The `scraping` subagent owns these
CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched.

## Boundaries

- The subagent drives the `04` verbs via tools; fetch/bypass/clean logic stays in Acquisition + `04`,
  callable headless.
- Humans get the agent; REST/MCP callers call `04` verbs directly, skipping this subagent.

## Work items

1. **[done]** Subagent scaffold: `subagents/builtins/scraping/` (`agent.py` / `tools/index.py` /
   `description.md` / `system_prompt.md`), packed via `pack_subagent`.
2. **[done]** CI playbook prompt: intent routing (block on ambiguous cadence), verb chains, "what changed" reasoning.
3. **[done]** Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`).
4. **[done]** `start_watch` / `stop_watch` / `refresh_watch` seam to `06`, bound to the current chat.
5. **[done]** Router registration: `description.md` for main-agent delegation.

## Tests

- A CI-/scraping-flavored request routes to `scraping`; a non-CI request does not.
- "compare X and Y" composes verbs and answers in plain language, persisting nothing.
- "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`.
- A follow-up re-invokes the verb and reports new items vs prior in-context outputs.
- "watch this weekly" routes to the ongoing mode, not a one-shot answer.

## Out of scope

- Verbs, doors → `04`/`05`. Ongoing/periodic mechanism → `06`.
- Richer multi-step playbooks, proactive suggestions, cross-topic synthesis.
- Frontend CI surfaces → frontend umbrella.

## Open questions

- None outstanding for the web verbs. `maps.*` composition reopens when the Maps actor lands.