diff --git a/plans/backend/revamp phases 4-7/00-overview.md b/plans/backend/revamp phases 4-7/00-overview.md new file mode 100644 index 000000000..8b6846c98 --- /dev/null +++ b/plans/backend/revamp phases 4-7/00-overview.md @@ -0,0 +1,113 @@ +# Phase 4 → end revamp — Overview & phase index (CI pivot · WIP) + +> **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small, +> single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace. +> This file is the map + the reconciliation + the subplan index. **End-to-end flow diagrams (stateless & +> stateful paths) live in `00b-pipeline-diagrams.md`.** +> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing). +> The revamp is **Phase 4 → end only**. We do **not** touch 1–3. + +## The two products + +``` +PRODUCT A — stateless utility Phase 04 (Capabilities + Access) + call a verb → get data → bill. Nothing persists. + +PRODUCT B — decision-grounded CI Phase 05 (Intelligence + Timeline) (+ 06 Triggers, 07 Orchestration) + a Tracker accumulates structured signal over time. The Timeline is the moat. +``` + +## The phases (this revamp) + +| Phase | Subplan(s) | Domain(s) | One line | Ships | +|-------|-----------|-----------|----------|-------| +| **04 — Capabilities & Access** | `04a-capabilities.md` · `04b-access.md` | ① + ② | typed verbs over Acquisition + the chat/REST/MCP doors | **Product A** (revenue day one) | +| **05 — Intelligence & Timeline** | `05a-timeline.md` · `05b-intelligence.md` | ④ + ③ | the Tracker, locked schema, hot loop + the 3-store delta moat | **Product B engine** | +| **06 — Triggers** | `06-triggers.md` | ⑤ | the pluggable refresh clock; recurrence+delivery = optional CI action on automations | recurrence + alerts | +| **07 — Orchestration** | `07-orchestration.md` | ⑥ | the `intelligence_agent` CI-expert subagent (intent routing, verb chains, Tracker crafting) | the human-facing CI experience | + +> **Build order = phase order.** `04a → 04b` (unblocks everything; ships Product A) → `05a → 05b` +> (state before the loop that writes it) → `06` (drive it) → `07` (front it). `04a` first because every +> other phase calls into the capability registry. + +## Domain ↔ phase map + +``` + FIXED (Phases 1–3) OUR SCOPE (Phase 04 → 07) + ┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐ + │ Acquisition │ │ 04a Capabilities typed verbs over Acquisition │ + │ proprietary/web_crawler │◄──┤ 04b Access chat · REST · MCP doors │ + │ CrawlOutcome · billing (03c) │ │ 05a Timeline 3-store delta state (moat) │ + │ │ │ 05b Intelligence Tracker · schema · hot loop │ + │ │ │ 06 Triggers pluggable refresh clock │ + │ │ │ 07 Orchestration CI-expert subagent + tools │ + └─────────────────────────────────┘ └──────────────────────────────────────────────┘ + data engine (untouched) stateless 04 → stateful 05 , driven by 06 , fronted by 07 +``` + +## Reconciliation with the old plans + +| Old plan | Fate | Where it goes | +|----------|------|---------------| +| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in `04b` (consume-user-MCP); the taxonomy work is not core | +| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in `04a` | +| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (`05b`) + Timeline tables (`05a`); no `pipeline`/`pipeline_runs` | +| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (`05b`); scheduling/runs/delivery → **reuse automations** via a CI action (`06`) — its selector + `AutomationRun`, **not** a rebuilt cron | +| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploads are routed to a dedicated **folder** and may feed the judge (`05b`) | + +**Net:** the old `05/06/07` pipeline+KB stack is replaced by the new `04`–`07`. KB indexing of *crawled* +data is out; *user uploads* still work (and gain a CI context-folder role). + +## Confirmed decisions (locked across phases) + +- **Natural language is the only human-facing surface.** Users never name verbs; the chat agent + understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot + (A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only + on the REST/MCP dev doors. +- MVP surface = Google Maps actor + general web crawler, exposed chat + REST (MCP fast-follow). +- Verbs namespaced per platform (`web.*`, `maps.*`); `web.scrape` takes a URL **array** (inline-or-job). +- **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state. +- CI is **decoupled from automations** (automations = one optional Trigger adapter). +- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity + per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check. +- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`) on + the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools. +- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, not a + new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and delivery. + CI core still runs with **zero** automations dependency (manual/agent/cron). +- **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing + idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state. +- **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first + provider; `maps.*` / `web.discover` register their own units there. +- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a **separate + effort**; `maps.*` verbs are contracts against it and don't block this design. +- **CI context files:** files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a + dedicated `Folder`, and may feed the materiality judge via KB retrieval. ("Don't index" applies only to + *crawled* data.) +- **Names:** primitive = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent). +- License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor. + +## Cross-phase open forks (need your call) + +1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? (`05b`) +2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. (`05a`) +3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). (`06`) + +*(Resolved: the "built-in scheduler tick" fork — reuse the automations schedule selector via a CI action +instead of building a tick. See `06`.)* + +> **Per-phase implementation-time questions** (refresh failure path, first-run baseline, tracker lifecycle, +> mid-run balance, concurrency lock, delivery payload shape) are intentionally **not** resolved here — they +> don't change any boundary, contract, or data shape, and are settled when each phase is built. + +## Subplan index (revamp) + +| Phase | Subplan file | Status | +|-------|--------------|--------| +| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams | +| 04 | `04a-capabilities.md` | drafted | +| 04 | `04b-access.md` | drafted | +| 05 | `05a-timeline.md` | drafted | +| 05 | `05b-intelligence.md` | drafted | +| 06 | `06-triggers.md` | drafted | +| 07 | `07-orchestration.md` | drafted | diff --git a/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md b/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md new file mode 100644 index 000000000..6e6e011e0 --- /dev/null +++ b/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md @@ -0,0 +1,202 @@ +# Pipeline diagrams — end-to-end (stateless & stateful paths) + +> Visual companion to `00-overview.md`. Shows the whole CI pipeline across Phases 04–07, with the +> **stateless** (Product A) and **stateful** (Product B) paths called out explicitly. Phase refs: +> `04a` Capabilities · `04b` Access · `05a` Timeline · `05b` Intelligence · `06` Triggers · `07` Orchestration. + +## 1. Domain map — the two products + +```mermaid +flowchart LR + subgraph FIXED["FIXED · Phases 1-3 (shipped, untouched)"] + ACQ["Acquisition
proprietary/web_crawler
CrawlOutcome"] + MET["Metering · 03c
WebCrawlCreditService"] + ID["Identity / Tenancy
+ API keys"] + RT["Chat runtime
deepagents · deliverable_wait"] + end + + subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] + CAP["04a Capabilities
typed verbs + registry + job store"] + ACC["04b Access
chat · REST · MCP doors"] + INT["05b Intelligence
Tracker · schema · hot loop"] + TL["05a Timeline
3-store delta state (moat)"] + TRG["06 Triggers
refresh clock"] + ORC["07 Orchestration
intelligence_agent subagent"] + end + + ACQ --> CAP + MET --> CAP + CAP --> ACC + ID --> ACC + RT --> ORC + ACC --> ORC + CAP --> INT + INT --> TL + TRG --> INT + ORC --> CAP + ORC --> INT + ORC --> TL + + classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7; + classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee; + class CAP,ACC,INT,TL,TRG,ORC a; + class ACQ,MET,ID,RT b; +``` + +- **Product A (stateless):** `04a Capabilities + 04b Access` — call a verb, get data, bill, nothing persists. +- **Product B (stateful):** `05b Intelligence + 05a Timeline` (driven by `06`, fronted by `07`) — a Tracker + accumulates structured signal; the Timeline is the moat. + +## 2. The fork — intent router (the only human-facing decision) + +```mermaid +flowchart TD + U(["User speaks in natural language"]) --> R{"Intent router
(in intelligence_agent prompt, 07)"} + R -->|"compare / find / what is / pull / right now"| A["ONE-SHOT
Product A · stateless"] + R -->|"watch / track / notify me / every week / over time"| B["STANDING CONCERN
Product B · stateful"] + R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] + Q -->|once| A + Q -->|keep watching| B + A --> P1["section 3 · Stateless path"] + B --> P2["section 4 · Stateful path"] +``` + +Raw verbs are exposed only on REST/MCP (devs/external agents). Humans never name a verb. + +## 3. Stateless path (Product A) — end to end + +```mermaid +sequenceDiagram + autonumber + participant U as User / Dev + participant D as Door 04b (chat · REST · MCP) + participant REG as Registry 04a + participant EX as Verb executor 04a + participant ACQ as Acquisition / Maps actor + participant BILL as Billing service (03c) + participant JOB as Job record (+Celery) + + U->>D: intent (chat) OR raw verb call (REST/MCP) + D->>D: validate input_schema · authn/authz · meter-gate + D->>REG: resolve verb + REG->>EX: same executor (all doors) + alt fast / small input -> inline + EX->>ACQ: crawl_url / search / place + ACQ-->>EX: data + EX->>BILL: charge billing_unit per success + EX-->>D: envelope {status: completed, results} + D-->>U: plain-language answer (chat) / JSON (REST) + else slow / large input -> job + EX->>JOB: dispatch, return {status: pending, job_id} + D-->>U: chat: deliverable_wait polls status
REST/MCP: GET /v1/jobs/{id} + JOB->>ACQ: work loop + JOB->>BILL: charge per success + JOB-->>D: status -> completed (+ results) + D-->>U: results inline / tracked card + end + Note over U,JOB: Nothing persists beyond chat history. No Tracker, no Timeline row. +``` + +## 4. Stateful path (Product B) — setup, then refresh, then read + +### 4a. Setup once — craft & lock the Tracker (05b) + +```mermaid +flowchart TD + S0(["Standing concern detected"]) --> S1["Bind capability + input
(maps.place X / web.scrape Y)"] + S1 --> S2["Sample fetch · ONE real verb call
(billed)"] + S2 --> S3["Agent drafts definition from decision + sample:
field_schema · materiality · identity_rule · notable_signals"] + S3 --> S4{"Human reviews in chat"} + S4 -->|add field X / tweak| S3 + S4 -->|looks good| S5["LOCK · versioned snapshot
status: draft -> active"] + S5 --> S6[("Tracker persisted
+ tracked_entities row (05a)")] +``` + +### 4b. The hot loop — refresh(tracker) (05b → writes 05a) + +```mermaid +flowchart TD + T(["Trigger fires refresh(tracker) · 06"]) --> C1["1 · Crawl: call bound capability (04a)"] + C1 --> C2{"2 · Content-hash
== stored hash?"} + C2 -->|identical| STOP["stamp last_checked_at · STOP
(zero LLM cost)"] + C2 -->|changed / first run| C3["3 · Fill: agent conforms raw -> locked field_schema
(extras -> notable_signals)"] + C3 --> C4["4 · Diff (code): new record vs entity_current_state -> raw deltas"] + C4 --> C5{"5 · Judge each delta"} + C5 -->|numeric / clear rule| C5a["code decides material/noise
(free, reproducible)"] + C5 -->|ambiguous / notable_signals| C5b["agent decides
(1 LLM call; may read CI context folder)"] + C5a --> C6{"material?"} + C5b --> C6 + C6 -->|yes| W["6 · Append entity_changes row
+ overwrite entity_current_state (05a)"] + C6 -->|no| N["only bump last_checked_at
(no change -> no row)"] + W --> DONE([done]) + N --> DONE + STOP --> DONE +``` + +### 4c. Read / answer — straight from the Timeline (05a) + +```mermaid +flowchart LR + Q(["'Is competitor X pulling ahead?'"]) --> A["intelligence_agent · query_timeline(tracker_id)"] + A --> TL[("entity_changes + entity_current_state")] + TL --> ANS["Answer from stored deltas
(NOT re-derived from chat history)"] +``` + +## 5. Triggers — who calls refresh(tracker) (06) + +```mermaid +flowchart LR + M["Manual
'refresh now' (chat/REST)"] --> RF["refresh(tracker)
headless unit of work · 05b"] + AG["Agent
calls refresh as a tool"] --> RF + XC["External cron
POST /v1/trackers/{id}/refresh"] --> RF + CA["CI automation ACTION
(existing automations)"] --> RF + RF --> TLW[("writes Timeline · 05a")] + + CA -. "schedule selector" .-> SEL["hardened cron selector"] + CA -. "run record + idempotency" .-> AR["AutomationRun (PENDING-gate)"] + CA -. "delivery" .-> DEL["alert / output to user"] + + classDef opt fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; + class CA,SEL,AR,DEL opt; +``` + +CI **core** (`refresh` + Timeline) has **zero** automations dependency — manual/agent/cron all work +standalone. The CI **action** is the *optional* adapter that buys recurrence + idempotency + delivery. + +## 6. The persisted state — Timeline data model (05a) + +```mermaid +erDiagram + TRACKER ||--|| TRACKED_ENTITY : "MVP 1:1 (multi-ready)" + TRACKED_ENTITY ||--|| ENTITY_CURRENT_STATE : "latest" + TRACKED_ENTITY ||--o{ ENTITY_CHANGES : "append-only log" + + TRACKER { + uuid id + text decision + json capability_binding + json definition_locked "field_schema·identity_rule·materiality (versioned)" + text status "draft|active" + } + TRACKED_ENTITY { + uuid id + uuid workspace_id + text entity_key "from identity_rule, unique per tracker" + ts first_seen_at + } + ENTITY_CURRENT_STATE { + uuid entity_id FK + json fields "conforms to field_schema" + text content_hash "powers step-2 pre-check" + ts last_checked_at + } + ENTITY_CHANGES { + uuid id + ts captured_at + json delta "{field:{from,to}}" + text materiality "material|notable" + text decided_by "code|agent (provenance)" + text source_ref + text note "why material" + } +``` diff --git a/plans/backend/revamp phases 4-7/04a-capabilities.md b/plans/backend/revamp phases 4-7/04a-capabilities.md new file mode 100644 index 000000000..8d7a01a4b --- /dev/null +++ b/plans/backend/revamp phases 4-7/04a-capabilities.md @@ -0,0 +1,177 @@ +# Phase 4a — Capabilities (typed verbs over Acquisition) + +> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7). +> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above +> calls into it. +> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome` +> contract, the proxy provider, and the `WebCrawlCreditService` billing seam. +> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies +> them. Locate foundation code by **symbol/grep**, not the line numbers cited. + +## Objective + +Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable +verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume +**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the +KB" with "capability that returns data."** A capability is a stateless function you *call and get data +back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached. + +## Current state (cited) + +- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content, + metadata, crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS` + (`app/proprietary/web_crawler/`). +- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` / + `successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`). +- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`). +- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the + substrate for `web.discover`. +- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope). + +## Target design + +### The verb set (MVP) — namespaced, nothing top-level + +Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor). +Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …). + +| Verb | Input → Output | Mode | Executes over | Bills (03c) | +|------|----------------|------|---------------|-------------| +| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success | +| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* | +| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place | +| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 | +| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page | + +### `web.scrape` — one array-friendly verb (no separate "batch") + +Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model: +"give me URLs, I give you per-URL content." + +### Execution mode — a property of the **result**, not two verbs + +A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**: + +``` +{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... } +``` + +- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline → + zero polling. (single page, one place, a search) +- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until + `completed`. (batch scrape, Maps search/reviews) + +So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller. + +- **Threshold** (how many URLs / how heavy before async) is **configurable**. +- A caller may pass `async: true` to **force** a job even for small input (agents that never block). +- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the + **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it. + +### The capability registry — single source of truth + +One registry entry per verb: + +``` +Capability { + name # dotted, e.g. "web.scrape", "maps.search" + input_schema # Pydantic + output_schema # Pydantic + mode # can-complete-inline? + job-capable? + executor # the async fn (wraps Acquisition / Maps actor) + billing_unit # how Metering charges this call +} +``` + +The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool, +REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly. +Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces. + +### Billing — open to the billing service (not hardcoded per verb) + +A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the +**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite. +`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service. + +- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit). +- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet). +- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged). +- `web.discover` → register a per-search unit (or mark free) **with the billing service**. + +The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs +stay pure; pricing stays pluggable. + +### The Maps actor (clarification) + +- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). +- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw + markdown — exactly what the Intelligence/Timeline layer needs to diff reliably. +- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on + the capability. The capability never knows what decision it serves. + +### Where it lives / license boundary + +- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store). +- It **imports from** the proprietary Acquisition engine but never moves into it. +- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*` + executors — same boundary rule Phase 3 set. + +## Work items + +1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import. +2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL + result shape; declare the `web_crawl` `billing_unit`. +3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed); + declare its `billing_unit` (or free). +4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs. +5. **Uniform envelope**: shared `completed|pending` result type returned by every executor. +6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor. +7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code). + +## Tests + +- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns + `pending` + `job_id`; the job completes to `results`. +- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail + the batch. +- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED` + free; disabling the billing flag makes it a no-op. +- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb. +- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set. + +## Risks / trade-offs + +- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when + the actor lands (doesn't block `web.*`). +- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never + grows back into `pipeline_runs`. +- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact + mid-run behavior is an implementation-time call. + +## Resolved decisions + +1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level. +2. One array-friendly `web.scrape` (no separate batch verb). +3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs. +4. Build the thin job model now (Maps search / batch scrape force it). +5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`). +6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service** + (verbs declare a `billing_unit`; `03c` is the first provider, new units register there). +7. Connector → capability is a **replacement** for these data sources, not an extension. + +## Out of scope (hand-offs) + +- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`. +- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless. +- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a + **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain + doesn't block on it. +- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces + (`linkedin.*`, `amazon.*`) — deferred. +- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing + fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`. + +## Open questions (carry forward) + +- Default async threshold for `web.scrape`. +- Whether `web.discover` is metered or free. diff --git a/plans/backend/revamp phases 4-7/04b-access.md b/plans/backend/revamp phases 4-7/04b-access.md new file mode 100644 index 000000000..3cea58d45 --- /dev/null +++ b/plans/backend/revamp phases 4-7/04b-access.md @@ -0,0 +1,136 @@ +# Phase 4b — Access / Surfaces (chat · REST · MCP doors) + +> Part of **Phase 4 — Capabilities & Access**. Sits on top of `04a-capabilities.md` (the registry it exposes). +> **Build after** `04a`. Together, `04a + 04b` ship **Product A** (stateless utility) — revenue day one. +> **Depends on** SHIPPED infra: Identity/Tenancy, **API keys**, the chat agent + its tool registry, the +> streaming layer, and Metering (`03c`). Access *reuses* them — it builds none net-new. +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. Locate code by **symbol/grep**, not the cited lines. + +## Objective + +Expose the capability registry (`04a`) to callers, **authenticated + metered**, through three doors — +**chat tools, REST + API keys, MCP server** — all **generated from the one registry** so the I/O +contract cannot drift between surfaces. Access contains **no business logic**: every door is the same +thin adapter. + +## Current state (cited) + +- **API keys** — existing per-workspace key infra (reuse; do **not** build net-new); billed to the + workspace owner via `03c`. +- **Chat agent + tool registry** — capability tools register *into* the existing registry; the seed is + `research/tools/scrape_webpage.py` (capability executor + access door + `03c` turn-accumulator billing). +- **Slow-job pattern** — `subagents/builtins/deliverables/deliverable_wait.py`: dispatch a Celery task, + poll the row's `status` until `READY`/`FAILED` (1.5s cadence), bounded by + `SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s); `deliverables/tools/podcast.py` is the + "return now + live card tracks progress" model. +- **BYO-`MCP_CONNECTOR`** (old `04a`) — the user's own external MCP tools inside our chat agent; its + routing-map gap is the one "connector" worth fixing, and it lands **here**. + +## Target design + +### The adapter shape (identical on every door) + +``` +parse input → validate against verb.input_schema → authn/authz → meter-gate (03c) + → call the SAME executor (04a) → serialize verb.output_schema → return the uniform envelope +``` + +### The three doors (locked order: chat → REST → MCP) + +| Door | Who | Auth | Status | +|------|-----|------|--------| +| **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) | +| **REST + API keys** | external developers (Product A) | **existing API-key infra** | **public day one** | +| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — chosen at implementation | **fast-follow** | + +REST being public day one is cheap precisely because the routes are **generated**, not hand-written — +it's a go-to-market choice, not an engineering cost. + +### Natural language is THE surface (verbs are internal) — non-negotiable + +The human-facing product is **the conversation**. A user **never** names a verb, fills an `input_schema`, +or knows "Product A vs B" exists — they describe a *need* in plain language and the agent does the rest. +Raw typed verbs are exposed only on the REST/MCP doors (devs/external agents). The chat agent owns three +responsibilities per message: **understand intent** → **pick & fill verbs** (incl. chains like +`discover → scrape`, `search → place → reviews`) → **answer in plain language** (results, not envelopes). + +### The intent router (the one new orchestration rule) + +The agent classifies each request along the stateless/stateful line **from the language**: + +``` +"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless) +"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start Tracker setup (Product B, stateful → 05b) +ambiguous → ask ONE question: "just this once, or should I keep watching it for you?" +``` + +This router is the friendly seam between the two products; it lives in the chat door (its prompt lives +in the `intelligence_agent`, `07`) and is the only human-facing decision point. + +### The two MCP directions (keep distinct) + +- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new): Streamable-HTTP `/mcp`, + stateless, bounded/paginated outputs, untrusted inputs, least-privilege (read-only verbs). +- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` (old 04a): the user's own external MCP tools inside our + chat agent. The 04a routing-gap fix lands **here**, not in Capabilities. + +### Chat ↔ slow jobs — reuse the existing background-worker pattern + +Do **not** invent a chat-async mechanism. A slow verb (`web.scrape` over many URLs, `maps.search`, +`maps.reviews`) invoked from chat dispatches the **job** (the `04a` job record) and uses the +`deliverable_wait` poll-until-terminal path; the capability **job record's `status`** is what the helper +polls. Most calls finish inside the poll window → results inline; genuinely long ones surface a tracked +card. REST/MCP expose the same job via `GET /v1/jobs/{id}` (and an MCP equivalent). + +## Work items + +1. **Door generator**: from the `04a` registry, emit (a) chat tool defs + handlers, (b) REST routes + + request/response models, (c) MCP tool schemas + handlers — one adapter shape. +2. **REST surface**: public routes + API-key auth (reuse existing) + the `03c` meter-gate; `GET /v1/jobs/{id}`. +3. **Chat tools**: generalize `scrape_webpage` into the registry-backed set; wire slow verbs through + `deliverable_wait` polling the `04a` job record's `status`. +4. **Intent router**: the A-vs-B classifier (its home is the `07` subagent prompt; the seam is here). +5. **MCP server (fast-follow)**: Streamable-HTTP `/mcp`, least-privilege; auth depth chosen at implementation. +6. **Consume-MCP fix**: repair the BYO-`MCP_CONNECTOR` routing-map gap (old 04a). + +## Tests + +- **Generated parity**: a verb added to the registry appears on chat + REST (+ MCP) with identical I/O. +- **Auth + meter**: REST without a valid key → 401; an over-budget call → blocked by the `03c` gate + before execute; a success charges once. +- **Chat slow verb**: a many-URL `web.scrape` returns inline when fast; surfaces a tracked card when long; + `GET /v1/jobs/{id}` reflects the same terminal status. +- **Intent router**: "compare X and Y" → one-shot; "watch X weekly" → Tracker setup handoff; ambiguous → + exactly one clarifying question. +- **Consume-MCP**: a configured BYO MCP tool is reachable by the chat agent (routing-gap closed). + +## Risks / trade-offs + +- **Public REST day one** → needs bounded inputs + per-key quotas / abuse posture (design alongside launch). +- **MCP auth depth** deferred to implementation (OAuth 2.1 vs bearer) — don't over-build before a consumer exists. +- **Serve-vs-consume MCP confusion** — keep the two directions explicitly separate in code and docs. + +## Resolved decisions + +0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the chat + agent understands intent, picks & fills verbs, answers in plain language; an intent router classifies + one-shot (A) vs standing-concern (B), asking one question only when ambiguous. Raw verbs live only on REST/MCP. +1. Three doors, generated from the capability registry; order chat → REST → MCP. +2. REST is **public day one** (cheap; go-to-market choice). +3. API keys: **reuse existing infra**, billed to workspace owner. +4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation. +5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the `04a` job record's `status`. +6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the old-04a fix is the consume side. + +## Out of scope (hand-offs) + +- **The verbs themselves** (executors, registry, billing units) → `04a`. +- **The CI subagent + its prompt/playbook** (where the intent router actually lives) → `07`. +- **Stateful flows** (Tracker crafting, refresh, timeline reads) → `05b`/`05a`, surfaced via chat tools in `07`. +- Legacy branded connectors stay only for backward-compat (separate hygiene task). + +## Open questions (carry forward) + +- MCP auth depth (decide at implementation). +- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas). +- Whether `web.discover` is metered or free (carried from `04a`). diff --git a/plans/backend/revamp phases 4-7/05a-timeline.md b/plans/backend/revamp phases 4-7/05a-timeline.md new file mode 100644 index 000000000..436257cd4 --- /dev/null +++ b/plans/backend/revamp phases 4-7/05a-timeline.md @@ -0,0 +1,125 @@ +# Phase 5a — Timeline (the moat asset) + +> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes +> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them. +> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not** +> the Knowledge Base (documents/embeddings) and **not** `automation_runs`. + +## Objective + +Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** — +accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no +change, no row.** Storage grows with the *rate of change*, not the number of runs. + +## Current state (cited) + +- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is + what this phase introduces. +- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero + publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog + (this is the **fact** analog, deliberately separate). +- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document( + exclude_metadata=True)` to compute the `content_hash` stored here. + +## Target design + +### The state / process split + +Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain +today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without +running the loop**. That separation is why Timeline is its own domain. + +### The three stores + +| Store | Role | Write pattern | +|-------|------|---------------| +| `tracked_entities` | stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`) | written **once** | +| `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run | +| `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten | + +**The timeline = the change log read in order.** To reconstruct a past point: take Current state and +replay deltas backward (north-star tooling; MVP just *stores* the deltas). + +### Data model sketch (new tables) + +``` +tracked_entities + id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at + # MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready. + +entity_current_state + entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema) + · content_hash · last_checked_at · updated_at + # overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2) + +entity_changes # the append-only change log + id · entity_id (FK) · tracker_id · captured_at + · delta JSONB # { field: { from, to } } + · materiality # material | notable(=notable_signals-sourced) + · decided_by # code | agent (audit of the materiality split) + · source_ref # url / blob key the change was observed from + · note TEXT NULL # optional agent rationale (the "why material") +``` + +- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`. +- `decided_by` records whether code or the agent ruled the change material (provenance seed). + +### What it is NOT + +- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.) +- **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store. +- **Not a resurrected `pipeline_runs`.** + +### Where it lives + +- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide + at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor). +- Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend). + +## Work items + +1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration. +2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop. +3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side. +4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit. +5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred). + +## Tests + +- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged. +- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`. +- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls. +- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row. +- **Read API**: `query_timeline` returns deltas in `captured_at` order. + +## Risks / trade-offs + +- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against — + baseline semantics (silent establish vs flood the log) are an implementation-time call. +- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very + chatty entities is deferred. +- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity + guard (lean: yes) but adds a write-path dependency on `05b`'s lock. + +## Resolved decisions + +1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`. +2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change. +3. CI-owned new tables; **not** KB, **not** `automation_runs`. +4. `content_hash` on current state powers the hot-loop cheap pre-check. +5. `decided_by` on changes records the code-vs-agent materiality provenance. +6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later). + +## Out of scope (hand-offs) + +- **The writer** (hot loop, materiality, schema lock) → `05b`. +- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the + resale/data-product surface → north star (deferred). +- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella. + +## Open questions (carry forward) + +- ORM home: `app/db.py` vs a dedicated `app/timeline/` package. +- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes). +- Retention / archival policy for very high-velocity entities (deferred). diff --git a/plans/backend/revamp phases 4-7/05b-intelligence.md b/plans/backend/revamp phases 4-7/05b-intelligence.md new file mode 100644 index 000000000..97b915d53 --- /dev/null +++ b/plans/backend/revamp phases 4-7/05b-intelligence.md @@ -0,0 +1,180 @@ +# Phase 5b — Intelligence (the decision-grounded engine) + +> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes). +> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the +> **Product B engine** (the Tracker, locked schema, hot loop, deltas). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the +> automations subsystem. +> **Name:** the standing-concern primitive is the **Tracker**. + +## Objective + +Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent +judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the +**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the +Access→Intelligence boundary stays pure functions. + +``` +STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists +STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state +``` + +## Current state (cited) + +- **Capability executors** (`04a`) — called directly by the loop (not through a door). +- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)` + produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`). +- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`) + + the automations executor's PENDING→running idempotency gate (safe under Celery `acks_late`); and the + chat **job record** (`04a`) + `deliverable_wait`. **No new run table.** +- **CI context uploads** — the existing folder-upload / `destination_folder` machinery + KB retrieval. + +## Target design + +### The primitive — `Tracker` + +A saved, decision-grounded subject that accumulates structured signal over time: + +| Field | Meaning | +|-------|---------| +| `decision` | the question being tracked toward ("is this competitor pulling ahead?") | +| `capability_binding` | which verb + input feeds it (`maps.place(X)`, `web.scrape([Y])`) | +| `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract | +| `status` | `draft` → `locked`/`active` | + +One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred +(the `05a` model stays multi-entity-ready so it's additive). + +### Setup (once) — the agent-designed schema flow (IN MVP) + +The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is +derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first): + +1. **Bind** a capability + input. +2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data. +3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types), + `materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable + entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field. +4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run. +5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how + `automations` snapshots `definition`). + +### The hot loop (per refresh) — `refresh(tracker)` + +1. **Crawl** — call the bound capability (`04a`) → raw data. +2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`, + **stop** (no LLM cost). +3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does + **not** invent fields. Unanticipated observations go into `notable_signals`. +4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas. +5. **Judge — the materiality split:** + - **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100% + reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`). + - **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new + `notable_signals` entry → one LLM call rules material/noise. +6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`. + **No change → no row.** + +**Worked example (`maps.place` refresh):** +``` +rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM) +review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM) +hours unchanged → no delta +description reworded → code: no rule → ASK AGENT → NOISE +⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick. +``` + +### Refresh execution & idempotency — ride the invoking surface (no new run table) + +`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever +surface invoked it**: + +- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`** + is the run record and the automations executor provides the **PENDING→running idempotency gate**. This is + exactly the rigor old `06` hand-built — reused, not re-written. +- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`. +- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via + the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No + run-level `charged_micros` ledger required for MVP. + +Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed. + +### User-supplied context files (the F idea, generalized) + +When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** — +uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies +only to *crawled* data.)** The CI-specific part is **organization + use**: + +- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload machinery). +- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder: + +``` +competitor price 12.00 → 9.90 + user's context file says "our price is 10.00" + → agent: competitor crossed *below our price* → MATERIAL (and explain why) +``` + +The user's private context **shapes what counts as material**. Reuses existing KB upload + folder + +retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now. + +### Where it lives / decoupling + +- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls + capability `executor`s directly. +- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is + the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler. + +## Work items + +1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`. +2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version. +3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback. +4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`). +5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table. +6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder. + +## Tests + +- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped. +- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields. +- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword → + `decided_by=agent`; sub-threshold → noise, no row. +- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state. +- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill. +- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame. + +## Risks / trade-offs + +- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an + implementation-time call (no architecture impact). +- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM. +- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version. + +## Resolved decisions + +1. `Tracker` is the standing-concern primitive; replaces "pipeline". +2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary. +3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned. +4. Single entity per Tracker for MVP. +5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. +6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. +7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic. +8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job record; + billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state. +9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed), + routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled* + data only. MVP-optional seam. + +## Out of scope (hand-offs) + +- **The state tables** (the three stores, content-hash, read API) → `05a`. +- **When refresh fires + recurrence/delivery** → `06` (Triggers). +- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`. +- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the + resale/data-product stage → north star (deferred). + +## Open questions (carry forward) + +- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?). +- Versioning policy on re-lock (new version vs in-place) — lean new version. +- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?). +- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope). diff --git a/plans/backend/revamp phases 4-7/06-triggers.md b/plans/backend/revamp phases 4-7/06-triggers.md new file mode 100644 index 000000000..00dd5008c --- /dev/null +++ b/plans/backend/revamp phases 4-7/06-triggers.md @@ -0,0 +1,111 @@ +# Phase 6 — Triggers (the pluggable refresh clock) + +> **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes +> `refresh(tracker)`). +> **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED +> automations subsystem (its schedule selector + `AutomationRun` + delivery). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no +> dependency on it, and **automations is at most one optional adapter, never required**. + +## Objective + +Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** — +and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and +the engine still works. This replaces the old Phase-6 cron scheduler — and the resolution is **not** to +rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path. + +## Current state (cited) + +- **`refresh(tracker)`** — the headless unit of work exposed by `05b`. +- **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED` + claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the + `croniter` util. +- **`AutomationRun`** — the existing run record + PENDING-gate (audit + idempotency). +- **Automations output/delivery** — the existing path that carries results to the user. +- **Access routes** (`04b`) — where the external-cron and manual REST endpoints live. + +## Target design + +### The adapters + +| Adapter | Fired by | MVP? | +|---------|----------|------| +| **Manual** | user "refresh now" (chat tool / REST) | ✅ | +| **Agent** | the in-app agent calls `refresh` as a tool | ✅ | +| **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) | +| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)` **and delivers** the material changes | ✅ (the in-app recurrence + alert path) | + +### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) + +The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes."** Instead of +building a bespoke tick, we **add a CI *action* to the existing automations subsystem**: + +- **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new + scheduler. **(closes old Gap B — scheduler rigor.)** +- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run + table. **(closes old Gap A — run/idempotency, see `05b`.)** +- **Delivery / alert** → the automation's existing **output/delivery** carries the material changes to the + user. **(closes old Gap E — alert delivery.)** + +**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery +that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action +too constraining, a thin CI-specific shape is the fallback — but the action is the default.) + +### Decoupling is preserved (automations is still optional) + +CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs +via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + delivery ++ audit for in-app users. So we honor "don't glue CI to automations" *and* get its machinery for free. + +### Where it lives + +- The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from + `app/intelligence/`. No new scheduler/Beat task. +- The **external-cron** and **REST manual** paths are just Access-door routes (`POST + /v1/trackers/{id}/refresh`) — `04b` plumbing. + +## Work items + +1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`. +2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`. +3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that + calls `refresh(tracker)` and routes the resulting material changes into automations delivery. +4. **Concurrency guard**: a per-Tracker in-flight lock (belt-and-suspenders over the automation run-gate). + +## Tests + +- **Decoupling**: `refresh(tracker)` works via manual/agent/external-cron with automations disabled entirely. +- **Recurring path**: a scheduled CI action fires `refresh` on cron and delivers material changes; an + unchanged refresh delivers nothing. +- **Idempotency**: a redelivered scheduled run (same `AutomationRun`) does not double-refresh. +- **External cron**: `POST /v1/trackers/{id}/refresh` triggers exactly one refresh; rejects bad auth. +- **Concurrency**: a second refresh while one is in flight is skipped/queued, not run concurrently. + +## Risks / trade-offs + +- **Action vs shape**: the CI action is the default; a thin CI automation shape is the fallback if the + action proves too constraining. +- **Delivered payload shape** (agent-summarized vs raw deltas since last fire) — implementation-time call. +- **Double-guarding concurrency**: the per-Tracker lock overlaps the automation run-gate, but the lock also + protects the manual/cron paths that don't go through automations. + +## Resolved decisions + +1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled. +2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery). +3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule + selector + `AutomationRun`; delivery reuses automations' output. (Closes old Gaps A/B/E.) +4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**. +5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron). + +## Out of scope (hand-offs) + +- **`refresh(tracker)` internals** (the hot loop, materiality) → `05b`. +- **The Timeline reads** the delivery summarizes → `05a`. +- **The chat tool surface** for manual/agent refresh → `07` (the `intelligence_agent` toolset). + +## Open questions (carry forward) + +- CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback). +- What the delivered payload looks like (summarized material changes vs raw deltas since last fire). +- Concurrency: per-Tracker lock granularity (like the connector indexing lock). diff --git a/plans/backend/revamp phases 4-7/07-orchestration.md b/plans/backend/revamp phases 4-7/07-orchestration.md new file mode 100644 index 000000000..13e8f4654 --- /dev/null +++ b/plans/backend/revamp phases 4-7/07-orchestration.md @@ -0,0 +1,123 @@ +# Phase 7 — Orchestration / Conversation (the CI-expert subagent) + +> **Phase 7** of the CI-pivot revamp — the **human-facing brain** that turns "natural language is the only +> surface" (`04b`, Decision 0) into a real deliverable. **Build last** — it sits atop `04`–`06`. +> **Depends on** all prior phases: `04a` verbs, `04b` doors + `deliverable_wait`, `05a`/`05b` Tracker + +> Timeline, `06` refresh triggers. +> **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming, +> citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin +> subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it. + +## Objective + +Ship the **`intelligence_agent`** — a net-new builtin CI-expert subagent — so users get the whole product +through plain conversation: intent routing (one-shot vs standing concern), verb composition, Tracker +crafting, and decision-grounded answering. The intent router + verb-composition + Tracker-crafting are +where "user-friendly" is won or lost, so the orchestration layer is **designed, not assumed**. + +## Current state (cited) + +- **Builtin subagent pattern** — `subagents/builtins//`: `agent.py` (`build_subagent(...)` → + `pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py` + (`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md` + (the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`. +- **Tool shape** — `research/tools/scrape_webpage.py` already demonstrates **tool = capability executor + + access door + 03c billing**: calls `WebCrawlerConnector.crawl_url`, bills via the turn accumulator + (`get_current_accumulator()` + `WebCrawlCreditService`), returns a typed dict. CI tools follow this exactly. +- **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`). + +## Target design + +### The `intelligence_agent` subagent (CI expert) + +A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to +`research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research +a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing +happen. It owns the **CI playbook** in `system_prompt.md`: + +1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") → call + verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; ambiguous → + ask the single clarifying question. +2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place → + maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand. +3. **Tracker crafting** — the conversational schema-design flow from `05b`: sample-fetch → propose + `field_schema` + materiality + identity → user validates & locks → versioned. +4. **Decision-grounded answering** — read the Timeline (`05a`) to answer "what changed / is X pulling + ahead?" from stored deltas, not by re-deriving from chat history. + +### The toolset (what `load_tools` returns) + +| Tool | Wraps | Mode | Billing | +|------|-------|------|---------| +| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | `03c` turn accumulator (as `scrape_webpage` already does) | +| `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed | +| `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — | +| `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call | +| `query_timeline(tracker_id, …)` | `05a` read API | inline | — | +| `list_trackers()` | `05a`/`05b` read | inline | — | + +- **Capability verbs are a shared tool module** (generated from the `04a` registry) — `research` can load + the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt. + (`scrape_webpage` is the seed; generalize it into the registry-backed set.) +- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the + existing `deliverable_wait` poll-until-terminal + live-card path (`04b`). + +### Boundaries + +- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `05a`/`05b` via tools; the hot loop, + materiality, and Timeline writes live in `05a`/`05b`, callable headless (so REST/MCP and Triggers reach the + same logic with no agent in the loop). +- **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this + subagent entirely and call `04a` verbs directly — they *want* explicitness. + +## Work items + +1. **Subagent scaffold**: `subagents/builtins/intelligence_agent/` with `agent.py` / `tools/index.py` / + `description.md` / `system_prompt.md`, packed via `pack_subagent`. +2. **CI playbook prompt**: intent routing (A/B + one clarifying question), verb chains, crafting flow, + decision-grounded answering. +3. **Shared verb tool module**: registry-backed (`04a`) capability tools, reusable by `research` too. +4. **Tracker/Timeline tools**: `craft_tracker` / `lock_tracker` / `update_tracker` / `refresh_tracker` / + `query_timeline` / `list_trackers`. +5. **Slow-verb wiring**: route job-mode tools through `deliverable_wait` + the live card. +6. **Router registration**: `description.md` so the main agent delegates CI-flavored requests here. + +## Tests + +- **Delegation**: a CI-flavored request routes to `intelligence_agent`; a non-CI request does not. +- **One-shot**: "compare X and Y" composes verbs and answers in plain language, persisting nothing. +- **Standing concern**: "watch X weekly" runs the crafting flow → lock → (refresh via `06`). +- **Crafting**: `craft_tracker` does a real sample-fetch and proposes a schema; `lock_tracker` versions it. +- **Answering**: `query_timeline` answers "what changed?" from stored deltas, not chat history. +- **Slow verb**: a multi-URL scrape surfaces a live card and returns terminal results. + +## Risks / trade-offs + +- **Prompt quality is the product**: intent-router/composition quality is a first-class deliverable, not + plumbing — budget iteration on the prompt. +- **One-shot ownership**: whether `research` keeps generic scraping or the `intelligence_agent` owns all + CI-flavored calls (lean: shared verb tools, CI agent owns the *playbook*). +- **Pre-frontend lock UX**: "review & lock" must render in pure chat for MVP (ties to `05b`'s open Q). + +## Resolved decisions + +1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not + a runtime rebuild. +2. Tools follow the `scrape_webpage` shape: capability executor + access door + `03c` billing. +3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds + Tracker/Timeline tools + the CI prompt. +4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`. +5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async. + +## Out of scope (hand-offs) + +- **The verbs, doors, engine, state, and triggers** → `04`/`05`/`06` (this phase only *orchestrates* them). +- **Richer multi-step CI playbooks** (auto competitor discovery → multi-Tracker setup), proactive "you + should watch this" suggestions, cross-Tracker synthesis → north star (deferred). +- **Frontend CI surfaces** (cards, dashboards) → frontend umbrella. + +## Open questions (carry forward) + +- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls + (lean: shared verb tools, `intelligence_agent` owns the *CI playbook*). +- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `05b`'s open Q).