From 10801058c5bb1ca9adc7896506841b924ff049d2 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:35:05 +0200 Subject: [PATCH 01/21] docs(revamp): add CI pivot overview & domain map --- plans/backend/revamp/00-overview.md | 40 +++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 plans/backend/revamp/00-overview.md diff --git a/plans/backend/revamp/00-overview.md b/plans/backend/revamp/00-overview.md new file mode 100644 index 000000000..1604406bc --- /dev/null +++ b/plans/backend/revamp/00-overview.md @@ -0,0 +1,40 @@ +# Phase 4 → end revamp — Overview & reconciliation (CI pivot · WIP) + +> **What this is.** The CI pivot re-architects the old "pipeline" phases (`04`–`07`) into small, +> single-responsibility domains. This file is the map + the reconciliation against the old plans. +> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl +> billing). The revamp is **Phase 4 → end only**. We do **not** touch 1–3. + +## The two products + +``` +PRODUCT A — stateless utility ① Capabilities + ② Access + call a verb → get data → bill. Nothing persists. + +PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Triggers to drive it) + a Lens accumulates structured signal over time. The Timeline is the moat. +``` + +## The domain map + +``` + FIXED (Phases 1–3) OUR SCOPE (Phase 4 → end) + ┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐ + │ Acquisition │ │ ① Capabilities typed verbs over Acquisition │ + │ proprietary/web_crawler │◄──┤ ② Access chat · REST · MCP doors │ + │ CrawlOutcome · billing (03c) │ │ ③ Intelligence Lens · schema · hot loop │ + │ │ │ ④ Timeline 3-store delta state (moat) │ + │ │ │ ⑤ Triggers pluggable refresh clock │ + │ │ │ ⑥ Orchestration CI-expert subagent + tools │ + └─────────────────────────────────┘ └──────────────────────────────────────────────┘ + data engine (untouched) stateless ①② → stateful ③④ , driven by ⑤ , fronted by ⑥ +``` + +| # | Domain | One line | Doc | +|---|--------|----------|-----| +| ① | Capabilities | Acquisition → typed callable verbs (`web.scrape`, `web.discover`, `maps.*`) | `01-capabilities.md` | +| ② | Access | expose verbs to callers, authed + metered (chat / REST / MCP) | `02-access.md` | +| ③ | Intelligence | the Lens, agent-designed locked schema, hot loop (agent judges, code computes) | `03-intelligence.md` | +| ④ | Timeline | durable time-shaped truth; deltas not snapshots; no change → no row | `04-timeline.md` | +| ⑤ | Triggers | when a Lens refreshes; `refresh(lens)` callers; recurrence+delivery = optional CI action on automations | `05-triggers.md` | +| ⑥ | Orchestration | the human-facing CI-expert subagent (intent routing, verb chains, Lens crafting) + its tools | `06-orchestration.md` | From 297a53efa6e90d4a891196babf07c022f53bdf9f Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:35:11 +0200 Subject: [PATCH 02/21] docs(revamp): add Capabilities domain (typed verbs) --- plans/backend/revamp/01-capabilities.md | 165 ++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 plans/backend/revamp/01-capabilities.md diff --git a/plans/backend/revamp/01-capabilities.md b/plans/backend/revamp/01-capabilities.md new file mode 100644 index 000000000..dc93e0d00 --- /dev/null +++ b/plans/backend/revamp/01-capabilities.md @@ -0,0 +1,165 @@ +# Domain ① — Capabilities (CI pivot revamp · WIP) + +> **WIP design doc.** Part of the Phase 4 → end revamp (the old "pipeline"). +> **Scope guardrail:** Phases 1–3 are **SHIPPED and FIXED** — the rename, and the proprietary +> crawler/proxy/billing/stealth/captcha stack. This domain *builds on* them and never modifies them. + +## Role in the universe + +``` +FIXED foundation (done): Acquisition (crawler moat) · Metering (03c) · Identity/Tenancy +OUR scope (this revamp): ▶ Capabilities ◀ → Access → Intelligence + Timeline → Triggers +``` + +Capabilities sits directly on the fixed foundation and is consumed by **Access** (the doors) and +the **Intelligence** hot loop. It is the first domain because everything above it calls into it. + +## Purpose + +Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, +callable verbs** that every door (chat / REST / MCP) and the Intelligence loop consume +**identically**. + +This is the core pivot of old Phase 4: **replace "connector that ingests into the KB" with +"capability that returns data."** A capability is a stateless function you *call and get data +back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached. + +## Foundation it builds on (do not touch) + +- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch; `{content, metadata, + crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS`. + (`app/proprietary/web_crawler/`.) +- `WebCrawlCreditService` (`03c`) — per-success metering + the `bill=` seam. +- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier. + +## The verb set (MVP) — namespaced, nothing top-level + +Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor). +Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …). + +| Verb | Input → Output | Mode | Executes over | Bills (03c) | +|------|----------------|------|---------------|-------------| +| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success | +| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* | +| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place | +| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 | +| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page | + +## `web.scrape` — one array-friendly verb (no separate "batch") + +Scraping always **welcomes an array of URLs**. There is no separate `scrape_batch`. One verb, +one mental model: "give me URLs, I give you per-URL content." + +## Execution mode — a property of the **result**, not two verbs + +A capability does **not** split into sync/async variants. Every call returns a **uniform result +envelope**: + +``` +{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...] , error?: ... } +``` + +- **inline (fast path):** small/fast input → finishes in-request → envelope returns `completed` + with `results` inline → **zero polling**. (single page, one place, a search) +- **job (slow path):** large/slow input → envelope returns `pending` + `job_id` → caller polls a + status endpoint until `completed`, then reads `results`. (batch scrape, Maps search/reviews) + +So "sync" is simply *a job that finished instantly*. Same verb, same response shape, no magic +branching for the caller to reason about. + +- **Threshold** (how many URLs / how heavy before it goes async) is **configurable**. +- A caller may pass `async: true` to **force** a job even for small input (agents that never want + to block). +- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the + **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it. + +## The capability registry — single source of truth + +One registry entry per verb: + +``` +Capability { + name # dotted, e.g. "web.scrape", "maps.search" + input_schema # Pydantic + output_schema # Pydantic + mode # can-complete-inline? + job-capable? + executor # the async fn (wraps Acquisition / Maps actor) + billing_unit # how Metering charges this call +} +``` + +The **three doors are generated from the registry**, not hand-written three times: + +- **chat tool** ← entry (tool def + handler) +- **REST route** ← entry (path + request/response models) +- **MCP tool** ← entry (MCP tool schema + handler) + +…and the **Intelligence** hot loop calls the *same* `executor` directly. Add a verb once → it +lights up on every surface; the I/O contract cannot drift between surfaces. + +## Where it lives / license boundary + +- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store). +- It **imports from** the proprietary Acquisition engine but never moves into it. +- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*` + executors — same boundary rule Phase 3 set (Apache code may import *from* proprietary, not move + *into* it). + +## Billing integration — open to the billing service (not hardcoded per verb) + +Billing is **not** baked into each verb. A capability only **declares a `billing_unit`** in its +registry entry; charging is delegated to the **billing service**, so adding/repricing a unit is a +billing-service concern, not a capability rewrite. `03c`'s `WebCrawlCreditService` is the **first +provider**; new units register with the same service. + +- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit). +- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet). +- captcha attempts → existing per-attempt `web_crawl_captcha` unit (unchanged). +- `web.discover` → register a per-search unit (or mark free) **with the billing service**. + +The point: the registry says *"this verb bills unit X"*; the **billing service owns what unit X +costs and how it settles.** Verbs stay pure; pricing stays pluggable. + +## The Maps actor (clarification) + +- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). +- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), + not raw markdown — which is exactly what the Intelligence/Timeline layer needs to diff reliably. +- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint + on the capability. The capability never knows what decision it serves. + +## Relationship to the drafted Phase 4 + +- **04b (source-discovery)** → **absorbed** as `web.discover` + relocating Linkup/Baidu keys from + per-connector `config` to platform env (SearXNG already env-based). Tavily/Serper dropped. +- **04a (connector taxonomy + MCP routing)** → **demoted to backward-compat hygiene.** The + capability registry is the new spine, so the Type-1/Type-2 *connector* registry is no longer + central. 04a's "turn off branded connectors" stays a small, separable cleanup for the legacy + chat agent; the **BYO-`MCP_CONNECTOR`** routing fix belongs to the **Access/Conversation** domain + (it lets the agent use the *user's* external tools — a different concept from our capabilities). + +## Deferred (not in MVP) + +- **Recursive site crawl** (follow links across a whole site) — doesn't exist today; the + "crawl a topic" story is `web.discover` → `web.scrape`. +- A generic **`web.extract`** (LLM-structured extraction as a standalone verb) — structured + extraction lives in the Intelligence loop for MVP; promote to a capability later if devs want it. +- Additional platform namespaces (`linkedin.*`, `amazon.*`, …). + +## Locked decisions + +1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level. +2. One array-friendly `web.scrape` (no separate batch verb). +3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs. +4. Build the thin job model now (Maps search / batch scrape force it). +5. One capability registry → generates chat/REST/MCP doors + feeds the Intelligence executor. +6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing + service** (verbs declare a `billing_unit`; `03c` is the first provider, new units register there). +7. Connector → capability is a **replacement** for these data sources, not an extension. + +## Open questions (carry forward) + +- Default async threshold for `web.scrape`. +- The **Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a + **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain + doesn't block on it. From a5fc8c2b7d2e268dd18312e22867ccb3d08eebef Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:35:42 +0200 Subject: [PATCH 03/21] docs(revamp): add Access domain (chat/REST/MCP doors) --- plans/backend/revamp/02-access.md | 99 +++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 plans/backend/revamp/02-access.md diff --git a/plans/backend/revamp/02-access.md b/plans/backend/revamp/02-access.md new file mode 100644 index 000000000..326b6a055 --- /dev/null +++ b/plans/backend/revamp/02-access.md @@ -0,0 +1,99 @@ +# Domain ② — Access / Surfaces (CI pivot revamp · WIP) + +> **WIP design doc.** Part of the Phase 4 → end revamp. Sits on top of Domain ① (Capabilities). +> **Scope guardrail:** Phases 1–3 are SHIPPED and FIXED. Identity/Tenancy, API keys, the chat +> agent + its tool registry, the streaming layer, and Metering (`03c`) already exist — Access +> *reuses* them. + +## Role in the universe + +``` +FIXED: Acquisition · Metering (03c) · Identity/Tenancy · API keys · chat agent · streaming +SCOPE: Capabilities → ▶ Access ◀ → Intelligence + Timeline → Triggers +``` + +Access is the set of **doors** onto the capability registry. It contains **no business logic** — +every door is the same thin adapter. + +## Purpose + +Expose the capability registry to callers, **authenticated + metered**. One adapter shape for every +verb, on every door: + +``` +parse input → validate against verb.input_schema → authn/authz → meter-gate (03c) + → call the SAME executor → serialize verb.output_schema → return the uniform envelope +``` + +## The three doors (locked order: chat → REST → MCP) + +| Door | Who | Auth | Status | +|------|-----|------|--------| +| **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) | +| **REST + API keys** | external developers (Product A) | **existing API-key infra** (reuse, do not build net-new) | **public day one** | +| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — *chosen at implementation* | **fast-follow** after chat+REST | + +All three are **generated from the one capability registry** (Domain ①). REST being public day one +is cheap precisely because the routes are generated, not hand-written — it's a go-to-market choice, +not an engineering cost. + +## The two MCP directions (keep distinct) + +- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new). "External agents gain + the real web." Remote Streamable-HTTP `/mcp`, stateless, bounded/paginated outputs, untrusted + inputs, least-privilege (read-only verbs). Auth depth decided at implementation. +- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` from old 04a: the user's *own* external MCP tools + inside our chat agent. This is the only "connector" worth keeping; the 04a routing-gap fix lands + **here** (Access/Conversation), not in Capabilities. + +## Chat ↔ slow jobs — reuse the existing background-worker pattern + +Do **not** invent a chat-async mechanism. The deliverables stack already solves it: + +- `subagents/builtins/deliverables/deliverable_wait.py` — a shared **poll-until-terminal** helper: + dispatch the Celery task, poll the row's `status` until `READY`/`FAILED` (1.5s cadence), return a + real terminal outcome. Bounded by `SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s) in + multi-agent mode. +- `deliverables/tools/podcast.py` — the "return now + a live card tracks progress" model for very + long work; streaming emission frames live under + `tasks/chat/streaming/handlers/tools/deliverables/...`. + +**Mapping to capabilities:** a slow verb (`web.scrape` over many URLs, `maps.search`, `maps.reviews`) +invoked from chat dispatches the **job** (Domain ① job record) and uses the `deliverable_wait` +poll-until-terminal path; the capability **job record's `status`** is what the helper polls (the +analogue of the podcast/artifact status row). Most calls finish inside the poll window → the tool +returns results inline; genuinely long ones surface a tracked card. REST/MCP doors expose the same +job via `GET /v1/jobs/{id}` (and an MCP equivalent). + +## Reused / fixed (not built here) + +- **API keys** — existing infra; keys scope to a workspace; billed to the workspace owner via `03c`. +- **Identity & Tenancy** — workspace/user/permission checks on every door. +- **Chat agent + tool registry** — capability tools are *registered into* the existing registry. +- **Streaming layer** — existing SSE/card emission for chat job progress. +- **Metering (`03c`)** — the balance gate before execute + charge after, on every door. + +## Relationship to the drafted Phase 4 + +- **04a BYO-`MCP_CONNECTOR` routing fix** → lands here (we *consume* the user's MCP tools). +- Old connector-config routes for data sources are **not** the surface anymore — the capability + REST/MCP/chat doors are. Legacy branded connectors stay only for backward-compat (separate + hygiene task). + +## Locked decisions + +1. Three doors, generated from the capability registry; order chat → REST → MCP. +2. REST is **public day one** (cheap; go-to-market choice). +3. API keys: **reuse existing infra**, billed to workspace owner. +4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation. +5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the + capability job record's `status`. +6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the 04a fix is the + consume side. + +## Open questions (carry forward) + +- MCP auth depth (decide at implementation). +- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas) — design alongside the + public launch. +- Whether `web.discover` is metered or free (carried from Domain ①). From f13a17d3df41bd580f99e6bbac792aff88a0dc56 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:36:23 +0200 Subject: [PATCH 04/21] docs(revamp): make natural language the human surface (intent router) --- plans/backend/revamp/02-access.md | 40 +++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/plans/backend/revamp/02-access.md b/plans/backend/revamp/02-access.md index 326b6a055..ad8d165af 100644 --- a/plans/backend/revamp/02-access.md +++ b/plans/backend/revamp/02-access.md @@ -37,6 +37,41 @@ All three are **generated from the one capability registry** (Domain ①). REST is cheap precisely because the routes are generated, not hand-written — it's a go-to-market choice, not an engineering cost. +## Natural language is THE surface (verbs are internal) — non-negotiable + +The human-facing product is **the conversation**. A user **never** names a verb, fills an +`input_schema`, or knows "Product A vs B" exists — they describe a *need* or a *worry* in plain +language and the agent does the rest. Verbs/schemas/jobs/deltas are things the agent manages **on the +user's behalf**. (The raw typed verbs are exposed only on the REST/MCP doors, which serve +**developers/external agents**, not humans — that's the whole reason those doors exist separately.) + +The chat agent therefore owns three responsibilities on every message: + +1. **Understand intent** (what does the user actually want?). +2. **Pick & fill the verbs** — infer URLs / queries / locations / place refs from the conversation and + compose one or more capability calls (incl. the natural chains, e.g. + `discover → scrape`, `search → place → reviews`). +3. **Answer in plain language** (results, not envelopes). + +### The intent router (the one new orchestration rule) + +The agent classifies each request along the stateless/stateful line **from the language**, so the +user never has to: + +``` +"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless) +"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start the Lens setup flow (Product B, stateful → ③) +``` + +- **One-shot** → orchestrate verbs now, synthesize an answer; nothing persists beyond chat. +- **Standing concern** → hand off to the Intelligence setup flow (`03`): sample-fetch → agent proposes + schema/thresholds/identity → user validates & locks → Lens runs on a trigger. +- **Ambiguous** → ask exactly **one** clarifying question — *"just this once, or should I keep watching + it for you?"* — which is the entire A-vs-B decision expressed in human terms. + +This router is the friendly seam between the two products; it lives in the chat door and is the only +human-facing decision point. + ## The two MCP directions (keep distinct) - **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new). "External agents gain @@ -82,6 +117,11 @@ job via `GET /v1/jobs/{id}` (and an MCP equivalent). ## Locked decisions +0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the + chat agent understands intent, picks & fills verbs, and answers in plain language. An **intent + router** classifies one-shot (Product A) vs standing-concern (Product B) from the language, asking + one clarifying question only when ambiguous. Raw verbs are exposed solely on REST/MCP (dev/agent + doors). 1. Three doors, generated from the capability registry; order chat → REST → MCP. 2. REST is **public day one** (cheap; go-to-market choice). 3. API keys: **reuse existing infra**, billed to workspace owner. From 04fa89fd4cbdc5b63b57d5c5eac8bbb683aed2a2 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:37:10 +0200 Subject: [PATCH 05/21] docs(revamp): add Intelligence domain (Lens + hot loop) --- plans/backend/revamp/03-intelligence.md | 114 ++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 plans/backend/revamp/03-intelligence.md diff --git a/plans/backend/revamp/03-intelligence.md b/plans/backend/revamp/03-intelligence.md new file mode 100644 index 000000000..946a77044 --- /dev/null +++ b/plans/backend/revamp/03-intelligence.md @@ -0,0 +1,114 @@ +# Domain ③ — Intelligence (the decision-grounded engine) (CI pivot revamp · WIP) + +> **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `04-timeline.md` (the state it writes). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** +> the automations subsystem. +> **Working name:** the standing-concern primitive is called a **Lens** (provisional — easily renamed). + +## The stateless / stateful line + +``` +STATELESS (Product A): ① Capabilities + ② Access → call → data → bill, nothing persists +STATEFUL (Product B): ③ Intelligence + ④ Timeline → the Timeline IS the state +``` + +Intelligence is the **process** that mutates state; Timeline (`04`) is the **state**. Everything +below the Access→Intelligence boundary stays pure functions. + +## Purpose + +Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent +judges, code computes.** This replaces the old "pipeline" as the *standing concern*. + +## The primitive — `Lens` + +A saved, decision-grounded subject that accumulates structured signal over time: + +| Field | Meaning | +|-------|---------| +| `decision` | the question being tracked toward ("is this competitor pulling ahead?") | +| `capability_binding` | which verb + input feeds it (`maps.place(X)`, `web.scrape([Y])`) | +| `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract | +| `status` | `draft` → `locked`/`active` | + +One **entity per Lens for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred +(the Timeline model stays multi-entity-ready so it's additive). + +## Setup (once) — the agent-designed schema flow (IN MVP) + +The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is +derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first, +no UI needed): + +1. **Bind** a capability + input (the thing to track). +2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data, + not a hallucinated shape. +3. **Agent drafts the `definition`** from `decision` + the sample: + - `field_schema` — the fields that matter + types (structured-output contract). + - `materiality` — per-field rules (numeric thresholds where possible; "ask agent" otherwise). + - `identity_rule` — the stable entity key (e.g. Maps `place_id`, canonical URL). + - a reserved **`notable_signals`** escape-hatch field for the unanticipated. +4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run. +5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how + `automations` snapshots `definition`). + +## The hot loop (per refresh) — `refresh(lens)` + +1. **Crawl** — call the bound capability (Domain ①) → raw data. +2. **Cheap pre-check** — content hash via the existing + `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)`; identical hash to the + stored `content_hash` → stamp `last_checked_at`, **stop** (no LLM cost). +3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does + **not** invent fields. Unanticipated observations go into `notable_signals`. +4. **Diff (code)** — deterministic compare of the new record vs Current state (`04`) → raw deltas. +5. **Judge — the materiality split:** + - **deterministic (code):** numeric / clear-cut rules from `materiality`, applied for free, 100% + reproducible. e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ price wobble + → noise`, `any hours change → material`. + - **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new + `notable_signals` entry, "does this matter *for the decision*?" → one LLM call rules + material/noise. +6. **Append** — if material: write a Change + update Current state (`04`). Else: only `last_checked_at`. + **No change → no row.** + +**Worked example (`maps.place` refresh):** +``` +rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM) +review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM) +hours unchanged → no delta +description reworded → code: no rule → ASK AGENT → NOISE +⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick. +``` + +## Where it lives / decoupling + +- New **Apache-2** package `app/intelligence/` (the schema-design agent, the hot loop, the materiality + evaluator). Calls capability `executor`s directly (not through a door). +- Exposes **`refresh(lens)`**. *Who* calls it (manual / agent / external cron / optional automation) is + the **Triggers** domain's concern — Intelligence has **no dependency** on any scheduler. + +## MVP cut vs north star + +- **MVP:** agent-designed-schema flow (conversational, sample-grounded, human-locked) · single entity + per Lens · the hot loop with content-hash pre-check + code-threshold + agent-on-ambiguous · + `refresh(lens)` fired manually / by agent / by external cron. +- **North star (deferred):** schema **auto-evolution** from recurring `notable_signals` · multi-entity + Lenses (`maps.search`) · backward-replay reconstruction · coverage-confidence · full + provenance/explainability · the resale/data-product stage. + +## Locked decisions + +1. `Lens` (provisional name) is the standing-concern primitive; replaces "pipeline". +2. Stateless (①②/Product A) vs stateful (③④/Product B) is the Access→Intelligence boundary. +3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned. +4. Single entity per Lens for MVP. +5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. +6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. +7. `app/intelligence/` Apache-2; `refresh(lens)` is trigger-agnostic. + +## Open questions (carry forward) + +- How the schema-design agent surfaces the "review & lock" step before the frontend exists (pure-chat + confirmation for MVP?). +- Versioning policy on re-lock (new version vs in-place) — lean new version. +- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?). From a15cd8953a8c0c1f2b1e7c1ba618d39784821904 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:37:55 +0200 Subject: [PATCH 06/21] docs(revamp): reuse run-record/idempotency + add CI context folder --- plans/backend/revamp/03-intelligence.md | 40 +++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/plans/backend/revamp/03-intelligence.md b/plans/backend/revamp/03-intelligence.md index 946a77044..d74db7360 100644 --- a/plans/backend/revamp/03-intelligence.md +++ b/plans/backend/revamp/03-intelligence.md @@ -87,6 +87,40 @@ description reworded → code: no rule → ASK AGENT → NOISE - Exposes **`refresh(lens)`**. *Who* calls it (manual / agent / external cron / optional automation) is the **Triggers** domain's concern — Intelligence has **no dependency** on any scheduler. +## Refresh execution & idempotency — ride the invoking surface (no new run table) + +`refresh(lens)` is a **headless unit of work**; the **run/audit record + idempotency live on whatever +surface invoked it**, so we do *not* rebuild old Phase-6 `pipeline_runs`: + +- **Recurring (in-app):** invoked by the **CI automation action** (`05`) → the existing + **`AutomationRun`** is the run record (status / error / timing / `step_results`) and the automations + executor already provides the **PENDING→running idempotency gate** (safe under Celery `acks_late` + redelivery). This is exactly the rigor old `06` hand-built — reused, not re-written. +- **Chat (manual / agent):** invoked via the chat **job record** (`01`) + `deliverable_wait` — status + lives there. +- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once + via the billing service (`01`); the **content-hash pre-check** (step 2) is what prevents needless + re-crawls/charges on an unchanged page. So no run-level `charged_micros` ledger is required for MVP. + +Net: the only genuinely new state is the **Timeline** (`04`); execution accounting is borrowed from +`AutomationRun` / the job record. + +## User-supplied context files (the F idea, generalized) + +A CI chat/Lens may have an associated **context folder** (a normal `Folder`): files the user uploads +*in that CI chat* (e.g. "our own price list", a competitor brochure) land there directly — **not** the +global KB. Those files are **decision context**, and the **judge step (5) may consult them** when +ruling materiality: + +``` +competitor price 12.00 → 9.90 + user's context file says "our price is 10.00" + → agent: competitor crossed *below our price* → MATERIAL (and explain why) +``` + +So the user's private context **shapes what counts as material** — a real differentiator, and it +reuses the existing `Folder`/upload machinery without resurrecting KB indexing. **MVP-optional** +(the loop works without it); design the seam now so judgement can read the folder later. + ## MVP cut vs north star - **MVP:** agent-designed-schema flow (conversational, sample-grounded, human-locked) · single entity @@ -105,6 +139,11 @@ description reworded → code: no rule → ASK AGENT → NOISE 5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. 6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. 7. `app/intelligence/` Apache-2; `refresh(lens)` is trigger-agnostic. +8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job + record; billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`04`) is + new state. +9. **CI context folder** (F): user files uploaded in a CI chat land in a dedicated `Folder` and may + feed the judge step; reuses existing upload machinery, **not** the KB. MVP-optional seam. ## Open questions (carry forward) @@ -112,3 +151,4 @@ description reworded → code: no rule → ASK AGENT → NOISE confirmation for MVP?). - Versioning policy on re-lock (new version vs in-place) — lean new version. - Where the schema-design agent itself runs (a setup capability? a chat sub-flow?). +- Context-folder → judge wiring (how much of the folder to load; per-Lens vs per-chat scope). From 89ad2cd992f9ae52fd95379bdd57e55775ec2fca Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:38:02 +0200 Subject: [PATCH 07/21] docs(revamp): add Timeline domain (3-store delta moat) --- plans/backend/revamp/04-timeline.md | 92 +++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 plans/backend/revamp/04-timeline.md diff --git a/plans/backend/revamp/04-timeline.md b/plans/backend/revamp/04-timeline.md new file mode 100644 index 000000000..f515f5327 --- /dev/null +++ b/plans/backend/revamp/04-timeline.md @@ -0,0 +1,92 @@ +# Domain ④ — Timeline (the moat asset) (CI pivot revamp · WIP) + +> **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `03-intelligence.md` (the process +> that writes it). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is +> **not** the Knowledge Base (documents/embeddings) and **not** `automation_runs`. + +## Purpose + +Durably store the **time-shaped truth** for each Lens. This is the asset: **time is the moat** — +accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — +no change, no row.** Storage grows with the *rate of change*, not the number of runs. + +## The state / process split + +Intelligence (`03`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain +today, future dashboards / alerts / the deferred resale product — read the Timeline **directly, +without running the loop**. That separation is the reason Timeline is its own domain. + +## The three stores + +| Store | Role | Write pattern | +|-------|------|---------------| +| `tracked_entities` | stable identity per tracked thing (the Lens's `identity_rule` → `entity_key`) | written **once** | +| `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run | +| `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten | + +**The timeline = the change log read in order.** To reconstruct a past point: take Current state and +replay deltas backward (north-star tooling; not built in MVP — we just *store* the deltas). + +## Data model sketch (new tables) + +``` +tracked_entities + id · workspace_id · lens_id (FK) · entity_key (unique per lens) · first_seen_at + # MVP: exactly one row per Lens (single-entity). Table stays multi-entity-ready. + +entity_current_state + entity_id (FK, unique) · lens_id · fields JSONB (latest, conforms to locked field_schema) + · content_hash · last_checked_at · updated_at + # overwritten each material run; content_hash powers the hot-loop cheap pre-check (03 step 2) + +entity_changes # the append-only change log + id · entity_id (FK) · lens_id · captured_at + · delta JSONB # { field: { from, to } } + · materiality # material | notable(=notable_signals-sourced) + · decided_by # code | agent (audit of the materiality split) + · source_ref # url / blob key the change was observed from + · note TEXT NULL # optional agent rationale (the "why material") +``` + +- **No change → no row** in `entity_changes`; an unchanged refresh only bumps + `entity_current_state.last_checked_at`. +- `decided_by` records whether code or the agent ruled the change material (provenance seed for the + north-star explainability work). + +## What it is NOT + +- **Not the KB** — no `Document` rows, no embeddings, no indexing pipeline. (The meeting's "don't + index crawled data" holds.) +- **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store. +- **Not a resurrected `pipeline_runs`.** + +## Where it lives + +- New CI-owned tables (in `app/db.py` alongside the other core entities, or a small `app/timeline/` + package — decide at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps + extractor, not here). +- Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend). + +## MVP cut vs north star + +- **MVP:** the three stores · single entity per Lens · deltas with `decided_by`/`source_ref` · + `content_hash` pre-check support. +- **North star (deferred):** backward-replay reconstruction queries · trend/series read APIs · + coverage-confidence metadata · multi-entity scale · the resale/data-product surface. + +## Locked decisions + +1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`. +2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change. +3. CI-owned new tables; **not** KB, **not** `automation_runs`. +4. `content_hash` on current state powers the hot-loop cheap pre-check. +5. `decided_by` on changes records the code-vs-agent materiality provenance. +6. Single entity per Lens for MVP; schema stays multi-entity-ready (additive later). + +## Open questions (carry forward) + +- ORM home: `app/db.py` vs a dedicated `app/timeline/` package. +- Whether `entity_current_state.fields` should be validated against the Lens's locked `field_schema` + at write time (lean: yes, cheap integrity guard). +- Retention / archival policy for very high-velocity entities (deferred). From 5d93d46f5d607f8d12c596f6b3513550c4984ac7 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:38:02 +0200 Subject: [PATCH 08/21] docs(revamp): add Triggers domain (recurrence via automations action) --- plans/backend/revamp/05-triggers.md | 72 +++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 plans/backend/revamp/05-triggers.md diff --git a/plans/backend/revamp/05-triggers.md b/plans/backend/revamp/05-triggers.md new file mode 100644 index 000000000..5818e43bd --- /dev/null +++ b/plans/backend/revamp/05-triggers.md @@ -0,0 +1,72 @@ +# Domain ⑤ — Triggers (the pluggable refresh clock) (CI pivot revamp · WIP) + +> **WIP design doc.** Part of the Phase 4 → end revamp. The thinnest domain. +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This domain is **decoupled** — Intelligence (`03`) +> has no dependency on it, and **automations is at most one optional adapter, never required**. + +## Purpose + +Decide **when** a Lens refreshes. Intelligence exposes a single entry point — **`refresh(lens)`** — +and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger +and the engine still works. + +This replaces Phase 6's cron scheduler — and the resolution is **not** to rebuild a scheduler at all, +but to **reuse the automations subsystem** for the in-app recurring path (it already has a hardened +clock + run record + delivery). A trigger only ever calls `refresh(lens)`. + +## The adapters + +| Adapter | Fired by | MVP? | +|---------|----------|------| +| **Manual** | user "refresh now" (chat tool / REST) | ✅ | +| **Agent** | the in-app agent calls `refresh` as a tool | ✅ | +| **External cron** | the user's own scheduler hits `POST /v1/lenses/{id}/refresh` | ✅ (zero infra on us) | +| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(lens)` **and delivers** the material changes | ✅ (the in-app recurrence + alert path) | + +## Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) + +The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes"**. Instead +of building a bespoke tick (which — done correctly — still needs `FOR UPDATE SKIP LOCKED` claiming, +`next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), we **add a CI *action* +to the existing automations subsystem**: + +- **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new + scheduler. **(closes the old Gap B — scheduler rigor.)** +- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new + run table. **(closes the old Gap A — run/idempotency, see `03`.)** +- **Delivery / alert** → the automation's existing **output/delivery** carries the material changes to + the user. **(closes the old Gap E — alert delivery.)** + +**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and +delivery that already exist. A single `refresh_lens` action reuses all of it. (If implementation finds +the action too constraining, a thin CI-specific shape is the fallback — but the action is the default.) + +### Decoupling is preserved (automations is still optional) + +CI **core** — `refresh(lens)` + Timeline (`03`/`04`) — has **zero** automations dependency and runs via +manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + delivery ++ audit for in-app users. So we honor "don't glue CI to automations" *and* get its machinery for free. + +## Where it lives / decoupling + +- The **CI action** lives with automations (its action registry); it imports `refresh(lens)` from + `app/intelligence/`. No new scheduler/Beat task. +- The **external-cron** and **REST manual** paths are just Access-door routes (`POST + /v1/lenses/{id}/refresh`) — Domain ② plumbing. + +## Locked decisions + +1. Intelligence exposes `refresh(lens)`; all triggers are callers. Fully decoupled. +2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery). +3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule + selector + `AutomationRun`; delivery reuses automations' output. (Closes old Gaps A/B/E.) +4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**. +5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron). + +## Open questions (carry forward) + +- CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback). +- What the delivered payload looks like (the material `entity_changes` since last fire — summarized by + the agent, or raw deltas). +- Concurrency: skip a refresh if the Lens already has one in flight (per-Lens lock, like the connector + indexing lock) — even with the automation run-gate, belt-and-suspenders. From d69be56ddbcd80cb41bd9217257fba93daca125f Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:38:02 +0200 Subject: [PATCH 09/21] docs(revamp): add Orchestration domain (CI-expert subagent) --- plans/backend/revamp/06-orchestration.md | 96 ++++++++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 plans/backend/revamp/06-orchestration.md diff --git a/plans/backend/revamp/06-orchestration.md b/plans/backend/revamp/06-orchestration.md new file mode 100644 index 000000000..abc8c1f9e --- /dev/null +++ b/plans/backend/revamp/06-orchestration.md @@ -0,0 +1,96 @@ +# Domain ⑥ — Orchestration / Conversation (the CI-expert subagent) (CI pivot revamp · WIP) + +> **WIP design doc.** Part of the Phase 4 → end revamp. This is the **human-facing brain** that turns +> Decision-0 ("natural language is the only surface", `02-access.md`) into a real deliverable. +> **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming, +> citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin +> subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it. + +## Why this is a first-class deliverable + +The intent router + verb-composition + Lens-crafting are where "user-friendly" is won or lost. Treating +the agent prompt/tooling as plumbing would make the product feel rigid. So the orchestration layer gets +designed, not assumed. + +## The pattern we reuse (verbatim) + +Builtin subagents live at `subagents/builtins//`: +``` +agent.py build_subagent(...) → pack_subagent(name, description, system_prompt, tools, ruleset, …) +tools/index.py NAME · RULESET · load_tools(dependencies) +description.md one-liner the router uses to delegate +system_prompt.md the playbook +(middleware) e.g. citation_state +``` +`research/tools/scrape_webpage.py` already shows the **tool = capability executor + access door + 03c +billing** shape: it calls `WebCrawlerConnector.crawl_url`, bills via the turn accumulator +(`get_current_accumulator()` + `WebCrawlCreditService`), returns a typed dict. Our capability/CI tools +follow this exactly. + +## What's new — the `analyst` subagent (CI expert) + +A new builtin `subagents/builtins/analyst/` (working name — the competitive-intelligence specialist), +peer to `research`/`deliverables`. The **main agent delegates** to it whenever the request is +CI-flavored (research a competitor, watch something, analyze a place's reviews); `description.md` is +what makes that routing happen. + +It owns the **CI playbook** in `system_prompt.md`: + +1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") + → call verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; + ambiguous → ask the single clarifying question. +2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place → + maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand. +3. **Lens crafting** (the "crafting" you flagged) — the conversational schema-design flow from `03`: + sample-fetch → propose `field_schema` + materiality + identity → user validates & locks → versioned. +4. **Decision-grounded answering** — read the Timeline (`04`) to answer "what changed / is X pulling + ahead?" from stored deltas, not by re-deriving from chat history. + +## The toolset (what `load_tools` returns) + +| Tool | Wraps | Mode | Billing | +|------|-------|------|---------| +| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | Domain ① executors | inline-or-job (slow → `deliverable_wait`) | `03c` turn accumulator (as `scrape_webpage` already does) | +| `craft_lens(decision, binding)` | `03` schema-design agent | inline (does the sample-fetch + proposes) | sample crawl billed | +| `lock_lens(draft)` / `update_lens` | `03` Lens persistence | inline | — | +| `refresh_lens(lens_id)` | `03` `refresh(lens)` | job → `deliverable_wait` | per capability call | +| `query_timeline(lens_id, …)` | `04` read API | inline | — | +| `list_lenses()` | `04`/`03` read | inline | — | + +- **Capability verbs are a shared tool module** (generated from the Domain ① registry) — `research` + can load the same ones; the `analyst` additionally loads the Lens/Timeline tools + the CI prompt. + (`scrape_webpage` is the seed; generalize it into the registry-backed set.) +- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_lens`) dispatch a job and use the + existing `deliverable_wait` poll-until-terminal + live-card path (`02-access.md`). + +## Boundaries + +- **Orchestration ≠ Intelligence.** The `analyst` *drives* `03`/`04` via tools; the hot loop, + materiality, and Timeline writes live in `03`/`04`, callable headless (so REST/MCP and Triggers reach + the same logic with no agent in the loop). +- **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this + subagent entirely and call Domain ① verbs directly — they *want* explicitness. + +## MVP cut vs north star + +- **MVP:** the `analyst` subagent + description/prompt · capability verb tools (registry-backed) · + `craft_lens`/`lock_lens`/`refresh_lens`/`query_timeline`/`list_lenses` · intent routing in-prompt. +- **North star (deferred):** richer multi-step CI playbooks (auto competitor discovery → multi-Lens + setup), proactive "you should watch this" suggestions, cross-Lens synthesis. + +## Locked decisions + +1. CI orchestration is a **net-new builtin subagent** (`analyst`, working name) on the existing + runtime — not a runtime rebuild. +2. Tools follow the `scrape_webpage` shape: capability executor + access door + `03c` billing. +3. Capability verbs are a **shared, registry-generated** tool module; the `analyst` adds Lens/Timeline + tools + the CI prompt. +4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `03`/`04`. +5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async. + +## Open questions (carry forward) + +- Subagent **name/persona** (`analyst`? `intelligence`? `scout`?). +- Does CI one-shot scraping stay in `research`, or does the `analyst` own all CI-flavored calls (lean: + shared verb tools, `analyst` owns the *CI playbook*). +- How `craft_lens`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `03`'s open Q). From dc6864d8d4e6855e1e7478e372422e11774a444d Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:38:44 +0200 Subject: [PATCH 10/21] docs(revamp): complete overview (reconciliation, sequence, decisions) --- plans/backend/revamp/00-overview.md | 63 +++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/plans/backend/revamp/00-overview.md b/plans/backend/revamp/00-overview.md index 1604406bc..0e848639f 100644 --- a/plans/backend/revamp/00-overview.md +++ b/plans/backend/revamp/00-overview.md @@ -38,3 +38,66 @@ PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Tri | ④ | Timeline | durable time-shaped truth; deltas not snapshots; no change → no row | `04-timeline.md` | | ⑤ | Triggers | when a Lens refreshes; `refresh(lens)` callers; recurrence+delivery = optional CI action on automations | `05-triggers.md` | | ⑥ | Orchestration | the human-facing CI-expert subagent (intent routing, verb chains, Lens crafting) + its tools | `06-orchestration.md` | + +## Reconciliation with the old plans + +| Old plan | Fate | Where it goes | +|----------|------|---------------| +| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in ② (consume-user-MCP); the taxonomy work is not core | +| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in ① | +| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Lens` (③) + Timeline tables (④); no `pipeline`/`pipeline_runs` | +| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (③); scheduling/runs/delivery → **reuse automations** via a CI action (⑤) — its selector + `AutomationRun`, **not** a rebuilt cron | +| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploaded files land in a dedicated **context `Folder`** (not the KB) and may feed the judge (`03`, F) | + +**Net:** the old `05/06/07` pipeline+KB stack is replaced by `①②③④⑤⑥`. KB indexing of *crawled* data is +out; *user uploads* still work (and gain a CI context-folder role). + +## Build sequence (proposed) + +1. **① Capabilities** — registry + the verb `executor`s over Acquisition (unblocks everything). +2. **② Access** — REST + API keys (day-one endpoints) and chat tools; MCP fast-follow. +3. **④ Timeline** — the three tables (state must exist before the loop writes it). +4. **③ Intelligence** — schema-design agent + hot loop + materiality evaluator. +5. **⑤ Triggers** — manual/agent/external-cron; then the CI **automation action** (recurrence + + delivery) reusing the automations selector + `AutomationRun`. +6. **⑥ Orchestration** — the `analyst` CI-expert subagent + tools + prompt (the human-facing seam; + built atop ①–④, can grow alongside them). + +## Consolidated open forks (need your call) + +1. Primitive **name** — using `Lens` as a working label; confirm or replace. +2. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? +3. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. +4. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). + +*(Resolved: the "built-in scheduler tick" fork — we reuse the automations schedule selector via a CI +action instead of building a tick. See `05`.)* + +## Confirmed decisions (locked across docs) + +- **Natural language is the only human-facing surface.** Users never name verbs; the chat agent + understands intent, composes the verbs, and answers in plain language. An **intent router** + classifies one-shot (A) vs standing-concern (B) from the wording (one clarifying question if + ambiguous). Raw verbs live only on the REST/MCP dev doors. +- MVP surface = Google Maps actor + general web crawler, exposed chat + REST (MCP fast-follow). +- Verbs namespaced per platform (`web.*`, `maps.*`); `web.scrape` takes a URL **array** (inline-or-job). +- **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state. +- CI is **decoupled from automations** (automations = one optional Trigger adapter). +- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single + entity per Lens; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash + pre-check before LLM spend. +- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`analyst`, working + name) on the existing chat runtime, with registry-backed verb tools + Lens/Timeline tools; intent + routing lives in its prompt, the headless logic stays in ③/④. +- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, + not a new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and + automations' delivery. CI core still runs with **zero** automations dependency (manual/agent/cron). +- **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing + idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state. +- **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first + provider; `maps.*` / `web.discover` register their own units there (pricing stays pluggable). +- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a + **separate effort**; `maps.*` verbs are contracts against it and don't block this design. +- **CI context files (F):** files uploaded in a CI chat land in a dedicated `Folder` (not the KB) and + may feed the materiality judge — the user's private context shapes what counts as material. +- License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor. From 3aeecc315af859a5b98b077486a20d7339869efe Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:42:52 +0200 Subject: [PATCH 11/21] docs(revamp): correct CI uploads to land in KB (folder-scoped) --- plans/backend/revamp/00-overview.md | 7 ++++--- plans/backend/revamp/03-intelligence.md | 24 +++++++++++++++--------- 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/plans/backend/revamp/00-overview.md b/plans/backend/revamp/00-overview.md index 0e848639f..36dd05d2c 100644 --- a/plans/backend/revamp/00-overview.md +++ b/plans/backend/revamp/00-overview.md @@ -47,7 +47,7 @@ PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Tri | `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in ① | | `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Lens` (③) + Timeline tables (④); no `pipeline`/`pipeline_runs` | | `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (③); scheduling/runs/delivery → **reuse automations** via a CI action (⑤) — its selector + `AutomationRun`, **not** a rebuilt cron | -| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploaded files land in a dedicated **context `Folder`** (not the KB) and may feed the judge (`03`, F) | +| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature** (uploads still become indexed KB Documents). For CI, uploads are routed to a dedicated **folder** and may feed the judge (`03`, F) | **Net:** the old `05/06/07` pipeline+KB stack is replaced by `①②③④⑤⑥`. KB indexing of *crawled* data is out; *user uploads* still work (and gain a CI context-folder role). @@ -98,6 +98,7 @@ action instead of building a tick. See `05`.)* provider; `maps.*` / `web.discover` register their own units there (pricing stays pluggable). - The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a **separate effort**; `maps.*` verbs are contracts against it and don't block this design. -- **CI context files (F):** files uploaded in a CI chat land in a dedicated `Folder` (not the KB) and - may feed the materiality judge — the user's private context shapes what counts as material. +- **CI context files (F):** files uploaded in a CI chat go into the **KB as normal** (indexed), routed + to a dedicated `Folder`, and may feed the materiality judge via KB retrieval — the user's private + context shapes what counts as material. ("Don't index" applies only to *crawled* data.) - License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor. diff --git a/plans/backend/revamp/03-intelligence.md b/plans/backend/revamp/03-intelligence.md index d74db7360..b3789ec66 100644 --- a/plans/backend/revamp/03-intelligence.md +++ b/plans/backend/revamp/03-intelligence.md @@ -107,19 +107,24 @@ Net: the only genuinely new state is the **Timeline** (`04`); execution accounti ## User-supplied context files (the F idea, generalized) -A CI chat/Lens may have an associated **context folder** (a normal `Folder`): files the user uploads -*in that CI chat* (e.g. "our own price list", a competitor brochure) land there directly — **not** the -global KB. Those files are **decision context**, and the **judge step (5) may consult them** when -ruling materiality: +When a user uploads a file *in a CI chat* (e.g. "our own price list", a competitor brochure), it goes +into the **KB as normal** — uploads create `Document`s and are indexed/embedded, exactly as today. +**(The "don't index" rule applies only to *crawled* data, not to user uploads.)** The CI-specific part +is purely **organization + use**: + +- **Routed to a dedicated folder** for that CI chat/Lens (reuse the existing folder-upload / + `destination_folder` machinery), so the chat's reference files are scoped together. +- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder — when ruling + materiality: ``` competitor price 12.00 → 9.90 + user's context file says "our price is 10.00" → agent: competitor crossed *below our price* → MATERIAL (and explain why) ``` -So the user's private context **shapes what counts as material** — a real differentiator, and it -reuses the existing `Folder`/upload machinery without resurrecting KB indexing. **MVP-optional** -(the loop works without it); design the seam now so judgement can read the folder later. +So the user's private context **shapes what counts as material** — a real differentiator — and it +reuses the existing KB upload + folder + retrieval machinery (nothing new). **MVP-optional** (the loop +works without it); design the seam now so the judge can read the folder later. ## MVP cut vs north star @@ -142,8 +147,9 @@ reuses the existing `Folder`/upload machinery without resurrecting KB indexing. 8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job record; billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`04`) is new state. -9. **CI context folder** (F): user files uploaded in a CI chat land in a dedicated `Folder` and may - feed the judge step; reuses existing upload machinery, **not** the KB. MVP-optional seam. +9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed), + routed to a dedicated `Folder`, and may feed the judge step via KB retrieval. The "don't index" + rule is for *crawled* data only. MVP-optional seam. ## Open questions (carry forward) From d5df87dafdbb1d555fe8155b543d80b2ef5ec825 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Tue, 30 Jun 2026 23:54:11 +0200 Subject: [PATCH 12/21] docs(revamp): rename primitive to Tracker, subagent to intelligence_agent --- plans/backend/revamp/00-overview.md | 28 +++++++-------- plans/backend/revamp/02-access.md | 4 +-- plans/backend/revamp/03-intelligence.md | 28 +++++++-------- plans/backend/revamp/04-timeline.md | 18 +++++----- plans/backend/revamp/05-triggers.md | 20 +++++------ plans/backend/revamp/06-orchestration.md | 43 ++++++++++++------------ 6 files changed, 70 insertions(+), 71 deletions(-) diff --git a/plans/backend/revamp/00-overview.md b/plans/backend/revamp/00-overview.md index 36dd05d2c..92bed3b4d 100644 --- a/plans/backend/revamp/00-overview.md +++ b/plans/backend/revamp/00-overview.md @@ -12,7 +12,7 @@ PRODUCT A — stateless utility ① Capabilities + ② Access call a verb → get data → bill. Nothing persists. PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Triggers to drive it) - a Lens accumulates structured signal over time. The Timeline is the moat. + a Tracker accumulates structured signal over time. The Timeline is the moat. ``` ## The domain map @@ -22,7 +22,7 @@ PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Tri ┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐ │ Acquisition │ │ ① Capabilities typed verbs over Acquisition │ │ proprietary/web_crawler │◄──┤ ② Access chat · REST · MCP doors │ - │ CrawlOutcome · billing (03c) │ │ ③ Intelligence Lens · schema · hot loop │ + │ CrawlOutcome · billing (03c) │ │ ③ Intelligence Tracker · schema · hot loop │ │ │ │ ④ Timeline 3-store delta state (moat) │ │ │ │ ⑤ Triggers pluggable refresh clock │ │ │ │ ⑥ Orchestration CI-expert subagent + tools │ @@ -34,10 +34,10 @@ PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Tri |---|--------|----------|-----| | ① | Capabilities | Acquisition → typed callable verbs (`web.scrape`, `web.discover`, `maps.*`) | `01-capabilities.md` | | ② | Access | expose verbs to callers, authed + metered (chat / REST / MCP) | `02-access.md` | -| ③ | Intelligence | the Lens, agent-designed locked schema, hot loop (agent judges, code computes) | `03-intelligence.md` | +| ③ | Intelligence | the Tracker, agent-designed locked schema, hot loop (agent judges, code computes) | `03-intelligence.md` | | ④ | Timeline | durable time-shaped truth; deltas not snapshots; no change → no row | `04-timeline.md` | -| ⑤ | Triggers | when a Lens refreshes; `refresh(lens)` callers; recurrence+delivery = optional CI action on automations | `05-triggers.md` | -| ⑥ | Orchestration | the human-facing CI-expert subagent (intent routing, verb chains, Lens crafting) + its tools | `06-orchestration.md` | +| ⑤ | Triggers | when a Tracker refreshes; `refresh(tracker)` callers; recurrence+delivery = optional CI action on automations | `05-triggers.md` | +| ⑥ | Orchestration | the human-facing CI-expert subagent (intent routing, verb chains, Tracker crafting) + its tools | `06-orchestration.md` | ## Reconciliation with the old plans @@ -45,7 +45,7 @@ PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Tri |----------|------|---------------| | `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in ② (consume-user-MCP); the taxonomy work is not core | | `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in ① | -| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Lens` (③) + Timeline tables (④); no `pipeline`/`pipeline_runs` | +| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (③) + Timeline tables (④); no `pipeline`/`pipeline_runs` | | `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (③); scheduling/runs/delivery → **reuse automations** via a CI action (⑤) — its selector + `AutomationRun`, **not** a rebuilt cron | | `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature** (uploads still become indexed KB Documents). For CI, uploads are routed to a dedicated **folder** and may feed the judge (`03`, F) | @@ -60,16 +60,16 @@ out; *user uploads* still work (and gain a CI context-folder role). 4. **③ Intelligence** — schema-design agent + hot loop + materiality evaluator. 5. **⑤ Triggers** — manual/agent/external-cron; then the CI **automation action** (recurrence + delivery) reusing the automations selector + `AutomationRun`. -6. **⑥ Orchestration** — the `analyst` CI-expert subagent + tools + prompt (the human-facing seam; +6. **⑥ Orchestration** — the `intelligence_agent` CI-expert subagent + tools + prompt (the human-facing seam; built atop ①–④, can grow alongside them). ## Consolidated open forks (need your call) -1. Primitive **name** — using `Lens` as a working label; confirm or replace. -2. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? -3. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. -4. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). +1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? +2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. +3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). +*(Resolved: primitive **name** = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent).)* *(Resolved: the "built-in scheduler tick" fork — we reuse the automations schedule selector via a CI action instead of building a tick. See `05`.)* @@ -84,10 +84,10 @@ action instead of building a tick. See `05`.)* - **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state. - CI is **decoupled from automations** (automations = one optional Trigger adapter). - Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single - entity per Lens; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash + entity per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check before LLM spend. -- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`analyst`, working - name) on the existing chat runtime, with registry-backed verb tools + Lens/Timeline tools; intent +- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`, working + name) on the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools; intent routing lives in its prompt, the headless logic stays in ③/④. - **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, not a new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and diff --git a/plans/backend/revamp/02-access.md b/plans/backend/revamp/02-access.md index ad8d165af..69d5cec60 100644 --- a/plans/backend/revamp/02-access.md +++ b/plans/backend/revamp/02-access.md @@ -60,12 +60,12 @@ user never has to: ``` "compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless) -"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start the Lens setup flow (Product B, stateful → ③) +"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start the Tracker setup flow (Product B, stateful → ③) ``` - **One-shot** → orchestrate verbs now, synthesize an answer; nothing persists beyond chat. - **Standing concern** → hand off to the Intelligence setup flow (`03`): sample-fetch → agent proposes - schema/thresholds/identity → user validates & locks → Lens runs on a trigger. + schema/thresholds/identity → user validates & locks → Tracker runs on a trigger. - **Ambiguous** → ask exactly **one** clarifying question — *"just this once, or should I keep watching it for you?"* — which is the entire A-vs-B decision expressed in human terms. diff --git a/plans/backend/revamp/03-intelligence.md b/plans/backend/revamp/03-intelligence.md index b3789ec66..f37d4c8cb 100644 --- a/plans/backend/revamp/03-intelligence.md +++ b/plans/backend/revamp/03-intelligence.md @@ -3,7 +3,7 @@ > **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `04-timeline.md` (the state it writes). > **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** > the automations subsystem. -> **Working name:** the standing-concern primitive is called a **Lens** (provisional — easily renamed). +> **Name:** the standing-concern primitive is the **Tracker**. ## The stateless / stateful line @@ -20,7 +20,7 @@ below the Access→Intelligence boundary stays pure functions. Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent judges, code computes.** This replaces the old "pipeline" as the *standing concern*. -## The primitive — `Lens` +## The primitive — `Tracker` A saved, decision-grounded subject that accumulates structured signal over time: @@ -31,7 +31,7 @@ A saved, decision-grounded subject that accumulates structured signal over time: | `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract | | `status` | `draft` → `locked`/`active` | -One **entity per Lens for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred +One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred (the Timeline model stays multi-entity-ready so it's additive). ## Setup (once) — the agent-designed schema flow (IN MVP) @@ -52,7 +52,7 @@ no UI needed): 5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how `automations` snapshots `definition`). -## The hot loop (per refresh) — `refresh(lens)` +## The hot loop (per refresh) — `refresh(tracker)` 1. **Crawl** — call the bound capability (Domain ①) → raw data. 2. **Cheap pre-check** — content hash via the existing @@ -84,12 +84,12 @@ description reworded → code: no rule → ASK AGENT → NOISE - New **Apache-2** package `app/intelligence/` (the schema-design agent, the hot loop, the materiality evaluator). Calls capability `executor`s directly (not through a door). -- Exposes **`refresh(lens)`**. *Who* calls it (manual / agent / external cron / optional automation) is +- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is the **Triggers** domain's concern — Intelligence has **no dependency** on any scheduler. ## Refresh execution & idempotency — ride the invoking surface (no new run table) -`refresh(lens)` is a **headless unit of work**; the **run/audit record + idempotency live on whatever +`refresh(tracker)` is a **headless unit of work**; the **run/audit record + idempotency live on whatever surface invoked it**, so we do *not* rebuild old Phase-6 `pipeline_runs`: - **Recurring (in-app):** invoked by the **CI automation action** (`05`) → the existing @@ -112,7 +112,7 @@ into the **KB as normal** — uploads create `Document`s and are indexed/embedde **(The "don't index" rule applies only to *crawled* data, not to user uploads.)** The CI-specific part is purely **organization + use**: -- **Routed to a dedicated folder** for that CI chat/Lens (reuse the existing folder-upload / +- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload / `destination_folder` machinery), so the chat's reference files are scoped together. - **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder — when ruling materiality: @@ -129,21 +129,21 @@ works without it); design the seam now so the judge can read the folder later. ## MVP cut vs north star - **MVP:** agent-designed-schema flow (conversational, sample-grounded, human-locked) · single entity - per Lens · the hot loop with content-hash pre-check + code-threshold + agent-on-ambiguous · - `refresh(lens)` fired manually / by agent / by external cron. + per Tracker · the hot loop with content-hash pre-check + code-threshold + agent-on-ambiguous · + `refresh(tracker)` fired manually / by agent / by external cron. - **North star (deferred):** schema **auto-evolution** from recurring `notable_signals` · multi-entity - Lenses (`maps.search`) · backward-replay reconstruction · coverage-confidence · full + Trackers (`maps.search`) · backward-replay reconstruction · coverage-confidence · full provenance/explainability · the resale/data-product stage. ## Locked decisions -1. `Lens` (provisional name) is the standing-concern primitive; replaces "pipeline". +1. `Tracker` is the standing-concern primitive; replaces "pipeline". 2. Stateless (①②/Product A) vs stateful (③④/Product B) is the Access→Intelligence boundary. 3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned. -4. Single entity per Lens for MVP. +4. Single entity per Tracker for MVP. 5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. 6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. -7. `app/intelligence/` Apache-2; `refresh(lens)` is trigger-agnostic. +7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic. 8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job record; billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`04`) is new state. @@ -157,4 +157,4 @@ works without it); design the seam now so the judge can read the folder later. confirmation for MVP?). - Versioning policy on re-lock (new version vs in-place) — lean new version. - Where the schema-design agent itself runs (a setup capability? a chat sub-flow?). -- Context-folder → judge wiring (how much of the folder to load; per-Lens vs per-chat scope). +- Context-folder → judge wiring (how much of the folder to load; per-Tracker vs per-chat scope). diff --git a/plans/backend/revamp/04-timeline.md b/plans/backend/revamp/04-timeline.md index f515f5327..d66bc23e6 100644 --- a/plans/backend/revamp/04-timeline.md +++ b/plans/backend/revamp/04-timeline.md @@ -7,7 +7,7 @@ ## Purpose -Durably store the **time-shaped truth** for each Lens. This is the asset: **time is the moat** — +Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** — accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no change, no row.** Storage grows with the *rate of change*, not the number of runs. @@ -21,7 +21,7 @@ without running the loop**. That separation is the reason Timeline is its own do | Store | Role | Write pattern | |-------|------|---------------| -| `tracked_entities` | stable identity per tracked thing (the Lens's `identity_rule` → `entity_key`) | written **once** | +| `tracked_entities` | stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`) | written **once** | | `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run | | `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten | @@ -32,16 +32,16 @@ replay deltas backward (north-star tooling; not built in MVP — we just *store* ``` tracked_entities - id · workspace_id · lens_id (FK) · entity_key (unique per lens) · first_seen_at - # MVP: exactly one row per Lens (single-entity). Table stays multi-entity-ready. + id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at + # MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready. entity_current_state - entity_id (FK, unique) · lens_id · fields JSONB (latest, conforms to locked field_schema) + entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema) · content_hash · last_checked_at · updated_at # overwritten each material run; content_hash powers the hot-loop cheap pre-check (03 step 2) entity_changes # the append-only change log - id · entity_id (FK) · lens_id · captured_at + id · entity_id (FK) · tracker_id · captured_at · delta JSONB # { field: { from, to } } · materiality # material | notable(=notable_signals-sourced) · decided_by # code | agent (audit of the materiality split) @@ -70,7 +70,7 @@ entity_changes # the append-only change log ## MVP cut vs north star -- **MVP:** the three stores · single entity per Lens · deltas with `decided_by`/`source_ref` · +- **MVP:** the three stores · single entity per Tracker · deltas with `decided_by`/`source_ref` · `content_hash` pre-check support. - **North star (deferred):** backward-replay reconstruction queries · trend/series read APIs · coverage-confidence metadata · multi-entity scale · the resale/data-product surface. @@ -82,11 +82,11 @@ entity_changes # the append-only change log 3. CI-owned new tables; **not** KB, **not** `automation_runs`. 4. `content_hash` on current state powers the hot-loop cheap pre-check. 5. `decided_by` on changes records the code-vs-agent materiality provenance. -6. Single entity per Lens for MVP; schema stays multi-entity-ready (additive later). +6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later). ## Open questions (carry forward) - ORM home: `app/db.py` vs a dedicated `app/timeline/` package. -- Whether `entity_current_state.fields` should be validated against the Lens's locked `field_schema` +- Whether `entity_current_state.fields` should be validated against the Tracker's locked `field_schema` at write time (lean: yes, cheap integrity guard). - Retention / archival policy for very high-velocity entities (deferred). diff --git a/plans/backend/revamp/05-triggers.md b/plans/backend/revamp/05-triggers.md index 5818e43bd..087724143 100644 --- a/plans/backend/revamp/05-triggers.md +++ b/plans/backend/revamp/05-triggers.md @@ -6,13 +6,13 @@ ## Purpose -Decide **when** a Lens refreshes. Intelligence exposes a single entry point — **`refresh(lens)`** — +Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** — and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and the engine still works. This replaces Phase 6's cron scheduler — and the resolution is **not** to rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path (it already has a hardened -clock + run record + delivery). A trigger only ever calls `refresh(lens)`. +clock + run record + delivery). A trigger only ever calls `refresh(tracker)`. ## The adapters @@ -20,8 +20,8 @@ clock + run record + delivery). A trigger only ever calls `refresh(lens)`. |---------|----------|------| | **Manual** | user "refresh now" (chat tool / REST) | ✅ | | **Agent** | the in-app agent calls `refresh` as a tool | ✅ | -| **External cron** | the user's own scheduler hits `POST /v1/lenses/{id}/refresh` | ✅ (zero infra on us) | -| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(lens)` **and delivers** the material changes | ✅ (the in-app recurrence + alert path) | +| **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) | +| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)` **and delivers** the material changes | ✅ (the in-app recurrence + alert path) | ## Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) @@ -38,25 +38,25 @@ to the existing automations subsystem**: the user. **(closes the old Gap E — alert delivery.)** **Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and -delivery that already exist. A single `refresh_lens` action reuses all of it. (If implementation finds +delivery that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action too constraining, a thin CI-specific shape is the fallback — but the action is the default.) ### Decoupling is preserved (automations is still optional) -CI **core** — `refresh(lens)` + Timeline (`03`/`04`) — has **zero** automations dependency and runs via +CI **core** — `refresh(tracker)` + Timeline (`03`/`04`) — has **zero** automations dependency and runs via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + delivery + audit for in-app users. So we honor "don't glue CI to automations" *and* get its machinery for free. ## Where it lives / decoupling -- The **CI action** lives with automations (its action registry); it imports `refresh(lens)` from +- The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from `app/intelligence/`. No new scheduler/Beat task. - The **external-cron** and **REST manual** paths are just Access-door routes (`POST - /v1/lenses/{id}/refresh`) — Domain ② plumbing. + /v1/trackers/{id}/refresh`) — Domain ② plumbing. ## Locked decisions -1. Intelligence exposes `refresh(lens)`; all triggers are callers. Fully decoupled. +1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled. 2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery). 3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule selector + `AutomationRun`; delivery reuses automations' output. (Closes old Gaps A/B/E.) @@ -68,5 +68,5 @@ manual / agent / external-cron. Automations is the **optional adapter** that add - CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback). - What the delivered payload looks like (the material `entity_changes` since last fire — summarized by the agent, or raw deltas). -- Concurrency: skip a refresh if the Lens already has one in flight (per-Lens lock, like the connector +- Concurrency: skip a refresh if the Tracker already has one in flight (per-Tracker lock, like the connector indexing lock) — even with the automation run-gate, belt-and-suspenders. diff --git a/plans/backend/revamp/06-orchestration.md b/plans/backend/revamp/06-orchestration.md index abc8c1f9e..81116a591 100644 --- a/plans/backend/revamp/06-orchestration.md +++ b/plans/backend/revamp/06-orchestration.md @@ -8,7 +8,7 @@ ## Why this is a first-class deliverable -The intent router + verb-composition + Lens-crafting are where "user-friendly" is won or lost. Treating +The intent router + verb-composition + Tracker-crafting are where "user-friendly" is won or lost. Treating the agent prompt/tooling as plumbing would make the product feel rigid. So the orchestration layer gets designed, not assumed. @@ -27,9 +27,9 @@ billing** shape: it calls `WebCrawlerConnector.crawl_url`, bills via the turn ac (`get_current_accumulator()` + `WebCrawlCreditService`), returns a typed dict. Our capability/CI tools follow this exactly. -## What's new — the `analyst` subagent (CI expert) +## What's new — the `intelligence_agent` subagent (CI expert) -A new builtin `subagents/builtins/analyst/` (working name — the competitive-intelligence specialist), +A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to `research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing happen. @@ -41,7 +41,7 @@ It owns the **CI playbook** in `system_prompt.md`: ambiguous → ask the single clarifying question. 2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand. -3. **Lens crafting** (the "crafting" you flagged) — the conversational schema-design flow from `03`: +3. **Tracker crafting** (the "crafting" you flagged) — the conversational schema-design flow from `03`: sample-fetch → propose `field_schema` + materiality + identity → user validates & locks → versioned. 4. **Decision-grounded answering** — read the Timeline (`04`) to answer "what changed / is X pulling ahead?" from stored deltas, not by re-deriving from chat history. @@ -51,21 +51,21 @@ It owns the **CI playbook** in `system_prompt.md`: | Tool | Wraps | Mode | Billing | |------|-------|------|---------| | capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | Domain ① executors | inline-or-job (slow → `deliverable_wait`) | `03c` turn accumulator (as `scrape_webpage` already does) | -| `craft_lens(decision, binding)` | `03` schema-design agent | inline (does the sample-fetch + proposes) | sample crawl billed | -| `lock_lens(draft)` / `update_lens` | `03` Lens persistence | inline | — | -| `refresh_lens(lens_id)` | `03` `refresh(lens)` | job → `deliverable_wait` | per capability call | -| `query_timeline(lens_id, …)` | `04` read API | inline | — | -| `list_lenses()` | `04`/`03` read | inline | — | +| `craft_tracker(decision, binding)` | `03` schema-design agent | inline (does the sample-fetch + proposes) | sample crawl billed | +| `lock_tracker(draft)` / `update_tracker` | `03` Tracker persistence | inline | — | +| `refresh_tracker(tracker_id)` | `03` `refresh(tracker)` | job → `deliverable_wait` | per capability call | +| `query_timeline(tracker_id, …)` | `04` read API | inline | — | +| `list_trackers()` | `04`/`03` read | inline | — | - **Capability verbs are a shared tool module** (generated from the Domain ① registry) — `research` - can load the same ones; the `analyst` additionally loads the Lens/Timeline tools + the CI prompt. + can load the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt. (`scrape_webpage` is the seed; generalize it into the registry-backed set.) -- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_lens`) dispatch a job and use the +- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the existing `deliverable_wait` poll-until-terminal + live-card path (`02-access.md`). ## Boundaries -- **Orchestration ≠ Intelligence.** The `analyst` *drives* `03`/`04` via tools; the hot loop, +- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `03`/`04` via tools; the hot loop, materiality, and Timeline writes live in `03`/`04`, callable headless (so REST/MCP and Triggers reach the same logic with no agent in the loop). - **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this @@ -73,24 +73,23 @@ It owns the **CI playbook** in `system_prompt.md`: ## MVP cut vs north star -- **MVP:** the `analyst` subagent + description/prompt · capability verb tools (registry-backed) · - `craft_lens`/`lock_lens`/`refresh_lens`/`query_timeline`/`list_lenses` · intent routing in-prompt. -- **North star (deferred):** richer multi-step CI playbooks (auto competitor discovery → multi-Lens - setup), proactive "you should watch this" suggestions, cross-Lens synthesis. +- **MVP:** the `intelligence_agent` subagent + description/prompt · capability verb tools (registry-backed) · + `craft_tracker`/`lock_tracker`/`refresh_tracker`/`query_timeline`/`list_trackers` · intent routing in-prompt. +- **North star (deferred):** richer multi-step CI playbooks (auto competitor discovery → multi-Tracker + setup), proactive "you should watch this" suggestions, cross-Tracker synthesis. ## Locked decisions -1. CI orchestration is a **net-new builtin subagent** (`analyst`, working name) on the existing +1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not a runtime rebuild. 2. Tools follow the `scrape_webpage` shape: capability executor + access door + `03c` billing. -3. Capability verbs are a **shared, registry-generated** tool module; the `analyst` adds Lens/Timeline +3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds Tracker/Timeline tools + the CI prompt. 4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `03`/`04`. 5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async. ## Open questions (carry forward) -- Subagent **name/persona** (`analyst`? `intelligence`? `scout`?). -- Does CI one-shot scraping stay in `research`, or does the `analyst` own all CI-flavored calls (lean: - shared verb tools, `analyst` owns the *CI playbook*). -- How `craft_lens`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `03`'s open Q). +- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls (lean: + shared verb tools, `intelligence_agent` owns the *CI playbook*). +- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `03`'s open Q). From d472693aacc50ac36cec7a153ce020db11bd491e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:06:46 +0200 Subject: [PATCH 13/21] docs(revamp): reshape Capabilities into Phase 4a --- plans/backend/revamp/01-capabilities.md | 165 ---------------- plans/backend/revamp/04a-capabilities.md | 177 ++++++++++++++++++ .../revamp/{02-access.md => 04b-access.md} | 0 .../{04-timeline.md => 05a-timeline.md} | 0 ...03-intelligence.md => 05b-intelligence.md} | 0 .../revamp/{05-triggers.md => 06-triggers.md} | 0 ...6-orchestration.md => 07-orchestration.md} | 0 7 files changed, 177 insertions(+), 165 deletions(-) delete mode 100644 plans/backend/revamp/01-capabilities.md create mode 100644 plans/backend/revamp/04a-capabilities.md rename plans/backend/revamp/{02-access.md => 04b-access.md} (100%) rename plans/backend/revamp/{04-timeline.md => 05a-timeline.md} (100%) rename plans/backend/revamp/{03-intelligence.md => 05b-intelligence.md} (100%) rename plans/backend/revamp/{05-triggers.md => 06-triggers.md} (100%) rename plans/backend/revamp/{06-orchestration.md => 07-orchestration.md} (100%) diff --git a/plans/backend/revamp/01-capabilities.md b/plans/backend/revamp/01-capabilities.md deleted file mode 100644 index dc93e0d00..000000000 --- a/plans/backend/revamp/01-capabilities.md +++ /dev/null @@ -1,165 +0,0 @@ -# Domain ① — Capabilities (CI pivot revamp · WIP) - -> **WIP design doc.** Part of the Phase 4 → end revamp (the old "pipeline"). -> **Scope guardrail:** Phases 1–3 are **SHIPPED and FIXED** — the rename, and the proprietary -> crawler/proxy/billing/stealth/captcha stack. This domain *builds on* them and never modifies them. - -## Role in the universe - -``` -FIXED foundation (done): Acquisition (crawler moat) · Metering (03c) · Identity/Tenancy -OUR scope (this revamp): ▶ Capabilities ◀ → Access → Intelligence + Timeline → Triggers -``` - -Capabilities sits directly on the fixed foundation and is consumed by **Access** (the doors) and -the **Intelligence** hot loop. It is the first domain because everything above it calls into it. - -## Purpose - -Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, -callable verbs** that every door (chat / REST / MCP) and the Intelligence loop consume -**identically**. - -This is the core pivot of old Phase 4: **replace "connector that ingests into the KB" with -"capability that returns data."** A capability is a stateless function you *call and get data -back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached. - -## Foundation it builds on (do not touch) - -- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch; `{content, metadata, - crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS`. - (`app/proprietary/web_crawler/`.) -- `WebCrawlCreditService` (`03c`) — per-success metering + the `bill=` seam. -- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier. - -## The verb set (MVP) — namespaced, nothing top-level - -Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor). -Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …). - -| Verb | Input → Output | Mode | Executes over | Bills (03c) | -|------|----------------|------|---------------|-------------| -| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success | -| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* | -| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place | -| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 | -| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page | - -## `web.scrape` — one array-friendly verb (no separate "batch") - -Scraping always **welcomes an array of URLs**. There is no separate `scrape_batch`. One verb, -one mental model: "give me URLs, I give you per-URL content." - -## Execution mode — a property of the **result**, not two verbs - -A capability does **not** split into sync/async variants. Every call returns a **uniform result -envelope**: - -``` -{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...] , error?: ... } -``` - -- **inline (fast path):** small/fast input → finishes in-request → envelope returns `completed` - with `results` inline → **zero polling**. (single page, one place, a search) -- **job (slow path):** large/slow input → envelope returns `pending` + `job_id` → caller polls a - status endpoint until `completed`, then reads `results`. (batch scrape, Maps search/reviews) - -So "sync" is simply *a job that finished instantly*. Same verb, same response shape, no magic -branching for the caller to reason about. - -- **Threshold** (how many URLs / how heavy before it goes async) is **configurable**. -- A caller may pass `async: true` to **force** a job even for small input (agents that never want - to block). -- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the - **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it. - -## The capability registry — single source of truth - -One registry entry per verb: - -``` -Capability { - name # dotted, e.g. "web.scrape", "maps.search" - input_schema # Pydantic - output_schema # Pydantic - mode # can-complete-inline? + job-capable? - executor # the async fn (wraps Acquisition / Maps actor) - billing_unit # how Metering charges this call -} -``` - -The **three doors are generated from the registry**, not hand-written three times: - -- **chat tool** ← entry (tool def + handler) -- **REST route** ← entry (path + request/response models) -- **MCP tool** ← entry (MCP tool schema + handler) - -…and the **Intelligence** hot loop calls the *same* `executor` directly. Add a verb once → it -lights up on every surface; the I/O contract cannot drift between surfaces. - -## Where it lives / license boundary - -- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store). -- It **imports from** the proprietary Acquisition engine but never moves into it. -- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*` - executors — same boundary rule Phase 3 set (Apache code may import *from* proprietary, not move - *into* it). - -## Billing integration — open to the billing service (not hardcoded per verb) - -Billing is **not** baked into each verb. A capability only **declares a `billing_unit`** in its -registry entry; charging is delegated to the **billing service**, so adding/repricing a unit is a -billing-service concern, not a capability rewrite. `03c`'s `WebCrawlCreditService` is the **first -provider**; new units register with the same service. - -- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit). -- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet). -- captcha attempts → existing per-attempt `web_crawl_captcha` unit (unchanged). -- `web.discover` → register a per-search unit (or mark free) **with the billing service**. - -The point: the registry says *"this verb bills unit X"*; the **billing service owns what unit X -costs and how it settles.** Verbs stay pure; pricing stays pluggable. - -## The Maps actor (clarification) - -- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). -- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), - not raw markdown — which is exactly what the Intelligence/Timeline layer needs to diff reliably. -- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint - on the capability. The capability never knows what decision it serves. - -## Relationship to the drafted Phase 4 - -- **04b (source-discovery)** → **absorbed** as `web.discover` + relocating Linkup/Baidu keys from - per-connector `config` to platform env (SearXNG already env-based). Tavily/Serper dropped. -- **04a (connector taxonomy + MCP routing)** → **demoted to backward-compat hygiene.** The - capability registry is the new spine, so the Type-1/Type-2 *connector* registry is no longer - central. 04a's "turn off branded connectors" stays a small, separable cleanup for the legacy - chat agent; the **BYO-`MCP_CONNECTOR`** routing fix belongs to the **Access/Conversation** domain - (it lets the agent use the *user's* external tools — a different concept from our capabilities). - -## Deferred (not in MVP) - -- **Recursive site crawl** (follow links across a whole site) — doesn't exist today; the - "crawl a topic" story is `web.discover` → `web.scrape`. -- A generic **`web.extract`** (LLM-structured extraction as a standalone verb) — structured - extraction lives in the Intelligence loop for MVP; promote to a capability later if devs want it. -- Additional platform namespaces (`linkedin.*`, `amazon.*`, …). - -## Locked decisions - -1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level. -2. One array-friendly `web.scrape` (no separate batch verb). -3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs. -4. Build the thin job model now (Maps search / batch scrape force it). -5. One capability registry → generates chat/REST/MCP doors + feeds the Intelligence executor. -6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing - service** (verbs declare a `billing_unit`; `03c` is the first provider, new units register there). -7. Connector → capability is a **replacement** for these data sources, not an extension. - -## Open questions (carry forward) - -- Default async threshold for `web.scrape`. -- The **Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a - **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain - doesn't block on it. diff --git a/plans/backend/revamp/04a-capabilities.md b/plans/backend/revamp/04a-capabilities.md new file mode 100644 index 000000000..8d7a01a4b --- /dev/null +++ b/plans/backend/revamp/04a-capabilities.md @@ -0,0 +1,177 @@ +# Phase 4a — Capabilities (typed verbs over Acquisition) + +> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7). +> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above +> calls into it. +> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome` +> contract, the proxy provider, and the `WebCrawlCreditService` billing seam. +> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies +> them. Locate foundation code by **symbol/grep**, not the line numbers cited. + +## Objective + +Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable +verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume +**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the +KB" with "capability that returns data."** A capability is a stateless function you *call and get data +back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached. + +## Current state (cited) + +- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content, + metadata, crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS` + (`app/proprietary/web_crawler/`). +- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` / + `successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`). +- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`). +- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the + substrate for `web.discover`. +- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope). + +## Target design + +### The verb set (MVP) — namespaced, nothing top-level + +Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor). +Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …). + +| Verb | Input → Output | Mode | Executes over | Bills (03c) | +|------|----------------|------|---------------|-------------| +| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success | +| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* | +| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place | +| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 | +| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page | + +### `web.scrape` — one array-friendly verb (no separate "batch") + +Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model: +"give me URLs, I give you per-URL content." + +### Execution mode — a property of the **result**, not two verbs + +A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**: + +``` +{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... } +``` + +- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline → + zero polling. (single page, one place, a search) +- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until + `completed`. (batch scrape, Maps search/reviews) + +So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller. + +- **Threshold** (how many URLs / how heavy before async) is **configurable**. +- A caller may pass `async: true` to **force** a job even for small input (agents that never block). +- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the + **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it. + +### The capability registry — single source of truth + +One registry entry per verb: + +``` +Capability { + name # dotted, e.g. "web.scrape", "maps.search" + input_schema # Pydantic + output_schema # Pydantic + mode # can-complete-inline? + job-capable? + executor # the async fn (wraps Acquisition / Maps actor) + billing_unit # how Metering charges this call +} +``` + +The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool, +REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly. +Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces. + +### Billing — open to the billing service (not hardcoded per verb) + +A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the +**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite. +`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service. + +- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit). +- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet). +- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged). +- `web.discover` → register a per-search unit (or mark free) **with the billing service**. + +The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs +stay pure; pricing stays pluggable. + +### The Maps actor (clarification) + +- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …). +- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw + markdown — exactly what the Intelligence/Timeline layer needs to diff reliably. +- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on + the capability. The capability never knows what decision it serves. + +### Where it lives / license boundary + +- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store). +- It **imports from** the proprietary Acquisition engine but never moves into it. +- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*` + executors — same boundary rule Phase 3 set. + +## Work items + +1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import. +2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL + result shape; declare the `web_crawl` `billing_unit`. +3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed); + declare its `billing_unit` (or free). +4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs. +5. **Uniform envelope**: shared `completed|pending` result type returned by every executor. +6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor. +7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code). + +## Tests + +- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns + `pending` + `job_id`; the job completes to `results`. +- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail + the batch. +- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED` + free; disabling the billing flag makes it a no-op. +- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb. +- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set. + +## Risks / trade-offs + +- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when + the actor lands (doesn't block `web.*`). +- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never + grows back into `pipeline_runs`. +- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact + mid-run behavior is an implementation-time call. + +## Resolved decisions + +1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level. +2. One array-friendly `web.scrape` (no separate batch verb). +3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs. +4. Build the thin job model now (Maps search / batch scrape force it). +5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`). +6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service** + (verbs declare a `billing_unit`; `03c` is the first provider, new units register there). +7. Connector → capability is a **replacement** for these data sources, not an extension. + +## Out of scope (hand-offs) + +- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`. +- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless. +- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a + **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain + doesn't block on it. +- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces + (`linkedin.*`, `amazon.*`) — deferred. +- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing + fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`. + +## Open questions (carry forward) + +- Default async threshold for `web.scrape`. +- Whether `web.discover` is metered or free. diff --git a/plans/backend/revamp/02-access.md b/plans/backend/revamp/04b-access.md similarity index 100% rename from plans/backend/revamp/02-access.md rename to plans/backend/revamp/04b-access.md diff --git a/plans/backend/revamp/04-timeline.md b/plans/backend/revamp/05a-timeline.md similarity index 100% rename from plans/backend/revamp/04-timeline.md rename to plans/backend/revamp/05a-timeline.md diff --git a/plans/backend/revamp/03-intelligence.md b/plans/backend/revamp/05b-intelligence.md similarity index 100% rename from plans/backend/revamp/03-intelligence.md rename to plans/backend/revamp/05b-intelligence.md diff --git a/plans/backend/revamp/05-triggers.md b/plans/backend/revamp/06-triggers.md similarity index 100% rename from plans/backend/revamp/05-triggers.md rename to plans/backend/revamp/06-triggers.md diff --git a/plans/backend/revamp/06-orchestration.md b/plans/backend/revamp/07-orchestration.md similarity index 100% rename from plans/backend/revamp/06-orchestration.md rename to plans/backend/revamp/07-orchestration.md From 129e9039a8538003a361ed898a09344e907f60b1 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:07:39 +0200 Subject: [PATCH 14/21] docs(revamp): reshape Access into Phase 4b --- plans/backend/revamp/04b-access.md | 185 ++++++++++++++--------------- 1 file changed, 91 insertions(+), 94 deletions(-) diff --git a/plans/backend/revamp/04b-access.md b/plans/backend/revamp/04b-access.md index 69d5cec60..3cea58d45 100644 --- a/plans/backend/revamp/04b-access.md +++ b/plans/backend/revamp/04b-access.md @@ -1,139 +1,136 @@ -# Domain ② — Access / Surfaces (CI pivot revamp · WIP) +# Phase 4b — Access / Surfaces (chat · REST · MCP doors) -> **WIP design doc.** Part of the Phase 4 → end revamp. Sits on top of Domain ① (Capabilities). -> **Scope guardrail:** Phases 1–3 are SHIPPED and FIXED. Identity/Tenancy, API keys, the chat -> agent + its tool registry, the streaming layer, and Metering (`03c`) already exist — Access -> *reuses* them. +> Part of **Phase 4 — Capabilities & Access**. Sits on top of `04a-capabilities.md` (the registry it exposes). +> **Build after** `04a`. Together, `04a + 04b` ship **Product A** (stateless utility) — revenue day one. +> **Depends on** SHIPPED infra: Identity/Tenancy, **API keys**, the chat agent + its tool registry, the +> streaming layer, and Metering (`03c`). Access *reuses* them — it builds none net-new. +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. Locate code by **symbol/grep**, not the cited lines. -## Role in the universe +## Objective -``` -FIXED: Acquisition · Metering (03c) · Identity/Tenancy · API keys · chat agent · streaming -SCOPE: Capabilities → ▶ Access ◀ → Intelligence + Timeline → Triggers -``` +Expose the capability registry (`04a`) to callers, **authenticated + metered**, through three doors — +**chat tools, REST + API keys, MCP server** — all **generated from the one registry** so the I/O +contract cannot drift between surfaces. Access contains **no business logic**: every door is the same +thin adapter. -Access is the set of **doors** onto the capability registry. It contains **no business logic** — -every door is the same thin adapter. +## Current state (cited) -## Purpose +- **API keys** — existing per-workspace key infra (reuse; do **not** build net-new); billed to the + workspace owner via `03c`. +- **Chat agent + tool registry** — capability tools register *into* the existing registry; the seed is + `research/tools/scrape_webpage.py` (capability executor + access door + `03c` turn-accumulator billing). +- **Slow-job pattern** — `subagents/builtins/deliverables/deliverable_wait.py`: dispatch a Celery task, + poll the row's `status` until `READY`/`FAILED` (1.5s cadence), bounded by + `SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s); `deliverables/tools/podcast.py` is the + "return now + live card tracks progress" model. +- **BYO-`MCP_CONNECTOR`** (old `04a`) — the user's own external MCP tools inside our chat agent; its + routing-map gap is the one "connector" worth fixing, and it lands **here**. -Expose the capability registry to callers, **authenticated + metered**. One adapter shape for every -verb, on every door: +## Target design + +### The adapter shape (identical on every door) ``` parse input → validate against verb.input_schema → authn/authz → meter-gate (03c) - → call the SAME executor → serialize verb.output_schema → return the uniform envelope + → call the SAME executor (04a) → serialize verb.output_schema → return the uniform envelope ``` -## The three doors (locked order: chat → REST → MCP) +### The three doors (locked order: chat → REST → MCP) | Door | Who | Auth | Status | |------|-----|------|--------| | **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) | -| **REST + API keys** | external developers (Product A) | **existing API-key infra** (reuse, do not build net-new) | **public day one** | -| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — *chosen at implementation* | **fast-follow** after chat+REST | +| **REST + API keys** | external developers (Product A) | **existing API-key infra** | **public day one** | +| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — chosen at implementation | **fast-follow** | -All three are **generated from the one capability registry** (Domain ①). REST being public day one -is cheap precisely because the routes are generated, not hand-written — it's a go-to-market choice, -not an engineering cost. +REST being public day one is cheap precisely because the routes are **generated**, not hand-written — +it's a go-to-market choice, not an engineering cost. -## Natural language is THE surface (verbs are internal) — non-negotiable +### Natural language is THE surface (verbs are internal) — non-negotiable -The human-facing product is **the conversation**. A user **never** names a verb, fills an -`input_schema`, or knows "Product A vs B" exists — they describe a *need* or a *worry* in plain -language and the agent does the rest. Verbs/schemas/jobs/deltas are things the agent manages **on the -user's behalf**. (The raw typed verbs are exposed only on the REST/MCP doors, which serve -**developers/external agents**, not humans — that's the whole reason those doors exist separately.) - -The chat agent therefore owns three responsibilities on every message: - -1. **Understand intent** (what does the user actually want?). -2. **Pick & fill the verbs** — infer URLs / queries / locations / place refs from the conversation and - compose one or more capability calls (incl. the natural chains, e.g. - `discover → scrape`, `search → place → reviews`). -3. **Answer in plain language** (results, not envelopes). +The human-facing product is **the conversation**. A user **never** names a verb, fills an `input_schema`, +or knows "Product A vs B" exists — they describe a *need* in plain language and the agent does the rest. +Raw typed verbs are exposed only on the REST/MCP doors (devs/external agents). The chat agent owns three +responsibilities per message: **understand intent** → **pick & fill verbs** (incl. chains like +`discover → scrape`, `search → place → reviews`) → **answer in plain language** (results, not envelopes). ### The intent router (the one new orchestration rule) -The agent classifies each request along the stateless/stateful line **from the language**, so the -user never has to: +The agent classifies each request along the stateless/stateful line **from the language**: ``` -"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless) -"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start the Tracker setup flow (Product B, stateful → ③) +"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless) +"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start Tracker setup (Product B, stateful → 05b) +ambiguous → ask ONE question: "just this once, or should I keep watching it for you?" ``` -- **One-shot** → orchestrate verbs now, synthesize an answer; nothing persists beyond chat. -- **Standing concern** → hand off to the Intelligence setup flow (`03`): sample-fetch → agent proposes - schema/thresholds/identity → user validates & locks → Tracker runs on a trigger. -- **Ambiguous** → ask exactly **one** clarifying question — *"just this once, or should I keep watching - it for you?"* — which is the entire A-vs-B decision expressed in human terms. +This router is the friendly seam between the two products; it lives in the chat door (its prompt lives +in the `intelligence_agent`, `07`) and is the only human-facing decision point. -This router is the friendly seam between the two products; it lives in the chat door and is the only -human-facing decision point. +### The two MCP directions (keep distinct) -## The two MCP directions (keep distinct) +- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new): Streamable-HTTP `/mcp`, + stateless, bounded/paginated outputs, untrusted inputs, least-privilege (read-only verbs). +- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` (old 04a): the user's own external MCP tools inside our + chat agent. The 04a routing-gap fix lands **here**, not in Capabilities. -- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new). "External agents gain - the real web." Remote Streamable-HTTP `/mcp`, stateless, bounded/paginated outputs, untrusted - inputs, least-privilege (read-only verbs). Auth depth decided at implementation. -- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` from old 04a: the user's *own* external MCP tools - inside our chat agent. This is the only "connector" worth keeping; the 04a routing-gap fix lands - **here** (Access/Conversation), not in Capabilities. +### Chat ↔ slow jobs — reuse the existing background-worker pattern -## Chat ↔ slow jobs — reuse the existing background-worker pattern +Do **not** invent a chat-async mechanism. A slow verb (`web.scrape` over many URLs, `maps.search`, +`maps.reviews`) invoked from chat dispatches the **job** (the `04a` job record) and uses the +`deliverable_wait` poll-until-terminal path; the capability **job record's `status`** is what the helper +polls. Most calls finish inside the poll window → results inline; genuinely long ones surface a tracked +card. REST/MCP expose the same job via `GET /v1/jobs/{id}` (and an MCP equivalent). -Do **not** invent a chat-async mechanism. The deliverables stack already solves it: +## Work items -- `subagents/builtins/deliverables/deliverable_wait.py` — a shared **poll-until-terminal** helper: - dispatch the Celery task, poll the row's `status` until `READY`/`FAILED` (1.5s cadence), return a - real terminal outcome. Bounded by `SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s) in - multi-agent mode. -- `deliverables/tools/podcast.py` — the "return now + a live card tracks progress" model for very - long work; streaming emission frames live under - `tasks/chat/streaming/handlers/tools/deliverables/...`. +1. **Door generator**: from the `04a` registry, emit (a) chat tool defs + handlers, (b) REST routes + + request/response models, (c) MCP tool schemas + handlers — one adapter shape. +2. **REST surface**: public routes + API-key auth (reuse existing) + the `03c` meter-gate; `GET /v1/jobs/{id}`. +3. **Chat tools**: generalize `scrape_webpage` into the registry-backed set; wire slow verbs through + `deliverable_wait` polling the `04a` job record's `status`. +4. **Intent router**: the A-vs-B classifier (its home is the `07` subagent prompt; the seam is here). +5. **MCP server (fast-follow)**: Streamable-HTTP `/mcp`, least-privilege; auth depth chosen at implementation. +6. **Consume-MCP fix**: repair the BYO-`MCP_CONNECTOR` routing-map gap (old 04a). -**Mapping to capabilities:** a slow verb (`web.scrape` over many URLs, `maps.search`, `maps.reviews`) -invoked from chat dispatches the **job** (Domain ① job record) and uses the `deliverable_wait` -poll-until-terminal path; the capability **job record's `status`** is what the helper polls (the -analogue of the podcast/artifact status row). Most calls finish inside the poll window → the tool -returns results inline; genuinely long ones surface a tracked card. REST/MCP doors expose the same -job via `GET /v1/jobs/{id}` (and an MCP equivalent). +## Tests -## Reused / fixed (not built here) +- **Generated parity**: a verb added to the registry appears on chat + REST (+ MCP) with identical I/O. +- **Auth + meter**: REST without a valid key → 401; an over-budget call → blocked by the `03c` gate + before execute; a success charges once. +- **Chat slow verb**: a many-URL `web.scrape` returns inline when fast; surfaces a tracked card when long; + `GET /v1/jobs/{id}` reflects the same terminal status. +- **Intent router**: "compare X and Y" → one-shot; "watch X weekly" → Tracker setup handoff; ambiguous → + exactly one clarifying question. +- **Consume-MCP**: a configured BYO MCP tool is reachable by the chat agent (routing-gap closed). -- **API keys** — existing infra; keys scope to a workspace; billed to the workspace owner via `03c`. -- **Identity & Tenancy** — workspace/user/permission checks on every door. -- **Chat agent + tool registry** — capability tools are *registered into* the existing registry. -- **Streaming layer** — existing SSE/card emission for chat job progress. -- **Metering (`03c`)** — the balance gate before execute + charge after, on every door. +## Risks / trade-offs -## Relationship to the drafted Phase 4 +- **Public REST day one** → needs bounded inputs + per-key quotas / abuse posture (design alongside launch). +- **MCP auth depth** deferred to implementation (OAuth 2.1 vs bearer) — don't over-build before a consumer exists. +- **Serve-vs-consume MCP confusion** — keep the two directions explicitly separate in code and docs. -- **04a BYO-`MCP_CONNECTOR` routing fix** → lands here (we *consume* the user's MCP tools). -- Old connector-config routes for data sources are **not** the surface anymore — the capability - REST/MCP/chat doors are. Legacy branded connectors stay only for backward-compat (separate - hygiene task). +## Resolved decisions -## Locked decisions - -0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the - chat agent understands intent, picks & fills verbs, and answers in plain language. An **intent - router** classifies one-shot (Product A) vs standing-concern (Product B) from the language, asking - one clarifying question only when ambiguous. Raw verbs are exposed solely on REST/MCP (dev/agent - doors). +0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the chat + agent understands intent, picks & fills verbs, answers in plain language; an intent router classifies + one-shot (A) vs standing-concern (B), asking one question only when ambiguous. Raw verbs live only on REST/MCP. 1. Three doors, generated from the capability registry; order chat → REST → MCP. 2. REST is **public day one** (cheap; go-to-market choice). 3. API keys: **reuse existing infra**, billed to workspace owner. 4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation. -5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the - capability job record's `status`. -6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the 04a fix is the - consume side. +5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the `04a` job record's `status`. +6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the old-04a fix is the consume side. + +## Out of scope (hand-offs) + +- **The verbs themselves** (executors, registry, billing units) → `04a`. +- **The CI subagent + its prompt/playbook** (where the intent router actually lives) → `07`. +- **Stateful flows** (Tracker crafting, refresh, timeline reads) → `05b`/`05a`, surfaced via chat tools in `07`. +- Legacy branded connectors stay only for backward-compat (separate hygiene task). ## Open questions (carry forward) - MCP auth depth (decide at implementation). -- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas) — design alongside the - public launch. -- Whether `web.discover` is metered or free (carried from Domain ①). +- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas). +- Whether `web.discover` is metered or free (carried from `04a`). From 3b92e40bf6c2be6427c4c35a3854ebba9a41aad3 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:08:28 +0200 Subject: [PATCH 15/21] docs(revamp): reshape Timeline into Phase 5a --- plans/backend/revamp/05a-timeline.md | 103 ++++++++++++++++++--------- 1 file changed, 68 insertions(+), 35 deletions(-) diff --git a/plans/backend/revamp/05a-timeline.md b/plans/backend/revamp/05a-timeline.md index d66bc23e6..436257cd4 100644 --- a/plans/backend/revamp/05a-timeline.md +++ b/plans/backend/revamp/05a-timeline.md @@ -1,23 +1,36 @@ -# Domain ④ — Timeline (the moat asset) (CI pivot revamp · WIP) +# Phase 5a — Timeline (the moat asset) -> **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `03-intelligence.md` (the process -> that writes it). -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is -> **not** the Knowledge Base (documents/embeddings) and **not** `automation_runs`. +> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes +> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them. +> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not** +> the Knowledge Base (documents/embeddings) and **not** `automation_runs`. -## Purpose +## Objective Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** — -accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — -no change, no row.** Storage grows with the *rate of change*, not the number of runs. +accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no +change, no row.** Storage grows with the *rate of change*, not the number of runs. -## The state / process split +## Current state (cited) -Intelligence (`03`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain -today, future dashboards / alerts / the deferred resale product — read the Timeline **directly, -without running the loop**. That separation is the reason Timeline is its own domain. +- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is + what this phase introduces. +- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero + publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog + (this is the **fact** analog, deliberately separate). +- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document( + exclude_metadata=True)` to compute the `content_hash` stored here. -## The three stores +## Target design + +### The state / process split + +Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain +today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without +running the loop**. That separation is why Timeline is its own domain. + +### The three stores | Store | Role | Write pattern | |-------|------|---------------| @@ -26,9 +39,9 @@ without running the loop**. That separation is the reason Timeline is its own do | `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten | **The timeline = the change log read in order.** To reconstruct a past point: take Current state and -replay deltas backward (north-star tooling; not built in MVP — we just *store* the deltas). +replay deltas backward (north-star tooling; MVP just *stores* the deltas). -## Data model sketch (new tables) +### Data model sketch (new tables) ``` tracked_entities @@ -38,7 +51,7 @@ tracked_entities entity_current_state entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema) · content_hash · last_checked_at · updated_at - # overwritten each material run; content_hash powers the hot-loop cheap pre-check (03 step 2) + # overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2) entity_changes # the append-only change log id · entity_id (FK) · tracker_id · captured_at @@ -49,33 +62,47 @@ entity_changes # the append-only change log · note TEXT NULL # optional agent rationale (the "why material") ``` -- **No change → no row** in `entity_changes`; an unchanged refresh only bumps - `entity_current_state.last_checked_at`. -- `decided_by` records whether code or the agent ruled the change material (provenance seed for the - north-star explainability work). +- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`. +- `decided_by` records whether code or the agent ruled the change material (provenance seed). -## What it is NOT +### What it is NOT -- **Not the KB** — no `Document` rows, no embeddings, no indexing pipeline. (The meeting's "don't - index crawled data" holds.) +- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.) - **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store. - **Not a resurrected `pipeline_runs`.** -## Where it lives +### Where it lives -- New CI-owned tables (in `app/db.py` alongside the other core entities, or a small `app/timeline/` - package — decide at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps - extractor, not here). +- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide + at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor). - Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend). -## MVP cut vs north star +## Work items -- **MVP:** the three stores · single entity per Tracker · deltas with `decided_by`/`source_ref` · - `content_hash` pre-check support. -- **North star (deferred):** backward-replay reconstruction queries · trend/series read APIs · - coverage-confidence metadata · multi-entity scale · the resale/data-product surface. +1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration. +2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop. +3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side. +4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit. +5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred). -## Locked decisions +## Tests + +- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged. +- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`. +- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls. +- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row. +- **Read API**: `query_timeline` returns deltas in `captured_at` order. + +## Risks / trade-offs + +- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against — + baseline semantics (silent establish vs flood the log) are an implementation-time call. +- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very + chatty entities is deferred. +- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity + guard (lean: yes) but adds a write-path dependency on `05b`'s lock. + +## Resolved decisions 1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`. 2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change. @@ -84,9 +111,15 @@ entity_changes # the append-only change log 5. `decided_by` on changes records the code-vs-agent materiality provenance. 6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later). +## Out of scope (hand-offs) + +- **The writer** (hot loop, materiality, schema lock) → `05b`. +- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the + resale/data-product surface → north star (deferred). +- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella. + ## Open questions (carry forward) - ORM home: `app/db.py` vs a dedicated `app/timeline/` package. -- Whether `entity_current_state.fields` should be validated against the Tracker's locked `field_schema` - at write time (lean: yes, cheap integrity guard). +- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes). - Retention / archival policy for very high-velocity entities (deferred). From 5c2428541def6513284e94bb3f28f34ec553119e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:09:32 +0200 Subject: [PATCH 16/21] docs(revamp): reshape Intelligence into Phase 5b --- plans/backend/revamp/05b-intelligence.md | 210 +++++++++++++---------- 1 file changed, 115 insertions(+), 95 deletions(-) diff --git a/plans/backend/revamp/05b-intelligence.md b/plans/backend/revamp/05b-intelligence.md index f37d4c8cb..97b915d53 100644 --- a/plans/backend/revamp/05b-intelligence.md +++ b/plans/backend/revamp/05b-intelligence.md @@ -1,26 +1,37 @@ -# Domain ③ — Intelligence (the decision-grounded engine) (CI pivot revamp · WIP) +# Phase 5b — Intelligence (the decision-grounded engine) -> **WIP design doc.** Part of the Phase 4 → end revamp. Pairs with `04-timeline.md` (the state it writes). -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** -> the automations subsystem. +> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes). +> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the +> **Product B engine** (the Tracker, locked schema, hot loop, deltas). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the +> automations subsystem. > **Name:** the standing-concern primitive is the **Tracker**. -## The stateless / stateful line - -``` -STATELESS (Product A): ① Capabilities + ② Access → call → data → bill, nothing persists -STATEFUL (Product B): ③ Intelligence + ④ Timeline → the Timeline IS the state -``` - -Intelligence is the **process** that mutates state; Timeline (`04`) is the **state**. Everything -below the Access→Intelligence boundary stays pure functions. - -## Purpose +## Objective Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent -judges, code computes.** This replaces the old "pipeline" as the *standing concern*. +judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the +**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the +Access→Intelligence boundary stays pure functions. -## The primitive — `Tracker` +``` +STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists +STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state +``` + +## Current state (cited) + +- **Capability executors** (`04a`) — called directly by the loop (not through a door). +- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)` + produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`). +- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`) + + the automations executor's PENDING→running idempotency gate (safe under Celery `acks_late`); and the + chat **job record** (`04a`) + `deliverable_wait`. **No new run table.** +- **CI context uploads** — the existing folder-upload / `destination_folder` machinery + KB retrieval. + +## Target design + +### The primitive — `Tracker` A saved, decision-grounded subject that accumulates structured signal over time: @@ -32,129 +43,138 @@ A saved, decision-grounded subject that accumulates structured signal over time: | `status` | `draft` → `locked`/`active` | One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred -(the Timeline model stays multi-entity-ready so it's additive). +(the `05a` model stays multi-entity-ready so it's additive). -## Setup (once) — the agent-designed schema flow (IN MVP) +### Setup (once) — the agent-designed schema flow (IN MVP) The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is -derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first, -no UI needed): +derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first): -1. **Bind** a capability + input (the thing to track). -2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data, - not a hallucinated shape. -3. **Agent drafts the `definition`** from `decision` + the sample: - - `field_schema` — the fields that matter + types (structured-output contract). - - `materiality` — per-field rules (numeric thresholds where possible; "ask agent" otherwise). - - `identity_rule` — the stable entity key (e.g. Maps `place_id`, canonical URL). - - a reserved **`notable_signals`** escape-hatch field for the unanticipated. +1. **Bind** a capability + input. +2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data. +3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types), + `materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable + entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field. 4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run. 5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how `automations` snapshots `definition`). -## The hot loop (per refresh) — `refresh(tracker)` +### The hot loop (per refresh) — `refresh(tracker)` -1. **Crawl** — call the bound capability (Domain ①) → raw data. -2. **Cheap pre-check** — content hash via the existing - `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)`; identical hash to the - stored `content_hash` → stamp `last_checked_at`, **stop** (no LLM cost). +1. **Crawl** — call the bound capability (`04a`) → raw data. +2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`, + **stop** (no LLM cost). 3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does **not** invent fields. Unanticipated observations go into `notable_signals`. -4. **Diff (code)** — deterministic compare of the new record vs Current state (`04`) → raw deltas. +4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas. 5. **Judge — the materiality split:** - - **deterministic (code):** numeric / clear-cut rules from `materiality`, applied for free, 100% - reproducible. e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ price wobble - → noise`, `any hours change → material`. + - **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100% + reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`). - **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new - `notable_signals` entry, "does this matter *for the decision*?" → one LLM call rules - material/noise. -6. **Append** — if material: write a Change + update Current state (`04`). Else: only `last_checked_at`. + `notable_signals` entry → one LLM call rules material/noise. +6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`. **No change → no row.** **Worked example (`maps.place` refresh):** ``` -rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM) -review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM) -hours unchanged → no delta -description reworded → code: no rule → ASK AGENT → NOISE +rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM) +review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM) +hours unchanged → no delta +description reworded → code: no rule → ASK AGENT → NOISE ⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick. ``` -## Where it lives / decoupling +### Refresh execution & idempotency — ride the invoking surface (no new run table) -- New **Apache-2** package `app/intelligence/` (the schema-design agent, the hot loop, the materiality - evaluator). Calls capability `executor`s directly (not through a door). -- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is - the **Triggers** domain's concern — Intelligence has **no dependency** on any scheduler. +`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever +surface invoked it**: -## Refresh execution & idempotency — ride the invoking surface (no new run table) +- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`** + is the run record and the automations executor provides the **PENDING→running idempotency gate**. This is + exactly the rigor old `06` hand-built — reused, not re-written. +- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`. +- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via + the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No + run-level `charged_micros` ledger required for MVP. -`refresh(tracker)` is a **headless unit of work**; the **run/audit record + idempotency live on whatever -surface invoked it**, so we do *not* rebuild old Phase-6 `pipeline_runs`: +Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed. -- **Recurring (in-app):** invoked by the **CI automation action** (`05`) → the existing - **`AutomationRun`** is the run record (status / error / timing / `step_results`) and the automations - executor already provides the **PENDING→running idempotency gate** (safe under Celery `acks_late` - redelivery). This is exactly the rigor old `06` hand-built — reused, not re-written. -- **Chat (manual / agent):** invoked via the chat **job record** (`01`) + `deliverable_wait` — status - lives there. -- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once - via the billing service (`01`); the **content-hash pre-check** (step 2) is what prevents needless - re-crawls/charges on an unchanged page. So no run-level `charged_micros` ledger is required for MVP. +### User-supplied context files (the F idea, generalized) -Net: the only genuinely new state is the **Timeline** (`04`); execution accounting is borrowed from -`AutomationRun` / the job record. +When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** — +uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies +only to *crawled* data.)** The CI-specific part is **organization + use**: -## User-supplied context files (the F idea, generalized) - -When a user uploads a file *in a CI chat* (e.g. "our own price list", a competitor brochure), it goes -into the **KB as normal** — uploads create `Document`s and are indexed/embedded, exactly as today. -**(The "don't index" rule applies only to *crawled* data, not to user uploads.)** The CI-specific part -is purely **organization + use**: - -- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload / - `destination_folder` machinery), so the chat's reference files are scoped together. -- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder — when ruling - materiality: +- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload machinery). +- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder: ``` competitor price 12.00 → 9.90 + user's context file says "our price is 10.00" → agent: competitor crossed *below our price* → MATERIAL (and explain why) ``` -So the user's private context **shapes what counts as material** — a real differentiator — and it -reuses the existing KB upload + folder + retrieval machinery (nothing new). **MVP-optional** (the loop -works without it); design the seam now so the judge can read the folder later. +The user's private context **shapes what counts as material**. Reuses existing KB upload + folder + +retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now. -## MVP cut vs north star +### Where it lives / decoupling -- **MVP:** agent-designed-schema flow (conversational, sample-grounded, human-locked) · single entity - per Tracker · the hot loop with content-hash pre-check + code-threshold + agent-on-ambiguous · - `refresh(tracker)` fired manually / by agent / by external cron. -- **North star (deferred):** schema **auto-evolution** from recurring `notable_signals` · multi-entity - Trackers (`maps.search`) · backward-replay reconstruction · coverage-confidence · full - provenance/explainability · the resale/data-product stage. +- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls + capability `executor`s directly. +- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is + the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler. -## Locked decisions +## Work items + +1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`. +2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version. +3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback. +4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`). +5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table. +6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder. + +## Tests + +- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped. +- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields. +- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword → + `decided_by=agent`; sub-threshold → noise, no row. +- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state. +- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill. +- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame. + +## Risks / trade-offs + +- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an + implementation-time call (no architecture impact). +- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM. +- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version. + +## Resolved decisions 1. `Tracker` is the standing-concern primitive; replaces "pipeline". -2. Stateless (①②/Product A) vs stateful (③④/Product B) is the Access→Intelligence boundary. +2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary. 3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned. 4. Single entity per Tracker for MVP. 5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous. 6. Content-hash pre-check short-circuits unchanged pages before any LLM spend. 7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic. -8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job - record; billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`04`) is - new state. +8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job record; + billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state. 9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed), - routed to a dedicated `Folder`, and may feed the judge step via KB retrieval. The "don't index" - rule is for *crawled* data only. MVP-optional seam. + routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled* + data only. MVP-optional seam. + +## Out of scope (hand-offs) + +- **The state tables** (the three stores, content-hash, read API) → `05a`. +- **When refresh fires + recurrence/delivery** → `06` (Triggers). +- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`. +- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the + resale/data-product stage → north star (deferred). ## Open questions (carry forward) -- How the schema-design agent surfaces the "review & lock" step before the frontend exists (pure-chat - confirmation for MVP?). +- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?). - Versioning policy on re-lock (new version vs in-place) — lean new version. - Where the schema-design agent itself runs (a setup capability? a chat sub-flow?). -- Context-folder → judge wiring (how much of the folder to load; per-Tracker vs per-chat scope). +- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope). From e5d670c609929053676feb102f20bf62794eafd5 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:10:14 +0200 Subject: [PATCH 17/21] docs(revamp): reshape Triggers into Phase 6 --- plans/backend/revamp/06-triggers.md | 105 +++++++++++++++++++--------- 1 file changed, 72 insertions(+), 33 deletions(-) diff --git a/plans/backend/revamp/06-triggers.md b/plans/backend/revamp/06-triggers.md index 087724143..00dd5008c 100644 --- a/plans/backend/revamp/06-triggers.md +++ b/plans/backend/revamp/06-triggers.md @@ -1,20 +1,32 @@ -# Domain ⑤ — Triggers (the pluggable refresh clock) (CI pivot revamp · WIP) +# Phase 6 — Triggers (the pluggable refresh clock) -> **WIP design doc.** Part of the Phase 4 → end revamp. The thinnest domain. -> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This domain is **decoupled** — Intelligence (`03`) -> has no dependency on it, and **automations is at most one optional adapter, never required**. +> **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes +> `refresh(tracker)`). +> **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED +> automations subsystem (its schedule selector + `AutomationRun` + delivery). +> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no +> dependency on it, and **automations is at most one optional adapter, never required**. -## Purpose +## Objective Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** — -and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger -and the engine still works. +and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and +the engine still works. This replaces the old Phase-6 cron scheduler — and the resolution is **not** to +rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path. -This replaces Phase 6's cron scheduler — and the resolution is **not** to rebuild a scheduler at all, -but to **reuse the automations subsystem** for the in-app recurring path (it already has a hardened -clock + run record + delivery). A trigger only ever calls `refresh(tracker)`. +## Current state (cited) -## The adapters +- **`refresh(tracker)`** — the headless unit of work exposed by `05b`. +- **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED` + claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the + `croniter` util. +- **`AutomationRun`** — the existing run record + PENDING-gate (audit + idempotency). +- **Automations output/delivery** — the existing path that carries results to the user. +- **Access routes** (`04b`) — where the external-cron and manual REST endpoints live. + +## Target design + +### The adapters | Adapter | Fired by | MVP? | |---------|----------|------| @@ -23,38 +35,61 @@ clock + run record + delivery). A trigger only ever calls `refresh(tracker)`. | **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) | | **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)` **and delivers** the material changes | ✅ (the in-app recurrence + alert path) | -## Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) +### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape) -The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes"**. Instead -of building a bespoke tick (which — done correctly — still needs `FOR UPDATE SKIP LOCKED` claiming, -`next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), we **add a CI *action* -to the existing automations subsystem**: +The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes."** Instead of +building a bespoke tick, we **add a CI *action* to the existing automations subsystem**: - **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new - scheduler. **(closes the old Gap B — scheduler rigor.)** -- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new - run table. **(closes the old Gap A — run/idempotency, see `03`.)** -- **Delivery / alert** → the automation's existing **output/delivery** carries the material changes to - the user. **(closes the old Gap E — alert delivery.)** + scheduler. **(closes old Gap B — scheduler rigor.)** +- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run + table. **(closes old Gap A — run/idempotency, see `05b`.)** +- **Delivery / alert** → the automation's existing **output/delivery** carries the material changes to the + user. **(closes old Gap E — alert delivery.)** -**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and -delivery that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds -the action too constraining, a thin CI-specific shape is the fallback — but the action is the default.) +**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery +that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action +too constraining, a thin CI-specific shape is the fallback — but the action is the default.) ### Decoupling is preserved (automations is still optional) -CI **core** — `refresh(tracker)` + Timeline (`03`/`04`) — has **zero** automations dependency and runs via -manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + delivery +CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs +via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + delivery + audit for in-app users. So we honor "don't glue CI to automations" *and* get its machinery for free. -## Where it lives / decoupling +### Where it lives - The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from `app/intelligence/`. No new scheduler/Beat task. - The **external-cron** and **REST manual** paths are just Access-door routes (`POST - /v1/trackers/{id}/refresh`) — Domain ② plumbing. + /v1/trackers/{id}/refresh`) — `04b` plumbing. -## Locked decisions +## Work items + +1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`. +2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`. +3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that + calls `refresh(tracker)` and routes the resulting material changes into automations delivery. +4. **Concurrency guard**: a per-Tracker in-flight lock (belt-and-suspenders over the automation run-gate). + +## Tests + +- **Decoupling**: `refresh(tracker)` works via manual/agent/external-cron with automations disabled entirely. +- **Recurring path**: a scheduled CI action fires `refresh` on cron and delivers material changes; an + unchanged refresh delivers nothing. +- **Idempotency**: a redelivered scheduled run (same `AutomationRun`) does not double-refresh. +- **External cron**: `POST /v1/trackers/{id}/refresh` triggers exactly one refresh; rejects bad auth. +- **Concurrency**: a second refresh while one is in flight is skipped/queued, not run concurrently. + +## Risks / trade-offs + +- **Action vs shape**: the CI action is the default; a thin CI automation shape is the fallback if the + action proves too constraining. +- **Delivered payload shape** (agent-summarized vs raw deltas since last fire) — implementation-time call. +- **Double-guarding concurrency**: the per-Tracker lock overlaps the automation run-gate, but the lock also + protects the manual/cron paths that don't go through automations. + +## Resolved decisions 1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled. 2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery). @@ -63,10 +98,14 @@ manual / agent / external-cron. Automations is the **optional adapter** that add 4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**. 5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron). +## Out of scope (hand-offs) + +- **`refresh(tracker)` internals** (the hot loop, materiality) → `05b`. +- **The Timeline reads** the delivery summarizes → `05a`. +- **The chat tool surface** for manual/agent refresh → `07` (the `intelligence_agent` toolset). + ## Open questions (carry forward) - CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback). -- What the delivered payload looks like (the material `entity_changes` since last fire — summarized by - the agent, or raw deltas). -- Concurrency: skip a refresh if the Tracker already has one in flight (per-Tracker lock, like the connector - indexing lock) — even with the automation run-gate, belt-and-suspenders. +- What the delivered payload looks like (summarized material changes vs raw deltas since last fire). +- Concurrency: per-Tracker lock granularity (like the connector indexing lock). From 4e77b8397e0303ae61d89a5e7a1cb3183b99a49e Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:11:04 +0200 Subject: [PATCH 18/21] docs(revamp): reshape Orchestration into Phase 7 --- plans/backend/revamp/07-orchestration.md | 150 ++++++++++++++--------- 1 file changed, 89 insertions(+), 61 deletions(-) diff --git a/plans/backend/revamp/07-orchestration.md b/plans/backend/revamp/07-orchestration.md index 81116a591..13e8f4654 100644 --- a/plans/backend/revamp/07-orchestration.md +++ b/plans/backend/revamp/07-orchestration.md @@ -1,95 +1,123 @@ -# Domain ⑥ — Orchestration / Conversation (the CI-expert subagent) (CI pivot revamp · WIP) +# Phase 7 — Orchestration / Conversation (the CI-expert subagent) -> **WIP design doc.** Part of the Phase 4 → end revamp. This is the **human-facing brain** that turns -> Decision-0 ("natural language is the only surface", `02-access.md`) into a real deliverable. +> **Phase 7** of the CI-pivot revamp — the **human-facing brain** that turns "natural language is the only +> surface" (`04b`, Decision 0) into a real deliverable. **Build last** — it sits atop `04`–`06`. +> **Depends on** all prior phases: `04a` verbs, `04b` doors + `deliverable_wait`, `05a`/`05b` Tracker + +> Timeline, `06` refresh triggers. > **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming, > citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin > subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it. -## Why this is a first-class deliverable +## Objective -The intent router + verb-composition + Tracker-crafting are where "user-friendly" is won or lost. Treating -the agent prompt/tooling as plumbing would make the product feel rigid. So the orchestration layer gets -designed, not assumed. +Ship the **`intelligence_agent`** — a net-new builtin CI-expert subagent — so users get the whole product +through plain conversation: intent routing (one-shot vs standing concern), verb composition, Tracker +crafting, and decision-grounded answering. The intent router + verb-composition + Tracker-crafting are +where "user-friendly" is won or lost, so the orchestration layer is **designed, not assumed**. -## The pattern we reuse (verbatim) +## Current state (cited) -Builtin subagents live at `subagents/builtins//`: -``` -agent.py build_subagent(...) → pack_subagent(name, description, system_prompt, tools, ruleset, …) -tools/index.py NAME · RULESET · load_tools(dependencies) -description.md one-liner the router uses to delegate -system_prompt.md the playbook -(middleware) e.g. citation_state -``` -`research/tools/scrape_webpage.py` already shows the **tool = capability executor + access door + 03c -billing** shape: it calls `WebCrawlerConnector.crawl_url`, bills via the turn accumulator -(`get_current_accumulator()` + `WebCrawlCreditService`), returns a typed dict. Our capability/CI tools -follow this exactly. +- **Builtin subagent pattern** — `subagents/builtins//`: `agent.py` (`build_subagent(...)` → + `pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py` + (`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md` + (the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`. +- **Tool shape** — `research/tools/scrape_webpage.py` already demonstrates **tool = capability executor + + access door + 03c billing**: calls `WebCrawlerConnector.crawl_url`, bills via the turn accumulator + (`get_current_accumulator()` + `WebCrawlCreditService`), returns a typed dict. CI tools follow this exactly. +- **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`). -## What's new — the `intelligence_agent` subagent (CI expert) +## Target design -A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), -peer to `research`/`deliverables`. The **main agent delegates** to it whenever the request is -CI-flavored (research a competitor, watch something, analyze a place's reviews); `description.md` is -what makes that routing happen. +### The `intelligence_agent` subagent (CI expert) -It owns the **CI playbook** in `system_prompt.md`: +A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to +`research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research +a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing +happen. It owns the **CI playbook** in `system_prompt.md`: -1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") - → call verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; - ambiguous → ask the single clarifying question. +1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") → call + verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; ambiguous → + ask the single clarifying question. 2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand. -3. **Tracker crafting** (the "crafting" you flagged) — the conversational schema-design flow from `03`: - sample-fetch → propose `field_schema` + materiality + identity → user validates & locks → versioned. -4. **Decision-grounded answering** — read the Timeline (`04`) to answer "what changed / is X pulling +3. **Tracker crafting** — the conversational schema-design flow from `05b`: sample-fetch → propose + `field_schema` + materiality + identity → user validates & locks → versioned. +4. **Decision-grounded answering** — read the Timeline (`05a`) to answer "what changed / is X pulling ahead?" from stored deltas, not by re-deriving from chat history. -## The toolset (what `load_tools` returns) +### The toolset (what `load_tools` returns) | Tool | Wraps | Mode | Billing | |------|-------|------|---------| -| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | Domain ① executors | inline-or-job (slow → `deliverable_wait`) | `03c` turn accumulator (as `scrape_webpage` already does) | -| `craft_tracker(decision, binding)` | `03` schema-design agent | inline (does the sample-fetch + proposes) | sample crawl billed | -| `lock_tracker(draft)` / `update_tracker` | `03` Tracker persistence | inline | — | -| `refresh_tracker(tracker_id)` | `03` `refresh(tracker)` | job → `deliverable_wait` | per capability call | -| `query_timeline(tracker_id, …)` | `04` read API | inline | — | -| `list_trackers()` | `04`/`03` read | inline | — | +| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | `03c` turn accumulator (as `scrape_webpage` already does) | +| `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed | +| `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — | +| `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call | +| `query_timeline(tracker_id, …)` | `05a` read API | inline | — | +| `list_trackers()` | `05a`/`05b` read | inline | — | -- **Capability verbs are a shared tool module** (generated from the Domain ① registry) — `research` - can load the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt. +- **Capability verbs are a shared tool module** (generated from the `04a` registry) — `research` can load + the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt. (`scrape_webpage` is the seed; generalize it into the registry-backed set.) - **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the - existing `deliverable_wait` poll-until-terminal + live-card path (`02-access.md`). + existing `deliverable_wait` poll-until-terminal + live-card path (`04b`). -## Boundaries +### Boundaries -- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `03`/`04` via tools; the hot loop, - materiality, and Timeline writes live in `03`/`04`, callable headless (so REST/MCP and Triggers reach - the same logic with no agent in the loop). +- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `05a`/`05b` via tools; the hot loop, + materiality, and Timeline writes live in `05a`/`05b`, callable headless (so REST/MCP and Triggers reach the + same logic with no agent in the loop). - **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this - subagent entirely and call Domain ① verbs directly — they *want* explicitness. + subagent entirely and call `04a` verbs directly — they *want* explicitness. -## MVP cut vs north star +## Work items -- **MVP:** the `intelligence_agent` subagent + description/prompt · capability verb tools (registry-backed) · - `craft_tracker`/`lock_tracker`/`refresh_tracker`/`query_timeline`/`list_trackers` · intent routing in-prompt. -- **North star (deferred):** richer multi-step CI playbooks (auto competitor discovery → multi-Tracker - setup), proactive "you should watch this" suggestions, cross-Tracker synthesis. +1. **Subagent scaffold**: `subagents/builtins/intelligence_agent/` with `agent.py` / `tools/index.py` / + `description.md` / `system_prompt.md`, packed via `pack_subagent`. +2. **CI playbook prompt**: intent routing (A/B + one clarifying question), verb chains, crafting flow, + decision-grounded answering. +3. **Shared verb tool module**: registry-backed (`04a`) capability tools, reusable by `research` too. +4. **Tracker/Timeline tools**: `craft_tracker` / `lock_tracker` / `update_tracker` / `refresh_tracker` / + `query_timeline` / `list_trackers`. +5. **Slow-verb wiring**: route job-mode tools through `deliverable_wait` + the live card. +6. **Router registration**: `description.md` so the main agent delegates CI-flavored requests here. -## Locked decisions +## Tests -1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing - runtime — not a runtime rebuild. +- **Delegation**: a CI-flavored request routes to `intelligence_agent`; a non-CI request does not. +- **One-shot**: "compare X and Y" composes verbs and answers in plain language, persisting nothing. +- **Standing concern**: "watch X weekly" runs the crafting flow → lock → (refresh via `06`). +- **Crafting**: `craft_tracker` does a real sample-fetch and proposes a schema; `lock_tracker` versions it. +- **Answering**: `query_timeline` answers "what changed?" from stored deltas, not chat history. +- **Slow verb**: a multi-URL scrape surfaces a live card and returns terminal results. + +## Risks / trade-offs + +- **Prompt quality is the product**: intent-router/composition quality is a first-class deliverable, not + plumbing — budget iteration on the prompt. +- **One-shot ownership**: whether `research` keeps generic scraping or the `intelligence_agent` owns all + CI-flavored calls (lean: shared verb tools, CI agent owns the *playbook*). +- **Pre-frontend lock UX**: "review & lock" must render in pure chat for MVP (ties to `05b`'s open Q). + +## Resolved decisions + +1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not + a runtime rebuild. 2. Tools follow the `scrape_webpage` shape: capability executor + access door + `03c` billing. -3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds Tracker/Timeline - tools + the CI prompt. -4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `03`/`04`. +3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds + Tracker/Timeline tools + the CI prompt. +4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`. 5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async. +## Out of scope (hand-offs) + +- **The verbs, doors, engine, state, and triggers** → `04`/`05`/`06` (this phase only *orchestrates* them). +- **Richer multi-step CI playbooks** (auto competitor discovery → multi-Tracker setup), proactive "you + should watch this" suggestions, cross-Tracker synthesis → north star (deferred). +- **Frontend CI surfaces** (cards, dashboards) → frontend umbrella. + ## Open questions (carry forward) -- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls (lean: - shared verb tools, `intelligence_agent` owns the *CI playbook*). -- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `03`'s open Q). +- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls + (lean: shared verb tools, `intelligence_agent` owns the *CI playbook*). +- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `05b`'s open Q). From 0b884d1774c386ec03e3eb3d1666f42c700c4534 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:11:56 +0200 Subject: [PATCH 19/21] docs(revamp): rewrite overview as the 04-07 phase index --- plans/backend/revamp/00-overview.md | 151 +++++++++++++++------------- 1 file changed, 79 insertions(+), 72 deletions(-) diff --git a/plans/backend/revamp/00-overview.md b/plans/backend/revamp/00-overview.md index 92bed3b4d..4cbfea102 100644 --- a/plans/backend/revamp/00-overview.md +++ b/plans/backend/revamp/00-overview.md @@ -1,104 +1,111 @@ -# Phase 4 → end revamp — Overview & reconciliation (CI pivot · WIP) +# Phase 4 → end revamp — Overview & phase index (CI pivot · WIP) -> **What this is.** The CI pivot re-architects the old "pipeline" phases (`04`–`07`) into small, -> single-responsibility domains. This file is the map + the reconciliation against the old plans. -> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl -> billing). The revamp is **Phase 4 → end only**. We do **not** touch 1–3. +> **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small, +> single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace. +> This file is the map + the reconciliation + the subplan index. +> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing). +> The revamp is **Phase 4 → end only**. We do **not** touch 1–3. ## The two products ``` -PRODUCT A — stateless utility ① Capabilities + ② Access +PRODUCT A — stateless utility Phase 04 (Capabilities + Access) call a verb → get data → bill. Nothing persists. -PRODUCT B — decision-grounded CI ③ Intelligence + ④ Timeline (+ ⑤ Triggers to drive it) +PRODUCT B — decision-grounded CI Phase 05 (Intelligence + Timeline) (+ 06 Triggers, 07 Orchestration) a Tracker accumulates structured signal over time. The Timeline is the moat. ``` -## The domain map +## The phases (this revamp) + +| Phase | Subplan(s) | Domain(s) | One line | Ships | +|-------|-----------|-----------|----------|-------| +| **04 — Capabilities & Access** | `04a-capabilities.md` · `04b-access.md` | ① + ② | typed verbs over Acquisition + the chat/REST/MCP doors | **Product A** (revenue day one) | +| **05 — Intelligence & Timeline** | `05a-timeline.md` · `05b-intelligence.md` | ④ + ③ | the Tracker, locked schema, hot loop + the 3-store delta moat | **Product B engine** | +| **06 — Triggers** | `06-triggers.md` | ⑤ | the pluggable refresh clock; recurrence+delivery = optional CI action on automations | recurrence + alerts | +| **07 — Orchestration** | `07-orchestration.md` | ⑥ | the `intelligence_agent` CI-expert subagent (intent routing, verb chains, Tracker crafting) | the human-facing CI experience | + +> **Build order = phase order.** `04a → 04b` (unblocks everything; ships Product A) → `05a → 05b` +> (state before the loop that writes it) → `06` (drive it) → `07` (front it). `04a` first because every +> other phase calls into the capability registry. + +## Domain ↔ phase map ``` - FIXED (Phases 1–3) OUR SCOPE (Phase 4 → end) + FIXED (Phases 1–3) OUR SCOPE (Phase 04 → 07) ┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐ - │ Acquisition │ │ ① Capabilities typed verbs over Acquisition │ - │ proprietary/web_crawler │◄──┤ ② Access chat · REST · MCP doors │ - │ CrawlOutcome · billing (03c) │ │ ③ Intelligence Tracker · schema · hot loop │ - │ │ │ ④ Timeline 3-store delta state (moat) │ - │ │ │ ⑤ Triggers pluggable refresh clock │ - │ │ │ ⑥ Orchestration CI-expert subagent + tools │ + │ Acquisition │ │ 04a Capabilities typed verbs over Acquisition │ + │ proprietary/web_crawler │◄──┤ 04b Access chat · REST · MCP doors │ + │ CrawlOutcome · billing (03c) │ │ 05a Timeline 3-store delta state (moat) │ + │ │ │ 05b Intelligence Tracker · schema · hot loop │ + │ │ │ 06 Triggers pluggable refresh clock │ + │ │ │ 07 Orchestration CI-expert subagent + tools │ └─────────────────────────────────┘ └──────────────────────────────────────────────┘ - data engine (untouched) stateless ①② → stateful ③④ , driven by ⑤ , fronted by ⑥ + data engine (untouched) stateless 04 → stateful 05 , driven by 06 , fronted by 07 ``` -| # | Domain | One line | Doc | -|---|--------|----------|-----| -| ① | Capabilities | Acquisition → typed callable verbs (`web.scrape`, `web.discover`, `maps.*`) | `01-capabilities.md` | -| ② | Access | expose verbs to callers, authed + metered (chat / REST / MCP) | `02-access.md` | -| ③ | Intelligence | the Tracker, agent-designed locked schema, hot loop (agent judges, code computes) | `03-intelligence.md` | -| ④ | Timeline | durable time-shaped truth; deltas not snapshots; no change → no row | `04-timeline.md` | -| ⑤ | Triggers | when a Tracker refreshes; `refresh(tracker)` callers; recurrence+delivery = optional CI action on automations | `05-triggers.md` | -| ⑥ | Orchestration | the human-facing CI-expert subagent (intent routing, verb chains, Tracker crafting) + its tools | `06-orchestration.md` | - ## Reconciliation with the old plans | Old plan | Fate | Where it goes | |----------|------|---------------| -| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in ② (consume-user-MCP); the taxonomy work is not core | -| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in ① | -| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (③) + Timeline tables (④); no `pipeline`/`pipeline_runs` | -| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (③); scheduling/runs/delivery → **reuse automations** via a CI action (⑤) — its selector + `AutomationRun`, **not** a rebuilt cron | -| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature** (uploads still become indexed KB Documents). For CI, uploads are routed to a dedicated **folder** and may feed the judge (`03`, F) | +| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in `04b` (consume-user-MCP); the taxonomy work is not core | +| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in `04a` | +| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (`05b`) + Timeline tables (`05a`); no `pipeline`/`pipeline_runs` | +| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (`05b`); scheduling/runs/delivery → **reuse automations** via a CI action (`06`) — its selector + `AutomationRun`, **not** a rebuilt cron | +| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploads are routed to a dedicated **folder** and may feed the judge (`05b`) | -**Net:** the old `05/06/07` pipeline+KB stack is replaced by `①②③④⑤⑥`. KB indexing of *crawled* data is -out; *user uploads* still work (and gain a CI context-folder role). +**Net:** the old `05/06/07` pipeline+KB stack is replaced by the new `04`–`07`. KB indexing of *crawled* +data is out; *user uploads* still work (and gain a CI context-folder role). -## Build sequence (proposed) - -1. **① Capabilities** — registry + the verb `executor`s over Acquisition (unblocks everything). -2. **② Access** — REST + API keys (day-one endpoints) and chat tools; MCP fast-follow. -3. **④ Timeline** — the three tables (state must exist before the loop writes it). -4. **③ Intelligence** — schema-design agent + hot loop + materiality evaluator. -5. **⑤ Triggers** — manual/agent/external-cron; then the CI **automation action** (recurrence + - delivery) reusing the automations selector + `AutomationRun`. -6. **⑥ Orchestration** — the `intelligence_agent` CI-expert subagent + tools + prompt (the human-facing seam; - built atop ①–④, can grow alongside them). - -## Consolidated open forks (need your call) - -1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? -2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. -3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). - -*(Resolved: primitive **name** = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent).)* -*(Resolved: the "built-in scheduler tick" fork — we reuse the automations schedule selector via a CI -action instead of building a tick. See `05`.)* - -## Confirmed decisions (locked across docs) +## Confirmed decisions (locked across phases) - **Natural language is the only human-facing surface.** Users never name verbs; the chat agent - understands intent, composes the verbs, and answers in plain language. An **intent router** - classifies one-shot (A) vs standing-concern (B) from the wording (one clarifying question if - ambiguous). Raw verbs live only on the REST/MCP dev doors. + understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot + (A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only + on the REST/MCP dev doors. - MVP surface = Google Maps actor + general web crawler, exposed chat + REST (MCP fast-follow). - Verbs namespaced per platform (`web.*`, `maps.*`); `web.scrape` takes a URL **array** (inline-or-job). - **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state. - CI is **decoupled from automations** (automations = one optional Trigger adapter). -- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single - entity per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash - pre-check before LLM spend. -- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`, working - name) on the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools; intent - routing lives in its prompt, the headless logic stays in ③/④. -- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, - not a new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and - automations' delivery. CI core still runs with **zero** automations dependency (manual/agent/cron). +- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity + per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check. +- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`) on + the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools. +- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, not a + new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and delivery. + CI core still runs with **zero** automations dependency (manual/agent/cron). - **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state. - **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first - provider; `maps.*` / `web.discover` register their own units there (pricing stays pluggable). -- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a - **separate effort**; `maps.*` verbs are contracts against it and don't block this design. -- **CI context files (F):** files uploaded in a CI chat go into the **KB as normal** (indexed), routed - to a dedicated `Folder`, and may feed the materiality judge via KB retrieval — the user's private - context shapes what counts as material. ("Don't index" applies only to *crawled* data.) + provider; `maps.*` / `web.discover` register their own units there. +- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a **separate + effort**; `maps.*` verbs are contracts against it and don't block this design. +- **CI context files:** files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a + dedicated `Folder`, and may feed the materiality judge via KB retrieval. ("Don't index" applies only to + *crawled* data.) +- **Names:** primitive = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent). - License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor. + +## Cross-phase open forks (need your call) + +1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? (`05b`) +2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. (`05a`) +3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). (`06`) + +*(Resolved: the "built-in scheduler tick" fork — reuse the automations schedule selector via a CI action +instead of building a tick. See `06`.)* + +> **Per-phase implementation-time questions** (refresh failure path, first-run baseline, tracker lifecycle, +> mid-run balance, concurrency lock, delivery payload shape) are intentionally **not** resolved here — they +> don't change any boundary, contract, or data shape, and are settled when each phase is built. + +## Subplan index (revamp) + +| Phase | Subplan file | Status | +|-------|--------------|--------| +| 04 | `04a-capabilities.md` | drafted | +| 04 | `04b-access.md` | drafted | +| 05 | `05a-timeline.md` | drafted | +| 05 | `05b-intelligence.md` | drafted | +| 06 | `06-triggers.md` | drafted | +| 07 | `07-orchestration.md` | drafted | From 93f7c591aeeb8d15b8b976fb0d5081a2f815cc25 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:13:45 +0200 Subject: [PATCH 20/21] docs(revamp): rename folder to 'revamp phases 4-7' --- plans/backend/{revamp => revamp phases 4-7}/00-overview.md | 0 plans/backend/{revamp => revamp phases 4-7}/04a-capabilities.md | 0 plans/backend/{revamp => revamp phases 4-7}/04b-access.md | 0 plans/backend/{revamp => revamp phases 4-7}/05a-timeline.md | 0 plans/backend/{revamp => revamp phases 4-7}/05b-intelligence.md | 0 plans/backend/{revamp => revamp phases 4-7}/06-triggers.md | 0 plans/backend/{revamp => revamp phases 4-7}/07-orchestration.md | 0 7 files changed, 0 insertions(+), 0 deletions(-) rename plans/backend/{revamp => revamp phases 4-7}/00-overview.md (100%) rename plans/backend/{revamp => revamp phases 4-7}/04a-capabilities.md (100%) rename plans/backend/{revamp => revamp phases 4-7}/04b-access.md (100%) rename plans/backend/{revamp => revamp phases 4-7}/05a-timeline.md (100%) rename plans/backend/{revamp => revamp phases 4-7}/05b-intelligence.md (100%) rename plans/backend/{revamp => revamp phases 4-7}/06-triggers.md (100%) rename plans/backend/{revamp => revamp phases 4-7}/07-orchestration.md (100%) diff --git a/plans/backend/revamp/00-overview.md b/plans/backend/revamp phases 4-7/00-overview.md similarity index 100% rename from plans/backend/revamp/00-overview.md rename to plans/backend/revamp phases 4-7/00-overview.md diff --git a/plans/backend/revamp/04a-capabilities.md b/plans/backend/revamp phases 4-7/04a-capabilities.md similarity index 100% rename from plans/backend/revamp/04a-capabilities.md rename to plans/backend/revamp phases 4-7/04a-capabilities.md diff --git a/plans/backend/revamp/04b-access.md b/plans/backend/revamp phases 4-7/04b-access.md similarity index 100% rename from plans/backend/revamp/04b-access.md rename to plans/backend/revamp phases 4-7/04b-access.md diff --git a/plans/backend/revamp/05a-timeline.md b/plans/backend/revamp phases 4-7/05a-timeline.md similarity index 100% rename from plans/backend/revamp/05a-timeline.md rename to plans/backend/revamp phases 4-7/05a-timeline.md diff --git a/plans/backend/revamp/05b-intelligence.md b/plans/backend/revamp phases 4-7/05b-intelligence.md similarity index 100% rename from plans/backend/revamp/05b-intelligence.md rename to plans/backend/revamp phases 4-7/05b-intelligence.md diff --git a/plans/backend/revamp/06-triggers.md b/plans/backend/revamp phases 4-7/06-triggers.md similarity index 100% rename from plans/backend/revamp/06-triggers.md rename to plans/backend/revamp phases 4-7/06-triggers.md diff --git a/plans/backend/revamp/07-orchestration.md b/plans/backend/revamp phases 4-7/07-orchestration.md similarity index 100% rename from plans/backend/revamp/07-orchestration.md rename to plans/backend/revamp phases 4-7/07-orchestration.md From 9f107ff52ce3a905bf2b6e74b76da6ef2beaadb3 Mon Sep 17 00:00:00 2001 From: CREDO23 Date: Wed, 1 Jul 2026 00:16:29 +0200 Subject: [PATCH 21/21] docs(revamp): add end-to-end pipeline diagrams --- .../backend/revamp phases 4-7/00-overview.md | 4 +- .../00b-pipeline-diagrams.md | 202 ++++++++++++++++++ 2 files changed, 205 insertions(+), 1 deletion(-) create mode 100644 plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md diff --git a/plans/backend/revamp phases 4-7/00-overview.md b/plans/backend/revamp phases 4-7/00-overview.md index 4cbfea102..8b6846c98 100644 --- a/plans/backend/revamp phases 4-7/00-overview.md +++ b/plans/backend/revamp phases 4-7/00-overview.md @@ -2,7 +2,8 @@ > **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small, > single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace. -> This file is the map + the reconciliation + the subplan index. +> This file is the map + the reconciliation + the subplan index. **End-to-end flow diagrams (stateless & +> stateful paths) live in `00b-pipeline-diagrams.md`.** > **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing). > The revamp is **Phase 4 → end only**. We do **not** touch 1–3. @@ -103,6 +104,7 @@ instead of building a tick. See `06`.)* | Phase | Subplan file | Status | |-------|--------------|--------| +| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams | | 04 | `04a-capabilities.md` | drafted | | 04 | `04b-access.md` | drafted | | 05 | `05a-timeline.md` | drafted | diff --git a/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md b/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md new file mode 100644 index 000000000..6e6e011e0 --- /dev/null +++ b/plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md @@ -0,0 +1,202 @@ +# Pipeline diagrams — end-to-end (stateless & stateful paths) + +> Visual companion to `00-overview.md`. Shows the whole CI pipeline across Phases 04–07, with the +> **stateless** (Product A) and **stateful** (Product B) paths called out explicitly. Phase refs: +> `04a` Capabilities · `04b` Access · `05a` Timeline · `05b` Intelligence · `06` Triggers · `07` Orchestration. + +## 1. Domain map — the two products + +```mermaid +flowchart LR + subgraph FIXED["FIXED · Phases 1-3 (shipped, untouched)"] + ACQ["Acquisition
proprietary/web_crawler
CrawlOutcome"] + MET["Metering · 03c
WebCrawlCreditService"] + ID["Identity / Tenancy
+ API keys"] + RT["Chat runtime
deepagents · deliverable_wait"] + end + + subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"] + CAP["04a Capabilities
typed verbs + registry + job store"] + ACC["04b Access
chat · REST · MCP doors"] + INT["05b Intelligence
Tracker · schema · hot loop"] + TL["05a Timeline
3-store delta state (moat)"] + TRG["06 Triggers
refresh clock"] + ORC["07 Orchestration
intelligence_agent subagent"] + end + + ACQ --> CAP + MET --> CAP + CAP --> ACC + ID --> ACC + RT --> ORC + ACC --> ORC + CAP --> INT + INT --> TL + TRG --> INT + ORC --> CAP + ORC --> INT + ORC --> TL + + classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7; + classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee; + class CAP,ACC,INT,TL,TRG,ORC a; + class ACQ,MET,ID,RT b; +``` + +- **Product A (stateless):** `04a Capabilities + 04b Access` — call a verb, get data, bill, nothing persists. +- **Product B (stateful):** `05b Intelligence + 05a Timeline` (driven by `06`, fronted by `07`) — a Tracker + accumulates structured signal; the Timeline is the moat. + +## 2. The fork — intent router (the only human-facing decision) + +```mermaid +flowchart TD + U(["User speaks in natural language"]) --> R{"Intent router
(in intelligence_agent prompt, 07)"} + R -->|"compare / find / what is / pull / right now"| A["ONE-SHOT
Product A · stateless"] + R -->|"watch / track / notify me / every week / over time"| B["STANDING CONCERN
Product B · stateful"] + R -->|ambiguous| Q["Ask ONE question:
'just once, or keep watching?'"] + Q -->|once| A + Q -->|keep watching| B + A --> P1["section 3 · Stateless path"] + B --> P2["section 4 · Stateful path"] +``` + +Raw verbs are exposed only on REST/MCP (devs/external agents). Humans never name a verb. + +## 3. Stateless path (Product A) — end to end + +```mermaid +sequenceDiagram + autonumber + participant U as User / Dev + participant D as Door 04b (chat · REST · MCP) + participant REG as Registry 04a + participant EX as Verb executor 04a + participant ACQ as Acquisition / Maps actor + participant BILL as Billing service (03c) + participant JOB as Job record (+Celery) + + U->>D: intent (chat) OR raw verb call (REST/MCP) + D->>D: validate input_schema · authn/authz · meter-gate + D->>REG: resolve verb + REG->>EX: same executor (all doors) + alt fast / small input -> inline + EX->>ACQ: crawl_url / search / place + ACQ-->>EX: data + EX->>BILL: charge billing_unit per success + EX-->>D: envelope {status: completed, results} + D-->>U: plain-language answer (chat) / JSON (REST) + else slow / large input -> job + EX->>JOB: dispatch, return {status: pending, job_id} + D-->>U: chat: deliverable_wait polls status
REST/MCP: GET /v1/jobs/{id} + JOB->>ACQ: work loop + JOB->>BILL: charge per success + JOB-->>D: status -> completed (+ results) + D-->>U: results inline / tracked card + end + Note over U,JOB: Nothing persists beyond chat history. No Tracker, no Timeline row. +``` + +## 4. Stateful path (Product B) — setup, then refresh, then read + +### 4a. Setup once — craft & lock the Tracker (05b) + +```mermaid +flowchart TD + S0(["Standing concern detected"]) --> S1["Bind capability + input
(maps.place X / web.scrape Y)"] + S1 --> S2["Sample fetch · ONE real verb call
(billed)"] + S2 --> S3["Agent drafts definition from decision + sample:
field_schema · materiality · identity_rule · notable_signals"] + S3 --> S4{"Human reviews in chat"} + S4 -->|add field X / tweak| S3 + S4 -->|looks good| S5["LOCK · versioned snapshot
status: draft -> active"] + S5 --> S6[("Tracker persisted
+ tracked_entities row (05a)")] +``` + +### 4b. The hot loop — refresh(tracker) (05b → writes 05a) + +```mermaid +flowchart TD + T(["Trigger fires refresh(tracker) · 06"]) --> C1["1 · Crawl: call bound capability (04a)"] + C1 --> C2{"2 · Content-hash
== stored hash?"} + C2 -->|identical| STOP["stamp last_checked_at · STOP
(zero LLM cost)"] + C2 -->|changed / first run| C3["3 · Fill: agent conforms raw -> locked field_schema
(extras -> notable_signals)"] + C3 --> C4["4 · Diff (code): new record vs entity_current_state -> raw deltas"] + C4 --> C5{"5 · Judge each delta"} + C5 -->|numeric / clear rule| C5a["code decides material/noise
(free, reproducible)"] + C5 -->|ambiguous / notable_signals| C5b["agent decides
(1 LLM call; may read CI context folder)"] + C5a --> C6{"material?"} + C5b --> C6 + C6 -->|yes| W["6 · Append entity_changes row
+ overwrite entity_current_state (05a)"] + C6 -->|no| N["only bump last_checked_at
(no change -> no row)"] + W --> DONE([done]) + N --> DONE + STOP --> DONE +``` + +### 4c. Read / answer — straight from the Timeline (05a) + +```mermaid +flowchart LR + Q(["'Is competitor X pulling ahead?'"]) --> A["intelligence_agent · query_timeline(tracker_id)"] + A --> TL[("entity_changes + entity_current_state")] + TL --> ANS["Answer from stored deltas
(NOT re-derived from chat history)"] +``` + +## 5. Triggers — who calls refresh(tracker) (06) + +```mermaid +flowchart LR + M["Manual
'refresh now' (chat/REST)"] --> RF["refresh(tracker)
headless unit of work · 05b"] + AG["Agent
calls refresh as a tool"] --> RF + XC["External cron
POST /v1/trackers/{id}/refresh"] --> RF + CA["CI automation ACTION
(existing automations)"] --> RF + RF --> TLW[("writes Timeline · 05a")] + + CA -. "schedule selector" .-> SEL["hardened cron selector"] + CA -. "run record + idempotency" .-> AR["AutomationRun (PENDING-gate)"] + CA -. "delivery" .-> DEL["alert / output to user"] + + classDef opt fill:#3a2f1f,stroke:#bf975b,color:#f7efe6; + class CA,SEL,AR,DEL opt; +``` + +CI **core** (`refresh` + Timeline) has **zero** automations dependency — manual/agent/cron all work +standalone. The CI **action** is the *optional* adapter that buys recurrence + idempotency + delivery. + +## 6. The persisted state — Timeline data model (05a) + +```mermaid +erDiagram + TRACKER ||--|| TRACKED_ENTITY : "MVP 1:1 (multi-ready)" + TRACKED_ENTITY ||--|| ENTITY_CURRENT_STATE : "latest" + TRACKED_ENTITY ||--o{ ENTITY_CHANGES : "append-only log" + + TRACKER { + uuid id + text decision + json capability_binding + json definition_locked "field_schema·identity_rule·materiality (versioned)" + text status "draft|active" + } + TRACKED_ENTITY { + uuid id + uuid workspace_id + text entity_key "from identity_rule, unique per tracker" + ts first_seen_at + } + ENTITY_CURRENT_STATE { + uuid entity_id FK + json fields "conforms to field_schema" + text content_hash "powers step-2 pre-check" + ts last_checked_at + } + ENTITY_CHANGES { + uuid id + ts captured_at + json delta "{field:{from,to}}" + text materiality "material|notable" + text decided_by "code|agent (provenance)" + text source_ref + text note "why material" + } +```