apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #1564 from CREDO23/revamp-phases4-7

Browse source 
[Docs] Re-architect Phases 4–7 (CI pipeline revamp)
This commit is contained in:
Rohan Verma 2026-06-30 16:42:22 -07:00 • committed by GitHub
commit 78da036c6b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 1167 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

113
plans/backend/revamp phases 4-7/00-overview.md Normal file
View file

@ -0,0 +1,113 @@
# Phase 4 → end revamp — Overview & phase index (CI pivot · WIP)
> **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small,
> single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace.
> This file is the map + the reconciliation + the subplan index. **End-to-end flow diagrams (stateless &
> stateful paths) live in `00b-pipeline-diagrams.md`.**
> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing).
> The revamp is **Phase 4 → end only**. We do **not** touch 1–3.
## The two products
```
PRODUCT A — stateless utility Phase 04 (Capabilities + Access)
call a verb → get data → bill. Nothing persists.
PRODUCT B — decision-grounded CI Phase 05 (Intelligence + Timeline) (+ 06 Triggers, 07 Orchestration)
a Tracker accumulates structured signal over time. The Timeline is the moat.
```
## The phases (this revamp)
| Phase | Subplan(s) | Domain(s) | One line | Ships |
|-------|-----------|-----------|----------|-------|
| **04 — Capabilities & Access** | `04a-capabilities.md` · `04b-access.md` | ① + ② | typed verbs over Acquisition + the chat/REST/MCP doors | **Product A** (revenue day one) |
| **05 — Intelligence & Timeline** | `05a-timeline.md` · `05b-intelligence.md` | ④ + ③ | the Tracker, locked schema, hot loop + the 3-store delta moat | **Product B engine** |
| **06 — Triggers** | `06-triggers.md` | ⑤ | the pluggable refresh clock; recurrence+delivery = optional CI action on automations | recurrence + alerts |
| **07 — Orchestration** | `07-orchestration.md` | ⑥ | the `intelligence_agent` CI-expert subagent (intent routing, verb chains, Tracker crafting) | the human-facing CI experience |
> **Build order = phase order.** `04a → 04b` (unblocks everything; ships Product A) → `05a → 05b`
> (state before the loop that writes it) → `06` (drive it) → `07` (front it). `04a` first because every
> other phase calls into the capability registry.
## Domain ↔ phase map
```
FIXED (Phases 1–3) OUR SCOPE (Phase 04 → 07)
┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐
│ Acquisition │ │ 04a Capabilities typed verbs over Acquisition │
│ proprietary/web_crawler │◄──┤ 04b Access chat · REST · MCP doors │
│ CrawlOutcome · billing (03c) │ │ 05a Timeline 3-store delta state (moat) │
│ │ │ 05b Intelligence Tracker · schema · hot loop │
│ │ │ 06 Triggers pluggable refresh clock │
│ │ │ 07 Orchestration CI-expert subagent + tools │
└─────────────────────────────────┘ └──────────────────────────────────────────────┘
data engine (untouched) stateless 04 → stateful 05 , driven by 06 , fronted by 07
```
## Reconciliation with the old plans
| Old plan | Fate | Where it goes |
|----------|------|---------------|
| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in `04b` (consume-user-MCP); the taxonomy work is not core |
| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in `04a` |
| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (`05b`) + Timeline tables (`05a`); no `pipeline`/`pipeline_runs` |
| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (`05b`); scheduling/runs/delivery → **reuse automations** via a CI action (`06`) — its selector + `AutomationRun`, **not** a rebuilt cron |
| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploads are routed to a dedicated **folder** and may feed the judge (`05b`) |
**Net:** the old `05/06/07` pipeline+KB stack is replaced by the new `04`–`07`. KB indexing of *crawled*
data is out; *user uploads* still work (and gain a CI context-folder role).
## Confirmed decisions (locked across phases)
- **Natural language is the only human-facing surface.** Users never name verbs; the chat agent
understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot
(A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only
on the REST/MCP dev doors.
- MVP surface = Google Maps actor + general web crawler, exposed chat + REST (MCP fast-follow).
- Verbs namespaced per platform (`web.*`, `maps.*`); `web.scrape` takes a URL **array** (inline-or-job).
- **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state.
- CI is **decoupled from automations** (automations = one optional Trigger adapter).
- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity
per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check.
- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`) on
the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools.
- **Recurrence + alert delivery = a CI *action* on the existing automations** (not a new scheduler, not a
new shape): reuses the hardened schedule selector, `AutomationRun` (audit + idempotency), and delivery.
CI core still runs with **zero** automations dependency (manual/agent/cron).
- **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing
idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state.
- **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first
provider; `maps.*` / `web.discover` register their own units there.
- The **Google Maps actor is net-new** (not in shipped Phases 1–3) and is designed/built as a **separate
effort**; `maps.*` verbs are contracts against it and don't block this design.
- **CI context files:** files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a
dedicated `Folder`, and may feed the materiality judge via KB retrieval. ("Don't index" applies only to
*crawled* data.)
- **Names:** primitive = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent).
- License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor.
## Cross-phase open forks (need your call)
1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? (`05b`)
2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. (`05a`)
3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). (`06`)
*(Resolved: the "built-in scheduler tick" fork — reuse the automations schedule selector via a CI action
instead of building a tick. See `06`.)*
> **Per-phase implementation-time questions** (refresh failure path, first-run baseline, tracker lifecycle,
> mid-run balance, concurrency lock, delivery payload shape) are intentionally **not** resolved here — they
> don't change any boundary, contract, or data shape, and are settled when each phase is built.
## Subplan index (revamp)
| Phase | Subplan file | Status |
|-------|--------------|--------|
| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams |
| 04 | `04a-capabilities.md` | drafted |
| 04 | `04b-access.md` | drafted |
| 05 | `05a-timeline.md` | drafted |
| 05 | `05b-intelligence.md` | drafted |
| 06 | `06-triggers.md` | drafted |
| 07 | `07-orchestration.md` | drafted |

202
plans/backend/revamp phases 4-7/00b-pipeline-diagrams.md Normal file
View file

@ -0,0 +1,202 @@
# Pipeline diagrams — end-to-end (stateless & stateful paths)
> Visual companion to `00-overview.md`. Shows the whole CI pipeline across Phases 04–07, with the
> **stateless** (Product A) and **stateful** (Product B) paths called out explicitly. Phase refs:
> `04a` Capabilities · `04b` Access · `05a` Timeline · `05b` Intelligence · `06` Triggers · `07` Orchestration.
## 1. Domain map — the two products
```mermaid
flowchart LR
subgraph FIXED["FIXED · Phases 1-3 (shipped, untouched)"]
ACQ["Acquisition<br/>proprietary/web_crawler<br/>CrawlOutcome"]
MET["Metering · 03c<br/>WebCrawlCreditService"]
ID["Identity / Tenancy<br/>+ API keys"]
RT["Chat runtime<br/>deepagents · deliverable_wait"]
end
subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"]
CAP["04a Capabilities<br/>typed verbs + registry + job store"]
ACC["04b Access<br/>chat · REST · MCP doors"]
INT["05b Intelligence<br/>Tracker · schema · hot loop"]
TL["05a Timeline<br/>3-store delta state (moat)"]
TRG["06 Triggers<br/>refresh clock"]
ORC["07 Orchestration<br/>intelligence_agent subagent"]
end
ACQ --> CAP
MET --> CAP
CAP --> ACC
ID --> ACC
RT --> ORC
ACC --> ORC
CAP --> INT
INT --> TL
TRG --> INT
ORC --> CAP
ORC --> INT
ORC --> TL
classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
class CAP,ACC,INT,TL,TRG,ORC a;
class ACQ,MET,ID,RT b;
```
- **Product A (stateless):** `04a Capabilities + 04b Access` — call a verb, get data, bill, nothing persists.
- **Product B (stateful):** `05b Intelligence + 05a Timeline` (driven by `06`, fronted by `07`) — a Tracker
accumulates structured signal; the Timeline is the moat.
## 2. The fork — intent router (the only human-facing decision)
```mermaid
flowchart TD
U(["User speaks in natural language"]) --> R{"Intent router<br/>(in intelligence_agent prompt, 07)"}
R -->|"compare / find / what is / pull / right now"| A["ONE-SHOT<br/>Product A · stateless"]
R -->|"watch / track / notify me / every week / over time"| B["STANDING CONCERN<br/>Product B · stateful"]
R -->|ambiguous| Q["Ask ONE question:<br/>'just once, or keep watching?'"]
Q -->|once| A
Q -->|keep watching| B
A --> P1["section 3 · Stateless path"]
B --> P2["section 4 · Stateful path"]
```
Raw verbs are exposed only on REST/MCP (devs/external agents). Humans never name a verb.
## 3. Stateless path (Product A) — end to end
```mermaid
sequenceDiagram
autonumber
participant U as User / Dev
participant D as Door 04b (chat · REST · MCP)
participant REG as Registry 04a
participant EX as Verb executor 04a
participant ACQ as Acquisition / Maps actor
participant BILL as Billing service (03c)
participant JOB as Job record (+Celery)
U->>D: intent (chat) OR raw verb call (REST/MCP)
D->>D: validate input_schema · authn/authz · meter-gate
D->>REG: resolve verb
REG->>EX: same executor (all doors)
alt fast / small input -> inline
EX->>ACQ: crawl_url / search / place
ACQ-->>EX: data
EX->>BILL: charge billing_unit per success
EX-->>D: envelope {status: completed, results}
D-->>U: plain-language answer (chat) / JSON (REST)
else slow / large input -> job
EX->>JOB: dispatch, return {status: pending, job_id}
D-->>U: chat: deliverable_wait polls status<br/>REST/MCP: GET /v1/jobs/{id}
JOB->>ACQ: work loop
JOB->>BILL: charge per success
JOB-->>D: status -> completed (+ results)
D-->>U: results inline / tracked card
end
Note over U,JOB: Nothing persists beyond chat history. No Tracker, no Timeline row.
```
## 4. Stateful path (Product B) — setup, then refresh, then read
### 4a. Setup once — craft & lock the Tracker (05b)
```mermaid
flowchart TD
S0(["Standing concern detected"]) --> S1["Bind capability + input<br/>(maps.place X / web.scrape Y)"]
S1 --> S2["Sample fetch · ONE real verb call<br/>(billed)"]
S2 --> S3["Agent drafts definition from decision + sample:<br/>field_schema · materiality · identity_rule · notable_signals"]
S3 --> S4{"Human reviews in chat"}
S4 -->|add field X / tweak| S3
S4 -->|looks good| S5["LOCK · versioned snapshot<br/>status: draft -> active"]
S5 --> S6[("Tracker persisted<br/>+ tracked_entities row (05a)")]
```
### 4b. The hot loop — refresh(tracker) (05b → writes 05a)
```mermaid
flowchart TD
T(["Trigger fires refresh(tracker) · 06"]) --> C1["1 · Crawl: call bound capability (04a)"]
C1 --> C2{"2 · Content-hash<br/>== stored hash?"}
C2 -->|identical| STOP["stamp last_checked_at · STOP<br/>(zero LLM cost)"]
C2 -->|changed / first run| C3["3 · Fill: agent conforms raw -> locked field_schema<br/>(extras -> notable_signals)"]
C3 --> C4["4 · Diff (code): new record vs entity_current_state -> raw deltas"]
C4 --> C5{"5 · Judge each delta"}
C5 -->|numeric / clear rule| C5a["code decides material/noise<br/>(free, reproducible)"]
C5 -->|ambiguous / notable_signals| C5b["agent decides<br/>(1 LLM call; may read CI context folder)"]
C5a --> C6{"material?"}
C5b --> C6
C6 -->|yes| W["6 · Append entity_changes row<br/>+ overwrite entity_current_state (05a)"]
C6 -->|no| N["only bump last_checked_at<br/>(no change -> no row)"]
W --> DONE([done])
N --> DONE
STOP --> DONE
```
### 4c. Read / answer — straight from the Timeline (05a)
```mermaid
flowchart LR
Q(["'Is competitor X pulling ahead?'"]) --> A["intelligence_agent · query_timeline(tracker_id)"]
A --> TL[("entity_changes + entity_current_state")]
TL --> ANS["Answer from stored deltas<br/>(NOT re-derived from chat history)"]
```
## 5. Triggers — who calls refresh(tracker) (06)
```mermaid
flowchart LR
M["Manual<br/>'refresh now' (chat/REST)"] --> RF["refresh(tracker)<br/>headless unit of work · 05b"]
AG["Agent<br/>calls refresh as a tool"] --> RF
XC["External cron<br/>POST /v1/trackers/{id}/refresh"] --> RF
CA["CI automation ACTION<br/>(existing automations)"] --> RF
RF --> TLW[("writes Timeline · 05a")]
CA -. "schedule selector" .-> SEL["hardened cron selector"]
CA -. "run record + idempotency" .-> AR["AutomationRun (PENDING-gate)"]
CA -. "delivery" .-> DEL["alert / output to user"]
classDef opt fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
class CA,SEL,AR,DEL opt;
```
CI **core** (`refresh` + Timeline) has **zero** automations dependency — manual/agent/cron all work
standalone. The CI **action** is the *optional* adapter that buys recurrence + idempotency + delivery.
## 6. The persisted state — Timeline data model (05a)
```mermaid
erDiagram
TRACKER ||--|| TRACKED_ENTITY : "MVP 1:1 (multi-ready)"
TRACKED_ENTITY ||--|| ENTITY_CURRENT_STATE : "latest"
TRACKED_ENTITY ||--o{ ENTITY_CHANGES : "append-only log"
TRACKER {
uuid id
text decision
json capability_binding
json definition_locked "field_schema·identity_rule·materiality (versioned)"
text status "draft|active"
}
TRACKED_ENTITY {
uuid id
uuid workspace_id
text entity_key "from identity_rule, unique per tracker"
ts first_seen_at
}
ENTITY_CURRENT_STATE {
uuid entity_id FK
json fields "conforms to field_schema"
text content_hash "powers step-2 pre-check"
ts last_checked_at
}
ENTITY_CHANGES {
uuid id
ts captured_at
json delta "{field:{from,to}}"
text materiality "material|notable"
text decided_by "code|agent (provenance)"
text source_ref
text note "why material"
}
```

177
plans/backend/revamp phases 4-7/04a-capabilities.md Normal file
View file

@ -0,0 +1,177 @@
# Phase 4a — Capabilities (typed verbs over Acquisition)
> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7).
> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above
> calls into it.
> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome`
> contract, the proxy provider, and the `WebCrawlCreditService` billing seam.
> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies
> them. Locate foundation code by **symbol/grep**, not the line numbers cited.
## Objective
Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable
verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume
**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the
KB" with "capability that returns data."** A capability is a stateless function you *call and get data
back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached.
## Current state (cited)
- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content,
metadata, crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS`
(`app/proprietary/web_crawler/`).
- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` /
`successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`).
- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`).
- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the
substrate for `web.discover`.
- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope).
## Target design
### The verb set (MVP) — namespaced, nothing top-level
Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor).
Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …).
| Verb | Input → Output | Mode | Executes over | Bills (03c) |
|------|----------------|------|---------------|-------------|
| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success |
| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* |
| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place |
| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 |
| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page |
### `web.scrape` — one array-friendly verb (no separate "batch")
Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model:
"give me URLs, I give you per-URL content."
### Execution mode — a property of the **result**, not two verbs
A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**:
```
{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... }
```
- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline →
zero polling. (single page, one place, a search)
- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until
`completed`. (batch scrape, Maps search/reviews)
So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller.
- **Threshold** (how many URLs / how heavy before async) is **configurable**.
- A caller may pass `async: true` to **force** a job even for small input (agents that never block).
- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the
**existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it.
### The capability registry — single source of truth
One registry entry per verb:
```
Capability {
name # dotted, e.g. "web.scrape", "maps.search"
input_schema # Pydantic
output_schema # Pydantic
mode # can-complete-inline? + job-capable?
executor # the async fn (wraps Acquisition / Maps actor)
billing_unit # how Metering charges this call
}
```
The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool,
REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly.
Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces.
### Billing — open to the billing service (not hardcoded per verb)
A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the
**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite.
`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service.
- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit).
- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet).
- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged).
- `web.discover` → register a per-search unit (or mark free) **with the billing service**.
The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs
stay pure; pricing stays pluggable.
### The Maps actor (clarification)
- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …).
- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw
markdown — exactly what the Intelligence/Timeline layer needs to diff reliably.
- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on
the capability. The capability never knows what decision it serves.
### Where it lives / license boundary
- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store).
- It **imports from** the proprietary Acquisition engine but never moves into it.
- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*`
executors — same boundary rule Phase 3 set.
## Work items
1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import.
2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL
result shape; declare the `web_crawl` `billing_unit`.
3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed);
declare its `billing_unit` (or free).
4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs.
5. **Uniform envelope**: shared `completed|pending` result type returned by every executor.
6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor.
7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code).
## Tests
- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns
`pending` + `job_id`; the job completes to `results`.
- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail
the batch.
- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED`
free; disabling the billing flag makes it a no-op.
- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb.
- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set.
## Risks / trade-offs
- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when
the actor lands (doesn't block `web.*`).
- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never
grows back into `pipeline_runs`.
- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact
mid-run behavior is an implementation-time call.
## Resolved decisions
1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level.
2. One array-friendly `web.scrape` (no separate batch verb).
3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs.
4. Build the thin job model now (Maps search / batch scrape force it).
5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`).
6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service**
(verbs declare a `billing_unit`; `03c` is the first provider, new units register there).
7. Connector → capability is a **replacement** for these data sources, not an extension.
## Out of scope (hand-offs)
- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`.
- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless.
- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a
**separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain
doesn't block on it.
- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces
(`linkedin.*`, `amazon.*`) — deferred.
- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing
fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`.
## Open questions (carry forward)
- Default async threshold for `web.scrape`.
- Whether `web.discover` is metered or free.

136
plans/backend/revamp phases 4-7/04b-access.md Normal file
View file

@ -0,0 +1,136 @@
# Phase 4b — Access / Surfaces (chat · REST · MCP doors)
> Part of **Phase 4 — Capabilities & Access**. Sits on top of `04a-capabilities.md` (the registry it exposes).
> **Build after** `04a`. Together, `04a + 04b` ship **Product A** (stateless utility) — revenue day one.
> **Depends on** SHIPPED infra: Identity/Tenancy, **API keys**, the chat agent + its tool registry, the
> streaming layer, and Metering (`03c`). Access *reuses* them — it builds none net-new.
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. Locate code by **symbol/grep**, not the cited lines.
## Objective
Expose the capability registry (`04a`) to callers, **authenticated + metered**, through three doors —
**chat tools, REST + API keys, MCP server** — all **generated from the one registry** so the I/O
contract cannot drift between surfaces. Access contains **no business logic**: every door is the same
thin adapter.
## Current state (cited)
- **API keys** — existing per-workspace key infra (reuse; do **not** build net-new); billed to the
workspace owner via `03c`.
- **Chat agent + tool registry** — capability tools register *into* the existing registry; the seed is
`research/tools/scrape_webpage.py` (capability executor + access door + `03c` turn-accumulator billing).
- **Slow-job pattern** — `subagents/builtins/deliverables/deliverable_wait.py`: dispatch a Celery task,
poll the row's `status` until `READY`/`FAILED` (1.5s cadence), bounded by
`SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s); `deliverables/tools/podcast.py` is the
"return now + live card tracks progress" model.
- **BYO-`MCP_CONNECTOR`** (old `04a`) — the user's own external MCP tools inside our chat agent; its
routing-map gap is the one "connector" worth fixing, and it lands **here**.
## Target design
### The adapter shape (identical on every door)
```
parse input → validate against verb.input_schema → authn/authz → meter-gate (03c)
→ call the SAME executor (04a) → serialize verb.output_schema → return the uniform envelope
```
### The three doors (locked order: chat → REST → MCP)
| Door | Who | Auth | Status |
|------|-----|------|--------|
| **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) |
| **REST + API keys** | external developers (Product A) | **existing API-key infra** | **public day one** |
| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — chosen at implementation | **fast-follow** |
REST being public day one is cheap precisely because the routes are **generated**, not hand-written —
it's a go-to-market choice, not an engineering cost.
### Natural language is THE surface (verbs are internal) — non-negotiable
The human-facing product is **the conversation**. A user **never** names a verb, fills an `input_schema`,
or knows "Product A vs B" exists — they describe a *need* in plain language and the agent does the rest.
Raw typed verbs are exposed only on the REST/MCP doors (devs/external agents). The chat agent owns three
responsibilities per message: **understand intent** → **pick & fill verbs** (incl. chains like
`discover → scrape`, `search → place → reviews`) → **answer in plain language** (results, not envelopes).
### The intent router (the one new orchestration rule)
The agent classifies each request along the stateless/stateful line **from the language**:
```
"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless)
"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start Tracker setup (Product B, stateful → 05b)
ambiguous → ask ONE question: "just this once, or should I keep watching it for you?"
```
This router is the friendly seam between the two products; it lives in the chat door (its prompt lives
in the `intelligence_agent`, `07`) and is the only human-facing decision point.
### The two MCP directions (keep distinct)
- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new): Streamable-HTTP `/mcp`,
stateless, bounded/paginated outputs, untrusted inputs, least-privilege (read-only verbs).
- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` (old 04a): the user's own external MCP tools inside our
chat agent. The 04a routing-gap fix lands **here**, not in Capabilities.
### Chat ↔ slow jobs — reuse the existing background-worker pattern
Do **not** invent a chat-async mechanism. A slow verb (`web.scrape` over many URLs, `maps.search`,
`maps.reviews`) invoked from chat dispatches the **job** (the `04a` job record) and uses the
`deliverable_wait` poll-until-terminal path; the capability **job record's `status`** is what the helper
polls. Most calls finish inside the poll window → results inline; genuinely long ones surface a tracked
card. REST/MCP expose the same job via `GET /v1/jobs/{id}` (and an MCP equivalent).
## Work items
1. **Door generator**: from the `04a` registry, emit (a) chat tool defs + handlers, (b) REST routes +
request/response models, (c) MCP tool schemas + handlers — one adapter shape.
2. **REST surface**: public routes + API-key auth (reuse existing) + the `03c` meter-gate; `GET /v1/jobs/{id}`.
3. **Chat tools**: generalize `scrape_webpage` into the registry-backed set; wire slow verbs through
`deliverable_wait` polling the `04a` job record's `status`.
4. **Intent router**: the A-vs-B classifier (its home is the `07` subagent prompt; the seam is here).
5. **MCP server (fast-follow)**: Streamable-HTTP `/mcp`, least-privilege; auth depth chosen at implementation.
6. **Consume-MCP fix**: repair the BYO-`MCP_CONNECTOR` routing-map gap (old 04a).
## Tests
- **Generated parity**: a verb added to the registry appears on chat + REST (+ MCP) with identical I/O.
- **Auth + meter**: REST without a valid key → 401; an over-budget call → blocked by the `03c` gate
before execute; a success charges once.
- **Chat slow verb**: a many-URL `web.scrape` returns inline when fast; surfaces a tracked card when long;
`GET /v1/jobs/{id}` reflects the same terminal status.
- **Intent router**: "compare X and Y" → one-shot; "watch X weekly" → Tracker setup handoff; ambiguous →
exactly one clarifying question.
- **Consume-MCP**: a configured BYO MCP tool is reachable by the chat agent (routing-gap closed).
## Risks / trade-offs
- **Public REST day one** → needs bounded inputs + per-key quotas / abuse posture (design alongside launch).
- **MCP auth depth** deferred to implementation (OAuth 2.1 vs bearer) — don't over-build before a consumer exists.
- **Serve-vs-consume MCP confusion** — keep the two directions explicitly separate in code and docs.
## Resolved decisions
0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the chat
agent understands intent, picks & fills verbs, answers in plain language; an intent router classifies
one-shot (A) vs standing-concern (B), asking one question only when ambiguous. Raw verbs live only on REST/MCP.
1. Three doors, generated from the capability registry; order chat → REST → MCP.
2. REST is **public day one** (cheap; go-to-market choice).
3. API keys: **reuse existing infra**, billed to workspace owner.
4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation.
5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the `04a` job record's `status`.
6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the old-04a fix is the consume side.
## Out of scope (hand-offs)
- **The verbs themselves** (executors, registry, billing units) → `04a`.
- **The CI subagent + its prompt/playbook** (where the intent router actually lives) → `07`.
- **Stateful flows** (Tracker crafting, refresh, timeline reads) → `05b`/`05a`, surfaced via chat tools in `07`.
- Legacy branded connectors stay only for backward-compat (separate hygiene task).
## Open questions (carry forward)
- MCP auth depth (decide at implementation).
- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas).
- Whether `web.discover` is metered or free (carried from `04a`).

125
plans/backend/revamp phases 4-7/05a-timeline.md Normal file
View file

@ -0,0 +1,125 @@
# Phase 5a — Timeline (the moat asset)
> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes
> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them.
> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline).
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not**
> the Knowledge Base (documents/embeddings) and **not** `automation_runs`.
## Objective
Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** —
accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no
change, no row.** Storage grows with the *rate of change*, not the number of runs.
## Current state (cited)
- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is
what this phase introduces.
- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero
publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog
(this is the **fact** analog, deliberately separate).
- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document(
exclude_metadata=True)` to compute the `content_hash` stored here.
## Target design
### The state / process split
Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain
today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without
running the loop**. That separation is why Timeline is its own domain.
### The three stores
| Store | Role | Write pattern |
|-------|------|---------------|
| `tracked_entities` | stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`) | written **once** |
| `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run |
| `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten |
**The timeline = the change log read in order.** To reconstruct a past point: take Current state and
replay deltas backward (north-star tooling; MVP just *stores* the deltas).
### Data model sketch (new tables)
```
tracked_entities
id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at
# MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready.
entity_current_state
entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema)
· content_hash · last_checked_at · updated_at
# overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2)
entity_changes # the append-only change log
id · entity_id (FK) · tracker_id · captured_at
· delta JSONB # { field: { from, to } }
· materiality # material | notable(=notable_signals-sourced)
· decided_by # code | agent (audit of the materiality split)
· source_ref # url / blob key the change was observed from
· note TEXT NULL # optional agent rationale (the "why material")
```
- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`.
- `decided_by` records whether code or the agent ruled the change material (provenance seed).
### What it is NOT
- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.)
- **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store.
- **Not a resurrected `pipeline_runs`.**
### Where it lives
- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide
at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor).
- Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend).
## Work items
1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration.
2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop.
3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side.
4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit.
5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred).
## Tests
- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged.
- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`.
- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls.
- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row.
- **Read API**: `query_timeline` returns deltas in `captured_at` order.
## Risks / trade-offs
- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against —
baseline semantics (silent establish vs flood the log) are an implementation-time call.
- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very
chatty entities is deferred.
- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity
guard (lean: yes) but adds a write-path dependency on `05b`'s lock.
## Resolved decisions
1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`.
2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change.
3. CI-owned new tables; **not** KB, **not** `automation_runs`.
4. `content_hash` on current state powers the hot-loop cheap pre-check.
5. `decided_by` on changes records the code-vs-agent materiality provenance.
6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later).
## Out of scope (hand-offs)
- **The writer** (hot loop, materiality, schema lock) → `05b`.
- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the
resale/data-product surface → north star (deferred).
- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella.
## Open questions (carry forward)
- ORM home: `app/db.py` vs a dedicated `app/timeline/` package.
- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes).
- Retention / archival policy for very high-velocity entities (deferred).

180
plans/backend/revamp phases 4-7/05b-intelligence.md Normal file
View file

@ -0,0 +1,180 @@
# Phase 5b — Intelligence (the decision-grounded engine)
> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes).
> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the
> **Product B engine** (the Tracker, locked schema, hot loop, deltas).
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the
> automations subsystem.
> **Name:** the standing-concern primitive is the **Tracker**.
## Objective
Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent
judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the
**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the
Access→Intelligence boundary stays pure functions.
```
STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists
STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state
```
## Current state (cited)
- **Capability executors** (`04a`) — called directly by the loop (not through a door).
- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)`
produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`).
- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`)
+ the automations executor's PENDING→running idempotency gate (safe under Celery `acks_late`); and the
chat **job record** (`04a`) + `deliverable_wait`. **No new run table.**
- **CI context uploads** — the existing folder-upload / `destination_folder` machinery + KB retrieval.
## Target design
### The primitive — `Tracker`
A saved, decision-grounded subject that accumulates structured signal over time:
| Field | Meaning |
|-------|---------|
| `decision` | the question being tracked toward ("is this competitor pulling ahead?") |
| `capability_binding` | which verb + input feeds it (`maps.place(X)`, `web.scrape([Y])`) |
| `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract |
| `status` | `draft` → `locked`/`active` |
One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred
(the `05a` model stays multi-entity-ready so it's additive).
### Setup (once) — the agent-designed schema flow (IN MVP)
The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is
derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first):
1. **Bind** a capability + input.
2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data.
3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types),
`materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable
entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field.
4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run.
5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how
`automations` snapshots `definition`).
### The hot loop (per refresh) — `refresh(tracker)`
1. **Crawl** — call the bound capability (`04a`) → raw data.
2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`,
**stop** (no LLM cost).
3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does
**not** invent fields. Unanticipated observations go into `notable_signals`.
4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas.
5. **Judge — the materiality split:**
- **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100%
reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`).
- **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new
`notable_signals` entry → one LLM call rules material/noise.
6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`.
**No change → no row.**
**Worked example (`maps.place` refresh):**
```
rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM)
review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM)
hours unchanged → no delta
description reworded → code: no rule → ASK AGENT → NOISE
⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick.
```
### Refresh execution & idempotency — ride the invoking surface (no new run table)
`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever
surface invoked it**:
- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`**
is the run record and the automations executor provides the **PENDING→running idempotency gate**. This is
exactly the rigor old `06` hand-built — reused, not re-written.
- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`.
- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via
the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No
run-level `charged_micros` ledger required for MVP.
Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed.
### User-supplied context files (the F idea, generalized)
When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** —
uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies
only to *crawled* data.)** The CI-specific part is **organization + use**:
- **Routed to a dedicated folder** for that CI chat/Tracker (reuse the existing folder-upload machinery).
- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder:
```
competitor price 12.00 → 9.90 + user's context file says "our price is 10.00"
→ agent: competitor crossed *below our price* → MATERIAL (and explain why)
```
The user's private context **shapes what counts as material**. Reuses existing KB upload + folder +
retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now.
### Where it lives / decoupling
- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls
capability `executor`s directly.
- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is
the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler.
## Work items
1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`.
2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version.
3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback.
4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`).
5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table.
6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder.
## Tests
- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped.
- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields.
- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword →
`decided_by=agent`; sub-threshold → noise, no row.
- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state.
- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill.
- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame.
## Risks / trade-offs
- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an
implementation-time call (no architecture impact).
- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM.
- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version.
## Resolved decisions
1. `Tracker` is the standing-concern primitive; replaces "pipeline".
2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary.
3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned.
4. Single entity per Tracker for MVP.
5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous.
6. Content-hash pre-check short-circuits unchanged pages before any LLM spend.
7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic.
8. **No new run table** — refresh audit/idempotency ride `AutomationRun` (recurring) or the chat job record;
billing idempotency is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state.
9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed),
routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled*
data only. MVP-optional seam.
## Out of scope (hand-offs)
- **The state tables** (the three stores, content-hash, read API) → `05a`.
- **When refresh fires + recurrence/delivery** → `06` (Triggers).
- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`.
- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the
resale/data-product stage → north star (deferred).
## Open questions (carry forward)
- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?).
- Versioning policy on re-lock (new version vs in-place) — lean new version.
- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?).
- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope).

111
plans/backend/revamp phases 4-7/06-triggers.md Normal file
View file

@ -0,0 +1,111 @@
# Phase 6 — Triggers (the pluggable refresh clock)
> **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes
> `refresh(tracker)`).
> **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED
> automations subsystem (its schedule selector + `AutomationRun` + delivery).
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no
> dependency on it, and **automations is at most one optional adapter, never required**.
## Objective
Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** —
and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and
the engine still works. This replaces the old Phase-6 cron scheduler — and the resolution is **not** to
rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path.
## Current state (cited)
- **`refresh(tracker)`** — the headless unit of work exposed by `05b`.
- **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED`
claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the
`croniter` util.
- **`AutomationRun`** — the existing run record + PENDING-gate (audit + idempotency).
- **Automations output/delivery** — the existing path that carries results to the user.
- **Access routes** (`04b`) — where the external-cron and manual REST endpoints live.
## Target design
### The adapters
| Adapter | Fired by | MVP? |
|---------|----------|------|
| **Manual** | user "refresh now" (chat tool / REST) | ✅ |
| **Agent** | the in-app agent calls `refresh` as a tool | ✅ |
| **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) |
| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)` **and delivers** the material changes | ✅ (the in-app recurrence + alert path) |
### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape)
The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes."** Instead of
building a bespoke tick, we **add a CI *action* to the existing automations subsystem**:
- **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new
scheduler. **(closes old Gap B — scheduler rigor.)**
- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run
table. **(closes old Gap A — run/idempotency, see `05b`.)**
- **Delivery / alert** → the automation's existing **output/delivery** carries the material changes to the
user. **(closes old Gap E — alert delivery.)**
**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery
that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action
too constraining, a thin CI-specific shape is the fallback — but the action is the default.)
### Decoupling is preserved (automations is still optional)
CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs
via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + delivery
+ audit for in-app users. So we honor "don't glue CI to automations" *and* get its machinery for free.
### Where it lives
- The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from
`app/intelligence/`. No new scheduler/Beat task.
- The **external-cron** and **REST manual** paths are just Access-door routes (`POST
/v1/trackers/{id}/refresh`) — `04b` plumbing.
## Work items
1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`.
2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`.
3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that
calls `refresh(tracker)` and routes the resulting material changes into automations delivery.
4. **Concurrency guard**: a per-Tracker in-flight lock (belt-and-suspenders over the automation run-gate).
## Tests
- **Decoupling**: `refresh(tracker)` works via manual/agent/external-cron with automations disabled entirely.
- **Recurring path**: a scheduled CI action fires `refresh` on cron and delivers material changes; an
unchanged refresh delivers nothing.
- **Idempotency**: a redelivered scheduled run (same `AutomationRun`) does not double-refresh.
- **External cron**: `POST /v1/trackers/{id}/refresh` triggers exactly one refresh; rejects bad auth.
- **Concurrency**: a second refresh while one is in flight is skipped/queued, not run concurrently.
## Risks / trade-offs
- **Action vs shape**: the CI action is the default; a thin CI automation shape is the fallback if the
action proves too constraining.
- **Delivered payload shape** (agent-summarized vs raw deltas since last fire) — implementation-time call.
- **Double-guarding concurrency**: the per-Tracker lock overlaps the automation run-gate, but the lock also
protects the manual/cron paths that don't go through automations.
## Resolved decisions
1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled.
2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery).
3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule
selector + `AutomationRun`; delivery reuses automations' output. (Closes old Gaps A/B/E.)
4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**.
5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron).
## Out of scope (hand-offs)
- **`refresh(tracker)` internals** (the hot loop, materiality) → `05b`.
- **The Timeline reads** the delivery summarizes → `05a`.
- **The chat tool surface** for manual/agent refresh → `07` (the `intelligence_agent` toolset).
## Open questions (carry forward)
- CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback).
- What the delivered payload looks like (summarized material changes vs raw deltas since last fire).
- Concurrency: per-Tracker lock granularity (like the connector indexing lock).

123
plans/backend/revamp phases 4-7/07-orchestration.md Normal file
View file

@ -0,0 +1,123 @@
# Phase 7 — Orchestration / Conversation (the CI-expert subagent)
> **Phase 7** of the CI-pivot revamp — the **human-facing brain** that turns "natural language is the only
> surface" (`04b`, Decision 0) into a real deliverable. **Build last** — it sits atop `04`–`06`.
> **Depends on** all prior phases: `04a` verbs, `04b` doors + `deliverable_wait`, `05a`/`05b` Tracker +
> Timeline, `06` refresh triggers.
> **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming,
> citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin
> subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it.
## Objective
Ship the **`intelligence_agent`** — a net-new builtin CI-expert subagent — so users get the whole product
through plain conversation: intent routing (one-shot vs standing concern), verb composition, Tracker
crafting, and decision-grounded answering. The intent router + verb-composition + Tracker-crafting are
where "user-friendly" is won or lost, so the orchestration layer is **designed, not assumed**.
## Current state (cited)
- **Builtin subagent pattern** — `subagents/builtins/<name>/`: `agent.py` (`build_subagent(...)` →
`pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py`
(`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md`
(the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`.
- **Tool shape** — `research/tools/scrape_webpage.py` already demonstrates **tool = capability executor +
access door + 03c billing**: calls `WebCrawlerConnector.crawl_url`, bills via the turn accumulator
(`get_current_accumulator()` + `WebCrawlCreditService`), returns a typed dict. CI tools follow this exactly.
- **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`).
## Target design
### The `intelligence_agent` subagent (CI expert)
A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to
`research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research
a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing
happen. It owns the **CI playbook** in `system_prompt.md`:
1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") → call
verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; ambiguous →
ask the single clarifying question.
2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place →
maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand.
3. **Tracker crafting** — the conversational schema-design flow from `05b`: sample-fetch → propose
`field_schema` + materiality + identity → user validates & locks → versioned.
4. **Decision-grounded answering** — read the Timeline (`05a`) to answer "what changed / is X pulling
ahead?" from stored deltas, not by re-deriving from chat history.
### The toolset (what `load_tools` returns)
| Tool | Wraps | Mode | Billing |
|------|-------|------|---------|
| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | `03c` turn accumulator (as `scrape_webpage` already does) |
| `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed |
| `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — |
| `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call |
| `query_timeline(tracker_id, …)` | `05a` read API | inline | — |
| `list_trackers()` | `05a`/`05b` read | inline | — |
- **Capability verbs are a shared tool module** (generated from the `04a` registry) — `research` can load
the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt.
(`scrape_webpage` is the seed; generalize it into the registry-backed set.)
- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the
existing `deliverable_wait` poll-until-terminal + live-card path (`04b`).
### Boundaries
- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `05a`/`05b` via tools; the hot loop,
materiality, and Timeline writes live in `05a`/`05b`, callable headless (so REST/MCP and Triggers reach the
same logic with no agent in the loop).
- **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this
subagent entirely and call `04a` verbs directly — they *want* explicitness.
## Work items
1. **Subagent scaffold**: `subagents/builtins/intelligence_agent/` with `agent.py` / `tools/index.py` /
`description.md` / `system_prompt.md`, packed via `pack_subagent`.
2. **CI playbook prompt**: intent routing (A/B + one clarifying question), verb chains, crafting flow,
decision-grounded answering.
3. **Shared verb tool module**: registry-backed (`04a`) capability tools, reusable by `research` too.
4. **Tracker/Timeline tools**: `craft_tracker` / `lock_tracker` / `update_tracker` / `refresh_tracker` /
`query_timeline` / `list_trackers`.
5. **Slow-verb wiring**: route job-mode tools through `deliverable_wait` + the live card.
6. **Router registration**: `description.md` so the main agent delegates CI-flavored requests here.
## Tests
- **Delegation**: a CI-flavored request routes to `intelligence_agent`; a non-CI request does not.
- **One-shot**: "compare X and Y" composes verbs and answers in plain language, persisting nothing.
- **Standing concern**: "watch X weekly" runs the crafting flow → lock → (refresh via `06`).
- **Crafting**: `craft_tracker` does a real sample-fetch and proposes a schema; `lock_tracker` versions it.
- **Answering**: `query_timeline` answers "what changed?" from stored deltas, not chat history.
- **Slow verb**: a multi-URL scrape surfaces a live card and returns terminal results.
## Risks / trade-offs
- **Prompt quality is the product**: intent-router/composition quality is a first-class deliverable, not
plumbing — budget iteration on the prompt.
- **One-shot ownership**: whether `research` keeps generic scraping or the `intelligence_agent` owns all
CI-flavored calls (lean: shared verb tools, CI agent owns the *playbook*).
- **Pre-frontend lock UX**: "review & lock" must render in pure chat for MVP (ties to `05b`'s open Q).
## Resolved decisions
1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not
a runtime rebuild.
2. Tools follow the `scrape_webpage` shape: capability executor + access door + `03c` billing.
3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds
Tracker/Timeline tools + the CI prompt.
4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`.
5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async.
## Out of scope (hand-offs)
- **The verbs, doors, engine, state, and triggers** → `04`/`05`/`06` (this phase only *orchestrates* them).
- **Richer multi-step CI playbooks** (auto competitor discovery → multi-Tracker setup), proactive "you
should watch this" suggestions, cross-Tracker synthesis → north star (deferred).
- **Frontend CI surfaces** (cards, dashboards) → frontend umbrella.
## Open questions (carry forward)
- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls
(lean: shared verb tools, `intelligence_agent` owns the *CI playbook*).
- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `05b`'s open Q).