docs(revamp): reshape Capabilities into Phase 4a

plans/backend/revamp/01-capabilities.md

@@ -1,165 +0,0 @@
-# Domain ① — Capabilities (CI pivot revamp · WIP)
-
-> **WIP design doc.** Part of the Phase 4 → end revamp (the old "pipeline").
-> **Scope guardrail:** Phases 1–3 are **SHIPPED and FIXED** — the rename, and the proprietary
-> crawler/proxy/billing/stealth/captcha stack. This domain *builds on* them and never modifies them.
-
-## Role in the universe
-
-```
-FIXED foundation (done):  Acquisition (crawler moat) · Metering (03c) · Identity/Tenancy
-OUR scope (this revamp):  ▶ Capabilities ◀ → Access → Intelligence + Timeline → Triggers
-```
-
-Capabilities sits directly on the fixed foundation and is consumed by **Access** (the doors) and
-the **Intelligence** hot loop. It is the first domain because everything above it calls into it.
-
-## Purpose
-
-Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed,
-callable verbs** that every door (chat / REST / MCP) and the Intelligence loop consume
-**identically**.
-
-This is the core pivot of old Phase 4: **replace "connector that ingests into the KB" with
-"capability that returns data."** A capability is a stateless function you *call and get data
-back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached.
-
-## Foundation it builds on (do not touch)
-
-- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch; `{content, metadata,
-  crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS`.
-  (`app/proprietary/web_crawler/`.)
-- `WebCrawlCreditService` (`03c`) — per-success metering + the `bill=` seam.
-- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier.
-
-## The verb set (MVP) — namespaced, nothing top-level
-
-Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor).
-Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …).
-
-| Verb | Input → Output | Mode | Executes over | Bills (03c) |
-|------|----------------|------|---------------|-------------|
-| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success |
-| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* |
-| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place |
-| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 |
-| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page |
-
-## `web.scrape` — one array-friendly verb (no separate "batch")
-
-Scraping always **welcomes an array of URLs**. There is no separate `scrape_batch`. One verb,
-one mental model: "give me URLs, I give you per-URL content."
-
-## Execution mode — a property of the **result**, not two verbs
-
-A capability does **not** split into sync/async variants. Every call returns a **uniform result
-envelope**:
-
-```
-{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...] , error?: ... }
-```
-
-- **inline (fast path):** small/fast input → finishes in-request → envelope returns `completed`
-  with `results` inline → **zero polling**. (single page, one place, a search)
-- **job (slow path):** large/slow input → envelope returns `pending` + `job_id` → caller polls a
-  status endpoint until `completed`, then reads `results`. (batch scrape, Maps search/reviews)
-
-So "sync" is simply *a job that finished instantly*. Same verb, same response shape, no magic
-branching for the caller to reason about.
-
-- **Threshold** (how many URLs / how heavy before it goes async) is **configurable**.
-- A caller may pass `async: true` to **force** a job even for small input (agents that never want
-  to block).
-- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the
-  **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it.
-
-## The capability registry — single source of truth
-
-One registry entry per verb:
-
-```
-Capability {
-  name            # dotted, e.g. "web.scrape", "maps.search"
-  input_schema    # Pydantic
-  output_schema   # Pydantic
-  mode            # can-complete-inline?  +  job-capable?
-  executor        # the async fn (wraps Acquisition / Maps actor)
-  billing_unit    # how Metering charges this call
-}
-```
-
-The **three doors are generated from the registry**, not hand-written three times:
-
-- **chat tool** ← entry (tool def + handler)
-- **REST route** ← entry (path + request/response models)
-- **MCP tool** ← entry (MCP tool schema + handler)
-
-…and the **Intelligence** hot loop calls the *same* `executor` directly. Add a verb once → it
-lights up on every surface; the I/O contract cannot drift between surfaces.
-
-## Where it lives / license boundary
-
-- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store).
-- It **imports from** the proprietary Acquisition engine but never moves into it.
-- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*`
-  executors — same boundary rule Phase 3 set (Apache code may import *from* proprietary, not move
-  *into* it).
-
-## Billing integration — open to the billing service (not hardcoded per verb)
-
-Billing is **not** baked into each verb. A capability only **declares a `billing_unit`** in its
-registry entry; charging is delegated to the **billing service**, so adding/repricing a unit is a
-billing-service concern, not a capability rewrite. `03c`'s `WebCrawlCreditService` is the **first
-provider**; new units register with the same service.
-
-- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit).
-- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet).
-- captcha attempts → existing per-attempt `web_crawl_captcha` unit (unchanged).
-- `web.discover` → register a per-search unit (or mark free) **with the billing service**.
-
-The point: the registry says *"this verb bills unit X"*; the **billing service owns what unit X
-costs and how it settles.** Verbs stay pure; pricing stays pluggable.
-
-## The Maps actor (clarification)
-
-- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …).
-- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`),
-  not raw markdown — which is exactly what the Intelligence/Timeline layer needs to diff reliably.
-- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint
-  on the capability. The capability never knows what decision it serves.
-
-## Relationship to the drafted Phase 4
-
-- **04b (source-discovery)** → **absorbed** as `web.discover` + relocating Linkup/Baidu keys from
-  per-connector `config` to platform env (SearXNG already env-based). Tavily/Serper dropped.
-- **04a (connector taxonomy + MCP routing)** → **demoted to backward-compat hygiene.** The
-  capability registry is the new spine, so the Type-1/Type-2 *connector* registry is no longer
-  central. 04a's "turn off branded connectors" stays a small, separable cleanup for the legacy
-  chat agent; the **BYO-`MCP_CONNECTOR`** routing fix belongs to the **Access/Conversation** domain
-  (it lets the agent use the *user's* external tools — a different concept from our capabilities).
-
-## Deferred (not in MVP)
-
-- **Recursive site crawl** (follow links across a whole site) — doesn't exist today; the
-  "crawl a topic" story is `web.discover` → `web.scrape`.
-- A generic **`web.extract`** (LLM-structured extraction as a standalone verb) — structured
-  extraction lives in the Intelligence loop for MVP; promote to a capability later if devs want it.
-- Additional platform namespaces (`linkedin.*`, `amazon.*`, …).
-
-## Locked decisions
-
-1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level.
-2. One array-friendly `web.scrape` (no separate batch verb).
-3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs.
-4. Build the thin job model now (Maps search / batch scrape force it).
-5. One capability registry → generates chat/REST/MCP doors + feeds the Intelligence executor.
-6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing
-   service** (verbs declare a `billing_unit`; `03c` is the first provider, new units register there).
-7. Connector → capability is a **replacement** for these data sources, not an extension.
-
-## Open questions (carry forward)
-
-- Default async threshold for `web.scrape`.
-- The **Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a
-  **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain
-  doesn't block on it.

plans/backend/revamp/04a-capabilities.md

@@ -0,0 +1,177 @@
+# Phase 4a — Capabilities (typed verbs over Acquisition)
+
+> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7).
+> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above
+> calls into it.
+> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome`
+> contract, the proxy provider, and the `WebCrawlCreditService` billing seam.
+> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies
+> them. Locate foundation code by **symbol/grep**, not the line numbers cited.
+
+## Objective
+
+Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable
+verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume
+**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the
+KB" with "capability that returns data."** A capability is a stateless function you *call and get data
+back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached.
+
+## Current state (cited)
+
+- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content,
+  metadata, crawler_type}` on `SUCCESS`; billable predicate single-sourced as `status == SUCCESS`
+  (`app/proprietary/web_crawler/`).
+- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` /
+  `successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`).
+- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`).
+- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the
+  substrate for `web.discover`.
+- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope).
+
+## Target design
+
+### The verb set (MVP) — namespaced, nothing top-level
+
+Two namespaces: **`web.*`** (generic crawler product) and **`maps.*`** (the one platform actor).
+Future platforms slot in as their own namespace (`linkedin.*`, `amazon.*`, …).
+
+| Verb | Input → Output | Mode | Executes over | Bills (03c) |
+|------|----------------|------|---------------|-------------|
+| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success |
+| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* |
+| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place |
+| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 |
+| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page |
+
+### `web.scrape` — one array-friendly verb (no separate "batch")
+
+Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model:
+"give me URLs, I give you per-URL content."
+
+### Execution mode — a property of the **result**, not two verbs
+
+A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**:
+
+```
+{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... }
+```
+
+- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline →
+  zero polling. (single page, one place, a search)
+- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until
+  `completed`. (batch scrape, Maps search/reviews)
+
+So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller.
+
+- **Threshold** (how many URLs / how heavy before async) is **configurable**.
+- A caller may pass `async: true` to **force** a job even for small input (agents that never block).
+- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the
+  **existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it.
+
+### The capability registry — single source of truth
+
+One registry entry per verb:
+
+```
+Capability {
+  name            # dotted, e.g. "web.scrape", "maps.search"
+  input_schema    # Pydantic
+  output_schema   # Pydantic
+  mode            # can-complete-inline?  +  job-capable?
+  executor        # the async fn (wraps Acquisition / Maps actor)
+  billing_unit    # how Metering charges this call
+}
+```
+
+The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool,
+REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly.
+Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces.
+
+### Billing — open to the billing service (not hardcoded per verb)
+
+A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the
+**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite.
+`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service.
+
+- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit).
+- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet).
+- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged).
+- `web.discover` → register a per-search unit (or mark free) **with the billing service**.
+
+The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs
+stay pure; pricing stays pluggable.
+
+### The Maps actor (clarification)
+
+- **Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …).
+- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw
+  markdown — exactly what the Intelligence/Timeline layer needs to diff reliably.
+- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on
+  the capability. The capability never knows what decision it serves.
+
+### Where it lives / license boundary
+
+- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store).
+- It **imports from** the proprietary Acquisition engine but never moves into it.
+- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*`
+  executors — same boundary rule Phase 3 set.
+
+## Work items
+
+1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import.
+2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL
+   result shape; declare the `web_crawl` `billing_unit`.
+3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed);
+   declare its `billing_unit` (or free).
+4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs.
+5. **Uniform envelope**: shared `completed|pending` result type returned by every executor.
+6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor.
+7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code).
+
+## Tests
+
+- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns
+  `pending` + `job_id`; the job completes to `results`.
+- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail
+  the batch.
+- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED`
+  free; disabling the billing flag makes it a no-op.
+- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb.
+- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set.
+
+## Risks / trade-offs
+
+- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when
+  the actor lands (doesn't block `web.*`).
+- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never
+  grows back into `pipeline_runs`.
+- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact
+  mid-run behavior is an implementation-time call.
+
+## Resolved decisions
+
+1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level.
+2. One array-friendly `web.scrape` (no separate batch verb).
+3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs.
+4. Build the thin job model now (Maps search / batch scrape force it).
+5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`).
+6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service**
+   (verbs declare a `billing_unit`; `03c` is the first provider, new units register there).
+7. Connector → capability is a **replacement** for these data sources, not an extension.
+
+## Out of scope (hand-offs)
+
+- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`.
+- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless.
+- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a
+  **separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain
+  doesn't block on it.
+- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces
+  (`linkedin.*`, `amazon.*`) — deferred.
+- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing
+  fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`.
+
+## Open questions (carry forward)
+
+- Default async threshold for `web.scrape`.
+- Whether `web.discover` is metered or free.