mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-08 22:22:17 +02:00
Merge pull request #1567 from CREDO23/feature-ci_phase-4-7
[Feat] CI: unified web.crawl + per-platform subagents; retire legacy connectors
This commit is contained in:
Rohan Verma
GitHub
commit
de04ef7111
187 changed files with 4512 additions and 5155 deletions
52
docker/docker-compose.watch-e2e.yml
Normal file
52
docker/docker-compose.watch-e2e.yml
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# =============================================================================
|
||||
# SurfSense — Isolated deps for the watch (Phase 6) manual E2E
|
||||
# =============================================================================
|
||||
# Fresh Postgres + Redis ONLY, with their own project name, volumes, and host
|
||||
# ports so they never collide with the day-to-day `surfsense-deps` stack.
|
||||
#
|
||||
# Backend + Celery run on the HOST (tests/e2e/run_backend.py / run_celery.py)
|
||||
# pointed at these ports via env:
|
||||
# DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5442/surfsense
|
||||
# CELERY_BROKER_URL / CELERY_RESULT_BACKEND / REDIS_APP_URL=redis://localhost:6389/0
|
||||
#
|
||||
# Up: docker compose -p surfsense-watch -f docker/docker-compose.watch-e2e.yml up -d
|
||||
# Down: docker compose -p surfsense-watch -f docker/docker-compose.watch-e2e.yml down -v
|
||||
# =============================================================================
|
||||
|
||||
name: surfsense-watch
|
||||
|
||||
services:
|
||||
db:
|
||||
image: pgvector/pgvector:pg17
|
||||
ports:
|
||||
- "5442:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
environment:
|
||||
- POSTGRES_USER=postgres
|
||||
- POSTGRES_PASSWORD=postgres
|
||||
- POSTGRES_DB=surfsense
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres -d surfsense"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
redis:
|
||||
image: redis:8-alpine
|
||||
ports:
|
||||
- "6389:6379"
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
command: redis-server --appendonly yes
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
name: surfsense-watch-postgres
|
||||
redis_data:
|
||||
name: surfsense-watch-redis
|
||||
|
|
@ -2,16 +2,30 @@
|
|||
|
||||
> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`).
|
||||
|
||||
This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities & Access**, **Phase 5 Intelligence & Timeline**, **Phase 6 Triggers**, **Phase 7 Orchestration** (canonical subplans in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was **superseded on 2026-06-30** — see the Architecture correction below. The earlier "Phase 5′ automation-enhancements" framing is also retired (folded into Phases 4/5/6 of the revamp).
|
||||
This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities** ([`04-capabilities.md`](04-capabilities.md)), **Phase 5 Access** ([`05-access.md`](05-access.md)), **Phase 6 Ongoing-Automation** ([`06-ongoing-automation.md`](06-ongoing-automation.md), design deferred), **Phase 7 Orchestration** ([`07-orchestration.md`](07-orchestration.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful **Tracker + Timeline + Triggers** design was superseded on **2026-07-01** (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below.
|
||||
|
||||
> SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases.
|
||||
|
||||
## Positioning
|
||||
|
||||
"NotebookLM for Competitive Intelligence" — each WorkSpace acts as a workspace for setting up competitive-intelligence-optimised notebooks.
|
||||
Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return **cleaned, AI-ready data** for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time.
|
||||
|
||||
## ⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native
|
||||
|
||||
> **This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6).** The canonical Phase 4–7 subplans (`04`–`07`, below) are re-cast to four phases:
|
||||
>
|
||||
> - **The product = scraper capabilities**, not a CI engine. `web.*` + `maps.*` typed verbs return cleaned, AI-ready data, exposed identically through **REST + API key**, an **MCP server**, and **chat**, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable).
|
||||
> - **Stateless.** No Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`. Memory = the **chat history**; the agent reasons over prior tool outputs in context to report "what changed."
|
||||
> - **Direct calls.** Capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling. Async results reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
|
||||
> - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6).
|
||||
> - **KB stays input-only**; crawled data is never indexed.
|
||||
>
|
||||
> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** · **07 Orchestration** (`scraping` subagent). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`.
|
||||
|
||||
## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted
|
||||
|
||||
> **Historical record.** The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the **2026-07-01 pivot** above.
|
||||
|
||||
> **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model:
|
||||
>
|
||||
> - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages.
|
||||
|
|
@ -20,35 +34,31 @@ This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shi
|
|||
> - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots.
|
||||
> - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source.
|
||||
>
|
||||
> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical revamp in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md):
|
||||
> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical subplans `04`–`07` (this framing was itself superseded by the 2026-07-01 pivot above):
|
||||
> - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.*
|
||||
> - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine.
|
||||
> - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path).
|
||||
> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `intelligence_agent` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language.
|
||||
> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `scraping` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language.
|
||||
>
|
||||
> **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors.
|
||||
|
||||
## Target architecture (corrected + revamped)
|
||||
## Target architecture (scraper-APIs-first, stateless)
|
||||
|
||||
> Full stateless/stateful end-to-end flow diagrams live in [`revamp phases 4-7/00b-pipeline-diagrams.md`](revamp%20phases%204-7/00b-pipeline-diagrams.md). Summary below.
|
||||
> Full end-to-end flow diagrams live in [`00b-pipeline-diagrams.md`](00b-pipeline-diagrams.md). Summary below.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (P8)"]]
|
||||
REG["Capability registry (P4a): web.scrape · web.discover (MVP) · <platform>.* scrapers (later) — typed verbs, bill at executor"]
|
||||
ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]]
|
||||
REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"]
|
||||
ACQ --> REG
|
||||
REG --> DOORS["Access doors (P4b): chat tools · REST+API key · MCP — generated from the registry"]
|
||||
DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · Tracker crafting"]
|
||||
REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"]
|
||||
DOORS --> AGENT["scraping subagent (P7): intent routing · verb chains · 'what changed' from chat history"]
|
||||
KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT
|
||||
CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT
|
||||
AGENT -->|one-shot / Product A| ANS["plain-language answer (nothing persists)"]
|
||||
AGENT -->|standing concern / Product B| TR["Tracker (P5b): locked schema · refresh hot loop"]
|
||||
REG --> TR
|
||||
TR --> TL[("Timeline (P5a): tracked_entities · entity_current_state · entity_changes — delta moat")]
|
||||
TRIG["Triggers (P6): manual · agent · external-cron · CI automation action"] --> TR
|
||||
AUTO["Automations (existing): schedule selector + AutomationRun (recurrence + audit; NO delivery)"] --> TRIG
|
||||
TL --> DELIVER["alerts on material change → app/notifications (Zero-synced)"]
|
||||
APIKEY["Platform API key (future)"] --> DOORS
|
||||
AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"]
|
||||
AGENT -.->|standing need| ONG["Ongoing-Automation (P6): persistent chat re-invokes verbs; memory = chat history"]
|
||||
ONG -.-> REG
|
||||
APIKEY["Platform API key"] --> DOORS
|
||||
```
|
||||
|
||||
<details>
|
||||
|
|
@ -78,14 +88,14 @@ flowchart TD
|
|||
|
||||
- Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps.
|
||||
- Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`.
|
||||
- **Capabilities, not connectors, are the core of Phase 4.** The crawler + search become typed **verbs** (MVP `web.scrape` + `web.discover`; per-platform scrapers like `maps.*` are later, uncommitted drop-ins) in a single **capability registry** that generates the chat/REST/MCP doors identically (revamp `04a`/`04b`). A capability is *call → data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into revamp `04b` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system. **~~pipeline-eligibility~~ removed.**
|
||||
- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent (or a Tracker binding) can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
|
||||
- **`Tracker` + Timeline are the standing-concern primitive** (replaces "pipeline"). A Tracker binds a capability + input and a locked, versioned `definition` (`field_schema`/`identity_rule`/`materiality`); its `refresh(tracker)` hot loop writes a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`). **Store deltas, no change → no row.** The Timeline — not the KB, not `automation_runs` — is the only new durable CI state (revamp `05a`/`05b`).
|
||||
- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via `WebCrawlCreditService` (03c) — so chat, automation/recurring, REST, MCP, and external-cron all meter **uniformly**. (Closes the confirmed gap where automation-run crawls billed nothing; see 03c + the Architecture correction.)
|
||||
- **Capabilities, not connectors, are the core of the product.** The crawler + search + Maps actor become typed **verbs** (`web.scrape`, `web.discover`, `maps.search/place/reviews`) in a single **capability registry** that generates the REST/MCP/chat doors identically (`04`/`05`). A capability is *call → cleaned data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into `05` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system.
|
||||
- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
|
||||
- **Stateless — the chat history is the memory.** No Timeline, no `Tracker`, no diff/materiality engine, no stored `entity_changes`. "What changed" is the agent reasoning over prior tool outputs already in context (`07`). Standing needs are served by a **persistent ongoing chat** that periodically re-invokes verbs (`06`, design deferred).
|
||||
- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via the billing service (`WebCrawlCreditService`, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter **uniformly**. `maps.*` registers a new per-place / per-page unit.
|
||||
- Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP.
|
||||
- MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point).
|
||||
- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines. The standing-concern need is met by the **`Tracker` + Timeline** (revamp `05`), and **recurrence + audit reuse the existing automations** (schedule selector + `AutomationRun`) via a CI **action** (revamp `06`) — **no new scheduler, no new run table**. **Delivery does NOT reuse automations** (no such path; `automation_runs.output` is never written) — alerts wire to the separate `app/notifications/` system. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper).
|
||||
- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by reading the **Timeline** (`query_timeline`, revamp `05a`), not by re-deriving from run logs. Recurring execution audit rides `AutomationRun` (already persisted + queryable). No separate pipeline-run context tool.
|
||||
- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a **persistent ongoing chat** re-invoking verbs (`06`, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper).
|
||||
- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by **reasoning over the chat history** (prior tool outputs already in context, `07`), not by re-deriving from run logs or a stored timeline.
|
||||
- **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**.
|
||||
- Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that.
|
||||
|
||||
|
|
@ -124,52 +134,50 @@ flowchart TD
|
|||
|
||||
The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans:
|
||||
|
||||
> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (revamp `04a`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture correction; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then revamp `04a` Capabilities → `04b` Access → `05` Intelligence & Timeline), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized.
|
||||
> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (`04`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then `04` Capabilities → `05` Access → `07` Orchestration), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized.
|
||||
|
||||
- **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran).
|
||||
- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability/per-Tracker override is left as a no-op seam (revamp `04a`/`05`), not built.
|
||||
- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture correction + revamp — see log): the charge moves to the **capability executor** (revamp `04a`), so **every** caller meters uniformly — chat, **automation/recurring** (the confirmed gap: `run_agent_task` establishes no turn accumulator today, so `scrape_webpage` bills nothing under automations), REST, MCP, and external-cron — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type).
|
||||
- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability override is left as a no-op seam (`04`), not built.
|
||||
- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture pivot + revamp — see log): the charge moves to the **capability executor** (`04`), so **every** caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type).
|
||||
- **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier.
|
||||
- **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account.
|
||||
- **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier.
|
||||
|
||||
### Phase 4 — Capabilities & Access (backend) [`subplans: revamp/04a–04b`]
|
||||
### Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`]
|
||||
|
||||
> **Canonical subplans:** [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md). Together they ship **Product A** (stateless utility) — revenue day one. **Build `04a` first** — every later phase calls the registry.
|
||||
> **Canonical subplan:** [`04-capabilities.md`](04-capabilities.md). **Build first** — every later phase calls the registry.
|
||||
|
||||
- **`revamp/04a-capabilities.md`** — Turn Acquisition (crawler + search + a future Maps actor) into a **small set of typed, callable verbs** over a single **capability registry**. MVP verbs = **only `web.scrape(urls[])`** (array-friendly, inline-or-job) **+ `web.discover(query, top_k)`** (SearXNG/Linkup/Baidu). **Platform-specific scrapers** (Maps/LinkedIn/Amazon/… — *examples, none committed*) are a family of **individual endpoints** added later: each is just another verb → agent tool + dev API-key REST endpoint, so `04a` only ships **contract stubs** for them, not executors. Uniform result **envelope** (`completed | pending + job_id`); slow calls use a **thin job record** + existing Celery (not a resurrected `pipeline_runs`). Each verb declares a **`billing_unit`**; charging is **delegated to the billing service** (03c is the first provider) and happens **at the executor** so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary.
|
||||
- **`revamp/04b-access.md`** — Expose the registry through **three generated doors** (order: chat → REST+API-key → MCP), each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → envelope`). **Natural language is the only human-facing surface** (users never name verbs); an **intent router** classifies one-shot (Product A) vs standing-concern (Product B). Slow chat verbs reuse the shipped `deliverable_wait` poll-until-terminal + live-card path. This is where the **BYO-`MCP_CONNECTOR` routing-gap fix** (from the old `04a`) lands (consume-user-MCP).
|
||||
- Turn Acquisition (crawler + search + Maps actor) into a **small set of typed, callable verbs** over a single **capability registry** — `web.scrape(urls[])`, `web.discover(query, top_k)`, `maps.search/place/reviews`. Each verb is a **direct async call that returns cleaned, AI-ready data** (no job store, no envelope, no polling). `maps.*` returns typed structured objects, not markdown. Each declares a **`billing_unit`**; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary.
|
||||
|
||||
**Locked model (MVP):**
|
||||
|
||||
- **Capabilities replace connectors as data sources.** `web.scrape` / `web.discover` (MVP) — and later per-platform scraper verbs — are *call → data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule.
|
||||
- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `revamp/04b`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**.
|
||||
- **Capabilities replace connectors as data sources.** `web.*` / `maps.*` are *call → cleaned data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule.
|
||||
- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `05`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**.
|
||||
- **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP.
|
||||
- **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook).
|
||||
- Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.)
|
||||
- The deferred "coming soon" platforms (LinkedIn, Amazon, Google, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as **Phase 8 native verbs / API endpoints**, not Type-1 KB connectors.
|
||||
- The **Google Maps actor is net-new**, proprietary, built as a separate effort; `maps.*` are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors.
|
||||
- Frontend connector/capability UI restructure is DEFERRED (frontend umbrella).
|
||||
|
||||
### Phase 5 — Intelligence & Timeline (backend) [`subplans: revamp/05a–05b`]
|
||||
### Phase 5 — Access (backend) [`subplan: 05-access.md`]
|
||||
|
||||
> **Canonical subplans:** [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md). Ships the **Product B engine**. **Depends on `04a`** (the verbs the loop calls). **Build `05a` (tables) before `05b` (the loop that writes them).** This *replaces* the old Pipelines Phases 5–7 — the durable CI state, and **the moat**.
|
||||
> **Canonical subplan:** [`05-access.md`](05-access.md). **Build after `04`.** Together `04 + 05` ship the scraper-API product — revenue day one.
|
||||
|
||||
- **`revamp/05a-timeline.md`** — The **3-store delta Timeline**: `tracked_entities` (stable identity, written once), `entity_current_state` (latest values + `content_hash` + `last_checked_at`, overwritten each run), `entity_changes` (append-only material deltas: `{field:{from,to}}`, `materiality`, `decided_by`, `source_ref`). **Store deltas, no change → no row** (storage ∝ rate of change). Explicitly **not the KB, not `automation_runs`, not a resurrected `pipeline_runs`.** New CI-owned Apache-2 tables + Alembic migration; write/read API for the `05b` loop and the read side. **This is where "diffable run history" actually lives** (superseding the old "structured `automation_runs.output`" idea).
|
||||
- **`revamp/05b-intelligence.md`** — The `Tracker` primitive + the **hot loop** `refresh(tracker)`: crawl (call the bound verb) → **content-hash pre-check** (identical → stamp `last_checked_at`, stop, no LLM) → **fill** raw data to the locked `field_schema` (structured output; extras → `notable_signals`) → **diff** (code) → **judge — the materiality split** (deterministic numeric/clear rules in code for free; agent only on ambiguous) → **append** a change + overwrite current state, else only `last_checked_at`. Setup = an **agent-designed, human-locked, versioned schema** flow (sample-fetch → draft `definition` → review/lock). **No new run table:** recurring audit/idempotency ride `AutomationRun`, manual rides the chat job record; **billing idempotency is per-capability-call** + the hash pre-check. Optional **CI context folder** (user files uploaded in a CI chat go into the KB as normal, routed to a folder, and can feed the judge). New Apache-2 package `app/intelligence/`.
|
||||
- Expose the registry through **three generated doors** — **REST + API key** (public day one), an **MCP server** (fast-follow), and **chat tools** — each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data`). The executor returns directly; there is **no envelope, no `deliverable_wait` polling**. Async results (e.g. the `06` ongoing mode) reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
|
||||
- **Natural language is the human surface** (raw verbs only on REST/MCP); the chat door classifies **one-shot** vs **keep-watching** and hands keep-watching to `06`. Two MCP directions kept distinct: **serve** our verbs as an MCP server (new door) vs **consume** the user's BYO `MCP_CONNECTOR` (the old-`04a` routing-gap fix lands here).
|
||||
|
||||
### Phase 6 — Triggers (backend) [`subplan: revamp/06-triggers.md`]
|
||||
### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`]
|
||||
|
||||
> **Canonical subplan:** [`revamp phases 4-7/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) — the thinnest phase. **Depends on `05b`** (`refresh(tracker)`). Decides **when** a Tracker refreshes; Intelligence never knows which trigger fired.
|
||||
> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **shipped.** A chat watch is an `Automation` bound to the current chat (`schedule` + `chat_message`); the durable checkpointer is the memory. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel).
|
||||
|
||||
- Adapters: **manual** (user "refresh now" via chat tool / REST), **agent** (the agent calls `refresh` as a tool), **external cron** (`POST /v1/trackers/{id}/refresh`, zero infra on us), and the **CI automation action** — recurrence reusing the **existing automations**: its hardened **schedule selector** (no new scheduler) + **`AutomationRun`** (no new run table). **Alert delivery is separate** — persist deltas to the Timeline, then emit an in-app notification via `app/notifications/` (automations have no delivery path; `run.output` is unwritten). Guard concurrency with a **per-Tracker lock** (the automation PENDING-gate is non-atomic). A single `refresh_tracker` action, **not** a new automation shape.
|
||||
- **Decoupling preserved:** CI core (`refresh` + Timeline) runs with **zero** automations dependency (manual/agent/external-cron); automations is the **optional** in-app recurrence+alert adapter. This is the correct home for the old "wire a run-now trigger" item.
|
||||
- Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls.
|
||||
|
||||
### Phase 7 — Orchestration (backend) [`subplan: revamp/07-orchestration.md`]
|
||||
### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`]
|
||||
|
||||
> **Canonical subplan:** [`revamp phases 4-7/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — the human-facing brain. **Build last** (atop `04`–`06`). We plug into the shipped multi-agent runtime; we don't rebuild it.
|
||||
> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Shipped for the web verbs** (atop `04`/`05`/`06`). We plug into the shipped multi-agent runtime; we don't rebuild it.
|
||||
|
||||
- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs standing concern, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), **Tracker crafting** (the `05b` schema-design flow in chat), and **decision-grounded answering** (read the Timeline via `query_timeline`, not chat history).
|
||||
- Toolset: registry-backed capability verbs (shared with `research`) + Tracker/Timeline tools (`craft_tracker`/`lock_tracker`/`refresh_tracker`/`query_timeline`/`list_trackers`); slow verbs reuse `deliverable_wait`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing).
|
||||
- Ship the builtin **`scraping`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, block on ambiguous cadence), **verb composition** (`web.discover → web.scrape`; `maps.*` chains deferred until the Maps actor exists), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store).
|
||||
- Toolset: registry-backed capability verbs + `start_watch`/`stop_watch`/`refresh_watch` bound to the current chat (`06`). Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`).
|
||||
|
||||
### (Future) Platform API-key + public MCP
|
||||
|
||||
|
|
@ -177,7 +185,7 @@ Platform **API-key** access to the capability verbs + a public **MCP server** ex
|
|||
|
||||
### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`]
|
||||
|
||||
NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Once the hardened fetch core (Phases 3a–3e) is solid, **platform scrapers** — a family of **individual scraping endpoints** (Google Maps/Local, LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **all examples, none committed**) — layer **on top** of it. **Each is simply another capability verb** (`revamp/04a`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha), callable by the chat agent, by automations, and by the future platform API key. Billed per call via `WebCrawlCreditService`.
|
||||
NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the **committed MVP actor** (a `maps.*` namespace, Phase 4). Beyond it, more **platform scrapers** — a family of **individual scraping endpoints** (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **examples, none committed**) — layer **on top** of the hardened fetch core once it's solid. **Each is simply another capability verb** (`04`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha). Billed per call via `WebCrawlCreditService`.
|
||||
|
||||
- **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream.
|
||||
- The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling.
|
||||
|
|
@ -189,7 +197,7 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
|
|||
- Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename.
|
||||
- Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs.
|
||||
- Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types.
|
||||
- CI + automations + positioning UI (**replaces the superseded "Pipelines UI"**): the CI experience (Tracker crafting/review-&-lock, Timeline/change views, "refresh now"), automation create/list/run-history, `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards, "NotebookLM for Competitive Intelligence" copy.
|
||||
- Scraper-API + chat + positioning UI (**replaces the superseded "Pipelines UI"**): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards.
|
||||
|
||||
## Open items to confirm during subplanning
|
||||
|
||||
|
|
@ -197,11 +205,12 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
|
|||
- ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with.
|
||||
- ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`).
|
||||
- ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP.
|
||||
- **Revamp cross-phase forks (from `revamp/00-overview.md`, decide during subplanning):** (1) schema **review-&-lock** UX pre-frontend — pure-chat confirmation for MVP? (`05b`); (2) Timeline ORM home — `app/db.py` vs a dedicated `app/timeline/` (`05a`); (3) recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback) (`06`); (4) is `web.discover` metered or free (`04a`); (5) MCP-serve auth depth — OAuth 2.1 vs bearer (`04b`).
|
||||
- **Revamp open items (decide during subplanning):** (1) the **Ongoing-Automation mechanism** (`06`, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large `web.scrape` / `maps.reviews` responses are bounded/streamed (`04`); (3) is `web.discover` metered or free (`04`); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (`05`); (5) public REST rate-limiting / abuse posture (`05`).
|
||||
|
||||
## Resolved decisions log
|
||||
|
||||
- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft in [`revamp phases 4-7/`](revamp%20phases%204-7/00-overview.md). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`.
|
||||
- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`scraping`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below.
|
||||
- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`scraping`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`.
|
||||
- **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths).
|
||||
- ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type.
|
||||
- Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).
|
||||
|
|
@ -239,17 +248,10 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
|
|||
| 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) |
|
||||
| 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default |
|
||||
| 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff |
|
||||
| 4 | `revamp phases 4-7/04a-capabilities.md` | **CANONICAL** · drafted — capability registry + MVP verbs `web.scrape`/`web.discover` (per-platform scrapers = later drop-ins); bill at executor. **← build next** |
|
||||
| 4 | `revamp phases 4-7/04b-access.md` | **CANONICAL** · drafted — generated chat/REST/MCP doors + intent router + BYO-MCP routing fix |
|
||||
| 5 | `revamp phases 4-7/05a-timeline.md` | **CANONICAL** · drafted — 3-store delta Timeline (the moat) |
|
||||
| 5 | `revamp phases 4-7/05b-intelligence.md` | **CANONICAL** · drafted — `Tracker` + refresh hot loop + materiality split |
|
||||
| 6 | `revamp phases 4-7/06-triggers.md` | **CANONICAL** · drafted — refresh clock; recurrence = CI action on automations |
|
||||
| 7 | `revamp phases 4-7/07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent |
|
||||
| — | `revamp phases 4-7/00-overview.md` · `00b-pipeline-diagrams.md` | revamp map + reconciliation + end-to-end flow diagrams |
|
||||
| 4 | `04a-connector-category.md` *(flat)* | ↪ **REDIRECT** — demoted to backward-compat hygiene; superseded by `revamp/04a`+`04b` |
|
||||
| 4 | `04b-source-discovery.md` *(flat)* | ↪ **REDIRECT** — absorbed as the `web.discover` verb in `revamp/04a` |
|
||||
| 5 | `05-pipelines-model.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05a`+`05b` (Tracker/Timeline) |
|
||||
| 6 | `06-pipelines-exec.md` *(flat)* | ❌ **SUPERSEDED** — replaced by `revamp/05b` (hot loop) + `revamp/06` (triggers) |
|
||||
| 7 | `07-upload-pipeline-kb.md` *(flat)* | ❌ **SUPERSEDED** — uploads populate the input-only KB (no pipeline wrapper) |
|
||||
| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) |
|
||||
| 4 | `04-capabilities.md` | **IMPLEMENTED (web verbs)** — capability registry + `web.scrape`/`web.discover` → cleaned data, bill at executor (top_k bounds, max_length truncation, per-attempt captcha metering). `maps.*` deferred (Maps actor) |
|
||||
| 5 | `05-access.md` | **IMPLEMENTED (REST + chat doors)** — generated REST/API-key + chat tools from the registry. MCP door + BYO-MCP routing fix deferred (MCP server) |
|
||||
| 6 | `06-ongoing-automation.md` | **IMPLEMENTED** — chat-native "keep watching": `Automation` bound to the chat (`schedule` + `chat_message`), worker-safe durable checkpointer, watch tools + REST controls |
|
||||
| 7 | `07-orchestration.md` | **IMPLEMENTED (web verbs)** — `scraping` subagent; intent routing, verb composition, "what changed" from chat history |
|
||||
|
||||
Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases").
|
||||
|
|
|
|||
109
plans/backend/00b-pipeline-diagrams.md
Normal file
109
plans/backend/00b-pipeline-diagrams.md
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
# Pipeline diagrams — end-to-end (scraper-APIs-first, stateless)
|
||||
|
||||
> Visual companion to `00-umbrella-plan.md`.
|
||||
> Phase refs: `04` Capabilities · `05` Access · `06` Ongoing-Automation (design deferred) · `07` Orchestration.
|
||||
|
||||
## 1. The shape — scraper APIs + a chat agent (no state)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph FIXED["FIXED · Phases 1-3 (shipped)"]
|
||||
ACQ["Acquisition<br/>crawler · proxy · stealth · captcha"]
|
||||
MET["Metering · 03c"]
|
||||
ID["Identity / API keys"]
|
||||
end
|
||||
|
||||
subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"]
|
||||
CAP["04 Capabilities<br/>web.* · maps.*<br/>cleaned, AI-ready output"]
|
||||
ACC["05 Access<br/>REST · MCP · chat doors"]
|
||||
ORC["07 Orchestration<br/>scraping subagent"]
|
||||
ONG["06 Ongoing-Automation<br/>(keep-watching)"]
|
||||
end
|
||||
|
||||
ACQ --> CAP
|
||||
MET --> CAP
|
||||
CAP --> ACC
|
||||
ID --> ACC
|
||||
ACC --> ORC
|
||||
ORC -. "standing need" .-> ONG
|
||||
ONG -. "re-invokes verbs" .-> CAP
|
||||
|
||||
classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
|
||||
classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
|
||||
classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
|
||||
class CAP,ACC,ORC a;
|
||||
class ACQ,MET,ID b;
|
||||
class ONG d;
|
||||
```
|
||||
|
||||
Memory for "what changed" = the chat history.
|
||||
|
||||
## 2. Doors — one registry, three surfaces (generated)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
REG["04 Capability registry<br/>web.scrape · web.discover · maps.search/place/reviews"]
|
||||
REG --> R["REST + API keys<br/>(public · OSS self-host · revenue)"]
|
||||
REG --> M["MCP server<br/>(external agents · fast-follow)"]
|
||||
REG --> C["Chat tools<br/>(in-app agent, 07)"]
|
||||
R --> EX["same executor · direct return · 03c billing"]
|
||||
M --> EX
|
||||
C --> EX
|
||||
```
|
||||
|
||||
## 3. One-shot scrape (the core path) — direct call, no polling
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant U as User / Dev
|
||||
participant D as Door 05 (REST · MCP · chat)
|
||||
participant EX as Verb executor 04
|
||||
participant ACQ as Acquisition / Maps actor
|
||||
participant BILL as Billing (03c)
|
||||
|
||||
U->>D: intent (chat) OR raw verb call (REST/MCP)
|
||||
D->>D: validate input_schema · authn/authz · meter-gate
|
||||
D->>EX: call executor (same for all doors)
|
||||
EX->>ACQ: crawl_url / maps / search
|
||||
ACQ-->>EX: cleaned, AI-ready data
|
||||
EX->>BILL: charge billing_unit per success
|
||||
EX-->>D: results (returned directly)
|
||||
D-->>U: plain-language answer (chat) / JSON (REST/MCP)
|
||||
Note over U,BILL: Stateless — nothing persists.
|
||||
```
|
||||
|
||||
## 4. Intent fork (in the agent, 07)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U(["User speaks in natural language"]) --> R{"Intent router<br/>(scraping prompt, 07)"}
|
||||
R -->|"find / compare / what is / right now"| A["ONE-SHOT<br/>compose verbs, answer (section 3)"]
|
||||
R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING<br/>start_watch -> Ongoing-Automation (06)"]
|
||||
R -->|ambiguous cadence| Q["Block: ask for the missing cadence / timezone"]
|
||||
Q -->|once| A
|
||||
Q -->|keep watching| B
|
||||
```
|
||||
|
||||
## 5. "What changed" — chat history is the memory
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
P["Prior tool outputs<br/>(already in chat context)"] --> AG["scraping subagent"]
|
||||
NEW["Fresh verb call (04)"] --> AG
|
||||
AG --> ANS["'Here's what's new since last time'<br/>(reasoned from chat history)"]
|
||||
```
|
||||
|
||||
## 6. Ongoing-Automation (06) — intent only, mechanism deferred
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
W["'Keep watching X'"] --> LOOP["Persistent ongoing chat<br/>periodically re-invokes verbs (04)"]
|
||||
LOOP --> OUT["results delivered into the session<br/>(write-then-sync / stream)"]
|
||||
OUT --> MEM["chat history = memory<br/>agent reports what's new"]
|
||||
LOOP -.->|OPEN| Q["periodicity driver? · delivery channel? · context-window limit?"]
|
||||
classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
|
||||
class W,LOOP,OUT,MEM d;
|
||||
```
|
||||
|
||||
> Section 6 is a **placeholder** — the periodic mechanism is designed separately (see `06-ongoing-automation.md`).
|
||||
|
|
@ -133,7 +133,7 @@ These do not move with a symbol rename; each is decided here.
|
|||
|
||||
- Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella.
|
||||
- Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred.
|
||||
- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04a-connector-category.md](04a-connector-category.md)).
|
||||
- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04-capabilities.md](04-capabilities.md)).
|
||||
- Enum VALUE migration and observability-key rename — deliberately deferred announced changes.
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)).
|
||||
|
||||
> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** (revamp [`04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, **automation/recurring**, REST, MCP, external-cron. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context.
|
||||
> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** ([`04-capabilities.md`](04-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, ongoing automation, REST, MCP. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context.
|
||||
|
||||
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
|
||||
> Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`).
|
||||
|
|
|
|||
89
plans/backend/04-capabilities.md
Normal file
89
plans/backend/04-capabilities.md
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
# Phase 4 — Capabilities (the scraper APIs)
|
||||
|
||||
> Build first; every other phase calls this registry. Sibling: `05-access.md` (the doors).
|
||||
> Reuses Phases 3a/3b/3c (shipped): `WebCrawlerConnector.crawl_url -> CrawlOutcome`, the proxy/stealth/
|
||||
> captcha tiers, and the `WebCrawlCreditService` billing seam. Locate code by symbol/grep.
|
||||
|
||||
## Objective
|
||||
|
||||
Expose the Acquisition engine as a small set of typed, stateless scraper verbs (`web.*`, `maps.*`) that
|
||||
return cleaned, AI-ready structured data. One capability = one function you call and get data back from.
|
||||
|
||||
## Verb set (MVP)
|
||||
|
||||
Two namespaces: `web.*` (generic crawler) and `maps.*` (Google Maps actor). Future platforms slot in as
|
||||
their own namespace.
|
||||
|
||||
| Verb | Input → Output (cleaned) | Executes over | Billing unit |
|
||||
|------|--------------------------|---------------|--------------|
|
||||
| `web.scrape(urls[])` | `[{url, status, content, metadata}]` | loop `crawl_url` | per success |
|
||||
| `web.discover(query, top_k)` | `[{url, title, snippet, provider}]` | search providers | per search / free |
|
||||
| `maps.search(query, location)` | `[place]` | Maps actor | per place |
|
||||
| `maps.place(place_id\|url)` | `place` | Maps actor | 1 |
|
||||
| `maps.reviews(place)` | `[review]` (paged) | Maps actor | per page |
|
||||
|
||||
`maps.*` returns typed structured objects (`{name, rating, review_count, hours, price_level, …}`).
|
||||
`web.scrape` takes a URL array.
|
||||
|
||||
## Execution
|
||||
|
||||
Each verb is a direct async call that returns its cleaned result. Slow verbs (large `web.scrape` arrays,
|
||||
`maps.reviews` paging) are bounded/streamed at the endpoint.
|
||||
|
||||
## Registry
|
||||
|
||||
One entry per verb, and the single source of truth the doors (`05`) and the agent (`07`) read from:
|
||||
|
||||
```
|
||||
Capability {
|
||||
name # dotted, e.g. "web.scrape", "maps.search"
|
||||
input_schema # Pydantic
|
||||
output_schema # Pydantic (cleaned, AI-ready)
|
||||
executor # async fn; wraps Acquisition / Maps actor; returns data directly
|
||||
billing_unit # how the billing service charges this call
|
||||
}
|
||||
```
|
||||
|
||||
Adding a verb once lights it up on every door.
|
||||
|
||||
## Billing
|
||||
|
||||
A verb declares a `billing_unit`; charging is delegated to the billing service (`03c` first provider).
|
||||
|
||||
- `web.scrape` → per `SUCCESS` (`web_crawl` unit).
|
||||
- `maps.*` → new per-place / per-page unit registered with the billing service.
|
||||
- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`).
|
||||
- `web.discover` → per-search unit (or free).
|
||||
|
||||
## Location
|
||||
|
||||
New Apache-2 package `app/capabilities/` (registry, schemas, executors) — open-source and self-hostable.
|
||||
Executors import the proprietary Acquisition engine and Maps extractor (`app/proprietary/…`).
|
||||
|
||||
## Work items
|
||||
|
||||
1. Registry package: `Capability` dataclass + registry, imported by `05` and `07`.
|
||||
2. `web.scrape` executor: loop `crawl_url` over the URL array → per-URL cleaned rows; `web_crawl` unit.
|
||||
3. `web.discover` executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit.
|
||||
4. `maps.*` contracts: input/output schemas + executor stubs against the Maps actor.
|
||||
5. Output normalization: each executor returns AI-ready structured output.
|
||||
6. Billing seam: honor `billing_unit` via the billing service.
|
||||
|
||||
## Tests
|
||||
|
||||
- `web.scrape([a,b,c])` returns one cleaned row per URL inline; partial failures don't fail the batch.
|
||||
- Each `SUCCESS` bills one `web_crawl` unit; `EMPTY`/`FAILED` free; disabling the flag no-ops.
|
||||
- A door and the agent hit the same executor for a verb.
|
||||
- `web.discover` returns `{url,title,snippet,provider}`; self-disables when no provider key is set.
|
||||
- `maps.*` returns typed structured objects against fixtures.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Doors → `05`. Keep-watching mode → `06`. Agent → `07`.
|
||||
- Google Maps actor build (incl. sourcing legality) — net-new, separate effort; `maps.*` are contracts.
|
||||
- Additional platform namespaces; recursive site crawl; a generic `web.extract` verb.
|
||||
|
||||
## Open questions
|
||||
|
||||
- `web.discover` metered vs free.
|
||||
- How large `web.scrape` / `maps.reviews` responses are bounded/streamed.
|
||||
|
|
@ -1,167 +0,0 @@
|
|||
# Phase 4a — Connector taxonomy (Type-1/Type-2) + availability gating + MCP routing fix
|
||||
|
||||
> ## ↪ REDIRECT / DEMOTED (2026-06-30) — superseded by the revamp
|
||||
> **The canonical Phase 4 is now [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) + [`04b-access.md`](revamp%20phases%204-7/04b-access.md)** (typed capability verbs + generated doors — see `00-umbrella-plan.md`). This file's connector-taxonomy work is **demoted to backward-compat hygiene**: it is **not** the core of Phase 4. Read the revamp first.
|
||||
>
|
||||
> What survives from this doc, and where it goes:
|
||||
> - **The one genuinely useful piece — the `MCP_CONNECTOR` (BYO MCP) routing-gap fix — moves to `revamp/04b`** (the "consume-user-MCP" work). That's the actionable carry-over.
|
||||
> - **Keeping legacy branded natives `MIGRATING` (off) + existing indexed KB docs searchable, and Obsidian/Circleback `DISABLED`** — still true; a small hygiene task, no longer a headline phase.
|
||||
> - **`is_pipeline_eligible` is dropped** (no pipelines); **Type-1 = file/KB-input only** (Drive/Dropbox/OneDrive + uploads → input-only KB); **`WEBCRAWLER_CONNECTOR` is RETIRED** (native `scrape_webpage` tool, now a `web.scrape` capability verb).
|
||||
>
|
||||
> The static-registry design below is retained for reference, but do not treat "connector two-type restructure" as the Phase 4 deliverable — the deliverable is the **capability registry** in `revamp/04a`.
|
||||
|
||||
> Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`.
|
||||
> Sibling: `04b-source-discovery.md` (web-search repurposing). Precondition: Phases 1–2 (rename) live.
|
||||
|
||||
> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename shifted them.
|
||||
|
||||
## Objective
|
||||
|
||||
Give connectors a first-class **Type-1 (Data Source) / Type-2 (MCP Tool)** taxonomy and an **availability** state, then gate creation, indexing, chat-subagent build, and (forward-compat) pipeline-eligibility off it — without a DB migration. Also fix the long-standing bug where generic `MCP_CONNECTOR` tools are discovered but dropped.
|
||||
|
||||
The taxonomy is **fully determined by `connector_type`** (every Notion row is the same category), so it is modeled as a **static code registry**, not a per-row column (resolved decision — avoids a migration + backfill drift across ~20 connector-create sites). `is_indexable` is **kept** (it is an orthogonal per-row capability that still gates the real `/index` + periodic machinery — e.g. Notion is `is_indexable=True` yet is a Type-2 tool).
|
||||
|
||||
## Locked model (MVP)
|
||||
|
||||
| Bucket | `category` | `availability` | Members | Behaviour |
|
||||
|--------|-----------|----------------|---------|-----------|
|
||||
| Functional data sources | `DATA_SOURCE` | `AVAILABLE` | `WEBCRAWLER_CONNECTOR`, `GOOGLE_DRIVE_CONNECTOR`, `COMPOSIO_GOOGLE_DRIVE_CONNECTOR`, `ONEDRIVE_CONNECTOR`, `DROPBOX_CONNECTOR` | create OK, `/index`+periodic OK, **pipeline-eligible** |
|
||||
| Functional tool | `MCP_TOOL` | `AVAILABLE` | `MCP_CONNECTOR` (generic BYO MCP server) | create OK, acts in chat, **no pipelines** |
|
||||
| Deprecated branded (indexers) | `MCP_TOOL` | `MIGRATING` | `NOTION_CONNECTOR`, `GITHUB_CONNECTOR`, `CONFLUENCE_CONNECTOR`, `BOOKSTACK_CONNECTOR`, `ELASTICSEARCH_CONNECTOR` | block new create, **disable `/index`+periodic**; disable subagent **only for Notion/Confluence** (GitHub/BookStack/Elasticsearch have no subagent); keep existing rows + already-indexed KB docs searchable |
|
||||
| Deprecated branded (act-only) | `MCP_TOOL` | `MIGRATING` | `SLACK_CONNECTOR`, `TEAMS_CONNECTOR`, `LINEAR_CONNECTOR`, `JIRA_CONNECTOR`, `CLICKUP_CONNECTOR`, `AIRTABLE_CONNECTOR`, `DISCORD_CONNECTOR`, `GOOGLE_GMAIL_CONNECTOR`, `GOOGLE_CALENDAR_CONNECTOR`, `LUMA_CONNECTOR`, `COMPOSIO_GMAIL_CONNECTOR`, `COMPOSIO_GOOGLE_CALENDAR_CONNECTOR` | block new create, **disable subagent** (no KB data to keep) |
|
||||
| Disabled | `DATA_SOURCE` | `DISABLED` | `OBSIDIAN_CONNECTOR`, `CIRCLEBACK_CONNECTOR` | block new create + subagent; not MCP-bound (distinct from MIGRATING). Obsidian is self-hosted-only today |
|
||||
| Removed → `04b` | n/a | n/a | `SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API` | enum values dropped + repurposed in `04b`. **04a treats them as `HIDDEN`** (excluded from taxonomy) so the registry is total until `04b` removes them |
|
||||
|
||||
**"Migrating" UX:** existing `MIGRATING` rows remain in the DB and (for the indexers) their already-indexed documents stay searchable via the always-on `knowledge_base` subagent; only the connector's own management (re-index, periodic, live subagent) is turned off, with a "Migrating to MCP soon" status surfaced by the API.
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
### The enum + model
|
||||
|
||||
- `class SearchSourceConnectorType(StrEnum)` — `db.py:85–117` (all **30** values, incl. the 5 search APIs `:86–90`, `MCP_CONNECTOR:113`, Composio variants `:115–117`). The registry's totality test must cover all 30.
|
||||
- `class SearchSourceConnector` — `db.py:1829`. Columns: `connector_type` (`:1867`), `is_indexable` (`:1868`, `Boolean default False`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`).
|
||||
- Schema: `schemas/search_source_connector.py` — `is_indexable: bool` (`:16`), validator "periodic only if indexable" (`:42–45`), `Update.is_indexable` (`:66`).
|
||||
|
||||
### Where indexability is currently set (per connector, scattered)
|
||||
|
||||
Create routes hardcode `is_indexable` per type: `True` for Notion (`notion_add_connector_route.py:409`), Confluence (`:376`), Google Drive (`:420`), Dropbox (`:373`), OneDrive (`:380`); `False` for Slack/Teams/Discord/Gmail/Calendar/Linear/Jira/ClickUp/Airtable/Luma (their `*_add_connector_route.py`), MCP (`search_source_connectors_routes.py:2696`). Composio uses `INDEXABLE_TOOLKITS` (`composio_routes.py:289,400`, `composio_service.py:99`). `oauth_connector_base.py:59,598` defaults `True`. **This scatter is exactly why category lives in a registry, not duplicated at each create site.**
|
||||
|
||||
### The `/index` dispatch (the indexable set, today)
|
||||
|
||||
`index_connector_content()` — `search_source_connectors_routes.py:717` — is an `if/elif` chain on `connector_type` that dispatches to a Celery task for: Notion (`:850`), GitHub (`:861`), Confluence (`:872`), BookStack (`:885`), Google Drive (`:898`), OneDrive (`:948`), Dropbox (`:995`), Elasticsearch (`~:1050`), WebCrawler (`:1058`). **Movers** (Notion/GitHub/Confluence/BookStack/Elasticsearch) lose this branch; the file-based + WebCrawler branches stay.
|
||||
|
||||
### The chat subagent maps + the routing gap
|
||||
|
||||
- `multi_agent_chat/constants.py` (the chat-package root `constants.py`, **not** under `subagents/`) — `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (`:5–24`, connector_type → subagent name; **note GitHub/BookStack/Elasticsearch have NO entry** — they are index-only, no chat subagent) and `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`:26–44`, subagent → required tokens; `deliverables`/`knowledge_base` require `frozenset()` = always built). The required tokens are a **mix of connector types and doc types** (e.g. `notion`→`NOTION_CONNECTOR`, but `dropbox`→`DROPBOX_FILE`, `google_drive`→`GOOGLE_DRIVE_FILE`, `onedrive`→`ONEDRIVE_FILE`).
|
||||
- Subagent exclusion: `subagents/registry.py` `get_subagents_to_exclude(available_connectors)` (`:136–152`) excludes a builder in **two** cases: (a) it is **absent from** `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`required_tokens is None`, `:145–147`) → excluded; (b) it has non-empty required tokens that don't intersect the available set (`:150–151`) → excluded. An **empty** `frozenset()` (deliverables/knowledge_base) is always kept (`:148–149`). `build_subagents(..., exclude=...)` (`:182–220`) additionally hard-skips `memory`/`research` (`:195`) and the names in `exclude`, then calls each builder with `mcp_tools=mcp.get(name)` (`:209`). `SUBAGENT_BUILDERS_BY_NAME` (`:92–112`) has **no `mcp` builder** today. **Consequence for §5:** adding an `mcp` builder *without* a `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP["mcp"]` entry would hit case (a) and exclude it forever — so §5's map entry is mandatory, not optional.
|
||||
- **CRITICAL — `available_connectors` is NOT raw connector types.** `factory.py:102–105` computes `connector_types = get_available_connectors(...)` then `available_connectors = map_connectors_to_searchable_types(connector_types)`. This **shared, already-mapped** list of searchable-type tokens feeds *multiple* consumers: `get_subagents_to_exclude` (`factory.py:253`, `main_agent/middleware/stack.py:202`), the on-demand KB-search tool (`subagents/builtins/knowledge_base/tools/search_knowledge_base.py:_search_types:53–63` does `types.update(available_connectors)` at `:61–62` — this is the post-`main`-merge replacement for the now-deleted `shared/middleware/knowledge_search.py`, which used to do the same in `_resolve_search_types`), `web_search`'s live-connector filter (`shared/tools/web_search.py`), and the `deliverables` `report` tool. **Filtering this list would break legacy KB-doc searchability and web search** — see Target §4.
|
||||
- `map_connectors_to_searchable_types` (`connector_searchable_types.py:65–100`) maps each configured connector type to a searchable token; **`MCP_CONNECTOR` is absent from `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`)**, so MCP connectors never produce a token in `available_connectors`.
|
||||
- **The bug:** `subagents/mcp_tools/index.py` `partition_mcp_tools_by_connector` (`:55–99`) routes each MCP tool via `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS.get(connector_type)`; `MCP_CONNECTOR` has **no entry**, so it hits the `connector_agent is None` branch and is **skipped with a warning** (`:88–95`). Generic user MCP servers therefore contribute zero tools today. (The bucket key is the agent name, e.g. `"mcp"`, consumed by `build_subagents` via `mcp.get("mcp")`.)
|
||||
|
||||
### Query-time searchable mapping (keeps legacy docs visible)
|
||||
|
||||
`main_agent/runtime/connector_searchable_types.py` `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`, unchanged by the `main` merge) maps connector types → searchable doc-types that scope the KB-search tool (via `available_connectors` → `search_knowledge_base.py:_search_types`). The MIGRATING indexers' entries (NOTION→`NOTION_CONNECTOR`, etc.) **stay** so already-indexed docs remain searchable. (WebCrawler→`CRAWLED_URL`, file connectors→`*_FILE` stay too.)
|
||||
|
||||
## Target design
|
||||
|
||||
### 1. The static registry (single source of truth)
|
||||
|
||||
New `app/connectors/connector_registry.py` (or `app/utils/connector_registry.py`):
|
||||
|
||||
```python
|
||||
class ConnectorCategory(StrEnum):
|
||||
DATA_SOURCE = "DATA_SOURCE"
|
||||
MCP_TOOL = "MCP_TOOL"
|
||||
|
||||
class ConnectorAvailability(StrEnum):
|
||||
AVAILABLE = "AVAILABLE" # usable now
|
||||
MIGRATING = "MIGRATING" # branded native, "moving to MCP soon"
|
||||
DISABLED = "DISABLED" # off for MVP, not MCP-bound (Obsidian/Circleback)
|
||||
HIDDEN = "HIDDEN" # not a real connector in the taxonomy (search APIs, until 04b removes them)
|
||||
|
||||
# connector_type -> (category, availability). Total over SearchSourceConnectorType.
|
||||
CONNECTOR_REGISTRY: dict[SearchSourceConnectorType, tuple[ConnectorCategory, ConnectorAvailability]] = { ... }
|
||||
|
||||
def get_category(ct) -> ConnectorCategory | None
|
||||
def get_availability(ct) -> ConnectorAvailability
|
||||
def is_creatable(ct) -> bool # availability == AVAILABLE
|
||||
def is_pipeline_eligible(ct) -> bool # category == DATA_SOURCE and availability == AVAILABLE
|
||||
def is_indexable_type(ct) -> bool # the file/web data sources that own a Celery indexer
|
||||
```
|
||||
|
||||
- **Totality guard (test):** assert every `SearchSourceConnectorType` member has a registry entry, so a newly added connector can't silently fall through gating.
|
||||
- Exposed on the connector read schema as **computed fields** (`category`, `availability`) so the API/frontend can label and filter without a DB column. (Promote to a column later only if Zero/SQL filtering needs it — see umbrella resolved log.)
|
||||
|
||||
### 2. Create gating
|
||||
|
||||
In the generic create handler `create_search_source_connector` (`search_source_connectors_routes.py:172`, `@router.post("/search-source-connectors")`) **and** the per-service add/OAuth routes (e.g. `notion_add_connector_route.py`, `luma_add_connector_route.py:51`, the Composio routes) reject when `not is_creatable(connector_type)` with a clear 4xx ("This connector is migrating to MCP and can't be added yet" / "disabled for MVP"). The existing per-type duplicate check (`:198–212`) is unaffected. Because the per-service routes each build their own connector, the gate must be applied in each (or in a shared helper they all call) — verify by grepping the `*_add_connector_route.py` set. This is the behavioural change that blocks new branded connectors; existing rows are untouched.
|
||||
|
||||
### 3. Index gating
|
||||
|
||||
Guard `index_connector_content` (`:717`) up front: if `not is_indexable_type(connector_type)`, return a 4xx/no-op ("indexing disabled — migrating to MCP"). This neutralizes the Notion/GitHub/Confluence/BookStack/Elasticsearch branches without deleting their (now-dead) code paths.
|
||||
|
||||
**Periodic path (must also be gated).** Create-gating blocks *new* MIGRATING rows, but **existing** MIGRATING connectors with `periodic_indexing_enabled=True` already have a `next_scheduled_at` and would keep firing via the meta-scheduler Beat task that polls `next_scheduled_at` every minute: `tasks/celery_tasks/schedule_checker_task.py` (due query `next_scheduled_at <= now` `:36`, dispatch loop `:88`, `task = task_map.get(connector.connector_type)` then `.delay(...)` `:118`). The first-run trigger is `create_periodic_schedule` (`utils/periodic_scheduler.py:30`, `task.delay(...)` `:97`). **Both paths re-dispatch through the same per-type Celery tasks** (`index_crawled_urls_task`, `index_notion_pages_task`, …), so the robust single chokepoint is to **gate the Celery index tasks at entry** by `is_indexable_type(connector_type)` — this covers the manual `/index` route, the first-run trigger, and the recurring meta-scheduler pass at once (a no-op early-return for MIGRATING types). (Aside: `schedule_checker_task` already auto-disables a `LIVE_CONNECTOR_TYPES` set at `:74-77` — an existing precedent for turning periodic off per type.) The schema validator "periodic only if indexable" (`schemas/search_source_connector.py:42–45`) is unchanged and unaffected (it keys off the per-row `is_indexable` bool, not the registry).
|
||||
|
||||
Note: **update** (`PUT /search-source-connectors/{id}`, `search_source_connectors_routes.py:371`) is intentionally **left allowed** for MIGRATING rows so existing users can still edit/disable them; only create + index are gated.
|
||||
|
||||
### 4. Subagent gating (turns off branded chat tools) — by NAME, not by token
|
||||
|
||||
> **Do NOT filter `available_connectors`.** It is a shared, already-mapped searchable-type list (see Current State). Stripping MIGRATING entries from it would (a) remove `NOTION_CONNECTOR`/etc. from the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`) → **break the very legacy-doc searchability this plan promises**, and (b) remove the live-search tokens → break `web_search` before 04b. The token list must stay intact.
|
||||
|
||||
Instead, **exclude the deprecated subagents by NAME**, derived from the registry. Extend `get_subagents_to_exclude` (`registry.py:136`) so that, in addition to its current token check, it also excludes any subagent whose mapped connector type(s) are all non-`AVAILABLE`:
|
||||
|
||||
- Build the reverse of `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (subagent_name → set of connector types). For each `SUBAGENT_BUILDERS_BY_NAME` entry, if it has connector type(s) and **every** one is non-`AVAILABLE` in the registry, add it to the excluded set unconditionally (regardless of tokens).
|
||||
- This is the single chokepoint used by both `factory.py:253` and `stack.py:202`, and feeds `main_prompt_registry_subagent_lines(exclude)`, so the deprecated specialists disappear from both the build and the prompt.
|
||||
|
||||
Net effect for MVP:
|
||||
|
||||
- **Excluded (MIGRATING):** `notion`, `confluence`, `slack`, `teams`, `linear`, `jira`, `clickup`, `airtable`, `discord`, `gmail`, `calendar`, `luma`. (GitHub/BookStack/Elasticsearch have no subagent — they are handled purely by §3 index gating; their indexed docs stay searchable via the untouched token list.)
|
||||
- **Kept (AVAILABLE):** `knowledge_base`, `deliverables`, `research`, `memory` (builtins), the new `mcp` subagent, **and the file-source specialists `google_drive`, `dropbox`, `onedrive`** — these map to `AVAILABLE` Type-1 connectors, so by the registry rule they are NOT deprecation-excluded; they remain token-gated as today (built only when that file connector is configured). **Decision:** keep them — a Type-1 data source can still own a chat specialist over its indexed files; the two-type split governs pipeline-eligibility + the MCP migration, not whether an AVAILABLE connector may have a subagent. (If we later want pure-ingestion file connectors with no specialist, flip them by adding a `subagent_chat` flag to the registry — out of scope here.)
|
||||
|
||||
### 5. MCP_CONNECTOR routing-gap fix (makes generic Type-2 work)
|
||||
|
||||
Two changes are needed — the routing map AND surfacing the token, or the subagent will still never build:
|
||||
|
||||
1. **Routing map** (`multi_agent_chat/constants.py`): add `"MCP_CONNECTOR": "mcp"` to `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (so `partition_mcp_tools_by_connector` stops hitting the `connector_agent is None` skip at `mcp_tools/index.py:88–95`), and `"mcp": frozenset({"MCP_CONNECTOR"})` to `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP`.
|
||||
2. **Builder**: add a generic `mcp` entry to `SUBAGENT_BUILDERS_BY_NAME` (`registry.py:92`); `build_subagents` will call it with `mcp_tools=mcp.get("mcp")` (the bucket key from step 1's map value).
|
||||
3. **Surface the token** (the gap step 4's predecessor missed): `MCP_CONNECTOR` is **not** in `_CONNECTOR_TYPE_TO_SEARCHABLE`, so `available_connectors` never contains it and the `mcp` subagent's required token `{MCP_CONNECTOR}` would never intersect → it would be excluded forever. Fix by adding `"MCP_CONNECTOR": "MCP_CONNECTOR"` to `_CONNECTOR_TYPE_TO_SEARCHABLE` (`connector_searchable_types.py:22`). Side effect: `MCP_CONNECTOR` joins the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`), where it matches zero indexed docs — harmless no-op. (Alternative if the no-op is undesirable: special-case `mcp` in `get_subagents_to_exclude` to include it whenever any configured connector is an `AVAILABLE` `MCP_TOOL`, passing the raw `connector_types` alongside. The searchable-map entry is simpler and preferred.)
|
||||
|
||||
### 6. Keep `is_indexable` semantics intact
|
||||
|
||||
No change to the column or its validator. `is_indexable` continues to gate the real `/index`+periodic machinery for the data sources; the registry's `is_indexable_type` is a **type-level** allowlist layered on top (a row must be both an indexable type AND `is_indexable=True`). This avoids reinterpreting historical rows.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Registry module** + `ConnectorCategory`/`ConnectorAvailability` enums + the total `CONNECTOR_REGISTRY` map + helper predicates (incl. reverse subagent-name→types helper for §4).
|
||||
2. **Schema**: add computed `category`/`availability` to the connector read schema (`schemas/search_source_connector.py`).
|
||||
3. **Create gating** in the generic create handler (`:172`) + every per-service add route (reject non-`AVAILABLE`).
|
||||
4. **Index gating** at the Celery index-task entry (`is_indexable_type`) — covers manual `/index`, first-run trigger, and meta-scheduler.
|
||||
5. **Subagent gating by name**: extend `get_subagents_to_exclude` to also exclude subagents whose mapped connector types are all non-`AVAILABLE` (do **not** filter the shared `available_connectors` token list).
|
||||
6. **MCP routing-gap fix**: `multi_agent_chat/constants.py` map entries + generic `mcp` builder in `SUBAGENT_BUILDERS_BY_NAME` + `MCP_CONNECTOR` token in `_CONNECTOR_TYPE_TO_SEARCHABLE` (so the subagent actually builds).
|
||||
7. **Tests** (below).
|
||||
|
||||
## Tests
|
||||
|
||||
- **Registry totality**: every `SearchSourceConnectorType` has an entry (guards future additions).
|
||||
- **Create gating**: creating any `MIGRATING`/`DISABLED` type → 4xx; `WEBCRAWLER`/file/`MCP_CONNECTOR` → OK; MCP multi-instance still works.
|
||||
- **Index gating**: index task entry for Notion/GitHub/Confluence/BookStack/Elasticsearch → no-op; WebCrawler/GDrive/OneDrive/Dropbox → runs. Include a case for an **existing** MIGRATING row with `periodic_indexing_enabled=True` + due `next_scheduled_at` → meta-scheduler triggers a no-op (does not re-index).
|
||||
- **Subagent gating by name**: with a (legacy) Notion + Slack connector configured, `get_subagents_to_exclude` excludes `notion`/`slack`; `knowledge_base`/`deliverables` always built; a configured `google_drive`/`dropbox`/`onedrive` (AVAILABLE) is **not** excluded (still token-gated).
|
||||
- **No collateral damage to the token list**: gating subagents must NOT change `available_connectors` — assert the KB-search tool still resolves the `NOTION_CONNECTOR` doc type (via `_search_types`) for the legacy connector (guards against the "filter the shared list" regression).
|
||||
- **MCP gap (two-part)**: (a) a configured `MCP_CONNECTOR` produces a `"mcp"` token in `map_connectors_to_searchable_types`; (b) its tools land in the `mcp` bucket and the `mcp` subagent is actually built (regression for both `mcp_tools/index.py:88–95` and the missing-token exclusion).
|
||||
- **Legacy searchability**: a search space with an existing (now-MIGRATING) Notion connector + indexed docs still returns those docs via the KB-search tool (`connector_searchable_types` Notion entry unchanged).
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Breaks current branded chat tooling (accepted).** Deprecating act-only natives removes their live chat subagents for existing users mid-pivot. Per the umbrella's "DB migrations carry users; backend behaviour can change" posture, this is acceptable; the "migrating to MCP soon" status sets expectations.
|
||||
- **Dead code paths left in place.** The Notion/GitHub/Confluence/BookStack/Elasticsearch `/index` branches + indexers remain but are gated off (not deleted) to keep the diff small and reversible; delete when the real MCP migration lands.
|
||||
- **No column = not SQL/Zero-filterable.** Frontend (deferred) filters via the API's computed fields. If Zero needs server-side filtering later, promoting the registry to a denormalized column is additive.
|
||||
- **Registry/`is_indexable` dual-gate.** Two truths (type-level allowlist + per-row bool) must agree; the totality test + index-gating test cover the seam.
|
||||
- **Shared `available_connectors` is a footgun.** It feeds subagent-exclusion, the KB-search tool's doc-type scope, web_search, and the deliverables `report` tool simultaneously. Subagent gating is therefore done by *name* (registry-driven), never by mutating that list; the "no collateral damage" test pins this. This also removes any 04a→04b sequencing hazard (04a no longer touches the live-search tokens that 04b later retires).
|
||||
- **File-source specialists kept.** `google_drive`/`dropbox`/`onedrive` stay as AVAILABLE chat specialists (decision in §4). If product wants pure-ingestion file connectors, that's an additive registry flag later.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- Search-API enum removal, key relocation, and the source-discovery endpoint → `04b`.
|
||||
- Pipeline tables + actually creating pipelines from Type-1 connectors → Phases 5–7 (`is_pipeline_eligible` is defined here for them to consume).
|
||||
- Real MCP migration of the branded connectors (re-point to MCP servers) → post-MVP.
|
||||
- Frontend connector UI restructure → frontend umbrella.
|
||||
|
|
@ -1,120 +0,0 @@
|
|||
# Phase 4b — Web-search repurposing + source-discovery endpoint
|
||||
|
||||
> ## ↪ REDIRECT / ABSORBED (2026-06-30) — superseded by the revamp
|
||||
> **This subplan is absorbed into the canonical [`revamp phases 4-7/04a-capabilities.md`](revamp%20phases%204-7/04a-capabilities.md) as the `web.discover` verb.** The substance still holds — drop the 5 search `connector_type` values (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) + their paths (destructive migration), move SearXNG/Linkup/Baidu to **platform env** config, and rewire the chat `web_search` tool onto them — but the "source-discovery endpoint" is now a **capability verb** (`web.discover(query, top_k)`) in the registry, exposed identically via chat/REST/MCP (not a bespoke route), and its output feeds the agent (or a Tracker binding), **not** "a pipeline." Read `revamp/04a` (`web.discover`) as canonical; use the provider/migration detail here as implementation reference.
|
||||
>
|
||||
> Part of **Phase 4 — Capabilities & Access (backend)**. See `00-umbrella-plan.md`.
|
||||
> Sibling (old): `04a-connector-category.md` (demoted to hygiene). Precondition: Phases 1–2 (rename) live.
|
||||
|
||||
> **Implementation note.** Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name; locate code by **symbol/grep**, not the absolute line numbers cited.
|
||||
|
||||
## Objective
|
||||
|
||||
Retire the five web-search **connector types** and re-cast the survivors as **platform-level providers** that power two things: (1) the existing chat `web_search` tool, and (2) a new **source-discovery endpoint** that, given a topic/competitor, suggests candidate URLs the user can feed into the Universal WebURL Crawler / a pipeline.
|
||||
|
||||
Resolved decisions driving this:
|
||||
|
||||
- **Drop all 5 search `connector_type` enum values** (`SERPER_API`, `TAVILY_API`, `SEARXNG_API`, `LINKUP_API`, `BAIDU_SEARCH_API`).
|
||||
- **Tavily + Serper removed entirely** (no provider, no code path).
|
||||
- **SearXNG / Linkup / Baidu survive as platform providers.** SearXNG is already platform/env-based; **Linkup + Baidu move from per-workspace connector `config` to platform/env config** (one app-wide key set, matching the "single provider app-wide" style from `03b`).
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
### The 5 search types are first-class connectors today
|
||||
|
||||
- Enum members `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` — `db.py:86–90` (`SERPER_API` is annotated "NOT IMPLEMENTED YET"; SearXNG is already platform-backed despite being an enum value).
|
||||
- Per-workspace search methods on `ConnectorService` read keys from the connector row's `config`: `search_tavily` (`connector_service.py:481`, key `TAVILY_API_KEY:510`), `search_searxng` (`:587`), `search_baidu` (`:614`), `search_linkup` (`:1968`, key `LINKUP_API_KEY:2000`).
|
||||
- Query-time routing: `connector_searchable_types.py:_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–26`) maps `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` to the web_search tool ("live search connectors"); the other connector types map to KB pre-search.
|
||||
|
||||
### The web_search tool already has a platform/per-workspace split
|
||||
|
||||
> **Post-`main`-merge note.** A `main` sync **consolidated web_search to a single tool**: the old research-subagent copy `subagents/builtins/research/tools/web_search.py` was **deleted**, and there is now **one** factory at `app/agents/chat/shared/tools/web_search.py`. The research subagent now imports it (`subagents/builtins/research/tools/index.py:10,24–26`). So everywhere below that said "both variants / mirror" is now **one file**. The same merge also retired the XML result blob in favor of `[n]`-citation rendering (see §3).
|
||||
|
||||
`web_search` tool factory `create_web_search_tool(search_space_id, available_connectors)` — the single canonical tool at `app/agents/chat/shared/tools/web_search.py`:
|
||||
- `_LIVE_SEARCH_CONNECTORS = {TAVILY_API, LINKUP_API, BAIDU_SEARCH_API}` (`:41–45`), `_LIVE_CONNECTOR_SPECS` → `ConnectorService.search_*` (`:47–51`), per-connector dispatch `_search_live_connector` (`:115–151`). The factory picks the active live connectors by intersecting `available_connectors` with `_LIVE_SEARCH_CONNECTORS` (`:163–167`).
|
||||
- At call time it fans out **in parallel** to platform SearXNG (`web_search_service.is_available()` / `.search()`, `:204–214`) **plus** each active per-workspace live connector (`:216–228`), then dedupes by URL (`:242–250`).
|
||||
- Platform SearXNG service: `app/services/web_search_service.py` — env-gated by `config.SEARXNG_DEFAULT_HOST`, in-process circuit breaker + Redis result cache; returns the `(result_obj, documents)` shape the tool consumes.
|
||||
|
||||
### Config
|
||||
|
||||
- `config.SEARXNG_DEFAULT_HOST = os.getenv("SEARXNG_DEFAULT_HOST")` — `config/__init__.py:558`. There are **no** platform env vars for Linkup/Baidu/Tavily today (their keys live in connector `config`).
|
||||
|
||||
## Target design
|
||||
|
||||
### 1. Platform search-provider config (env)
|
||||
|
||||
Add platform env knobs alongside `SEARXNG_DEFAULT_HOST` (`config/__init__.py:~558`):
|
||||
|
||||
- `LINKUP_API_KEY = os.getenv("LINKUP_API_KEY")`
|
||||
- `BAIDU_SEARCH_API_KEY = os.getenv("BAIDU_SEARCH_API_KEY")` (+ the optional knobs `search_baidu` reads from connector `config` today — `BAIDU_MODEL` / `BAIDU_SEARCH_SOURCE` / `BAIDU_ENABLE_DEEP_SEARCH`, per `validators.py:507-514`; the required key is `BAIDU_API_KEY` in the per-workspace config, relocated to this platform env var — port from its current `config` reads).
|
||||
- (SearXNG unchanged.) Document all three in `.env.example` as optional; each provider self-disables when its key/host is unset (mirrors `web_search_service.is_available()`).
|
||||
|
||||
No Tavily/Serper env (removed).
|
||||
|
||||
### 2. A platform search-providers service (consolidate the survivors)
|
||||
|
||||
Introduce `app/services/web_search_service.py`-level functions (or a small `search_providers/` package) so all three survivors share one platform shape `(result_obj, documents)`:
|
||||
|
||||
- `searxng.search(...)` — the existing `web_search_service.search` (unchanged).
|
||||
- `linkup.search(...)` / `baidu.search(...)` — **port the bodies of `ConnectorService.search_linkup`/`search_baidu`** but source the API key from `config.*` (platform) instead of `connector.config` (per-workspace), and drop the `search_space_id`/connector lookups.
|
||||
- Each is `is_available()`-gated on its env key. Tavily (`search_tavily`) is deleted.
|
||||
|
||||
This removes per-workspace coupling: web search + discovery no longer depend on any connector row.
|
||||
|
||||
### 3. Rewire the chat `web_search` tool (single tool)
|
||||
|
||||
- Drop the `_LIVE_SEARCH_CONNECTORS` / `_LIVE_CONNECTOR_SPECS` / `_search_live_connector` mechanism in the **single** tool `shared/tools/web_search.py` (`:41–51`, `:115–151`, and the factory's live-connector intersection `:163–167`, plus the per-workspace fan-out `:216–228`). Instead, fan out to the **platform providers** that are `is_available()` (SearXNG + Linkup + Baidu), all keyless from the caller's view.
|
||||
- Keep the existing parallel-gather + URL-dedupe (`:233–250`). **Note:** the `main` merge replaced the old XML result blob with `[n]`-citation rendering — the tool now builds `RenderableDocument`s (`_to_renderable_web_documents:66–112`) and returns them via `render_web_results` + `load_registry`, persisting the `citation_registry` on state (`:252–281`). Leave that render/registry path **unchanged**; only the *provider sourcing* (SearXNG+Linkup+Baidu, env-keyed) changes.
|
||||
- **Call-site churn — keep the `available_connectors` parameter in the factory signature but stop using it for provider selection** (providers are now env-derived). This avoids touching every caller: `main_agent/tools/registry.py:24,39`, `subagents/builtins/research/tools/index.py:10,24–26`, and `anonymous_chat/agent.py:127` (passes `available_connectors=None` already). **`deliverables` is no longer a web_search caller** (its KB/web tooling was removed in the `main` merge) — drop it from the churn list. `search_space_id` likewise stays for logging only. (A later cleanup can remove the now-dead params.)
|
||||
- Net behaviour: web search works platform-wide whenever any provider env key is set; it no longer requires the workspace to have a search connector configured (so anonymous chat gains web search too).
|
||||
|
||||
### 4. Source-discovery endpoint (the new capability)
|
||||
|
||||
New route (e.g. `routes/source_discovery_routes.py`, mounted under `/api/v1/workspaces/{workspace_id}/...` to match the renamed surface): `POST .../source-discovery` taking `{ query/topic, top_k }` and returning a **ranked list of candidate URLs** (url, title, snippet, provider) for the user to add to the WebURL Crawler / a pipeline.
|
||||
|
||||
- Implementation = call the platform providers (§2) in parallel, dedupe by URL (reuse the tool's dedupe), and return URL-centric results (plain `{url,title,snippet,provider}` — **not** the chat tool's `[n]`-cited `render_web_results` output, which is for in-conversation citation, not URL suggestion).
|
||||
- Auth: standard workspace access check (same dependency as other workspace routes).
|
||||
- This is **backend-only**; the UX (a "find sources" affordance when configuring a crawler/pipeline) is deferred to the frontend umbrella.
|
||||
- Optional thin reuse: the chat `web_search` `_web_search_impl` and this endpoint can share a `discover_urls(query, top_k) -> list[UrlCandidate]` core.
|
||||
|
||||
### 5. Drop the 5 connector types
|
||||
|
||||
- **Remove the enum members** `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` from `SearchSourceConnectorType` (`db.py:86–90`).
|
||||
- **Remove their connector code paths**: `ConnectorService.search_tavily` (`:481`, delete) and the per-workspace `search_searxng` (`:587`)/`search_baidu` (`:614`)/`search_linkup` (`:1968`) (replace with the platform service §2; delete the connector-config variants that read `connector.config[...]_API_KEY`, e.g. `:510`, `:2000`). Remove the three search entries from `connector_searchable_types.py:24–26`. Remove the `*_API` entries from the connector-config validation dict (`utils/validators.py:490–514`, the 5 keys in `connector_rules`: `SERPER_API:490`/`TAVILY_API:491`/`SEARXNG_API:492-505`/`LINKUP_API:506`/`BAIDU_SEARCH_API:507-514`) and `04a`'s `HIDDEN` registry entries for these types. **Grep-guard:** no remaining references to the 5 enum names or `search_tavily` anywhere.
|
||||
- **Migration (this phase is NOT migration-free):** existing connector rows of these 5 types must be handled before/with the enum change:
|
||||
- Alembic migration: `DELETE FROM search_source_connectors WHERE connector_type IN ('SERPER_API','TAVILY_API','SEARXNG_API','LINKUP_API','BAIDU_SEARCH_API');` (FKs cascade; these rows only held search API keys, now relocated to env). Self-hosted operators re-add Linkup/Baidu keys via env — call out in the migration docstring + `.env.example`.
|
||||
- **PG enum handling:** dropping a value from a Postgres enum type requires a type recreate. Lowest-risk option: after deleting the rows, leave the now-unused labels in the PG enum type (harmless orphans) and just remove them from the Python `StrEnum` so no new rows can use them. If a clean type is wanted, do the standard "create new enum → `ALTER COLUMN ... TYPE` with a `USING` cast → drop old" dance in the migration (heavier; only if desired). Recommend the orphan-label approach for MVP.
|
||||
- Also scrub `document.document_type` / search rows that referenced `TAVILY_API`/`LINKUP_API`/`BAIDU_SEARCH_API` as a **doc type** only if such persisted docs exist (live-search results were ephemeral, not indexed — verify; expectation is none).
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Config**: `LINKUP_API_KEY`, `BAIDU_SEARCH_API_KEY` (+ Baidu host/region) in `Config` + `.env.example`; keep `SEARXNG_DEFAULT_HOST`.
|
||||
2. **Platform providers**: port Linkup/Baidu search to env-keyed platform functions sharing SearXNG's `(result_obj, documents)` shape; delete Tavily.
|
||||
3. **Rewire `web_search`** (the single shared tool `shared/tools/web_search.py`) to fan out to platform providers; drop `available_connectors`/live-connector plumbing; keep dedupe + the `[n]`-citation render path.
|
||||
4. **Source-discovery endpoint**: `POST .../source-discovery` → ranked URL candidates; shared `discover_urls()` core.
|
||||
5. **Drop the 5 enum values** + remove `search_*` connector methods, `connector_searchable_types` search entries, validators, and `04a` HIDDEN entries.
|
||||
6. **Migration**: delete the 5 connector types' rows; handle the PG enum (orphan-label approach); docstring notes the env re-keying.
|
||||
7. **Tests** (below).
|
||||
|
||||
## Tests
|
||||
|
||||
- **Provider availability**: each of SearXNG/Linkup/Baidu self-disables when its env key/host is unset; `web_search` returns "not available" only when all are unset.
|
||||
- **web_search rewire**: with `LINKUP_API_KEY` set (no search connector rows), `web_search` still returns Linkup results; results from multiple providers dedupe by URL.
|
||||
- **No per-workspace dependency**: a workspace with zero connectors still gets web search + source-discovery results.
|
||||
- **Source-discovery endpoint**: returns URL candidates `{url,title,snippet,provider}`; enforces workspace access; empty when no providers configured.
|
||||
- **Enum removal**: `SearchSourceConnectorType` no longer has the 5 values; creating one → 422; the migration deletes existing rows; reading remaining connectors is unaffected.
|
||||
- **Tavily gone**: no import/path references `search_tavily`/`TAVILY_API` remain (grep guard).
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Per-workspace search keys become platform-wide (accepted).** Workspaces can no longer bring their own Linkup/Baidu keys; one app-wide key set serves everyone (matches the single-provider posture). Self-hosted must move keys to env — surfaced in migration docs.
|
||||
- **Destructive migration.** Deleting the 5 connector types' rows is irreversible (downgrade can recreate rows but not their secrets). Gate behind the standard backup/runbook; the data is just API-key config now living in env.
|
||||
- **PG enum orphan labels.** Leaving unused enum labels is cosmetically untidy but avoids a risky type-recreate; documented as a deliberate trade-off.
|
||||
- **Sequencing with 04a.** 04a marks the 5 types `HIDDEN` so the taxonomy stays total in the interim; 04b removes them. If 04b ships first, 04a's HIDDEN bucket is simply never populated — both orders are safe, but `04a → 04b` is the intended order.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- Taxonomy/gating/MCP-routing fix → `04a`.
|
||||
- Using discovered URLs to actually create a pipeline/crawl → Phases 5–7 (this endpoint only *suggests* URLs).
|
||||
- Source-discovery UX (the "find sources" affordance) → frontend umbrella.
|
||||
- Crawl billing for any crawl the user starts from a discovered URL → already covered by `03c` (the crawl path bills regardless of how the URL was found).
|
||||
85
plans/backend/05-access.md
Normal file
85
plans/backend/05-access.md
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
# Phase 5 — Access / Surfaces (REST · MCP · chat doors)
|
||||
|
||||
> Build after `04`. Together `04 + 05` ship the scraper-API product.
|
||||
> Reuses shipped infra: Identity/Tenancy, API keys, the chat agent + tool registry, the streaming layer,
|
||||
> Metering (`03c`), and Rocicorp Zero reactive sync. Locate code by symbol/grep.
|
||||
|
||||
## Objective
|
||||
|
||||
Expose the `04` capability registry through three doors — REST + API keys, an MCP server, and chat tools —
|
||||
each generated from the one registry so the I/O contract can't drift. Every door is the same thin adapter
|
||||
with no business logic.
|
||||
|
||||
## Adapter shape (identical on every door)
|
||||
|
||||
```
|
||||
parse input → validate against verb.input_schema → authn/authz → meter-gate (03c)
|
||||
→ call the SAME executor (04) → serialize verb.output_schema → return cleaned data
|
||||
```
|
||||
|
||||
The executor returns directly (`04`).
|
||||
|
||||
## The three doors
|
||||
|
||||
| Door | Who | Auth | Status |
|
||||
|------|-----|------|--------|
|
||||
| REST + API keys | external developers | existing API-key infra | public day one |
|
||||
| MCP server | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 or bearer (at implementation) | fast-follow |
|
||||
| Chat tools | the in-app agent (`07`) | existing session + workspace | seed exists (`scrape_webpage`) |
|
||||
|
||||
Routes are generated, so public REST on day one costs nothing extra.
|
||||
|
||||
## Chat surface
|
||||
|
||||
The in-app product is the conversation: the user describes a need in plain language and the agent (`07`)
|
||||
picks and fills verbs. Raw typed verbs are exposed on REST/MCP for devs and external agents. The chat
|
||||
door classifies intent and routes:
|
||||
|
||||
```
|
||||
"find / compare / what is / pull / right now" → ONE-SHOT → call verbs, answer now
|
||||
"watch / track / notify me / keep an eye / over time" → KEEP-WATCHING → hand to the ongoing mode (06)
|
||||
```
|
||||
|
||||
## MCP — two directions
|
||||
|
||||
- **Serve MCP** (new door): our capabilities as a remote MCP server — Streamable-HTTP `/mcp`, stateless,
|
||||
bounded/paginated outputs, read-only verbs (least privilege).
|
||||
- **Consume MCP** (reuse): the BYO-`MCP_CONNECTOR` — the user's own external MCP tools inside our chat
|
||||
agent. Fix its routing-map gap here.
|
||||
|
||||
## Result delivery
|
||||
|
||||
A chat tool call returns its data inline in the turn. Asynchronous results (e.g. the `06` ongoing mode
|
||||
writing between turns) reach the client via write-then-sync / stream: the worker writes rows and the
|
||||
client receives them over the existing SSE stream or a Zero-published table.
|
||||
|
||||
## Work items
|
||||
|
||||
1. Door generator: from the `04` registry emit (a) REST routes + models, (b) MCP tool schemas + handlers,
|
||||
(c) chat tool defs + handlers.
|
||||
2. REST surface: public routes + API-key auth (reuse) + `03c` meter-gate.
|
||||
3. Chat tools: generalize `scrape_webpage` into the registry-backed set (direct-return).
|
||||
4. Intent routing: classify one-shot vs keep-watching; hand keep-watching to `06`.
|
||||
5. MCP server (fast-follow): Streamable-HTTP `/mcp`, least-privilege; auth depth at implementation.
|
||||
6. Consume-MCP fix: repair the BYO-`MCP_CONNECTOR` routing-map gap.
|
||||
|
||||
## Tests
|
||||
|
||||
- A verb added to the registry appears on REST + MCP + chat with identical I/O.
|
||||
- REST without a valid key → 401; an over-budget call → blocked by the `03c` gate; a success charges once.
|
||||
- A chat scrape returns cleaned data inline in the same turn.
|
||||
- "compare X and Y" → one-shot; "watch X weekly" → routed to `06`.
|
||||
- A configured BYO MCP tool is reachable by the chat agent.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Verbs, executors, billing units → `04`.
|
||||
- Keep-watching mechanism + its delivery channel → `06`.
|
||||
- The CI subagent + its prompt → `07`.
|
||||
- Legacy branded connectors (backward-compat hygiene, separate task).
|
||||
|
||||
## Open questions
|
||||
|
||||
- MCP auth depth (decide at implementation).
|
||||
- Public REST rate-limiting / abuse posture.
|
||||
- Async delivery channel for `06` outputs: SSE stream vs Zero-published messages table (settled with `06`).
|
||||
|
|
@ -1,321 +0,0 @@
|
|||
# Phase 5 — Pipelines data model (tables, schemas, migration, Zero, CRUD + run routes)
|
||||
|
||||
> # ❌ SUPERSEDED (2026-06-30)
|
||||
> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, the CI pivot no longer uses a Pipelines concept: the KB is **input-only** (user files), the crawler/search are **platform-native tools**, connectors are **MCP**, and the recurring-fetch + run-history need is met by the existing **automations** system (`app/automations/`). There are **no `pipelines`/`pipeline_runs` tables**. This document is retained **for historical context only** — do **not** implement it. **The replacement is the canonical revamp Phase 5 — [`revamp phases 4-7/05a-timeline.md`](revamp%20phases%204-7/05a-timeline.md) + [`05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: the `Tracker` primitive + a 3-store delta **Timeline** (the durable CI state / moat). "Diffable run history" lives in the Timeline's `entity_changes`, **not** in `automation_runs.output`. (The earlier "Phase 5′" framing is retired.)
|
||||
|
||||
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 5).
|
||||
> Precondition: Phases 1–2 (rename) live, Phase 4a (connector registry) live. Siblings ahead: `06-pipelines-exec.md` (run engine + scheduling), `07-upload-pipeline-kb.md` (uploads-as-pipeline).
|
||||
|
||||
> **Implementation note (post-rename).** This phase adds **brand-new** tables, created *after* the rename, so they use the canonical names natively: table `workspaces`, column `workspace_id`, ORM class `Workspace`. Phases 1–2 are now **SHIPPED**, so the live code uses the `workspace_*` names. Existing code cited below that still shows the **pre-rename** names (`searchspaces`/`search_space_id`/`SearchSpace`) was captured pre-rename — substitute the `workspace_*` equivalent and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phase 3/4 migrations shift them).
|
||||
|
||||
## Objective
|
||||
|
||||
Introduce two first-class tables — `pipelines` and `pipeline_runs` — plus their enums, Pydantic schemas, one Alembic migration, the backend Zero-publication entries, and the HTTP routes (CRUD + manual-run trigger + list/get runs). **This phase is data-model + API surface only.** The actual run **engine** (invoke connector fetch → optional KB save → write run record), scheduling tick, crawl-billing wiring, and chat-agent context exposure are **Phase 6**; wiring file upload onto an "Uploads" pipeline is **Phase 7**. The manual-run endpoint here **creates a `PipelineRun` row and enqueues the Phase-6 task** (which is a stub until Phase 6 lands), so the surface is testable end-to-end without the engine.
|
||||
|
||||
A **Pipeline** = a saved, runnable fetch over a Type-1 data source: it references a connector + per-pipeline config + an optional cron schedule + a KB destination (`save_to_kb` + `destination_folder_id`). A **PipelineRun** = one immutable execution record (manual / scheduled / upload), carrying status, timing, doc counts, error, an optional raw-result blob ref, and a billing idempotency field.
|
||||
|
||||
## Locked model (MVP)
|
||||
|
||||
| Concept | Decision |
|
||||
|---------|----------|
|
||||
| Tables | `pipelines` (mutable definition) + `pipeline_runs` (append-only execution record). Mirrors the `automations` / `automation_runs` split. |
|
||||
| Where the ORM lives | In **`app/db.py`** alongside `Folder`/`Document`/`SearchSourceConnector`/`Log` (the other core, Zero-published, cross-referenced entities), **not** a separate package. Rationale: needs `back_populates` on `Workspace`/`User`/`SearchSourceConnector`/`Folder`, a Zero entry, and a `schemas/` Pydantic mirror — all lowest-friction in `db.py`. (The `automations` package avoided `db.py` but pays for it with string-relationships + import-time model registration; pipelines are more core, like connectors.) |
|
||||
| Pipeline → connector | `connector_id` **nullable**, FK `ON DELETE CASCADE`. A connector-backed pipeline points at a Type-1 source; **`NULL` = a non-connector pipeline** (the Phase-7 "Uploads" pipeline). Eligibility (`DATA_SOURCE` + `AVAILABLE`) is enforced at create via Phase 4a's `is_pipeline_eligible` (app-level, not a DB constraint — the registry is code) **and re-checked at run time in Phase 6** (the static registry can flip a type to `MIGRATING` on deploy, stranding existing pipelines). |
|
||||
| Ownership | `created_by_id` FK `ON DELETE SET NULL`, **nullable** — creator metadata only (workspace-shared, like `Folder`/`Automation`), not an owner. Billing targets the workspace owner (03c). |
|
||||
| Schedule representation | `schedule_cron VARCHAR NULL` (NULL = manual-only) + `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` + `enabled BOOLEAN` + `next_scheduled_at TIMESTAMPTZ NULL` (computed by Phase 6's tick). Chosen over a bare `frequency_minutes` int to match the **automations** schedule model and reuse its cron util (`automations/triggers/builtin/schedule/cron.py`), which takes **both** a cron and a timezone (`compute_next_fire_at(cron, timezone, …)`). |
|
||||
| KB destination | `save_to_kb BOOLEAN` + `destination_folder_id` FK `ON DELETE SET NULL` (nullable). `save_to_kb=true` with NULL folder ⇒ index to workspace root (`documents.folder_id` is nullable). No DB CHECK couples them (an FK `SET NULL` would violate a conditional CHECK); the pairing is validated at the API only. |
|
||||
| Run status enum | `pipeline_run_status`: `pending`, `running`, `succeeded`, `failed`, `cancelled` (mirrors `automation_run_status` minus `timed_out`). |
|
||||
| Run trigger enum | `pipeline_run_trigger`: `manual`, `scheduled`, `upload`. |
|
||||
| Periodic coexistence | **COEXIST, do not migrate** (resolves umbrella open item §2 — see "Periodic-indexing coexistence" below). |
|
||||
| Zero publication | Publish **both tables full-row** (`None`), mirroring `folders` / `search_source_connectors`. Avoids the column-list `_0_version` seam entirely; neither table has a bulky column. |
|
||||
| Naming | The CI **`Pipeline`** entity (a saved, runnable fetch) is distinct from the existing **`IndexingPipelineService`** (`app/indexing_pipeline/`, the KB-ingest service). They coexist but do **not** share a code path for the MVP: Phase 6's WebURL executor reuses the crawler's own KB-write path (`index_crawled_urls`, extended with `folder_id`), **not** `IndexingPipelineService` (which is the file/Composio ingest path — and is what Phase 7's uploads flow builds on). Name routes/services `pipeline_*` (not `indexing_pipeline_*`) to keep the two legible. |
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
### The precedent: `automations` / `automation_runs`
|
||||
|
||||
The cleanest existing analogue to model against.
|
||||
|
||||
- Editable parent: `Automation` — `automations/persistence/models/automation.py:24`. Pattern: `BaseModel, TimestampMixin`; `search_space_id` FK CASCADE (`:27`), `created_by_user_id` FK SET NULL (`:34`), a native enum `status` with `values_callable=lambda x: [e.value for e in x]` (`:44–54`), a JSONB `definition` (`:56`), an explicit `updated_at` with `onupdate` (`:60–66`), and `runs = relationship(..., cascade="all, delete-orphan", passive_deletes=True)` (`:76–81`).
|
||||
- Append-only child: `AutomationRun` — `automations/persistence/models/run.py:20`. `automation_id` FK CASCADE (`:23`), native enum `status` default `pending` (`:37–47`), JSONB `error` (`:60`), nullable `started_at`/`finished_at` (`:62–63`).
|
||||
- Enums: `automations/persistence/enums/run_status.py:8` (`RunStatus(StrEnum)` = pending/running/succeeded/failed/cancelled/timed_out).
|
||||
- Migration: `alembic/versions/144_add_automation_tables.py` — the **exact template** for this phase: idempotent `CREATE TYPE ... DO $$ IF NOT EXISTS` guards (`:31–76`), `CREATE TABLE IF NOT EXISTS` with inline FKs (`:79–96`), `CREATE INDEX IF NOT EXISTS` per FK/status/created_at (`:97–111`), a **partial "due" index** for the schedule scan (`ix_automation_triggers_due`, `:144–152`), and symmetric `downgrade()` dropping indexes→tables→types (`:190–213`).
|
||||
- Run routes: `automations/api/run.py` — `GET /automations/{id}/runs` (list, paginated `limit/offset`, `:13–30`) + `GET /automations/{id}/runs/{run_id}` (detail, `:33–44`). The exact shape to mirror for pipeline runs.
|
||||
|
||||
### The other core tables (relationship + column conventions)
|
||||
|
||||
- Base: `TimestampMixin` (gives `created_at`, `db.py:487`) + `BaseModel` (`id SERIAL PK`, `db.py:498`). Tables wanting a mutable `updated_at` declare it explicitly (see `Folder.updated_at` `db.py:1333–1339`, with `onupdate` + `index=True`).
|
||||
- `SearchSourceConnector` — `db.py:1829`. `connector_type` native enum (`:1867`), `is_indexable` (`:1868`), periodic fields `periodic_indexing_enabled`/`indexing_frequency_minutes`/`next_scheduled_at` (`:1880–1882`), `search_space_id` FK CASCADE (`:1884`), `user_id` FK CASCADE (`:1891`), `documents = relationship(...)` (`:1897`). **Pipelines FK this table** (`connector_id`).
|
||||
- `Folder` — `db.py:1310`. `search_space_id` FK CASCADE (`:1321`), `created_by_id` FK SET NULL (`:1327`). **Pipelines FK this table** (`destination_folder_id`, SET NULL).
|
||||
- `Document.folder_id` is nullable (root/unfiled docs exist) — confirmed by `DOCUMENT_COLS` including `folder_id` (`zero_publication.py:32`) and `Document.folder` `passive_deletes=True` (`db.py:1345`). Lets `save_to_kb` work with a NULL destination.
|
||||
- `Permission` enum has `CONNECTORS_CREATE/READ/UPDATE/DELETE` (`db.py:345–348`); routes authz via `check_permission(session, auth, search_space_id, Permission.X.value, msg)` (`folders_routes.py:42–48`) with `auth = Depends(get_auth_context)` + `session = Depends(get_async_session)` (`folders_routes.py:36–37`). **Reuse the `CONNECTORS_*` permissions** for pipelines (no new permission needed for MVP).
|
||||
|
||||
### Schema conventions
|
||||
|
||||
- `schemas/base.py`: `IDModel` (`id:int`, `:11`) + `TimestampModel` (`created_at`, `:6`), both `ConfigDict(from_attributes=True)`.
|
||||
- `schemas/search_source_connector.py`: `Base` → `Create(Base)` → `Update(BaseModel, all-Optional)` → `Read(Base, IDModel, TimestampModel)` with `model_config = ConfigDict(from_attributes=True)` and `field_validator`/`model_validator` for config consistency (`:24–56`). The shape to mirror for `schemas/pipeline.py`.
|
||||
|
||||
### Zero publication mechanics
|
||||
|
||||
- `zero_publication.py`: `ZERO_PUBLICATION` map (`:81–94`; line numbers post-`main`-merge, which added `automations`/`new_chat_threads` entries) is the single source of truth; `None` ⇒ publish full row (e.g. `folders`, `search_source_connectors`), a list ⇒ column subset (e.g. `AUTOMATION_RUN_COLS` `:44–53`). `apply_publication(conn)` (`:163`) reconciles via `ALTER PUBLICATION ... SET TABLE`; a migration just calls it (template: `159_publish_podcasts_to_zero.py:21–22`). `_format_table_entry` omits a table until it physically exists with all its canonical columns (`:125–152`), so the migration **must create the tables before** calling `apply_publication`. The `_0_version` allowlist (`{"documents","user","podcasts"}`, `:118`) applies **only to column-list tables** — irrelevant here since we publish full-row.
|
||||
|
||||
### The meta-scheduler we coexist with (Phase-6 reuse target)
|
||||
|
||||
- `tasks/celery_tasks/schedule_checker_task.py:17` — `check_periodic_schedules_task`, runs every minute, scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33–39`) and dispatches per-type Celery tasks. This is the **connector-level** periodic path and is **untouched** by Phase 5. Phase 6 adds a sibling scan over `pipelines.next_scheduled_at`.
|
||||
- `utils/periodic_scheduler.py:30` — `create_periodic_schedule` (first-run trigger helper). Reference for Phase 6's pipeline scheduler.
|
||||
|
||||
### Alembic head
|
||||
|
||||
As of the latest `main` sync the head is **`169`** (`alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py`; chain `166→167→168→169`, sequential integer-prefixed files). By the time Phase 5 is implemented, Phase 1 (rename) and Phase 4b (search-enum drop) have each added a migration ahead of `169`. **Set `down_revision` to the then-current head** — verify with `alembic heads`; do not hardcode a number.
|
||||
|
||||
## Target design
|
||||
|
||||
### 1. Enums (`db.py`, near the other `StrEnum`s)
|
||||
|
||||
```python
|
||||
class PipelineRunStatus(StrEnum):
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
SUCCEEDED = "succeeded"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
|
||||
class PipelineRunTrigger(StrEnum):
|
||||
MANUAL = "manual"
|
||||
SCHEDULED = "scheduled"
|
||||
UPLOAD = "upload"
|
||||
```
|
||||
|
||||
Mapped with native PG enum types `pipeline_run_status` / `pipeline_run_trigger` using the `values_callable=lambda x: [e.value for e in x]` convention (matches `Automation.status` `automation.py:46–49`, so the DB stores lowercase values, not the Python member names).
|
||||
|
||||
### 2. `pipelines` table (`db.py`)
|
||||
|
||||
```python
|
||||
class Pipeline(BaseModel, TimestampMixin):
|
||||
__tablename__ = "pipelines"
|
||||
|
||||
name = Column(String(200), nullable=False, index=True)
|
||||
config = Column(JSONB, nullable=False, default=dict, server_default=text("'{}'::jsonb")) # per-pipeline overrides (URL list, crawl opts, proxy override seam)
|
||||
|
||||
save_to_kb = Column(Boolean, nullable=False, default=False, server_default="false")
|
||||
|
||||
# Schedule (NULL cron = manual-only). next_scheduled_at is owned/written by Phase 6's tick.
|
||||
schedule_cron = Column(String(120), nullable=True)
|
||||
schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC") # cron util needs a tz (compute_next_fire_at)
|
||||
enabled = Column(Boolean, nullable=False, default=True, server_default="true")
|
||||
next_scheduled_at = Column(TIMESTAMP(timezone=True), nullable=True)
|
||||
|
||||
# NULL connector_id = non-connector pipeline (Phase-7 Uploads).
|
||||
connector_id = Column(Integer, ForeignKey("search_source_connectors.id", ondelete="CASCADE"), nullable=True, index=True)
|
||||
destination_folder_id = Column(Integer, ForeignKey("folders.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
workspace_id = Column(Integer, ForeignKey("workspaces.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
# Creator metadata, NOT owner: pipelines are workspace-shared (like folders/automations),
|
||||
# so member deletion must NOT nuke them — SET NULL, nullable. Billing targets the workspace
|
||||
# owner (03c), not this field. Phase 6 resolves the acting user as created_by ?? workspace owner.
|
||||
created_by_id = Column(UUID(as_uuid=True), ForeignKey("user.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
|
||||
updated_at = Column(TIMESTAMP(timezone=True), nullable=False,
|
||||
default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC), index=True)
|
||||
|
||||
connector = relationship("SearchSourceConnector", back_populates="pipelines")
|
||||
destination_folder = relationship("Folder")
|
||||
workspace = relationship("Workspace", back_populates="pipelines")
|
||||
created_by = relationship("User", back_populates="pipelines")
|
||||
runs = relationship("PipelineRun", back_populates="pipeline", cascade="all, delete-orphan", passive_deletes=True)
|
||||
```
|
||||
|
||||
Add the inverse `pipelines = relationship("Pipeline", back_populates=...)` on **`Workspace`**, **`User`**, and **`SearchSourceConnector`** (mirror how `SearchSpace.search_source_connectors` / `User.search_source_connectors` are declared at `db.py:1887–1894`; the `created_by`/SET-NULL side mirrors `Folder.created_by` `db.py:1344` and `Automation.created_by` `automation.py:69`). `destination_folder` is intentionally one-directional (no back_populates on `Folder`) to keep the folder model lean.
|
||||
|
||||
> **Ownership decision (review).** Earlier draft used connector-style `user_id NOT NULL CASCADE`. Corrected to `created_by_id … SET NULL` because pipelines are **workspace-shared** content (closer to `Folder`/`Automation` than to the per-user `SearchSourceConnector`): removing a member must not delete the workspace's pipelines or erase run-history audit. For *connector-backed* pipelines this also avoids surprising double-cascade (the connector's own `user_id CASCADE` would already drop them on member deletion via `connector_id`); for the Phase-7 Uploads pipeline (`connector_id` NULL) it's the only thing keeping the pipeline alive after its creator leaves.
|
||||
|
||||
> **FK target naming:** Phases 1–2 are **SHIPPED**, so the table is `workspaces` and the column `workspace_id` — author the FK against `workspaces` / `workspace_id`. (The earlier "if sequencing slips before the rename" caveat is now moot.)
|
||||
|
||||
### 3. `pipeline_runs` table (`db.py`)
|
||||
|
||||
```python
|
||||
class PipelineRun(BaseModel, TimestampMixin):
|
||||
__tablename__ = "pipeline_runs"
|
||||
|
||||
pipeline_id = Column(Integer, ForeignKey("pipelines.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
|
||||
status = Column(SQLAlchemyEnum(PipelineRunStatus, name="pipeline_run_status",
|
||||
values_callable=lambda x: [e.value for e in x]),
|
||||
nullable=False, default=PipelineRunStatus.PENDING,
|
||||
server_default=PipelineRunStatus.PENDING.value, index=True)
|
||||
trigger = Column(SQLAlchemyEnum(PipelineRunTrigger, name="pipeline_run_trigger",
|
||||
values_callable=lambda x: [e.value for e in x]), nullable=False, index=True)
|
||||
|
||||
# Result accounting (written by Phase 6).
|
||||
documents_indexed = Column(Integer, nullable=True)
|
||||
crawls_attempted = Column(Integer, nullable=True)
|
||||
crawls_succeeded = Column(Integer, nullable=True)
|
||||
error = Column(JSONB, nullable=True)
|
||||
|
||||
# Raw fetch blob ref for save_to_kb=false runs (Phase 6 writes via file_storage). Just a key/path string.
|
||||
result_blob_key = Column(String, nullable=True)
|
||||
|
||||
# Crawl-billing idempotency (carry-over from 03c / umbrella Phase 6 §105): micro-USD charged for this run.
|
||||
charged_micros = Column(BigInteger, nullable=True)
|
||||
|
||||
started_at = Column(TIMESTAMP(timezone=True), nullable=True)
|
||||
finished_at = Column(TIMESTAMP(timezone=True), nullable=True)
|
||||
|
||||
pipeline = relationship("Pipeline", back_populates="runs")
|
||||
```
|
||||
|
||||
These extra columns (`crawls_*`, `result_blob_key`, `charged_micros`) are added **now** so Phase 6 needs **no second migration** — they sit unused until the engine writes them. (`charged_micros` directly satisfies umbrella Phase 6 line 105's "record `charged_micros` on the `PipelineRun` for idempotency".)
|
||||
|
||||
> **Out of scope (provenance):** linking individual `documents` back to the `pipeline_run` that produced them (a `documents.pipeline_run_id` column) is **not** in this phase — it would touch the existing `documents` table. Defer to Phase 6/7 if run-level provenance is wanted.
|
||||
|
||||
### 4. Migration (one file, `144`-shaped)
|
||||
|
||||
Chain after the then-current head. In `upgrade()`:
|
||||
|
||||
1. `CREATE TYPE pipeline_run_status` + `pipeline_run_trigger` behind `DO $$ IF NOT EXISTS` guards (copy `144:31–76`).
|
||||
2. `CREATE TABLE IF NOT EXISTS pipelines (...)` with inline FKs: `workspace_id → workspaces(id) ON DELETE CASCADE` (NOT NULL), `connector_id → search_source_connectors(id) ON DELETE CASCADE` (nullable), `destination_folder_id → folders(id) ON DELETE SET NULL` (nullable), `created_by_id → "user"(id) ON DELETE SET NULL` (nullable; note the quoted `"user"` table). `id SERIAL PK`, `created_at`/`updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()`, `config JSONB NOT NULL DEFAULT '{}'::jsonb`, `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'`.
|
||||
3. `CREATE TABLE IF NOT EXISTS pipeline_runs (...)`.
|
||||
4. Indexes (`CREATE INDEX IF NOT EXISTS`): per FK (`workspace_id`, `created_by_id`, `connector_id`, `destination_folder_id`, `pipeline_id`), plus `status`, `created_at`, `updated_at`, `name`, `trigger`. Add the **partial "due" index** for Phase 6's tick:
|
||||
|
||||
```sql
|
||||
CREATE INDEX IF NOT EXISTS ix_pipelines_due
|
||||
ON pipelines (next_scheduled_at)
|
||||
WHERE enabled = true AND next_scheduled_at IS NOT NULL;
|
||||
```
|
||||
|
||||
5. `from app.zero_publication import apply_publication; apply_publication(op.get_bind())` **after** the tables exist (so `_format_table_entry` includes them). Requires the `ZERO_PUBLICATION` code change (§6) to be in the tree.
|
||||
|
||||
`downgrade()`: drop indexes → `pipeline_runs` → `pipelines` → both types (symmetric to `144:190–213`). (The publication reconcile is intentionally not reversed — historical publication shapes are immutable, per `159:25–27`.)
|
||||
|
||||
### 5. Pydantic schemas (`app/schemas/pipeline.py`, exported via `app/schemas/__init__.py`)
|
||||
|
||||
```python
|
||||
class PipelineBase(BaseModel):
|
||||
name: str
|
||||
connector_id: int | None = None
|
||||
config: dict[str, Any] = {}
|
||||
save_to_kb: bool = False
|
||||
destination_folder_id: int | None = None
|
||||
schedule_cron: str | None = None
|
||||
schedule_timezone: str = "UTC"
|
||||
enabled: bool = True
|
||||
|
||||
class PipelineCreate(PipelineBase):
|
||||
workspace_id: int # required on create (path/body), like FolderCreate.search_space_id
|
||||
|
||||
class PipelineUpdate(BaseModel): # all-Optional, like SearchSourceConnectorUpdate
|
||||
name: str | None = None
|
||||
config: dict[str, Any] | None = None
|
||||
save_to_kb: bool | None = None
|
||||
destination_folder_id: int | None = None
|
||||
schedule_cron: str | None = None
|
||||
schedule_timezone: str | None = None
|
||||
enabled: bool | None = None
|
||||
# connector_id is immutable after create (re-create to re-point); not in Update.
|
||||
|
||||
class PipelineRead(PipelineBase, IDModel, TimestampModel):
|
||||
workspace_id: int
|
||||
created_by_id: uuid.UUID | None = None
|
||||
next_scheduled_at: datetime | None = None
|
||||
updated_at: datetime
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
class PipelineRunRead(IDModel, TimestampModel):
|
||||
pipeline_id: int
|
||||
status: PipelineRunStatus
|
||||
trigger: PipelineRunTrigger
|
||||
documents_indexed: int | None = None
|
||||
crawls_attempted: int | None = None
|
||||
crawls_succeeded: int | None = None
|
||||
error: dict[str, Any] | None = None
|
||||
charged_micros: int | None = None
|
||||
started_at: datetime | None = None
|
||||
finished_at: datetime | None = None
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
class PipelineRunList(BaseModel):
|
||||
items: list[PipelineRunRead]
|
||||
total: int
|
||||
```
|
||||
|
||||
**Validators:**
|
||||
|
||||
- `schedule_cron` + `schedule_timezone` (when cron not None): validate together with the existing automations cron util `validate_cron(cron, timezone)` (`automations/triggers/builtin/schedule/cron.py:15`) so a bad expression **or** bad IANA timezone 422s at create, not at the Phase-6 tick. (`schedule_timezone` is consumed by Phase 6's `compute_next_fire_at`; default `"UTC"`.)
|
||||
- The `save_to_kb` ⇄ `destination_folder_id` pairing and **connector eligibility** are validated in the **route** (they need a DB session: folder-in-workspace check + `is_pipeline_eligible(connector.connector_type)`), not in the pure schema validator — mirroring how `folders_routes` validates parent-in-search-space in the handler (`folders_routes.py:54–58`).
|
||||
|
||||
### 6. Zero publication (`zero_publication.py`)
|
||||
|
||||
Add to `ZERO_PUBLICATION` (`:81–94`), both **full-row**:
|
||||
|
||||
```python
|
||||
"pipelines": None,
|
||||
"pipeline_runs": None,
|
||||
```
|
||||
|
||||
Full-row (like `folders`/`search_source_connectors`) — no `_0_version` allowlist edit needed, no column-drift migrations later. (If a bulky column is ever added and must be excluded, switch that table to an explicit COLS list and handle the `_0_version` seam at `:118` then.) `verify_publication` will then expect both tables; the migration's `apply_publication` call (§4) reconciles them.
|
||||
|
||||
> **Client sync is not live until the frontend lands (safe).** Publishing backend-side only makes the rows *available* to Zero; nothing syncs to a client until the (deferred) frontend Zero schema + permissions include these tables. Publishing now is harmless and matches the existing pattern (e.g. `automation_runs` was published by migration before/independent of its UI). This entry exists so the chat-agent run-history context (Phase 6) and a later Pipelines UI get push for free.
|
||||
|
||||
### 7. Routes (`app/routes/pipelines_routes.py`, registered in `routes/__init__.py`)
|
||||
|
||||
Follow `folders_routes` for authz (`check_permission` + `get_auth_context` + `get_async_session`) and `automations/api/run.py` for the run-list shape. Register the router in `routes/__init__.py` near `search_source_connectors_router` (`:105`).
|
||||
|
||||
| Method + path | Behaviour | AuthZ |
|
||||
|---------------|-----------|-------|
|
||||
| `POST /pipelines` | Create. Validate: workspace access; if `connector_id`, load it, assert same workspace + `is_pipeline_eligible` (4xx otherwise); if `destination_folder_id`, assert folder in same workspace; `save_to_kb=true` allowed with NULL folder (→ root). Set `created_by_id = auth.user.id`. Persist; return `PipelineRead`. | `CONNECTORS_CREATE` |
|
||||
| `GET /pipelines?workspace_id=` | List pipelines in a workspace. | `CONNECTORS_READ` |
|
||||
| `GET /pipelines/{id}` | Read one. | `CONNECTORS_READ` |
|
||||
| `PUT /pipelines/{id}` | Update (all-optional). Re-validate folder-in-workspace + cron if changed. | `CONNECTORS_UPDATE` |
|
||||
| `DELETE /pipelines/{id}` | Delete (runs cascade). | `CONNECTORS_DELETE` |
|
||||
| `POST /pipelines/{id}/run` | **Manual trigger.** Insert a `PipelineRun(status=pending, trigger=manual)`, enqueue the Phase-6 Celery task with the run id, return the run. Until Phase 6, the task is a stub (no-op/marks failed) — the row + enqueue path are what this phase delivers. (Concurrency control — block a second run while one is in flight — is deferred to Phase 6, reusing the connector indexing Redis locks `utils/indexing_locks.py`.) | `CONNECTORS_UPDATE` |
|
||||
| `GET /pipelines/{id}/runs?limit=&offset=` | List runs newest-first, paginated (`limit` 1–200 default 50, `offset` ≥0) → `PipelineRunList`. Mirror `automations/api/run.py:13–30`. | `CONNECTORS_READ` |
|
||||
| `GET /pipelines/{id}/runs/{run_id}` | Run detail. | `CONNECTORS_READ` |
|
||||
|
||||
All handlers scope by the pipeline's `workspace_id` and 404 on cross-workspace access. A thin `PipelineService` (mirroring `automations` `RunService`) is optional; for MVP, inline session logic in the routes (as `folders_routes` does) is acceptable.
|
||||
|
||||
## Periodic-indexing coexistence (resolves umbrella open item §2)
|
||||
|
||||
**Decision: COEXIST for MVP — do NOT migrate connector periodic config into pipelines.**
|
||||
|
||||
- The existing `SearchSourceConnector.periodic_indexing_enabled` / `next_scheduled_at` path and its meta-scheduler (`schedule_checker_task.py`) **stay fully functional and untouched** in Phases 5–6. Pipelines are **purely additive**: a new `pipelines.next_scheduled_at` scan (Phase 6) runs **alongside** the connector scan.
|
||||
- **Rationale:** (a) lowest risk + smallest diff (no backfill of existing connector schedules into pipeline rows, no data migration of live schedules); (b) only the WebURL crawler is the MVP pipeline executor (Phase 6) — file sources keep using their connector-level periodic path; (c) consistent with the umbrella posture "DB migrations carry users; backend behaviour can change incrementally."
|
||||
- **Known overlap (flagged for Phase 6, not solved here):** a single `WEBCRAWLER_CONNECTOR` could have BOTH connector-level periodic indexing AND a pipeline wrapping it → double crawl + **double bill**. The data model permits it; the *guard* is Phase 6's responsibility. **Recommendation for Phase 6:** when a pipeline is created/enabled over a connector, treat the **pipeline as authoritative** and set that connector's `periodic_indexing_enabled=False` (single scheduler owns each connector). Recorded here so Phase 6 implements the de-dup; Phase 5 only needs the columns to support either path.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Enums** `PipelineRunStatus` + `PipelineRunTrigger` in `db.py`.
|
||||
2. **ORM** `Pipeline` + `PipelineRun` in `db.py`; inverse `pipelines` relationships on `Workspace`, `User`, `SearchSourceConnector`.
|
||||
3. **Migration**: 2 types + 2 tables + indexes (incl. `ix_pipelines_due`) + `apply_publication`; symmetric downgrade.
|
||||
4. **Schemas** `app/schemas/pipeline.py` (+ export in `schemas/__init__.py`); cron syntactic validator.
|
||||
5. **Zero**: add `pipelines`/`pipeline_runs` (full-row) to `ZERO_PUBLICATION`.
|
||||
6. **Routes** `pipelines_routes.py` (CRUD + `/run` + `/runs` list/detail); register in `routes/__init__.py`; reuse `CONNECTORS_*` permissions.
|
||||
7. **Phase-6 task stub**: a named Celery task `run_pipeline(run_id)` in **`app/pipelines/tasks.py`** (the module Phase 6 fleshes out) that `/run` enqueues — no-op/marks the run `failed` with an "engine not implemented" error until Phase 6 fills it in (keeps the endpoint honest and testable). Register it so it's dispatchable: add `"app.pipelines.tasks"` to the Celery `include` list (`celery_app.py`). Phase 6 replaces the stub body in-place (same task name + module → `/run`'s import is stable) and adds the scheduler task + Beat entry + queue routing.
|
||||
8. **Tests** (below).
|
||||
|
||||
## Tests
|
||||
|
||||
- **Migration round-trip**: upgrade creates both tables + both enum types + `ix_pipelines_due`; `zero_publication --verify` reports no mismatch (both tables published full-row); downgrade drops cleanly.
|
||||
- **Create gating (cross-link to 04a)**: `POST /pipelines` with an `AVAILABLE` `DATA_SOURCE` connector (WebURL/GDrive/OneDrive/Dropbox) → 201; with a `MIGRATING`/`MCP_TOOL`/`DISABLED` connector → 4xx (`is_pipeline_eligible` false); with `connector_id=None` → allowed (Uploads case). **(Superseded in Phase 7: once the system-managed Uploads pipeline exists, user `POST` with `connector_id=None` is rejected 4xx and the Uploads row is auto-created on upload — see `07` §7. Update this test in Phase 7.)**
|
||||
- **Folder validation**: `destination_folder_id` in another workspace → 4xx; `save_to_kb=true` with NULL folder → allowed.
|
||||
- **Cron validation**: invalid `schedule_cron` → 422; NULL cron → allowed (manual-only).
|
||||
- **Cascade**: deleting a connector deletes its pipelines + their runs; deleting a folder SET-NULLs `destination_folder_id` (pipeline survives); deleting a workspace removes pipelines + runs.
|
||||
- **Manual run**: `POST /pipelines/{id}/run` inserts a `pending`/`manual` run and enqueues the task; `GET /pipelines/{id}/runs` returns it newest-first with correct `total`.
|
||||
- **AuthZ**: cross-workspace access to any pipeline/run route → 404; missing `CONNECTORS_*` permission → 403.
|
||||
- **Zero shape**: `expected_publication_shape` includes `pipelines`/`pipeline_runs` as full-row.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **`db.py` growth.** Adds two more models to an already-large module. Accepted for relationship/Zero/schema ergonomics + consistency with the other core tables; a later extraction to a `pipelines` package is additive.
|
||||
- **Coexisting schedulers (double-bill window).** Until Phase 6 adds the de-dup guard, a connector with both its own periodic indexing and a wrapping pipeline can crawl/bill twice. Documented above; Phase 6 owns the fix. MVP exposure is small (operator-created overlap only).
|
||||
- **Eligibility enforced in code, not SQL.** `is_pipeline_eligible` is a registry (04a) check at the route, so a direct DB insert could bypass it. Same trade-off 04a already accepted (no DB column for the taxonomy).
|
||||
- **Pre-built unused columns.** `crawls_*` / `result_blob_key` / `charged_micros` ship empty in Phase 5. Deliberate — avoids a second migration in Phase 6.
|
||||
- **`connector_id` CASCADE drops run history.** Deleting a connector erases its pipelines' audit trail. If audit retention becomes a requirement, switch to `SET NULL` + a discriminator to distinguish Uploads from orphaned (additive change).
|
||||
- **`created_by_id` nullable shifts work to Phase 6.** SET-NULL preserves shared pipelines past member deletion (the goal) but means the run engine cannot assume a creator — it must resolve `created_by ?? workspace owner` for the indexing actor + billing target. Flagged in the Phase-6 handoff; harmless until then (no engine).
|
||||
- **Static-registry strand.** A pipeline created over an `AVAILABLE` connector can later reference a type the registry flips to `MIGRATING` on a deploy (create-time check passed; the row persists). Phase 6 must re-check `is_pipeline_eligible` at run time and fail the run cleanly. The data model intentionally does not encode eligibility, so no migration is needed when the registry changes.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- Run **engine**, scheduling tick over `pipelines.next_scheduled_at`, crawl-billing wiring (write `charged_micros`/`crawls_*`), raw-blob persistence, and chat-agent run-history context → **Phase 6** (`06-pipelines-exec.md`). Phase 6 also owns three guards this model only *enables*: (a) **run-time eligibility re-check** (`is_pipeline_eligible` — a deploy can flip a connector type to `MIGRATING`, stranding existing pipelines → fail the run cleanly, don't crash); (b) **acting-user resolution** (`created_by_id` can be NULL after member deletion → resolve `created_by ?? workspace owner` for indexing + billing); (c) **concurrency** (Redis indexing lock per pipeline/connector).
|
||||
- File upload creating/using an "Uploads" pipeline (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) + generalizing opt-in KB save → **Phase 7** (`07-upload-pipeline-kb.md`).
|
||||
- Document→run provenance column → deferred (Phase 6/7 if needed).
|
||||
- Frontend Pipelines UI (list/create/configure/run-history/manual run) → frontend umbrella.
|
||||
- Public pay-as-you-go API over Type-1 pipelines → post-MVP (umbrella "Deferred").
|
||||
65
plans/backend/06-ongoing-automation.md
Normal file
65
plans/backend/06-ongoing-automation.md
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
# Phase 6 — Ongoing Automation (chat-native "keep watching")
|
||||
|
||||
> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `scraping` subagent).
|
||||
> Reuses the existing chat + automations machinery; adds no parallel engine.
|
||||
|
||||
## Objective
|
||||
|
||||
Support "keep watching": a persistent chat where the agent periodically re-invokes scraper verbs and drops
|
||||
results into the session. "What changed" is the agent reading its own prior turns from the durable
|
||||
checkpoint — no diff store.
|
||||
|
||||
## Mechanism
|
||||
|
||||
A **chat watch is an `Automation` bound to the current chat** — nothing more. No dedicated thread, no
|
||||
thread "kind", no schema change. Start it and the chat gains a `schedule` trigger that re-posts the
|
||||
question on a cadence; stop it and the automation is deleted and the chat is a normal chat again. The
|
||||
chat's own checkpoint is the memory.
|
||||
|
||||
- **`chat_message` action** — params `{ thread_id, message }`. Its handler drains
|
||||
`stream_new_chat(user_query=message, chat_id=thread_id, …)` under `AuthContext.system(creator,
|
||||
source="automation")` against the **current chat**. Auto-approve (system auth; CI verbs are read-only
|
||||
and don't interrupt).
|
||||
- **Durable memory + delivery** — `stream_new_chat` persists messages to `new_chat_messages` (already
|
||||
Zero-synced to the UI) and advances the shared Postgres checkpointer keyed by `chat_id`. A scheduled run
|
||||
has no SSE client; it runs server-side and is delivered via the persisted rows. "What changed" is the
|
||||
agent reading the chat's own prior turns.
|
||||
- **Worker-safe checkpointer** — the shared `AsyncPostgresSaver` pool binds connections to the loop that
|
||||
opened them, but Celery uses a fresh loop per task (`PoolTimeout`). Dispose the checkpointer pool per
|
||||
task in `run_async_celery_task`, mirroring `_dispose_shared_db_engine`, so a worker can use the *durable*
|
||||
checkpointer (not `InMemorySaver`) that "what changed" requires.
|
||||
- **`start_watch`** — a `scraping` subagent tool that binds a watch to the *current* chat: it distills
|
||||
the recurring question + cadence and creates the automation (`schedule` + `chat_message(thread_id =
|
||||
current chat)`).
|
||||
- **"Is this chat watched?"** — derived: an active automation with a `chat_message` action targeting the
|
||||
chat. No stored flag.
|
||||
- **Controls** — run-now = trigger a run; stop = delete the automation (chat reverts to normal).
|
||||
- **Concurrency** — the checkpointer is single-writer per thread; a tick skips if the prior turn on that
|
||||
chat is still running (DB `ai_responding` flag).
|
||||
|
||||
## Work items
|
||||
|
||||
1. Durable checkpointer in workers: `_dispose_shared_checkpointer_pool` in `run_async_celery_task`
|
||||
(before + after), mirroring the SQLAlchemy engine dispose. **[done]**
|
||||
2. `chat_message` action: params + factory + handler (drains `stream_new_chat`); concurrency guard. **[done]**
|
||||
3. Watch service: create (bind `schedule` + `chat_message` automation to a chat) / stop (delete) /
|
||||
find-for-thread (is-watched) / run-now. **[done]**
|
||||
4. `start_watch` tool on the `scraping` subagent (+ prompt line); binds to the current chat. **[done]**
|
||||
5. Controls — chat tools (`stop_watch`, `refresh_watch`) + REST (`GET /automations/watches`,
|
||||
`POST /automations/{id}/run`; stop = `DELETE /automations/{id}`). **[done]**
|
||||
|
||||
## Tests
|
||||
|
||||
- `run_async_celery_task` disposes the checkpointer pool before and after a task. **[done]**
|
||||
- `chat_message` drains a turn on the given thread; skips when one is in-flight. **[done]**
|
||||
- Watch service: create binds a `schedule` + `chat_message` automation to the chat; stop deletes it;
|
||||
find-for-thread filters by plan; run-now launches the schedule trigger. **[done]**
|
||||
- `start_watch` / `stop_watch` / `refresh_watch` tools act on the current chat from tool context. **[done]**
|
||||
- Watch routes registered on the automations router. **[done]**
|
||||
- Integration (running stack): two scheduled runs on one chat — run 2 sees run 1 in checkpoint history.
|
||||
|
||||
## Deferred / out of scope
|
||||
|
||||
- Zero delivery-cost optimization (signal-column + REST fetch vs full-content sync) — app-wide, separate.
|
||||
- Server-side turns surviving browser navigation for *interactive* chat ("zombie streaming").
|
||||
- `start_watch` cadence UX refinements. Verbs/doors → `04`/`05`; agent playbook → `07`.
|
||||
|
|
@ -1,350 +0,0 @@
|
|||
# Phase 6 — Pipeline execution + scheduling (run engine, crawl billing, blob, chat context)
|
||||
|
||||
> # ❌ SUPERSEDED (2026-06-30)
|
||||
> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no pipeline run engine or scheduler. The existing **automations** system already runs the full chat agent (with the native `web_search`/`scrape_webpage` tools) on **cron/event** triggers and persists **`automation_runs`** — which is exactly what this phase was building. Retained **for historical context only** — do **not** implement it. **Its still-relevant ideas survive in the canonical revamp:** (1) **billing crawls done outside chat** → now handled at the **capability executor** ([`revamp/04a`](revamp%20phases%204-7/04a-capabilities.md)), so recurring/automation runs meter correctly; (2) the refresh **hot loop** (crawl → diff → judge → append) → [`revamp/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md), writing the **Timeline** ([`revamp/05a`](revamp%20phases%204-7/05a-timeline.md)) — not `automation_runs.output`; (3) **recurrence + "run now"** → [`revamp/06-triggers.md`](revamp%20phases%204-7/06-triggers.md) (reuse automations via a CI action). The `bill=False` seam and `index_crawled_urls` `folder_id`/`stats` params described below are **not needed** (no pipeline path).
|
||||
|
||||
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 6).
|
||||
> Precondition: Phase 5 (`05-pipelines-model.md`) live — `pipelines` / `pipeline_runs` tables, schemas, Zero, CRUD + `/run` (currently enqueues a **stub** `run_pipeline(run_id)`). Depends on `03a-crawler-core.md` (`crawl_url` SUCCESS signal + `crawls_succeeded` counter) and `03c-crawl-billing.md` (`WebCrawlCreditService`). Sibling ahead: `07-upload-pipeline-kb.md` (uploads-as-pipeline).
|
||||
|
||||
> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace` / `workspaces`. Citations below that still show `search_space_id` / `SearchSpace` / `searchspaces` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated; e.g. the `get_pipeline_runs` tool's dep key is `workspace_id`, matching the renamed deps dict). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–5 shift them).
|
||||
|
||||
## Objective
|
||||
|
||||
Make pipelines actually **run**. Phase 5 shipped the data model + a stub task; this phase fills in:
|
||||
|
||||
1. **Run engine** — turn a `PipelineRun` row into work: resolve the pipeline + connector, re-check eligibility, acquire a lock, crawl the source (WebURL crawler = the only MVP executor), optionally save to the KB destination folder, persist a raw-result blob otherwise, write the run record (status/timing/counts/error).
|
||||
2. **Crawl billing wiring** (carry-over from `03c`) — bill `crawls_succeeded` to the **workspace owner** via `WebCrawlCreditService` for **both** KB-save branches (a `save_to_kb=false` run must not crawl for free); record `charged_micros` on the run for idempotency.
|
||||
3. **Scheduling** — a Celery Beat tick that fires due `pipelines.next_scheduled_at` (cron), modeled on the **automations** schedule selector (cron + `FOR UPDATE SKIP LOCKED` + self-heal). Plus the **de-dup guard** Phase 5 deferred (a pipeline over a connector disables that connector's own periodic indexing).
|
||||
4. **Chat-agent context** — a read-only main-agent **tool** (`get_pipeline_runs`) exposing recent run history scoped to the active workspace (umbrella default = tool).
|
||||
|
||||
This is backend-only; the Pipelines UI is deferred to the frontend umbrella.
|
||||
|
||||
## Locked decisions (MVP)
|
||||
|
||||
| Concept | Decision |
|
||||
|---------|----------|
|
||||
| MVP executor | **WebURL crawler only.** `connector.connector_type == WEBCRAWLER_CONNECTOR`. Any other Type-1 type (GDrive/OneDrive/Dropbox) → run fails cleanly with `"executor not implemented for <type>"` (their connector-level periodic path still works; pipelines over them are a Phase-7+ concern). `connector_id IS NULL` (Phase-7 Uploads) → not run by this engine (uploads register their own run in Phase 7). |
|
||||
| Run-engine shape | Mirror **automations**: thin Celery task → `run_async_celery_task` → async `execute_pipeline_run(session, run_id)` (`automations/tasks/execute_run.py:19-33`). PENDING-gated + idempotent terminal no-op (`automations/runtime/executor.py:29-30`). Status lifecycle `pending → running → succeeded/failed` with `started_at`/`finished_at` (mirrors `repository.mark_running/mark_succeeded/mark_failed`). |
|
||||
| KB-save path | **Reuse `index_crawled_urls`** (the battle-tested 2-phase indexer) for `save_to_kb=true`, extended with a `folder_id` param (set on created/updated `Document`s → destination folder). `save_to_kb=true` + NULL folder → root (existing behaviour; `documents.folder_id` nullable). |
|
||||
| Non-KB path | `save_to_kb=false` → a **fetch-only** crawl loop (no `Document` rows): crawl each URL, collect `{url, content, metadata}`, persist as **one JSON blob** via `file_storage` backend, store `result_blob_key` on the run. (The 2-phase indexer is `Document`-centric and cannot "not persist"; a separate small loop is cleaner than bending it.) |
|
||||
| Billing ownership | **Run engine owns billing for the pipeline path** (pre-check on `len(urls)` + charge on `crawls_succeeded` + record `charged_micros`), calling the crawler with **billing suppressed** (`bill=False`). The connector `/index` + connector-periodic paths keep `03c`'s in-indexer billing (`bill=True`). This keeps `charged_micros` run-idempotency clean and bills both KB branches identically. **(03c coordination — see below.)** |
|
||||
| Scheduler | New `pipeline_schedule_select` Beat task modeled on `automations/triggers/builtin/schedule/selector.py` (cron, `FOR UPDATE SKIP LOCKED`, self-heal of NULL `next_scheduled_at`), **not** the simpler connector `frequency_minutes` checker — because Phase 5 chose `schedule_cron`. Reuses the existing `croniter` util (`automations/triggers/builtin/schedule/cron.py`). |
|
||||
| Schedule timezone | Cron needs a timezone (`compute_next_fire_at(cron, timezone, …)`). Phase 5's model has only `schedule_cron`. **Add `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` to `pipelines`** (small additive amendment to `05` — see "Required 05 amendment"). Matches how automations store cron **and** timezone (`schedule/selector.py:86-89`). |
|
||||
| Concurrency | Per-pipeline Redis lock + (when connector-backed) the **existing connector lock** `utils/indexing_locks.py` so a pipeline run and any residual connector index can't double-crawl the same connector. |
|
||||
| Chat context | A main-agent **tool** `get_pipeline_runs` (registry pattern, `main_agent/tools/registry.py:85-102`), opening its own `shielded_async_session()` (like `KnowledgeTreeMiddleware`) and scoping to the build-time `workspace_id`. Always-on middleware injection is **deferred** (token cost; tool is on-demand). |
|
||||
| Code location | New `app/pipelines/` package: `engine.py` (`execute_pipeline_run`), `tasks.py` (Celery `run_pipeline` + `pipeline_schedule_select`), `scheduler.py` (the tick), `storage.py` (blob key). The Phase-5 stub `run_pipeline` is **replaced** by the real task here; the `/run` route + scheduler enqueue it. (ORM stays in `db.py` per Phase 5; only the *engine* is a package, mirroring `app/automations/`.) |
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
### Run-engine precedent (automations)
|
||||
|
||||
- **Task wrapper**: `automations/tasks/execute_run.py:19-33` — `@celery_app.task(name=..., bind=True)` → `run_async_celery_task(lambda: _impl(run_id))`; `_impl` opens `get_celery_session_maker()()` and calls `execute_run(session, run_id)`, rolling back + re-raising on failure.
|
||||
- **Launch**: `automations/dispatch/launch.py:43-60` — create the child row `status=PENDING`, `session.add` + `commit` + `refresh`, then `task.apply_async(args=[run.id], time_limit=…)`. (Phase 5's `/run` already does the create+enqueue; this phase makes the task real.)
|
||||
- **Executor lifecycle**: `automations/runtime/executor.py:23-75` — `load run; if run.status != PENDING: return` (idempotent terminal no-op, `:29-30`); `mark_running` + `commit` (`:46-47`); on bad snapshot → `mark_failed` + commit (`:35-44`); terminal `mark_succeeded`/`mark_failed` each `commit`.
|
||||
|
||||
### The MVP executor: `webcrawler_indexer.index_crawled_urls`
|
||||
|
||||
- Signature `webcrawler_indexer.py:44-53`: `(session, connector_id, search_space_id, user_id, start_date=None, end_date=None, update_last_indexed=True, on_heartbeat_callback=None) -> (int, str|None)`.
|
||||
- Reads URLs from `connector.config["INITIAL_URLS"]` via `parse_webcrawler_urls` (`:118-119`).
|
||||
- **Creates `Document` rows with NO `folder_id`** (`:222-239`) → today everything lands at workspace root. Updates existing docs in place. Crawls via `crawler.crawl_url(url)` (`:297`; `03a` changes this signature/return). Tracks `documents_indexed` / `documents_updated` / counts; final commit at `:432`; returns `(total_processed, error)` (`:477`).
|
||||
- **No billing today** — `03c` adds the owner pre-check + `charge_credits` inside this function (`03c` §2). **This phase adds a `bill: bool = True` seam** so the pipeline path can suppress it (plus `folder_id` / `urls` params + a `stats` out-param — see "Crawler changes"; the positional return is left unchanged so the shared wrapper keeps working).
|
||||
- Connector entrypoint chain (the path the pipeline engine **bypasses**, because it's notification-centric, not run-centric): `connector_tasks.py:418-454` (`index_crawled_urls_task` → `_index_crawled_urls` → `run_web_page_indexing`) → a `_run_indexing_with_notifications(...)` **call site** (e.g. the WebCrawler one at `search_source_connectors_routes.py:~2524`). The wrapper itself is **defined at `:1296`** (acquires the connector lock at `:1334`, length-unpacks the indexer return at `:1503`).
|
||||
|
||||
### KB-ingest service (folder support already exists there)
|
||||
|
||||
- `IndexingPipelineService` (`indexing_pipeline/indexing_pipeline_service.py:82`) already threads a destination folder via `ConnectorDocument.folder_id` (`:273-274`, `:301-302`, `:326`). **But the webcrawler indexer does not use this service** — it writes `Document`s directly. So the umbrella's "route through `IndexingPipelineService` into the destination folder" is accurate for **file/Composio** sources, **not** the crawler. For the WebURL MVP, the minimal correct move is to thread `folder_id` into `index_crawled_urls` (not refactor the crawler onto `IndexingPipelineService`). Unifying them is a post-MVP cleanup.
|
||||
|
||||
### Scheduler precedent (cron) + Beat registration
|
||||
|
||||
- **Selector** `automations/triggers/builtin/schedule/selector.py`: tick (`:47-65`) → self-heal NULL `next_fire_at` (`:68-98`) → claim due rows `FOR UPDATE SKIP LOCKED` + advance via `compute_next_fire_at` + set `last_fired_at` (`:101-150`) → `_start_one` launches each (`:153-183`). `_TICK_BATCH=200` cap (`:35`).
|
||||
- **Cron util** `automations/triggers/builtin/schedule/cron.py`: `validate_cron(cron, timezone)` (`:15`) and `compute_next_fire_at(cron, timezone, *, after)` (`:28`, returns UTC). **Requires a timezone** → drives the 05 amendment.
|
||||
- **Beat source** `automations/triggers/builtin/schedule/source.py:14-20` — a `BEAT_SCHEDULE` dict (`crontab(minute="*")`) merged into the app at `celery_app.py:323` (`**SCHEDULE_BEAT_SCHEDULE`). The app's beat dict is `celery_app.py:268-324`; the worker `include` list is `:180-198`; task routing (`index_crawled_urls → CONNECTORS_QUEUE`) is `:237-255` (`:246`).
|
||||
- **Connector meta-scheduler we coexist with**: `schedule_checker_task.py:17` scans `SearchSourceConnector` where `periodic_indexing_enabled AND next_scheduled_at <= now` (`:33-39`), with a Redis-lock guard `is_connector_indexing_locked` (`:93`). Untouched; the pipeline tick is a sibling.
|
||||
|
||||
### Locks, storage, chat tooling
|
||||
|
||||
- **Locks** `utils/indexing_locks.py`: `acquire_connector_indexing_lock(connector_id)->bool` (`:24`), `release_…` (`:37`), `is_…_locked` (`:43`), keyed `indexing:connector_lock:{id}`, TTL `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`.
|
||||
- **Storage** `file_storage/service.py:39,46`: `get_storage_backend()` (from `file_storage/factory.py`) → `backend.put(key, data, content_type=…)` / `open_stream(key)` / `delete(key)`. Keys are built per-purpose (`file_storage/keys.py:11`, `build_document_file_key`). For runs we add a sibling key builder (no `document_files` row — runs aren't documents).
|
||||
- **Tool registry** `main_agent/tools/registry.py:85-102` — `_MAIN_AGENT_TOOL_FACTORIES: {name: (factory, required_dep_names)}`; `build_main_agent_tools(deps, …)` resolves deps + builds (`:105-147`). Display metadata lives in `shared/tools/catalog`; the main-agent name list in `main_agent/tools/index` (per the registry docstring `:1-14`). Read-only DB pattern for a chat surface: `KnowledgeTreeMiddleware` opens `shielded_async_session()` and filters by its build-time `search_space_id` (`knowledge_tree/middleware.py:199-206`).
|
||||
- **Workspace owner** (billing target): `SearchSpace.user_id` (`03c` §2.1 resolves the owner with a direct `select SearchSpace.user_id where id == search_space_id`).
|
||||
|
||||
## Target design
|
||||
|
||||
### 1. `execute_pipeline_run(session, run_id)` — the engine (`app/pipelines/engine.py`)
|
||||
|
||||
```
|
||||
load run (pipeline + connector eager); if run is None: raise
|
||||
if run.status != PENDING: return # status gate; also no-ops acks_late redelivery (executor.py:29-30)
|
||||
pipeline = run.pipeline
|
||||
mark_running(run): status=RUNNING, started_at=now; commit
|
||||
# ^ committed BEFORE any crawl/charge → a redelivered/duplicate task sees RUNNING and returns (executor.py:46-47)
|
||||
|
||||
pipeline_lock = connector_lock = False
|
||||
connector = None
|
||||
try:
|
||||
# --- guards: fail cleanly, never crash the worker (Phase-5 hand-offs (a)/(b)) ---
|
||||
if pipeline.connector_id is None: # Uploads pipeline — Phase 7, not here
|
||||
return fail(run, "uploads pipeline has no fetch executor")
|
||||
connector = pipeline.connector
|
||||
if not is_pipeline_eligible(connector.connector_type): # 04a runtime re-check (strand guard)
|
||||
return fail(run, f"connector type {connector.connector_type} no longer eligible")
|
||||
if connector.connector_type != WEBCRAWLER_CONNECTOR:
|
||||
return fail(run, f"executor not implemented for {connector.connector_type}")
|
||||
|
||||
owner_id = workspace_owner_id(session, pipeline.workspace_id) # select Workspace.user_id (03c §2.1)
|
||||
actor_user_id = str(pipeline.created_by_id or owner_id) # Phase-5 hand-off (b)
|
||||
urls = parse_webcrawler_urls(pipeline.config.get("INITIAL_URLS") or connector.config.get("INITIAL_URLS"))
|
||||
if not urls:
|
||||
return fail(run, "no URLs to crawl")
|
||||
|
||||
# --- concurrency: take both; release only what we took (Phase-5 hand-off (c)) ---
|
||||
pipeline_lock = acquire_pipeline_indexing_lock(pipeline.id)
|
||||
if not pipeline_lock:
|
||||
return fail(run, "a run for this pipeline is already in progress")
|
||||
connector_lock = acquire_connector_indexing_lock(connector.id)
|
||||
if not connector_lock:
|
||||
return fail(run, "connector is currently indexing; retry later")
|
||||
|
||||
# --- billing pre-flight (run-level; 03c statics). len(urls) is a safe upper bound. ---
|
||||
svc = WebCrawlCreditService(session)
|
||||
if WebCrawlCreditService.billing_enabled() and run.charged_micros is None:
|
||||
await svc.check_credits(owner_id, len(urls)) # InsufficientCreditsError → except → fail("out of crawl credit")
|
||||
|
||||
# --- fetch ---
|
||||
if pipeline.save_to_kb:
|
||||
stats = {} # out-param sink — NON-breaking (see §3)
|
||||
await index_crawled_urls(session, connector.id, pipeline.workspace_id, actor_user_id,
|
||||
urls=urls, folder_id=pipeline.destination_folder_id,
|
||||
bill=False, stats=stats, update_last_indexed=True)
|
||||
documents_indexed = stats["documents_indexed"]
|
||||
crawls_attempted = stats["crawls_attempted"]
|
||||
crawls_succeeded = stats["crawls_succeeded"]
|
||||
result_blob_key = None
|
||||
else:
|
||||
outcomes = await crawl_urls_fetch_only(urls) # list[dict] from CrawlOutcome (03a)
|
||||
crawls_attempted = len(urls)
|
||||
crawls_succeeded = sum(1 for o in outcomes if o["status"] == "success")
|
||||
documents_indexed = 0
|
||||
result_blob_key = await persist_run_blob(run, outcomes)
|
||||
|
||||
# --- charge: stamp run + add audit, THEN charge (its single commit flushes all three atomically) ---
|
||||
if WebCrawlCreditService.billing_enabled() and crawls_succeeded > 0 and run.charged_micros is None:
|
||||
run.charged_micros = WebCrawlCreditService.successes_to_micros(crawls_succeeded)
|
||||
record_token_usage(session, usage_type="web_crawl", workspace_id=pipeline.workspace_id,
|
||||
user_id=owner_id, cost_micros=run.charged_micros,
|
||||
call_details={"urls": len(urls), "successes": crawls_succeeded,
|
||||
"pipeline_id": pipeline.id, "run_id": run.id},
|
||||
message_id=None) # add only — does not commit (03c §2.3)
|
||||
await svc.charge_credits(owner_id, crawls_succeeded) # debits + COMMITS run+audit+balance (03c §2.4)
|
||||
|
||||
mark_succeeded(run): status=SUCCEEDED, finished_at=now,
|
||||
documents_indexed, crawls_attempted, crawls_succeeded, result_blob_key; commit
|
||||
except InsufficientCreditsError:
|
||||
return fail(run, "out of crawl credit") # run is RUNNING→FAILED; no debit happened
|
||||
except Exception as e:
|
||||
return fail(run, {"message": str(e), "type": type(e).__name__})
|
||||
finally:
|
||||
if connector_lock: release_connector_indexing_lock(connector.id)
|
||||
if pipeline_lock: release_pipeline_indexing_lock(pipeline.id)
|
||||
```
|
||||
|
||||
`fail(run, err)` = set `status=FAILED`, `finished_at=now`, `error={...}`, `commit`, return — so the `finally` still runs and releases only the locks actually taken. `connector.id` in `finally` is reached only when `connector_lock is True`, which implies `connector` was resolved. **Idempotency rests on the status gate**, not `charged_micros`: `mark_running` commits `RUNNING` before any crawl/charge, so an `acks_late` redelivery (`celery_app.py:228-229`) or a manual re-enqueue sees `RUNNING`/terminal and returns. `charged_micros` is the belt-and-suspenders guard reserved for a future stuck-`RUNNING` re-driver (none in MVP).
|
||||
|
||||
Notes:
|
||||
- **`workspace_owner_id`** = `select Workspace.user_id where id == workspace_id` (`03c` §2.1 pattern).
|
||||
- **`is_pipeline_eligible`** is the 04a registry helper — re-checked here at run time (a deploy can flip a type to `MIGRATING`; the create-time check in Phase 5 doesn't protect the row forever).
|
||||
- **URL source**: per-pipeline `config["INITIAL_URLS"]` overrides the connector's (the `config` JSONB "per-pipeline overrides" Phase 5 reserved); fall back to the connector's `INITIAL_URLS`.
|
||||
|
||||
### 2. Celery tasks (`app/pipelines/tasks.py`)
|
||||
|
||||
```python
|
||||
@celery_app.task(name="run_pipeline", bind=True)
|
||||
def run_pipeline(self, run_id: int) -> None:
|
||||
return run_async_celery_task(lambda: _run(run_id)) # execute_run.py:19-22 shape
|
||||
|
||||
async def _run(run_id: int) -> None:
|
||||
async with get_celery_session_maker()() as session:
|
||||
try:
|
||||
await execute_pipeline_run(session, run_id)
|
||||
except Exception:
|
||||
logger.exception("pipeline run %d failed unexpectedly", run_id)
|
||||
await session.rollback(); raise
|
||||
```
|
||||
|
||||
- `run_pipeline` **replaces** the Phase-5 stub of the same name; Phase 5's `/run` route already enqueues `run_pipeline(run.id)` (no route change needed).
|
||||
- **Routing/registration**: ensure `"app.pipelines.tasks"` is in the `include` list (`celery_app.py:180-198`) — Phase 5 already added it for the stub; no-op if present — and route `"run_pipeline": {"queue": CONNECTORS_QUEUE}` (it crawls — same queue as `index_crawled_urls`, `celery_app.py:246`). Give the `apply_async`/`delay` a `time_limit` (the connectors queue already has an 8h hard cap, `celery_app.py:219`).
|
||||
|
||||
### 3. Crawler changes — `index_crawled_urls` (coordination with `03a`/`03c`)
|
||||
|
||||
Four additive params, **all default-compatible** with the existing connector path:
|
||||
|
||||
1. **`folder_id: int | None = None`** — set on each created `Document` (`:222-239`) and on the existing-doc update branch, so KB-save lands in the destination folder. Default `None` = root (today's behaviour).
|
||||
2. **`urls: list[str] | None = None`** — when provided, crawl this list instead of re-reading `connector.config["INITIAL_URLS"]` (`:118-119`). **Required** for the per-pipeline `config["INITIAL_URLS"]` override (a Phase-5 `config` capability) and to keep the engine's `len(urls)` billing pre-check consistent with what's actually crawled. Default `None` = read connector config (today's behaviour).
|
||||
3. **`bill: bool = True`** — gate `03c`'s in-indexer `check_credits` + `charge_credits` + `record_token_usage`. The pipeline engine calls `bill=False` and bills at the run level; the connector `/index` + periodic paths keep `bill=True`. *(Small addition to `03c`'s §2 wiring — see "Cross-plan coordination".)*
|
||||
4. **`stats: dict | None = None`** — an **out-param sink**, NOT a return-shape change. When provided, the indexer populates `stats["documents_indexed"|"crawls_attempted"|"crawls_succeeded"]`. **Do NOT widen the return tuple:** the shared `_run_indexing_with_notifications` wrapper unpacks every indexer's return **by length** (`search_source_connectors_routes.py:1499-1507`: `if len(result) == 3: a,b,c = result else: a,b = result`), so a 4-tuple would raise `ValueError` for *every* connector. The 2-/3-tuple return stays exactly as-is (wrapper untouched); the engine passes a fresh `stats` dict and reads it after the call (the wrapper passes none → `None` → no-op). `03a` work-item 3 already reflects this (it exposes `crawls_succeeded` via **task metadata** only, with this `stats` sink owned by Phase 6 — not via the positional return).
|
||||
|
||||
The engine **bypasses** `_run_indexing_with_notifications` (it acquires the connector lock itself at `:1334` — going through the wrapper would self-deadlock on that same lock, and the wrapper's connector-`Notification` UI is the wrong surface for a pipeline run, which has its own `PipelineRun` record). It calls `index_crawled_urls` directly with no heartbeat callback. (Consequence: pipeline runs don't emit connector-indexing notifications and aren't touched by `cleanup_stale_indexing_notifications` — fine; run status lives on `pipeline_runs`.)
|
||||
|
||||
### 4. Fetch-only path + blob (`app/pipelines/engine.py` + `app/pipelines/storage.py`)
|
||||
|
||||
For `save_to_kb=false`:
|
||||
|
||||
```python
|
||||
async def crawl_urls_fetch_only(urls):
|
||||
crawler = WebCrawlerConnector() # 03a: no firecrawl ctor arg
|
||||
out = []
|
||||
for url in urls:
|
||||
outcome = await crawler.crawl_url(url) # 03a: CrawlOutcome(status, result, error, tier)
|
||||
out.append({
|
||||
"url": url,
|
||||
"status": outcome.status.value, # CrawlOutcomeStatus: success | empty | failed
|
||||
"content": (outcome.result or {}).get("content", ""),
|
||||
"metadata": (outcome.result or {}).get("metadata", {}),
|
||||
"error": outcome.error,
|
||||
})
|
||||
return out
|
||||
```
|
||||
|
||||
> **03a return shape.** `03a` is **committed to the `CrawlOutcome` dataclass** (`status`/`result`/`error`/`tier`) — the tuple option was dropped precisely because this fetch-only path consumes `.status`/`.result`/`.error` as attributes. So the unpack above is the firm contract; no adaptation needed.
|
||||
|
||||
```python
|
||||
# storage.py
|
||||
def build_pipeline_run_result_key(*, workspace_id: int, run_id: int) -> str:
|
||||
return f"pipeline_runs/{workspace_id}/{run_id}/result.json"
|
||||
|
||||
async def persist_run_blob(run, outcomes) -> str:
|
||||
key = build_pipeline_run_result_key(workspace_id=run.pipeline.workspace_id, run_id=run.id)
|
||||
data = json.dumps({"runs": outcomes}).encode()
|
||||
await get_storage_backend().put(key, data, content_type="application/json")
|
||||
return key # → run.result_blob_key
|
||||
```
|
||||
|
||||
- One blob per run (single configured backend, like document files). `result_blob_key` (Phase 5 column) stores the key; reads resolve via `get_storage_backend()`.
|
||||
- **Billing still applies** here — `crawls_succeeded` is counted from `outcomes`, so non-KB runs cost exactly the same as KB runs (umbrella line 105). This is the whole reason the run engine, not the indexer, owns pipeline billing.
|
||||
|
||||
### 5. Scheduler tick (`app/pipelines/scheduler.py` + `tasks.py`)
|
||||
|
||||
A near-copy of the automations selector, retargeted at `pipelines`:
|
||||
|
||||
```python
|
||||
@celery_app.task(name="pipeline_schedule_select")
|
||||
def pipeline_schedule_select() -> None:
|
||||
return run_async_celery_task(_tick)
|
||||
|
||||
async def _tick():
|
||||
async with get_celery_session_maker()() as session:
|
||||
now = datetime.now(UTC)
|
||||
await _self_heal_null_next(session, now) # enabled + cron + next_scheduled_at IS NULL → compute
|
||||
claims = await _claim_due(session, now) # enabled + next_scheduled_at <= now, FOR UPDATE SKIP LOCKED, advance
|
||||
for c in claims:
|
||||
await _start_run(session, c) # create PipelineRun(trigger=scheduled, status=pending) + run_pipeline.apply_async
|
||||
```
|
||||
|
||||
- **`_claim_due`**: `select Pipeline where enabled AND connector_id IS NOT NULL AND schedule_cron IS NOT NULL AND next_scheduled_at IS NOT NULL AND next_scheduled_at <= now ORDER BY next_scheduled_at LIMIT 200 FOR UPDATE SKIP LOCKED` (uses Phase 5's partial `ix_pipelines_due`). The `connector_id IS NOT NULL` filter is **defense-in-depth** so a non-connector pipeline (the Phase-7 Uploads pipeline) can **never** be scheduled into perpetually-failing runs even if a cron is somehow set on it — the engine would `fail` every such run (`§1`); this stops them at the selector. (Phase 7 also guards the API; this is the belt to that suspenders.) Advance `next_scheduled_at = compute_next_fire_at(schedule_cron, schedule_timezone, after=now)`; on `InvalidCronError` → `enabled=False` (self-disable, like `selector.py:90-96,131-137`). Commit, then create the run + enqueue (mirror `_start_one` reload-after-commit, `selector.py:153-183`).
|
||||
- **`_self_heal_null_next`**: backfills `next_scheduled_at` for enabled cron pipelines missing it (fresh inserts, restored rows) — so Phase 5's create can leave it NULL and the tick fills it within ≤1 min.
|
||||
- **Duplicate suppression**: in `_start_run`, skip creating a scheduled run if the pipeline already has a non-terminal run (`status IN (pending, running)`) — mirrors the connector checker's in-progress guard (`schedule_checker_task.py:100-116`). Without it, a pipeline whose run outlasts its cron interval piles up `scheduled` runs that immediately `fail` on the lock (noisy, though harmless). `next_scheduled_at` is still advanced (the fire isn't retried — `catchup=False`, like the automations selector).
|
||||
- **Beat entry**: add `"pipeline-schedule-select": {"task": "pipeline_schedule_select", "schedule": crontab(minute="*"), "options": {"expires": 50}}` to `celery_app.py:268-324` (default fast queue, like `check_periodic_schedules`). (`"app.pipelines.tasks"` is already in `include` from Phase 5's stub, so `pipeline_schedule_select` in the same module is auto-registered.)
|
||||
- **`schedule_cron` edits** (Phase 5 `PUT`): when `schedule_cron` (or timezone) changes, **null `next_scheduled_at`** so the tick recomputes (small `PUT`-handler addition — see Required 05 amendment). NULL cron ⇒ manual-only (tick ignores it).
|
||||
|
||||
### 6. De-dup guard (resolves Phase-5 deferred double-bill)
|
||||
|
||||
When a pipeline that **wraps a connector** is created or enabled (Phase 5 routes), set that connector's `periodic_indexing_enabled = False` (+ `next_scheduled_at = None`) so a single scheduler owns each connector (the **pipeline is authoritative**, per Phase 5's recommendation). Two touch points:
|
||||
|
||||
- Phase 6 owns the rule; implement it in the Phase-5 `POST /pipelines` and `PUT /pipelines/{id}` handlers (when `connector_id` is set and `enabled`). This is a tiny handler addition (Phase 5 explicitly handed it here).
|
||||
- Belt-and-suspenders: the run engine already takes the **connector lock** during a run, so even if both scan, only one crawl proceeds at a time (the connector checker skips on a held lock, `schedule_checker_task.py:93`).
|
||||
|
||||
### 7. Chat-agent run-history tool (`main_agent/tools/pipeline_runs.py`)
|
||||
|
||||
A read-only tool registered in the main-agent registry (umbrella default = tool):
|
||||
|
||||
```python
|
||||
def create_get_pipeline_runs_tool(*, workspace_id: int) -> BaseTool:
|
||||
@tool
|
||||
async def get_pipeline_runs(pipeline_id: int | None = None, limit: int = 20) -> str:
|
||||
"""Recent pipeline run history for THIS workspace: name, status, trigger,
|
||||
schedule, counts, started/finished. Read-only."""
|
||||
async with shielded_async_session() as session: # knowledge_tree/middleware.py:199 pattern
|
||||
q = (select(PipelineRun, Pipeline.name, Pipeline.schedule_cron)
|
||||
.join(Pipeline, PipelineRun.pipeline_id == Pipeline.id)
|
||||
.where(Pipeline.workspace_id == workspace_id))
|
||||
if pipeline_id is not None:
|
||||
q = q.where(Pipeline.id == pipeline_id)
|
||||
rows = (await session.execute(
|
||||
q.order_by(PipelineRun.created_at.desc()).limit(min(limit, 100)))).all()
|
||||
return render_compact(rows) # name • status • trigger • finished_at • docs/crawls
|
||||
return get_pipeline_runs
|
||||
```
|
||||
|
||||
- **Registration**: add `"get_pipeline_runs": (_build_get_pipeline_runs_tool, ("workspace_id",))` to `_MAIN_AGENT_TOOL_FACTORIES` (`registry.py:85-102`) — the dep key is `workspace_id`, matching the post-rename deps dict; add the name to `main_agent/tools/index`; add display metadata to `shared/tools/catalog`.
|
||||
- **Workspace scope is structural** — the tool only ever sees its build-time `workspace_id`; no cross-workspace leakage (same guarantee `KnowledgeTreeMiddleware` relies on).
|
||||
- **Anonymous / no-workspace turns**: exclude the tool when there's no `workspace_id` (anonymous chat has no workspace/pipelines), the same way `KnowledgeTreeMiddleware` is CLOUD-only (`knowledge_tree/middleware.py:130-131`). The registry factory requires `workspace_id`, so just don't enable it in the anonymous tool set.
|
||||
- The tool reads `pipelines`/`pipeline_runs` directly (no Zero dependency), so it works server-side immediately even though client Zero sync is dormant until the frontend lands (Phase 5 note).
|
||||
- **Deferred**: an always-on `<pipeline_activity>` middleware injection (à la `KnowledgeTreeMiddleware`) — adds per-turn token cost; revisit if the agent under-uses the tool.
|
||||
|
||||
### 8. (Optional) blob read endpoint
|
||||
|
||||
Phase 5 stores `result_blob_key` but exposes no reader. Add `GET /pipelines/{id}/runs/{run_id}/result` → stream the blob via `get_storage_backend().open_stream(run.result_blob_key)` (404 if NULL), `CONNECTORS_READ`, workspace-scoped — mirroring `file_storage/api.py:69-90`. Lets API consumers + the chat tool fetch non-KB results. (Small; include if cheap, else defer to the public-API phase.)
|
||||
|
||||
## Required `05` amendment (timezone + recompute)
|
||||
|
||||
Small, additive. **Items 1–3 (the `schedule_timezone` column) are already folded into `05`** (model/migration/schema/validator) — listed here for traceability. **Item 4 is Phase-6 route logic** added on top of Phase 5's handlers (Work items 5–6):
|
||||
|
||||
1. *(done in `05` §2)* **Model**: `schedule_timezone = Column(String(64), nullable=False, default="UTC", server_default="UTC")` on `Pipeline`.
|
||||
2. *(done in `05` §4)* **Migration**: `schedule_timezone VARCHAR(64) NOT NULL DEFAULT 'UTC'` in `CREATE TABLE pipelines`.
|
||||
3. *(done in `05` §5)* **Schema**: `schedule_timezone: str = "UTC"` on `PipelineBase` + `PipelineUpdate`; `(schedule_cron, schedule_timezone)` validated together via `validate_cron(cron, tz)` (`cron.py:15`).
|
||||
4. **Routes** (Phase 6 adds to `05` §7 handlers): on `PUT` when `schedule_cron`/`schedule_timezone` changes, set `next_scheduled_at = None` (tick self-heals). On `POST`/`PUT` with `connector_id` + `enabled`, apply the §6 de-dup guard.
|
||||
|
||||
(If we'd rather not touch `05`: hardcode `"UTC"` everywhere in Phase 6 and skip the column — but that loses operator-meaningful local schedules and diverges from automations. The column is the right call.)
|
||||
|
||||
## Cross-plan coordination
|
||||
|
||||
- **`03a`** — Phase 6 calls `WebCrawlerConnector.crawl_url(url)` directly in the fetch-only path and relies on `03a`'s `CrawlOutcome.status` (`SUCCESS`/`EMPTY`/`FAILED`, dataclass form — already locked in `03a`). `03a` work-item 3 is **already aligned**: it surfaces the `crawls_succeeded` counter via **task metadata** only (not by widening `index_crawled_urls`'s positional return — the shared `_run_indexing_with_notifications` wrapper unpacks by tuple length, `:1499-1507`). Phase 6 owns the `folder_id`/`urls`/`stats` params; `03a` owns the counter + `CrawlOutcome`.
|
||||
- **`03c`** — add the **`bill: bool = True`** parameter to the in-indexer billing (so the pipeline path can suppress it). `03c`'s `WebCrawlCreditService` statics (`billing_enabled()`, `successes_to_micros(n)`, `check_credits`, `charge_credits`) are reused verbatim by the run engine. `03c`'s risk note already anticipated this: *"Phase 5's `PipelineRun` can persist `charged_micros` for stronger idempotency"* — Phase 6 is where that lands.
|
||||
- **`05`** — the four amendments above; `run_pipeline` task name + `/run` enqueue already exist (stub → real); `charged_micros`/`crawls_*`/`result_blob_key` columns already exist (pre-built in Phase 5).
|
||||
- **`04a`** — `is_pipeline_eligible(connector_type)` is imported for the run-time re-check.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **`app/pipelines/` package**: `engine.py` (`execute_pipeline_run` + `mark_running/succeeded/fail` helpers + `workspace_owner_id` + `crawl_urls_fetch_only`), `tasks.py` (`run_pipeline`, `pipeline_schedule_select`), `scheduler.py` (`_tick`/`_self_heal_null_next`/`_claim_due`/`_start_run`), `storage.py` (blob key + `persist_run_blob`).
|
||||
2. **Pipeline lock helpers**: add `acquire_pipeline_indexing_lock(id)` / `release_pipeline_indexing_lock(id)` to `utils/indexing_locks.py` (key `indexing:pipeline_lock:{id}`), mirroring the connector lock (`:19-40`).
|
||||
3. **Crawler changes**: `index_crawled_urls` gains `folder_id` + `urls` + `bill` + a `stats` **out-param** (populates `documents_indexed`/`crawls_attempted`/`crawls_succeeded`). **No return-shape change → no caller changes** (the shared wrapper's 2-/3-tuple unpack stays valid). Coordinate with `03a`/`03c` (see "Cross-plan coordination").
|
||||
4. **Celery wiring** (`celery_app.py`): ensure `"app.pipelines.tasks"` is in `include` (added in Phase 5 for the stub); route `run_pipeline → CONNECTORS_QUEUE`; add the `pipeline-schedule-select` Beat entry.
|
||||
5. **De-dup guard** in Phase-5 `POST`/`PUT /pipelines` (disable wrapped connector's periodic indexing).
|
||||
6. **`05` amendment**: `schedule_timezone` column/migration/schema/validator + `next_scheduled_at` reset on cron edit.
|
||||
7. **Chat tool** `get_pipeline_runs` + registry/index/catalog registration.
|
||||
8. **(Optional)** `GET …/runs/{run_id}/result` blob reader.
|
||||
9. **Tests** (below).
|
||||
|
||||
## Tests
|
||||
|
||||
- **Manual run, KB-save**: WebURL pipeline `save_to_kb=true` + `destination_folder_id` → run goes `pending→running→succeeded`, `Document`s created **in that folder**, `documents_indexed`/`crawls_*` set, `finished_at` set.
|
||||
- **Manual run, non-KB**: `save_to_kb=false` → **no** `Document`s, `result_blob_key` set + blob readable, `crawls_*` set. Billing **still** charges `crawls_succeeded` (parity with KB run).
|
||||
- **Billing**: enabled + sufficient → owner debited `crawls_succeeded * 1000`, `charged_micros` stamped, one `web_crawl` `TokenUsage`; **insufficient → run fails pre-crawl (FAILED), no debit**; disabled (self-hosted) → no charge, no `web_crawl` row, `charged_micros` stays NULL.
|
||||
- **Idempotency / lifecycle**: re-enqueuing a `RUNNING` or terminal run → immediate no-op (`status != PENDING` gate) → **no second charge** (the primary guard; `mark_running` commits before any charge); engine never raises out of `fail()` paths (bad type / NULL connector / no URLs / ineligible / lock contention) — all land the run in `FAILED` and release locks.
|
||||
- **Eligibility re-check**: pipeline over a connector whose type is now `MIGRATING` → run fails cleanly (not crash).
|
||||
- **Scheduler**: a due cron pipeline → tick creates a `scheduled` run + advances `next_scheduled_at` to the next cron match (in `schedule_timezone`); NULL `next_scheduled_at` self-heals; invalid cron self-disables (`enabled=False`); manual-only (NULL cron) never fires.
|
||||
- **De-dup**: creating/enabling a pipeline over a connector flips that connector's `periodic_indexing_enabled=False`; concurrent connector-checker tick skips on the held lock.
|
||||
- **Concurrency**: a second run while one is in-flight → fails with "already in progress" (locks held).
|
||||
- **Chat tool**: `get_pipeline_runs` returns this workspace's runs only; another workspace's runs never appear; respects `limit`.
|
||||
- **Counts contract (regression-critical)**: `index_crawled_urls` keeps its existing 2-/3-tuple return (the shared `_run_indexing_with_notifications` length-unpack still works for *all* connectors); when a `stats` dict is passed it's populated with `documents_indexed`/`crawls_attempted`/`crawls_succeeded`; `bill=False` suppresses in-indexer billing; connector path (`bill=True`, no `stats`) behaves byte-for-byte as before. Add a wrapper test asserting webcrawler indexing through `_run_indexing_with_notifications` still unpacks cleanly.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Billing split across two call sites** (indexer for connector path, run engine for pipeline path), unified only by `03c`'s shared statics. Accepted: it's what makes `charged_micros` run-idempotency clean and keeps both KB branches billed identically. The `bill` flag is the single seam.
|
||||
- **Stuck `RUNNING` runs.** A crash *after* `mark_running` but *before* terminal leaves `status=RUNNING` forever (the `status != PENDING` gate then no-ops any redelivery — same property as automations). No double-charge (gate + `charged_micros`), but the run never completes and its locks rely on Redis TTL to free. A stale-run sweeper (à la `cleanup_stale_indexing_notifications`, `celery_app.py:279-285`) is deferred; MVP exposure is a rare worker crash.
|
||||
- **Two crawl code paths** (`index_crawled_urls` for KB vs `crawl_urls_fetch_only` for non-KB). Minor duplication of the crawl loop; deliberate, since forcing the `Document`-centric 2-phase indexer into a "don't persist" mode is messier. Both share `WebCrawlerConnector.crawl_url` + the same billing math.
|
||||
- **WebURL-only executor.** A pipeline over GDrive/OneDrive/Dropbox is creatable (Phase 5 lets `AVAILABLE DATA_SOURCE`) but fails at run time ("executor not implemented"). Acceptable for MVP (those keep their connector-periodic path); generalize the executor in Phase 7+.
|
||||
- **De-dup guard is one-way.** Creating/enabling a pipeline disables the wrapped connector's `periodic_indexing_enabled`, but **disabling or deleting that pipeline does NOT restore it** — so a connector can end up indexed by *neither* path (data goes stale) until the user re-enables connector periodic indexing manually. MVP-acceptable (operator action); a symmetric restore-on-disable is a small follow-up. Flag in the (deferred) UI copy.
|
||||
- **Blob backend assumption.** `result_blob_key` stores only the key (no backend name, unlike `document_files.storage_backend`). Fine for the single configured backend; if multi-backend ever lands, add a backend column to `pipeline_runs` (additive).
|
||||
- **Non-KB blob built in memory.** `crawl_urls_fetch_only` accumulates every URL's extracted content into one JSON blob before upload — a large URL list (or huge pages) inflates worker memory. Acceptable for MVP URL counts; if it bites, stream to the backend or cap per-run URLs (a `config` knob). Also: non-KB runs have **no dedup** (every run re-crawls + re-bills all URLs) — that's inherent to "don't persist", and the user opted out of KB.
|
||||
- **Lock lifetime vs run length.** The Redis locks use `CONNECTOR_INDEXING_LOCK_TTL_SECONDS`; a crawl run longer than the TTL could let a second run start. Pre-existing property of the connector lock (the connectors queue caps at 8h); revisit the TTL if pipeline runs routinely exceed it.
|
||||
- **Two minute-level Beat scans** (connector checker + pipeline tick) now run every minute. Both are indexed (`ix_pipelines_due` partial index; connector scan on its own predicate) + batch-capped (200), so cost is negligible.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **Uploads pipeline execution** (`connector_id=NULL`, `trigger=upload`, always `save_to_kb`) → **Phase 7** (`07-upload-pipeline-kb.md`). This engine explicitly fails NULL-connector runs.
|
||||
- **Non-WebURL executors** (GDrive/OneDrive/Dropbox pipelines) → Phase 7+.
|
||||
- **Always-on chat context middleware** (`<pipeline_activity>` injection) → deferred (tool-only for MVP).
|
||||
- **Document→run provenance** (`documents.pipeline_run_id`) → deferred (touches `documents`).
|
||||
- **Public pay-as-you-go API over pipelines** + **public MCP KB server** → post-MVP (umbrella "Deferred").
|
||||
- **Frontend** Pipelines UI (run-history, manual run, schedule editor) → frontend umbrella.
|
||||
76
plans/backend/07-orchestration.md
Normal file
76
plans/backend/07-orchestration.md
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# Phase 7 — Orchestration / Conversation (the CI-expert subagent)
|
||||
|
||||
> **Status (2026-07-02): shipped for the web verbs.** The `scraping` subagent, its tools, prompt, and
|
||||
> router blurb are live; `web.discover`/`web.scrape` compose and the "keep watching" handoff (`06`) is
|
||||
> wired. `maps.*` composition is deferred until the Maps actor exists.
|
||||
> Sits atop `04`/`05` (and `06` for the ongoing mode). The multi-agent chat runtime (deepagents, subagent
|
||||
> dispatch, streaming, citation middleware) is shipped. Net-new here = one builtin subagent + its tools +
|
||||
> its prompt. Locate code by symbol/grep.
|
||||
|
||||
## Objective
|
||||
|
||||
Ship the `scraping` subagent — a builtin CI/scraper-expert that delivers the product through
|
||||
plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand
|
||||
standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's
|
||||
new vs prior tool outputs already in context.
|
||||
|
||||
## The subagent
|
||||
|
||||
Builtin `subagents/builtins/scraping/`, peer to `research`/`deliverables`, packed via
|
||||
`pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py`
|
||||
(`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook).
|
||||
The main agent delegates CI-/scraping-flavored requests to it via `description.md`.
|
||||
|
||||
## The playbook (`system_prompt.md`)
|
||||
|
||||
1. **Intent routing** — one-shot ("compare/find/what is") → compose verbs & answer; standing concern
|
||||
("watch/track/notify when/weekly") → hand to `06`; ambiguous → ask one question ("just once, or keep
|
||||
watching?").
|
||||
2. **Verb composition** — chains `web.discover → web.scrape` and `maps.search → maps.place → maps.reviews`;
|
||||
infer URLs/queries/locations from context.
|
||||
3. **"What changed"** — re-invoke the verb and compare against prior tool outputs in the chat history,
|
||||
summarizing what's new.
|
||||
|
||||
## The toolset (`load_tools`)
|
||||
|
||||
| Tool | Wraps | Billing |
|
||||
|------|-------|---------|
|
||||
| capability verbs — `web.scrape`, `web.discover` shipped; `maps.search`/`maps.place`/`maps.reviews` deferred (Maps actor) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) |
|
||||
| `start_watch` / `stop_watch` / `refresh_watch` | bind a watch to the current chat via `06` | per re-invocation |
|
||||
|
||||
Verbs reach the subagent through the chat door generator `access/chat.py::build_capability_tools` (`05`),
|
||||
which turns registry verbs (`04`) into tools with owner-wallet billing. The `scraping` subagent owns these
|
||||
CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- The subagent drives the `04` verbs via tools; fetch/bypass/clean logic stays in Acquisition + `04`,
|
||||
callable headless.
|
||||
- Humans get the agent; REST/MCP callers call `04` verbs directly, skipping this subagent.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **[done]** Subagent scaffold: `subagents/builtins/scraping/` (`agent.py` / `tools/index.py` /
|
||||
`description.md` / `system_prompt.md`), packed via `pack_subagent`.
|
||||
2. **[done]** CI playbook prompt: intent routing (block on ambiguous cadence), verb chains, "what changed" reasoning.
|
||||
3. **[done]** Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`).
|
||||
4. **[done]** `start_watch` / `stop_watch` / `refresh_watch` seam to `06`, bound to the current chat.
|
||||
5. **[done]** Router registration: `description.md` for main-agent delegation.
|
||||
|
||||
## Tests
|
||||
|
||||
- A CI-/scraping-flavored request routes to `scraping`; a non-CI request does not.
|
||||
- "compare X and Y" composes verbs and answers in plain language, persisting nothing.
|
||||
- "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`.
|
||||
- A follow-up re-invokes the verb and reports new items vs prior in-context outputs.
|
||||
- "watch this weekly" routes to the ongoing mode, not a one-shot answer.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Verbs, doors → `04`/`05`. Ongoing/periodic mechanism → `06`.
|
||||
- Richer multi-step playbooks, proactive suggestions, cross-topic synthesis.
|
||||
- Frontend CI surfaces → frontend umbrella.
|
||||
|
||||
## Open questions
|
||||
|
||||
- None outstanding for the web verbs. `maps.*` composition reopens when the Maps actor lands.
|
||||
|
|
@ -1,195 +0,0 @@
|
|||
# Phase 7 — File upload as a pipeline + KB-save-secondary
|
||||
|
||||
> # ❌ SUPERSEDED (2026-06-30)
|
||||
> **This entire phase is dropped.** Per the **Architecture correction** in `00-umbrella-plan.md`, there is no "Uploads pipeline" and no upload `PipelineRun`. Uploads simply populate the **input-only KB** through the existing upload routes (an upload creates a `Document` — that *is* KB membership). The umbrella's KB-input invariant ("uploads always land in the KB") is inherent and needs no pipeline wrapper. Retained **for historical context only** — do **not** implement it. **The CI-relevant successor is the "context folder" idea in [`revamp phases 4-7/05b-intelligence.md`](revamp%20phases%204-7/05b-intelligence.md)**: files uploaded in a CI chat go into the KB as normal (indexed), routed to a dedicated folder, and may feed the materiality judge via KB retrieval. (Canonical Phase 7 is now **Orchestration** — [`revamp/07-orchestration.md`](revamp%20phases%204-7/07-orchestration.md) — unrelated to uploads.)
|
||||
|
||||
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 7) — the final backend phase.
|
||||
> Precondition: Phase 5 (`05-pipelines-model.md`) live (`pipelines`/`pipeline_runs` tables, `PipelineRunTrigger.UPLOAD`, `connector_id` nullable) and Phase 6 (`06-pipelines-exec.md`) live (run engine, which **explicitly fails `connector_id IS NULL` runs**). No sibling ahead — this closes the backend umbrella.
|
||||
|
||||
> **Implementation note (post-rename).** Phases 1–2 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace`. Citations below that still show `search_space_id` / `SearchSpace` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 3–6 shift them).
|
||||
|
||||
## Objective
|
||||
|
||||
Close the loop on the umbrella's positioning — *"User file uploads also do a pipeline entry and register a run that files were uploaded by user; only file uploads are always saved in the knowledge base"* — by:
|
||||
|
||||
1. **Uploads-as-pipeline** — give every workspace a singleton **"Uploads" pipeline** (`connector_id = NULL`, `save_to_kb = true`), lazily created, and make each upload request **register a `PipelineRun(trigger=upload)`** so uploads show up in pipeline run history (and in the Phase-6 chat `get_pipeline_runs` tool) exactly like connector fetches.
|
||||
2. **KB-save-secondary (confirm + close)** — the opt-in `save_to_kb` + `destination_folder_id` machinery for *connector* pipelines already shipped in Phases 5–6; this phase records the **inverse invariant for uploads** (uploads **always** save to KB) and verifies nothing in the pivot accidentally made KB-save the *default* for connector pipelines.
|
||||
|
||||
**The core design fact (drives everything):** uploads are **NOT executed by the Phase-6 run engine**. Phase 6's `execute_pipeline_run` fails any `connector_id IS NULL` run with `"uploads pipeline has no fetch executor"` (`06` §1). The existing upload routes remain the executor (they already create `Document`s + dispatch ETL); Phase 7 only **attaches an audit `PipelineRun`** to that flow and ensures the Uploads pipeline row exists. No `run_pipeline` enqueue, no engine change.
|
||||
|
||||
## Locked decisions (MVP)
|
||||
|
||||
| Concept | Decision |
|
||||
|---------|----------|
|
||||
| Uploads pipeline | **One singleton per workspace**, `connector_id = NULL`, `save_to_kb = true`, `schedule_cron = NULL` (never scheduled), `enabled = true`, `name = "Uploads"`. **Lazily get-or-created** on first upload (not at workspace creation — avoids backfilling every existing workspace). |
|
||||
| Uniqueness / race-safety | A **partial unique index** `ON pipelines (workspace_id) WHERE connector_id IS NULL` makes "one Uploads pipeline per workspace" a DB invariant and lets get-or-create be race-safe via `INSERT … ON CONFLICT DO NOTHING` + re-`SELECT`. (Phase 5 already reserves `connector_id IS NULL` to mean "Uploads", so this index is exactly that semantic.) Small Phase-7 migration. |
|
||||
| Run lifecycle | **Audit record, created terminal at request time.** The upload route inserts `PipelineRun(trigger=upload, status=succeeded, documents_indexed=<accepted file count>, started_at=finished_at=now())`. Per-file ETL outcomes continue to live on `Document.status` (`pending→ready/failed`, already Zero-synced). Rationale below. |
|
||||
| Why not track per-file ETL into the run | The `fileupload` path fans out **N independent per-file Celery tasks** (`task_dispatcher.py:44`) with **no batch barrier**, and accurate "which docs belong to this run" needs the **`documents.pipeline_run_id` provenance column that Phase 5 deferred** (`05` §3 "Out of scope (provenance)"). Building a barrier is out of MVP scope; the run is an *upload event* record, matching the umbrella's "register a run that files were uploaded". |
|
||||
| Billing | **No crawl billing for uploads** — uploads aren't crawls. `charged_micros` / `crawls_*` stay `NULL` on upload runs. Existing ETL/embedding credit behaviour (if any) is unchanged and orthogonal (`03c` only governs `web_crawl`). |
|
||||
| Folder placement | The Uploads pipeline's `destination_folder_id` stays **NULL** — it is a singleton and cannot encode per-upload folders. Folder placement remains owned by the **existing upload code** (`fileupload` → root; `folder-upload` → its created/declared folder). The run/pipeline is **descriptive**, not the folder driver (unlike connector pipelines, where the engine reads `destination_folder_id`). |
|
||||
| Surfaces wired | `POST /documents/fileupload` and `POST /documents/folder-upload` (both create `Document`s). The desktop precheck route `POST /documents/folder-mtime-check` (`documents_routes.py:1533`) creates nothing and is **not** wired; `folder-unlink` / `folder-sync-finalize` are delete/sync lifecycle, also out. |
|
||||
| Connector-pipeline KB default | **Unchanged from Phase 5:** `save_to_kb` defaults `False` (`05` §2 `server_default="false"`). This phase only asserts uploads set it `true`; it does **not** flip the connector default. |
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
### The upload routes (the executors we attach to)
|
||||
|
||||
- **`POST /documents/fileupload`** — `documents_routes.py:126-332`. Permission `DOCUMENTS_CREATE` (`:163-169`). Phase 1: create `Document(search_space_id, document_type=FILE, status=pending, created_by_id, …)` per file, dedup by `unique_identifier_hash` (`:217-266`), `commit` (`:278`). Phase 1.5: `store_document_file(...)` per doc (`:289-297`). Phase 2: `dispatcher.dispatch_file_processing(document_id=…)` per file (`:307-316`). Returns `document_ids` / `skipped_duplicates` (`:318-325`). **Creates no folder (root), no run record today.** `files_to_process` (`:213`) = the accepted (newly-queued) set; `skipped_duplicates` (`:214`) = already-`READY` dupes.
|
||||
- **`POST /documents/folder-upload`** — `documents_routes.py:1598-1748`. Permission `DOCUMENTS_CREATE` (`:1623-1629`). Creates/reuses a root `Folder` (`:1673-1704`), writes temps, then dispatches **one** batch task `index_uploaded_folder_files_task.delay(...)` (`document_tasks.py:1403`) which has its own notification + heartbeat (`:1446-1457`) — i.e. a **real completion barrier** exists on this path (unlike `fileupload`). Returns `root_folder_id` + `file_count` (`:1743-1748`).
|
||||
- **Dispatcher** — `task_dispatcher.py:12-52`: `dispatch_file_processing(...)` → `process_file_upload_with_document_task.delay(document_id=…)`, **one task per file**, fire-and-forget (no barrier).
|
||||
|
||||
### Phase 5/6 hand-offs this phase consumes
|
||||
|
||||
- `Pipeline.connector_id` **nullable**, `NULL = "non-connector pipeline (the Phase-7 Uploads pipeline)"` (`05` §"Pipeline → connector"; model `05` §2 `:105-106`). `PipelineRunTrigger.UPLOAD` already exists (`05` §1). `PipelineRun` has `trigger`, `status`, `documents_indexed`, `started_at`, `finished_at`, `charged_micros` (nullable) (`05` §3).
|
||||
- **Phase 6 engine refuses uploads**: `if pipeline.connector_id is None: return fail(run, "uploads pipeline has no fetch executor")` (`06` §1) — confirms uploads must be recorded by the route, **not** enqueued to `run_pipeline`.
|
||||
- **Chat tool** `get_pipeline_runs` (`06` §7) selects `PipelineRun` joined to `Pipeline` by `workspace_id` — so upload runs surface to the agent for free once they exist.
|
||||
- **De-dup guard** (`06` §6) only touches **connector-backed** pipelines (`when connector_id is set`); the Uploads pipeline (`connector_id NULL`) is untouched by it. The **scheduler** (`06` §5 `_claim_due`) ignores the Uploads pipeline **two ways** — `schedule_cron IS NOT NULL` (NULL cron) **and** `connector_id IS NOT NULL` (the defense-in-depth filter `06` added at Phase 7's request). Both confirmed safe.
|
||||
|
||||
### Document model
|
||||
|
||||
- `Document` — `db.py:1348`: `document_type` (`:1352`, `DocumentType.FILE`), `unique_identifier_hash` (`:1365`), `folder_id` **nullable** (`:1389`, root/unfiled), `created_by_id` (`:1397`), plus `search_space_id`/`status`. KB membership = a `Document` row exists; uploads always create one ⇒ uploads are always "in the KB" by construction.
|
||||
|
||||
## Target design
|
||||
|
||||
### 1. `get_or_create_uploads_pipeline(session, workspace_id, created_by_id)` — `app/pipelines/uploads.py`
|
||||
|
||||
```python
|
||||
async def get_or_create_uploads_pipeline(session, *, workspace_id: int, created_by_id) -> Pipeline:
|
||||
# fast path
|
||||
existing = (await session.execute(
|
||||
select(Pipeline).where(Pipeline.workspace_id == workspace_id,
|
||||
Pipeline.connector_id.is_(None)))).scalar_one_or_none()
|
||||
if existing:
|
||||
return existing
|
||||
# race-safe create (partial unique index ux_pipelines_uploads_per_workspace)
|
||||
stmt = (pg_insert(Pipeline.__table__)
|
||||
.values(workspace_id=workspace_id, connector_id=None, name="Uploads",
|
||||
save_to_kb=True, enabled=True, schedule_cron=None,
|
||||
created_by_id=created_by_id, config={})
|
||||
.on_conflict_do_nothing(index_elements=[Pipeline.workspace_id],
|
||||
index_where=Pipeline.connector_id.is_(None)))
|
||||
await session.execute(stmt)
|
||||
await session.flush()
|
||||
return (await session.execute(
|
||||
select(Pipeline).where(Pipeline.workspace_id == workspace_id,
|
||||
Pipeline.connector_id.is_(None)))).scalar_one()
|
||||
```
|
||||
|
||||
- Lazy (first upload), idempotent, race-safe. `created_by_id` = the uploading user (becomes the pipeline's creator metadata; `SET NULL` on member deletion per Phase 5).
|
||||
- Lives in the `app/pipelines/` package (created in Phase 6) so the upload route imports it without a route↔route dependency.
|
||||
- **Implementation gotcha — core insert bypasses ORM defaults.** `Pipeline.updated_at`/`created_at` use **Python-side** `default=`/`onupdate=` callables (`05` §2 `:114-115`), which a core `pg_insert` does **not** fire. The `.values(...)` must set `created_at`/`updated_at` explicitly (e.g. `now`) — or implement get-or-create as **ORM-insert + `flush` inside a `session.begin_nested()` SAVEPOINT, catching `IntegrityError` → rollback savepoint → re-`SELECT`** (avoids the core/ORM default mismatch entirely; the SAVEPOINT keeps the route's outer transaction intact). Either is fine; the SAVEPOINT form is closer to the ORM conventions used elsewhere.
|
||||
|
||||
### 2. Register an upload run (shared helper) — `app/pipelines/uploads.py`
|
||||
|
||||
```python
|
||||
async def record_upload_run(session, *, workspace_id, created_by_id, documents_indexed: int) -> PipelineRun:
|
||||
pipeline = await get_or_create_uploads_pipeline(session, workspace_id=workspace_id,
|
||||
created_by_id=created_by_id)
|
||||
now = datetime.now(UTC)
|
||||
run = PipelineRun(pipeline_id=pipeline.id, trigger=PipelineRunTrigger.UPLOAD,
|
||||
status=PipelineRunStatus.SUCCEEDED,
|
||||
documents_indexed=documents_indexed, started_at=now, finished_at=now)
|
||||
session.add(run)
|
||||
return run # caller commits (folds into the route's existing commit)
|
||||
```
|
||||
|
||||
- **Terminal-at-creation** (locked decision): records the upload event with the accepted file count. `crawls_*` / `charged_micros` left NULL (not a crawl).
|
||||
- Caller does **not** await any Celery task — the run is purely the route's own DB write, folded into the route's existing transaction (no extra round-trips, no new failure mode that could block the upload).
|
||||
|
||||
### 3. Wire `POST /documents/fileupload`
|
||||
|
||||
After Phase 2 dispatch (`documents_routes.py:307-316`), before building the response, add:
|
||||
|
||||
```python
|
||||
if files_to_process: # only when something was actually queued
|
||||
await record_upload_run(session, workspace_id=workspace_id,
|
||||
created_by_id=str(user.id),
|
||||
documents_indexed=len(files_to_process))
|
||||
await session.commit()
|
||||
```
|
||||
|
||||
- Count = `len(files_to_process)` (the newly-created/re-queued docs), **excluding** `skipped_duplicates` (already-READY no-ops). An all-duplicate upload records **no** run (nothing ingested).
|
||||
- **Best-effort, non-blocking — and the try/except is mandatory, not cosmetic.** The route has an **outer `except Exception` → `session.rollback()` → HTTP 500** (`documents_routes.py:328-332`). If `record_upload_run` raised unguarded, it would hit that handler and **500 an upload whose documents are already committed + dispatched**. So the call MUST be wrapped in its **own inner `try/except`** that swallows: on failure `await session.rollback()` (discards only the run insert — documents were committed at `:278`/`:304`) + `logger.warning(...)`, then fall through to the response. A run-record write must **never** fail the upload. (Mirrors the `store_document_file` best-effort pattern at `:298-303`.)
|
||||
|
||||
### 4. Wire `POST /documents/folder-upload`
|
||||
|
||||
Folder-upload dispatches one batch task, so record the run at request time with the file count (`documents_routes.py` before the `return` at `:1743`):
|
||||
|
||||
```python
|
||||
await record_upload_run(session, workspace_id=workspace_id,
|
||||
created_by_id=str(user.id), documents_indexed=len(files))
|
||||
await session.commit() # best-effort / logged, same as §3
|
||||
```
|
||||
|
||||
- Uses `len(files)` (folder-upload doesn't pre-dedup in the route; dedup happens inside the batch task).
|
||||
- **Optional richer accounting (deferred):** because this path *has* a barrier (`index_uploaded_folder_files_task`), a later enhancement could create the run `pending` here and flip it `succeeded`/`failed` from inside the task (it already manages a notification lifecycle). MVP keeps it uniform with `fileupload` (terminal-at-request). Flagged, not built.
|
||||
|
||||
### 5. Migration (one small file) — partial unique index
|
||||
|
||||
```sql
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_pipelines_uploads_per_workspace
|
||||
ON pipelines (workspace_id) WHERE connector_id IS NULL;
|
||||
```
|
||||
|
||||
- Chain after the then-current head (verify `alembic heads`). Enforces the singleton + powers `ON CONFLICT`. `downgrade()` drops it.
|
||||
- **Pre-flight (safety):** if any workspace somehow already has >1 `connector_id IS NULL` pipeline (shouldn't, pre-Phase-7), the `CREATE UNIQUE INDEX` fails — the migration docstring notes a dedup step (keep the oldest, delete extras) just in case. Expectation: none exist (uploads pipelines are introduced *here*).
|
||||
|
||||
### 6. KB-save-secondary verification (no new code, a guard)
|
||||
|
||||
- Uploads pipeline hardcodes `save_to_kb=True` (§1) — uploads always land in the KB (a `Document` row), matching the umbrella invariant.
|
||||
- Confirm Phase 5's connector-pipeline default is still `save_to_kb=False` (`05` §2) — i.e. KB-save is **opt-in for connectors, mandatory for uploads**. Add a test asserting both defaults so a future change can't silently make connector KB-save the default.
|
||||
|
||||
### 7. Guard the generic Phase-5 CRUD against the system Uploads pipeline (the gap)
|
||||
|
||||
The Uploads pipeline is a **system-managed** row (`connector_id IS NULL`, auto-created by §1). Phase 5's generic `/pipelines` CRUD doesn't know it's special, which opens three footguns — Phase 7 closes them:
|
||||
|
||||
| Phase-5 endpoint | Problem if left generic | Phase-7 guard |
|
||||
|------------------|-------------------------|---------------|
|
||||
| `POST /pipelines` | Phase 5 **allows `connector_id=None`** (`05` §7 create + test). A user could hand-create a rogue/duplicate non-connector pipeline. | **Reject `connector_id is None` with 4xx** — non-connector pipelines are system-managed (auto-created on upload), not user-created. This **supersedes** Phase 5's "`connector_id=None` → allowed" create test (update it). The partial unique index (§5) is the DB backstop. |
|
||||
| `POST /pipelines/{id}/run` | Manual-running the Uploads pipeline enqueues `run_pipeline`, which the Phase-6 engine **fails** (`"uploads pipeline has no fetch executor"`) → a confusing **FAILED** manual run every click. | **Reject `connector_id IS NULL` with 4xx** ("uploads are triggered by uploading files, not a manual run") — before creating the run row / enqueuing. |
|
||||
| `PUT /pipelines/{id}` | Setting `schedule_cron` on the Uploads pipeline makes the scheduler eligible-by-cron → **perpetually-failing scheduled runs** (engine refuses null-connector). Editing `save_to_kb`/`destination_folder_id` is silently **ineffective** (uploads are route-driven, not engine-driven). | **Reject schedule edits (`schedule_cron`/`schedule_timezone`) on `connector_id IS NULL` pipelines with 4xx.** (`connector_id` is already immutable per Phase 5.) Optionally reject all PUTs to the Uploads pipeline; minimally block the scheduling fields. Belt-and-suspenders: Phase 6's `_claim_due` also filters `connector_id IS NOT NULL` (`06` §5). |
|
||||
| `DELETE /pipelines/{id}` | Deleting the Uploads pipeline cascades its run history. | **Allowed** (MVP): it is **lazily re-created** on the next upload (§1); only past upload-run history is lost. Note in UI copy later; an optional hard block is a small follow-up. |
|
||||
|
||||
Net: after Phase 7, the only non-connector pipeline that can exist is the **system Uploads pipeline**, it is never user-created / manually-run / scheduled, and the scheduler ignores it two ways (NULL cron *and* the `connector_id IS NOT NULL` filter).
|
||||
|
||||
## Work items
|
||||
|
||||
1. **`app/pipelines/uploads.py`**: `get_or_create_uploads_pipeline` (race-safe via `ON CONFLICT`) + `record_upload_run` (terminal upload run).
|
||||
2. **Migration**: partial unique index `ux_pipelines_uploads_per_workspace ON pipelines(workspace_id) WHERE connector_id IS NULL`; docstring dedup pre-flight note; `downgrade` drops it.
|
||||
3. **Wire `fileupload`** (`documents_routes.py:126-332`): call `record_upload_run(documents_indexed=len(files_to_process))` after dispatch, in an **inner** best-effort try/except (must not reach the route's outer 500 handler), only when `files_to_process` non-empty.
|
||||
4. **Wire `folder-upload`** (`documents_routes.py:1598-1748`): call `record_upload_run(documents_indexed=len(files))` before the response, same inner best-effort guard.
|
||||
5. **Guard the generic CRUD** (§7): `POST /pipelines` rejects `connector_id is None`; `/run` rejects `connector_id IS NULL`; `PUT` rejects schedule edits on `connector_id IS NULL`. Update Phase 5's "`connector_id=None` → allowed" create test accordingly. (Phase 6's `_claim_due` `connector_id IS NOT NULL` filter is the scheduler backstop.)
|
||||
6. **Tests** (below).
|
||||
|
||||
## Tests
|
||||
|
||||
- **First upload creates the pipeline**: `fileupload` into a fresh workspace → exactly one `connector_id IS NULL` pipeline named "Uploads" with `save_to_kb=True`; a `PipelineRun(trigger=upload, status=succeeded, documents_indexed=N)`.
|
||||
- **Subsequent uploads reuse it**: a second `fileupload` → still one Uploads pipeline, two upload runs. Concurrency: two simultaneous first-uploads (race) → still exactly one pipeline (partial-unique + `ON CONFLICT`), two runs.
|
||||
- **Duplicate-only upload**: re-uploading already-READY files (`skipped_duplicates == len(files)`, `files_to_process == []`) → **no** run recorded.
|
||||
- **folder-upload**: records one upload run with `documents_indexed=len(files)`; the existing batch task/folder behaviour is unchanged.
|
||||
- **No crawl billing**: upload runs have `charged_micros IS NULL`, `crawls_* IS NULL`; no `web_crawl` `TokenUsage` row is written.
|
||||
- **Engine still refuses uploads**: enqueuing `run_pipeline` for an Uploads pipeline's run (or a NULL-connector pipeline) → Phase-6 engine fails it cleanly (regression guard for `06` §1). Confirms uploads are never engine-executed.
|
||||
- **Scheduler ignores uploads (two ways)**: the Phase-6 tick never selects the Uploads pipeline — both because `schedule_cron IS NULL` and because `_claim_due` filters `connector_id IS NOT NULL`. Add a case: even with a cron forced onto a NULL-connector row (direct DB), `_claim_due` still skips it.
|
||||
- **CRUD guards (§7)**: `POST /pipelines` with `connector_id=None` → 4xx; `POST /pipelines/{uploads_id}/run` → 4xx (no FAILED run created); `PUT` setting `schedule_cron` on the Uploads pipeline → 4xx; `DELETE` of the Uploads pipeline → allowed, and the next upload re-creates it.
|
||||
- **Chat tool surfaces uploads**: `get_pipeline_runs` returns the upload runs for the workspace (trigger=upload), scoped correctly.
|
||||
- **KB defaults guard**: Uploads pipeline `save_to_kb=True`; a connector pipeline created without `save_to_kb` defaults `False` (§6).
|
||||
- **Best-effort isolation**: a forced failure inside `record_upload_run` does not fail the upload request (documents still created + dispatched, 2xx returned).
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Run is an upload *event*, not per-file ETL truth.** `status=succeeded` + `documents_indexed=<accepted count>` is stamped at request time; individual files may still fail ETL afterward (visible on `Document.status`, not reflected back onto the run). Accepted for MVP — accurate per-file roll-up needs the deferred `documents.pipeline_run_id` provenance column. Documented; the folder-upload barrier is the natural place to add real status later (§4).
|
||||
- **Singleton Uploads pipeline ⇒ no per-upload folder on the pipeline.** `destination_folder_id` stays NULL; folder placement stays in the upload code. Fine because the upload run is descriptive, not engine-driven. If product later wants "uploads to folder X" as a first-class pipeline, that's a connector-less pipeline variant (additive).
|
||||
- **Partial-unique index assumes `connector_id IS NULL ⇔ Uploads`.** True for MVP (Phase 5 reserved NULL for Uploads, and Phase 6 fails NULL-connector runs). If a *second* kind of connector-less pipeline is ever introduced, replace the index predicate with an explicit `kind`/discriminator column (additive migration) — flagged so the NULL-overload doesn't silently block it.
|
||||
- **Best-effort run write can be skipped on error.** If `record_upload_run` raises (e.g. transient DB error) the upload still succeeds but no run is logged — an audit gap, never a data-loss or user-facing failure. Acceptable: the `Document` rows (the actual KB content) are the source of truth; the run is supplementary history.
|
||||
- **No backfill for historical uploads.** Pre-Phase-7 uploads have no upload runs (pipeline is created lazily on the next upload). Accepted — run history starts accumulating from rollout; no migration backfills synthetic runs.
|
||||
- **Upload run doesn't capture the uploader.** `PipelineRun` has no user column (Phase 5), so the run records *what/when/how-many*, not *who* — the per-document uploader is still on `Document.created_by_id`, and the pipeline's `created_by` is just whoever triggered the first-ever upload. If per-run attribution is wanted, add a nullable `triggered_by_id` to `pipeline_runs` (additive Phase-5 model change); deferred.
|
||||
- **`enabled=True` on a never-scheduled pipeline.** The Uploads pipeline is `enabled=True` with NULL cron — harmless (scheduler ignores it via NULL cron *and* the `connector_id IS NOT NULL` filter); `enabled` only ever gates scheduling, which uploads never use. Left `True` so a future UI can show it as an active pipeline.
|
||||
- **CRUD-guard coupling to Phase 5.** Phase 7 tightens Phase 5's create contract (`connector_id=None` now rejected) and adds `/run` + `PUT` guards. This is a documented evolution (Phase 7 ships after 5/6); the Phase-5 create test is updated here, not left contradictory.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **`documents.pipeline_run_id` provenance** (per-file → run linkage, accurate ETL roll-up, folder-upload barrier-driven status) → post-MVP (touches `documents`).
|
||||
- **Engine execution of uploads** — intentionally never; uploads are route-recorded (`06` fails NULL-connector runs by design).
|
||||
- **Migrating connector periodic indexing into pipelines** — still COEXIST (Phase 5 decision); not revisited here.
|
||||
- **Public pay-as-you-go API / MCP KB server** → post-MVP (umbrella "Deferred").
|
||||
- **Frontend** — surfacing the Uploads pipeline + upload runs in the Pipelines UI, "uploads always saved to KB" copy → frontend umbrella.
|
||||
|
|
@ -1,128 +0,0 @@
|
|||
# Phase 4 → end revamp — Overview & phase index (CI pivot · WIP)
|
||||
|
||||
> **What this is.** The CI pivot re-architects the old "pipeline" Phases `04`–`07` into small,
|
||||
> single-responsibility domains, now packaged as **4 phases (04–07)** that mirror the slots they replace.
|
||||
> This file is the map + the reconciliation + the subplan index. **End-to-end flow diagrams (stateless &
|
||||
> stateful paths) live in `00b-pipeline-diagrams.md`.**
|
||||
> **Scope guardrail:** Phases **1–3 are SHIPPED/FIXED** (rename/DB, proxy/captcha/stealth, crawl billing).
|
||||
> The revamp is **Phase 4 → end only**. We do **not** touch 1–3.
|
||||
>
|
||||
> **Citation verification — 2026-06-30 (principal-engineer pass).** All load-bearing code citations in this
|
||||
> revamp were re-checked against `surfsense_backend`. Confirmed accurate: the schedule selector
|
||||
> (`selector.py` — `FOR UPDATE SKIP LOCKED`, `next_fire_at`, self-heal, `catchup=False`, `croniter`), the
|
||||
> `AutomationRun` model/fields, `format_to_structured_document(exclude_metadata=True)` content-hashing, the
|
||||
> MCP routing gap (`constants.py::CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS`), the connector enum, and the
|
||||
> `MANUAL` trigger placeholder. **Corrected during this pass:** (a) automations have **no** delivery/
|
||||
> notification path and `automation_runs.output` is never written — CI alerts wire to `app/notifications/`
|
||||
> (`06`/`05b`); (b) the automation PENDING-gate is **not atomic** — the per-Tracker lock is the real
|
||||
> concurrency guard (`06`/`05b`); (c) folder upload uses `root_folder_id` (not `destination_folder`) and KB
|
||||
> folder scoping goes through `referenced_document_ids → SearchScope.document_ids` (`05b`); (d) the billable
|
||||
> predicate is `SUCCESS and outcome.result`, to be single-sourced in the executor (`04a`).
|
||||
|
||||
## The two products
|
||||
|
||||
```
|
||||
PRODUCT A — stateless utility Phase 04 (Capabilities + Access)
|
||||
call a verb → get data → bill. Nothing persists.
|
||||
|
||||
PRODUCT B — decision-grounded CI Phase 05 (Intelligence + Timeline) (+ 06 Triggers, 07 Orchestration)
|
||||
a Tracker accumulates structured signal over time. The Timeline is the moat.
|
||||
```
|
||||
|
||||
## The phases (this revamp)
|
||||
|
||||
| Phase | Subplan(s) | Domain(s) | One line | Ships |
|
||||
|-------|-----------|-----------|----------|-------|
|
||||
| **04 — Capabilities & Access** | `04a-capabilities.md` · `04b-access.md` | ① + ② | typed verbs over Acquisition + the chat/REST/MCP doors | **Product A** (revenue day one) |
|
||||
| **05 — Intelligence & Timeline** | `05a-timeline.md` · `05b-intelligence.md` | ④ + ③ | the Tracker, locked schema, hot loop + the 3-store delta moat | **Product B engine** |
|
||||
| **06 — Triggers** | `06-triggers.md` | ⑤ | the pluggable refresh clock; recurrence+delivery = optional CI action on automations | recurrence + alerts |
|
||||
| **07 — Orchestration** | `07-orchestration.md` | ⑥ | the `intelligence_agent` CI-expert subagent (intent routing, verb chains, Tracker crafting) | the human-facing CI experience |
|
||||
|
||||
> **Build order = phase order.** `04a → 04b` (unblocks everything; ships Product A) → `05a → 05b`
|
||||
> (state before the loop that writes it) → `06` (drive it) → `07` (front it). `04a` first because every
|
||||
> other phase calls into the capability registry.
|
||||
|
||||
## Domain ↔ phase map
|
||||
|
||||
```
|
||||
FIXED (Phases 1–3) OUR SCOPE (Phase 04 → 07)
|
||||
┌─────────────────────────────────┐ ┌──────────────────────────────────────────────┐
|
||||
│ Acquisition │ │ 04a Capabilities typed verbs over Acquisition │
|
||||
│ proprietary/web_crawler │◄──┤ 04b Access chat · REST · MCP doors │
|
||||
│ CrawlOutcome · billing (03c) │ │ 05a Timeline 3-store delta state (moat) │
|
||||
│ │ │ 05b Intelligence Tracker · schema · hot loop │
|
||||
│ │ │ 06 Triggers pluggable refresh clock │
|
||||
│ │ │ 07 Orchestration CI-expert subagent + tools │
|
||||
└─────────────────────────────────┘ └──────────────────────────────────────────────┘
|
||||
data engine (untouched) stateless 04 → stateful 05 , driven by 06 , fronted by 07
|
||||
```
|
||||
|
||||
## Reconciliation with the old plans
|
||||
|
||||
| Old plan | Fate | Where it goes |
|
||||
|----------|------|---------------|
|
||||
| `04a-connector-category.md` | **demoted to hygiene** | the genuine MCP-routing fix survives in `04b` (consume-user-MCP); the taxonomy work is not core |
|
||||
| `04b-source-discovery.md` | **absorbed** | becomes the `web.discover` capability in `04a` |
|
||||
| `05-pipelines-model.md` | **dissolved** | the "pipeline" concept → the `Tracker` (`05b`) + Timeline tables (`05a`); no `pipeline`/`pipeline_runs` |
|
||||
| `06-pipelines-exec.md` | **dissolved** | execution → the hot loop (`05b`); scheduling/runs/delivery → **reuse automations** via a CI action (`06`) — its selector + `AutomationRun`, **not** a rebuilt cron |
|
||||
| `07-upload-pipeline-kb.md` | **dropped (crawl→KB + uploads-as-pipeline audit)** | "don't index *crawled* data" holds. **User file-upload-to-KB remains a pre-existing, untouched feature.** For CI, uploads are routed to a dedicated **folder** and may feed the judge (`05b`) |
|
||||
|
||||
**Net:** the old `05/06/07` pipeline+KB stack is replaced by the new `04`–`07`. KB indexing of *crawled*
|
||||
data is out; *user uploads* still work (and gain a CI context-folder role).
|
||||
|
||||
## Confirmed decisions (locked across phases)
|
||||
|
||||
- **Natural language is the only human-facing surface.** Users never name verbs; the chat agent
|
||||
understands intent, composes verbs, answers in plain language. An **intent router** classifies one-shot
|
||||
(A) vs standing-concern (B) from the wording (one clarifying question if ambiguous). Raw verbs live only
|
||||
on the REST/MCP dev doors.
|
||||
- MVP surface = the **general web crawler + search** (`web.scrape` + `web.discover`), exposed chat + REST (MCP fast-follow). **Platform-specific scrapers are a family of individual endpoints added incrementally — none (incl. Google Maps) is committed for MVP;** each is just another verb → agent tool + dev API-key REST endpoint.
|
||||
- Verbs namespaced per platform (`web.*`, and `<platform>.*` per scraper, e.g. `maps.*` as an example); `web.scrape` takes a URL **array** (inline-or-job).
|
||||
- **Don't index** crawled data into the KB; the Timeline (deltas) is the only persisted CI state.
|
||||
- CI is **decoupled from automations** (automations = one optional Trigger adapter).
|
||||
- Intelligence: **agent-designed schema** (sample-grounded, human-locked, versioned) in MVP; single entity
|
||||
per Tracker; materiality = code thresholds (numeric/clear) + agent on ambiguous; content-hash pre-check.
|
||||
- Orchestration is a first-class deliverable: a **net-new CI-expert subagent** (`intelligence_agent`) on
|
||||
the existing chat runtime, with registry-backed verb tools + Tracker/Timeline tools.
|
||||
- **Recurrence = a CI *action* on the existing automations** (not a new scheduler, not a new shape):
|
||||
reuses the hardened schedule selector + `AutomationRun` (audit). The PENDING-gate is **non-atomic**, so a
|
||||
**per-Tracker lock** is the real concurrency guard. **Alert delivery is separate** — automations have no
|
||||
delivery path (`run.output` unwritten); alerts ride `app/notifications/`. CI core still runs with **zero**
|
||||
automations dependency (manual/agent/cron).
|
||||
- **No new run table:** refresh audit/idempotency ride `AutomationRun` or the chat job record; billing
|
||||
idempotency is per-capability-call + the content-hash pre-check. Only the Timeline is new state.
|
||||
- **Billing is delegated to the billing service:** verbs declare a `billing_unit`; `03c` is the first
|
||||
provider; `maps.*` / `web.discover` register their own units there.
|
||||
- **Any platform actor** (e.g. a Maps actor — illustrative, **not committed for MVP**) is net-new (not in
|
||||
shipped Phases 1–3) and would be designed/built as a **separate effort**; per-platform verbs (`<platform>.*`,
|
||||
e.g. `maps.*`) are contracts against it and don't block this design.
|
||||
- **CI context files:** files uploaded in a CI chat go into the **KB as normal** (indexed), routed to a
|
||||
dedicated `Folder`, and may feed the materiality judge via KB retrieval. ("Don't index" applies only to
|
||||
*crawled* data.)
|
||||
- **Names:** primitive = `Tracker`, subagent = `intelligence_agent` (Intelligence Agent).
|
||||
- License: new domains **Apache-2**; the moat stays in proprietary Acquisition + the Maps extractor.
|
||||
|
||||
## Cross-phase open forks (need your call)
|
||||
|
||||
1. Schema **review-&-lock** UX before frontend exists — pure-chat confirmation for MVP? (`05b`)
|
||||
2. Timeline ORM home — `app/db.py` vs dedicated `app/timeline/`. (`05a`)
|
||||
3. Recurrence = CI **action** on automations (default) vs a thin CI automation **shape** (fallback). (`06`)
|
||||
|
||||
*(Resolved: the "built-in scheduler tick" fork — reuse the automations schedule selector via a CI action
|
||||
instead of building a tick. See `06`.)*
|
||||
|
||||
> **Per-phase implementation-time questions** (refresh failure path, first-run baseline, tracker lifecycle,
|
||||
> mid-run balance, concurrency lock, delivery payload shape) are intentionally **not** resolved here — they
|
||||
> don't change any boundary, contract, or data shape, and are settled when each phase is built.
|
||||
|
||||
## Subplan index (revamp)
|
||||
|
||||
| Phase | Subplan file | Status |
|
||||
|-------|--------------|--------|
|
||||
| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams |
|
||||
| 04 | `04a-capabilities.md` | drafted |
|
||||
| 04 | `04b-access.md` | drafted |
|
||||
| 05 | `05a-timeline.md` | drafted |
|
||||
| 05 | `05b-intelligence.md` | drafted |
|
||||
| 06 | `06-triggers.md` | drafted |
|
||||
| 07 | `07-orchestration.md` | drafted |
|
||||
|
|
@ -1,204 +0,0 @@
|
|||
# Pipeline diagrams — end-to-end (stateless & stateful paths)
|
||||
|
||||
> Visual companion to `00-overview.md`. Shows the whole CI pipeline across Phases 04–07, with the
|
||||
> **stateless** (Product A) and **stateful** (Product B) paths called out explicitly. Phase refs:
|
||||
> `04a` Capabilities · `04b` Access · `05a` Timeline · `05b` Intelligence · `06` Triggers · `07` Orchestration.
|
||||
|
||||
## 1. Domain map — the two products
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph FIXED["FIXED · Phases 1-3 (shipped, untouched)"]
|
||||
ACQ["Acquisition<br/>proprietary/web_crawler<br/>CrawlOutcome"]
|
||||
MET["Metering · 03c<br/>WebCrawlCreditService"]
|
||||
ID["Identity / Tenancy<br/>+ API keys"]
|
||||
RT["Chat runtime<br/>deepagents · deliverable_wait"]
|
||||
end
|
||||
|
||||
subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"]
|
||||
CAP["04a Capabilities<br/>typed verbs + registry + job store"]
|
||||
ACC["04b Access<br/>chat · REST · MCP doors"]
|
||||
INT["05b Intelligence<br/>Tracker · schema · hot loop"]
|
||||
TL["05a Timeline<br/>3-store delta state (moat)"]
|
||||
TRG["06 Triggers<br/>refresh clock"]
|
||||
ORC["07 Orchestration<br/>intelligence_agent subagent"]
|
||||
end
|
||||
|
||||
ACQ --> CAP
|
||||
MET --> CAP
|
||||
CAP --> ACC
|
||||
ID --> ACC
|
||||
RT --> ORC
|
||||
ACC --> ORC
|
||||
CAP --> INT
|
||||
INT --> TL
|
||||
TRG --> INT
|
||||
ORC --> CAP
|
||||
ORC --> INT
|
||||
ORC --> TL
|
||||
|
||||
classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
|
||||
classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
|
||||
class CAP,ACC,INT,TL,TRG,ORC a;
|
||||
class ACQ,MET,ID,RT b;
|
||||
```
|
||||
|
||||
- **Product A (stateless):** `04a Capabilities + 04b Access` — call a verb, get data, bill, nothing persists.
|
||||
- **Product B (stateful):** `05b Intelligence + 05a Timeline` (driven by `06`, fronted by `07`) — a Tracker
|
||||
accumulates structured signal; the Timeline is the moat.
|
||||
|
||||
## 2. The fork — intent router (the only human-facing decision)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
U(["User speaks in natural language"]) --> R{"Intent router<br/>(in intelligence_agent prompt, 07)"}
|
||||
R -->|"compare / find / what is / pull / right now"| A["ONE-SHOT<br/>Product A · stateless"]
|
||||
R -->|"watch / track / notify me / every week / over time"| B["STANDING CONCERN<br/>Product B · stateful"]
|
||||
R -->|ambiguous| Q["Ask ONE question:<br/>'just once, or keep watching?'"]
|
||||
Q -->|once| A
|
||||
Q -->|keep watching| B
|
||||
A --> P1["section 3 · Stateless path"]
|
||||
B --> P2["section 4 · Stateful path"]
|
||||
```
|
||||
|
||||
Raw verbs are exposed only on REST/MCP (devs/external agents). Humans never name a verb.
|
||||
|
||||
## 3. Stateless path (Product A) — end to end
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant U as User / Dev
|
||||
participant D as Door 04b (chat · REST · MCP)
|
||||
participant REG as Registry 04a
|
||||
participant EX as Verb executor 04a
|
||||
participant ACQ as Acquisition / Maps actor
|
||||
participant BILL as Billing service (03c)
|
||||
participant JOB as Job record (+Celery)
|
||||
|
||||
U->>D: intent (chat) OR raw verb call (REST/MCP)
|
||||
D->>D: validate input_schema · authn/authz · meter-gate
|
||||
D->>REG: resolve verb
|
||||
REG->>EX: same executor (all doors)
|
||||
alt fast / small input -> inline
|
||||
EX->>ACQ: crawl_url / search / place
|
||||
ACQ-->>EX: data
|
||||
EX->>BILL: charge billing_unit per success
|
||||
EX-->>D: envelope {status: completed, results}
|
||||
D-->>U: plain-language answer (chat) / JSON (REST)
|
||||
else slow / large input -> job
|
||||
EX->>JOB: dispatch, return {status: pending, job_id}
|
||||
D-->>U: chat: deliverable_wait polls status<br/>REST/MCP: GET /v1/jobs/{id}
|
||||
JOB->>ACQ: work loop
|
||||
JOB->>BILL: charge per success
|
||||
JOB-->>D: status -> completed (+ results)
|
||||
D-->>U: results inline / tracked card
|
||||
end
|
||||
Note over U,JOB: Nothing persists beyond chat history. No Tracker, no Timeline row.
|
||||
```
|
||||
|
||||
## 4. Stateful path (Product B) — setup, then refresh, then read
|
||||
|
||||
### 4a. Setup once — craft & lock the Tracker (05b)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
S0(["Standing concern detected"]) --> S1["Bind capability + input<br/>(maps.place X / web.scrape Y)"]
|
||||
S1 --> S2["Sample fetch · ONE real verb call<br/>(billed)"]
|
||||
S2 --> S3["Agent drafts definition from decision + sample:<br/>field_schema · materiality · identity_rule · notable_signals"]
|
||||
S3 --> S4{"Human reviews in chat"}
|
||||
S4 -->|add field X / tweak| S3
|
||||
S4 -->|looks good| S5["LOCK · versioned snapshot<br/>status: draft -> active"]
|
||||
S5 --> S6[("Tracker persisted<br/>+ tracked_entities row (05a)")]
|
||||
```
|
||||
|
||||
### 4b. The hot loop — refresh(tracker) (05b → writes 05a)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
T(["Trigger fires refresh(tracker) · 06"]) --> C1["1 · Crawl: call bound capability (04a)"]
|
||||
C1 --> C2{"2 · Content-hash<br/>== stored hash?"}
|
||||
C2 -->|identical| STOP["stamp last_checked_at · STOP<br/>(zero LLM cost)"]
|
||||
C2 -->|changed / first run| C3["3 · Fill: agent conforms raw -> locked field_schema<br/>(extras -> notable_signals)"]
|
||||
C3 --> C4["4 · Diff (code): new record vs entity_current_state -> raw deltas"]
|
||||
C4 --> C5{"5 · Judge each delta"}
|
||||
C5 -->|numeric / clear rule| C5a["code decides material/noise<br/>(free, reproducible)"]
|
||||
C5 -->|ambiguous / notable_signals| C5b["agent decides<br/>(1 LLM call; may read CI context folder)"]
|
||||
C5a --> C6{"material?"}
|
||||
C5b --> C6
|
||||
C6 -->|yes| W["6 · Append entity_changes row<br/>+ overwrite entity_current_state (05a)"]
|
||||
C6 -->|no| N["only bump last_checked_at<br/>(no change -> no row)"]
|
||||
W --> DONE([done])
|
||||
N --> DONE
|
||||
STOP --> DONE
|
||||
```
|
||||
|
||||
### 4c. Read / answer — straight from the Timeline (05a)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Q(["'Is competitor X pulling ahead?'"]) --> A["intelligence_agent · query_timeline(tracker_id)"]
|
||||
A --> TL[("entity_changes + entity_current_state")]
|
||||
TL --> ANS["Answer from stored deltas<br/>(NOT re-derived from chat history)"]
|
||||
```
|
||||
|
||||
## 5. Triggers — who calls refresh(tracker) (06)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
M["Manual<br/>'refresh now' (chat/REST)"] --> RF["refresh(tracker)<br/>headless unit of work · 05b"]
|
||||
AG["Agent<br/>calls refresh as a tool"] --> RF
|
||||
XC["External cron<br/>POST /v1/trackers/{id}/refresh"] --> RF
|
||||
CA["CI automation ACTION<br/>(existing automations)"] --> RF
|
||||
RF --> TLW[("writes Timeline · 05a")]
|
||||
|
||||
CA -. "schedule selector" .-> SEL["hardened cron selector"]
|
||||
CA -. "run record + idempotency" .-> AR["AutomationRun (PENDING-gate, non-atomic → + per-Tracker lock)"]
|
||||
RF -. "on material change" .-> DEL["alert via app/notifications (Zero-synced)"]
|
||||
|
||||
classDef opt fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
|
||||
class CA,SEL,AR opt;
|
||||
```
|
||||
|
||||
CI **core** (`refresh` + Timeline) has **zero** automations dependency — manual/agent/cron all work
|
||||
standalone. The CI **action** is the *optional* adapter that buys recurrence + audit. **Delivery is not
|
||||
from automations** (no such path) — alerts on material change ride the separate `app/notifications/`
|
||||
system; concurrency is guarded by a per-Tracker lock (the automation PENDING-gate is non-atomic).
|
||||
|
||||
## 6. The persisted state — Timeline data model (05a)
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
TRACKER ||--|| TRACKED_ENTITY : "MVP 1:1 (multi-ready)"
|
||||
TRACKED_ENTITY ||--|| ENTITY_CURRENT_STATE : "latest"
|
||||
TRACKED_ENTITY ||--o{ ENTITY_CHANGES : "append-only log"
|
||||
|
||||
TRACKER {
|
||||
uuid id
|
||||
text decision
|
||||
json capability_binding
|
||||
json definition_locked "field_schema·identity_rule·materiality (versioned)"
|
||||
text status "draft|active"
|
||||
}
|
||||
TRACKED_ENTITY {
|
||||
uuid id
|
||||
uuid workspace_id
|
||||
text entity_key "from identity_rule, unique per tracker"
|
||||
ts first_seen_at
|
||||
}
|
||||
ENTITY_CURRENT_STATE {
|
||||
uuid entity_id FK
|
||||
json fields "conforms to field_schema"
|
||||
text content_hash "powers step-2 pre-check"
|
||||
ts last_checked_at
|
||||
}
|
||||
ENTITY_CHANGES {
|
||||
uuid id
|
||||
ts captured_at
|
||||
json delta "{field:{from,to}}"
|
||||
text materiality "material|notable"
|
||||
text decided_by "code|agent (provenance)"
|
||||
text source_ref
|
||||
text note "why material"
|
||||
}
|
||||
```
|
||||
|
|
@ -1,198 +0,0 @@
|
|||
# Phase 4a — Capabilities (typed verbs over Acquisition)
|
||||
|
||||
> Part of **Phase 4 — Capabilities & Access** (the CI-pivot revamp of the old "pipeline" Phases 4–7).
|
||||
> Sibling: `04b-access.md` (the doors that expose these verbs). **Build first** — everything above
|
||||
> calls into it.
|
||||
> **Depends on** Phases 3a/3b/3c (SHIPPED): the `WebCrawlerConnector.crawl_url -> CrawlOutcome`
|
||||
> contract, the proxy provider, and the `WebCrawlCreditService` billing seam.
|
||||
> **Scope guardrail:** Phases 1–3 are SHIPPED/FIXED. This domain *builds on* them and never modifies
|
||||
> them. Locate foundation code by **symbol/grep**, not the line numbers cited.
|
||||
|
||||
## Objective
|
||||
|
||||
Turn the fixed Acquisition engine (+ a new Google Maps actor) into a **small set of typed, callable
|
||||
verbs** that every door (chat / REST / MCP) and the Intelligence hot loop (`05b`) consume
|
||||
**identically**. This is the core pivot of the old Phase 4: **replace "connector that ingests into the
|
||||
KB" with "capability that returns data."** A capability is a stateless function you *call and get data
|
||||
back from* — no `SearchSourceConnector` row to configure, no KB write, no schedule attached.
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
- `WebCrawlerConnector.crawl_url(url) -> CrawlOutcome` — single-URL fetch returning `{content,
|
||||
metadata, crawler_type}` on `SUCCESS`. `CrawlOutcomeStatus.SUCCESS` documents the billable signal, but
|
||||
**operationally every caller checks `status == SUCCESS and outcome.result`** (see `scrape_webpage.py` and
|
||||
`webcrawler_indexer.py`) — the predicate is duplicated, not one helper. **The executor (below) should
|
||||
single-source it** in a shared `is_billable(outcome)` helper (`app/proprietary/web_crawler/`).
|
||||
- `WebCrawlCreditService` (`03c`) — per-success metering with the static `billing_enabled()` /
|
||||
`successes_to_micros()` seam (`app/services/web_crawl_credit_service.py`).
|
||||
- Proxy / stealth / captcha tiers — all behind `crawl_url`; callers never see the tier (`03b`/`03d`/`03e`).
|
||||
- Web-search providers (SearXNG/Linkup/Baidu) and the source-discovery core from old `04b` — the
|
||||
substrate for `web.discover`.
|
||||
- **No Maps actor exists** — it is net-new, proprietary, and built as a separate effort (see Out of scope).
|
||||
|
||||
## Target design
|
||||
|
||||
### The verb set (MVP) — namespaced, nothing top-level
|
||||
|
||||
Two namespaces: **`web.*`** (generic crawler product) and **`<platform>.*`** (per-platform scrapers).
|
||||
Future platforms slot in as their own namespace (`maps.*`, `linkedin.*`, `amazon.*`, …).
|
||||
|
||||
> **Scope clarification (2026-06-30).** Platform-specific scrapers are a **family of individual scraping
|
||||
> endpoints**, **not** one committed integration — **no specific platform (incl. Google Maps) is committed
|
||||
> for MVP**; `maps.*` below is an **illustrative example** of the pattern. Each such scraper is *just
|
||||
> another capability verb*: adding one lights up (a) an **agent tool** (chat) and (b) a **dev-callable REST
|
||||
> endpoint behind the platform API key** — same executor, same billing, zero new machinery. **The MVP
|
||||
> builds only `web.scrape` + `web.discover` executors**; every `<platform>.*` row is a **contract stub**
|
||||
> demonstrating that the registry + generated doors (`04b`) make per-platform endpoints a drop-in.
|
||||
|
||||
| Verb | Input → Output | Mode | Executes over | Bills (03c) |
|
||||
|------|----------------|------|---------------|-------------|
|
||||
| `web.scrape(urls[])` | → `[{url, status, content, metadata}]` | inline-or-job | loop `crawl_url` | per success |
|
||||
| `web.discover(query, top_k)` | → `[{url, title, snippet, provider}]` | inline | search providers (SearXNG/Linkup/Baidu) | per search *(or free — open)* |
|
||||
| `maps.search(query, location)` | → `[place]` | job | Maps actor *(new, proprietary)* | per place |
|
||||
| `maps.place(place_id\|url)` | → `place` (structured) | inline | Maps actor *(new, proprietary)* | 1 |
|
||||
| `maps.reviews(place)` | → `[review]` (paged) | job | Maps actor *(new, proprietary)* | per page |
|
||||
|
||||
### `web.scrape` — one array-friendly verb (no separate "batch")
|
||||
|
||||
Scraping always **welcomes an array of URLs**. There is no `scrape_batch`. One verb, one mental model:
|
||||
"give me URLs, I give you per-URL content."
|
||||
|
||||
### Execution mode — a property of the **result**, not two verbs
|
||||
|
||||
A capability does **not** split into sync/async variants. Every call returns a **uniform envelope**:
|
||||
|
||||
```
|
||||
{ status: "completed" | "pending", job_id?: str, progress?: {done, total}, results?: [...], error?: ... }
|
||||
```
|
||||
|
||||
- **inline (fast path):** small/fast input finishes in-request → `completed` with `results` inline →
|
||||
zero polling. (single page, one place, a search)
|
||||
- **job (slow path):** large/slow input → `pending` + `job_id` → caller polls a status endpoint until
|
||||
`completed`. (batch scrape, Maps search/reviews)
|
||||
|
||||
So "sync" is simply *a job that finished instantly*. Same verb, same shape, no branching for the caller.
|
||||
|
||||
- **Threshold** (how many URLs / how heavy before async) is **configurable**.
|
||||
- A caller may pass `async: true` to **force** a job even for small input (agents that never block).
|
||||
- **Only infra this requires:** a thin **job record** (`id, status, progress, result_ref`) + the
|
||||
**existing Celery workers**. *Not* a resurrected `pipeline_runs`. Sync verbs need none of it.
|
||||
|
||||
### The capability registry — single source of truth
|
||||
|
||||
One registry entry per verb:
|
||||
|
||||
```
|
||||
Capability {
|
||||
name # dotted, e.g. "web.scrape", "maps.search"
|
||||
input_schema # Pydantic
|
||||
output_schema # Pydantic
|
||||
mode # can-complete-inline? + job-capable?
|
||||
executor # the async fn (wraps Acquisition / Maps actor)
|
||||
billing_unit # how Metering charges this call
|
||||
}
|
||||
```
|
||||
|
||||
The **three doors are generated from the registry** (`04b`), not hand-written three times — chat tool,
|
||||
REST route, MCP tool — and the **Intelligence** hot loop (`05b`) calls the *same* `executor` directly.
|
||||
Add a verb once → it lights up on every surface; the I/O contract cannot drift between surfaces.
|
||||
|
||||
### Billing — open to the billing service (not hardcoded per verb)
|
||||
|
||||
A capability only **declares a `billing_unit`** in its registry entry; charging is delegated to the
|
||||
**billing service**, so adding/repricing a unit is a billing-service concern, not a capability rewrite.
|
||||
`03c`'s `WebCrawlCreditService` is the **first provider**; new units register with the same service.
|
||||
|
||||
- `web.scrape` → per `SUCCESS` (existing `web_crawl` unit).
|
||||
- `maps.*` → register a new per-place / per-page unit **with the billing service** (same wallet).
|
||||
- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`, unchanged).
|
||||
- `web.discover` → register a per-search unit (or mark free) **with the billing service**.
|
||||
|
||||
The registry says *"this verb bills unit X"*; the **billing service owns what unit X costs.** Verbs
|
||||
stay pure; pricing stays pluggable.
|
||||
|
||||
> **Charge at the executor, not the turn accumulator (locked 2026-06-30).** The billable charge fires
|
||||
> **inside the capability executor** (this phase), so *every* caller meters identically — chat,
|
||||
> **automation/recurring**, REST, MCP, external-cron. This is deliberate: a code review confirmed the
|
||||
> automation path (`run_agent_task` → `agent_task`) establishes **no chat turn accumulator**, so the
|
||||
> existing `scrape_webpage` billing (which no-ops when `get_current_accumulator()` is `None`) would let
|
||||
> **automation-run crawls bill nothing**. The interactive chat **turn accumulator stays only as an
|
||||
> optional presentation fold** (so a chat turn still shows the crawl line on its bill) — it is **not** the
|
||||
> charging mechanism. Billing idempotency is **per capability call** (+ the `05b` content-hash pre-check),
|
||||
> not per run.
|
||||
|
||||
### Platform scrapers (illustrated with a Maps actor — example only, not committed)
|
||||
|
||||
- Each platform scraper is a **standalone endpoint**; the one shown here (a hypothetical Maps actor) is
|
||||
**Google-Maps-agnostic** — works for *any* place type (restaurants, gyms, hotels, retail, …).
|
||||
- Returns **typed structured objects** (`{name, rating, review_count, hours, price_level, …}`), not raw
|
||||
markdown — exactly what the Intelligence/Timeline layer needs to diff reliably.
|
||||
- The "restaurant" use case is an **Intelligence-domain wedge** (one Tracker), **not** a constraint on
|
||||
the capability. The capability never knows what decision it serves.
|
||||
|
||||
### Where it lives / license boundary
|
||||
|
||||
- New **Apache-2** package `app/capabilities/` (registry, schemas, executors, the thin job store).
|
||||
- It **imports from** the proprietary Acquisition engine but never moves into it.
|
||||
- The **Maps extractor logic is proprietary** (`app/proprietary/...`), consumed by the `maps.*`
|
||||
executors — same boundary rule Phase 3 set.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Registry**: `app/capabilities/` package — `Capability` dataclass + a registry that other domains import.
|
||||
2. **`web.scrape` executor**: loop `crawl_url` over a URL array; map each `CrawlOutcome` to the per-URL
|
||||
result shape; declare the `web_crawl` `billing_unit`.
|
||||
3. **`web.discover` executor**: wrap the `04b` source-discovery core (SearXNG/Linkup/Baidu, env-keyed);
|
||||
declare its `billing_unit` (or free).
|
||||
4. **Job store**: thin job record (`id, status, progress, result_ref`) + Celery dispatch for job-mode verbs.
|
||||
5. **Uniform envelope**: shared `completed|pending` result type returned by every executor.
|
||||
6. **`maps.*` contracts**: input/output schemas + executor stubs against the (separate) Maps actor.
|
||||
7. **Billing seam**: `billing_unit` declaration honored via the billing service (no per-verb price code).
|
||||
|
||||
## Tests
|
||||
|
||||
- **Envelope**: a small `web.scrape` returns `completed` inline; a large/`async:true` one returns
|
||||
`pending` + `job_id`; the job completes to `results`.
|
||||
- **Array semantics**: `web.scrape([a, b, c])` returns one per-URL row each; partial failures don't fail
|
||||
the batch.
|
||||
- **Billing**: each `SUCCESS` bills exactly one `web_crawl` unit via the billing service; `EMPTY`/`FAILED`
|
||||
free; disabling the billing flag makes it a no-op.
|
||||
- **Registry → executor parity**: the Intelligence loop and a door hit the *same* executor for the same verb.
|
||||
- **`web.discover`**: returns `{url,title,snippet,provider}`; self-disables when no provider env key is set.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Maps actor is a hard external dependency** for `maps.*` — contracts ship now, executors light up when
|
||||
the actor lands (doesn't block `web.*`).
|
||||
- **Job store vs no store**: a thin job record is unavoidable for slow verbs; keep it minimal so it never
|
||||
grows back into `pipeline_runs`.
|
||||
- **Per-success billing on batches** can exhaust a wallet mid-run — pre-check is an upper bound; exact
|
||||
mid-run behavior is an implementation-time call.
|
||||
|
||||
## Resolved decisions
|
||||
|
||||
1. Verbs: `web.scrape` · `web.discover` · `maps.search` · `maps.place` · `maps.reviews`. Namespaced; nothing top-level.
|
||||
2. One array-friendly `web.scrape` (no separate batch verb).
|
||||
3. Execution mode = result property (uniform `completed`/`pending` envelope), not separate verbs.
|
||||
4. Build the thin job model now (Maps search / batch scrape force it).
|
||||
5. One capability registry → generates chat/REST/MCP doors (`04b`) + feeds the Intelligence executor (`05b`).
|
||||
6. `app/capabilities/` Apache-2; Maps extractor proprietary; billing **delegated to the billing service**
|
||||
(verbs declare a `billing_unit`; `03c` is the first provider, new units register there).
|
||||
7. Connector → capability is a **replacement** for these data sources, not an extension.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **The doors** (chat/REST/MCP adapters, auth, metering gate) → `04b`.
|
||||
- **Stateful accumulation** (Tracker/Timeline) → `05a`/`05b`; capabilities stay stateless.
|
||||
- **The Google Maps actor** is **net-new** (not part of shipped Phases 1–3) — designed/built as a
|
||||
**separate effort** (incl. sourcing legality). `maps.*` verbs are contracts against it; this domain
|
||||
doesn't block on it.
|
||||
- **Recursive site crawl**, a generic **`web.extract`** verb, and additional platform namespaces
|
||||
(`linkedin.*`, `amazon.*`) — deferred.
|
||||
- **04a (old, connector taxonomy)** → demoted to backward-compat hygiene; the BYO-`MCP_CONNECTOR` routing
|
||||
fix belongs to `04b` (consume-user-MCP). **04b (old, source-discovery)** → absorbed here as `web.discover`.
|
||||
|
||||
## Open questions (carry forward)
|
||||
|
||||
- Default async threshold for `web.scrape`.
|
||||
- Whether `web.discover` is metered or free.
|
||||
|
|
@ -1,136 +0,0 @@
|
|||
# Phase 4b — Access / Surfaces (chat · REST · MCP doors)
|
||||
|
||||
> Part of **Phase 4 — Capabilities & Access**. Sits on top of `04a-capabilities.md` (the registry it exposes).
|
||||
> **Build after** `04a`. Together, `04a + 04b` ship **Product A** (stateless utility) — revenue day one.
|
||||
> **Depends on** SHIPPED infra: Identity/Tenancy, **API keys**, the chat agent + its tool registry, the
|
||||
> streaming layer, and Metering (`03c`). Access *reuses* them — it builds none net-new.
|
||||
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. Locate code by **symbol/grep**, not the cited lines.
|
||||
|
||||
## Objective
|
||||
|
||||
Expose the capability registry (`04a`) to callers, **authenticated + metered**, through three doors —
|
||||
**chat tools, REST + API keys, MCP server** — all **generated from the one registry** so the I/O
|
||||
contract cannot drift between surfaces. Access contains **no business logic**: every door is the same
|
||||
thin adapter.
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
- **API keys** — existing per-workspace key infra (reuse; do **not** build net-new); billed to the
|
||||
workspace owner via `03c`.
|
||||
- **Chat agent + tool registry** — capability tools register *into* the existing registry; the seed is
|
||||
`research/tools/scrape_webpage.py` (capability executor + access door + `03c` turn-accumulator billing).
|
||||
- **Slow-job pattern** — `subagents/builtins/deliverables/deliverable_wait.py`: dispatch a Celery task,
|
||||
poll the row's `status` until `READY`/`FAILED` (1.5s cadence), bounded by
|
||||
`SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS` (default 300s); `deliverables/tools/podcast.py` is the
|
||||
"return now + live card tracks progress" model.
|
||||
- **BYO-`MCP_CONNECTOR`** (old `04a`) — the user's own external MCP tools inside our chat agent; its
|
||||
routing-map gap is the one "connector" worth fixing, and it lands **here**.
|
||||
|
||||
## Target design
|
||||
|
||||
### The adapter shape (identical on every door)
|
||||
|
||||
```
|
||||
parse input → validate against verb.input_schema → authn/authz → meter-gate (03c)
|
||||
→ call the SAME executor (04a) → serialize verb.output_schema → return the uniform envelope
|
||||
```
|
||||
|
||||
### The three doors (locked order: chat → REST → MCP)
|
||||
|
||||
| Door | Who | Auth | Status |
|
||||
|------|-----|------|--------|
|
||||
| **Chat tools** | in-app agent (Product B delivery + interactive) | existing session + workspace | partly exists (`scrape_webpage`) |
|
||||
| **REST + API keys** | external developers (Product A) | **existing API-key infra** | **public day one** |
|
||||
| **MCP server** | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 **or** bearer — chosen at implementation | **fast-follow** |
|
||||
|
||||
REST being public day one is cheap precisely because the routes are **generated**, not hand-written —
|
||||
it's a go-to-market choice, not an engineering cost.
|
||||
|
||||
### Natural language is THE surface (verbs are internal) — non-negotiable
|
||||
|
||||
The human-facing product is **the conversation**. A user **never** names a verb, fills an `input_schema`,
|
||||
or knows "Product A vs B" exists — they describe a *need* in plain language and the agent does the rest.
|
||||
Raw typed verbs are exposed only on the REST/MCP doors (devs/external agents). The chat agent owns three
|
||||
responsibilities per message: **understand intent** → **pick & fill verbs** (incl. chains like
|
||||
`discover → scrape`, `search → place → reviews`) → **answer in plain language** (results, not envelopes).
|
||||
|
||||
### The intent router (the one new orchestration rule)
|
||||
|
||||
The agent classifies each request along the stateless/stateful line **from the language**:
|
||||
|
||||
```
|
||||
"compare / find / what is / pull / summarize / right now" → ONE-SHOT → call verbs, answer (Product A, stateless)
|
||||
"watch / track / notify me when / every week / keep an eye / over time" → STANDING → start Tracker setup (Product B, stateful → 05b)
|
||||
ambiguous → ask ONE question: "just this once, or should I keep watching it for you?"
|
||||
```
|
||||
|
||||
This router is the friendly seam between the two products; it lives in the chat door (its prompt lives
|
||||
in the `intelligence_agent`, `07`) and is the only human-facing decision point.
|
||||
|
||||
### The two MCP directions (keep distinct)
|
||||
|
||||
- **We *serve* MCP** — our capabilities as a remote MCP server (door #3, new): Streamable-HTTP `/mcp`,
|
||||
stateless, bounded/paginated outputs, untrusted inputs, least-privilege (read-only verbs).
|
||||
- **We *consume* MCP** — the BYO-`MCP_CONNECTOR` (old 04a): the user's own external MCP tools inside our
|
||||
chat agent. The 04a routing-gap fix lands **here**, not in Capabilities.
|
||||
|
||||
### Chat ↔ slow jobs — reuse the existing background-worker pattern
|
||||
|
||||
Do **not** invent a chat-async mechanism. A slow verb (`web.scrape` over many URLs, `maps.search`,
|
||||
`maps.reviews`) invoked from chat dispatches the **job** (the `04a` job record) and uses the
|
||||
`deliverable_wait` poll-until-terminal path; the capability **job record's `status`** is what the helper
|
||||
polls. Most calls finish inside the poll window → results inline; genuinely long ones surface a tracked
|
||||
card. REST/MCP expose the same job via `GET /v1/jobs/{id}` (and an MCP equivalent).
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Door generator**: from the `04a` registry, emit (a) chat tool defs + handlers, (b) REST routes +
|
||||
request/response models, (c) MCP tool schemas + handlers — one adapter shape.
|
||||
2. **REST surface**: public routes + API-key auth (reuse existing) + the `03c` meter-gate; `GET /v1/jobs/{id}`.
|
||||
3. **Chat tools**: generalize `scrape_webpage` into the registry-backed set; wire slow verbs through
|
||||
`deliverable_wait` polling the `04a` job record's `status`.
|
||||
4. **Intent router**: the A-vs-B classifier (its home is the `07` subagent prompt; the seam is here).
|
||||
5. **MCP server (fast-follow)**: Streamable-HTTP `/mcp`, least-privilege; auth depth chosen at implementation.
|
||||
6. **Consume-MCP fix**: repair the BYO-`MCP_CONNECTOR` routing-map gap (old 04a).
|
||||
|
||||
## Tests
|
||||
|
||||
- **Generated parity**: a verb added to the registry appears on chat + REST (+ MCP) with identical I/O.
|
||||
- **Auth + meter**: REST without a valid key → 401; an over-budget call → blocked by the `03c` gate
|
||||
before execute; a success charges once.
|
||||
- **Chat slow verb**: a many-URL `web.scrape` returns inline when fast; surfaces a tracked card when long;
|
||||
`GET /v1/jobs/{id}` reflects the same terminal status.
|
||||
- **Intent router**: "compare X and Y" → one-shot; "watch X weekly" → Tracker setup handoff; ambiguous →
|
||||
exactly one clarifying question.
|
||||
- **Consume-MCP**: a configured BYO MCP tool is reachable by the chat agent (routing-gap closed).
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Public REST day one** → needs bounded inputs + per-key quotas / abuse posture (design alongside launch).
|
||||
- **MCP auth depth** deferred to implementation (OAuth 2.1 vs bearer) — don't over-build before a consumer exists.
|
||||
- **Serve-vs-consume MCP confusion** — keep the two directions explicitly separate in code and docs.
|
||||
|
||||
## Resolved decisions
|
||||
|
||||
0. **Natural language is the only human-facing surface.** Users never name verbs/schemas/jobs; the chat
|
||||
agent understands intent, picks & fills verbs, answers in plain language; an intent router classifies
|
||||
one-shot (A) vs standing-concern (B), asking one question only when ambiguous. Raw verbs live only on REST/MCP.
|
||||
1. Three doors, generated from the capability registry; order chat → REST → MCP.
|
||||
2. REST is **public day one** (cheap; go-to-market choice).
|
||||
3. API keys: **reuse existing infra**, billed to workspace owner.
|
||||
4. MCP server is a **fast-follow**; auth depth (OAuth 2.1 vs bearer) chosen at implementation.
|
||||
5. Chat ↔ slow jobs: **reuse `deliverable_wait` poll-until-terminal + live card**, polling the `04a` job record's `status`.
|
||||
6. "Serve MCP" (our tools out) vs "consume MCP" (BYO tools in) are distinct; the old-04a fix is the consume side.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **The verbs themselves** (executors, registry, billing units) → `04a`.
|
||||
- **The CI subagent + its prompt/playbook** (where the intent router actually lives) → `07`.
|
||||
- **Stateful flows** (Tracker crafting, refresh, timeline reads) → `05b`/`05a`, surfaced via chat tools in `07`.
|
||||
- Legacy branded connectors stay only for backward-compat (separate hygiene task).
|
||||
|
||||
## Open questions (carry forward)
|
||||
|
||||
- MCP auth depth (decide at implementation).
|
||||
- Public REST rate-limiting / abuse posture (bounded inputs, per-key quotas).
|
||||
- Whether `web.discover` is metered or free (carried from `04a`).
|
||||
|
|
@ -1,125 +0,0 @@
|
|||
# Phase 5a — Timeline (the moat asset)
|
||||
|
||||
> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05b-intelligence.md` (the process that writes
|
||||
> this state). **Build first** within Phase 5 — the tables must exist before the hot loop writes them.
|
||||
> **Depends on** `04a` (the verbs the loop calls) and Phase 1 (the DB / migration baseline).
|
||||
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. The Timeline is **CI-owned, new tables** — it is **not**
|
||||
> the Knowledge Base (documents/embeddings) and **not** `automation_runs`.
|
||||
|
||||
## Objective
|
||||
|
||||
Durably store the **time-shaped truth** for each Tracker. This is the asset: **time is the moat** —
|
||||
accumulated history a later entrant cannot re-create. Design rule: **store deltas, not snapshots — no
|
||||
change, no row.** Storage grows with the *rate of change*, not the number of runs.
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
- **No CI state exists today** — crawled data is explicitly *not* indexed; the only persisted CI state is
|
||||
what this phase introduces.
|
||||
- **Reference models** for shape/placement: connectors/folders ORM in `app/db.py` (full-row Zero
|
||||
publication, like folders/connectors); `automations`/`automation_runs` as the *orchestration* analog
|
||||
(this is the **fact** analog, deliberately separate).
|
||||
- **Hot-loop pre-check** (`05b` step 2) reads `WebCrawlerConnector.format_to_structured_document(
|
||||
exclude_metadata=True)` to compute the `content_hash` stored here.
|
||||
|
||||
## Target design
|
||||
|
||||
### The state / process split
|
||||
|
||||
Intelligence (`05b`) is the only **writer** (via the hot loop). **Readers** — the Conversation domain
|
||||
today, future dashboards/alerts/the deferred resale product — read the Timeline **directly, without
|
||||
running the loop**. That separation is why Timeline is its own domain.
|
||||
|
||||
### The three stores
|
||||
|
||||
| Store | Role | Write pattern |
|
||||
|-------|------|---------------|
|
||||
| `tracked_entities` | stable identity per tracked thing (the Tracker's `identity_rule` → `entity_key`) | written **once** |
|
||||
| `entity_current_state` | latest values + `content_hash` + `last_checked_at` per entity | **overwritten** each run |
|
||||
| `entity_changes` (the change log) | append-only material deltas | **appended**, never overwritten |
|
||||
|
||||
**The timeline = the change log read in order.** To reconstruct a past point: take Current state and
|
||||
replay deltas backward (north-star tooling; MVP just *stores* the deltas).
|
||||
|
||||
### Data model sketch (new tables)
|
||||
|
||||
```
|
||||
tracked_entities
|
||||
id · workspace_id · tracker_id (FK) · entity_key (unique per tracker) · first_seen_at
|
||||
# MVP: exactly one row per Tracker (single-entity). Table stays multi-entity-ready.
|
||||
|
||||
entity_current_state
|
||||
entity_id (FK, unique) · tracker_id · fields JSONB (latest, conforms to locked field_schema)
|
||||
· content_hash · last_checked_at · updated_at
|
||||
# overwritten each material run; content_hash powers the hot-loop cheap pre-check (05b step 2)
|
||||
|
||||
entity_changes # the append-only change log
|
||||
id · entity_id (FK) · tracker_id · captured_at
|
||||
· delta JSONB # { field: { from, to } }
|
||||
· materiality # material | notable(=notable_signals-sourced)
|
||||
· decided_by # code | agent (audit of the materiality split)
|
||||
· source_ref # url / blob key the change was observed from
|
||||
· note TEXT NULL # optional agent rationale (the "why material")
|
||||
```
|
||||
|
||||
- **No change → no row** in `entity_changes`; an unchanged refresh only bumps `last_checked_at`.
|
||||
- `decided_by` records whether code or the agent ruled the change material (provenance seed).
|
||||
|
||||
### What it is NOT
|
||||
|
||||
- **Not the KB** — no `Document` rows, no embeddings, no indexing. ("Don't index crawled data" holds.)
|
||||
- **Not `automation_runs`** — that's an orchestration artifact; this is the durable fact store.
|
||||
- **Not a resurrected `pipeline_runs`.**
|
||||
|
||||
### Where it lives
|
||||
|
||||
- New CI-owned tables (in `app/db.py` alongside core entities, or a small `app/timeline/` package — decide
|
||||
at write-up). **Apache-2** (it stores facts; the moat is in Acquisition + the Maps extractor).
|
||||
- Published to Zero full-row later if/when a UI needs live sync (deferred with the frontend).
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Models + migration**: `tracked_entities` / `entity_current_state` / `entity_changes` + Alembic migration.
|
||||
2. **Write API**: `upsert_current_state(...)` (overwrite) and `append_change(...)` (insert) used by the `05b` loop.
|
||||
3. **Read API**: `query_timeline(tracker_id, …)` + `get_current_state(entity_id)` for the conversation/read side.
|
||||
4. **Content-hash field**: store + expose `content_hash` so `05b` step 2 can short-circuit.
|
||||
5. **Zero publication entry** (full-row) — wired but inert until a UI consumes it (deferred).
|
||||
|
||||
## Tests
|
||||
|
||||
- **No change → no row**: an unchanged refresh bumps `last_checked_at` only; `entity_changes` count is unchanged.
|
||||
- **Append-only log**: a material refresh inserts exactly one `entity_changes` row and overwrites `entity_current_state`.
|
||||
- **Provenance**: `decided_by` is `code` for threshold rules and `agent` for ambiguous calls.
|
||||
- **Identity**: `entity_key` is unique per tracker; re-refresh of the same entity reuses the row.
|
||||
- **Read API**: `query_timeline` returns deltas in `captured_at` order.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **First-run baseline**: with no prior `entity_current_state`, the diff has nothing to compare against —
|
||||
baseline semantics (silent establish vs flood the log) are an implementation-time call.
|
||||
- **High-velocity entities**: append-only growth is bounded by change rate, but retention/archival for very
|
||||
chatty entities is deferred.
|
||||
- **Schema-validate-on-write**: validating `fields` against the locked `field_schema` is a cheap integrity
|
||||
guard (lean: yes) but adds a write-path dependency on `05b`'s lock.
|
||||
|
||||
## Resolved decisions
|
||||
|
||||
1. Three stores: `tracked_entities` / `entity_current_state` / `entity_changes`.
|
||||
2. Store deltas, not snapshots; **no change → no row**; storage ∝ rate of change.
|
||||
3. CI-owned new tables; **not** KB, **not** `automation_runs`.
|
||||
4. `content_hash` on current state powers the hot-loop cheap pre-check.
|
||||
5. `decided_by` on changes records the code-vs-agent materiality provenance.
|
||||
6. Single entity per Tracker for MVP; schema stays multi-entity-ready (additive later).
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **The writer** (hot loop, materiality, schema lock) → `05b`.
|
||||
- **Backward-replay reconstruction**, trend/series read APIs, coverage-confidence, multi-entity scale, the
|
||||
resale/data-product surface → north star (deferred).
|
||||
- **Live UI sync** (Zero consumption, dashboards) → frontend umbrella.
|
||||
|
||||
## Open questions (carry forward)
|
||||
|
||||
- ORM home: `app/db.py` vs a dedicated `app/timeline/` package.
|
||||
- Whether `entity_current_state.fields` is validated against the locked `field_schema` at write time (lean: yes).
|
||||
- Retention / archival policy for very high-velocity entities (deferred).
|
||||
|
|
@ -1,191 +0,0 @@
|
|||
# Phase 5b — Intelligence (the decision-grounded engine)
|
||||
|
||||
> Part of **Phase 5 — Intelligence & Timeline**. Sibling: `05a-timeline.md` (the state it writes).
|
||||
> **Build after** `05a` (the tables) and `04a` (the verbs it calls). Together, `05a + 05b` ship the
|
||||
> **Product B engine** (the Tracker, locked schema, hot loop, deltas).
|
||||
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This is net-new and is **not** the KB and **not** the
|
||||
> automations subsystem.
|
||||
> **Name:** the standing-concern primitive is the **Tracker**.
|
||||
|
||||
## Objective
|
||||
|
||||
Turn repeated capability calls into **decision-relevant structured signal**. The motto: **the agent
|
||||
judges, code computes.** This replaces the old "pipeline" as the *standing concern*: Intelligence is the
|
||||
**process** that mutates state; Timeline (`05a`) is the **state**. Everything below the
|
||||
Access→Intelligence boundary stays pure functions.
|
||||
|
||||
```
|
||||
STATELESS (Product A): 04a Capabilities + 04b Access → call → data → bill, nothing persists
|
||||
STATEFUL (Product B): 05b Intelligence + 05a Timeline → the Timeline IS the state
|
||||
```
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
- **Capability executors** (`04a`) — called directly by the loop (not through a door).
|
||||
- **Content-hash pre-check** — `WebCrawlerConnector.format_to_structured_document(exclude_metadata=True)`
|
||||
produces the stable text the loop hashes against `entity_current_state.content_hash` (`05a`).
|
||||
- **Run/audit substrate to reuse** — the existing **`AutomationRun`** (status/error/timing/`step_results`;
|
||||
note the executor writes results to `step_results[n].result`, **not** `run.output`/`artifacts`, which stay
|
||||
unwritten) + the automations executor's PENDING→running gate (`execute_run()` early-returns unless
|
||||
`status == PENDING`; `acks_late` is global). **Caveat:** the gate is **not atomic** (no
|
||||
`UPDATE … WHERE status='pending'`), so it stops terminal/post-`mark_running` redelivery but **not**
|
||||
concurrent double-claim — pair it with the per-Tracker lock (`06`). Plus the chat **job record** (`04a`) +
|
||||
`deliverable_wait`. **No new run table.**
|
||||
- **CI context uploads** — the existing folder-upload machinery (`POST /documents/folder-upload`, param
|
||||
`root_folder_id` — there is no `destination_folder`) + KB retrieval scoped via mention-pins →
|
||||
`referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids` (no direct `folder_id` on search).
|
||||
|
||||
## Target design
|
||||
|
||||
### The primitive — `Tracker`
|
||||
|
||||
A saved, decision-grounded subject that accumulates structured signal over time:
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `decision` | the question being tracked toward ("is this competitor pulling ahead?") |
|
||||
| `capability_binding` | which verb + input feeds it (`maps.place(X)`, `web.scrape([Y])`) |
|
||||
| `definition` (locked, versioned) | `{ field_schema, identity_rule, materiality }` — the agent-drafted, human-locked contract |
|
||||
| `status` | `draft` → `locked`/`active` |
|
||||
|
||||
One **entity per Tracker for MVP** (one place / one URL). Multi-entity (`maps.search → many`) is deferred
|
||||
(the `05a` model stays multi-entity-ready so it's additive).
|
||||
|
||||
### Setup (once) — the agent-designed schema flow (IN MVP)
|
||||
|
||||
The product must not be rigid: **we cannot author one schema that serves everyone**, so the schema is
|
||||
derived from the *user's* decision by an agent and locked by the human. Conversationally (chat-first):
|
||||
|
||||
1. **Bind** a capability + input.
|
||||
2. **Sample fetch** — one real capability call so the agent drafts against *actual* returned data.
|
||||
3. **Agent drafts the `definition`** from `decision` + the sample: `field_schema` (fields + types),
|
||||
`materiality` (numeric thresholds where possible; "ask agent" otherwise), `identity_rule` (stable
|
||||
entity key — Maps `place_id`, canonical URL), and a reserved **`notable_signals`** escape-hatch field.
|
||||
4. **Human reviews & locks** (in chat: "looks good" / "add field X"). Locked ⇒ stable run-to-run.
|
||||
5. **Versioned** — a locked `definition` is a snapshot; edits create a new version (mirrors how
|
||||
`automations` snapshots `definition`).
|
||||
|
||||
### The hot loop (per refresh) — `refresh(tracker)`
|
||||
|
||||
1. **Crawl** — call the bound capability (`04a`) → raw data.
|
||||
2. **Cheap pre-check** — content hash; identical to the stored `content_hash` → stamp `last_checked_at`,
|
||||
**stop** (no LLM cost).
|
||||
3. **Fill** — agent conforms raw data to the **locked `field_schema`** via structured output; it does
|
||||
**not** invent fields. Unanticipated observations go into `notable_signals`.
|
||||
4. **Diff (code)** — deterministic compare of the new record vs Current state (`05a`) → raw deltas.
|
||||
5. **Judge — the materiality split:**
|
||||
- **deterministic (code):** numeric/clear-cut rules from `materiality`, applied for free, 100%
|
||||
reproducible (e.g. `rating Δ≥0.2 → material`, `review_count Δ≥10 → material`, `1¢ wobble → noise`).
|
||||
- **agent (only on ambiguous):** anything a rule can't decide — reworded `description`, a new
|
||||
`notable_signals` entry → one LLM call rules material/noise.
|
||||
6. **Append** — if material: write a Change + update Current state (`05a`). Else: only `last_checked_at`.
|
||||
**No change → no row.**
|
||||
|
||||
**Worked example (`maps.place` refresh):**
|
||||
```
|
||||
rating 4.4 → 4.3 (Δ0.1) → code: < 0.2 → NOISE (no LLM)
|
||||
review_count 312 → 470 → code: ≥ 10 → MATERIAL (no LLM)
|
||||
hours unchanged → no delta
|
||||
description reworded → code: no rule → ASK AGENT → NOISE
|
||||
⇒ one Change row (review spike); one cheap LLM call; zero LLM on the rating tick.
|
||||
```
|
||||
|
||||
### Refresh execution & idempotency — ride the invoking surface (no new run table)
|
||||
|
||||
`refresh(tracker)` is a **headless unit of work**; the run/audit record + idempotency live on **whatever
|
||||
surface invoked it**:
|
||||
|
||||
- **Recurring (in-app):** invoked by the **CI automation action** (`06`) → the existing **`AutomationRun`**
|
||||
is the run record and the automations executor's PENDING-gate handles redelivery. Reused, not re-written —
|
||||
but the gate is **not atomic** against concurrent claims, so the **per-Tracker lock (`06`)** is the actual
|
||||
double-refresh guard.
|
||||
- **Chat (manual / agent):** invoked via the chat **job record** (`04a`) + `deliverable_wait`.
|
||||
- **Billing idempotency is per *capability call*, not per run:** each `executor` bills a success once via
|
||||
the billing service (`04a`); the content-hash pre-check (step 2) prevents needless re-crawls/charges. No
|
||||
run-level `charged_micros` ledger required for MVP.
|
||||
|
||||
Net: the only genuinely new state is the **Timeline** (`05a`); execution accounting is borrowed.
|
||||
|
||||
### User-supplied context files (the F idea, generalized)
|
||||
|
||||
When a user uploads a file *in a CI chat* (e.g. "our own price list"), it goes into the **KB as normal** —
|
||||
uploads create `Document`s and are indexed/embedded, exactly as today. **(The "don't index" rule applies
|
||||
only to *crawled* data.)** The CI-specific part is **organization + use**:
|
||||
|
||||
- **Routed to a dedicated folder** for that CI chat/Tracker (reuse `POST /documents/folder-upload` with
|
||||
`root_folder_id`; uploads land as `Document`s with `folder_id`).
|
||||
- **The judge step (5) may consult them** — retrieved from the KB, scoped to that folder via the mention-pin
|
||||
path (`referenced_document_ids(folder_ids=…)` → `SearchScope.document_ids`), since search has no direct
|
||||
`folder_id` filter:
|
||||
|
||||
```
|
||||
competitor price 12.00 → 9.90 + user's context file says "our price is 10.00"
|
||||
→ agent: competitor crossed *below our price* → MATERIAL (and explain why)
|
||||
```
|
||||
|
||||
The user's private context **shapes what counts as material**. Reuses existing KB upload + folder +
|
||||
retrieval machinery (nothing new). **MVP-optional** (the loop works without it); design the seam now.
|
||||
|
||||
### Where it lives / decoupling
|
||||
|
||||
- New **Apache-2** package `app/intelligence/` (schema-design agent, hot loop, materiality evaluator). Calls
|
||||
capability `executor`s directly.
|
||||
- Exposes **`refresh(tracker)`**. *Who* calls it (manual / agent / external cron / optional automation) is
|
||||
the **Triggers** domain's concern (`06`) — Intelligence has **no dependency** on any scheduler.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Tracker model + persistence**: `decision` · `capability_binding` · versioned locked `definition` · `status`.
|
||||
2. **Schema-design flow**: bind → sample-fetch → agent-drafts `definition` → human review/lock → version.
|
||||
3. **Materiality evaluator**: deterministic rule engine (numeric/clear) + the agent-on-ambiguous fallback.
|
||||
4. **The hot loop** `refresh(tracker)`: crawl → hash pre-check → fill → diff → judge → append (writes `05a`).
|
||||
5. **Idempotency wiring**: ride `AutomationRun` (recurring) / chat job record (manual) — no new run table.
|
||||
6. **Context-folder seam**: optional KB-retrieval hook into the judge, scoped to the Tracker's folder.
|
||||
|
||||
## Tests
|
||||
|
||||
- **Pre-check short-circuit**: identical content hash → no fill, no LLM, only `last_checked_at` bumped.
|
||||
- **Fill conforms to lock**: extra observed fields land in `notable_signals`, never invent schema fields.
|
||||
- **Materiality split**: numeric Δ over threshold → `decided_by=code, material`; ambiguous reword →
|
||||
`decided_by=agent`; sub-threshold → noise, no row.
|
||||
- **Append semantics**: a material run writes one `entity_changes` row + overwrites current state.
|
||||
- **Idempotency**: a redelivered recurring refresh (same `AutomationRun`) does not double-write or double-bill.
|
||||
- **Context folder (optional)**: judge ruling flips when a user context file changes the decision frame.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Refresh failure path** (capability returns `FAILED`/partial): skip vs retain vs retry vs alert — an
|
||||
implementation-time call (no architecture impact).
|
||||
- **Agent fill cost** on changed pages: bounded by the hash pre-check; only changed content reaches the LLM.
|
||||
- **Schema lock rigidity**: locking trades flexibility for run-to-run stability; re-lock creates a new version.
|
||||
|
||||
## Resolved decisions
|
||||
|
||||
1. `Tracker` is the standing-concern primitive; replaces "pipeline".
|
||||
2. Stateless (`04`/Product A) vs stateful (`05`/Product B) is the Access→Intelligence boundary.
|
||||
3. **Agent-designed schema flow is in MVP** (not hand-authored) — sample-grounded, human-locked, versioned.
|
||||
4. Single entity per Tracker for MVP.
|
||||
5. Materiality = deterministic numeric/clear rules in code + agent only on ambiguous.
|
||||
6. Content-hash pre-check short-circuits unchanged pages before any LLM spend.
|
||||
7. `app/intelligence/` Apache-2; `refresh(tracker)` is trigger-agnostic.
|
||||
8. **No new run table** — refresh audit rides `AutomationRun` (recurring) or the chat job record; redelivery
|
||||
is stopped by the PENDING-gate **plus a per-Tracker lock** (the gate is non-atomic); billing idempotency
|
||||
is per-capability-call + the content-hash gate. Only Timeline (`05a`) is new state.
|
||||
9. **CI context folder** (F): user files uploaded in a CI chat go into the **KB as normal** (indexed),
|
||||
routed to a dedicated `Folder`, and may feed the judge via KB retrieval. "Don't index" is for *crawled*
|
||||
data only. MVP-optional seam.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **The state tables** (the three stores, content-hash, read API) → `05a`.
|
||||
- **When refresh fires + recurrence/delivery** → `06` (Triggers).
|
||||
- **The human-facing crafting/answering experience** (the `intelligence_agent` subagent + prompt) → `07`.
|
||||
- **Schema auto-evolution**, multi-entity Trackers, backward-replay, coverage-confidence, the
|
||||
resale/data-product stage → north star (deferred).
|
||||
|
||||
## Open questions (carry forward)
|
||||
|
||||
- How the schema-design agent surfaces "review & lock" before the frontend exists (pure-chat confirmation?).
|
||||
- Versioning policy on re-lock (new version vs in-place) — lean new version.
|
||||
- Where the schema-design agent itself runs (a setup capability? a chat sub-flow?).
|
||||
- Context-folder → judge wiring (how much to load; per-Tracker vs per-chat scope).
|
||||
|
|
@ -1,131 +0,0 @@
|
|||
# Phase 6 — Triggers (the pluggable refresh clock)
|
||||
|
||||
> **Phase 6** of the CI-pivot revamp — the thinnest phase. **Build after** Phase 5 (`05b` exposes
|
||||
> `refresh(tracker)`).
|
||||
> **Depends on** `05b` (`refresh(tracker)`), `04b` (the REST manual/cron routes), and the SHIPPED
|
||||
> automations subsystem (its schedule selector + `AutomationRun`) plus `app/notifications/` for alert delivery.
|
||||
> **Scope guardrail:** Phases 1–3 SHIPPED/FIXED. This phase is **decoupled** — Intelligence (`05b`) has no
|
||||
> dependency on it, and **automations is at most one optional adapter, never required**.
|
||||
|
||||
## Objective
|
||||
|
||||
Decide **when** a Tracker refreshes. Intelligence exposes a single entry point — **`refresh(tracker)`** —
|
||||
and every trigger is just a caller. Intelligence never knows which trigger fired; remove any trigger and
|
||||
the engine still works. This replaces the old Phase-6 cron scheduler — and the resolution is **not** to
|
||||
rebuild a scheduler at all, but to **reuse the automations subsystem** for the in-app recurring path.
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
- **`refresh(tracker)`** — the headless unit of work exposed by `05b`.
|
||||
- **Automations schedule selector** — the already-hardened cron selector (`FOR UPDATE SKIP LOCKED`
|
||||
claiming, `next_fire_at` advance, self-heal, duplicate-run suppression, `catchup=False`), reusing the
|
||||
`croniter` util. *(Verified: `app/automations/triggers/builtin/schedule/selector.py::_claim_due_triggers`
|
||||
+ `schedule/cron.py::compute_next_fire_at`.)*
|
||||
- **`AutomationRun`** — the existing run record + a PENDING-gate. *(Verified with a caveat: the gate is
|
||||
`execute_run()` early-returning unless `status == PENDING` + `mark_running`, and `acks_late` /
|
||||
`task_reject_on_worker_lost` are set globally — so **terminal redelivery and post-`mark_running`
|
||||
redelivery are no-ops**. But there is **no atomic `UPDATE … WHERE status='pending'` claim**, so two
|
||||
concurrent workers can both read PENDING → double-run. The per-Tracker lock below is therefore the
|
||||
primary concurrency guard, not just belt-and-suspenders.)*
|
||||
- **Delivery ≠ automations (correction).** Automations have **no** push/email/notification path, and
|
||||
`automation_runs.output` is **never written** by the executor (results live in `step_results[n].result`);
|
||||
today "delivery" is only **pull via `GET /automations/{id}/runs/{run_id}`** + **Zero dashboard sync**
|
||||
(`zero_publication.AUTOMATION_RUN_COLS`). In-app alerts must wire to the **separate `app/notifications/`
|
||||
system** — `NotificationService.create_notification(...)` (DB-backed, Zero-synced, typed
|
||||
`NotificationType`). So alert delivery is a **small net-new wiring**, not a free automations reuse.
|
||||
- **Access routes** (`04b`) — where the external-cron and manual REST endpoints live.
|
||||
|
||||
## Target design
|
||||
|
||||
### The adapters
|
||||
|
||||
| Adapter | Fired by | MVP? |
|
||||
|---------|----------|------|
|
||||
| **Manual** | user "refresh now" (chat tool / REST) | ✅ |
|
||||
| **Agent** | the in-app agent calls `refresh` as a tool | ✅ |
|
||||
| **External cron** | the user's own scheduler hits `POST /v1/trackers/{id}/refresh` | ✅ (zero infra on us) |
|
||||
| **CI automation action** | a **CI action on the existing automations** — its schedule trigger fires `refresh(tracker)`, persists deltas to the Timeline, **and emits an in-app alert** via `app/notifications/` | ✅ (the in-app recurrence + alert path) |
|
||||
|
||||
### Recurrence + delivery — a CI action on existing automations (NOT a new scheduler, NOT a new shape)
|
||||
|
||||
The SMB competitor-watch buyer needs **in-app recurrence** *and* **"tell me when it changes."** Instead of
|
||||
building a bespoke tick, we **add a CI *action* to the existing automations subsystem**:
|
||||
|
||||
- **Schedule** → the automation's existing **schedule trigger** (the already-hardened selector). No new
|
||||
scheduler. **(closes old Gap B — scheduler rigor.)**
|
||||
- **Run record + idempotency** → the automation's existing **`AutomationRun`** + PENDING-gate. No new run
|
||||
table. **(closes old Gap A — run/idempotency)** — *but* pair it with the per-Tracker lock (the gate is
|
||||
non-atomic; see Current state + `05b`).
|
||||
- **Delivery / alert** → **not** free from automations. Persist the material change to the **Timeline**
|
||||
(`05a`), then emit an in-app alert via **`app/notifications/`** (`NotificationService.create_notification`
|
||||
with a new CI notification type; Zero-synced to the UI). **(old Gap E — alert delivery — is *mostly*
|
||||
closed but needs this small wiring, not zero-cost reuse.)**
|
||||
|
||||
**Why a CI *action*, not a new automation shape:** a new shape would duplicate triggers, runs, and delivery
|
||||
that already exist. A single `refresh_tracker` action reuses all of it. (If implementation finds the action
|
||||
too constraining, a thin CI-specific shape is the fallback — but the action is the default.)
|
||||
|
||||
### Decoupling is preserved (automations is still optional)
|
||||
|
||||
CI **core** — `refresh(tracker)` + Timeline (`05a`/`05b`) — has **zero** automations dependency and runs
|
||||
via manual / agent / external-cron. Automations is the **optional adapter** that adds recurrence + audit
|
||||
for in-app users (alerts ride `app/notifications/`, not automations). So we honor "don't glue CI to
|
||||
automations" *and* get its machinery for free.
|
||||
|
||||
### Where it lives
|
||||
|
||||
- The **CI action** lives with automations (its action registry); it imports `refresh(tracker)` from
|
||||
`app/intelligence/`. No new scheduler/Beat task.
|
||||
- The **external-cron** and **REST manual** paths are just Access-door routes (`POST
|
||||
/v1/trackers/{id}/refresh`) — `04b` plumbing.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Manual / agent triggers**: a `refresh_tracker(tracker_id)` chat tool + REST route → `refresh(tracker)`.
|
||||
2. **External-cron route**: `POST /v1/trackers/{id}/refresh` (API-key authed) → `refresh(tracker)`.
|
||||
3. **CI automation action**: register a `refresh_tracker` action in the automations action registry that
|
||||
calls `refresh(tracker)` and, on material change, emits an in-app alert via `app/notifications/`
|
||||
(`NotificationService.create_notification`, new CI notification type). *(Not "automations delivery" —
|
||||
that path doesn't exist; see Current state.)*
|
||||
4. **Concurrency guard**: a per-Tracker in-flight lock — the **primary** guard against concurrent
|
||||
double-refresh, since the automation PENDING-gate is non-atomic (also protects the manual/cron paths
|
||||
that don't go through automations).
|
||||
|
||||
## Tests
|
||||
|
||||
- **Decoupling**: `refresh(tracker)` works via manual/agent/external-cron with automations disabled entirely.
|
||||
- **Recurring path**: a scheduled CI action fires `refresh` on cron and delivers material changes; an
|
||||
unchanged refresh delivers nothing.
|
||||
- **Idempotency**: a redelivered scheduled run (same `AutomationRun`) does not double-refresh.
|
||||
- **External cron**: `POST /v1/trackers/{id}/refresh` triggers exactly one refresh; rejects bad auth.
|
||||
- **Concurrency**: a second refresh while one is in flight is skipped/queued, not run concurrently.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Action vs shape**: the CI action is the default; a thin CI automation shape is the fallback if the
|
||||
action proves too constraining.
|
||||
- **Delivered payload shape** (agent-summarized vs raw deltas since last fire) — implementation-time call.
|
||||
- **Double-guarding concurrency**: the per-Tracker lock overlaps the automation run-gate, but the lock also
|
||||
protects the manual/cron paths that don't go through automations.
|
||||
|
||||
## Resolved decisions
|
||||
|
||||
1. Intelligence exposes `refresh(tracker)`; all triggers are callers. Fully decoupled.
|
||||
2. Adapters: manual · agent · external-cron · **CI automation action** (recurrence + delivery).
|
||||
3. **No bespoke scheduler and no new run table** — the recurring path reuses the automations schedule
|
||||
selector + `AutomationRun` (Closes old Gaps A/B). **Delivery does NOT reuse automations** (no such
|
||||
path; `run.output` is unwritten) — alerts go through `app/notifications/` (Gap E: small wiring).
|
||||
4. Recurrence is a **CI *action*** on the existing automations, **not a new automation shape**.
|
||||
5. CI core stays runnable with **zero** automations dependency (manual/agent/external-cron).
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **`refresh(tracker)` internals** (the hot loop, materiality) → `05b`.
|
||||
- **The Timeline reads** the delivery summarizes → `05a`.
|
||||
- **The chat tool surface** for manual/agent refresh → `07` (the `intelligence_agent` toolset).
|
||||
|
||||
## Open questions (carry forward)
|
||||
|
||||
- CI **action** vs a thin CI-specific automation **shape** (default: action; shape is the fallback).
|
||||
- What the delivered payload looks like (summarized material changes vs raw deltas since last fire).
|
||||
- Concurrency: per-Tracker lock granularity (like the connector indexing lock).
|
||||
|
|
@ -1,126 +0,0 @@
|
|||
# Phase 7 — Orchestration / Conversation (the CI-expert subagent)
|
||||
|
||||
> **Phase 7** of the CI-pivot revamp — the **human-facing brain** that turns "natural language is the only
|
||||
> surface" (`04b`, Decision 0) into a real deliverable. **Build last** — it sits atop `04`–`06`.
|
||||
> **Depends on** all prior phases: `04a` verbs, `04b` doors + `deliverable_wait`, `05a`/`05b` Tracker +
|
||||
> Timeline, `06` refresh triggers.
|
||||
> **Scope guardrail:** the multi-agent chat **runtime** (deepagents, subagent dispatch, streaming,
|
||||
> citation middleware, `deliverable_wait`) is SHIPPED/FIXED. What's **net-new here** is one builtin
|
||||
> subagent + its tools + its prompt. We *plug into* the runtime; we don't rebuild it.
|
||||
|
||||
## Objective
|
||||
|
||||
Ship the **`intelligence_agent`** — a net-new builtin CI-expert subagent — so users get the whole product
|
||||
through plain conversation: intent routing (one-shot vs standing concern), verb composition, Tracker
|
||||
crafting, and decision-grounded answering. The intent router + verb-composition + Tracker-crafting are
|
||||
where "user-friendly" is won or lost, so the orchestration layer is **designed, not assumed**.
|
||||
|
||||
## Current state (cited)
|
||||
|
||||
- **Builtin subagent pattern** — `subagents/builtins/<name>/`: `agent.py` (`build_subagent(...)` →
|
||||
`pack_subagent(name, description, system_prompt, tools, ruleset, …)`), `tools/index.py`
|
||||
(`NAME · RULESET · load_tools(dependencies)`), `description.md` (router one-liner), `system_prompt.md`
|
||||
(the playbook), optional middleware (e.g. `citation_state`). Peers: `research`, `deliverables`.
|
||||
- **Tool shape** — `research/tools/scrape_webpage.py` shows the **tool = capability executor + access door**
|
||||
pattern (calls `WebCrawlerConnector.crawl_url`, returns a typed dict). **Billing note:** its current
|
||||
`get_current_accumulator()` fold is **interactive-chat-only** and no-ops outside a chat turn — so CI tools
|
||||
do **not** copy that as the charging path. Per `04a` (locked), **the charge fires in the capability
|
||||
executor**; the chat turn accumulator stays only as an optional presentation fold. This is what makes the
|
||||
recurring `06 → refresh → verb → crawl` path (no chat turn) bill correctly.
|
||||
- **Slow-job path** — `deliverable_wait` poll-until-terminal + the podcast-style live card (`04b`).
|
||||
|
||||
## Target design
|
||||
|
||||
### The `intelligence_agent` subagent (CI expert)
|
||||
|
||||
A new builtin `subagents/builtins/intelligence_agent/` (the competitive-intelligence specialist), peer to
|
||||
`research`/`deliverables`. The **main agent delegates** to it whenever the request is CI-flavored (research
|
||||
a competitor, watch something, analyze a place's reviews); `description.md` is what makes that routing
|
||||
happen. It owns the **CI playbook** in `system_prompt.md`:
|
||||
|
||||
1. **Intent routing (A vs B)** — the Decision-0 rule, in-prompt: one-shot ("compare/find/what is") → call
|
||||
verbs & answer; standing concern ("watch/track/notify when/weekly") → run the crafting flow; ambiguous →
|
||||
ask the single clarifying question.
|
||||
2. **Verb composition** — the chains: `web.discover → web.scrape`, `maps.search → maps.place →
|
||||
maps.reviews`; infer URLs/queries/locations from context so the user never supplies them by hand.
|
||||
3. **Tracker crafting** — the conversational schema-design flow from `05b`: sample-fetch → propose
|
||||
`field_schema` + materiality + identity → user validates & locks → versioned.
|
||||
4. **Decision-grounded answering** — read the Timeline (`05a`) to answer "what changed / is X pulling
|
||||
ahead?" from stored deltas, not by re-deriving from chat history.
|
||||
|
||||
### The toolset (what `load_tools` returns)
|
||||
|
||||
| Tool | Wraps | Mode | Billing |
|
||||
|------|-------|------|---------|
|
||||
| capability verbs (`web.scrape`, `web.discover`, `maps.search`, `maps.place`, `maps.reviews`) | `04a` executors | inline-or-job (slow → `deliverable_wait`) | at the `04a` executor (per-call); chat turn accumulator = optional presentation fold only |
|
||||
| `craft_tracker(decision, binding)` | `05b` schema-design agent | inline (sample-fetch + proposes) | sample crawl billed |
|
||||
| `lock_tracker(draft)` / `update_tracker` | `05b` Tracker persistence | inline | — |
|
||||
| `refresh_tracker(tracker_id)` | `05b` `refresh(tracker)` (via `06`) | job → `deliverable_wait` | per capability call |
|
||||
| `query_timeline(tracker_id, …)` | `05a` read API | inline | — |
|
||||
| `list_trackers()` | `05a`/`05b` read | inline | — |
|
||||
|
||||
- **Capability verbs are a shared tool module** (generated from the `04a` registry) — `research` can load
|
||||
the same ones; the `intelligence_agent` additionally loads the Tracker/Timeline tools + the CI prompt.
|
||||
(`scrape_webpage` is the seed; generalize it into the registry-backed set.)
|
||||
- **Slow verbs** (`maps.search`, multi-URL `web.scrape`, `refresh_tracker`) dispatch a job and use the
|
||||
existing `deliverable_wait` poll-until-terminal + live-card path (`04b`).
|
||||
|
||||
### Boundaries
|
||||
|
||||
- **Orchestration ≠ Intelligence.** The `intelligence_agent` *drives* `05a`/`05b` via tools; the hot loop,
|
||||
materiality, and Timeline writes live in `05a`/`05b`, callable headless (so REST/MCP and Triggers reach the
|
||||
same logic with no agent in the loop).
|
||||
- **Humans get the agent; machines get raw verbs.** REST/MCP callers (devs/external agents) skip this
|
||||
subagent entirely and call `04a` verbs directly — they *want* explicitness.
|
||||
|
||||
## Work items
|
||||
|
||||
1. **Subagent scaffold**: `subagents/builtins/intelligence_agent/` with `agent.py` / `tools/index.py` /
|
||||
`description.md` / `system_prompt.md`, packed via `pack_subagent`.
|
||||
2. **CI playbook prompt**: intent routing (A/B + one clarifying question), verb chains, crafting flow,
|
||||
decision-grounded answering.
|
||||
3. **Shared verb tool module**: registry-backed (`04a`) capability tools, reusable by `research` too.
|
||||
4. **Tracker/Timeline tools**: `craft_tracker` / `lock_tracker` / `update_tracker` / `refresh_tracker` /
|
||||
`query_timeline` / `list_trackers`.
|
||||
5. **Slow-verb wiring**: route job-mode tools through `deliverable_wait` + the live card.
|
||||
6. **Router registration**: `description.md` so the main agent delegates CI-flavored requests here.
|
||||
|
||||
## Tests
|
||||
|
||||
- **Delegation**: a CI-flavored request routes to `intelligence_agent`; a non-CI request does not.
|
||||
- **One-shot**: "compare X and Y" composes verbs and answers in plain language, persisting nothing.
|
||||
- **Standing concern**: "watch X weekly" runs the crafting flow → lock → (refresh via `06`).
|
||||
- **Crafting**: `craft_tracker` does a real sample-fetch and proposes a schema; `lock_tracker` versions it.
|
||||
- **Answering**: `query_timeline` answers "what changed?" from stored deltas, not chat history.
|
||||
- **Slow verb**: a multi-URL scrape surfaces a live card and returns terminal results.
|
||||
|
||||
## Risks / trade-offs
|
||||
|
||||
- **Prompt quality is the product**: intent-router/composition quality is a first-class deliverable, not
|
||||
plumbing — budget iteration on the prompt.
|
||||
- **One-shot ownership**: whether `research` keeps generic scraping or the `intelligence_agent` owns all
|
||||
CI-flavored calls (lean: shared verb tools, CI agent owns the *playbook*).
|
||||
- **Pre-frontend lock UX**: "review & lock" must render in pure chat for MVP (ties to `05b`'s open Q).
|
||||
|
||||
## Resolved decisions
|
||||
|
||||
1. CI orchestration is a **net-new builtin subagent** (`intelligence_agent`) on the existing runtime — not
|
||||
a runtime rebuild.
|
||||
2. Tools follow the `scrape_webpage` executor+door shape, but **billing fires in the `04a` executor** (not the chat turn accumulator, which is interactive-only and would skip automation/cron runs).
|
||||
3. Capability verbs are a **shared, registry-generated** tool module; the `intelligence_agent` adds
|
||||
Tracker/Timeline tools + the CI prompt.
|
||||
4. Intent routing (A vs B) lives **in the subagent prompt**; the headless logic stays in `05a`/`05b`.
|
||||
5. Slow verbs reuse `deliverable_wait`; nothing new for chat-async.
|
||||
|
||||
## Out of scope (hand-offs)
|
||||
|
||||
- **The verbs, doors, engine, state, and triggers** → `04`/`05`/`06` (this phase only *orchestrates* them).
|
||||
- **Richer multi-step CI playbooks** (auto competitor discovery → multi-Tracker setup), proactive "you
|
||||
should watch this" suggestions, cross-Tracker synthesis → north star (deferred).
|
||||
- **Frontend CI surfaces** (cards, dashboards) → frontend umbrella.
|
||||
|
||||
## Open questions (carry forward)
|
||||
|
||||
- Does CI one-shot scraping stay in `research`, or does the `intelligence_agent` own all CI-flavored calls
|
||||
(lean: shared verb tools, `intelligence_agent` owns the *CI playbook*).
|
||||
- How `craft_tracker`'s "review & lock" renders pre-frontend (pure-chat confirmation — ties to `05b`'s open Q).
|
||||
|
|
@ -60,6 +60,13 @@ WHATSAPP_BRIDGE_URL=http://whatsapp-bridge:9929
|
|||
# Only uncomment if running the backend outside Docker (e.g. uvicorn on host).
|
||||
# SEARXNG_DEFAULT_HOST=http://localhost:8888
|
||||
|
||||
# web.discover fallback providers (env-keyed). SearXNG above is preferred; these
|
||||
# are used when it is not configured. web.discover self-disables if none are set.
|
||||
# LINKUP_API_KEY=
|
||||
# BAIDU_API_KEY=
|
||||
# BAIDU_MODEL=ernie-3.5-8k
|
||||
# BAIDU_SEARCH_SOURCE=baidu_search_v2
|
||||
|
||||
# Periodic task interval
|
||||
# # Run every minute (default)
|
||||
# SCHEDULE_CHECKER_INTERVAL=1m
|
||||
|
|
|
|||
|
|
@ -26,6 +26,9 @@ CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS: dict[str, str] = {
|
|||
SUBAGENT_TO_REQUIRED_CONNECTOR_MAP: dict[str, frozenset[str]] = {
|
||||
"deliverables": frozenset(),
|
||||
"knowledge_base": frozenset(),
|
||||
"web_crawler": frozenset(),
|
||||
"youtube": frozenset(),
|
||||
"google_maps": frozenset(),
|
||||
"airtable": frozenset({"AIRTABLE_CONNECTOR"}),
|
||||
"calendar": frozenset({"GOOGLE_CALENDAR_CONNECTOR"}),
|
||||
"clickup": frozenset({"CLICKUP_CONNECTOR"}),
|
||||
|
|
|
|||
|
|
@ -40,7 +40,6 @@ _CONNECTOR_TYPE_TO_SEARCHABLE: dict[str, str] = {
|
|||
"AIRTABLE_CONNECTOR": "AIRTABLE_CONNECTOR",
|
||||
"LUMA_CONNECTOR": "LUMA_CONNECTOR",
|
||||
"ELASTICSEARCH_CONNECTOR": "ELASTICSEARCH_CONNECTOR",
|
||||
"WEBCRAWLER_CONNECTOR": "CRAWLED_URL", # Maps to document type
|
||||
"BOOKSTACK_CONNECTOR": "BOOKSTACK_CONNECTOR",
|
||||
"CIRCLEBACK_CONNECTOR": "CIRCLEBACK", # Connector type differs from document type
|
||||
"OBSIDIAN_CONNECTOR": "OBSIDIAN_CONNECTOR",
|
||||
|
|
|
|||
|
|
@ -18,11 +18,11 @@ from requests import Session
|
|||
from scrapling.fetchers import AsyncFetcher
|
||||
from youtube_transcript_api import YouTubeTranscriptApi
|
||||
|
||||
from app.proprietary.platforms.youtube.url_resolver import get_youtube_video_id
|
||||
from app.proprietary.web_crawler import (
|
||||
CrawlOutcomeStatus,
|
||||
WebCrawlerConnector,
|
||||
)
|
||||
from app.tasks.document_processors.youtube_processor import get_youtube_video_id
|
||||
from app.utils.proxy import get_proxy_url, get_requests_proxies
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
"""``google_maps`` builtin subagent: structured Google Maps place/review data."""
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
"""``research`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
|
||||
"""``google_maps`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
@ -7,9 +7,6 @@ from typing import Any
|
|||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.middleware.citation_state import (
|
||||
build_citation_state_mw,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
|
||||
read_md_file,
|
||||
)
|
||||
|
|
@ -31,15 +28,9 @@ def build_subagent(
|
|||
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
|
||||
description = (
|
||||
read_md_file(__package__, "description").strip()
|
||||
or "Handles research tasks for this workspace."
|
||||
or "Pulls structured data from Google Maps — places and their reviews."
|
||||
)
|
||||
system_prompt = read_md_file(__package__, "system_prompt").strip()
|
||||
# web_search registers WEB_RESULT citations via Command(update=...); the
|
||||
# citation-state middleware declares the channel so those [n] merge back up.
|
||||
middleware_with_citations = {
|
||||
**(middleware_stack or {}),
|
||||
"citation_state": build_citation_state_mw(),
|
||||
}
|
||||
return pack_subagent(
|
||||
name=NAME,
|
||||
description=description,
|
||||
|
|
@ -48,5 +39,5 @@ def build_subagent(
|
|||
ruleset=RULESET,
|
||||
dependencies=dependencies,
|
||||
model=model,
|
||||
middleware_stack=middleware_with_citations,
|
||||
middleware_stack=middleware_stack,
|
||||
)
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
Google Maps specialist: pulls structured data from Google Maps — finds places by search query (optionally scoped to a location), or resolves a Maps URL / place ID into place details (name, address, category, phone, website, rating, review count, coordinates, opening hours), and fetches a place's reviews (author, text, star rating, owner response, dates). Also compares fresh Maps data against earlier findings in this chat.
|
||||
Use whenever the task involves Google Maps places, local businesses, or a google.com/maps link. Triggers include "find <business type> near/in X", "get details for this place", "phone/address/hours/rating of X", "how many reviews", "get the reviews for this place", "what are people saying about this business", and comparisons against earlier Maps results in this chat. Not for general web pages (use the web crawling specialist) or YouTube (use the YouTube specialist).
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
You are the SurfSense Google Maps sub-agent.
|
||||
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
|
||||
|
||||
<goal>
|
||||
Answer the delegated question from live Google Maps data gathered with your verbs, comparing against earlier results already in this conversation when the task calls for it.
|
||||
</goal>
|
||||
|
||||
<available_tools>
|
||||
- `google_maps_scrape`
|
||||
- `google_maps_reviews`
|
||||
</available_tools>
|
||||
|
||||
<playbook>
|
||||
- Finding places on a topic: call `google_maps_scrape` with `search_queries`, adding `location` to scope them (e.g. "coffee shops" in "Austin, USA").
|
||||
- Known place links or IDs: call `google_maps_scrape` with the links in `urls` or the IDs in `place_ids`.
|
||||
- Need richer detail (opening hours, popular times, extra contact info): set `include_details=true`.
|
||||
- Reviews / sentiment on specific places: call `google_maps_reviews` with the place `urls` or `place_ids`.
|
||||
- Batch multiple queries, URLs, or place IDs into one call rather than many single-item calls.
|
||||
- Comparison requests: pull the current values, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new).
|
||||
</playbook>
|
||||
|
||||
<tool_policy>
|
||||
- Use only tools in `<available_tools>`.
|
||||
- An item whose `status` is not `success` returned no data — report it unavailable, never invent it.
|
||||
- Report only deltas you can point to in the evidence. Never fabricate facts, counts, quotes, ratings, or URLs.
|
||||
</tool_policy>
|
||||
|
||||
<out_of_scope>
|
||||
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
|
||||
- Non-Maps web pages belong to the web crawling specialist; YouTube belongs to the YouTube specialist.
|
||||
</out_of_scope>
|
||||
|
||||
<safety>
|
||||
- Report uncertainty explicitly when evidence is incomplete or conflicting.
|
||||
- Never present unverified claims as facts.
|
||||
</safety>
|
||||
|
||||
<failure_policy>
|
||||
- Underspecified request — no usable search query, URL, or place ID — return `status=blocked` with the missing fields.
|
||||
- Tool failure: return `status=error` with a concise recovery `next_step`.
|
||||
- No useful evidence: return `status=blocked` with a narrower query or the URLs/IDs you still need.
|
||||
</failure_policy>
|
||||
|
||||
<output_contract>
|
||||
Return **only** one JSON object (no markdown/prose):
|
||||
{
|
||||
"status": "success" | "partial" | "blocked" | "error",
|
||||
"action_summary": string,
|
||||
"evidence": {
|
||||
"findings": string[],
|
||||
"sources": string[],
|
||||
"confidence": "high" | "medium" | "low"
|
||||
},
|
||||
"next_step": string | null,
|
||||
"missing_fields": string[] | null,
|
||||
"assumptions": string[] | null
|
||||
}
|
||||
<include snippet="output_contract_base"/>
|
||||
Route-specific rules:
|
||||
- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw payloads.
|
||||
- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once.
|
||||
</output_contract>
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
"""``google_maps`` sub-agent tools: the Google Maps scrape + reviews capability verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
|
||||
from app.capabilities.core.access.agent import build_capability_tools
|
||||
from app.capabilities.google_maps.reviews.definition import GOOGLE_MAPS_REVIEWS
|
||||
from app.capabilities.google_maps.scrape.definition import GOOGLE_MAPS_SCRAPE
|
||||
|
||||
NAME = "google_maps"
|
||||
|
||||
RULESET = Ruleset(origin=NAME, rules=[])
|
||||
|
||||
_CI_VERBS = [GOOGLE_MAPS_SCRAPE, GOOGLE_MAPS_REVIEWS]
|
||||
|
||||
|
||||
def load_tools(
|
||||
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
|
||||
) -> list[BaseTool]:
|
||||
d = {**(dependencies or {}), **kwargs}
|
||||
return build_capability_tools(
|
||||
workspace_id=d.get("workspace_id"),
|
||||
capabilities=_CI_VERBS,
|
||||
)
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Specialist for external research.
|
||||
Use whenever a task requires finding sources on the web and extracting evidence to answer documentation questions.
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
You are the SurfSense research operations sub-agent.
|
||||
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
|
||||
|
||||
<goal>
|
||||
Gather and synthesize evidence using SurfSense research tools with clear citations and uncertainty reporting.
|
||||
</goal>
|
||||
|
||||
<available_tools>
|
||||
- `web_search`
|
||||
- `scrape_webpage`
|
||||
</available_tools>
|
||||
|
||||
<tool_policy>
|
||||
- Use only tools in `<available_tools>`.
|
||||
- Prefer primary and recent sources when recency matters.
|
||||
- If the delegated request is underspecified, return `status=blocked` with the missing research constraints.
|
||||
- Never fabricate facts, citations, URLs, or quote text.
|
||||
</tool_policy>
|
||||
|
||||
<citations>
|
||||
`web_search` returns a `<web_results>` block whose results are each prefixed with a bracketed label — `[1]`, `[2]`, `[3]`. That `[n]` is the citation label. When a finding came from a specific result, append its `[n]` to that finding, copying the label **exactly** as shown. The caller relays these labels verbatim and the server resolves each one, so a wrong number silently breaks the citation.
|
||||
|
||||
- Use the exact `[n]` shown next to the result you actually used; never renumber, guess, or invent a label.
|
||||
- Before emitting an `[n]`, confirm that bracketed label appears in the `web_search` output this turn. If you can't see it, omit it.
|
||||
- Write the bare label `[n]` only — no `[citation:…]` wrapper, no markdown links.
|
||||
- Several results behind one finding → each in its own brackets with nothing between: `[1][2]`.
|
||||
- `scrape_webpage` returns raw page text with no `[n]` labels; a fact drawn only from a scrape carries no citation (report the URL in `evidence.sources` instead).
|
||||
</citations>
|
||||
|
||||
<out_of_scope>
|
||||
- Do not execute connector mutations (email/calendar/docs/chat writes) or deliverable generation.
|
||||
</out_of_scope>
|
||||
|
||||
<safety>
|
||||
- Report uncertainty explicitly when evidence is incomplete or conflicting.
|
||||
- Never present unverified claims as facts.
|
||||
</safety>
|
||||
|
||||
<failure_policy>
|
||||
- On tool failure, return `status=error` with a concise recovery `next_step`.
|
||||
- On no useful evidence, return `status=blocked` with recommended narrower filters.
|
||||
</failure_policy>
|
||||
|
||||
<output_contract>
|
||||
Return **only** one JSON object (no markdown/prose):
|
||||
{
|
||||
"status": "success" | "partial" | "blocked" | "error",
|
||||
"action_summary": string,
|
||||
"evidence": {
|
||||
"findings": string[],
|
||||
"sources": string[],
|
||||
"confidence": "high" | "medium" | "low"
|
||||
},
|
||||
"next_step": string | null,
|
||||
"missing_fields": string[] | null,
|
||||
"assumptions": string[] | null
|
||||
}
|
||||
<include snippet="output_contract_base"/>
|
||||
Route-specific rules:
|
||||
- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact. Append the supporting `[n]` to each finding drawn from a `web_search` result. Do not paste raw paragraphs, scraped pages, or quote blocks.
|
||||
- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once. (Citations travel as `[n]`; `sources` is for transparency and for scrape-only facts that carry no `[n]`.)
|
||||
</output_contract>
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
"""Research-stage tools: web search (shared) and scrape."""
|
||||
|
||||
from app.agents.chat.shared.tools.web_search import create_web_search_tool
|
||||
|
||||
from .scrape_webpage import create_scrape_webpage_tool
|
||||
|
||||
__all__ = [
|
||||
"create_scrape_webpage_tool",
|
||||
"create_web_search_tool",
|
||||
]
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
"""``research`` native tools and (empty) permission ruleset."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
|
||||
from app.agents.chat.shared.tools.web_search import create_web_search_tool
|
||||
|
||||
from .scrape_webpage import create_scrape_webpage_tool
|
||||
|
||||
NAME = "research"
|
||||
|
||||
RULESET = Ruleset(origin=NAME, rules=[])
|
||||
|
||||
|
||||
def load_tools(
|
||||
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
|
||||
) -> list[BaseTool]:
|
||||
d = {**(dependencies or {}), **kwargs}
|
||||
return [
|
||||
create_web_search_tool(
|
||||
workspace_id=d.get("workspace_id"),
|
||||
available_connectors=d.get("available_connectors"),
|
||||
),
|
||||
create_scrape_webpage_tool(),
|
||||
]
|
||||
|
|
@ -1,347 +0,0 @@
|
|||
"""Scrape pages via WebCrawlerConnector; YouTube URLs use the transcript API instead of HTML crawl."""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from fake_useragent import UserAgent
|
||||
from langchain_core.tools import tool
|
||||
from requests import Session
|
||||
from scrapling.fetchers import AsyncFetcher
|
||||
from youtube_transcript_api import YouTubeTranscriptApi
|
||||
|
||||
from app.proprietary.web_crawler import (
|
||||
CrawlOutcomeStatus,
|
||||
WebCrawlerConnector,
|
||||
)
|
||||
from app.tasks.document_processors.youtube_processor import get_youtube_video_id
|
||||
from app.utils.proxy import get_proxy_url, get_requests_proxies
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _bill_successful_scrape() -> None:
|
||||
"""Fold one successful crawl into the current chat turn's bill (Phase 3c).
|
||||
|
||||
The cost rides the turn accumulator and settles at the premium
|
||||
``finalize_credit`` step — no separate wallet hit. Free / BYOK / anonymous
|
||||
turns (which never reserve/finalize) still record the line in the
|
||||
breakdown but are never debited. No-op when crawl billing is disabled or
|
||||
there is no active turn (e.g. non-chat callers).
|
||||
"""
|
||||
from app.services.token_tracking_service import get_current_accumulator
|
||||
from app.services.web_crawl_credit_service import WebCrawlCreditService
|
||||
|
||||
if not WebCrawlCreditService.billing_enabled():
|
||||
return
|
||||
acc = get_current_accumulator()
|
||||
if acc is None:
|
||||
return
|
||||
acc.add(
|
||||
model="web_crawl",
|
||||
prompt_tokens=0,
|
||||
completion_tokens=0,
|
||||
total_tokens=0,
|
||||
cost_micros=WebCrawlCreditService.successes_to_micros(1),
|
||||
call_kind="web_crawl",
|
||||
)
|
||||
|
||||
|
||||
def _bill_captcha_attempts(outcome) -> None:
|
||||
"""Fold captcha solve attempts (Phase 3d) into the current chat turn's bill.
|
||||
|
||||
Per *attempt*, not per success: a solve that didn't rescue the crawl still
|
||||
cost real solver money, so this runs before the success/failure branch.
|
||||
Mirrors :func:`_bill_successful_scrape` (rides the turn accumulator, settles
|
||||
at ``finalize_credit``; record-only on free/anonymous turns). No-op when
|
||||
captcha billing is off, there were no attempts, or no turn is active.
|
||||
"""
|
||||
from app.services.token_tracking_service import get_current_accumulator
|
||||
from app.services.web_crawl_credit_service import WebCrawlCreditService
|
||||
|
||||
if not WebCrawlCreditService.captcha_billing_enabled():
|
||||
return
|
||||
attempts = getattr(outcome, "captcha_attempts", 0) or 0
|
||||
if attempts <= 0:
|
||||
return
|
||||
acc = get_current_accumulator()
|
||||
if acc is None:
|
||||
return
|
||||
acc.add(
|
||||
model="web_crawl_captcha",
|
||||
prompt_tokens=0,
|
||||
completion_tokens=0,
|
||||
total_tokens=0,
|
||||
cost_micros=WebCrawlCreditService.captcha_solves_to_micros(attempts),
|
||||
call_kind="web_crawl_captcha",
|
||||
)
|
||||
|
||||
|
||||
def extract_domain(url: str) -> str:
|
||||
"""Extract the domain from a URL."""
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
domain = parsed.netloc
|
||||
if domain.startswith("www."):
|
||||
domain = domain[4:]
|
||||
return domain
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def generate_scrape_id(url: str) -> str:
|
||||
"""Generate a unique ID for a scraped webpage."""
|
||||
hash_val = hashlib.md5(url.encode()).hexdigest()[:12]
|
||||
return f"scrape-{hash_val}"
|
||||
|
||||
|
||||
def truncate_content(content: str, max_length: int = 50000) -> tuple[str, bool]:
|
||||
"""
|
||||
Truncate content to a maximum length.
|
||||
|
||||
Returns:
|
||||
Tuple of (truncated_content, was_truncated)
|
||||
"""
|
||||
if len(content) <= max_length:
|
||||
return content, False
|
||||
|
||||
# Prefer truncating at a sentence/paragraph boundary.
|
||||
truncated = content[:max_length]
|
||||
last_period = truncated.rfind(".")
|
||||
last_newline = truncated.rfind("\n\n")
|
||||
|
||||
boundary = max(last_period, last_newline)
|
||||
if boundary > max_length * 0.8: # only if the boundary isn't too far back
|
||||
truncated = content[: boundary + 1]
|
||||
|
||||
return truncated + "\n\n[Content truncated...]", True
|
||||
|
||||
|
||||
async def _scrape_youtube_video(
|
||||
url: str, video_id: str, max_length: int
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Fetch YouTube video metadata and transcript via the YouTubeTranscriptApi.
|
||||
|
||||
Returns a result dict in the same shape as the regular scrape_webpage output.
|
||||
"""
|
||||
scrape_id = generate_scrape_id(url)
|
||||
domain = "youtube.com"
|
||||
|
||||
# --- Video metadata via oEmbed ---
|
||||
residential_proxies = get_requests_proxies()
|
||||
|
||||
params = {
|
||||
"format": "json",
|
||||
"url": f"https://www.youtube.com/watch?v={video_id}",
|
||||
}
|
||||
oembed_url = "https://www.youtube.com/oembed"
|
||||
|
||||
try:
|
||||
oembed_fetch_start = time.perf_counter()
|
||||
oembed_page = await AsyncFetcher.get(
|
||||
oembed_url,
|
||||
params=params,
|
||||
proxy=get_proxy_url(),
|
||||
stealthy_headers=True,
|
||||
)
|
||||
logger.info(
|
||||
"[scrape_webpage][perf] source=oembed video=%s status=%s fetch_ms=%.1f",
|
||||
video_id,
|
||||
getattr(oembed_page, "status", None),
|
||||
(time.perf_counter() - oembed_fetch_start) * 1000,
|
||||
)
|
||||
video_data = oembed_page.json()
|
||||
except Exception:
|
||||
video_data = {}
|
||||
|
||||
title = video_data.get("title", "YouTube Video")
|
||||
author = video_data.get("author_name", "Unknown")
|
||||
|
||||
# --- Transcript via YouTubeTranscriptApi ---
|
||||
try:
|
||||
transcript_fetch_start = time.perf_counter()
|
||||
ua = UserAgent()
|
||||
http_client = Session()
|
||||
http_client.headers.update({"User-Agent": ua.random})
|
||||
if residential_proxies:
|
||||
http_client.proxies.update(residential_proxies)
|
||||
ytt_api = YouTubeTranscriptApi(http_client=http_client)
|
||||
|
||||
# Pick the first transcript (video's primary language) rather than
|
||||
# defaulting to English.
|
||||
transcript_list = ytt_api.list(video_id)
|
||||
transcript = next(iter(transcript_list))
|
||||
captions = transcript.fetch()
|
||||
|
||||
logger.info(
|
||||
"[scrape_webpage][perf] source=transcript video=%s fetch_ms=%.1f",
|
||||
video_id,
|
||||
(time.perf_counter() - transcript_fetch_start) * 1000,
|
||||
)
|
||||
logger.info(
|
||||
f"[scrape_webpage] Fetched transcript for {video_id} "
|
||||
f"in {transcript.language} ({transcript.language_code})"
|
||||
)
|
||||
|
||||
transcript_segments = []
|
||||
for line in captions:
|
||||
start_time = line.start
|
||||
duration = line.duration
|
||||
text = line.text
|
||||
timestamp = f"[{start_time:.2f}s-{start_time + duration:.2f}s]"
|
||||
transcript_segments.append(f"{timestamp} {text}")
|
||||
transcript_text = "\n".join(transcript_segments)
|
||||
except Exception as e:
|
||||
logger.warning(f"[scrape_webpage] No transcript for video {video_id}: {e}")
|
||||
transcript_text = f"No captions available for this video. Error: {e!s}"
|
||||
|
||||
content = f"# {title}\n\n**Author:** {author}\n**Video ID:** {video_id}\n\n## Transcript\n\n{transcript_text}"
|
||||
|
||||
content, was_truncated = truncate_content(content, max_length)
|
||||
word_count = len(content.split())
|
||||
|
||||
description = f"YouTube video by {author}"
|
||||
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": title,
|
||||
"description": description,
|
||||
"content": content,
|
||||
"domain": domain,
|
||||
"word_count": word_count,
|
||||
"was_truncated": was_truncated,
|
||||
"crawler_type": "youtube_transcript",
|
||||
"author": author,
|
||||
}
|
||||
|
||||
|
||||
def create_scrape_webpage_tool():
|
||||
"""
|
||||
Factory function to create the scrape_webpage tool.
|
||||
|
||||
Returns:
|
||||
A configured tool function for scraping webpages.
|
||||
"""
|
||||
|
||||
@tool
|
||||
async def scrape_webpage(
|
||||
url: str,
|
||||
max_length: int = 50000,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Scrape and extract the main content from a webpage.
|
||||
|
||||
Use this tool when the user wants you to read, summarize, or answer
|
||||
questions about a specific webpage's content. This tool actually
|
||||
fetches and reads the full page content. For YouTube video URLs it
|
||||
fetches the transcript directly instead of crawling the page.
|
||||
|
||||
Common triggers:
|
||||
- "Read this article and summarize it"
|
||||
- "What does this page say about X?"
|
||||
- "Summarize this blog post for me"
|
||||
- "Tell me the key points from this article"
|
||||
- "What's in this webpage?"
|
||||
|
||||
Args:
|
||||
url: The URL of the webpage to scrape (must be HTTP/HTTPS)
|
||||
max_length: Maximum content length to return (default: 50000 chars)
|
||||
|
||||
Returns:
|
||||
A dictionary containing:
|
||||
- id: Unique identifier for this scrape
|
||||
- assetId: The URL (for deduplication)
|
||||
- kind: "article" (type of content)
|
||||
- href: The URL to open when clicked
|
||||
- title: Page title
|
||||
- description: Brief description or excerpt
|
||||
- content: The extracted main content (markdown format)
|
||||
- domain: The domain name
|
||||
- word_count: Approximate word count
|
||||
- was_truncated: Whether content was truncated
|
||||
- error: Error message (if scraping failed)
|
||||
"""
|
||||
scrape_id = generate_scrape_id(url)
|
||||
domain = extract_domain(url)
|
||||
|
||||
if not url.startswith(("http://", "https://")):
|
||||
url = f"https://{url}"
|
||||
|
||||
try:
|
||||
# YouTube URLs use the transcript API instead of crawling.
|
||||
video_id = get_youtube_video_id(url)
|
||||
if video_id:
|
||||
return await _scrape_youtube_video(url, video_id, max_length)
|
||||
|
||||
connector = WebCrawlerConnector()
|
||||
outcome = await connector.crawl_url(url)
|
||||
|
||||
# 03d: bill any captcha attempts (even if the crawl ultimately failed).
|
||||
_bill_captcha_attempts(outcome)
|
||||
|
||||
if outcome.status is not CrawlOutcomeStatus.SUCCESS or not outcome.result:
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": domain or "Webpage",
|
||||
"domain": domain,
|
||||
"error": outcome.error or "No content returned from crawler",
|
||||
}
|
||||
|
||||
result = outcome.result
|
||||
_bill_successful_scrape()
|
||||
content = result.get("content", "")
|
||||
metadata = result.get("metadata", {})
|
||||
|
||||
title = metadata.get("title", "")
|
||||
if not title:
|
||||
title = domain or url.split("/")[-1] or "Webpage"
|
||||
|
||||
description = metadata.get("description", "")
|
||||
if not description and content:
|
||||
first_para = content.split("\n\n")[0] if content else ""
|
||||
description = (
|
||||
first_para[:300] + "..." if len(first_para) > 300 else first_para
|
||||
)
|
||||
|
||||
content, was_truncated = truncate_content(content, max_length)
|
||||
word_count = len(content.split())
|
||||
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": title,
|
||||
"description": description,
|
||||
"content": content,
|
||||
"domain": domain,
|
||||
"word_count": word_count,
|
||||
"was_truncated": was_truncated,
|
||||
"crawler_type": result.get("crawler_type", "unknown"),
|
||||
"author": metadata.get("author"),
|
||||
"date": metadata.get("date"),
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
error_message = str(e)
|
||||
logger.error(f"[scrape_webpage] Error scraping {url}: {error_message}")
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": domain or "Webpage",
|
||||
"domain": domain,
|
||||
"error": f"Failed to scrape: {error_message[:100]}",
|
||||
}
|
||||
|
||||
return scrape_webpage
|
||||
|
|
@ -0,0 +1 @@
|
|||
"""``web_crawler`` builtin subagent: crawl single URLs or spider whole sites."""
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
"""``web_crawler`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
|
||||
read_md_file,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import (
|
||||
pack_subagent,
|
||||
)
|
||||
|
||||
from .tools.index import NAME, RULESET, load_tools
|
||||
|
||||
|
||||
def build_subagent(
|
||||
*,
|
||||
dependencies: dict[str, Any],
|
||||
model: BaseChatModel | None = None,
|
||||
middleware_stack: dict[str, Any] | None = None,
|
||||
mcp_tools: list[BaseTool] | None = None,
|
||||
) -> SurfSenseSubagentSpec:
|
||||
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
|
||||
description = (
|
||||
read_md_file(__package__, "description").strip()
|
||||
or "Crawls live public web pages and whole sites for this workspace."
|
||||
)
|
||||
system_prompt = read_md_file(__package__, "system_prompt").strip()
|
||||
return pack_subagent(
|
||||
name=NAME,
|
||||
description=description,
|
||||
system_prompt=system_prompt,
|
||||
tools=tools,
|
||||
ruleset=RULESET,
|
||||
dependencies=dependencies,
|
||||
model=model,
|
||||
middleware_stack=middleware_stack,
|
||||
)
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
Web crawling specialist: fetches live public web pages by URL and can spider a whole website (follow its links to a given depth), returning clean markdown, page metadata, and crawl provenance. Also compares freshly crawled data against earlier findings in this chat.
|
||||
Use whenever the task needs current content pulled from the open web rather than the workspace's own documents or connectors. Triggers include "scrape", "crawl", "fetch this URL/page", "read this website", "crawl this site", "get the pages under X", "check the price/stock/listing", and "what changed vs earlier in this chat". Not for YouTube links (use the youtube specialist) or for searching the web to discover unknown URLs.
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
You are the SurfSense web crawling sub-agent.
|
||||
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
|
||||
|
||||
<goal>
|
||||
Answer the delegated question from live web evidence gathered with `web_crawl`, comparing against earlier results already in this conversation when the task calls for it.
|
||||
</goal>
|
||||
|
||||
<available_tools>
|
||||
- `web_crawl`
|
||||
</available_tools>
|
||||
|
||||
<playbook>
|
||||
- Single page(s): call `web_crawl` with the URL(s) in `startUrls` and `maxCrawlDepth=0`.
|
||||
- Whole site / "pages under X": set `maxCrawlDepth` to 1+ to follow links, and cap the run with `maxCrawlPages`. The crawl stays on the start URL's site.
|
||||
- Batch known URLs into one `web_crawl` call (pass them all in `startUrls`) rather than many single-URL calls.
|
||||
- Keep depth and page caps as small as the task allows — each fetched page is billable.
|
||||
- Comparison requests: crawl the current values, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new).
|
||||
</playbook>
|
||||
|
||||
<tool_policy>
|
||||
- Use only tools in `<available_tools>`.
|
||||
- A `web_crawl` item whose `status` is not `success` returned no content — report it unavailable, never invent it.
|
||||
- Report only deltas you can point to in the evidence. Never fabricate facts, URLs, prices, or quotes.
|
||||
</tool_policy>
|
||||
|
||||
<out_of_scope>
|
||||
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
|
||||
- YouTube URLs belong to the youtube specialist, not here.
|
||||
</out_of_scope>
|
||||
|
||||
<safety>
|
||||
- Report uncertainty explicitly when evidence is incomplete or conflicting.
|
||||
- Never present unverified claims as facts.
|
||||
</safety>
|
||||
|
||||
<failure_policy>
|
||||
- Underspecified request — no usable URL to start from — return `status=blocked` with the missing fields.
|
||||
- Tool failure: return `status=error` with a concise recovery `next_step`.
|
||||
- No useful evidence: return `status=blocked` with the URLs you still need or a narrower scope.
|
||||
</failure_policy>
|
||||
|
||||
<output_contract>
|
||||
Return **only** one JSON object (no markdown/prose):
|
||||
{
|
||||
"status": "success" | "partial" | "blocked" | "error",
|
||||
"action_summary": string,
|
||||
"evidence": {
|
||||
"findings": string[],
|
||||
"sources": string[],
|
||||
"confidence": "high" | "medium" | "low"
|
||||
},
|
||||
"next_step": string | null,
|
||||
"missing_fields": string[] | null,
|
||||
"assumptions": string[] | null
|
||||
}
|
||||
<include snippet="output_contract_base"/>
|
||||
Route-specific rules:
|
||||
- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw crawled pages.
|
||||
- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once.
|
||||
</output_contract>
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
"""``web_crawler`` sub-agent tools: the unified web.crawl capability verb."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
|
||||
from app.capabilities.core.access.agent import build_capability_tools
|
||||
from app.capabilities.web.crawl.definition import WEB_CRAWL
|
||||
|
||||
NAME = "web_crawler"
|
||||
|
||||
RULESET = Ruleset(origin=NAME, rules=[])
|
||||
|
||||
_CI_VERBS = [WEB_CRAWL]
|
||||
|
||||
|
||||
def load_tools(
|
||||
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
|
||||
) -> list[BaseTool]:
|
||||
d = {**(dependencies or {}), **kwargs}
|
||||
return build_capability_tools(
|
||||
workspace_id=d.get("workspace_id"),
|
||||
capabilities=_CI_VERBS,
|
||||
)
|
||||
|
|
@ -0,0 +1 @@
|
|||
"""``youtube`` builtin subagent: structured YouTube video/channel/comment data."""
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
"""``youtube`` route: ``SurfSenseSubagentSpec`` builder for deepagents."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.language_models import BaseChatModel
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.md_file_reader import (
|
||||
read_md_file,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.spec import SurfSenseSubagentSpec
|
||||
from app.agents.chat.multi_agent_chat.subagents.shared.subagent_builder import (
|
||||
pack_subagent,
|
||||
)
|
||||
|
||||
from .tools.index import NAME, RULESET, load_tools
|
||||
|
||||
|
||||
def build_subagent(
|
||||
*,
|
||||
dependencies: dict[str, Any],
|
||||
model: BaseChatModel | None = None,
|
||||
middleware_stack: dict[str, Any] | None = None,
|
||||
mcp_tools: list[BaseTool] | None = None,
|
||||
) -> SurfSenseSubagentSpec:
|
||||
tools = [*load_tools(dependencies=dependencies), *(mcp_tools or [])]
|
||||
description = (
|
||||
read_md_file(__package__, "description").strip()
|
||||
or "Pulls structured data from YouTube videos, channels, playlists, and comments."
|
||||
)
|
||||
system_prompt = read_md_file(__package__, "system_prompt").strip()
|
||||
return pack_subagent(
|
||||
name=NAME,
|
||||
description=description,
|
||||
system_prompt=system_prompt,
|
||||
tools=tools,
|
||||
ruleset=RULESET,
|
||||
dependencies=dependencies,
|
||||
model=model,
|
||||
middleware_stack=middleware_stack,
|
||||
)
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
YouTube specialist: pulls structured data from YouTube — videos, channels, playlists, shorts, and hashtags (title, views, likes, publish date, channel info, description, optional subtitles), finds videos by search query, and fetches a video's comments and replies. Also compares fresh YouTube data against earlier findings in this chat.
|
||||
Use whenever the task involves YouTube content or a youtube.com/youtu.be link. Triggers include "get this YouTube video/channel/playlist", "find videos about X on YouTube", "how many views/likes", "get the transcript/subtitles", "get the comments on this video", "what are people saying about this video", and comparisons against earlier YouTube results in this chat. Not for general web pages (use the web crawling specialist for non-YouTube URLs).
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
You are the SurfSense YouTube sub-agent.
|
||||
You receive delegated instructions from a supervisor agent and return structured results for supervisor synthesis.
|
||||
|
||||
<goal>
|
||||
Answer the delegated question from live YouTube data gathered with your verbs, comparing against earlier results already in this conversation when the task calls for it.
|
||||
</goal>
|
||||
|
||||
<available_tools>
|
||||
- `youtube_scrape`
|
||||
- `youtube_comments`
|
||||
</available_tools>
|
||||
|
||||
<playbook>
|
||||
- Known video/channel/playlist/shorts/hashtag links: call `youtube_scrape` with the links in `urls`.
|
||||
- Finding videos on a topic: call `youtube_scrape` with `search_queries`.
|
||||
- Comments / sentiment on specific videos: call `youtube_comments` with the video `urls`.
|
||||
- Batch multiple URLs (or queries) into one call rather than many single-item calls.
|
||||
- Comparison requests: pull the current values, compare against prior values already in this conversation's earlier tool results, and report concrete deltas (added, removed, old -> new).
|
||||
</playbook>
|
||||
|
||||
<tool_policy>
|
||||
- Use only tools in `<available_tools>`.
|
||||
- An item whose `status` is not `success` returned no data — report it unavailable, never invent it.
|
||||
- Report only deltas you can point to in the evidence. Never fabricate facts, counts, quotes, or URLs.
|
||||
</tool_policy>
|
||||
|
||||
<out_of_scope>
|
||||
- Do not generate deliverables or perform connector mutations; return findings for the supervisor to act on.
|
||||
- Non-YouTube web pages belong to the web crawling specialist, not here.
|
||||
</out_of_scope>
|
||||
|
||||
<safety>
|
||||
- Report uncertainty explicitly when evidence is incomplete or conflicting.
|
||||
- Never present unverified claims as facts.
|
||||
</safety>
|
||||
|
||||
<failure_policy>
|
||||
- Underspecified request — no usable URL or search query — return `status=blocked` with the missing fields.
|
||||
- Tool failure: return `status=error` with a concise recovery `next_step`.
|
||||
- No useful evidence: return `status=blocked` with a narrower query or the URLs you still need.
|
||||
</failure_policy>
|
||||
|
||||
<output_contract>
|
||||
Return **only** one JSON object (no markdown/prose):
|
||||
{
|
||||
"status": "success" | "partial" | "blocked" | "error",
|
||||
"action_summary": string,
|
||||
"evidence": {
|
||||
"findings": string[],
|
||||
"sources": string[],
|
||||
"confidence": "high" | "medium" | "low"
|
||||
},
|
||||
"next_step": string | null,
|
||||
"missing_fields": string[] | null,
|
||||
"assumptions": string[] | null
|
||||
}
|
||||
<include snippet="output_contract_base"/>
|
||||
Route-specific rules:
|
||||
- `evidence.findings`: max 10 entries, each a single sentence stating one distinct fact or delta. Do not paste raw payloads.
|
||||
- `evidence.sources`: max 10 URLs, one per finding when applicable. List each URL once.
|
||||
</output_contract>
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
"""``youtube`` sub-agent tools: the YouTube scrape + comments capability verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.tools import BaseTool
|
||||
|
||||
from app.agents.chat.multi_agent_chat.shared.permissions import Ruleset
|
||||
from app.capabilities.core.access.agent import build_capability_tools
|
||||
from app.capabilities.youtube.comments.definition import YOUTUBE_COMMENTS
|
||||
from app.capabilities.youtube.scrape.definition import YOUTUBE_SCRAPE
|
||||
|
||||
NAME = "youtube"
|
||||
|
||||
RULESET = Ruleset(origin=NAME, rules=[])
|
||||
|
||||
_CI_VERBS = [YOUTUBE_SCRAPE, YOUTUBE_COMMENTS]
|
||||
|
||||
|
||||
def load_tools(
|
||||
*, dependencies: dict[str, Any] | None = None, **kwargs: Any
|
||||
) -> list[BaseTool]:
|
||||
d = {**(dependencies or {}), **kwargs}
|
||||
return build_capability_tools(
|
||||
workspace_id=d.get("workspace_id"),
|
||||
capabilities=_CI_VERBS,
|
||||
)
|
||||
|
|
@ -15,14 +15,20 @@ from app.agents.chat.multi_agent_chat.constants import (
|
|||
from app.agents.chat.multi_agent_chat.subagents.builtins.deliverables.agent import (
|
||||
build_subagent as build_deliverables_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.google_maps.agent import (
|
||||
build_subagent as build_google_maps_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.knowledge_base.agent import (
|
||||
build_subagent as build_knowledge_base_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.memory.agent import (
|
||||
build_subagent as build_memory_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.research.agent import (
|
||||
build_subagent as build_research_subagent,
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.web_crawler.agent import (
|
||||
build_subagent as build_web_crawler_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.builtins.youtube.agent import (
|
||||
build_subagent as build_youtube_subagent,
|
||||
)
|
||||
from app.agents.chat.multi_agent_chat.subagents.connectors.airtable.agent import (
|
||||
build_subagent as build_airtable_subagent,
|
||||
|
|
@ -99,6 +105,7 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = {
|
|||
"dropbox": build_dropbox_subagent,
|
||||
"gmail": build_gmail_subagent,
|
||||
"google_drive": build_google_drive_subagent,
|
||||
"google_maps": build_google_maps_subagent,
|
||||
"jira": build_jira_subagent,
|
||||
"knowledge_base": build_knowledge_base_subagent,
|
||||
"linear": build_linear_subagent,
|
||||
|
|
@ -106,9 +113,10 @@ SUBAGENT_BUILDERS_BY_NAME: dict[str, SubagentBuilder] = {
|
|||
"memory": build_memory_subagent,
|
||||
"notion": build_notion_subagent,
|
||||
"onedrive": build_onedrive_subagent,
|
||||
"research": build_research_subagent,
|
||||
"slack": build_slack_subagent,
|
||||
"teams": build_teams_subagent,
|
||||
"web_crawler": build_web_crawler_subagent,
|
||||
"youtube": build_youtube_subagent,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -119,7 +127,7 @@ def _route_resource_package(builder: SubagentBuilder) -> str:
|
|||
|
||||
def main_prompt_registry_subagent_lines(exclude: list[str]) -> list[tuple[str, str]]:
|
||||
"""(name, description) for registry specialists included for **task** (same rules as ``build_subagents``)."""
|
||||
banned = frozenset(("memory", "research")) | frozenset(exclude)
|
||||
banned = frozenset(("memory",)) | frozenset(exclude)
|
||||
rows: list[tuple[str, str]] = []
|
||||
for name in sorted(SUBAGENT_BUILDERS_BY_NAME):
|
||||
if name in banned:
|
||||
|
|
@ -189,10 +197,10 @@ def build_subagents(
|
|||
disabled_tools: list[str] | None = None,
|
||||
ask_kb_tool: BaseTool | None = None,
|
||||
) -> list[SubAgent]:
|
||||
"""Build registry subagents; skip memory/research; skip names in exclude."""
|
||||
"""Build registry subagents; skip memory; skip names in exclude."""
|
||||
mcp = mcp_tools_by_agent or {}
|
||||
specs: list[SubAgent] = []
|
||||
excluded = ["memory", "research"]
|
||||
excluded = ["memory"]
|
||||
if exclude:
|
||||
excluded.extend(exclude)
|
||||
disabled_names = frozenset(disabled_tools or ())
|
||||
|
|
|
|||
|
|
@ -84,16 +84,9 @@ async def build_dependencies(
|
|||
connector_service = await setup_connector_service(
|
||||
session, workspace_id=workspace_id
|
||||
)
|
||||
# Per-task InMemorySaver: the shared Postgres checkpointer's connection
|
||||
# pool binds connections to the loop that opened them, but Celery uses a
|
||||
# fresh loop per task, so the next task hangs 30s on a dead-loop connection
|
||||
# (`PoolTimeout`). InMemorySaver has no pool and dies with the task — fine
|
||||
# while runs are one-shot (the checkpoint only spans one graph execution).
|
||||
#
|
||||
# TODO(checkpointer): when runs need durability (crash-resume or HITL
|
||||
# interrupt/resume across tasks), dispose the checkpointer pool around each
|
||||
# Celery task in `run_async_celery_task` — as `_dispose_shared_db_engine`
|
||||
# already does for the SQLAlchemy pool — then use the shared checkpointer.
|
||||
# One-shot runs on a fresh thread don't outlive a single execution, so an
|
||||
# in-memory checkpointer suffices. Ongoing chat-thread turns that need
|
||||
# durable memory use the shared Postgres checkpointer via stream_new_chat.
|
||||
checkpointer = InMemorySaver()
|
||||
return AgentDependencies(
|
||||
llm=llm,
|
||||
|
|
|
|||
5
surfsense_backend/app/capabilities/__init__.py
Normal file
5
surfsense_backend/app/capabilities/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""Scraper capability registry — typed, stateless verbs. See plans/backend/04-capabilities.md."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
__all__: list[str] = []
|
||||
30
surfsense_backend/app/capabilities/core/__init__.py
Normal file
30
surfsense_backend/app/capabilities/core/__init__.py
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
"""Capability framework kernel: registry contracts, store, billing, and access doors."""
|
||||
|
||||
from app.capabilities.core.billing import charge_capability, gate_capability
|
||||
from app.capabilities.core.store import (
|
||||
all_capabilities,
|
||||
get_capability,
|
||||
register_capability,
|
||||
)
|
||||
from app.capabilities.core.types import (
|
||||
BillableInput,
|
||||
BillableOutput,
|
||||
BillingUnit,
|
||||
Capability,
|
||||
CapabilityContext,
|
||||
Executor,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"BillableInput",
|
||||
"BillableOutput",
|
||||
"BillingUnit",
|
||||
"Capability",
|
||||
"CapabilityContext",
|
||||
"Executor",
|
||||
"all_capabilities",
|
||||
"charge_capability",
|
||||
"gate_capability",
|
||||
"get_capability",
|
||||
"register_capability",
|
||||
]
|
||||
|
|
@ -0,0 +1 @@
|
|||
"""Access doors (05): thin adapters that expose the 04 registry as REST/MCP/agent."""
|
||||
52
surfsense_backend/app/capabilities/core/access/agent.py
Normal file
52
surfsense_backend/app/capabilities/core/access/agent.py
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
"""Generate the agent door from the capability registry (05).
|
||||
|
||||
One LangChain tool per verb; each runs the same thin adapter as the REST door
|
||||
(``access/rest.py``): meter-gate -> executor -> charge. The tool returns the
|
||||
verb's serialized output so the model can reason over it; UI cards are the SSE
|
||||
emission handler's job, not this generator's.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from langchain_core.tools import BaseTool, StructuredTool
|
||||
|
||||
from app.capabilities.core.billing import charge_capability, gate_capability
|
||||
from app.capabilities.core.store import all_capabilities
|
||||
from app.capabilities.core.types import Capability, CapabilityContext
|
||||
from app.db import async_session_maker
|
||||
from app.services.web_crawl_credit_service import InsufficientCreditsError
|
||||
|
||||
|
||||
def build_capability_tools(
|
||||
*,
|
||||
workspace_id: int,
|
||||
capabilities: list[Capability] | None = None,
|
||||
) -> list[BaseTool]:
|
||||
"""Emit one tool per verb (defaults to the whole registry)."""
|
||||
caps = capabilities if capabilities is not None else all_capabilities()
|
||||
return [_capability_tool(cap, workspace_id) for cap in caps]
|
||||
|
||||
|
||||
def _capability_tool(capability: Capability, workspace_id: int) -> BaseTool:
|
||||
input_model = capability.input_schema
|
||||
unit = capability.billing_unit
|
||||
executor = capability.executor
|
||||
|
||||
async def _run(**kwargs: object) -> dict | str:
|
||||
payload = input_model(**kwargs)
|
||||
async with async_session_maker() as session:
|
||||
ctx = CapabilityContext(session=session, workspace_id=workspace_id)
|
||||
try:
|
||||
await gate_capability(payload, unit, ctx)
|
||||
except InsufficientCreditsError as exc:
|
||||
return str(exc)
|
||||
output = await executor(payload)
|
||||
await charge_capability(output, unit, ctx)
|
||||
return output.model_dump()
|
||||
|
||||
return StructuredTool.from_function(
|
||||
coroutine=_run,
|
||||
name=capability.name.replace(".", "_"),
|
||||
description=capability.description,
|
||||
args_schema=input_model,
|
||||
)
|
||||
65
surfsense_backend/app/capabilities/core/access/rate_limit.py
Normal file
65
surfsense_backend/app/capabilities/core/access/rate_limit.py
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
"""Per-workspace rate limit for the capability doors (05).
|
||||
|
||||
A secondary abuse guard; the credit meter-gate (03c) is the primary control.
|
||||
Fixed-window over Redis (shared across workers) with a per-worker in-memory
|
||||
fallback when Redis is unavailable — mirroring the auth-endpoint limiter.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from threading import Lock
|
||||
|
||||
from fastapi import HTTPException, Request, status
|
||||
|
||||
from app.config import config
|
||||
|
||||
CAPABILITY_RATE_LIMIT_PER_MINUTE = 120
|
||||
_WINDOW_SECONDS = 60
|
||||
_KEY_PREFIX = "surfsense:capability_rate_limit"
|
||||
|
||||
_redis = None
|
||||
_memory: dict[str, list[float]] = defaultdict(list)
|
||||
_memory_lock = Lock()
|
||||
|
||||
|
||||
def _redis_client():
|
||||
global _redis
|
||||
if _redis is None:
|
||||
import redis
|
||||
|
||||
_redis = redis.from_url(config.REDIS_APP_URL, decode_responses=True)
|
||||
return _redis
|
||||
|
||||
|
||||
def _incr_memory(key: str, window_seconds: int) -> int:
|
||||
now = time.monotonic()
|
||||
with _memory_lock:
|
||||
hits = [t for t in _memory[key] if now - t < window_seconds]
|
||||
hits.append(now)
|
||||
_memory[key] = hits
|
||||
return len(hits)
|
||||
|
||||
|
||||
def _incr(key: str, window_seconds: int) -> int:
|
||||
"""Increment the window counter for ``key`` and return the new count."""
|
||||
try:
|
||||
client = _redis_client()
|
||||
count = int(client.incr(key))
|
||||
if count == 1:
|
||||
client.expire(key, window_seconds)
|
||||
return count
|
||||
except Exception:
|
||||
return _incr_memory(key, window_seconds)
|
||||
|
||||
|
||||
async def enforce_capability_rate_limit(request: Request) -> None:
|
||||
"""Cap requests per workspace per minute; raise 429 when exceeded."""
|
||||
workspace_id = request.path_params.get("workspace_id")
|
||||
count = _incr(f"{_KEY_PREFIX}:{workspace_id}", _WINDOW_SECONDS)
|
||||
if count > CAPABILITY_RATE_LIMIT_PER_MINUTE:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
detail="Rate limit exceeded for this workspace. Try again shortly.",
|
||||
)
|
||||
80
surfsense_backend/app/capabilities/core/access/rest.py
Normal file
80
surfsense_backend/app/capabilities/core/access/rest.py
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
"""Generate the REST door from the capability registry (05).
|
||||
|
||||
One typed ``POST`` per verb; each runs the same thin adapter:
|
||||
authn -> workspace authz -> meter-gate -> executor -> charge -> typed output.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.auth.context import AuthContext
|
||||
from app.capabilities.core.access.rate_limit import enforce_capability_rate_limit
|
||||
from app.capabilities.core.billing import charge_capability, gate_capability
|
||||
from app.capabilities.core.store import all_capabilities
|
||||
from app.capabilities.core.types import Capability, CapabilityContext
|
||||
from app.db import get_async_session
|
||||
from app.exceptions import ExternalServiceError, SurfSenseError
|
||||
from app.services.web_crawl_credit_service import InsufficientCreditsError
|
||||
from app.users import get_auth_context
|
||||
from app.utils.rbac import check_workspace_access
|
||||
|
||||
|
||||
def build_capabilities_router(
|
||||
capabilities: list[Capability] | None = None,
|
||||
) -> APIRouter:
|
||||
"""Emit one typed route per verb (defaults to the whole registry)."""
|
||||
router = APIRouter(tags=["capabilities"])
|
||||
caps = capabilities if capabilities is not None else all_capabilities()
|
||||
for capability in caps:
|
||||
_register_verb(router, capability)
|
||||
return router
|
||||
|
||||
|
||||
def _register_verb(router: APIRouter, capability: Capability) -> None:
|
||||
input_model = capability.input_schema
|
||||
output_model = capability.output_schema
|
||||
unit = capability.billing_unit
|
||||
executor = capability.executor
|
||||
|
||||
async def endpoint(
|
||||
workspace_id: int,
|
||||
payload: input_model,
|
||||
session: AsyncSession = Depends(get_async_session),
|
||||
auth: AuthContext = Depends(get_auth_context),
|
||||
):
|
||||
await check_workspace_access(session, auth, workspace_id)
|
||||
ctx = CapabilityContext(session=session, workspace_id=workspace_id)
|
||||
try:
|
||||
await gate_capability(payload, unit, ctx)
|
||||
except InsufficientCreditsError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_402_PAYMENT_REQUIRED,
|
||||
detail={
|
||||
"error_code": "insufficient_credits",
|
||||
"message": str(exc),
|
||||
"balance_micros": exc.balance_micros,
|
||||
"required_micros": exc.required_micros,
|
||||
},
|
||||
) from exc
|
||||
|
||||
try:
|
||||
output = await executor(payload)
|
||||
except (SurfSenseError, HTTPException):
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ExternalServiceError(
|
||||
f"The '{capability.name}' capability failed due to an upstream error.",
|
||||
code="CAPABILITY_UPSTREAM_ERROR",
|
||||
) from exc
|
||||
|
||||
await charge_capability(output, unit, ctx)
|
||||
return output
|
||||
|
||||
router.add_api_route(
|
||||
f"/workspaces/{{workspace_id}}/capabilities/{capability.name}",
|
||||
endpoint,
|
||||
methods=["POST"],
|
||||
response_model=output_model,
|
||||
name=f"capability:{capability.name}",
|
||||
dependencies=[Depends(enforce_capability_rate_limit)],
|
||||
)
|
||||
132
surfsense_backend/app/capabilities/core/billing.py
Normal file
132
surfsense_backend/app/capabilities/core/billing.py
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
"""Charge the workspace owner per billable success at the capability executor (03c)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.capabilities.core.types import (
|
||||
BillableInput,
|
||||
BillableOutput,
|
||||
BillingUnit,
|
||||
CapabilityContext,
|
||||
)
|
||||
from app.config import config
|
||||
from app.services.token_tracking_service import record_token_usage
|
||||
from app.services.web_crawl_credit_service import WebCrawlCreditService
|
||||
from app.utils.captcha import captcha_enabled
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
async def gate_capability(
|
||||
payload: BillableInput, unit: BillingUnit | None, ctx: CapabilityContext
|
||||
) -> None:
|
||||
"""Pre-flight: block an over-budget owner before the executor runs (03c).
|
||||
|
||||
Raises ``InsufficientCreditsError`` when the wallet can't cover the input's
|
||||
worst-case ``estimated_units``. ``None`` unit = free = no gate.
|
||||
"""
|
||||
if unit is None:
|
||||
return
|
||||
if unit is BillingUnit.WEB_CRAWL:
|
||||
await _gate_web_crawl(ctx, payload.estimated_units)
|
||||
|
||||
|
||||
async def _gate_web_crawl(ctx: CapabilityContext, estimated_successes: int) -> None:
|
||||
"""Reserve the worst-case cost: crawl successes + worst-case captcha attempts.
|
||||
|
||||
Captcha budget is only reserved when solving is actually enabled — with
|
||||
solving off, attempts can never happen, so reserving would wrongly block a
|
||||
run for captcha that will never be attempted. Mirrors the indexer path (3d).
|
||||
"""
|
||||
service = WebCrawlCreditService(ctx.session)
|
||||
crawl_on = service.billing_enabled()
|
||||
captcha_on = service.captcha_billing_enabled() and captcha_enabled()
|
||||
if not crawl_on and not captcha_on:
|
||||
return
|
||||
owner_user_id = await _resolve_workspace_owner(ctx.session, ctx.workspace_id)
|
||||
if owner_user_id is None:
|
||||
return
|
||||
|
||||
required_micros = 0
|
||||
if crawl_on:
|
||||
required_micros += service.successes_to_micros(estimated_successes)
|
||||
if captcha_on:
|
||||
worst_case_attempts = estimated_successes * config.CAPTCHA_MAX_ATTEMPTS_PER_URL
|
||||
required_micros += service.captcha_solves_to_micros(worst_case_attempts)
|
||||
await service.check_balance(owner_user_id, required_micros)
|
||||
|
||||
|
||||
async def charge_capability(
|
||||
output: BillableOutput, unit: BillingUnit | None, ctx: CapabilityContext
|
||||
) -> None:
|
||||
"""Bill the workspace owner for this result's billable successes (03c). ``None`` = free.
|
||||
|
||||
For crawl-backed verbs this also bills any captcha *attempts* (Phase 3d) as a
|
||||
separate per-attempt unit — the solver charges per attempt even when the crawl
|
||||
ultimately failed, so it can't ride the per-success crawl meter.
|
||||
"""
|
||||
if unit is None:
|
||||
return
|
||||
if unit is BillingUnit.WEB_CRAWL:
|
||||
await _charge_web_crawl(ctx, output.billable_units)
|
||||
await _charge_captcha(ctx, getattr(output, "captcha_attempts", 0))
|
||||
|
||||
|
||||
async def _charge_web_crawl(ctx: CapabilityContext, successes: int) -> None:
|
||||
if successes <= 0:
|
||||
return
|
||||
service = WebCrawlCreditService(ctx.session)
|
||||
if not service.billing_enabled():
|
||||
return
|
||||
owner_user_id = await _resolve_workspace_owner(ctx.session, ctx.workspace_id)
|
||||
if owner_user_id is None:
|
||||
return
|
||||
# Stage the audit row before charge_credits' commit flushes both.
|
||||
await record_token_usage(
|
||||
ctx.session,
|
||||
usage_type="web_crawl",
|
||||
workspace_id=ctx.workspace_id,
|
||||
user_id=owner_user_id,
|
||||
cost_micros=service.successes_to_micros(successes),
|
||||
call_details={"successes": successes},
|
||||
)
|
||||
await service.charge_credits(owner_user_id, successes)
|
||||
|
||||
|
||||
async def _charge_captcha(ctx: CapabilityContext, attempts: int) -> None:
|
||||
if attempts <= 0:
|
||||
return
|
||||
service = WebCrawlCreditService(ctx.session)
|
||||
if not service.captcha_billing_enabled():
|
||||
return
|
||||
owner_user_id = await _resolve_workspace_owner(ctx.session, ctx.workspace_id)
|
||||
if owner_user_id is None:
|
||||
return
|
||||
# Stage the audit row before charge_captcha's commit flushes both.
|
||||
await record_token_usage(
|
||||
ctx.session,
|
||||
usage_type="web_crawl_captcha",
|
||||
workspace_id=ctx.workspace_id,
|
||||
user_id=owner_user_id,
|
||||
cost_micros=service.captcha_solves_to_micros(attempts),
|
||||
call_details={"attempts": attempts},
|
||||
)
|
||||
await service.charge_captcha(owner_user_id, attempts)
|
||||
|
||||
|
||||
async def _resolve_workspace_owner(
|
||||
session: AsyncSession, workspace_id: int
|
||||
) -> UUID | None:
|
||||
"""The ``user_id`` that owns ``workspace_id`` (the crawl payer, not the caller)."""
|
||||
from app.db import Workspace
|
||||
|
||||
result = await session.execute(
|
||||
select(Workspace.user_id).where(Workspace.id == workspace_id)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
20
surfsense_backend/app/capabilities/core/store.py
Normal file
20
surfsense_backend/app/capabilities/core/store.py
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
"""In-process capability registry, populated at import by each verb's ``definition.py``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core.types import Capability
|
||||
|
||||
_REGISTRY: dict[str, Capability] = {}
|
||||
|
||||
|
||||
def register_capability(capability: Capability) -> None:
|
||||
"""Add (or replace) a verb by name."""
|
||||
_REGISTRY[capability.name] = capability
|
||||
|
||||
|
||||
def get_capability(name: str) -> Capability:
|
||||
return _REGISTRY[name]
|
||||
|
||||
|
||||
def all_capabilities() -> list[Capability]:
|
||||
return list(_REGISTRY.values())
|
||||
55
surfsense_backend/app/capabilities/core/types.py
Normal file
55
surfsense_backend/app/capabilities/core/types.py
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
"""``Capability`` registry contracts shared by every verb."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from enum import StrEnum
|
||||
from typing import TYPE_CHECKING, Any, Protocol
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
class BillingUnit(StrEnum):
|
||||
"""The meter a verb charges on (priced by the billing service, 03c). ``None`` = free."""
|
||||
|
||||
WEB_CRAWL = "web_crawl"
|
||||
|
||||
|
||||
class BillableInput(Protocol):
|
||||
"""A billed verb's input that reports its worst-case unit count for pre-flight."""
|
||||
|
||||
@property
|
||||
def estimated_units(self) -> int: ...
|
||||
|
||||
|
||||
class BillableOutput(Protocol):
|
||||
"""A capability output that reports its own billable count."""
|
||||
|
||||
@property
|
||||
def billable_units(self) -> int: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CapabilityContext:
|
||||
"""Request-scoped deps a capability call needs beyond its typed input."""
|
||||
|
||||
session: AsyncSession
|
||||
workspace_id: int
|
||||
|
||||
|
||||
Executor = Callable[[Any], Awaitable[Any]]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Capability:
|
||||
"""One typed verb; the source of truth the doors (05) and agent (07) read."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
input_schema: type[BaseModel]
|
||||
output_schema: type[BaseModel]
|
||||
executor: Executor
|
||||
billing_unit: BillingUnit | None
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
"""``google_maps.*`` namespace: platform-native Google Maps data verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.google_maps.reviews import definition as _reviews # noqa: F401
|
||||
from app.capabilities.google_maps.scrape import definition as _scrape # noqa: F401
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
"""``google_maps.reviews`` verb: Maps place URLs / place IDs → review items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
"""``google_maps.reviews`` capability registration (free — see 04-capabilities open item)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import Capability, register_capability
|
||||
from app.capabilities.google_maps.reviews.executor import build_reviews_executor
|
||||
from app.capabilities.google_maps.reviews.schemas import ReviewsInput, ReviewsOutput
|
||||
|
||||
GOOGLE_MAPS_REVIEWS = Capability(
|
||||
name="google_maps.reviews",
|
||||
description=(
|
||||
"Fetch public reviews for one or more Google Maps places. Give it place "
|
||||
"URLs or place IDs; returns structured review items with author, text, "
|
||||
"star rating, like count, owner response, and timestamps. Use it to "
|
||||
"gauge sentiment or pull recent feedback on specific places."
|
||||
),
|
||||
input_schema=ReviewsInput,
|
||||
output_schema=ReviewsOutput,
|
||||
executor=build_reviews_executor(),
|
||||
billing_unit=None,
|
||||
)
|
||||
|
||||
register_capability(GOOGLE_MAPS_REVIEWS)
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
"""``google_maps.reviews`` executor: verb input → scraper → review items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.google_maps.reviews.schemas import ReviewsInput, ReviewsOutput
|
||||
from app.exceptions import ForbiddenError
|
||||
from app.proprietary.platforms.google_maps import (
|
||||
GoogleMapsReviewsInput,
|
||||
scrape_reviews,
|
||||
)
|
||||
from app.proprietary.platforms.google_maps.scraper import SignInRequiredError
|
||||
|
||||
ReviewsFn = Callable[[GoogleMapsReviewsInput], Awaitable[list[dict]]]
|
||||
|
||||
|
||||
def build_reviews_executor(scrape_fn: ReviewsFn | None = None) -> Executor:
|
||||
"""Bind the executor to a reviews scraper fn (defaults to the proprietary actor)."""
|
||||
scrape_fn = scrape_fn or scrape_reviews
|
||||
|
||||
async def execute(payload: ReviewsInput) -> ReviewsOutput:
|
||||
actor_input = GoogleMapsReviewsInput(
|
||||
startUrls=[{"url": url} for url in payload.urls],
|
||||
placeIds=payload.place_ids,
|
||||
maxReviews=payload.max_reviews,
|
||||
reviewsSort=payload.sort_by,
|
||||
reviewsStartDate=payload.start_date,
|
||||
language=payload.language,
|
||||
)
|
||||
try:
|
||||
items = await scrape_fn(actor_input)
|
||||
except SignInRequiredError as exc:
|
||||
raise ForbiddenError(
|
||||
f"Google sign in required: {exc}", code="GOOGLE_SIGNIN_REQUIRED"
|
||||
) from exc
|
||||
return ReviewsOutput(items=items)
|
||||
|
||||
return execute
|
||||
|
|
@ -0,0 +1,68 @@
|
|||
"""``google_maps.reviews`` I/O contracts.
|
||||
|
||||
A lean surface over ``GoogleMapsReviewsInput``; the scraper's ``ReviewItem`` is
|
||||
reused verbatim as the output element.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.proprietary.platforms.google_maps import ReviewItem
|
||||
|
||||
MAX_MAPS_REVIEW_SOURCES = 20
|
||||
"""Per-call cap on urls + place_ids: bounds how many places one request harvests."""
|
||||
|
||||
|
||||
class ReviewsInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_MAPS_REVIEW_SOURCES,
|
||||
description=(
|
||||
"Google Maps place URLs to fetch reviews for. Provide these OR "
|
||||
"place_ids (at least one is required)."
|
||||
),
|
||||
)
|
||||
place_ids: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_MAPS_REVIEW_SOURCES,
|
||||
description=(
|
||||
"Known Google place IDs (ChIJ...) to fetch reviews for. Provide "
|
||||
"these OR urls."
|
||||
),
|
||||
)
|
||||
max_reviews: int = Field(
|
||||
default=20,
|
||||
ge=1,
|
||||
le=100_000,
|
||||
description="Max reviews to return per place.",
|
||||
)
|
||||
sort_by: Literal["newest", "mostRelevant", "highestRanking", "lowestRanking"] = (
|
||||
Field(
|
||||
default="newest",
|
||||
description="Review ordering.",
|
||||
)
|
||||
)
|
||||
language: str = Field(
|
||||
default="en",
|
||||
description="Review language code, e.g. 'en', 'fr'.",
|
||||
)
|
||||
start_date: str | None = Field(
|
||||
default=None,
|
||||
description="Only reviews on/after this ISO date, e.g. '2024-01-01'.",
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_a_source(self) -> ReviewsInput:
|
||||
if not (self.urls or self.place_ids):
|
||||
raise ValueError("Provide at least one of 'urls' or 'place_ids'.")
|
||||
return self
|
||||
|
||||
|
||||
class ReviewsOutput(BaseModel):
|
||||
items: list[ReviewItem] = Field(
|
||||
default_factory=list,
|
||||
description="One item per review, in the scraper's emission order.",
|
||||
)
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
"""``google_maps.scrape`` verb: search queries / Maps URLs / place IDs → place items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
"""``google_maps.scrape`` capability registration (free — see 04-capabilities open item)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import Capability, register_capability
|
||||
from app.capabilities.google_maps.scrape.executor import build_scrape_executor
|
||||
from app.capabilities.google_maps.scrape.schemas import ScrapeInput, ScrapeOutput
|
||||
|
||||
GOOGLE_MAPS_SCRAPE = Capability(
|
||||
name="google_maps.scrape",
|
||||
description=(
|
||||
"Scrape public Google Maps places. Give it search queries (optionally "
|
||||
"scoped by location), Google Maps URLs, or place IDs, and it returns "
|
||||
"structured place items — name, address, category, phone, website, "
|
||||
"rating, review count, coordinates, and opening hours. Set "
|
||||
"include_details for richer detail-page fields, or max_reviews/"
|
||||
"max_images to attach reviews and photos per place."
|
||||
),
|
||||
input_schema=ScrapeInput,
|
||||
output_schema=ScrapeOutput,
|
||||
executor=build_scrape_executor(),
|
||||
billing_unit=None,
|
||||
)
|
||||
|
||||
register_capability(GOOGLE_MAPS_SCRAPE)
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
"""``google_maps.scrape`` executor: verb input → scraper → place items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.google_maps.scrape.schemas import ScrapeInput, ScrapeOutput
|
||||
from app.exceptions import ForbiddenError
|
||||
from app.proprietary.platforms.google_maps import (
|
||||
GoogleMapsScrapeInput,
|
||||
scrape_places,
|
||||
)
|
||||
from app.proprietary.platforms.google_maps.scraper import SignInRequiredError
|
||||
|
||||
ScrapeFn = Callable[[GoogleMapsScrapeInput], Awaitable[list[dict]]]
|
||||
|
||||
|
||||
def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor:
|
||||
"""Bind the executor to a scraper fn (defaults to the proprietary actor)."""
|
||||
scrape_fn = scrape_fn or scrape_places
|
||||
|
||||
async def execute(payload: ScrapeInput) -> ScrapeOutput:
|
||||
actor_input = GoogleMapsScrapeInput(
|
||||
searchStringsArray=payload.search_queries,
|
||||
startUrls=[{"url": url} for url in payload.urls],
|
||||
placeIds=payload.place_ids,
|
||||
locationQuery=payload.location,
|
||||
maxCrawledPlacesPerSearch=payload.max_places,
|
||||
language=payload.language,
|
||||
scrapePlaceDetailPage=payload.include_details,
|
||||
maxReviews=payload.max_reviews,
|
||||
maxImages=payload.max_images,
|
||||
)
|
||||
try:
|
||||
items = await scrape_fn(actor_input)
|
||||
except SignInRequiredError as exc:
|
||||
raise ForbiddenError(
|
||||
f"Google sign in required: {exc}", code="GOOGLE_SIGNIN_REQUIRED"
|
||||
) from exc
|
||||
return ScrapeOutput(items=items)
|
||||
|
||||
return execute
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
"""``google_maps.scrape`` I/O contracts.
|
||||
|
||||
A lean, agent-friendly surface over ``GoogleMapsScrapeInput``
|
||||
(``app/proprietary/platforms/google_maps``). The executor maps this to the full
|
||||
scraper input; the scraper's ``PlaceItem`` is reused verbatim as the output
|
||||
element.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.proprietary.platforms.google_maps import PlaceItem
|
||||
|
||||
MAX_MAPS_SOURCES = 20
|
||||
"""Per-call cap on queries + urls + place_ids: bounds a sync request's fan-out."""
|
||||
|
||||
|
||||
class ScrapeInput(BaseModel):
|
||||
search_queries: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_MAPS_SOURCES,
|
||||
description=(
|
||||
"Google Maps search terms (e.g. 'coffee shops', 'dentist'); each "
|
||||
"returns up to max_places. Provide these OR urls OR place_ids "
|
||||
"(at least one is required). Pair with location to scope a search."
|
||||
),
|
||||
)
|
||||
urls: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_MAPS_SOURCES,
|
||||
description=(
|
||||
"Google Maps URLs — a place page (/maps/place/...) or a search "
|
||||
"results URL. Provide these OR search_queries OR place_ids."
|
||||
),
|
||||
)
|
||||
place_ids: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_MAPS_SOURCES,
|
||||
description=(
|
||||
"Known Google place IDs (ChIJ...) to fetch directly. Provide these "
|
||||
"OR search_queries OR urls."
|
||||
),
|
||||
)
|
||||
location: str | None = Field(
|
||||
default=None,
|
||||
description="Location to scope search_queries to, e.g. 'New York, USA'.",
|
||||
)
|
||||
max_places: int = Field(
|
||||
default=10,
|
||||
ge=1,
|
||||
le=1000,
|
||||
description="Max places to return per search query.",
|
||||
)
|
||||
language: str = Field(
|
||||
default="en",
|
||||
description="Result language code, e.g. 'en', 'fr'.",
|
||||
)
|
||||
include_details: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Also fetch each place's detail page — opening hours, popular "
|
||||
"times, extra contact info (slower; more requests)."
|
||||
),
|
||||
)
|
||||
max_reviews: int = Field(
|
||||
default=0,
|
||||
ge=0,
|
||||
le=100_000,
|
||||
description="Reviews to attach per place (0 = none).",
|
||||
)
|
||||
max_images: int = Field(
|
||||
default=0,
|
||||
ge=0,
|
||||
description="Images to attach per place (0 = none).",
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_a_source(self) -> ScrapeInput:
|
||||
if not (self.search_queries or self.urls or self.place_ids):
|
||||
raise ValueError(
|
||||
"Provide at least one of 'search_queries', 'urls', or 'place_ids'."
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class ScrapeOutput(BaseModel):
|
||||
items: list[PlaceItem] = Field(
|
||||
default_factory=list,
|
||||
description="One place item per result, in the scraper's emission order.",
|
||||
)
|
||||
5
surfsense_backend/app/capabilities/web/__init__.py
Normal file
5
surfsense_backend/app/capabilities/web/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""``web.*`` namespace: the unified web content crawler verb."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.web.crawl import definition as _crawl # noqa: F401
|
||||
3
surfsense_backend/app/capabilities/web/crawl/__init__.py
Normal file
3
surfsense_backend/app/capabilities/web/crawl/__init__.py
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
"""``web.crawl`` verb: single-URL scrape or same-site spider → one row per page."""
|
||||
|
||||
from __future__ import annotations
|
||||
25
surfsense_backend/app/capabilities/web/crawl/definition.py
Normal file
25
surfsense_backend/app/capabilities/web/crawl/definition.py
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
"""``web.crawl`` capability registration."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import BillingUnit, Capability, register_capability
|
||||
from app.capabilities.web.crawl.executor import build_crawl_executor
|
||||
from app.capabilities.web.crawl.schemas import CrawlInput, CrawlOutput
|
||||
|
||||
WEB_CRAWL = Capability(
|
||||
name="web.crawl",
|
||||
description=(
|
||||
"Scrape a single web page or crawl a whole website. Give it one or more "
|
||||
"startUrls. Set maxCrawlDepth=0 to fetch just those URLs, or higher to "
|
||||
"also follow the links on each page (depth 1 = the start pages plus the "
|
||||
"pages they link to, and so on) — staying on the same site and stopping "
|
||||
"at maxCrawlPages. Returns one item per fetched page with clean markdown "
|
||||
"content, metadata (title, description), and crawl provenance."
|
||||
),
|
||||
input_schema=CrawlInput,
|
||||
output_schema=CrawlOutput,
|
||||
executor=build_crawl_executor(),
|
||||
billing_unit=BillingUnit.WEB_CRAWL,
|
||||
)
|
||||
|
||||
register_capability(WEB_CRAWL)
|
||||
65
surfsense_backend/app/capabilities/web/crawl/executor.py
Normal file
65
surfsense_backend/app/capabilities/web/crawl/executor.py
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
"""``web.crawl`` executor: seeds → site spider → one cleaned item per fetched page.
|
||||
|
||||
Boundary owned elsewhere: the crawl frontier/fetch live in the proprietary engine
|
||||
(``app.proprietary.web_crawler``). This executor only maps the engine's
|
||||
``CrawlPage`` list onto the typed ``CrawlOutput`` (status labels, truncation, and
|
||||
the captcha telemetry the billing seam reads).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.web.crawl.schemas import (
|
||||
CrawlInput,
|
||||
CrawlItem,
|
||||
CrawlMeta,
|
||||
CrawlOutput,
|
||||
)
|
||||
from app.proprietary.web_crawler import (
|
||||
CrawlOutcomeStatus,
|
||||
CrawlPage,
|
||||
WebCrawlerConnector,
|
||||
crawl_site,
|
||||
)
|
||||
|
||||
_STATUS_LABEL: dict[CrawlOutcomeStatus, str] = {
|
||||
CrawlOutcomeStatus.SUCCESS: "success",
|
||||
CrawlOutcomeStatus.EMPTY: "empty",
|
||||
CrawlOutcomeStatus.FAILED: "failed",
|
||||
}
|
||||
|
||||
|
||||
def build_crawl_executor(engine: WebCrawlerConnector | None = None) -> Executor:
|
||||
"""Build the ``web.crawl`` executor, optionally over an injected engine (tests)."""
|
||||
crawler = engine or WebCrawlerConnector()
|
||||
|
||||
async def execute(payload: CrawlInput) -> CrawlOutput:
|
||||
pages = await crawl_site(
|
||||
crawler,
|
||||
payload.startUrls,
|
||||
max_crawl_depth=payload.maxCrawlDepth,
|
||||
max_crawl_pages=payload.maxCrawlPages,
|
||||
)
|
||||
return CrawlOutput(
|
||||
items=[_to_item(page, payload.maxLength) for page in pages],
|
||||
captcha_attempts=sum(page.captcha_attempts for page in pages),
|
||||
captcha_solved=sum(1 for page in pages if page.captcha_solved),
|
||||
)
|
||||
|
||||
return execute
|
||||
|
||||
|
||||
def _to_item(page: CrawlPage, max_length: int) -> CrawlItem:
|
||||
content = page.content[:max_length] if page.content is not None else None
|
||||
return CrawlItem(
|
||||
url=page.url,
|
||||
status=_STATUS_LABEL[page.status],
|
||||
crawl=CrawlMeta(
|
||||
loadedUrl=page.loaded_url or page.url,
|
||||
depth=page.depth,
|
||||
referrerUrl=page.referrer,
|
||||
),
|
||||
markdown=content,
|
||||
metadata=page.metadata,
|
||||
error=page.error,
|
||||
)
|
||||
114
surfsense_backend/app/capabilities/web/crawl/schemas.py
Normal file
114
surfsense_backend/app/capabilities/web/crawl/schemas.py
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
# ruff: noqa: N815 - public field names intentionally use camelCase
|
||||
"""``web.crawl`` I/O contracts.
|
||||
|
||||
A Website Content Crawler-style surface: one verb that either scrapes the given
|
||||
URLs (``maxCrawlDepth == 0``) or spiders their site (``maxCrawlDepth > 0``),
|
||||
bounded by ``maxCrawlPages`` and kept on the seed's site.
|
||||
|
||||
Fields are trimmed to what the proprietary engine honors today. Knobs the engine
|
||||
handles automatically (crawler type, proxy, dynamic-render waits) are
|
||||
intentionally omitted, as are features we haven't built (URL globs, output
|
||||
formats, click actions, PII handling).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
MAX_START_URLS = 20
|
||||
"""Per-call cap on seed URLs: bounds a synchronous request's fan-out (05)."""
|
||||
|
||||
MAX_CRAWL_DEPTH = 5
|
||||
"""Deepest link distance a spider will follow from a start URL."""
|
||||
|
||||
MAX_CRAWL_PAGES = 200
|
||||
"""Hard ceiling on pages fetched per call (protects the wallet and the run)."""
|
||||
|
||||
|
||||
class CrawlInput(BaseModel):
|
||||
startUrls: list[str] = Field(
|
||||
min_length=1,
|
||||
max_length=MAX_START_URLS,
|
||||
description=(
|
||||
"Seed URLs to crawl. With maxCrawlDepth=0 only these are fetched; "
|
||||
"with a higher depth they are also the entry points for the spider."
|
||||
),
|
||||
)
|
||||
maxCrawlDepth: int = Field(
|
||||
default=0,
|
||||
ge=0,
|
||||
le=MAX_CRAWL_DEPTH,
|
||||
description=(
|
||||
"How many link-hops to follow from each start URL. 0 = scrape only "
|
||||
"the start URLs (no spidering); 1 = also their linked pages; etc. "
|
||||
"The spider stays on the start URL's site."
|
||||
),
|
||||
)
|
||||
maxCrawlPages: int = Field(
|
||||
default=10,
|
||||
ge=1,
|
||||
le=MAX_CRAWL_PAGES,
|
||||
description=(
|
||||
"Maximum number of pages to fetch in total (start URLs included). "
|
||||
"The crawl stops once this many pages have been fetched."
|
||||
),
|
||||
)
|
||||
maxLength: int = Field(
|
||||
default=50_000,
|
||||
ge=1,
|
||||
description="Maximum characters of cleaned markdown kept per page (truncates beyond).",
|
||||
)
|
||||
|
||||
@property
|
||||
def estimated_units(self) -> int:
|
||||
"""Worst-case billable pages for the pre-flight gate (03c)."""
|
||||
if self.maxCrawlDepth == 0:
|
||||
return len(self.startUrls)
|
||||
return self.maxCrawlPages
|
||||
|
||||
|
||||
class CrawlMeta(BaseModel):
|
||||
loadedUrl: str = Field(description="The URL actually fetched for this page.")
|
||||
depth: int = Field(
|
||||
description="Link distance from a start URL (0 for a start URL itself)."
|
||||
)
|
||||
referrerUrl: str | None = Field(
|
||||
default=None,
|
||||
description="The page this URL was discovered on (null for start URLs).",
|
||||
)
|
||||
|
||||
|
||||
class CrawlItem(BaseModel):
|
||||
url: str = Field(description="The requested URL for this page.")
|
||||
status: Literal["success", "empty", "failed"] = Field(
|
||||
description="success = content returned; empty = fetched but no content; failed = could not fetch."
|
||||
)
|
||||
crawl: CrawlMeta | None = Field(
|
||||
default=None, description="Crawl provenance (loaded URL, depth, referrer)."
|
||||
)
|
||||
markdown: str | None = Field(
|
||||
default=None, description="Cleaned page content as markdown (null unless success)."
|
||||
)
|
||||
metadata: dict[str, str] | None = Field(
|
||||
default=None, description="Page metadata such as title and description."
|
||||
)
|
||||
error: str | None = Field(
|
||||
default=None, description="Failure reason when status is not success."
|
||||
)
|
||||
|
||||
|
||||
class CrawlOutput(BaseModel):
|
||||
items: list[CrawlItem] = Field(
|
||||
default_factory=list,
|
||||
description="One item per fetched page, in crawl (BFS) order.",
|
||||
)
|
||||
# Billing-only telemetry; excluded from the wire shape (mirrors web.scrape).
|
||||
captcha_attempts: int = Field(default=0, exclude=True)
|
||||
captcha_solved: int = Field(default=0, exclude=True)
|
||||
|
||||
@property
|
||||
def billable_units(self) -> int:
|
||||
"""Successful pages are the metered unit (03c)."""
|
||||
return sum(1 for item in self.items if item.status == "success")
|
||||
6
surfsense_backend/app/capabilities/youtube/__init__.py
Normal file
6
surfsense_backend/app/capabilities/youtube/__init__.py
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
"""``youtube.*`` namespace: platform-native YouTube data verbs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.youtube.comments import definition as _comments # noqa: F401
|
||||
from app.capabilities.youtube.scrape import definition as _scrape # noqa: F401
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
"""``youtube.comments`` verb: video URLs → comment items (+ replies)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
"""``youtube.comments`` capability registration (free — see 04-capabilities open item)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import Capability, register_capability
|
||||
from app.capabilities.youtube.comments.executor import build_comments_executor
|
||||
from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput
|
||||
|
||||
YOUTUBE_COMMENTS = Capability(
|
||||
name="youtube.comments",
|
||||
description=(
|
||||
"Fetch public comments (and their replies) for one or more YouTube "
|
||||
"videos. Give it the video URLs; returns structured comment items with "
|
||||
"author, text, like count, reply relationships, and timestamps. Use it "
|
||||
"to gauge sentiment or pull discussion on specific videos."
|
||||
),
|
||||
input_schema=CommentsInput,
|
||||
output_schema=CommentsOutput,
|
||||
executor=build_comments_executor(),
|
||||
billing_unit=None,
|
||||
)
|
||||
|
||||
register_capability(YOUTUBE_COMMENTS)
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
"""``youtube.comments`` executor: verb input → scraper → comment items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.youtube.comments.schemas import CommentsInput, CommentsOutput
|
||||
from app.proprietary.platforms.youtube import (
|
||||
YouTubeCommentsInput,
|
||||
scrape_comments,
|
||||
)
|
||||
|
||||
CommentsFn = Callable[[YouTubeCommentsInput], Awaitable[list[dict]]]
|
||||
|
||||
|
||||
def build_comments_executor(scrape_fn: CommentsFn | None = None) -> Executor:
|
||||
"""Bind the executor to a comments scraper fn (defaults to the proprietary actor)."""
|
||||
scrape_fn = scrape_fn or scrape_comments
|
||||
|
||||
async def execute(payload: CommentsInput) -> CommentsOutput:
|
||||
actor_input = YouTubeCommentsInput(
|
||||
startUrls=[{"url": url} for url in payload.urls],
|
||||
maxComments=payload.max_comments,
|
||||
sortCommentsBy=payload.sort_by,
|
||||
)
|
||||
items = await scrape_fn(actor_input)
|
||||
return CommentsOutput(items=items)
|
||||
|
||||
return execute
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
"""``youtube.comments`` I/O contracts.
|
||||
|
||||
A lean surface over ``YouTubeCommentsInput``; the scraper's ``CommentItem`` is
|
||||
reused verbatim as the output element.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.proprietary.platforms.youtube import CommentItem
|
||||
|
||||
MAX_COMMENT_VIDEOS = 20
|
||||
"""Per-call cap on how many video URLs one request may harvest comments from."""
|
||||
|
||||
|
||||
class CommentsInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
min_length=1,
|
||||
max_length=MAX_COMMENT_VIDEOS,
|
||||
description="YouTube video URLs to fetch comments (and replies) for (1-20).",
|
||||
)
|
||||
max_comments: int = Field(
|
||||
default=20,
|
||||
ge=1,
|
||||
le=100_000,
|
||||
description=(
|
||||
"Max items returned per video, counting both top-level comments and "
|
||||
"their replies."
|
||||
),
|
||||
)
|
||||
sort_by: Literal["TOP_COMMENTS", "NEWEST_FIRST"] = Field(
|
||||
default="NEWEST_FIRST",
|
||||
description="Comment ordering: most-liked first, or most-recent first.",
|
||||
)
|
||||
|
||||
|
||||
class CommentsOutput(BaseModel):
|
||||
items: list[CommentItem] = Field(
|
||||
default_factory=list,
|
||||
description="One item per comment or reply, in the scraper's emission order.",
|
||||
)
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
"""``youtube.scrape`` verb: YouTube URLs / search queries → video items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
"""``youtube.scrape`` capability registration (free — see 04-capabilities open item)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.capabilities.core import Capability, register_capability
|
||||
from app.capabilities.youtube.scrape.executor import build_scrape_executor
|
||||
from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput
|
||||
|
||||
YOUTUBE_SCRAPE = Capability(
|
||||
name="youtube.scrape",
|
||||
description=(
|
||||
"Scrape public YouTube data. Give it YouTube URLs (video, channel, "
|
||||
"playlist, shorts, or hashtag) and/or search queries, and it returns "
|
||||
"structured video items — title, views, likes, publish date, channel "
|
||||
"info, description, and optionally subtitles. Use search_queries to "
|
||||
"discover videos, or urls to pull a known video/channel/playlist."
|
||||
),
|
||||
input_schema=ScrapeInput,
|
||||
output_schema=ScrapeOutput,
|
||||
executor=build_scrape_executor(),
|
||||
billing_unit=None,
|
||||
)
|
||||
|
||||
register_capability(YOUTUBE_SCRAPE)
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
"""``youtube.scrape`` executor: verb input → scraper → video items."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from app.capabilities.core import Executor
|
||||
from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput
|
||||
from app.proprietary.platforms.youtube import (
|
||||
YouTubeScrapeInput,
|
||||
scrape_youtube,
|
||||
)
|
||||
|
||||
ScrapeFn = Callable[[YouTubeScrapeInput], Awaitable[list[dict]]]
|
||||
|
||||
|
||||
def build_scrape_executor(scrape_fn: ScrapeFn | None = None) -> Executor:
|
||||
"""Bind the executor to a scraper fn (defaults to the proprietary actor)."""
|
||||
scrape_fn = scrape_fn or scrape_youtube
|
||||
|
||||
async def execute(payload: ScrapeInput) -> ScrapeOutput:
|
||||
# Channels emit three content types; cap each at the caller's max_results
|
||||
# so a channel scrape isn't silently limited to plain videos only.
|
||||
actor_input = YouTubeScrapeInput(
|
||||
startUrls=[{"url": url} for url in payload.urls],
|
||||
searchQueries=payload.search_queries,
|
||||
maxResults=payload.max_results,
|
||||
maxResultsShorts=payload.max_results,
|
||||
maxResultStreams=payload.max_results,
|
||||
downloadSubtitles=payload.download_subtitles,
|
||||
subtitlesLanguage=payload.subtitles_language,
|
||||
)
|
||||
items = await scrape_fn(actor_input)
|
||||
return ScrapeOutput(items=items)
|
||||
|
||||
return execute
|
||||
66
surfsense_backend/app/capabilities/youtube/scrape/schemas.py
Normal file
66
surfsense_backend/app/capabilities/youtube/scrape/schemas.py
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
"""``youtube.scrape`` I/O contracts.
|
||||
|
||||
A lean, agent-friendly surface over ``YouTubeScrapeInput``
|
||||
(``app/proprietary/platforms/youtube``). The executor maps this to the full
|
||||
scraper input; the scraper's ``VideoItem`` is reused verbatim as the output
|
||||
element.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
|
||||
from app.proprietary.platforms.youtube import VideoItem
|
||||
|
||||
MAX_YOUTUBE_SOURCES = 20
|
||||
"""Per-call cap on URLs + queries: bounds a synchronous request's fan-out (05)."""
|
||||
|
||||
|
||||
class ScrapeInput(BaseModel):
|
||||
urls: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_YOUTUBE_SOURCES,
|
||||
description=(
|
||||
"YouTube URLs to scrape: video, channel (/@handle or /channel/UC...), "
|
||||
"playlist (?list=...), shorts, or hashtag pages. Provide these OR "
|
||||
"search_queries (at least one is required)."
|
||||
),
|
||||
)
|
||||
search_queries: list[str] = Field(
|
||||
default_factory=list,
|
||||
max_length=MAX_YOUTUBE_SOURCES,
|
||||
description=(
|
||||
"Search terms to run on YouTube; each returns up to max_results videos. "
|
||||
"Provide these OR urls (at least one is required)."
|
||||
),
|
||||
)
|
||||
max_results: int = Field(
|
||||
default=10,
|
||||
ge=1,
|
||||
le=1000,
|
||||
description=(
|
||||
"Max items to return per source and per content type (videos, shorts, "
|
||||
"streams are capped independently for a channel)."
|
||||
),
|
||||
)
|
||||
download_subtitles: bool = Field(
|
||||
default=False,
|
||||
description="Also fetch each video's subtitle track (slower; more requests).",
|
||||
)
|
||||
subtitles_language: str = Field(
|
||||
default="en",
|
||||
description="Subtitle language code (e.g. 'en', 'fr'). Used when download_subtitles is true.",
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _require_a_source(self) -> ScrapeInput:
|
||||
if not self.urls and not self.search_queries:
|
||||
raise ValueError("Provide at least one of 'urls' or 'search_queries'.")
|
||||
return self
|
||||
|
||||
|
||||
class ScrapeOutput(BaseModel):
|
||||
items: list[VideoItem] = Field(
|
||||
default_factory=list,
|
||||
description="One video item per result, in the scraper's emission order.",
|
||||
)
|
||||
|
|
@ -244,7 +244,6 @@ celery_app.conf.update(
|
|||
"index_google_gmail_messages": {"queue": CONNECTORS_QUEUE},
|
||||
"index_google_drive_files": {"queue": CONNECTORS_QUEUE},
|
||||
"index_elasticsearch_documents": {"queue": CONNECTORS_QUEUE},
|
||||
"index_crawled_urls": {"queue": CONNECTORS_QUEUE},
|
||||
"index_bookstack_pages": {"queue": CONNECTORS_QUEUE},
|
||||
"index_composio_connector": {"queue": CONNECTORS_QUEUE},
|
||||
"index_obsidian_attachment": {"queue": CONNECTORS_QUEUE},
|
||||
|
|
|
|||
|
|
@ -554,8 +554,12 @@ class Config:
|
|||
os.getenv("SURFSENSE_CONNECTOR_DISCOVERY_TTL_SECONDS", "30")
|
||||
)
|
||||
|
||||
# Platform web search (SearXNG)
|
||||
# Platform web search providers for the web.discover capability (env-keyed).
|
||||
SEARXNG_DEFAULT_HOST = os.getenv("SEARXNG_DEFAULT_HOST")
|
||||
LINKUP_API_KEY = os.getenv("LINKUP_API_KEY")
|
||||
BAIDU_API_KEY = os.getenv("BAIDU_API_KEY")
|
||||
BAIDU_MODEL = os.getenv("BAIDU_MODEL", "ernie-3.5-8k")
|
||||
BAIDU_SEARCH_SOURCE = os.getenv("BAIDU_SEARCH_SOURCE", "baidu_search_v2")
|
||||
|
||||
SURFSENSE_PUBLIC_URL = os.getenv("SURFSENSE_PUBLIC_URL")
|
||||
NEXT_FRONTEND_URL = os.getenv("NEXT_FRONTEND_URL") or SURFSENSE_PUBLIC_URL
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
"""Platform-specific crawl/extraction actors (non-Apache-2; see ../LICENSE).
|
||||
"""Platform-native crawl/extraction actors (non-Apache-2; see ../LICENSE).
|
||||
|
||||
Scaffolded for future work (Phase 8 — Platform Actors). Each platform (e.g.
|
||||
maps, professional networks) will get its own subpackage that reuses the shared
|
||||
fetch strategies in ``app.proprietary.web_crawler`` under a platform-specific
|
||||
structured extractor. Empty for now by design.
|
||||
One subpackage per platform (e.g. ``youtube``), each a structured extractor for
|
||||
that platform's data. Actors may reuse the shared fetch strategies in
|
||||
``app.proprietary.web_crawler`` but expose their own platform-specific I/O.
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ No API keys, no Apify account, no headless browser on the happy path.
|
|||
## Quick start
|
||||
|
||||
```python
|
||||
from app.proprietary.scrapers.youtube import (
|
||||
from app.proprietary.platforms.youtube import (
|
||||
YouTubeScrapeInput, scrape_youtube,
|
||||
YouTubeCommentsInput, scrape_comments,
|
||||
)
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue