mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-06 22:12:12 +02:00

CREDO23 0944d9bb78 docs(plans): re-cast Phase 4-7 to stateless scraper-APIs, flatten subplans

2026-07-01 14:51:29 +02:00

54 KiB

Raw Blame History

CI Pivot MVP — Umbrella Plan

Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (plans/backend/).

This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as four revamped phases — Phase 4 Capabilities (04-capabilities.md), Phase 5 Access (05-access.md), Phase 6 Ongoing-Automation (06-ongoing-automation.md, design deferred), Phase 7 Orchestration (07-orchestration.md). NOTE: the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful Tracker + Timeline + Triggers design was superseded on 2026-07-01 (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below.

SCOPE: This umbrella currently covers the BACKEND only (surfsense_backend). Frontend (surfsense_web) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases.

Positioning

Premium, open-source, self-hostable scraper APIs (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return cleaned, AI-ready data for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time.

⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native

This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6). The canonical Phase 4–7 subplans (04–07, below) are re-cast to four phases:

The product = scraper capabilities, not a CI engine. web.* + maps.* typed verbs return cleaned, AI-ready data, exposed identically through REST + API key, an MCP server, and chat, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable).

Stateless. No Timeline (3-store delta), no Tracker, no diff/materiality engine, no stored entity_changes. Memory = the chat history; the agent reasons over prior tool outputs in context to report "what changed."

Direct calls. Capabilities return their result directly — no job store, no completed | pending + job_id envelope, no deliverable_wait polling. Async results reach the client via write-then-sync / stream (existing SSE or a Zero-published table).

Automation = a persistent ongoing chat that periodically re-invokes verbs. This replaces the Triggers subsystem; the periodic mechanism is design-deferred (Phase 6).

KB stays input-only; crawled data is never indexed.

Phase mapping: 04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration (intelligence_agent). The subplans 05a-timeline/05b-intelligence/06-triggers are removed (history in git); 04a/04b are folded into 04/05.

⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted

Historical record. The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the 2026-07-01 pivot above.

This supersedes the original Pipelines paradigm (Phases 5–7). During Phase 4 discussion we concluded that the "sync data into the KB, then operate over it" model is wrong for competitive intelligence. Verified against references/opencode — a coding agent that pulls context live via read/grep/webfetch/websearch tools (packages/opencode/src/tool/) and persists sessions, never a scraped corpus; its websearch tool even live-crawls at query time (tool/websearch.ts). The corrected model:

Knowledge Base = input-only — the user's personal files/context that enriches the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). Nothing scrapes into it. Its job is to be searchable user-provided context, not a dump of fetched pages.

Web search + the WebURL crawler = platform-native agent tools (already web_search + scrape_webpage on the main agent, main_agent/tools/index.py), always available; later exposed to developers via a platform API key. They are NOT connectors and NOT pipelines.

Connectors = MCP tools (Type-2) + file/KB-input connectors (the only Type-1 that survives). All branded natives → MCP.

Automations are the scheduling + run-history substrate. An automation run already invokes the full chat agent (with the crawler/search tools) on a cron/event trigger and persists run history (automation_runs) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's insight per run, not raw snapshots.

WEBCRAWLER_CONNECTOR is retired — the crawler is a native tool, not a connector/data-source.

Net effect on this umbrella: the old Pipelines stack (old Phases 5–7) is dropped and Phases 4–7 are re-cast by the canonical subplans 04–07 (this framing was itself superseded by the 2026-07-01 pivot above):

Phase 4 — Capabilities & Access (revamp/04a + 04b): turn the crawler + search into typed, callable verbs (MVP = web.scrape + web.discover; per-platform scrapers like maps.*/linkedin.* are later, uncommitted drop-ins) over a single capability registry, exposed identically through chat / REST+API-key / MCP doors. Ships Product A (stateless utility) — revenue day one. The old 04a-connector-category is demoted to backward-compat hygiene; the old 04b-source-discovery is absorbed as the web.discover verb.

Phase 5 — Intelligence & Timeline (revamp/05a + 05b): the Tracker primitive + a 3-store delta Timeline (tracked_entities/entity_current_state/entity_changes, "store deltas, no change → no row"). This is the durable CI state and the moat — it replaces "pipelines" as the standing concern. Ships the Product B engine.

Phase 6 — Triggers (revamp/06): decide when a Tracker refreshes. refresh(tracker) is trigger-agnostic; recurrence is a CI action on the existing automations (reuse its schedule selector + AutomationRun — no new scheduler, no new run table). Alert delivery is separate (via app/notifications/; automations have no delivery path).

Phase 7 — Orchestration (revamp/07): a net-new intelligence_agent subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language.

KB stays input-only; WEBCRAWLER_CONNECTOR retired; crawler/search are native tools (later a platform API key). The three correction decisions land in their real homes: bill crawls at the capability executor (revamp/04a, works uniformly across chat/automation/REST/MCP/cron — see 03c below), diffable run history = the Timeline delta store (revamp/05a, not automation_runs.output), and "run now" = a Trigger adapter (revamp/06). Phase 8 platform scrapers re-slot as a family of individual native endpoints — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (none, incl. Google Maps, committed for MVP; they're examples of the pattern), not pipeline executors.

Target architecture (scraper-APIs-first, stateless)

Full end-to-end flow diagrams live in 00b-pipeline-diagrams.md. Summary below.

flowchart TD
  ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]]
  REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"]
  ACQ --> REG
  REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"]
  DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · 'what changed' from chat history"]
  KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT
  CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT
  AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"]
  AGENT -.->|standing need| ONG["Ongoing-Automation (P6, DEFERRED): persistent chat re-invokes verbs; memory = chat history"]
  ONG -.-> REG
  APIKEY["Platform API key"] --> DOORS

Original (superseded) Pipelines-centric architecture

flowchart TD
  WS[WorkSpace] --> CONN[Connectors]
  WS --> PIPE[Pipelines]
  WS --> KB[(Knowledge Base: documents + chunks)]
  CONN --> T1[Type 1: Data Sources - pull]
  CONN --> T2[Type 2: MCP Tools - act]
  T1 --> WEB[Universal WebURL Crawler - functional]
  T1 --> PLAT[Platform connectors - coming soon]
  T1 --> UP[File Upload]
  PIPE --> RUN[PipelineRun history]
  PIPE -->|"save_to_kb + destination folder"| KB
  RUN -->|"manual or cron"| T1
  T2 --> CHAT[Chat / Automations]
  CHAT --> DELIV[Deliverables: audio/video/report/image]
  RUN -->|"read-only context"| CHAT

Decisions locked

Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps.
Canonical names (proposed defaults): DB table workspaces, column workspace_id, RBAC tables workspace_roles / workspace_memberships / workspace_invites, API base /workspaces (consolidating today's /searchspaces vs /search-spaces split), URL segment [workspace_id], settings folder workspace-settings, TS type Workspace.
Capabilities, not connectors, are the core of the product. The crawler + search + Maps actor become typed verbs (web.scrape, web.discover, maps.search/place/reviews) in a single capability registry that generates the REST/MCP/chat doors identically (04/05). A capability is call → cleaned data → bill; no SearchSourceConnector row, no KB write, no schedule. The old connector two-type taxonomy is demoted to backward-compat hygiene (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the MCP_CONNECTOR (BYO MCP) routing-gap fix, moves into 05 (consume-user-MCP). Surviving connectors: file/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + BYO MCP_CONNECTOR (act in chat). WEBCRAWLER_CONNECTOR RETIRED (native tool); branded natives stay MIGRATING (off); Obsidian/Circleback DISABLED. Artifacts stay in the existing deliverables system.
Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the web.discover capability (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to web.scrape. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
Stateless — the chat history is the memory. No Timeline, no Tracker, no diff/materiality engine, no stored entity_changes. "What changed" is the agent reasoning over prior tool outputs already in context (07). Standing needs are served by a persistent ongoing chat that periodically re-invokes verbs (06, design deferred).
Bill at the capability executor. Each verb declares a billing_unit; the executor charges the workspace owner once per billable success via the billing service (WebCrawlCreditService, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter uniformly. maps.* registers a new per-place / per-page unit.
Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP.
MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are DISABLED — they're tagged MIGRATING (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point).
~~Pipeline and PipelineRun are new first-class tables.~~ SUPERSEDED — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a persistent ongoing chat re-invoking verbs (06, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper).
~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by reasoning over the chat history (prior tool outputs already in context, 07), not by re-deriving from run logs or a stored timeline.
Web search + WebURL crawler are platform-native (agent tools now; developer platform-API-key access later). The KB is input-only.
Deferred (post-MVP): platform scraper implementations (Phase 8 — Platform actors, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as native tools / API endpoints, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that.

Platform connector research list (deferred build, MVP = "coming soon")

LinkedIn — people profiles (discovery by keyword/company), company info, job listings.
Amazon — product (ASIN), search (keyword), pricing; reviews secondary.
Google — Web Search (organic SERP), AI Overviews, Maps/Local (discover by location).
Instagram — profiles first, then posts; discover profiles by username/keyword.
Zillow / Redfin — full property listings (discover by search URL/filters); Zillow price history.
Walmart — product, search; zipcode-localized pricing premium variant.
eBay — search by keyword/category; price-comparison/resale feeds.
Crunchbase — company info, search by keyword (B2B lead-gen / investor research).
TikTok / YouTube — profiles/channels, posts/videos; discover by keyword/hashtag; TikTok Shop.
Indeed / Glassdoor — job listings (discover by keyword in location), company reviews.

Backend phases (active — this umbrella)

Phase 1 — Rename foundation (DB) [`subplan: 01-rename-db.md`]

✅ SHIPPED (2026-06-27) · branch feat/rename-searchspace-to-workspace · PR #1546 (merged to ci_mvp via #1562). Migration 170 (chains 169, current head) does the physical rename + Zero-publication reconcile. Shipped atomically with Phase 2. As-built record + re-runnable verification live in 01-rename-db.md. Deploy caveat: zero-cache replica reset (ZERO_AUTO_RESET) required; from-scratch alembic upgrade head stays pre-existing-broken (rev 23 conflict — separate baseline-squash task), only the 169→170 path is verified.

Alembic migration: rename searchspaces -> workspaces; rename search_space_id -> workspace_id on ~20 child tables; rename RBAC tables and their FKs; rename indexes/constraints (uq_searchspace_*, idx_documents_search_space_id, etc.); update Rocicorp Zero publication column lists (backend-owned publication definition; frontend Zero schema rename happens in the later frontend umbrella).
Decide transition strategy: hard cutover (simplest for MVP) vs temporary API aliases for clients.
Key files: surfsense_backend/app/db.py, surfsense_backend/alembic/versions/ (new migration).

Phase 2 — Rename backend (code + API) [`subplan: 02-rename-backend.md`]

✅ SHIPPED (2026-06-27) · branch feat/rename-searchspace-to-workspace · PR #1546 (merged to ci_mvp via #1562). Symbolic rename across app/ + tests/ (Phase-1 shim dropped), API consolidated to /workspaces (legacy /searchspaces · /search-spaces · /search-space all retired/404). Verified (ground-truth git grep, 2026-06-29): every residual search_space/SEARCH_SPACE is a documented carve-out — enum values (ConnectionScope/ChatVisibility.SEARCH_SPACE), the 'SEARCH_SPACE' CHECK literal (now paired with workspace_id), Celery wire names (delete_search_space_background, ai_sort_search_space), OTel key search_space.id, and the SEARCH_SPACE_FORBIDDEN error code; alembic/versions/ untouched except 168+170. Suite: 3016 passed, 1 skipped. As-built record in 02-rename-backend.md. Clients are intentionally broken until the frontend/satellite umbrellas land (hard cutover).

Rename models/schemas/services/routes/agents/tasks identifiers: SearchSpace* -> Workspace*, search_space_id -> workspace_id.
Consolidate API to /workspaces and fix the /searchspaces vs /search-spaces inconsistency.
High-touch files: routes/search_spaces_routes.py, routes/rbac_routes.py, utils/rbac.py (check_search_space_access), schemas/search_space.py, plus search_space_id threading through agents/Redis keys/storage paths (documents/{id}/...).

Phase 3 — WebURL Crawler & Crawl Billing (backend) [`subplans: 03a–03f`]

The Universal WebURL Crawler is the flagship Type-1 data source (the moat). This phase hardens it into an in-house, best-effort "undetectable, captcha-bypassing" crawler on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds stealth hardening (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds opt-in captcha solving. All tiers plug in behind a single FetchStrategy seam returning CrawlOutcome, so callers never depend on how a page was fetched — that seam is what lets the moat grow (and lets a deferred paid-unblocker tier drop in later by config). Strategy decision (recorded in the log): CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. Logged-in/account-based bypass is out of scope (public data only this MVP). It is broken into focused subplans:

Sequencing within Phase 3 (critical path vs hardening). Only 03a + 03b + 03c are on the MVP critical path. The crawler (03a CrawlOutcome) + proxy provider (03b) + 03c WebCrawlCreditService billing seam are consumed by the capability verbs (04) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). 03d (captcha), 03e (stealth hardening), and 03f (test harness) are hardening/measurement that nothing downstream imports — they tune the same seam behind the same contract, so they can land in parallel with, or after, Phase 4 without blocking the pivot. Recommended build order: 03a → 03b → 03c (then 04 Capabilities → 05 Access → 07 Orchestration), with 03e → 03d → 03f slotted in whenever crawler robustness is prioritized.

03a-crawler-core.md (✅ IMPLEMENTED — ci_mvp @ 5c36cd3; crawler moved to app/proprietary/web_crawler/, impersonate="chrome" + solve_cloudflare=True shipped) — Standardize the fetch layer on Scrapling. Remove Firecrawl entirely (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran).
03b-proxy-expansion.md (✅ IMPLEMENTED — ci_mvp @ 6226012) — Add a BYO CustomProxyProvider (the only new provider — no branded vendors) alongside anonymous_proxies, selectable via a single, app-wide Config.PROXY_PROVIDER. Add bounded client-side rotation+retry via Scrapling's ProxyRotator/is_proxy_error only when the active provider is pool-backed (CUSTOM_PROXY_URLS); single-endpoint providers (incl. anonymous_proxies) stay the default and no-op the retry. No per-connector/per-crawl selection (one provider app-wide); a per-capability override is left as a no-op seam (04), not built.
03c-crawl-billing.md (✅ IMPLEMENTED — ci_mvp @ 17bdb0682) — Charge crawl credits at $1 / 1000 successful requests = 1000 micro-USD per successful crawl (config-driven via WEB_CRAWL_MICROS_PER_SUCCESS, retunable with no code change), drawn from the existing credit wallet (credit_micros_balance), gated by a new WEB_CRAWL_CREDIT_BILLING_ENABLED flag (off for self-hosted). Billing surfaces (reframed by the Architecture pivot + revamp — see log): the charge moves to the capability executor (04), so every caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the workspace owner via WebCrawlCreditService. The interactive chat turn accumulator becomes an optional presentation fold (so chat still shows the crawl line on the turn bill), not the charging mechanism. The original connector/pipeline-indexer billing branch (webcrawler_indexer) is now vestigial (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The WebCrawlCreditService (mirrors EtlCreditService's gate → check_credits → charge_credits) is unchanged. No DB migration (uses the existing free-form web_crawl usage_type).
03e-stealth-hardening.md — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. Geoip coherence (match browser locale/timezone_id to the proxy's exit geo), fingerprint flags (hide_canvas/block_webrtc), persistent per-domain profiles (user_data_dir), headed execution under Xvfb, real fonts in the worker image, and DIY behavioral humanization via page_action (the Chromium engine has no built-in humanize). Adds a block classifier (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + per-domain strategy memory (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does not build — the deferred paid-unblocker FetchStrategy (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, not top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier.
03d-captcha-solving.md (✅ IMPLEMENTED — ci_mvp; captchatools page_action + per-attempt web_crawl_captcha billing, off by default) — Covers the captcha types Scrapling does not (reCAPTCHA v2/v3, hCaptcha, image) via captchatools, opt-in + off by default. captchatools is itself the provider registry (new_harvester(solving_site=…) across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do not rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher page_action that detects the sitekey, harvests a token (egressing from the same proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (03a), and page_action runs after solve_cloudflare, so the tiers compose. The billing asymmetry is now resolved: a separate per-attempt web_crawl_captcha unit (solvers charge per attempt regardless of crawl success), attached via a WebCrawlCreditService seam in 03c. ErrNoBalance stops solving (no retry loop → avoids IP bans). Requires a paid solver account.
03f-undetectability-testing.md (✅ IMPLEMENTED + first baseline — ci_mvp, MANUAL-only, no CI gating; lives under app/proprietary/web_crawler/testbench/) — A manual scorecard harness that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's bin/cloaktest suite) to quantify the free-stack ceiling over time. Two labeled axes: Suite S (stealth/anti-bot) — browser-tier (bot.sannysoft, bot.incolumitas, CreepJS, deviceandbrowserinfo, FingerprintJS demo, reCAPTCHA-v3 score, fingerprint-scan/Castle.js, browserscan, cloudflare-challenge exercising solve_cloudflare, iphey geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (tls.peet.ws, informational not a gate), and proxy/leak checks (httpbin/ip, WebRTC/DNS); Suite E (extraction correctness) — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. Every detection site is auto-graded from its real DOM verdict (parsers written against captured dumps; INFO reserved for TLS/IP/manual rows). Reuses 03d's page_action+closure-cell mechanism. Adopts CloakBrowser's bars as aspirational (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording our actual numbers as the baseline (the whole results/ tree is gitignored, run-local). First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (hasHeadlessWorkerUA worker-UA leak → 03e Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy fpscanner WEBDRIVER only). The scorecard is the documented trigger for flipping 03e's deferred paid-unblocker tier.

Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`]

Canonical subplan: 04-capabilities.md. Build first — every later phase calls the registry.

Turn Acquisition (crawler + search + Maps actor) into a small set of typed, callable verbs over a single capability registry — web.scrape(urls[]), web.discover(query, top_k), maps.search/place/reviews. Each verb is a direct async call that returns cleaned, AI-ready data (no job store, no envelope, no polling). maps.* returns typed structured objects, not markdown. Each declares a billing_unit; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package app/capabilities/; Maps extractor stays proprietary.

Locked model (MVP):

Capabilities replace connectors as data sources. web.* / maps.* are call → cleaned data → bill verbs — no SearchSourceConnector row, no KB write, no schedule.
Surviving connectors = file/KB-input + BYO MCP. File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the input-only KB. BYO MCP_CONNECTOR (act in chat) is the only functional Type-2; its routing-gap fix lands in 05. WEBCRAWLER_CONNECTOR RETIRED. The old connector two-type taxonomy (04a-connector-category) is demoted to backward-compat hygiene.
Deprecated (MIGRATING, off for MVP): every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay searchable; new-create//index/periodic/subagents off. Real MCP migration is post-MVP.
Disabled (DISABLED): Obsidian (plugin push) and Circleback (meeting webhook).
Web search: drop all 5 search-API connector_type values (SERPER_API/TAVILY_API/SEARXNG_API/LINKUP_API/BAIDU_SEARCH_API); survivors (SearXNG/Linkup/Baidu) become the web.discover provider set keyed by platform env (Linkup/Baidu keys move off per-connector config). Destructive migration deletes the 5 types' rows. (Absorbs the old 04b-source-discovery.)
The Google Maps actor is net-new, proprietary, built as a separate effort; maps.* are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors.
Frontend connector/capability UI restructure is DEFERRED (frontend umbrella).

Phase 5 — Access (backend) [`subplan: 05-access.md`]

Canonical subplan: 05-access.md. Build after 04. Together 04 + 05 ship the scraper-API product — revenue day one.

Expose the registry through three generated doors — REST + API key (public day one), an MCP server (fast-follow), and chat tools — each the same thin adapter (parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data). The executor returns directly; there is no envelope, no deliverable_wait polling. Async results (e.g. the 06 ongoing mode) reach the client via write-then-sync / stream (existing SSE or a Zero-published table).
Natural language is the human surface (raw verbs only on REST/MCP); the chat door classifies one-shot vs keep-watching and hands keep-watching to 06. Two MCP directions kept distinct: serve our verbs as an MCP server (new door) vs consume the user's BYO MCP_CONNECTOR (the old-04a routing-gap fix lands here).

Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`]

Canonical subplan: 06-ongoing-automation.md — ⚠️ design deferred. The periodic mechanism is designed separately, after 04/05/07. Depends on 04 (the verbs it re-invokes) and 05 (the chat surface + delivery channel).

Support "keep watching" with no stateful storage: a persistent, ongoing chat where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls.

Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`]

Canonical subplan: 07-orchestration.md — the human-facing brain. Build last (atop 04/05, and 06 for the ongoing mode). We plug into the shipped multi-agent runtime; we don't rebuild it.

Ship the net-new builtin intelligence_agent subagent (peer to research/deliverables): intent routing (one-shot vs keep-watching, one clarifying question when ambiguous), verb composition (web.discover → web.scrape, maps.search → maps.place → maps.reviews), and "what changed" from the chat history (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store).
Toolset: registry-backed capability verbs (shared with research) + a deferred start_watch handoff to 06. Tools follow the shipped scrape_webpage shape (executor + door + 03c billing), direct-return (no deliverable_wait).

(Future) Platform API-key + public MCP

Platform API-key access to the capability verbs + a public MCP server exposing the input-only KB — post-MVP (see Phase 8 + Deferred).

Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`]

NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the committed MVP actor (a maps.* namespace, Phase 4). Beyond it, more platform scrapers — a family of individual scraping endpoints (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; examples, none committed) — layer on top of the hardened fetch core once it's solid. Each is simply another capability verb (04), so registering one automatically yields (a) an agent tool and (b) a dev-callable REST endpoint behind the platform API key — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the 03a crawler core (proxies + 03e hardening + 03d captcha). Billed per call via WebCrawlCreditService.

Public data only at first — discovery/extraction of publicly visible pages. Logged-in/account-based bypass is explicitly deferred beyond Phase 8's first cut; it needs sticky/static proxies + credential management (03b static-proxy hand-off) and is the higher-risk, later workstream.
The deferred paid-unblocker tier (03e §8) is the fallback for any platform whose anti-bot exceeds the in-house ceiling.

Deferred — Frontend & client phases (separate umbrella, planned LATER)

These are recorded for continuity but are NOT planned in this umbrella. They start once the backend phases above are working.

Frontend rename + i18n: route segment [search_space_id] -> [workspace_id], search-space-settings/ -> workspace-settings/, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (messages/{en,zh,es,pt,hi}.json), plus frontend Zero schema rename.
Satellite/client apps + docs rename: surfsense_desktop, surfsense_obsidian, surfsense_browser_extension, surfsense_evals, README/docs.
Connector two-type UI: restructure connector-popup and connector-constants.ts into the two labeled types.
Scraper-API + chat + positioning UI (replaces the superseded "Pipelines UI"): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), web.discover source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards.

Open items to confirm during subplanning

~~Rename transition: hard cutover vs temporary API aliases~~ RESOLVED: HARD CUTOVER (see resolved log + 02-rename-backend.md). The frontend is rebuilt against the corrected backend in its own umbrella; backend is verified via tests/OpenAPI, not the old UI.
~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ MOOT (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with.
~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ MOOT (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (automation_runs).
~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged MIGRATING and turned OFF for MVP (not re-pointed to MCP yet); only the generic MCP_CONNECTOR is a functional Type-2. Real MCP re-pointing is post-MVP.
Revamp open items (decide during subplanning): (1) the Ongoing-Automation mechanism (06, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large web.scrape / maps.reviews responses are bounded/streamed (04); (3) is web.discover metered or free (04); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (05); (5) public REST rate-limiting / abuse posture (05).

Resolved decisions log

PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped. The product is the scraper capabilities (web.* + maps.* typed verbs → cleaned, AI-ready data), exposed as REST + API key / MCP / chat generated from one registry — premium, OSS, self-hostable, the revenue driver. Stateless: no Timeline (3-store delta), no Tracker, no diff/materiality engine, no stored entity_changes; memory = the chat history (the agent reasons over prior tool outputs for "what changed"). Direct calls: capabilities return their result directly — no job store, no completed | pending + job_id envelope, no deliverable_wait polling; async delivery is write-then-sync / stream. Automation = a persistent ongoing chat re-invoking verbs (replaces Triggers; mechanism design-deferred). KB stays input-only. Phase re-cast: 04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration (intelligence_agent). Subplans 05a-timeline/05b-intelligence/06-triggers removed (git history); 04a/04b folded into 04/05. This supersedes the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below.
REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; revamp phases 4-7/ is the source of truth. A principal-engineer review found the flat 04a–07 files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a revamp phases 4-7/ folder, since flattened into 04–07). We adopt the revamp: Phase 4 = Capabilities & Access (typed verbs + generated doors; old 04a-connector-category demoted to hygiene, old 04b-source-discovery absorbed as web.discover), Phase 5 = Intelligence & Timeline (the Tracker + 3-store delta Timeline — the moat), Phase 6 = Triggers (reuse automations via a CI action), Phase 7 = Orchestration (intelligence_agent). The three architecture-correction decisions land in their real homes and one is revised: (1) bill at the capability executor — a code-verified gap review confirmed run_agent_task (automation agent_task) sets up no turn accumulator, so today scrape_webpage bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) "diffable run history" = the Timeline delta store (revamp/05a), superseding the earlier "structured automation_runs.output" decision. (3) "run now" = a Trigger adapter (revamp/06). The flat 04a/04b/05/06/07 files are retained with redirect banners; revamp/ filenames win on any number collision. Build order: 04a → 04b → 05a → 05b → 06 → 07.
ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted. The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against references/opencode (live tool-fetched context + persisted sessions, no scraped corpus; tool/websearch.ts live-crawls at query time). Decisions: (1) KB is input-only — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) Web search + WebURL crawler are platform-native tools (already web_search/scrape_webpage), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) Connectors = MCP tools + file/KB-input connectors; all branded natives → MCP. (4) Automations are the scheduling + run-history substrate (an agent_task run already invokes the full chat agent with the crawler/search tools on cron/event and persists automation_runs). (5) WEBCRAWLER_CONNECTOR RETIRED. Consequences: Phases 5/6/7 SUPERSEDED (docs kept with banners); Phase 4a reframed (drop is_pipeline_eligible; Type-1 = file/KB-input only; WEBCRAWLER retired); Phase 4b stands; new Phase 5′ automation-enhancement workstream (bill automation-run crawls; structured run.output; wire "run now"); Phase 8 actors re-slot as native tools/API. 03c connector-indexer billing branch becomes vestigial (billing lives on chat-turn / automation-run / API paths).
~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type.
Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).
Obsidian + Circleback: disabled for MVP.
MCP-availability audit: BookStack, Elasticsearch, Luma all have MCP available -> eligible for Type-2 (so deprecated as MIGRATING, not DISABLED). For MVP they are turned off pending the post-MVP MCP re-point, like the other branded natives.
Phase 4 connector taxonomy: modeled as a STATIC code registry (connector_type -> category/availability), NOT a DB column — no migration in 04a; is_indexable is KEPT (orthogonal). Only the generic MCP_CONNECTOR is a functional Type-2 for MVP; all branded natives (indexers + act-only, incl. Composio Gmail/Calendar) are MIGRATING (new-create blocked, /index+periodic+their subagents off, existing KB docs stay searchable). The MCP_CONNECTOR subagent routing-map gap (constants.py) is fixed in 04a.
Phase 4 search APIs: all 5 enum values dropped (SERPER_API/TAVILY_API/SEARXNG_API/LINKUP_API/BAIDU_SEARCH_API) in 04b. Survivors (SearXNG/Linkup/Baidu) become PLATFORM providers keyed by env (Linkup/Baidu keys move from per-connector config to env — app-wide, not per-workspace). 04b carries a destructive migration deleting the 5 connector types' rows.
Phase 4 structure: split into 04a (taxonomy/gating/MCP-fix, no migration) and 04b (search repurposing + source-discovery endpoint, with migration); intended order 04a -> 04b (both orders safe).
Rename transition policy: HARD CUTOVER of the external API (paths + JSON field names) in Phase 2 — no backward-compat aliases. Rationale: the frontend is (re)built against the corrected backend later, so there is no old client to keep alive; backend correctness is verified via the test suite + OpenAPI rather than the existing UI.
Crawler code location & licensing boundary (decided during 03a impl): the WebURL crawler engine — and future Phase-8 platform actors — live under surfsense_backend/app/proprietary/, a non-Apache-2 license boundary (its own LICENSE, currently an all-rights-reserved placeholder; the repo root stays Apache-2). 03a's WebCrawlerConnector / CrawlOutcome / CrawlOutcomeStatus moved to app/proprietary/web_crawler/ (public API re-exported from its __init__); the 3 Apache-2 callers (webcrawler indexer + both chat scrape_webpage tools) import from app.proprietary.web_crawler import .... Rule: everything under app/proprietary/** is non-Apache-2; Apache-2 code may import from it but not move into it. Rationale: keep the moat under a clearly-bounded, swappable license.
Proxy code placement (decided during 03b): the proxy provider package (app/utils/proxy/ — base/registry/anonymous_proxies/CustomProxyProvider/rotation) stays Apache-2 shared infra, NOT proprietary. Rationale: it's consumed by Apache-2 features unrelated to the moat (YouTube transcript route + indexer, chat tools' YouTube branch), and CustomProxyProvider is a thin wrapper over Scrapling's public ProxyRotator. Only the crawl-ladder-coupled rotation-retry (app/proprietary/web_crawler/connector.py::_run_tier_with_proxy_retry) lives under the boundary. Boundary test: code goes in app/proprietary/ only if used exclusively by the moat (applies to 03e's geoip/sticky-proxy hardening too — the bypass-specific tuning is proprietary; the generic provider plumbing is not).
WebURL Crawler framework: STANDARDIZE on Scrapling; remove Firecrawl entirely (no other scraping frameworks now or planned). Scrapling's StealthyFetcher (patchright-Chromium as of 0.4.9 — not Camoufox) handles Cloudflare; 03e stealth-hardening minimizes challenges; captcha-tools (03d) covers the rest. All fetch tiers sit behind a FetchStrategy seam returning CrawlOutcome (callers never depend on the tier).
Crawl billing: reuse the existing credit wallet (credit_micros_balance) with a new web_crawl usage_type. Price: $1 / 1000 successful requests (1000 micro-USD per success). Connector/pipeline crawls bill the workspace owner; chat scrapes fold their crawl cost into the already-billed chat turn. Gated by WEB_CRAWL_CREDIT_BILLING_ENABLED (off for self-hosted); no DB migration required.
Billable unit: one unit per URL that returns usable extracted content, regardless of how many internal fallback tiers were attempted (not per HTTP fetch, not per URL-processed).
Captcha solving (captcha-tools): ACTIVE (no longer deferred) — sequenced last in Phase 3 (03d), after 03e hardening, opt-in + off by default. Cloudflare stays in-framework (03a); reCAPTCHA/hCaptcha/image use captchatools. Billing asymmetry RESOLVED → option (a): a separate per-attempt web_crawl_captcha unit (WEB_CRAWL_CAPTCHA_* knobs on WebCrawlCreditService), since solvers charge per attempt regardless of crawl success. ErrNoBalance halts solving (no retry-loop IP bans).
Crawler stealth strategy (the moat): CloakBrowser REJECTED (source-patched Chromium binary requires an OEM/SaaS license incompatible with our model). External unblocker APIs DEFERRED (ZenRows/ScrapFly/Bright Data) — pre-wired as a config-flagged FetchStrategy (03e §8) but not built. Plan: maintain an in-house bypass moat for ~4–6 months (Scrapling stealth + residential proxies + 03e hardening + 03d captcha), then move hostile/top-tier-fingerprinted targets (DataDome/Kasada/reCAPTCHA-Enterprise) to a paid tier if demand/maintenance justifies it. Realistic ceiling acknowledged: in-house beats Cloudflare + the moderate long tail, not top-tier behavioral fingerprinting.
Authenticated/logged-in scraping: OUT OF SCOPE this MVP (public data only). Sticky/static proxies + credential management are deferred and paired with the future platform actors (03b static-proxy hand-off + Phase 8).
Phase 3 stealth-hardening subplan 03e ADDED: geoip locale/tz coherence, hide_canvas/block_webrtc, persistent per-domain profiles, headed+Xvfb, fonts, DIY humanization (page_action; Chromium engine has no built-in humanize), a block classifier, and per-domain strategy memory (Redis, no migration).
Phase 3 test-harness subplan 03f ADDED: manual-only (no CI/automated gating now) undetectability + extraction scorecard, modeled on CloakBrowser's bin/cloaktest. Two labeled axes (Suite S stealth + Suite E extraction) so they scale independently. Drives the real Scrapling tiers (browser + curl_cffi HTTP/TLS), reuses 03d's page_action+closure-cell for JS-object verdicts. TLS JA3/JA4 parity = informational axis, not a hard gate. Adopt CloakBrowser bars as aspirational; record our actual free-stack numbers as the committed baseline. The scorecard is the documented evidence/trigger for flipping 03e's deferred paid-unblocker tier.
Licensing placement of 03e/03f code (decided during 03f impl, applying the §boundary test): the stealth kwargs builder + geoip coherence (03e bypass tuning) live proprietary at app/proprietary/web_crawler/stealth.py; the block classifier (passive telemetry, public markers) stays Apache-2 at app/utils/crawl/classifier.py (direct analog of the captcha split: proprietary captcha.py logic + Apache-2 app/utils/captcha/ config). The 03f scorecard harness moved whole to app/proprietary/web_crawler/testbench/ (run python -m app.proprietary.web_crawler.testbench) — it's the moat's measurement tool and can't be cleanly half-moved (a proprietary Suite S would back-import generic scaffolding from scripts/, a forbidden app→scripts direction). The scripts/e2e_phase3_crawl_billing.py billing e2e stays in scripts/ (Apache-2) since it exercises billing, not the stealth moat.
Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7.
[SUPERSEDED — Architecture correction 2026-06-30; no pipelines] Phase 5 pipelines data model: two new tables pipelines (mutable) + pipeline_runs (append-only), modeled on automations/automation_runs; ORM lives in db.py next to connectors/folders. connector_id nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's is_pipeline_eligible. Schedule = schedule_cron + schedule_timezone (default UTC) + next_scheduled_at (cron, matching automations). pipeline_runs pre-includes charged_micros/crawls_*/result_blob_key so Phase 6 needs no extra migration. Both tables published to Zero full-row (like folders/connectors). Routes reuse CONNECTORS_* permissions. Phase 5 ships the data model + API surface only; the /run endpoint enqueues a Phase-6 task stub.
[SUPERSEDED — Architecture correction 2026-06-30; uploads just populate the input-only KB] Phase 7 uploads-as-pipeline: a singleton "Uploads" pipeline per workspace (connector_id NULL, save_to_kb=true), lazily get-or-created (race-safe via a partial unique index ON pipelines(workspace_id) WHERE connector_id IS NULL). Each fileupload/folder-upload request writes a terminal audit PipelineRun(trigger=upload, status=succeeded, documents_indexed=<accepted count>) — uploads are route-recorded, not engine-executed (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an inner try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; charged_micros NULL). Per-file ETL truth stays on Document.status; accurate roll-up needs the deferred documents.pipeline_run_id provenance. Connector save_to_kb default stays False (opt-in for connectors, mandatory for uploads). Phase 7 also guards Phase-5's generic CRUD against the system Uploads pipeline: POST /pipelines rejects connector_id=None (supersedes Phase 5's permissive create), /run and schedule-PUT reject NULL-connector pipelines, and Phase 6's scheduler _claim_due filters connector_id IS NOT NULL as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See 07-upload-pipeline-kb.md.
[SUPERSEDED — Architecture correction 2026-06-30; automations are the run engine] Phase 6 pipeline execution: run engine mirrors automations (thin Celery run_pipeline(run_id) → execute_pipeline_run; PENDING-gated, idempotent terminal no-op; pending→running→succeeded/failed with timing/counts/error). MVP executor = WebURL crawler only (other types fail cleanly). save_to_kb=true reuses index_crawled_urls extended with a folder_id param (lands in the destination folder); save_to_kb=false runs a fetch-only loop and persists one JSON blob via file_storage (result_blob_key). Crawl billing is owned by the run engine for the pipeline path (pre-check on len(urls) + charge crawls_succeeded + idempotent charged_micros), calling the crawler with a new bill=False seam (the connector /index+periodic paths keep 03c's in-indexer bill=True) — so non-KB runs are billed identically. Scheduler = a pipeline_schedule_select Beat tick modeled on the automations cron selector (cron + FOR UPDATE SKIP LOCKED + self-heal, using the existing croniter util), plus the de-dup guard (a pipeline over a connector disables that connector's periodic_indexing_enabled). Chat context = the get_pipeline_runs tool. Carries a small additive 05 amendment: a schedule_timezone column (cron util needs a tz). See 06-pipelines-exec.md.

Subplan index (backend)

Phase	Subplan file	Status
1	`01-rename-db.md`	SHIPPED (2026-06-27, PR #1546/#1562; migration `170`)
2	`02-rename-backend.md`	SHIPPED (2026-06-27, PR #1546/#1562)
3	`03a-crawler-core.md`	IMPLEMENTED (`ci_mvp` @ `5c36cd3`) — Firecrawl removed, Scrapling-only 3-tier `CrawlOutcome`, crawler relocated to `app/proprietary/web_crawler/`
3	`03b-proxy-expansion.md`	IMPLEMENTED (`ci_mvp` @ `6226012`) — `CustomProxyProvider` (BYO single/pool) + registry + bounded rotation-retry
3	`03c-crawl-billing.md`	IMPLEMENTED (`ci_mvp` @ `17bdb0682`) — `WebCrawlCreditService` (config-driven price) + indexer wiring + chat-turn fold; functional e2e green
3	`03e-stealth-hardening.md`	Slice A IMPLEMENTED (`ci_mvp`) — stealth kwargs builder + geoip coherence (proprietary `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker)
3	`03d-captcha-solving.md`	IMPLEMENTED (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default
3	`03f-undetectability-testing.md`	IMPLEMENTED (`ci_mvp`) — manual scorecard under the proprietary boundary at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff
—	`00b-pipeline-diagrams.md`	end-to-end flow diagrams (companion to this umbrella)
4	`04-capabilities.md`	CANONICAL · drafted — capability registry + verbs `web.scrape`/`web.discover`/`maps.` → cleaned data; bill at executor. ← build next*
5	`05-access.md`	CANONICAL · drafted — generated REST/MCP/chat doors + intent router + BYO-MCP routing fix
6	`06-ongoing-automation.md`	CANONICAL · design deferred — chat-native "keep watching" (periodic re-invocation)
7	`07-orchestration.md`	CANONICAL · drafted — `intelligence_agent` subagent; "what changed" from chat history

Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases").

54 KiB Raw Blame History Unescape Escape