SurfSense/plans/backend/00-umbrella-plan.md

# CI Pivot MVP — Umbrella Plan

> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`).

This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities** ([`04-capabilities.md`](04-capabilities.md)), **Phase 5 Access** ([`05-access.md`](05-access.md)), **Phase 6 Ongoing-Automation** ([`06-ongoing-automation.md`](06-ongoing-automation.md), design deferred), **Phase 7 Orchestration** ([`07-orchestration.md`](07-orchestration.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful **Tracker + Timeline + Triggers** design was superseded on **2026-07-01** (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below.

> SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases.

## Positioning

Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return **cleaned, AI-ready data** for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time.

## ⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native

> **This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6).** The canonical Phase 4–7 subplans (`04`–`07`, below) are re-cast to four phases:
>
> - **The product = scraper capabilities**, not a CI engine. `web.*` + `maps.*` typed verbs return cleaned, AI-ready data, exposed identically through **REST + API key**, an **MCP server**, and **chat**, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable).
> - **Stateless.** No Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`. Memory = the **chat history**; the agent reasons over prior tool outputs in context to report "what changed."
> - **Direct calls.** Capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling. Async results reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
> - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6).
> - **KB stays input-only**; crawled data is never indexed.
>
> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** (deferred) · **07 Orchestration** (`intelligence_agent`). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`.

## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted

> **Historical record.** The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the **2026-07-01 pivot** above.

> **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model:
>
> - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages.
> - **Web search + the WebURL crawler = platform-native agent tools** (already `web_search` + `scrape_webpage` on the main agent, `main_agent/tools/index.py`), always available; later exposed to developers via a platform **API key**. They are **NOT connectors and NOT pipelines**.
> - **Connectors = MCP tools** (Type-2) **+ file/KB-input connectors** (the only Type-1 that survives). All branded natives → MCP.
> - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots.
> - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source.
>
> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical subplans `04`–`07` (this framing was itself superseded by the 2026-07-01 pivot above):
> - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.*
> - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine.
> - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path).
> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `intelligence_agent` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language.
>
> **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors.

## Target architecture (scraper-APIs-first, stateless)

> Full end-to-end flow diagrams live in [`00b-pipeline-diagrams.md`](00b-pipeline-diagrams.md). Summary below.

```mermaid
flowchart TD
  ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]]
  REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"]
  ACQ --> REG
  REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"]
  DOORS --> AGENT["intelligence_agent subagent (P7): intent routing · verb chains · 'what changed' from chat history"]
  KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT
  CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT
  AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"]
  AGENT -.->|standing need| ONG["Ongoing-Automation (P6, DEFERRED): persistent chat re-invokes verbs; memory = chat history"]
  ONG -.-> REG
  APIKEY["Platform API key"] --> DOORS
```

<details>
<summary>Original (superseded) Pipelines-centric architecture</summary>

```mermaid
flowchart TD
  WS[WorkSpace] --> CONN[Connectors]
  WS --> PIPE[Pipelines]
  WS --> KB[(Knowledge Base: documents + chunks)]
  CONN --> T1[Type 1: Data Sources - pull]
  CONN --> T2[Type 2: MCP Tools - act]
  T1 --> WEB[Universal WebURL Crawler - functional]
  T1 --> PLAT[Platform connectors - coming soon]
  T1 --> UP[File Upload]
  PIPE --> RUN[PipelineRun history]
  PIPE -->|"save_to_kb + destination folder"| KB
  RUN -->|"manual or cron"| T1
  T2 --> CHAT[Chat / Automations]
  CHAT --> DELIV[Deliverables: audio/video/report/image]
  RUN -->|"read-only context"| CHAT
```

</details>

## Decisions locked

- Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps.
- Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`.
- **Capabilities, not connectors, are the core of the product.** The crawler + search + Maps actor become typed **verbs** (`web.scrape`, `web.discover`, `maps.search/place/reviews`) in a single **capability registry** that generates the REST/MCP/chat doors identically (`04`/`05`). A capability is *call → cleaned data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into `05` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system.
- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
- **Stateless — the chat history is the memory.** No Timeline, no `Tracker`, no diff/materiality engine, no stored `entity_changes`. "What changed" is the agent reasoning over prior tool outputs already in context (`07`). Standing needs are served by a **persistent ongoing chat** that periodically re-invokes verbs (`06`, design deferred).
- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via the billing service (`WebCrawlCreditService`, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter **uniformly**. `maps.*` registers a new per-place / per-page unit.
- Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP.
- MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point).
- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a **persistent ongoing chat** re-invoking verbs (`06`, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper).
- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by **reasoning over the chat history** (prior tool outputs already in context, `07`), not by re-deriving from run logs or a stored timeline.
- **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**.
- Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that.

## Platform connector research list (deferred build, MVP = "coming soon")

- LinkedIn — people profiles (discovery by keyword/company), company info, job listings.
- Amazon — product (ASIN), search (keyword), pricing; reviews secondary.
- Google — Web Search (organic SERP), AI Overviews, Maps/Local (discover by location).
- Instagram — profiles first, then posts; discover profiles by username/keyword.
- Zillow / Redfin — full property listings (discover by search URL/filters); Zillow price history.
- Walmart — product, search; zipcode-localized pricing premium variant.
- eBay — search by keyword/category; price-comparison/resale feeds.
- Crunchbase — company info, search by keyword (B2B lead-gen / investor research).
- TikTok / YouTube — profiles/channels, posts/videos; discover by keyword/hashtag; TikTok Shop.
- Indeed / Glassdoor — job listings (discover by keyword in location), company reviews.

## Backend phases (active — this umbrella)

### Phase 1 — Rename foundation (DB) [`subplan: 01-rename-db.md`]

> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Migration `170` (chains `169`, current head) does the physical rename + Zero-publication reconcile. Shipped atomically with Phase 2. As-built record + re-runnable verification live in `01-rename-db.md`. **Deploy caveat:** zero-cache replica reset (`ZERO_AUTO_RESET`) required; from-scratch `alembic upgrade head` stays pre-existing-broken (rev 23 conflict — separate baseline-squash task), only the `169→170` path is verified.

- Alembic migration: rename `searchspaces` -> `workspaces`; rename `search_space_id` -> `workspace_id` on ~20 child tables; rename RBAC tables and their FKs; rename indexes/constraints (`uq_searchspace_*`, `idx_documents_search_space_id`, etc.); update Rocicorp Zero publication column lists (backend-owned `publication` definition; frontend Zero schema rename happens in the later frontend umbrella).
- Decide transition strategy: hard cutover (simplest for MVP) vs temporary API aliases for clients.
- Key files: `surfsense_backend/app/db.py`, `surfsense_backend/alembic/versions/` (new migration).

### Phase 2 — Rename backend (code + API) [`subplan: 02-rename-backend.md`]

> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Symbolic rename across `app/` + `tests/` (Phase-1 shim dropped), API consolidated to `/workspaces` (legacy `/searchspaces` · `/search-spaces` · `/search-space` all retired/404). Verified (ground-truth `git grep`, 2026-06-29): every residual `search_space`/`SEARCH_SPACE` is a documented carve-out — enum values (`ConnectionScope`/`ChatVisibility.SEARCH_SPACE`), the `'SEARCH_SPACE'` CHECK literal (now paired with `workspace_id`), Celery wire names (`delete_search_space_background`, `ai_sort_search_space`), OTel key `search_space.id`, and the `SEARCH_SPACE_FORBIDDEN` error code; `alembic/versions/` untouched except `168`+`170`. Suite: `3016 passed, 1 skipped`. As-built record in `02-rename-backend.md`. **Clients are intentionally broken** until the frontend/satellite umbrellas land (hard cutover).

- Rename models/schemas/services/routes/agents/tasks identifiers: `SearchSpace*` -> `Workspace*`, `search_space_id` -> `workspace_id`.
- Consolidate API to `/workspaces` and fix the `/searchspaces` vs `/search-spaces` inconsistency.
- High-touch files: `routes/search_spaces_routes.py`, `routes/rbac_routes.py`, `utils/rbac.py` (`check_search_space_access`), `schemas/search_space.py`, plus `search_space_id` threading through agents/Redis keys/storage paths (`documents/{id}/...`).

### Phase 3 — WebURL Crawler & Crawl Billing (backend) [`subplans: 03a–03f`]

The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans:

> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (`04`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then `04` Capabilities → `05` Access → `07` Orchestration), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized.

- **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran).
- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability override is left as a no-op seam (`04`), not built.
- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture pivot + revamp — see log): the charge moves to the **capability executor** (`04`), so **every** caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type).
- **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier.
- **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account.
- **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier.

### Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`]

> **Canonical subplan:** [`04-capabilities.md`](04-capabilities.md). **Build first** — every later phase calls the registry.

- Turn Acquisition (crawler + search + Maps actor) into a **small set of typed, callable verbs** over a single **capability registry** — `web.scrape(urls[])`, `web.discover(query, top_k)`, `maps.search/place/reviews`. Each verb is a **direct async call that returns cleaned, AI-ready data** (no job store, no envelope, no polling). `maps.*` returns typed structured objects, not markdown. Each declares a **`billing_unit`**; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary.

**Locked model (MVP):**

- **Capabilities replace connectors as data sources.** `web.*` / `maps.*` are *call → cleaned data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule.
- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `05`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**.
- **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP.
- **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook).
- Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.)
- The **Google Maps actor is net-new**, proprietary, built as a separate effort; `maps.*` are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors.
- Frontend connector/capability UI restructure is DEFERRED (frontend umbrella).

### Phase 5 — Access (backend) [`subplan: 05-access.md`]

> **Canonical subplan:** [`05-access.md`](05-access.md). **Build after `04`.** Together `04 + 05` ship the scraper-API product — revenue day one.

- Expose the registry through **three generated doors** — **REST + API key** (public day one), an **MCP server** (fast-follow), and **chat tools** — each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data`). The executor returns directly; there is **no envelope, no `deliverable_wait` polling**. Async results (e.g. the `06` ongoing mode) reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
- **Natural language is the human surface** (raw verbs only on REST/MCP); the chat door classifies **one-shot** vs **keep-watching** and hands keep-watching to `06`. Two MCP directions kept distinct: **serve** our verbs as an MCP server (new door) vs **consume** the user's BYO `MCP_CONNECTOR` (the old-`04a` routing-gap fix lands here).

### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`]

> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **⚠️ design deferred.** The periodic mechanism is designed separately, after `04`/`05`/`07`. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel).

- Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls.

### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`]

> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Build last** (atop `04`/`05`, and `06` for the ongoing mode). We plug into the shipped multi-agent runtime; we don't rebuild it.

- Ship the net-new builtin **`intelligence_agent`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, one clarifying question when ambiguous), **verb composition** (`web.discover → web.scrape`, `maps.search → maps.place → maps.reviews`), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store).
- Toolset: registry-backed capability verbs (shared with `research`) + a deferred `start_watch` handoff to `06`. Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`).

### (Future) Platform API-key + public MCP

Platform **API-key** access to the capability verbs + a public **MCP server** exposing the input-only KB — post-MVP (see Phase 8 + Deferred).

### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`]

NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the **committed MVP actor** (a `maps.*` namespace, Phase 4). Beyond it, more **platform scrapers** — a family of **individual scraping endpoints** (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **examples, none committed**) — layer **on top** of the hardened fetch core once it's solid. **Each is simply another capability verb** (`04`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha). Billed per call via `WebCrawlCreditService`.

- **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream.
- The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling.

## Deferred — Frontend & client phases (separate umbrella, planned LATER)

These are recorded for continuity but are NOT planned in this umbrella. They start once the backend phases above are working.

- Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename.
- Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs.
- Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types.
- Scraper-API + chat + positioning UI (**replaces the superseded "Pipelines UI"**): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards.

## Open items to confirm during subplanning

- ~~Rename transition: hard cutover vs temporary API aliases~~ RESOLVED: HARD CUTOVER (see resolved log + 02-rename-backend.md). The frontend is rebuilt against the corrected backend in its own umbrella; backend is verified via tests/OpenAPI, not the old UI.
- ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with.
- ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`).
- ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP.
- **Revamp open items (decide during subplanning):** (1) the **Ongoing-Automation mechanism** (`06`, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large `web.scrape` / `maps.reviews` responses are bounded/streamed (`04`); (3) is `web.discover` metered or free (`04`); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (`05`); (5) public REST rate-limiting / abuse posture (`05`).

## Resolved decisions log

- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`intelligence_agent`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below.
- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`intelligence_agent`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`.
- **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths).
- ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type.
- Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).
- Obsidian + Circleback: disabled for MVP.
- MCP-availability audit: BookStack, Elasticsearch, Luma all have MCP available -> eligible for Type-2 (so deprecated as `MIGRATING`, not `DISABLED`). For MVP they are turned off pending the post-MVP MCP re-point, like the other branded natives.
- Phase 4 connector taxonomy: modeled as a STATIC code registry (`connector_type` -> category/availability), NOT a DB column — no migration in 04a; `is_indexable` is KEPT (orthogonal). Only the generic `MCP_CONNECTOR` is a functional Type-2 for MVP; all branded natives (indexers + act-only, incl. Composio Gmail/Calendar) are `MIGRATING` (new-create blocked, `/index`+periodic+their subagents off, existing KB docs stay searchable). The `MCP_CONNECTOR` subagent routing-map gap (`constants.py`) is fixed in 04a.
- Phase 4 search APIs: all 5 enum values dropped (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) in 04b. Survivors (SearXNG/Linkup/Baidu) become PLATFORM providers keyed by env (Linkup/Baidu keys move from per-connector `config` to env — app-wide, not per-workspace). 04b carries a destructive migration deleting the 5 connector types' rows.
- Phase 4 structure: split into 04a (taxonomy/gating/MCP-fix, no migration) and 04b (search repurposing + source-discovery endpoint, with migration); intended order 04a -> 04b (both orders safe).
- Rename transition policy: HARD CUTOVER of the external API (paths + JSON field names) in Phase 2 — no backward-compat aliases. Rationale: the frontend is (re)built against the corrected backend later, so there is no old client to keep alive; backend correctness is verified via the test suite + OpenAPI rather than the existing UI.
- Crawler code location & licensing boundary (decided during 03a impl): the WebURL crawler engine — and future Phase-8 platform actors — live under `surfsense_backend/app/proprietary/`, a **non-Apache-2 license boundary** (its own `LICENSE`, currently an all-rights-reserved placeholder; the repo root stays Apache-2). 03a's `WebCrawlerConnector` / `CrawlOutcome` / `CrawlOutcomeStatus` moved to `app/proprietary/web_crawler/` (public API re-exported from its `__init__`); the 3 Apache-2 callers (webcrawler indexer + both chat `scrape_webpage` tools) import `from app.proprietary.web_crawler import ...`. Rule: everything under `app/proprietary/**` is non-Apache-2; Apache-2 code may import *from* it but not move *into* it. Rationale: keep the moat under a clearly-bounded, swappable license.
- Proxy code placement (decided during 03b): the proxy provider package (`app/utils/proxy/` — base/registry/`anonymous_proxies`/`CustomProxyProvider`/rotation) **stays Apache-2 shared infra**, NOT proprietary. Rationale: it's consumed by Apache-2 features unrelated to the moat (YouTube transcript route + indexer, chat tools' YouTube branch), and `CustomProxyProvider` is a thin wrapper over Scrapling's *public* `ProxyRotator`. Only the crawl-ladder-coupled rotation-retry (`app/proprietary/web_crawler/connector.py::_run_tier_with_proxy_retry`) lives under the boundary. **Boundary test:** code goes in `app/proprietary/` only if used *exclusively* by the moat (applies to 03e's geoip/sticky-proxy hardening too — the bypass-specific tuning is proprietary; the generic provider plumbing is not).
- WebURL Crawler framework: STANDARDIZE on Scrapling; **remove Firecrawl entirely** (no other scraping frameworks now or planned). Scrapling's `StealthyFetcher` (patchright-Chromium as of 0.4.9 — **not** Camoufox) handles Cloudflare; `03e` stealth-hardening minimizes challenges; captcha-tools (`03d`) covers the rest. All fetch tiers sit behind a `FetchStrategy` seam returning `CrawlOutcome` (callers never depend on the tier).
- Crawl billing: reuse the existing credit wallet (`credit_micros_balance`) with a new `web_crawl` usage_type. Price: **$1 / 1000 successful requests** (1000 micro-USD per success). Connector/pipeline crawls bill the **workspace owner**; chat scrapes fold their crawl cost into the already-billed chat turn. Gated by `WEB_CRAWL_CREDIT_BILLING_ENABLED` (off for self-hosted); no DB migration required.
- Billable unit: one unit per URL that returns usable extracted content, regardless of how many internal fallback tiers were attempted (not per HTTP fetch, not per URL-processed).
- Captcha solving (captcha-tools): **ACTIVE** (no longer deferred) — sequenced last in Phase 3 (`03d`), **after** `03e` hardening, **opt-in + off by default**. Cloudflare stays in-framework (`03a`); reCAPTCHA/hCaptcha/image use `captchatools`. **Billing asymmetry RESOLVED → option (a): a separate per-attempt `web_crawl_captcha` unit** (`WEB_CRAWL_CAPTCHA_*` knobs on `WebCrawlCreditService`), since solvers charge per attempt regardless of crawl success. `ErrNoBalance` halts solving (no retry-loop IP bans).
- Crawler stealth strategy (the moat): **CloakBrowser REJECTED** (source-patched Chromium binary requires an OEM/SaaS license incompatible with our model). **External unblocker APIs DEFERRED** (ZenRows/ScrapFly/Bright Data) — pre-wired as a config-flagged `FetchStrategy` (`03e §8`) but not built. Plan: maintain an **in-house bypass moat for ~4–6 months** (Scrapling stealth + residential proxies + `03e` hardening + `03d` captcha), then move hostile/top-tier-fingerprinted targets (DataDome/Kasada/reCAPTCHA-Enterprise) to a paid tier if demand/maintenance justifies it. Realistic ceiling acknowledged: in-house beats Cloudflare + the moderate long tail, not top-tier behavioral fingerprinting.
- Authenticated/logged-in scraping: **OUT OF SCOPE this MVP** (public data only). Sticky/static proxies + credential management are deferred and paired with the future platform actors (`03b` static-proxy hand-off + Phase 8).
- Phase 3 stealth-hardening subplan `03e` ADDED: geoip locale/tz coherence, `hide_canvas`/`block_webrtc`, persistent per-domain profiles, headed+Xvfb, fonts, DIY humanization (`page_action`; Chromium engine has no built-in `humanize`), a block classifier, and per-domain strategy memory (Redis, no migration).
- Phase 3 test-harness subplan `03f` ADDED: **manual-only** (no CI/automated gating now) undetectability + extraction scorecard, modeled on CloakBrowser's `bin/cloaktest`. Two labeled axes (Suite S stealth + Suite E extraction) so they scale independently. Drives the **real** Scrapling tiers (browser + curl_cffi HTTP/TLS), reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. **TLS JA3/JA4 parity = informational axis, not a hard gate.** Adopt CloakBrowser bars as aspirational; record our actual free-stack numbers as the committed baseline. The scorecard is the documented evidence/trigger for flipping `03e`'s deferred paid-unblocker tier.
- Licensing placement of 03e/03f code (decided during 03f impl, applying the §boundary test): the **stealth kwargs builder + geoip coherence** (`03e` bypass tuning) live **proprietary** at `app/proprietary/web_crawler/stealth.py`; the **block classifier** (passive telemetry, public markers) stays **Apache-2** at `app/utils/crawl/classifier.py` (direct analog of the captcha split: proprietary `captcha.py` logic + Apache-2 `app/utils/captcha/` config). The **03f scorecard harness** moved whole to `app/proprietary/web_crawler/testbench/` (run `python -m app.proprietary.web_crawler.testbench`) — it's the moat's measurement tool and can't be cleanly half-moved (a proprietary Suite S would back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). The `scripts/e2e_phase3_crawl_billing.py` billing e2e stays in `scripts/` (Apache-2) since it exercises billing, not the stealth moat.
- Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7.
- **[SUPERSEDED — Architecture correction 2026-06-30; no pipelines]** Phase 5 pipelines data model: two new tables `pipelines` (mutable) + `pipeline_runs` (append-only), modeled on `automations`/`automation_runs`; ORM lives in `db.py` next to connectors/folders. `connector_id` nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's `is_pipeline_eligible`. Schedule = `schedule_cron` + `schedule_timezone` (default UTC) + `next_scheduled_at` (cron, matching automations). `pipeline_runs` pre-includes `charged_micros`/`crawls_*`/`result_blob_key` so Phase 6 needs no extra migration. Both tables published to Zero **full-row** (like folders/connectors). Routes reuse `CONNECTORS_*` permissions. Phase 5 ships the data model + API surface only; the `/run` endpoint enqueues a Phase-6 task stub.
- **[SUPERSEDED — Architecture correction 2026-06-30; uploads just populate the input-only KB]** Phase 7 uploads-as-pipeline: a **singleton "Uploads" pipeline** per workspace (`connector_id NULL`, `save_to_kb=true`), lazily get-or-created (race-safe via a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL`). Each `fileupload`/`folder-upload` request writes a **terminal audit `PipelineRun(trigger=upload, status=succeeded, documents_indexed=<accepted count>)`** — uploads are **route-recorded, not engine-executed** (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an **inner** try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; `charged_micros` NULL). Per-file ETL truth stays on `Document.status`; accurate roll-up needs the deferred `documents.pipeline_run_id` provenance. Connector `save_to_kb` default stays `False` (opt-in for connectors, mandatory for uploads). Phase 7 also **guards Phase-5's generic CRUD** against the system Uploads pipeline: `POST /pipelines` rejects `connector_id=None` (supersedes Phase 5's permissive create), `/run` and schedule-`PUT` reject NULL-connector pipelines, and Phase 6's scheduler `_claim_due` filters `connector_id IS NOT NULL` as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See `07-upload-pipeline-kb.md`.
- **[SUPERSEDED — Architecture correction 2026-06-30; automations are the run engine]** Phase 6 pipeline execution: run engine mirrors **automations** (thin Celery `run_pipeline(run_id)` → `execute_pipeline_run`; PENDING-gated, idempotent terminal no-op; `pending→running→succeeded/failed` with timing/counts/error). MVP executor = **WebURL crawler only** (other types fail cleanly). `save_to_kb=true` reuses `index_crawled_urls` extended with a `folder_id` param (lands in the destination folder); `save_to_kb=false` runs a **fetch-only** loop and persists one JSON blob via `file_storage` (`result_blob_key`). **Crawl billing is owned by the run engine** for the pipeline path (pre-check on `len(urls)` + charge `crawls_succeeded` + idempotent `charged_micros`), calling the crawler with a new `bill=False` seam (the connector `/index`+periodic paths keep `03c`'s in-indexer `bill=True`) — so non-KB runs are billed identically. Scheduler = a `pipeline_schedule_select` Beat tick modeled on the automations cron **selector** (cron + `FOR UPDATE SKIP LOCKED` + self-heal, using the existing `croniter` util), plus the **de-dup guard** (a pipeline over a connector disables that connector's `periodic_indexing_enabled`). Chat context = the `get_pipeline_runs` tool. Carries a small additive `05` amendment: a `schedule_timezone` column (cron util needs a tz). See `06-pipelines-exec.md`.

## Subplan index (backend)

| Phase | Subplan file | Status |
|-------|--------------|--------|
| 1 | `01-rename-db.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562); migration `170`) |
| 2 | `02-rename-backend.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562)) |
| 3 | `03a-crawler-core.md` | **IMPLEMENTED** (`ci_mvp` @ `5c36cd3`) — Firecrawl removed, Scrapling-only 3-tier `CrawlOutcome`, crawler relocated to `app/proprietary/web_crawler/` |
| 3 | `03b-proxy-expansion.md` | **IMPLEMENTED** (`ci_mvp` @ `6226012`) — `CustomProxyProvider` (BYO single/pool) + registry + bounded rotation-retry |
| 3 | `03c-crawl-billing.md` | **IMPLEMENTED** (`ci_mvp` @ `17bdb0682`) — `WebCrawlCreditService` (config-driven price) + indexer wiring + chat-turn fold; functional e2e green |
| 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) |
| 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default |
| 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff |
| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) |
| 4 | `04-capabilities.md` | **CANONICAL** · drafted — capability registry + verbs `web.scrape`/`web.discover`/`maps.*` → cleaned data; bill at executor. **← build next** |
| 5 | `05-access.md` | **CANONICAL** · drafted — generated REST/MCP/chat doors + intent router + BYO-MCP routing fix |
| 6 | `06-ongoing-automation.md` | **CANONICAL** · **design deferred** — chat-native "keep watching" (periodic re-invocation) |
| 7 | `07-orchestration.md` | **CANONICAL** · drafted — `intelligence_agent` subagent; "what changed" from chat history |

Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases").