diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md
deleted file mode 100644
index cfcda1489..000000000
--- a/plans/backend/00-umbrella-plan.md
+++ /dev/null
@@ -1,257 +0,0 @@
-# CI Pivot MVP — Umbrella Plan
-
-> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`).
-
-This is the high-level roadmap. It is sequenced: rename first (Phases 1–2, shipped), then the WebURL crawler moat (Phase 3, shipped), then the CI product itself as **four revamped phases** — **Phase 4 Capabilities** ([`04-capabilities.md`](04-capabilities.md)), **Phase 5 Access** ([`05-access.md`](05-access.md)), **Phase 6 Ongoing-Automation** ([`06-ongoing-automation.md`](06-ongoing-automation.md), design deferred), **Phase 7 Orchestration** ([`07-orchestration.md`](07-orchestration.md)). **NOTE:** the original "connectors → Pipelines" plan (old Phases 4–7) was superseded on 2026-06-30, and the stateful **Tracker + Timeline + Triggers** design was superseded on **2026-07-01** (scraper-APIs-first, stateless, chat-native) — see the Architecture corrections below.
-
-> SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases.
-
-## Positioning
-
-Premium, open-source, self-hostable **scraper APIs** (Bright Data / Firecrawl / Scrapfly-class) that solve the hard parts — bypass, proxies, stealth — and return **cleaned, AI-ready data** for Google Maps + general web. A chat agent uses those APIs to answer competitive-intelligence questions and, for standing needs, keeps watching over time.
-
-## ⚠️ Architecture pivot (2026-07-01) — scraper-APIs-first, stateless, chat-native
-
-> **This supersedes the stateful Tracker/Timeline/Triggers design (the 2026-06-30 revamp's Phases 5–6).** The canonical Phase 4–7 subplans (`04`–`07`, below) are re-cast to four phases:
->
-> - **The product = scraper capabilities**, not a CI engine. `web.*` + `maps.*` typed verbs return cleaned, AI-ready data, exposed identically through **REST + API key**, an **MCP server**, and **chat**, all generated from one capability registry. These endpoints are the revenue driver (premium, OSS, self-hostable).
-> - **Stateless.** No Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`. Memory = the **chat history**; the agent reasons over prior tool outputs in context to report "what changed."
-> - **Direct calls.** Capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling. Async results reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
-> - **Automation = a persistent ongoing chat** that periodically re-invokes verbs. This replaces the `Triggers` subsystem; the periodic mechanism is **design-deferred** (Phase 6).
-> - **KB stays input-only**; crawled data is never indexed.
->
-> **Phase mapping:** **04 Capabilities** · **05 Access** · **06 Ongoing-Automation** · **07 Orchestration** (`scraping` subagent). The subplans `05a-timeline`/`05b-intelligence`/`06-triggers` are **removed** (history in git); `04a`/`04b` are folded into `04`/`05`.
-
-## ⚠️ Architecture correction (2026-06-30) — Pipelines dropped; automations + input-only KB adopted
-
-> **Historical record.** The pipelines-drop + input-only-KB decisions below still hold. The stateful Tracker/Timeline/Triggers framing in its "Net effect" bullets was itself superseded by the **2026-07-01 pivot** above.
-
-> **This supersedes the original Pipelines paradigm (Phases 5–7).** During Phase 4 discussion we concluded that the *"sync data into the KB, then operate over it"* model is wrong for competitive intelligence. Verified against `references/opencode` — a coding agent that pulls context **live** via `read`/`grep`/`webfetch`/`websearch` tools (`packages/opencode/src/tool/`) and persists **sessions**, never a scraped corpus; its `websearch` tool even live-crawls at query time (`tool/websearch.ts`). The corrected model:
->
-> - **Knowledge Base = input-only** — the user's personal files/context that *enriches* the agent. Filled by file uploads + file connectors (Google Drive / Dropbox / OneDrive). **Nothing scrapes into it.** Its job is to be searchable *user-provided* context, not a dump of fetched pages.
-> - **Web search + the WebURL crawler = platform-native agent tools** (already `web_search` + `scrape_webpage` on the main agent, `main_agent/tools/index.py`), always available; later exposed to developers via a platform **API key**. They are **NOT connectors and NOT pipelines**.
-> - **Connectors = MCP tools** (Type-2) **+ file/KB-input connectors** (the only Type-1 that survives). All branded natives → MCP.
-> - **Automations are the scheduling + run-history substrate.** An automation run already invokes the *full chat agent* (with the crawler/search tools) on a **cron/event** trigger and persists **run history** (`automation_runs`) — exactly the recurring-fetch + monitoring loop Pipelines were reinventing. You store the agent's *insight per run*, not raw snapshots.
-> - **`WEBCRAWLER_CONNECTOR` is retired** — the crawler is a native tool, not a connector/data-source.
->
-> **Net effect on this umbrella:** the old **Pipelines** stack (old Phases 5–7) is dropped and **Phases 4–7 are re-cast** by the canonical subplans `04`–`07` (this framing was itself superseded by the 2026-07-01 pivot above):
-> - **Phase 4 — Capabilities & Access** (`revamp/04a` + `04b`): turn the crawler + search into **typed, callable verbs** (MVP = `web.scrape` + `web.discover`; per-platform scrapers like `maps.*`/`linkedin.*` are *later, uncommitted* drop-ins) over a single **capability registry**, exposed identically through **chat / REST+API-key / MCP** doors. Ships **Product A** (stateless utility) — revenue day one. *The old `04a-connector-category` is demoted to backward-compat hygiene; the old `04b-source-discovery` is absorbed as the `web.discover` verb.*
-> - **Phase 5 — Intelligence & Timeline** (`revamp/05a` + `05b`): the `Tracker` primitive + a **3-store delta Timeline** (`tracked_entities`/`entity_current_state`/`entity_changes`, "store deltas, no change → no row"). This is the durable CI state and **the moat** — it replaces "pipelines" as the *standing concern*. Ships the **Product B** engine.
-> - **Phase 6 — Triggers** (`revamp/06`): decide **when** a Tracker refreshes. `refresh(tracker)` is trigger-agnostic; recurrence is a **CI action on the existing automations** (reuse its schedule selector + `AutomationRun` — no new scheduler, no new run table). Alert **delivery is separate** (via `app/notifications/`; automations have no delivery path).
-> - **Phase 7 — Orchestration** (`revamp/07`): a net-new `scraping` subagent (intent routing one-shot-vs-standing, verb chains, Tracker crafting, decision-grounded answering) so the whole product is reachable in plain language.
->
-> **KB stays input-only; `WEBCRAWLER_CONNECTOR` retired; crawler/search are native tools (later a platform API key).** The three correction decisions land in their real homes: **bill crawls at the capability executor** (`revamp/04a`, works uniformly across chat/automation/REST/MCP/cron — see 03c below), **diffable run history = the Timeline delta store** (`revamp/05a`, *not* `automation_runs.output`), and **"run now" = a Trigger adapter** (`revamp/06`). Phase 8 platform scrapers re-slot as a family of **individual native endpoints** — each one a capability verb → an agent tool + a dev API-key REST endpoint — added incrementally (**none, incl. Google Maps, committed for MVP**; they're examples of the pattern), not pipeline executors.
-
-## Target architecture (scraper-APIs-first, stateless)
-
-> Full end-to-end flow diagrams live in [`00b-pipeline-diagrams.md`](00b-pipeline-diagrams.md). Summary below.
-
-```mermaid
-flowchart TD
- ACQ[["Acquisition (Phase 3, shipped): WebURL crawler + proxy/stealth/captcha + 03c billing · + Maps actor (new)"]]
- REG["Capability registry (P4): web.scrape · web.discover · maps.search/place/reviews — typed verbs, cleaned AI-ready output, bill at executor"]
- ACQ --> REG
- REG --> DOORS["Access doors (P5): REST + API key · MCP server · chat tools — generated from the registry"]
- DOORS --> AGENT["scraping subagent (P7): intent routing · verb chains · 'what changed' from chat history"]
- KB[("Knowledge Base — input-only: user files/context (uploads + Drive/Dropbox/OneDrive)")] --> AGENT
- CONN["Connectors = MCP tools (BYO) + file/KB-input"] --> AGENT
- AGENT -->|one-shot| ANS["plain-language answer (nothing persists)"]
- AGENT -.->|standing need| ONG["Ongoing-Automation (P6): persistent chat re-invokes verbs; memory = chat history"]
- ONG -.-> REG
- APIKEY["Platform API key"] --> DOORS
-```
-
-
-Original (superseded) Pipelines-centric architecture
-
-```mermaid
-flowchart TD
- WS[WorkSpace] --> CONN[Connectors]
- WS --> PIPE[Pipelines]
- WS --> KB[(Knowledge Base: documents + chunks)]
- CONN --> T1[Type 1: Data Sources - pull]
- CONN --> T2[Type 2: MCP Tools - act]
- T1 --> WEB[Universal WebURL Crawler - functional]
- T1 --> PLAT[Platform connectors - coming soon]
- T1 --> UP[File Upload]
- PIPE --> RUN[PipelineRun history]
- PIPE -->|"save_to_kb + destination folder"| KB
- RUN -->|"manual or cron"| T1
- T2 --> CHAT[Chat / Automations]
- CHAT --> DELIV[Deliverables: audio/video/report/image]
- RUN -->|"read-only context"| CHAT
-```
-
-
-
-## Decisions locked
-
-- Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps.
-- Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`.
-- **Capabilities, not connectors, are the core of the product.** The crawler + search + Maps actor become typed **verbs** (`web.scrape`, `web.discover`, `maps.search/place/reviews`) in a single **capability registry** that generates the REST/MCP/chat doors identically (`04`/`05`). A capability is *call → cleaned data → bill*; no `SearchSourceConnector` row, no KB write, no schedule. The **old connector two-type taxonomy is demoted to backward-compat hygiene** (kept so existing rows/KB docs stay searchable) — the one genuinely useful piece, the **`MCP_CONNECTOR` (BYO MCP) routing-gap fix**, moves into `05` (consume-user-MCP). Surviving connectors: **file/KB-input** (Google Drive native+Composio, OneDrive, Dropbox, uploads → the input-only KB) + **BYO `MCP_CONNECTOR`** (act in chat). **`WEBCRAWLER_CONNECTOR` RETIRED** (native tool); branded natives stay `MIGRATING` (off); Obsidian/Circleback `DISABLED`. Artifacts stay in the existing `deliverables` system.
-- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as the **`web.discover` capability** (a platform-level search provider set): given a topic/competitor, suggest candidate URLs the agent can feed to `web.scrape`. Not a standalone connector type, does not index data. NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
-- **Stateless — the chat history is the memory.** No Timeline, no `Tracker`, no diff/materiality engine, no stored `entity_changes`. "What changed" is the agent reasoning over prior tool outputs already in context (`07`). Standing needs are served by a **persistent ongoing chat** that periodically re-invokes verbs (`06`, design deferred).
-- **Bill at the capability executor.** Each verb declares a `billing_unit`; the executor charges the workspace owner once per billable success via the billing service (`WebCrawlCreditService`, 03c, first provider) — so chat, ongoing automation, REST, and MCP all meter **uniformly**. `maps.*` registers a new per-place / per-page unit.
-- Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP.
-- MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are `DISABLED` — they're tagged `MIGRATING` (turned off for MVP like the other branded natives, pending the post-MVP MCP re-point).
-- ~~`Pipeline` and `PipelineRun` are new first-class tables.~~ **SUPERSEDED** — no Pipelines, and (2026-07-01) no Tracker/Timeline either. Standing needs are a **persistent ongoing chat** re-invoking verbs (`06`, deferred); no new scheduler, no new run/state table. Uploads simply populate the input-only KB (existing upload code, no pipeline wrapper).
-- ~~The chat agent gets read-only access to pipeline run history~~ → the agent answers "what changed?" by **reasoning over the chat history** (prior tool outputs already in context, `07`), not by re-deriving from run logs or a stored timeline.
-- **Web search + WebURL crawler are platform-native** (agent tools now; developer platform-API-key access later). The KB is **input-only**.
-- Deferred (post-MVP): platform scraper implementations (**Phase 8 — Platform actors**, public-data-only first; built in-house on the Phase-3 fetch core) surfaced as **native tools / API endpoints**, public pay-as-you-go API over the crawler/search, public MCP server exposing the KB. Logged-in/account-based scraping deferred beyond that.
-
-## Platform connector research list (deferred build, MVP = "coming soon")
-
-- LinkedIn — people profiles (discovery by keyword/company), company info, job listings.
-- Amazon — product (ASIN), search (keyword), pricing; reviews secondary.
-- Google — Web Search (organic SERP), AI Overviews, Maps/Local (discover by location).
-- Instagram — profiles first, then posts; discover profiles by username/keyword.
-- Zillow / Redfin — full property listings (discover by search URL/filters); Zillow price history.
-- Walmart — product, search; zipcode-localized pricing premium variant.
-- eBay — search by keyword/category; price-comparison/resale feeds.
-- Crunchbase — company info, search by keyword (B2B lead-gen / investor research).
-- TikTok / YouTube — profiles/channels, posts/videos; discover by keyword/hashtag; TikTok Shop.
-- Indeed / Glassdoor — job listings (discover by keyword in location), company reviews.
-
-## Backend phases (active — this umbrella)
-
-### Phase 1 — Rename foundation (DB) [`subplan: 01-rename-db.md`]
-
-> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Migration `170` (chains `169`, current head) does the physical rename + Zero-publication reconcile. Shipped atomically with Phase 2. As-built record + re-runnable verification live in `01-rename-db.md`. **Deploy caveat:** zero-cache replica reset (`ZERO_AUTO_RESET`) required; from-scratch `alembic upgrade head` stays pre-existing-broken (rev 23 conflict — separate baseline-squash task), only the `169→170` path is verified.
-
-- Alembic migration: rename `searchspaces` -> `workspaces`; rename `search_space_id` -> `workspace_id` on ~20 child tables; rename RBAC tables and their FKs; rename indexes/constraints (`uq_searchspace_*`, `idx_documents_search_space_id`, etc.); update Rocicorp Zero publication column lists (backend-owned `publication` definition; frontend Zero schema rename happens in the later frontend umbrella).
-- Decide transition strategy: hard cutover (simplest for MVP) vs temporary API aliases for clients.
-- Key files: `surfsense_backend/app/db.py`, `surfsense_backend/alembic/versions/` (new migration).
-
-### Phase 2 — Rename backend (code + API) [`subplan: 02-rename-backend.md`]
-
-> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Symbolic rename across `app/` + `tests/` (Phase-1 shim dropped), API consolidated to `/workspaces` (legacy `/searchspaces` · `/search-spaces` · `/search-space` all retired/404). Verified (ground-truth `git grep`, 2026-06-29): every residual `search_space`/`SEARCH_SPACE` is a documented carve-out — enum values (`ConnectionScope`/`ChatVisibility.SEARCH_SPACE`), the `'SEARCH_SPACE'` CHECK literal (now paired with `workspace_id`), Celery wire names (`delete_search_space_background`, `ai_sort_search_space`), OTel key `search_space.id`, and the `SEARCH_SPACE_FORBIDDEN` error code; `alembic/versions/` untouched except `168`+`170`. Suite: `3016 passed, 1 skipped`. As-built record in `02-rename-backend.md`. **Clients are intentionally broken** until the frontend/satellite umbrellas land (hard cutover).
-
-- Rename models/schemas/services/routes/agents/tasks identifiers: `SearchSpace*` -> `Workspace*`, `search_space_id` -> `workspace_id`.
-- Consolidate API to `/workspaces` and fix the `/searchspaces` vs `/search-spaces` inconsistency.
-- High-touch files: `routes/search_spaces_routes.py`, `routes/rbac_routes.py`, `utils/rbac.py` (`check_search_space_access`), `schemas/search_space.py`, plus `search_space_id` threading through agents/Redis keys/storage paths (`documents/{id}/...`).
-
-### Phase 3 — WebURL Crawler & Crawl Billing (backend) [`subplans: 03a–03f`]
-
-The Universal WebURL Crawler is the flagship Type-1 data source (**the moat**). This phase hardens it into an **in-house, best-effort "undetectable, captcha-bypassing" crawler** on a single framework (Scrapling): it standardizes the fetch layer, generalizes proxy support, introduces pay-as-you-go crawl credits, adds **stealth hardening** (geoip coherence, persistent profiles, headed/Xvfb, fonts, humanization, a block classifier + per-domain strategy memory), and adds **opt-in captcha solving**. All tiers plug in behind a single `FetchStrategy` seam returning `CrawlOutcome`, so callers never depend on *how* a page was fetched — that seam is what lets the moat grow (and lets a **deferred paid-unblocker tier** drop in later by config). **Strategy decision (recorded in the log):** CloakBrowser is rejected on licensing and external unblocker APIs are deferred; we hold an in-house bypass moat for ~4–6 months, then move hostile targets to a paid tier if demand/maintenance justifies it. **Logged-in/account-based bypass is out of scope** (public data only this MVP). It is broken into focused subplans:
-
-> **Sequencing within Phase 3 (critical path vs hardening).** Only **`03a` + `03b` + `03c`** are on the **MVP critical path**. The crawler (`03a` `CrawlOutcome`) + proxy provider (`03b`) + `03c` `WebCrawlCreditService` billing seam are consumed by the **capability verbs** (`04`) that back every chat / automation / REST / MCP path (the crawler is a platform-native tool — see Architecture pivot; the old "Phases 4–7 pipelines consume this" framing is superseded). **`03d` (captcha), `03e` (stealth hardening), and `03f` (test harness) are hardening/measurement that nothing downstream imports** — they tune the *same* seam behind the *same* contract, so they can land **in parallel with, or after, Phase 4** without blocking the pivot. Recommended build order: `03a → 03b → 03c` (then `04` Capabilities → `05` Access → `07` Orchestration), with `03e → 03d → 03f` slotted in whenever crawler robustness is prioritized.
-
-- **`03a-crawler-core.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `5c36cd3`; crawler moved to `app/proprietary/web_crawler/`, `impersonate="chrome"` + `solve_cloudflare=True` shipped)* — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran).
-- **`03b-proxy-expansion.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `6226012`)* — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-capability override is left as a no-op seam (`04`), not built.
-- **`03c-crawl-billing.md`** *(✅ IMPLEMENTED — `ci_mvp` @ `17bdb0682`)* — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl** (config-driven via `WEB_CRAWL_MICROS_PER_SUCCESS`, retunable with no code change), drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Billing surfaces (**reframed** by the Architecture pivot + revamp — see log): the charge moves to the **capability executor** (`04`), so **every** caller meters uniformly — chat, ongoing automation, REST, and MCP — billing the **workspace owner** via `WebCrawlCreditService`. The interactive **chat turn accumulator** becomes an *optional presentation fold* (so chat still shows the crawl line on the turn bill), **not** the charging mechanism. The original **connector/pipeline-indexer** billing branch (`webcrawler_indexer`) is now **vestigial** (WEBCRAWLER connector retired, no pipelines) — kept dormant, not deleted. The `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`) is unchanged. No DB migration (uses the existing free-form `web_crawl` usage_type).
-- **`03e-stealth-hardening.md`** — The in-house "undetectable" layer on top of Scrapling's default patchright-Chromium stealth. **Geoip coherence** (match browser `locale`/`timezone_id` to the proxy's exit geo), **fingerprint flags** (`hide_canvas`/`block_webrtc`), **persistent per-domain profiles** (`user_data_dir`), **headed execution under Xvfb**, **real fonts** in the worker image, and **DIY behavioral humanization** via `page_action` (the Chromium engine has no built-in `humanize`). Adds a **block classifier** (label Cloudflare/DataDome/Kasada/captcha/empty from the response) + **per-domain strategy memory** (Redis, no migration) so the ladder learns the known-good tier per domain. Defines — but does **not** build — the **deferred paid-unblocker `FetchStrategy`** (ZenRows/ScrapFly/Bright Data) as the config-flagged escape hatch, with its own (later) billing. Honest ceiling: defeats Cloudflare + the moderate long tail, **not** top-tier behavioral fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred tier.
-- **`03d-captcha-solving.md`** *(✅ IMPLEMENTED — `ci_mvp`; `captchatools` page_action + per-attempt `web_crawl_captcha` billing, off by default)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`, **opt-in + off by default**. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token (egressing from the **same** proxy IP as the crawl), and injects it. Scrapling already handles Cloudflare Turnstile (`03a`), and `page_action` runs **after** `solve_cloudflare`, so the tiers compose. The **billing asymmetry** is now **resolved**: a **separate per-attempt** `web_crawl_captcha` unit (solvers charge per attempt regardless of crawl success), attached via a `WebCrawlCreditService` seam in `03c`. `ErrNoBalance` stops solving (no retry loop → avoids IP bans). Requires a paid solver account.
-- **`03f-undetectability-testing.md`** *(✅ IMPLEMENTED + first baseline — `ci_mvp`, MANUAL-only, no CI gating; lives under `app/proprietary/web_crawler/testbench/`)* — A **manual scorecard harness** that drives the real Scrapling tiers against the industry-standard detection + sandbox sites (modeled on CloakBrowser's `bin/cloaktest` suite) to **quantify the free-stack ceiling** over time. Two labeled axes: **Suite S (stealth/anti-bot)** — browser-tier (`bot.sannysoft`, `bot.incolumitas`, CreepJS, `deviceandbrowserinfo`, FingerprintJS demo, reCAPTCHA-v3 score, `fingerprint-scan`/Castle.js, `browserscan`, **`cloudflare-challenge`** exercising `solve_cloudflare`, **`iphey`** geoip-coherence), HTTP/TLS-tier JA3/JA4 parity (`tls.peet.ws`, **informational** not a gate), and proxy/leak checks (`httpbin/ip`, WebRTC/DNS); **Suite E (extraction correctness)** — toscrape/scrapethissite sandboxes for the HTTP vs JS (DynamicFetcher) tiers. **Every detection site is auto-graded from its real DOM verdict** (parsers written against captured dumps; `INFO` reserved for TLS/IP/manual rows). Reuses `03d`'s `page_action`+closure-cell mechanism. Adopts CloakBrowser's bars as **aspirational** (sannysoft 0 fails, CreepJS ≤30%, reCAPTCHA ≥0.7) while recording **our actual numbers as the baseline** (the whole `results/` tree is **gitignored**, run-local). **First baseline (2026-06-30, headless, rotating residential, captcha OFF): Suite S 6 PASS / 4 FAIL** — PASS sannysoft, deviceandbrowserinfo, reCAPTCHA-v3 (0.9), BrowserScan, fingerprint-scan (35/100), cloudflare-challenge; FAIL CreepJS (`hasHeadlessWorkerUA` worker-UA leak → `03e` Slice-B candidate), FingerprintJS Pro (commercial ceiling), iphey ("Unreliable" → expected geoip-coherence fix), incolumitas (legacy `fpscanner WEBDRIVER` only). The scorecard is the documented **trigger** for flipping `03e`'s deferred paid-unblocker tier.
-
-### Phase 4 — Capabilities (backend) [`subplan: 04-capabilities.md`]
-
-> **Canonical subplan:** [`04-capabilities.md`](04-capabilities.md). **Build first** — every later phase calls the registry.
-
-- Turn Acquisition (crawler + search + Maps actor) into a **small set of typed, callable verbs** over a single **capability registry** — `web.scrape(urls[])`, `web.discover(query, top_k)`, `maps.search/place/reviews`. Each verb is a **direct async call that returns cleaned, AI-ready data** (no job store, no envelope, no polling). `maps.*` returns typed structured objects, not markdown. Each declares a **`billing_unit`**; charging is delegated to the billing service (03c first provider) at the executor so every door meters uniformly. New Apache-2 package `app/capabilities/`; Maps extractor stays proprietary.
-
-**Locked model (MVP):**
-
-- **Capabilities replace connectors as data sources.** `web.*` / `maps.*` are *call → cleaned data → bill* verbs — no `SearchSourceConnector` row, no KB write, no schedule.
-- **Surviving connectors = file/KB-input + BYO MCP.** File/KB-input (Google Drive native+Composio, OneDrive, Dropbox, uploads) put the user's personal files into the **input-only** KB. BYO `MCP_CONNECTOR` (act in chat) is the only functional Type-2; its routing-gap fix lands in `05`. **`WEBCRAWLER_CONNECTOR` RETIRED.** The old connector two-type taxonomy (`04a-connector-category`) is **demoted to backward-compat hygiene**.
-- **Deprecated (`MIGRATING`, off for MVP):** every branded native — indexers (Notion, GitHub, Confluence, BookStack, Elasticsearch) + act-only (Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar, Luma, + Composio Gmail/Calendar). Existing rows + already-indexed KB docs stay **searchable**; new-create/`/index`/periodic/subagents off. Real MCP migration is post-MVP.
-- **Disabled (`DISABLED`):** Obsidian (plugin push) and Circleback (meeting webhook).
-- Web search: **drop all 5 search-API `connector_type` values** (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`); survivors (SearXNG/Linkup/Baidu) become the `web.discover` provider set keyed by **platform env** (Linkup/Baidu keys move off per-connector `config`). Destructive migration deletes the 5 types' rows. (Absorbs the old `04b-source-discovery`.)
-- The **Google Maps actor is net-new**, proprietary, built as a separate effort; `maps.*` are contracts against it. Other "coming soon" platforms (LinkedIn, Amazon, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor) re-slot as later namespaces, not Type-1 KB connectors.
-- Frontend connector/capability UI restructure is DEFERRED (frontend umbrella).
-
-### Phase 5 — Access (backend) [`subplan: 05-access.md`]
-
-> **Canonical subplan:** [`05-access.md`](05-access.md). **Build after `04`.** Together `04 + 05` ship the scraper-API product — revenue day one.
-
-- Expose the registry through **three generated doors** — **REST + API key** (public day one), an **MCP server** (fast-follow), and **chat tools** — each the same thin adapter (`parse → validate → authn/authz → 03c meter-gate → same executor → serialize → cleaned data`). The executor returns directly; there is **no envelope, no `deliverable_wait` polling**. Async results (e.g. the `06` ongoing mode) reach the client via **write-then-sync / stream** (existing SSE or a Zero-published table).
-- **Natural language is the human surface** (raw verbs only on REST/MCP); the chat door classifies **one-shot** vs **keep-watching** and hands keep-watching to `06`. Two MCP directions kept distinct: **serve** our verbs as an MCP server (new door) vs **consume** the user's BYO `MCP_CONNECTOR` (the old-`04a` routing-gap fix lands here).
-
-### Phase 6 — Ongoing-Automation (backend) [`subplan: 06-ongoing-automation.md`]
-
-> **Canonical subplan:** [`06-ongoing-automation.md`](06-ongoing-automation.md) — **shipped.** A chat watch is an `Automation` bound to the current chat (`schedule` + `chat_message`); the durable checkpointer is the memory. **Depends on `04`** (the verbs it re-invokes) and `05` (the chat surface + delivery channel).
-
-- Support **"keep watching"** with no stateful storage: a **persistent, ongoing chat** where the agent periodically re-invokes scraper verbs and drops results into the session; the agent reports "what's new" by reading the chat history. Open questions (resolved together): periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, loop owner, stop/cost controls.
-
-### Phase 7 — Orchestration (backend) [`subplan: 07-orchestration.md`]
-
-> **Canonical subplan:** [`07-orchestration.md`](07-orchestration.md) — the human-facing brain. **Shipped for the web verbs** (atop `04`/`05`/`06`). We plug into the shipped multi-agent runtime; we don't rebuild it.
-
-- Ship the builtin **`scraping`** subagent (peer to `research`/`deliverables`): **intent routing** (one-shot vs keep-watching, block on ambiguous cadence), **verb composition** (`web.discover → web.scrape`; `maps.*` chains deferred until the Maps actor exists), and **"what changed" from the chat history** (re-invoke a verb, compare against prior tool outputs already in context — no timeline, no diff store).
-- Toolset: registry-backed capability verbs + `start_watch`/`stop_watch`/`refresh_watch` bound to the current chat (`06`). Tools follow the shipped `scrape_webpage` shape (executor + door + 03c billing), **direct-return** (no `deliverable_wait`).
-
-### (Future) Platform API-key + public MCP
-
-Platform **API-key** access to the capability verbs + a public **MCP server** exposing the input-only KB — post-MVP (see Phase 8 + Deferred).
-
-### Phase 8 — Platform actors (FUTURE — post-MVP, public data only) [`subplan: TBD`]
-
-NOT planned in this umbrella; recorded so the Phase-3 architecture stays aimed at it. Google Maps is the **committed MVP actor** (a `maps.*` namespace, Phase 4). Beyond it, more **platform scrapers** — a family of **individual scraping endpoints** (LinkedIn public profiles/companies, Amazon products, etc. — see "Platform connector research list"; **examples, none committed**) — layer **on top** of the hardened fetch core once it's solid. **Each is simply another capability verb** (`04`), so registering one automatically yields (a) an **agent tool** and (b) a **dev-callable REST endpoint behind the platform API key** — same executor, same billing, no new machinery. Each is a per-platform structured extractor built in-house "Apify-style" on the `03a` crawler core (proxies + `03e` hardening + `03d` captcha). Billed per call via `WebCrawlCreditService`.
-
-- **Public data only** at first — discovery/extraction of publicly visible pages. **Logged-in/account-based bypass is explicitly deferred** beyond Phase 8's first cut; it needs sticky/static proxies + credential management (`03b` static-proxy hand-off) and is the higher-risk, later workstream.
-- The **deferred paid-unblocker tier** (`03e §8`) is the fallback for any platform whose anti-bot exceeds the in-house ceiling.
-
-## Deferred — Frontend & client phases (separate umbrella, planned LATER)
-
-These are recorded for continuity but are NOT planned in this umbrella. They start once the backend phases above are working.
-
-- Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename.
-- Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs.
-- Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types.
-- Scraper-API + chat + positioning UI (**replaces the superseded "Pipelines UI"**): the developer surface for the scraper endpoints (API keys, usage/billing), the chat experience (one-shot answers + "keep watching" ongoing chats), `web.discover` source-suggestion UX, input-only KB framing (uploads + file connectors), MCP connector management, "coming soon" platform cards.
-
-## Open items to confirm during subplanning
-
-- ~~Rename transition: hard cutover vs temporary API aliases~~ RESOLVED: HARD CUTOVER (see resolved log + 02-rename-backend.md). The frontend is rebuilt against the corrected backend in its own umbrella; backend is verified via tests/OpenAPI, not the old UI.
-- ~~Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP.~~ **MOOT** (Architecture correction — no pipelines). Connector periodic indexing stays as-is for file sources; there is no pipeline scheduler to coexist with.
-- ~~Chat agent run-history access: tool vs middleware injection vs both (default: tool).~~ **MOOT** (Architecture correction — no pipeline runs). The agent's recurring work is automations, whose history is already persisted + queryable (`automation_runs`).
-- ~~Type-2 MCP migration depth~~ RESOLVED (Phase 4): branded natives are tagged `MIGRATING` and turned OFF for MVP (not re-pointed to MCP yet); only the generic `MCP_CONNECTOR` is a functional Type-2. Real MCP re-pointing is post-MVP.
-- **Revamp open items (decide during subplanning):** (1) the **Ongoing-Automation mechanism** (`06`, deferred) — periodicity driver, delivery channel (SSE vs Zero-published table), context-window limit, stop/cost controls; (2) how large `web.scrape` / `maps.reviews` responses are bounded/streamed (`04`); (3) is `web.discover` metered or free (`04`); (4) MCP-serve auth depth — OAuth 2.1 vs bearer (`05`); (5) public REST rate-limiting / abuse posture (`05`).
-
-## Resolved decisions log
-
-- **PIVOT (2026-07-01) — scraper-APIs-first, stateless, chat-native; Tracker/Timeline/Triggers dropped.** The product is the **scraper capabilities** (`web.*` + `maps.*` typed verbs → cleaned, AI-ready data), exposed as **REST + API key / MCP / chat** generated from one registry — premium, OSS, self-hostable, the revenue driver. **Stateless:** no Timeline (3-store delta), no `Tracker`, no diff/materiality engine, no stored `entity_changes`; memory = the **chat history** (the agent reasons over prior tool outputs for "what changed"). **Direct calls:** capabilities return their result directly — no job store, no `completed | pending + job_id` envelope, no `deliverable_wait` polling; async delivery is **write-then-sync / stream**. **Automation = a persistent ongoing chat** re-invoking verbs (replaces `Triggers`; mechanism **design-deferred**). KB stays **input-only**. Phase re-cast: **04 Capabilities · 05 Access · 06 Ongoing-Automation (deferred) · 07 Orchestration** (`scraping`). Subplans `05a-timeline`/`05b-intelligence`/`06-triggers` **removed** (git history); `04a`/`04b` folded into `04`/`05`. This **supersedes** the 2026-06-30 revamp's Tracker/Timeline/Triggers decisions below.
-- **REVAMP ADOPTED as canonical (2026-06-30) — Phases 4–7 re-cast; `revamp phases 4-7/` is the source of truth.** A principal-engineer review found the flat `04a–07` files and my earlier "Phase 5′" framing had been overtaken by a more complete engineer draft (then in a `revamp phases 4-7/` folder, since flattened into `04`–`07`). We adopt the revamp: **Phase 4 = Capabilities & Access** (typed verbs + generated doors; old `04a-connector-category` demoted to hygiene, old `04b-source-discovery` absorbed as `web.discover`), **Phase 5 = Intelligence & Timeline** (the `Tracker` + 3-store delta Timeline — the moat), **Phase 6 = Triggers** (reuse automations via a CI action), **Phase 7 = Orchestration** (`scraping`). The three architecture-correction decisions land in their real homes and one is revised: (1) **bill at the capability executor** — a code-verified gap review confirmed `run_agent_task` (automation `agent_task`) sets up **no turn accumulator**, so today `scrape_webpage` bills nothing under automations; billing moves to the executor so chat/automation/REST/MCP/cron meter uniformly (chat turn accumulator becomes optional presentation). (2) **"diffable run history" = the Timeline delta store** (`revamp/05a`), **superseding** the earlier "structured `automation_runs.output`" decision. (3) **"run now" = a Trigger adapter** (`revamp/06`). The flat `04a/04b/05/06/07` files are retained with redirect banners; **`revamp/` filenames win** on any number collision. Build order: `04a → 04b → 05a → 05b → 06 → 07`.
-- **ARCHITECTURE CORRECTION (2026-06-30) — Pipelines dropped; automations + input-only KB adopted.** The "sync into KB → operate over it" paradigm was judged wrong for CI. Verified against `references/opencode` (live tool-fetched context + persisted sessions, no scraped corpus; `tool/websearch.ts` live-crawls at query time). Decisions: (1) **KB is input-only** — user's personal files/context (uploads + Drive/Dropbox/OneDrive); nothing scrapes into it. (2) **Web search + WebURL crawler are platform-native tools** (already `web_search`/`scrape_webpage`), later a developer platform-API-key surface — NOT connectors, NOT pipelines. (3) **Connectors = MCP tools + file/KB-input connectors**; all branded natives → MCP. (4) **Automations are the scheduling + run-history substrate** (an `agent_task` run already invokes the full chat agent with the crawler/search tools on cron/event and persists `automation_runs`). (5) **`WEBCRAWLER_CONNECTOR` RETIRED.** Consequences: **Phases 5/6/7 SUPERSEDED** (docs kept with banners); **Phase 4a reframed** (drop `is_pipeline_eligible`; Type-1 = file/KB-input only; WEBCRAWLER retired); **Phase 4b stands**; new **Phase 5′** automation-enhancement workstream (bill automation-run crawls; structured `run.output`; wire "run now"); **Phase 8** actors re-slot as native tools/API. `03c` connector-indexer billing branch becomes **vestigial** (billing lives on chat-turn / automation-run / API paths).
-- ~~Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines)~~ — still repurposed as a source-discovery helper (04b), but "for pipelines" → "for the user / the native crawler tool" (pipelines dropped); not a standalone connector type.
-- Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).
-- Obsidian + Circleback: disabled for MVP.
-- MCP-availability audit: BookStack, Elasticsearch, Luma all have MCP available -> eligible for Type-2 (so deprecated as `MIGRATING`, not `DISABLED`). For MVP they are turned off pending the post-MVP MCP re-point, like the other branded natives.
-- Phase 4 connector taxonomy: modeled as a STATIC code registry (`connector_type` -> category/availability), NOT a DB column — no migration in 04a; `is_indexable` is KEPT (orthogonal). Only the generic `MCP_CONNECTOR` is a functional Type-2 for MVP; all branded natives (indexers + act-only, incl. Composio Gmail/Calendar) are `MIGRATING` (new-create blocked, `/index`+periodic+their subagents off, existing KB docs stay searchable). The `MCP_CONNECTOR` subagent routing-map gap (`constants.py`) is fixed in 04a.
-- Phase 4 search APIs: all 5 enum values dropped (`SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API`) in 04b. Survivors (SearXNG/Linkup/Baidu) become PLATFORM providers keyed by env (Linkup/Baidu keys move from per-connector `config` to env — app-wide, not per-workspace). 04b carries a destructive migration deleting the 5 connector types' rows.
-- Phase 4 structure: split into 04a (taxonomy/gating/MCP-fix, no migration) and 04b (search repurposing + source-discovery endpoint, with migration); intended order 04a -> 04b (both orders safe).
-- Rename transition policy: HARD CUTOVER of the external API (paths + JSON field names) in Phase 2 — no backward-compat aliases. Rationale: the frontend is (re)built against the corrected backend later, so there is no old client to keep alive; backend correctness is verified via the test suite + OpenAPI rather than the existing UI.
-- Crawler code location & licensing boundary (decided during 03a impl): the WebURL crawler engine — and future Phase-8 platform actors — live under `surfsense_backend/app/proprietary/`, a **non-Apache-2 license boundary** (its own `LICENSE`, currently an all-rights-reserved placeholder; the repo root stays Apache-2). 03a's `WebCrawlerConnector` / `CrawlOutcome` / `CrawlOutcomeStatus` moved to `app/proprietary/web_crawler/` (public API re-exported from its `__init__`); the 3 Apache-2 callers (webcrawler indexer + both chat `scrape_webpage` tools) import `from app.proprietary.web_crawler import ...`. Rule: everything under `app/proprietary/**` is non-Apache-2; Apache-2 code may import *from* it but not move *into* it. Rationale: keep the moat under a clearly-bounded, swappable license.
-- Proxy code placement (decided during 03b): the proxy provider package (`app/utils/proxy/` — base/registry/`anonymous_proxies`/`CustomProxyProvider`/rotation) **stays Apache-2 shared infra**, NOT proprietary. Rationale: it's consumed by Apache-2 features unrelated to the moat (YouTube transcript route + indexer, chat tools' YouTube branch), and `CustomProxyProvider` is a thin wrapper over Scrapling's *public* `ProxyRotator`. Only the crawl-ladder-coupled rotation-retry (`app/proprietary/web_crawler/connector.py::_run_tier_with_proxy_retry`) lives under the boundary. **Boundary test:** code goes in `app/proprietary/` only if used *exclusively* by the moat (applies to 03e's geoip/sticky-proxy hardening too — the bypass-specific tuning is proprietary; the generic provider plumbing is not).
-- WebURL Crawler framework: STANDARDIZE on Scrapling; **remove Firecrawl entirely** (no other scraping frameworks now or planned). Scrapling's `StealthyFetcher` (patchright-Chromium as of 0.4.9 — **not** Camoufox) handles Cloudflare; `03e` stealth-hardening minimizes challenges; captcha-tools (`03d`) covers the rest. All fetch tiers sit behind a `FetchStrategy` seam returning `CrawlOutcome` (callers never depend on the tier).
-- Crawl billing: reuse the existing credit wallet (`credit_micros_balance`) with a new `web_crawl` usage_type. Price: **$1 / 1000 successful requests** (1000 micro-USD per success). Connector/pipeline crawls bill the **workspace owner**; chat scrapes fold their crawl cost into the already-billed chat turn. Gated by `WEB_CRAWL_CREDIT_BILLING_ENABLED` (off for self-hosted); no DB migration required.
-- Billable unit: one unit per URL that returns usable extracted content, regardless of how many internal fallback tiers were attempted (not per HTTP fetch, not per URL-processed).
-- Captcha solving (captcha-tools): **ACTIVE** (no longer deferred) — sequenced last in Phase 3 (`03d`), **after** `03e` hardening, **opt-in + off by default**. Cloudflare stays in-framework (`03a`); reCAPTCHA/hCaptcha/image use `captchatools`. **Billing asymmetry RESOLVED → option (a): a separate per-attempt `web_crawl_captcha` unit** (`WEB_CRAWL_CAPTCHA_*` knobs on `WebCrawlCreditService`), since solvers charge per attempt regardless of crawl success. `ErrNoBalance` halts solving (no retry-loop IP bans).
-- Crawler stealth strategy (the moat): **CloakBrowser REJECTED** (source-patched Chromium binary requires an OEM/SaaS license incompatible with our model). **External unblocker APIs DEFERRED** (ZenRows/ScrapFly/Bright Data) — pre-wired as a config-flagged `FetchStrategy` (`03e §8`) but not built. Plan: maintain an **in-house bypass moat for ~4–6 months** (Scrapling stealth + residential proxies + `03e` hardening + `03d` captcha), then move hostile/top-tier-fingerprinted targets (DataDome/Kasada/reCAPTCHA-Enterprise) to a paid tier if demand/maintenance justifies it. Realistic ceiling acknowledged: in-house beats Cloudflare + the moderate long tail, not top-tier behavioral fingerprinting.
-- Authenticated/logged-in scraping: **OUT OF SCOPE this MVP** (public data only). Sticky/static proxies + credential management are deferred and paired with the future platform actors (`03b` static-proxy hand-off + Phase 8).
-- Phase 3 stealth-hardening subplan `03e` ADDED: geoip locale/tz coherence, `hide_canvas`/`block_webrtc`, persistent per-domain profiles, headed+Xvfb, fonts, DIY humanization (`page_action`; Chromium engine has no built-in `humanize`), a block classifier, and per-domain strategy memory (Redis, no migration).
-- Phase 3 test-harness subplan `03f` ADDED: **manual-only** (no CI/automated gating now) undetectability + extraction scorecard, modeled on CloakBrowser's `bin/cloaktest`. Two labeled axes (Suite S stealth + Suite E extraction) so they scale independently. Drives the **real** Scrapling tiers (browser + curl_cffi HTTP/TLS), reuses `03d`'s `page_action`+closure-cell for JS-object verdicts. **TLS JA3/JA4 parity = informational axis, not a hard gate.** Adopt CloakBrowser bars as aspirational; record our actual free-stack numbers as the committed baseline. The scorecard is the documented evidence/trigger for flipping `03e`'s deferred paid-unblocker tier.
-- Licensing placement of 03e/03f code (decided during 03f impl, applying the §boundary test): the **stealth kwargs builder + geoip coherence** (`03e` bypass tuning) live **proprietary** at `app/proprietary/web_crawler/stealth.py`; the **block classifier** (passive telemetry, public markers) stays **Apache-2** at `app/utils/crawl/classifier.py` (direct analog of the captcha split: proprietary `captcha.py` logic + Apache-2 `app/utils/captcha/` config). The **03f scorecard harness** moved whole to `app/proprietary/web_crawler/testbench/` (run `python -m app.proprietary.web_crawler.testbench`) — it's the moat's measurement tool and can't be cleanly half-moved (a proprietary Suite S would back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). The `scripts/e2e_phase3_crawl_billing.py` billing e2e stays in `scripts/` (Apache-2) since it exercises billing, not the stealth moat.
-- Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7.
-- **[SUPERSEDED — Architecture correction 2026-06-30; no pipelines]** Phase 5 pipelines data model: two new tables `pipelines` (mutable) + `pipeline_runs` (append-only), modeled on `automations`/`automation_runs`; ORM lives in `db.py` next to connectors/folders. `connector_id` nullable (NULL = Phase-7 Uploads), eligibility enforced at create via 04a's `is_pipeline_eligible`. Schedule = `schedule_cron` + `schedule_timezone` (default UTC) + `next_scheduled_at` (cron, matching automations). `pipeline_runs` pre-includes `charged_micros`/`crawls_*`/`result_blob_key` so Phase 6 needs no extra migration. Both tables published to Zero **full-row** (like folders/connectors). Routes reuse `CONNECTORS_*` permissions. Phase 5 ships the data model + API surface only; the `/run` endpoint enqueues a Phase-6 task stub.
-- **[SUPERSEDED — Architecture correction 2026-06-30; uploads just populate the input-only KB]** Phase 7 uploads-as-pipeline: a **singleton "Uploads" pipeline** per workspace (`connector_id NULL`, `save_to_kb=true`), lazily get-or-created (race-safe via a partial unique index `ON pipelines(workspace_id) WHERE connector_id IS NULL`). Each `fileupload`/`folder-upload` request writes a **terminal audit `PipelineRun(trigger=upload, status=succeeded, documents_indexed=)`** — uploads are **route-recorded, not engine-executed** (Phase 6 fails NULL-connector runs by design; existing upload code stays the executor). Best-effort via an **inner** try/except (never 5xx the upload — the route's outer handler would otherwise 500 an already-committed upload). No crawl billing (uploads aren't crawls; `charged_micros` NULL). Per-file ETL truth stays on `Document.status`; accurate roll-up needs the deferred `documents.pipeline_run_id` provenance. Connector `save_to_kb` default stays `False` (opt-in for connectors, mandatory for uploads). Phase 7 also **guards Phase-5's generic CRUD** against the system Uploads pipeline: `POST /pipelines` rejects `connector_id=None` (supersedes Phase 5's permissive create), `/run` and schedule-`PUT` reject NULL-connector pipelines, and Phase 6's scheduler `_claim_due` filters `connector_id IS NOT NULL` as a backstop (so the Uploads pipeline can never be manually-run or scheduled into perpetually-failing runs). See `07-upload-pipeline-kb.md`.
-- **[SUPERSEDED — Architecture correction 2026-06-30; automations are the run engine]** Phase 6 pipeline execution: run engine mirrors **automations** (thin Celery `run_pipeline(run_id)` → `execute_pipeline_run`; PENDING-gated, idempotent terminal no-op; `pending→running→succeeded/failed` with timing/counts/error). MVP executor = **WebURL crawler only** (other types fail cleanly). `save_to_kb=true` reuses `index_crawled_urls` extended with a `folder_id` param (lands in the destination folder); `save_to_kb=false` runs a **fetch-only** loop and persists one JSON blob via `file_storage` (`result_blob_key`). **Crawl billing is owned by the run engine** for the pipeline path (pre-check on `len(urls)` + charge `crawls_succeeded` + idempotent `charged_micros`), calling the crawler with a new `bill=False` seam (the connector `/index`+periodic paths keep `03c`'s in-indexer `bill=True`) — so non-KB runs are billed identically. Scheduler = a `pipeline_schedule_select` Beat tick modeled on the automations cron **selector** (cron + `FOR UPDATE SKIP LOCKED` + self-heal, using the existing `croniter` util), plus the **de-dup guard** (a pipeline over a connector disables that connector's `periodic_indexing_enabled`). Chat context = the `get_pipeline_runs` tool. Carries a small additive `05` amendment: a `schedule_timezone` column (cron util needs a tz). See `06-pipelines-exec.md`.
-
-## Subplan index (backend)
-
-| Phase | Subplan file | Status |
-|-------|--------------|--------|
-| 1 | `01-rename-db.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562); migration `170`) |
-| 2 | `02-rename-backend.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562)) |
-| 3 | `03a-crawler-core.md` | **IMPLEMENTED** (`ci_mvp` @ `5c36cd3`) — Firecrawl removed, Scrapling-only 3-tier `CrawlOutcome`, crawler relocated to `app/proprietary/web_crawler/` |
-| 3 | `03b-proxy-expansion.md` | **IMPLEMENTED** (`ci_mvp` @ `6226012`) — `CustomProxyProvider` (BYO single/pool) + registry + bounded rotation-retry |
-| 3 | `03c-crawl-billing.md` | **IMPLEMENTED** (`ci_mvp` @ `17bdb0682`) — `WebCrawlCreditService` (config-driven price) + indexer wiring + chat-turn fold; functional e2e green |
-| 3 | `03e-stealth-hardening.md` | **Slice A IMPLEMENTED** (`ci_mvp`) — stealth kwargs builder + geoip coherence (**proprietary** `app/proprietary/web_crawler/stealth.py`) + additive block classifier (Apache-2 `app/utils/crawl/`; `CrawlOutcome.block_type`, incl. static-tier 4xx) + Xvfb/fonts in image; Slices B/C deferred (WebGL spoof, humanization, persistent profiles, strategy memory, paid-unblocker) |
-| 3 | `03d-captcha-solving.md` | **IMPLEMENTED** (`ci_mvp`) — `captchatools` page_action (proprietary) + Apache-2 config + per-attempt `web_crawl_captcha` billing; off by default |
-| 3 | `03f-undetectability-testing.md` | **IMPLEMENTED** (`ci_mvp`) — manual scorecard under the **proprietary boundary** at `app/proprietary/web_crawler/testbench/` (`python -m app.proprietary.web_crawler.testbench`); Suite S (stealth, shipped builder) + Suite E (extraction via real `crawl_url`) + scorecard JSON/MD baseline diff |
-| — | `00b-pipeline-diagrams.md` | end-to-end flow diagrams (companion to this umbrella) |
-| 4 | `04-capabilities.md` | **IMPLEMENTED (web verbs)** — capability registry + `web.scrape`/`web.discover` → cleaned data, bill at executor (top_k bounds, max_length truncation, per-attempt captcha metering). `maps.*` deferred (Maps actor) |
-| 5 | `05-access.md` | **IMPLEMENTED (REST + chat doors)** — generated REST/API-key + chat tools from the registry. MCP door + BYO-MCP routing fix deferred (MCP server) |
-| 6 | `06-ongoing-automation.md` | **IMPLEMENTED** — chat-native "keep watching": `Automation` bound to the chat (`schedule` + `chat_message`), worker-safe durable checkpointer, watch tools + REST controls |
-| 7 | `07-orchestration.md` | **IMPLEMENTED (web verbs)** — `scraping` subagent; intent routing, verb composition, "what changed" from chat history |
-
-Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases").
diff --git a/plans/backend/00b-pipeline-diagrams.md b/plans/backend/00b-pipeline-diagrams.md
deleted file mode 100644
index b199fbf1b..000000000
--- a/plans/backend/00b-pipeline-diagrams.md
+++ /dev/null
@@ -1,109 +0,0 @@
-# Pipeline diagrams — end-to-end (scraper-APIs-first, stateless)
-
-> Visual companion to `00-umbrella-plan.md`.
-> Phase refs: `04` Capabilities · `05` Access · `06` Ongoing-Automation (design deferred) · `07` Orchestration.
-
-## 1. The shape — scraper APIs + a chat agent (no state)
-
-```mermaid
-flowchart LR
- subgraph FIXED["FIXED · Phases 1-3 (shipped)"]
- ACQ["Acquisition
crawler · proxy · stealth · captcha"]
- MET["Metering · 03c"]
- ID["Identity / API keys"]
- end
-
- subgraph SCOPE["OUR SCOPE · Phase 04 -> 07"]
- CAP["04 Capabilities
web.* · maps.*
cleaned, AI-ready output"]
- ACC["05 Access
REST · MCP · chat doors"]
- ORC["07 Orchestration
scraping subagent"]
- ONG["06 Ongoing-Automation
(keep-watching)"]
- end
-
- ACQ --> CAP
- MET --> CAP
- CAP --> ACC
- ID --> ACC
- ACC --> ORC
- ORC -. "standing need" .-> ONG
- ONG -. "re-invokes verbs" .-> CAP
-
- classDef a fill:#22314f,stroke:#5b7fbf,color:#e6edf7;
- classDef b fill:#1f3a2e,stroke:#4f9d76,color:#e6f7ee;
- classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
- class CAP,ACC,ORC a;
- class ACQ,MET,ID b;
- class ONG d;
-```
-
-Memory for "what changed" = the chat history.
-
-## 2. Doors — one registry, three surfaces (generated)
-
-```mermaid
-flowchart TD
- REG["04 Capability registry
web.scrape · web.discover · maps.search/place/reviews"]
- REG --> R["REST + API keys
(public · OSS self-host · revenue)"]
- REG --> M["MCP server
(external agents · fast-follow)"]
- REG --> C["Chat tools
(in-app agent, 07)"]
- R --> EX["same executor · direct return · 03c billing"]
- M --> EX
- C --> EX
-```
-
-## 3. One-shot scrape (the core path) — direct call, no polling
-
-```mermaid
-sequenceDiagram
- autonumber
- participant U as User / Dev
- participant D as Door 05 (REST · MCP · chat)
- participant EX as Verb executor 04
- participant ACQ as Acquisition / Maps actor
- participant BILL as Billing (03c)
-
- U->>D: intent (chat) OR raw verb call (REST/MCP)
- D->>D: validate input_schema · authn/authz · meter-gate
- D->>EX: call executor (same for all doors)
- EX->>ACQ: crawl_url / maps / search
- ACQ-->>EX: cleaned, AI-ready data
- EX->>BILL: charge billing_unit per success
- EX-->>D: results (returned directly)
- D-->>U: plain-language answer (chat) / JSON (REST/MCP)
- Note over U,BILL: Stateless — nothing persists.
-```
-
-## 4. Intent fork (in the agent, 07)
-
-```mermaid
-flowchart TD
- U(["User speaks in natural language"]) --> R{"Intent router
(scraping prompt, 07)"}
- R -->|"find / compare / what is / right now"| A["ONE-SHOT
compose verbs, answer (section 3)"]
- R -->|"watch / track / notify me / over time"| B["KEEP-WATCHING
start_watch -> Ongoing-Automation (06)"]
- R -->|ambiguous cadence| Q["Block: ask for the missing cadence / timezone"]
- Q -->|once| A
- Q -->|keep watching| B
-```
-
-## 5. "What changed" — chat history is the memory
-
-```mermaid
-flowchart LR
- P["Prior tool outputs
(already in chat context)"] --> AG["scraping subagent"]
- NEW["Fresh verb call (04)"] --> AG
- AG --> ANS["'Here's what's new since last time'
(reasoned from chat history)"]
-```
-
-## 6. Ongoing-Automation (06) — intent only, mechanism deferred
-
-```mermaid
-flowchart LR
- W["'Keep watching X'"] --> LOOP["Persistent ongoing chat
periodically re-invokes verbs (04)"]
- LOOP --> OUT["results delivered into the session
(write-then-sync / stream)"]
- OUT --> MEM["chat history = memory
agent reports what's new"]
- LOOP -.->|OPEN| Q["periodicity driver? · delivery channel? · context-window limit?"]
- classDef d fill:#3a2f1f,stroke:#bf975b,color:#f7efe6;
- class W,LOOP,OUT,MEM d;
-```
-
-> Section 6 is a **placeholder** — the periodic mechanism is designed separately (see `06-ongoing-automation.md`).
diff --git a/plans/backend/01-rename-db.md b/plans/backend/01-rename-db.md
deleted file mode 100644
index 2f6e77401..000000000
--- a/plans/backend/01-rename-db.md
+++ /dev/null
@@ -1,213 +0,0 @@
-# Subplan 01 — Rename foundation (DB)
-
-Part of [00-umbrella-plan.md](00-umbrella-plan.md), Phase 1. Backend only (`surfsense_backend`).
-
-> **Status: SHIPPED** · as of 2026-06-27 · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)
-> Last commit of this phase: `49d3001fb` (DB rename through migration 170). Phase 1 + Phase 2 shipped as one atomic PR.
-> The sections below are the **original design/rationale**; the as-built state + how to re-verify live in the [Implementation record](#implementation-record-as-built) at the bottom. Ground truth for files/commits is the PR, not this doc.
-
-## Goal
-
-Physically rename the `SearchSpace` schema to `WorkSpace` in PostgreSQL via a single Alembic migration, and update the ORM's physical mapping + the Zero publication so the app keeps booting and running — WITHOUT yet touching the ~250 backend files that reference the `search_space_id` Python attribute (that symbolic rename is Phase 2).
-
-Canonical target names:
-
-- Table `searchspaces` -> `workspaces`
-- Tables `search_space_roles` / `search_space_memberships` / `search_space_invites` -> `workspace_roles` / `workspace_memberships` / `workspace_invites`
-- Column `search_space_id` -> `workspace_id` (all child tables); `external_chat_accounts.owner_search_space_id` -> `owner_workspace_id`
-- Named constraints/indexes containing `searchspace` -> `workspace` (see inventory)
-
-## The central coupling decision (read first)
-
-A physical column rename instantly breaks every line of code that reads the old name. Today that is large:
-
-- `search_space_id` / `searchspaces` appear in ~250 backend files outside `db.py` (Grep over `surfsense_backend/app`, excluding `db.py`). Almost all are Python ORM attribute reads like `Document.search_space_id` or `.search_space_id`.
-
-To keep Phase 1 self-contained and low-risk, decouple the PHYSICAL rename from the SYMBOLIC rename using an explicit column-name mapping shim:
-
-- Keep the Python attribute name `search_space_id`, but bind it to the new physical column:
- - Today: `search_space_id = Column(Integer, ForeignKey("searchspaces.id", ondelete="CASCADE"), ...)` ([surfsense_backend/app/db.py](surfsense_backend/app/db.py) line 1382 etc.)
- - Phase 1: `search_space_id = Column("workspace_id", Integer, ForeignKey("workspaces.id", ondelete="CASCADE"), ...)`
-- Result: the physical DB is fully renamed, the ORM maps the unchanged Python attribute to the new column, and the ~250 callers keep working untouched.
-- Phase 2 then renames the Python attribute (`search_space_id` -> `workspace_id`), the class (`SearchSpace` -> `Workspace`), schemas, routes/API, and drops the explicit `"workspace_id"` first-arg shim.
-
-CONFIRMED: ship the shim approach. (Alternative big-bang rename rejected as too risky.)
-
-## Current-state inventory (all citations from code)
-
-### The table itself
-
-- `class SearchSpace(... __tablename__ = "searchspaces")` — [surfsense_backend/app/db.py](surfsense_backend/app/db.py) line 1688.
-
-### Child tables with a `search_space_id` FK to `searchspaces.id`
-
-All declared `ForeignKey("searchspaces.id", ondelete="CASCADE")` unless noted. Locations in [surfsense_backend/app/db.py](surfsense_backend/app/db.py):
-
-- `new_chat_threads` (605), `external_chat_bindings` (897), `token_usage` (1113), `folders` (1318), `documents` (1382), `video_presentations` (1506), `reports` (1533), `connections` (1561, nullable), `image_generations` (1672), `search_source_connectors` (1878), `logs` (1905), `search_space_roles` (2045), `search_space_memberships` (2076), `search_space_invites` (2118), `prompts` (2176), `agent_action_log` (2495), `document_revisions` (2566), `folder_revisions` (2607), `agent_permission_rules` (2643).
-- `external_chat_accounts.owner_search_space_id` (785-786, nullable) — note the distinct column name.
-
-In other model modules:
-
-- `document_files` — [surfsense_backend/app/file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) line 32.
-- `automations` — [surfsense_backend/app/automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) line 27.
-- `notifications` — [surfsense_backend/app/notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) line 50.
-- `podcasts` — [surfsense_backend/app/podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) line 71.
-
-### Named constraints / indexes that embed the old name
-
-- `uq_searchspace_user_connector_type_name` on `search_source_connectors` — [db.py](surfsense_backend/app/db.py) line 1831.
-- `uq_searchspace_role_name` on `search_space_roles` — line 2032.
-- `uq_user_searchspace_membership` on `search_space_memberships` — line 2069.
-- `idx_documents_search_space_id` and `idx_documents_search_space_updated` (raw `CREATE INDEX CONCURRENTLY` DDL in a module-level list) — lines 2821-2830.
-- `ix_external_chat_bindings_search_space_state` — line 978-979.
-
-These reference `search_space_id` by COLUMN but the index/constraint NAME does not embed "searchspace" (definitions auto-follow a column rename; names can stay):
-
-- `ix_notifications_user_space_created` — [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) line 36-41.
-- `uq_agent_permission_rules_scope` — [db.py](surfsense_backend/app/db.py) line 2680.
-
-### CHECK constraint referencing the column AND a scope literal
-
-- `ck_connections_scope_owner` on `connections` — [db.py](surfsense_backend/app/db.py) lines 1578-1584:
- - `"(scope = 'GLOBAL' AND search_space_id IS NULL ...) OR (scope = 'SEARCH_SPACE' AND search_space_id IS NOT NULL ...) OR (scope = 'USER' ...)"`.
- - The physical CHECK expression auto-follows a column rename (Postgres tracks by attnum), but the model's `CheckConstraint(...)` text must be updated to `workspace_id` for create_all/autogenerate consistency.
- - The scope string literal `'SEARCH_SPACE'` is a `ConnectionScope` value, not a column — see Confirmed decisions.
-
-### Zero publication (single source of truth)
-
-- [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py) is "the single source of truth for `zero_publication`" (module docstring, lines 1-11). Publication changes "should update `ZERO_PUBLICATION` and call `apply_publication()` from a migration" (lines 3-5).
-- `search_space_id` appears in **four** published column lists (a recent `main` merge added the `automations`/`new_chat_threads` entries — previously only `documents`/`podcasts`):
- - `DOCUMENT_COLS` line 31.
- - `AUTOMATION_COLS` line 57 (`["id", "search_space_id"]`).
- - `NEW_CHAT_THREAD_COLS` line 62 (`["id", "search_space_id"]`).
- - `PODCAST_COLS` line 76.
- - (`AUTOMATION_RUN_COLS` lines 44-53 does **not** contain the column — `automation_runs` is scoped via `automation_id`, so it auto-follows and needs no edit.)
-- These are the only **four** published tables with column lists that include the column (`ZERO_PUBLICATION` map, lines 81-94; other entries are `None` = all columns and auto-follow). All four (`documents`/`automations`/`new_chat_threads`/`podcasts`) columns must be updated to `workspace_id`, then `apply_publication(op.get_bind())` re-run.
-- Reconcile pattern to copy: [surfsense_backend/alembic/versions/155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) (calls `apply_publication(op.get_bind())`, no-op downgrade).
-
-## Migration design
-
-### Conventions (from the repo)
-
-- Revision ids are plain integers; the new migration chains after the **then-current head** — verify with `alembic heads` at implementation time, since the head advances as other PRs merge. As of the latest `main` sync the head is `169` ([surfsense_backend/alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py](surfsense_backend/alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py); chain `166→167→168→169`), so the new file would be `170_rename_searchspace_to_workspace.py` with `revision="170"`, `down_revision="169"`. (Note: Phase 4b and Phase 5 each add a migration too; whichever lands first takes the next integer — chain by actual head, not by these illustrative numbers.)
-- Migrations run one-per-transaction under an advisory lock — [surfsense_backend/alembic/env.py](surfsense_backend/alembic/env.py) lines 77, 80-95. So all renames below land atomically.
-- Raw `op.execute("ALTER TABLE ...")` is the house style (e.g. [165_add_chunk_position.py](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 39-49).
-
-### upgrade() steps (order matters)
-
-1. Rename tables:
- - `ALTER TABLE searchspaces RENAME TO workspaces;`
- - `ALTER TABLE search_space_roles RENAME TO workspace_roles;`
- - `ALTER TABLE search_space_memberships RENAME TO workspace_memberships;`
- - `ALTER TABLE search_space_invites RENAME TO workspace_invites;`
- - FKs and PKs follow the table automatically (referenced by OID). Sequence `searchspaces_id_seq` and `*_pkey` keep old names — CONFIRMED: also `ALTER SEQUENCE searchspaces_id_seq RENAME TO workspaces_id_seq;` and `ALTER INDEX searchspaces_pkey RENAME TO workspaces_pkey;` (plus the RBAC tables' `*_id_seq` / `*_pkey`) for a clean final schema.
-2. Rename columns on every child table (use `ALTER TABLE RENAME COLUMN search_space_id TO workspace_id;` for the full list in the inventory; plus `external_chat_accounts.owner_search_space_id -> owner_workspace_id`).
- - Metadata-only, fast even on large tables (same property the repo relies on in [165](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 36-41).
- - ORDERING NOTE: the RBAC tables were renamed in step 1, so their column rename must target the NEW table names (`workspace_roles` / `workspace_memberships` / `workspace_invites`), not the old ones.
-3. Rename the named constraints/indexes. Use defensive/idempotent forms because some objects are created at runtime, not by migrations (see review finding 4):
- - Constraints (no `IF EXISTS` for `RENAME CONSTRAINT`; wrap each in a guarded `DO` block that checks `pg_constraint`):
- - `uq_searchspace_user_connector_type_name` -> `uq_workspace_user_connector_type_name` (on `search_source_connectors`).
- - `uq_searchspace_role_name` -> `uq_workspace_role_name` (on `workspace_roles`).
- - `uq_user_searchspace_membership` -> `uq_user_workspace_membership` (on `workspace_memberships`).
- - Indexes (these two are created by the runtime `setup_indexes()` startup routine, NOT a migration, so they may or may not exist depending on `DB_BOOTSTRAP_ON_STARTUP`; use `IF EXISTS`):
- - `ALTER INDEX IF EXISTS idx_documents_search_space_id RENAME TO idx_documents_workspace_id;`
- - `ALTER INDEX IF EXISTS idx_documents_search_space_updated RENAME TO idx_documents_workspace_updated;`
- - `ALTER INDEX IF EXISTS ix_external_chat_bindings_search_space_state RENAME TO ix_external_chat_bindings_workspace_state;`
-4. Zero publication. DECISION: we will recreate the Zero replication (reset the zero-cache replica) as part of this deploy, so consumer recovery is handled out-of-band and finding 3 is de-risked. Follow the repo's established publication patterns — do NOT use raw `DROP`/`CREATE PUBLICATION` (forbidden by [116_create_zero_publication.py](surfsense_backend/alembic/versions/116_create_zero_publication.py) lines 8-17; reintroduces bug #1355). Sequence inside the migration:
- - (Safe ordering) Neutralize the column-list dependency surgically so the RENAME is unconditionally permitted: `ALTER PUBLICATION zero_publication DROP TABLE documents, automations, new_chat_threads, podcasts;` (all **four** column-list tables). This removes ONLY their column-list dependency and is far safer than re-emitting the whole member set via `SET TABLE` (which is a full replacement and would drop any table accidentally omitted from the hand-written list, including the quoted `"user"` table). `DROP TABLE` from a publication is within the blessed ALTER-PUBLICATION family — it is NOT the forbidden raw `DROP/CREATE PUBLICATION`.
- - Rename the columns (steps 1-2).
- - Update canonical `DOCUMENT_COLS`/`AUTOMATION_COLS`/`NEW_CHAT_THREAD_COLS`/`PODCAST_COLS` to `workspace_id` in [zero_publication.py](surfsense_backend/app/zero_publication.py) (31, 57, 62, 76), then call `apply_publication(op.get_bind())` (blessed plain `ALTER ... SET TABLE`, as used by [155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py)) to re-add all four dropped tables with the narrowed `workspace_id` column lists and reconcile the full member set in one shot.
- - Reference template for forcing a schema-change event (only if event triggers are unavailable): the `COMMENT ON PUBLICATION` bookend trio in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) lines 104-137. With event triggers installed (current setup), `apply_publication` is sufficient.
- - Replica recreate (operational, out-of-band): reset the `surfsense-zero-cache` replica (delete the volume / rely on `ZERO_AUTO_RESET=true`) so zero-cache does a fresh initial sync from the corrected publication. This is the "recreate replication" step and the primary consumer-recovery mechanism.
- - INTERLOCK FOOTGUN (review finding 2): if any of the canonical `DOCUMENT_COLS`/`AUTOMATION_COLS`/`NEW_CHAT_THREAD_COLS`/`PODCAST_COLS` are NOT updated to `workspace_id` before `apply_publication` runs, `_format_table_entry` ([zero_publication.py](surfsense_backend/app/zero_publication.py) lines 148-149) silently DROPS the mismatched table(s) from the publication (no error). Gate with `--verify` (see Verification).
-5. No DDL needed for the `ck_connections_scope_owner` CHECK: the physical expression auto-follows the column rename (Postgres tracks the column by attnum). The model's `CheckConstraint(...)` text is updated separately as an ORM edit (see ORM edits) so create_all/autogenerate stay consistent.
-
-### downgrade()
-
-Provide a full reverse (rename `workspaces` -> `searchspaces`, columns back, constraints/indexes back, then restore the OLD publication shape). Note the repo sometimes uses no-op downgrades for publication-only migrations ([155](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) lines 22-23), but a structural rename should be reversible.
-
-CRITICAL (review finding 7 — do NOT call `apply_publication()` in downgrade): `apply_publication()` reads the LIVE canonical in `zero_publication.py`, which after this change is `workspace_id`. In a downgrade the column is back to `search_space_id`, so `apply_publication` would find `workspace_id` missing and silently DROP the four column-list tables (`documents`/`automations`/`new_chat_threads`/`podcasts`) from the publication — the finding-2 footgun, self-inflicted. Instead, the downgrade must restore the old shape with HARDCODED `search_space_id` column lists (for all four tables) via a plain `ALTER PUBLICATION ... SET TABLE`, mirroring the constants-style approach in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) (which embeds literal `DOCUMENT_COLS` rather than importing the live module). Use the same DROP-then-SET neutralize sequence in reverse.
-
-COHERENCE (review finding 6): even with a correct downgrade SQL, `downgrade()` only produces a consistent RUNNING system if the PRIOR code revision is redeployed alongside it — under the shim the model maps the attribute to physical `workspace_id`, which no longer exists after a pure DB downgrade. Document rollback = revert code + schema together as one operation.
-
-## ORM / source edits in this phase (mapping only)
-
-These are required so the ORM matches the renamed physical schema; they are NOT the broad symbolic rename. Explicitly OUT of scope for Phase 1: relationship attribute names (`User.search_spaces`, `back_populates="search_space"`, etc.) and class names (`SearchSpace`, `SearchSpaceRole`, ...). They stay as-is here and are renamed in Phase 2 — leaving them untouched is what keeps the ~250 callers green.
-
-1. [surfsense_backend/app/db.py](surfsense_backend/app/db.py):
- - `__tablename__` for `SearchSpace` (1688), `SearchSpaceRole` (2027), `SearchSpaceMembership` (2064), `SearchSpaceInvite` (2113) -> new table names.
- - Every `ForeignKey("searchspaces.id", ...)` -> `ForeignKey("workspaces.id", ...)`; `ForeignKey("search_space_roles.id"...)` -> `workspace_roles.id`; `ForeignKey("search_space_invites.id"...)` -> `workspace_invites.id`.
- - Apply the explicit-name shim on each `search_space_id` column: `Column("workspace_id", ...)`; and `owner_search_space_id = Column("owner_workspace_id", ...)`.
- - Update ONLY the `name=` kwarg of constraints/indexes (1831, 2032, 2069, and the Index at 978-979). CRITICAL (review finding 1): do NOT change the inner column-reference strings (`"search_space_id"` at 1827, 2030, 2068, 2674, 979) — under the shim the column's `.key` is still `search_space_id`, and declarative `__table_args__` resolves these strings against `Table.c` by `.key`. Changing them to `"workspace_id"` raises a config-time error and the app won't boot.
- - Update the runtime raw index DDL strings + names in `_INDEX_DEFINITIONS` (~2821-2831; +1 line vs the pre-merge cite after `main`'s `RefreshToken` edit — locate by grep): both the index name and the physical column (`search_space_id` -> `workspace_id`). These must match the migration's renamed indexes and ship in the same release; `setup_indexes()` ([db.py](surfsense_backend/app/db.py) ~line 2856) re-creates them under the new name on next boot when `DB_BOOTSTRAP_ON_STARTUP` is true.
- - Update `ck_connections_scope_owner` text (1579-1582) `search_space_id` -> `workspace_id`.
- - Update `ix_external_chat_bindings_search_space_state` name + the surrounding Index (978-979).
-2. Module models — `__tablename__` unaffected, but apply the column shim + FK target string:
- - [file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) (32), [automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) (27), [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) (50, plus the index at 36-41), [podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) (71).
-3. [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py): `DOCUMENT_COLS` (31), `AUTOMATION_COLS` (57), `NEW_CHAT_THREAD_COLS` (62), and `PODCAST_COLS` (76) `search_space_id` -> `workspace_id`. (The `_expected_columns` special-casing on line 118 keys off table names `{"documents","user","podcasts"}`, not the column, so no change there — and `automations`/`new_chat_threads` aren't in that `_0_version` allowlist anyway.)
-4. Runtime raw-SQL audit (defensive): grep `surfsense_backend/app` for any `text(...)` / hardcoded `"searchspaces"` or `search_space_id` strings that execute at runtime (not Python attribute access) and fix them here, since the shim only covers ORM attribute access. Expectation is few; the ~250-file footprint is overwhelmingly attribute access handled by the shim.
-
-## Confirmed decisions
-
-- Strategy: SHIM approach (physical DB rename now; Python attribute kept via `Column("workspace_id", ...)`; symbolic attribute/class/API rename deferred to Phase 2).
-- Scope literal `'SEARCH_SPACE'` (the `ConnectionScope` value in `ck_connections_scope_owner`): KEEP the string value as-is; only fix the `search_space_id` column reference in the CHECK. No enum change, no data UPDATE.
-- Cosmetic names: RENAME auto-named sequences/PK indexes (`searchspaces_id_seq` -> `workspaces_id_seq`, `searchspaces_pkey` -> `workspaces_pkey`, and the RBAC tables' `*_id_seq`/`*_pkey`) for a clean final schema.
-- Transition: HARD CUTOVER at the DB layer (no backward-compat old column/view). Client apps hit the API (Phase 2), not the DB, so the DB rename does not affect them. (Phase 2 RESOLVED: hard cutover, no API aliases — see [02-rename-backend.md](02-rename-backend.md).)
-- Zero: RECREATE THE REPLICATION (reset the zero-cache replica / `ZERO_AUTO_RESET`) on deploy for consumer recovery. Publication mutated only via the blessed `apply_publication` / `ALTER ... SET TABLE` path (never raw DROP/CREATE PUBLICATION, per migration 116). See migration step 4.
-
-## Release coupling (review finding 5)
-
-The Alembic migration, the `db.py` edits (shim + constraint `name=` + `_INDEX_DEFINITIONS` + `ck_connections_scope_owner` text), the 4 module-model edits, and `zero_publication.py` MUST ship as ONE atomic release. There is no safe intermediate state where some are updated and others are not.
-
-## Verification & rollout
-
-- HARD GATE: `python -m app.zero_publication --verify` (CLI at [zero_publication.py](surfsense_backend/app/zero_publication.py) ~lines 266-269; `verify_publication` defined at ~203) must report verified, in CI and immediately post-migrate. This is the guard against the interlock footgun (finding 2).
-- Boot the API + a Celery worker; run a smoke chat + a document upload to exercise `documents.workspace_id` and the publication; confirm Zero clients still sync `documents`/`podcasts`.
-- AUTOGENERATE DRIFT CHECK (strong, cheap): after the change, `alembic revision --autogenerate` must produce an EMPTY diff. A non-empty diff means the ORM and migrated DB disagree — i.e. a missed `ForeignKey("searchspaces.id")` string, a constraint `name=` mismatch, or a forgotten `Column("workspace_id", ...)` shim. This single check catches most subtle misses.
-- Confirm `alembic upgrade head` then `alembic downgrade -1` then `upgrade head` round-trips on a staging copy. NOTE: this validates SQL reversibility only — it is NOT an app-health check, because new code cannot run against the downgraded schema (finding 6). The downgraded intermediate state must show `documents`/`podcasts` still present in the publication with `search_space_id` lists (proves the hardcoded downgrade publication shape from finding 7 works).
-- Run the full backend test suite; the shim keeps attribute-based tests green — watch specifically for fixtures/raw SQL that hardcode the table name `searchspaces`.
-- Pre-flight check: confirm no DB views/materialized views depend on the renamed objects (they auto-follow a rename, but verify so a later `pg_dump` diff holds no surprises).
-- Raw-SQL audit done-check: grep `surfsense_backend/app` for `text(...)` or string literals containing `searchspaces` / `search_space_id` that execute at runtime; current audit found none beyond ORM attribute/dict-key usage (which the shim covers).
-
-## Risks (with review findings)
-
-- RENAME COLUMN under an active Zero publication column list (finding 3) — de-risked: step 4 neutralizes the column-list dependency (publish documents/podcasts as ALL columns) before the rename, and the zero-cache replica is recreated on deploy, so consumers re-sync cleanly.
-- Silent publication drop if canonical col lists aren't updated (finding 2) — mitigated by the `--verify` hard gate.
-- Zero consumers recover via the planned replica recreate (reset `surfsense-zero-cache` / `ZERO_AUTO_RESET=true`), giving a fresh initial sync from the corrected publication; no raw DROP/CREATE PUBLICATION (bug #1355).
-- Constraint/index column-ref strings accidentally changed under the shim (finding 1) — config-time boot failure; mitigated by the explicit "only change name=" instruction.
-- Runtime-created indexes may not exist at migrate time (finding 4) — mitigated by `IF EXISTS` renames + guarded `DO` blocks for constraints.
-- Missing a `ForeignKey("searchspaces.id")` string leaves SQLAlchemy metadata pointing at a non-existent table — fails fast at mapper config (`NoReferencedTableError`); the inventory list above is the checklist.
-- Partial/non-atomic release (finding 5) or code-less downgrade (finding 6) — mitigated by shipping all edits in one release and treating rollback as code+schema together.
-- `downgrade()` calling `apply_publication()` against the live (new) canonical (finding 7) — would silently re-drop `documents`/`podcasts`; mitigated by hardcoding the old `search_space_id` publication shape in `downgrade()` (143-style), never importing the live module.
-
-## Out of scope (later phases)
-
-- Phase 2: rename Python attribute `search_space_id` -> `workspace_id`, class `SearchSpace` -> `Workspace`, Pydantic schemas, API routes/paths (`/searchspaces` + `/search-spaces` -> `/workspaces`), Redis keys, storage path segments; drop the explicit `Column("workspace_id", ...)` shim.
-- Frontend Zero schema, route segment, i18n — deferred frontend umbrella.
-
----
-
-## Implementation record (as-built)
-
-### Deviations from the plan
-
-- **Migration 168 idempotency fixed up-front**: the plan flagged from-scratch alembic as pre-existing-broken; we made 168 idempotent so the rename starts from a clean head, but did **not** take on the full baseline-squash. From-scratch `alembic upgrade head` therefore stays pre-existing-broken (rev 23 conflict); only the existing-DB `169 -> 170` path is in scope/verified.
-- **Cosmetic sequence/PK renames applied** (the plan's confirmed decision): `searchspaces_id_seq -> workspaces_id_seq`, `searchspaces_pkey -> workspaces_pkey`, plus the RBAC tables' `*_id_seq` / `*_pkey`, for a clean final schema.
-- Everything else implemented exactly as designed: shim via `Column("workspace_id", ...)`; publication mutated **only** through the blessed `apply_publication` path (no raw DROP/CREATE PUBLICATION, per migration 116); `__table_args__` inner column-reference strings deliberately **left** as `search_space_id` (flipped in Phase 2).
-
-### Carve-outs as-shipped (Phase 1)
-
-- `'SEARCH_SPACE'` CHECK literal in `ck_connections_scope_owner` kept; only the `search_space_id` column reference flipped to `workspace_id`. No enum/data migration.
-
-### Verify current state (re-runnable)
-
-Each line is a command + the last captured result. Re-run to confirm the *current* truth instead of trusting the snapshot. Schema gates assume a DB at rev `170` (locally `surfsense_oldshape`) with `AUTH_TYPE=LOCAL` (so the env-gated `oauth_account` table isn't in the ORM metadata).
-
-- **Schema drift (ORM ↔ physical @170)** —
- `AUTH_TYPE=LOCAL DATABASE_URL=postgresql+asyncpg://…@localhost:5432/surfsense_oldshape uv run alembic check`
- → last (2026-06-27): `No new upgrade operations detected.` (With `AUTH_TYPE=GOOGLE` the only delta is the env-gated `oauth_account` table — unrelated to the rename.)
-- **Publication columns** —
- `rg -n 'workspace_id|search_space_id' app/zero_publication.py`
- → last (2026-06-27): all four lists = `workspace_id`, zero `search_space_id`.
-
-Validated during the build run (2026-06-26), **not** reproducible on a `create_all`-built snapshot (it has no `zero_publication` object): `python -m app.zero_publication --verify` → verified (interlock-footgun guard, finding 2); `alembic upgrade 170 → downgrade -1 → upgrade 170` round-trip (SQL reversibility; downgrade restores the `search_space_id` publication shape via hardcoded lists, per finding 7).
diff --git a/plans/backend/02-rename-backend.md b/plans/backend/02-rename-backend.md
deleted file mode 100644
index cc56df3f0..000000000
--- a/plans/backend/02-rename-backend.md
+++ /dev/null
@@ -1,188 +0,0 @@
-# Subplan 02 — Rename backend (code + API)
-
-Part of [00-umbrella-plan.md](00-umbrella-plan.md), Phase 2. Backend only (`surfsense_backend`).
-
-> **Status: SHIPPED** · as of 2026-06-27 · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)
-> Last commit of this phase: `902b3374e` (backend code/API rename, Waves A–F + carve-outs). Phase 1 + Phase 2 shipped as one atomic PR.
-> The sections below are the **original design/rationale**; the as-built state + how to re-verify live in the [Implementation record](#implementation-record-as-built) at the bottom. Ground truth for files/commits is the PR, not this doc.
-
-## Goal
-
-Remove the Phase 1 shim and complete the SYMBOLIC rename `SearchSpace -> Workspace` / `search_space_id -> workspace_id` across the ~150 backend files, then consolidate the three live URL spellings (`/searchspaces`, `/search-spaces`, `/search-space`) onto a single canonical `/workspaces`. After this phase the physical DB (Phase 1) and the Python/API surface speak the same name.
-
-Precondition: [01-rename-db.md](01-rename-db.md) is merged and live. Phase 1 left the Python attribute as `search_space_id` mapped to the physical column via `Column("workspace_id", ...)`; Phase 2 flips the attribute itself and drops that explicit-name shim.
-
-## Surface area (cited; counts are current-tree, will shift after Phase 1)
-
-Measured over `surfsense_backend/app` (Grep, case-insensitive `search.?space`):
-
-- `search_space` — ~150 files, ~2,800 line matches (dominated by ORM attribute access).
-- `SearchSpace` (class/relationship target) — ~38 files, ~450 matches.
-- Heaviest files: [services/connector_service.py](surfsense_backend/app/services/connector_service.py) (158), [routes/search_source_connectors_routes.py](surfsense_backend/app/routes/search_source_connectors_routes.py) (138), [routes/rbac_routes.py](surfsense_backend/app/routes/rbac_routes.py) (142), [routes/model_connections_routes.py](surfsense_backend/app/routes/model_connections_routes.py) (132), [routes/new_chat_routes.py](surfsense_backend/app/routes/new_chat_routes.py) (133), [routes/documents_routes.py](surfsense_backend/app/routes/documents_routes.py) (121), [db.py](surfsense_backend/app/db.py) (119).
-- Plus ~120 test files under `surfsense_backend/tests` mirror these patterns (rename in the same release).
-
-This is mostly mechanical (an automated symbol rename handles the bulk), but a fixed set of STRING-LITERAL contracts and EXTERNAL contracts are NOT safe to blind-replace — they get explicit decisions in this plan (see "String-literal & external contracts").
-
-## Transition policy: HARD CUTOVER (confirmed)
-
-The umbrella defers all frontend/client work, and the frontend will be (re)built against the corrected backend in its own umbrella rather than kept alive in lockstep. So this phase HARD-CUTS the external API — no backward-compat aliases.
-
-- URL paths: rename `/api/v1/searchspaces...`, `/api/v1/search-spaces/...`, `/api/v1/...search-space...` outright to the canonical `/api/v1/workspaces...` ([routes/search_spaces_routes.py](surfsense_backend/app/routes/search_spaces_routes.py) 73-373; mount at [app.py](surfsense_backend/app/app.py) 993, `crud_router` under `/api/v1`).
-- JSON field names: rename `search_space_id` -> `workspace_id` in request/response bodies outright (e.g. [schemas/new_chat.py](surfsense_backend/app/schemas/new_chat.py), [schemas/rbac_schemas.py](surfsense_backend/app/schemas/rbac_schemas.py), [schemas/documents.py](surfsense_backend/app/schemas/documents.py)). No `populate_by_name` alias plumbing.
-
-CONSEQUENCE (accepted): the existing deployed frontend breaks against the renamed API until its umbrella lands. Backend correctness is therefore verified INDEPENDENTLY of the old UI — via the test suite, OpenAPI/`/docs`, and direct API calls — not by clicking through the current frontend. (Alias/dual-serve was considered and rejected: it adds plumbing to keep a frontend alive that is being redesigned anyway.)
-
-## Rename waves (order matters)
-
-### Wave A — ORM core in [db.py](surfsense_backend/app/db.py)
-
-1. Classes: `SearchSpace` (1688), `SearchSpaceRole` (2021/2027), `SearchSpaceMembership` (2058/2064), `SearchSpaceInvite` (2107/2113) -> `Workspace`, `WorkspaceRole`, `WorkspaceMembership`, `WorkspaceInvite`.
-2. Drop the Phase 1 shim on every column: `search_space_id = Column("workspace_id", Integer, ForeignKey("workspaces.id", ...))` -> `workspace_id = Column(Integer, ForeignKey("workspaces.id", ...))` (the explicit `"workspace_id"` first arg is now redundant because attribute name == column name). Same for `owner_search_space_id` -> `owner_workspace_id` (785-787).
-3. INVERSE OF PHASE-1 FINDING 1: now that the attribute `.key` becomes `workspace_id`, the `__table_args__` inner column-reference strings MUST flip to `"workspace_id"` to match — `UniqueConstraint("search_space_id", ...)` -> `UniqueConstraint("workspace_id", ...)` at 1826-1832 (`uq_workspace_user_connector_type_name`), 2029-2033 (`uq_workspace_role_name`), 2066-2070 (`uq_user_workspace_membership`), and the `Index(... "search_space_id" ...)` at 978-979. (Phase 1 deliberately left these as `search_space_id`; Phase 2 completes them. Getting this wrong = config-time boot failure, same failure mode as Phase 1 finding 1, just mirrored.)
-4. Relationships (~35 pairs): rename the attribute names and both ends of `back_populates`, and the target-class strings:
- - Child side: `search_space = relationship("SearchSpace", back_populates="...")` -> `workspace = relationship("Workspace", back_populates="...")` (e.g. Folder 1340, Document 1421, Connection 948).
- - Hub side on `Workspace`: `back_populates="search_space"` -> `"workspace"` for folders (1726-1728), documents, threads, podcasts, video_presentations, reports, image_generations, logs, notifications, connectors, connections, automations, roles, memberships, invites.
- - User side (both auth branches ~2205 and ~2337): `search_spaces = relationship("SearchSpace", ...)` -> `workspaces = relationship("Workspace", ...)`; `search_space_memberships` -> `workspace_memberships`; invites likewise.
- - External chat: `owner_search_space = relationship("SearchSpace", foreign_keys=[owner_search_space_id])` (819-820) -> `owner_workspace = relationship("Workspace", foreign_keys=[owner_workspace_id])`.
-5. CHECK text on `connections`: `ck_connections_scope_owner` (1578-1581) `search_space_id` -> `workspace_id` in the SQL string. (Phase 1 already did this if shipped; re-confirm. The scope literal `'SEARCH_SPACE'` stays — see enum carve-out.)
-6. Runtime index DDL in `_INDEX_DEFINITIONS` (2820-2830): already renamed to `workspace_id`/`idx_documents_workspace_id` in Phase 1 — confirm no `search_space` remains.
-
-### Wave B — satellite ORM modules (mirror Wave A; drop shim, flip strings)
-
-- [automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) (27-29 column, 68 relationship).
-- [file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) (32-34).
-- [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) (50-52 column, 72 relationship, plus the index `ix_notifications_user_space_created` 36-41 — definition auto-follows; rename only if you want the name clean).
-- [podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) (71-72).
-
-### Wave C — Pydantic schemas
-
-- Dedicated module [schemas/search_space.py](surfsense_backend/app/schemas/search_space.py): classes `SearchSpaceBase/Create/Update/Read/WithStats` -> `Workspace*`. Rename file to `schemas/workspace.py`. Update re-exports in [schemas/__init__.py](surfsense_backend/app/schemas/__init__.py) (106-111, and `UserSearchSpaceAccess` ~87).
-- Field `search_space_id` -> `workspace_id` across: new_chat, rbac_schemas (+ `search_space_name`, `UserSearchSpaceAccess`), documents, folders, image_generation, model_connections, search_source_connector, logs, stripe, prompts, chat_comments, video_presentations, reports, obsidian_plugin, podcasts/api/schemas, automations schemas, notifications/api/schemas.
-- Under hard cutover the serialized JSON key changes outright to `workspace_id` (no alias). Update fixtures/contract tests accordingly.
-
-### Wave D — services / utils / tasks / agents / gateway / event_bus (the mechanical bulk)
-
-Pure attribute/param/function symbol rename `search_space_id -> workspace_id`, `search_space -> workspace`, plus function renames:
-- RBAC helpers [utils/rbac.py](surfsense_backend/app/utils/rbac.py): `check_search_space_access` (129), `is_search_space_owner` (160), `get_search_space_with_access_check` (180) -> `*_workspace_*`.
-- Validator [utils/validators.py](surfsense_backend/app/utils/validators.py): `validate_search_space_id` (16).
-- Retrievers [retriever/documents_hybrid_search.py](surfsense_backend/app/retriever/documents_hybrid_search.py), [retriever/chunks_hybrid_search.py](surfsense_backend/app/retriever/chunks_hybrid_search.py) (~26 each, filter clauses).
-- Connector service [services/connector_service.py](surfsense_backend/app/services/connector_service.py) (ctor param + `self.search_space_id`, + the in-process caches keyed by id — see literals).
-- Indexers under `app/tasks/connector_indexers/*`, `indexing_pipeline/*`, `services/*/kb_sync_service.py`.
-- Agents: dep-dict KEYS like `"search_space_id": d["search_space_id"]` (e.g. [agents/.../subagents/connectors/google_drive/tools/index.py](surfsense_backend/app/agents/chat/multi_agent_chat/subagents/connectors/google_drive/tools/index.py) 28; `main_agent/runtime/factory.py` 137); class `SearchSpaceSkillsBackend` (`.../main_agent/skills/backends.py` 184).
-- Gateway: [gateway/inbox_processor.py](surfsense_backend/app/gateway/inbox_processor.py), [gateway/agent_invoke.py](surfsense_backend/app/gateway/agent_invoke.py), [gateway/auth_invariant.py](surfsense_backend/app/gateway/auth_invariant.py).
-- Event bus: `Event.search_space_id` field ([event_bus/event.py](surfsense_backend/app/event_bus/event.py) 36), `publish(... search_space_id=...)` ([event_bus/bus.py](surfsense_backend/app/event_bus/bus.py) 43-49), payload-dict key in [event_bus/events/document_entered_folder.py](surfsense_backend/app/event_bus/events/document_entered_folder.py) 74-76 — see literals decision.
-
-### Wave E — routes: rename handlers + consolidate the three URL spellings
-
-Canonical: `/workspaces` (umbrella decision). Current spellings to retire:
-- `/searchspaces` (no hyphen): [search_spaces_routes.py](surfsense_backend/app/routes/search_spaces_routes.py) (73-373), [rbac_routes.py](surfsense_backend/app/routes/rbac_routes.py), [agent_permissions_route.py](surfsense_backend/app/routes/agent_permissions_route.py), [team_memory_routes.py](surfsense_backend/app/routes/team_memory_routes.py).
-- `/search-spaces` (hyphen, plural): [editor_routes.py](surfsense_backend/app/routes/editor_routes.py), [notes_routes.py](surfsense_backend/app/routes/notes_routes.py), [export_routes.py](surfsense_backend/app/routes/export_routes.py), [model_connections_routes.py](surfsense_backend/app/routes/model_connections_routes.py) (`/model-roles`).
-- `/search-space` (hyphen, singular): [logs_routes.py](surfsense_backend/app/routes/logs_routes.py) 252, [gateway_webhook_routes.py](surfsense_backend/app/routes/gateway_webhook_routes.py) 992/1022, webhook path `{search_space_id}` in [circleback_webhook_route.py](surfsense_backend/app/routes/circleback_webhook_route.py) 215/315.
-- Rename the route files (`search_spaces_routes.py` -> `workspaces_routes.py`) and handler fns (`create_search_space` etc.) and the import/include in [routes/__init__.py](surfsense_backend/app/routes/__init__.py) (63, 73).
-
-Mechanism: routers use `APIRouter()` with the path fully spelled in each decorator (no per-router prefix), mounted under `/api/v1` at [app.py](surfsense_backend/app/app.py) 993. So consolidation = edit each decorator string to `/workspaces/{workspace_id}/...` outright (hard cutover; no alias routers).
-
-### Wave F — tests + fixtures
-
-Rename `surfsense_backend/tests/**`; watch for fixtures that hardcode the table name `searchspaces`/JSON key `search_space_id` (those depend on the transition policy for API tests).
-
-## String-literal & external contracts (explicit decisions — NOT blind-replace)
-
-These do not move with a symbol rename; each is decided here.
-
-1. Enum VALUES `'SEARCH_SPACE'` — `ConnectionScope.SEARCH_SPACE` ([db.py](surfsense_backend/app/db.py) 204-207, stored in `connections.scope` via `SQLAlchemyEnum`, line 1558) and `ChatVisibility.SEARCH_SPACE` (510-520, stored in `new_chat_threads.visibility`, 596-597). DECISION: KEEP the enum value strings as-is. They are persisted in Postgres and exposed in JSON; renaming the value needs a data migration + PG enum-type alter for zero benefit. This matches Phase 1's decision to keep the `'SEARCH_SPACE'` CHECK literal. (Optionally rename only the Python member to `WORKSPACE` while keeping `= "SEARCH_SPACE"` — deferred; not worth the churn for MVP.)
-2. Celery task `name=` strings — `"delete_search_space_background"` ([tasks/celery_tasks/document_tasks.py](surfsense_backend/app/tasks/celery_tasks/document_tasks.py) 206), `"ai_sort_search_space"` (1546). These are the WIRE NAME between producer and worker; tasks dispatch via `.delay()`/`send_task` (e.g. event trigger `send_task(TASK_NAME, ...)` in [automations/triggers/builtin/event/source.py](surfsense_backend/app/automations/triggers/builtin/event/source.py) 19). DECISION: KEEP the `name=` strings unchanged; freely rename the Python function symbols (`ai_sort_search_space_task` etc.). A rolling deploy with renamed wire names would orphan in-flight messages. (If a cosmetic rename is wanted later, do it with a dual-register + queue-drain, out of scope here.)
-3. Redis key literals — `surfsense:spawn_paused:{search_space_id}` (`tasks/.../spawn_paused.py`) and `ai_sort:search_space:{search_space_id}:lock` ([document_tasks.py](surfsense_backend/app/tasks/celery_tasks/document_tasks.py) 1542). DECISION: rename literals to `workspace` (these hold short-lived locks / an ops toggle). Accept that any in-flight lock / paused-flag resets at deploy (locks are seconds-long; paused-flag is an ops action). UPDATE the runbook in `.env.example` (the `redis-cli SET surfsense:spawn_paused:` doc, ~503-504).
-4. Event payload key `search_space_id` — `Event` is `model_dump`ed and sent to Celery for automation event triggers, and trigger filters read the key. DECISION: rename the field to `workspace_id` (internal, and triggers are re-evaluated continuously); accept transient loss of any event enqueued across the deploy boundary (fire-and-forget). Drain the event queue during the maintenance window to be safe.
-5. OpenTelemetry attribute `search_space.id` + metric label ([observability/otel.py](surfsense_backend/app/observability/otel.py) 263-264/305-306, [observability/metrics.py](surfsense_backend/app/observability/metrics.py) 537-542). DECISION: KEEP the OTel/metric KEY `search_space.id` for now (dashboards/alerts depend on it); rename only the Python params. Schedule the observability-key rename as a separate, announced change. (Carve-out to avoid silently breaking alerting.)
-6. Notification dedup/operation IDs embedding `{search_space_id}` (e.g. `doc_..._{search_space_id}_...`, `insufficient_credits_{search_space_id}_...`) and frontend deep-link strings like `/dashboard/{search_space_id}/buy-more`. DECISION: the ID is a numeric value, not the literal word — leave format strings as-is functionally; the embedded VALUE is unchanged. The `/dashboard/{id}/...` deep link points at a FRONTEND route still named `[search_space_id]` (deferred umbrella) — KEEP it until the frontend segment renames, else links 404.
-7. Storage path builders — `documents/{search_space_id}/...` ([file_storage/keys.py](surfsense_backend/app/file_storage/keys.py) 20-26) and `podcasts/{search_space_id}/...` ([podcasts/storage.py](surfsense_backend/app/podcasts/storage.py) 22-25). The path segment is the numeric ID; the literal word `search_space` is NOT in stored object keys. DECISION: rename the param only; NO blob migration needed; existing objects keep resolving.
-8. `SearchSourceConnector` — contains "search" but is the connectors table, a different concept (the word "search" here is unrelated to `SearchSpace`). OUT OF SCOPE: this rename does NOT touch it, and Phase 4 does not rename the class either — Phase 4 adds a Type-1/Type-2 taxonomy via a static `connector_type`→(category, availability) registry (no new column) and KEEPS `is_indexable` (`db.py:1868`). Leave `SearchSourceConnector` as-is.
-9. Historical Alembic migrations (`surfsense_backend/alembic/versions/*`) — ~20 files embed `searchspaces` / `search_space_id` as raw-SQL string literals (e.g. `23_associate_connectors_with_search_spaces.py`, `41_backfill_rbac_for_existing_searchspaces.py`, `40_move_llm_preferences_to_searchspace.py`). DECISION: NEVER rewrite these. They are an immutable replay log that intentionally references the schema as it existed at that revision; rewriting them corrupts history and breaks a clean `alembic upgrade` from zero. Verified safe: no migration imports the ORM classes (only `from app.db import Base` in `env.py` / `0_initial_schema.py`), so the class rename does not touch them. Phase 1's rename migration (the next integer after the live head — `170` at time of writing, since `main` advanced the head to `169`, but chain by actual `alembic heads`) is the single transition point; migrations after it use the new names. The scripted rename scope is `app/` + `tests/` ONLY.
-10. LangGraph persisted state channel key `search_space_id` — `input_state = {..., "search_space_id": search_space_id, ...}` ([tasks/chat/streaming/flows/new_chat/input_state.py](surfsense_backend/app/tasks/chat/streaming/flows/new_chat/input_state.py) 131) is a CHECKPOINTED state channel. The resume_chat flow reads persisted state (`agent.aget_state({"configurable": {"thread_id": ...}})`, [flows/resume_chat/resume_routing.py](surfsense_backend/app/tasks/chat/streaming/flows/resume_chat/resume_routing.py) 50) and HITL interrupts (`surfsense_resume_value`, `HumanReview`) can sit pending across a deploy. DECISION: rename the channel key to `workspace_id` for consistency, and ACCEPT that any chat thread paused at a HITL interrupt BEFORE the cutover must be restarted after (the old checkpoint carries `search_space_id`, the new graph reads `workspace_id`). Operationally: drain/resolve pending interrupts before deploy if practical. (Alternative — keep the channel key as a carve-out — rejected: leaves a lone `search_space_id` in otherwise-renamed agent state for a transient, conversation-scoped value. The `configurable` keys are `thread_id` / `surfsense_resume_value`, not `search_space_id`, so only this state channel is affected.)
-11. User-facing default literal — `users.py` seeds the default workspace `name="My Search Space"` (line 158, persisted + shown to users) and logs "Created default search space" (207). DECISION: rename the default name to "My Workspace" (backend-created seed value, not caught by a symbol rename); log strings are cosmetic. Also update the Redis-key assertion in `tests/unit/services/test_ai_sort_task_dedupe.py:14` when literal 3 is renamed.
-
-## Execution approach
-
-- Automate the safe bulk: a scripted symbol rename (IDE rename / `ast-grep`-style) for `search_space_id -> workspace_id`, `search_space -> workspace`, `SearchSpace -> Workspace`. SCOPE STRICTLY to `app/` + `tests/` — NEVER `alembic/versions/` (carve-out 9: immutable replay log). EXCLUDE the other carve-outs above (enum values, Celery `name=`, OTel keys, frontend deep-links, default-name literal). Run per-wave, not all at once, so review is tractable.
-- Then hand-apply the carve-outs and the routing consolidation.
-- One atomic release (same coupling rule as Phase 1): ORM + schemas + routes + literals ship together; there is no half-renamed steady state.
-
-## Verification & rollout
-
-- `alembic revision --autogenerate` must produce an EMPTY diff: confirms the renamed ORM (classes/attrs/constraint inner-strings) still matches the Phase 1 physical schema. A diff means a missed attribute flip or a stale `__table_args__` string.
-- Boot API + Celery worker; verify via OpenAPI/`/docs` + direct API calls (NOT the old frontend): create workspace, chat, document upload, an RBAC invite/membership op, an automation event trigger, and a podcast — exercising the renamed relationships (`workspace`, `workspaces`), tasks, and event payloads.
-- Hard-cutover contract check: confirm `/workspaces...` resolves and the three legacy spellings now 404; confirm bodies use `workspace_id` only.
-- Full backend test suite green; grep `surfsense_backend/app` for residual `search_space`/`SearchSpace` and confirm every remaining hit is an intentional carve-out (enum value, Celery `name=`, OTel key, frontend deep-link).
-- Confirm `alembic/versions/*` was NOT modified by the rename (git diff shows zero changes under that path) — guard against carve-out 9.
-- Resume check (carve-out 10): start a chat, trigger a HITL interrupt, deploy/rename, then verify NEW interrupts resume on `workspace_id`; accept that pre-cutover pending interrupts are restarted. Ideally drain pending interrupts before deploy.
-- Observability check: `search_space.id` OTel attribute still emitted (carve-out 5) so dashboards keep working.
-
-## Risks
-
-- Stale `__table_args__` inner string after attribute flip (Wave A.3) — config-time boot failure; mirror of Phase 1 finding 1. Mitigated by the autogenerate empty-diff gate + boot smoke test.
-- Relationship rename misses one `back_populates` end — mapper-config error at startup; the pair list in Wave A.4 is the checklist.
-- Hard cutover breaks the existing frontend in the interim (accepted) — mitigated by verifying the backend via tests/OpenAPI/API calls, and by (re)building the frontend against the new contract in its umbrella. Do not rely on the old UI to smoke-test during this window.
-- Renaming a Celery `name=` or the event payload key without a drain — orphaned/dropped in-flight messages; mitigated by the carve-out (keep task names) + queue-drain for events.
-- Enum value migration scope creep — explicitly OUT (carve-out 1); keep `'SEARCH_SPACE'` values.
-- Scripted rename corrupting historical Alembic migrations (carve-out 9) — would break `alembic upgrade` from zero / replay history; mitigated by scoping the rename to `app/`+`tests/` and the git-diff guard on `alembic/versions/`.
-- Renaming the persisted LangGraph state channel key (carve-out 10) — breaks resume of threads paused at a HITL interrupt across the cutover; mitigated by draining interrupts pre-deploy and accepting restart of any stragglers (state is conversation-scoped/ephemeral).
-- Sheer size (~150 app + ~120 test files) — mitigated by per-wave scripted rename + the empty-diff and residual-grep gates.
-
-## Out of scope (later phases / umbrella)
-
-- Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella.
-- Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred.
-- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04-capabilities.md](04-capabilities.md)).
-- Enum VALUE migration and observability-key rename — deliberately deferred announced changes.
-
----
-
-## Implementation record (as-built)
-
-Whole-PR footprint (vs `ci_mvp`): **452 files, +5317/-4781** — 325 under `app/`, 124 under `tests/`, and **exactly 2** `alembic/versions/` files (`168` idempotency + `170` rename), confirming the immutable-replay-log carve-out (#9) held.
-
-The plan's Waves A–F were executed in order, as described above; what follows records only what diverged from that design and the evidence that it landed cleanly.
-
-### Deviations from the plan
-
-- **Bulk rename executed via scoped Python codemods** (multiple passes), strictly limited to `app/` + `tests/` (never `alembic/versions/`), instead of IDE/`ast-grep`. Same scope rule and same carve-out exclusions as designed.
-- **camelCase `searchSpaceId` -> `workspaceId`**: a camelCase payload key (in `app/agents/chat/multi_agent_chat/main_agent/middleware/kb_persistence/middleware.py`) was renamed too — decided internal/transient, so it follows the hard cutover. The plan enumerated snake/Pascal/Title/hyphen variants; this camelCase variant was caught in a follow-up codemod pass (residual `searchSpaceId` in `app/` now = 0).
-- All other carve-outs honored exactly as decided (below).
-
-### Carve-outs as-shipped (verified by residual grep)
-
-KEPT (intentional):
-- Enum VALUES `'SEARCH_SPACE'` (`ConnectionScope`, `ChatVisibility`) — value migration deferred (carve-out 1).
-- Celery wire task names `"delete_search_space_background"`, `"ai_sort_search_space"` (carve-out 2). Python functions renamed (`ai_sort_workspace_task`, `delete_workspace_task`); producer and worker agree because dispatch is via `.delay()` on the task object (no hardcoded `send_task("ai_sort_search_space")`).
-- OTel/metric key `search_space.id` (carve-out 5) — 3 sites (`observability/otel.py` x2, `observability/metrics.py`); dashboards/alerts depend on it.
-- API error code `"SEARCH_SPACE_FORBIDDEN"` (external contract string).
-- `SearchSourceConnector` class/table (carve-out 8, unrelated taxonomy).
-- Historical `alembic/versions/*` (carve-out 9, immutable replay log).
-
-RENAMED (accepted transient deploy cost):
-- Redis keys `surfsense:spawn_paused:{workspace_id}`, `ai_sort:workspace:{workspace_id}:lock` (carve-out 3) + `.env.example` runbook.
-- Event payload key (carve-out 4) and LangGraph state-channel key (carve-out 10) -> `workspace_id`; default seed name -> `"My Workspace"` (carve-out 11).
-
-### Verify current state (re-runnable)
-
-Each line is a command + the last captured result. Re-run to confirm the *current* truth instead of trusting the snapshot. Run from `surfsense_backend/`.
-
-- **Full backend suite** — `uv run pytest tests/unit tests/integration`
- → last (2026-06-27): `3016 passed, 1 skipped` in ~101s. The skip is a Windows-only event-loop test (`tests/unit/tasks/test_celery_async_runner.py:302`).
-- **Schema drift @170** — `AUTH_TYPE=LOCAL DATABASE_URL=…surfsense_oldshape uv run alembic check`
- → last (2026-06-27): `No new upgrade operations detected.`
-- **Route hard-cutover** — load `app.app:app` and inspect `{r.path for r in app.routes}`
- → last (2026-06-27): legacy `/searchspaces` `/search-spaces` `/search-space` = 0 each; canonical `/workspaces` = 26 (of 278 routes).
-- **Residual symbols** — `rg -ni 'search.?space' app`
- → last (2026-06-27): 52 hits, all carve-outs (enum `SEARCH_SPACE` values dominate; 3× OTel `search_space.id`; 2× Celery wire names; 1× `SEARCH_SPACE_FORBIDDEN`). No stray attribute/relationship/route residue.
-- **`alembic/versions/` immutability** — `git diff --stat ci_mvp..HEAD -- surfsense_backend/alembic/versions`
- → last (2026-06-27): only `168` + `170` (guard against carve-out 9).
-
-### Known caveats at deploy (team-acknowledged)
-
-- **Zero-cache replica reset** required on deploy (`ZERO_AUTO_RESET`) — consequence of the schema rename.
-- **From-scratch `alembic upgrade head` stays pre-existing-broken** (rev 23 conflict); the existing-DB `169 -> 170` path is verified. Separate baseline-squash task.
-- **Planned-downtime deploy**: drain Celery/event queues and restart workers before cutover (renamed event payload / LangGraph state-channel / Redis keys). In-flight state across the boundary is discarded by design — acceptable given the downtime window.
-- **Live Celery worker not exercised end-to-end** (the broker is mocked in tests). The task code paths are covered by unit/integration tests; a real worker + Redis smoke is the one verification not yet run.
-- **Clients are intentionally broken** by the hard cutover until the frontend/satellite umbrellas land — backend correctness is verified via the suite + OpenAPI/route introspection + direct API calls, not the old UI.
diff --git a/plans/backend/03a-crawler-core.md b/plans/backend/03a-crawler-core.md
deleted file mode 100644
index 7460b5be7..000000000
--- a/plans/backend/03a-crawler-core.md
+++ /dev/null
@@ -1,145 +0,0 @@
-# Phase 3a — WebURL Crawler core (Scrapling-only) + success semantics
-
-> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> Sibling subplans: `03b-proxy-expansion.md`, `03c-crawl-billing.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md`, `03f-undetectability-testing.md` (manual scorecard).
-
-> **Implementation note (applies to all Phase-3 plans).** Phases 1–2 are **SHIPPED** (2026-06-27) — `SearchSpace`→`Workspace` and `search_space_id`→`workspace_id` are renamed everywhere, so the **live code already says `workspace_*`**. Where citations below use the **old** `search_space_*`/`SearchSpace` names (written pre-rename), substitute the `workspace_*` equivalent and **grep the new name** (grepping the old name now returns nothing). Apply every edit by **symbol/grep** (e.g. `firecrawl_api_key`, `crawl_url`, `FIRECRAWL_API_KEY`), **not** by the absolute line numbers cited here — the rename (and `03a`'s own Firecrawl removal) shifted them.
-
-> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `5c36cd3`).** All 5 work items done: Firecrawl removed repo-wide, `crawl_url` is the 3-tier Scrapling ladder returning `CrawlOutcome` (`impersonate="chrome"` on the static tier + `solve_cloudflare=True` on the stealthy tier), `crawls_succeeded` added to the indexer (positional return unchanged — surfaced in task metadata), both `scrape_webpage` tools updated, unit tests added/passing. Crawler relocated to `app/proprietary/web_crawler/connector.py` under the non-Apache-2 boundary (see umbrella decisions log + `app/proprietary/README.md`).
-
-## Objective
-
-Make the Universal WebURL Crawler a **single-framework (Scrapling) component** with deterministic per-URL outcome semantics and a clean, explicit **"successful crawl" signal** that Phase 3c bills on.
-
-Two hard requirements from the decisions log:
-
-1. **Remove Firecrawl entirely.** No other scraping framework now or planned. Scrapling's `StealthyFetcher` can bypass Cloudflare Turnstile when invoked with `solve_cloudflare=True` (see tier design below); captcha-tools (`03d`) covers reCAPTCHA/hCaptcha, and `03e` adds the broader stealth-hardening that makes the crawler "undetectable" as far as free tooling reaches.
-2. **One billable unit = one URL that yields usable extracted content**, regardless of how many internal fallback tiers ran. `03a` must expose that signal; `03c` meters it.
-
-This subplan does NOT touch proxy rotation (→ `03b`), credit metering (→ `03c`), or captcha (→ `03d`).
-
-## Current state (cited)
-
-### The crawler
-
-> **Post-implementation note (location moved).** 03a is implemented. The crawler now lives at `app/proprietary/web_crawler/connector.py` (public API re-exported from `app/proprietary/web_crawler/__init__.py`), under the non-Apache-2 license boundary `app/proprietary/` (see `app/proprietary/README.md` + `LICENSE`). Importers use `from app.proprietary.web_crawler import WebCrawlerConnector, CrawlOutcomeStatus`. The file:line references below describe the *original* `app/connectors/webcrawler_connector.py` surface that 03a refactored away (kept for historical context).
-
-`app/connectors/webcrawler_connector.py` — `WebCrawlerConnector.crawl_url()` is a 4-tier fallback ladder:
-
-1. **Firecrawl** (premium, if `firecrawl_api_key` set) — `_crawl_with_firecrawl()` (lines 89–100, 223–272), imports `from firecrawl import AsyncFirecrawlApp` (line 22).
-2. Scrapling `AsyncFetcher` (static HTTP, curl_cffi) — `_crawl_with_async_fetcher()` (lines 274–310).
-3. Scrapling `DynamicFetcher` (browser, run in a thread) — `_crawl_with_dynamic()` (lines 312–339).
-4. Scrapling `StealthyFetcher` (patchright-Chromium anti-bot, run in a thread) — `_crawl_with_stealthy()` (lines 341–369).
-
-> **Engine note (do not say "Camoufox").** As of the pinned `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`), `StealthyFetcher` is **"completely stealthy built on top of Chromium"** (`references/Scrapling/scrapling/fetchers/stealth_chrome.py:8`) driven by **patchright** (`references/Scrapling/scrapling/engines/_browsers/_stealth.py:8–9`), **not** Camoufox — Scrapling removed Camoufox (zero matches in the 0.4.9 tree; `uv.lock` carries `patchright`, no `camoufox`). The default Playwright `channel` is `"chromium"` (patchright), or `"chrome"` when `real_chrome=True` (`_browsers/_base.py:469`). Stealth = the compiled-in flag set `DEFAULT_ARGS + STEALTH_ARGS` (`engines/constants.py:24–99`, incl. `--disable-blink-features=AutomationControlled` `:94`) + a persistent context by default (`_stealth.py:90–93`) — this is **runtime/config-level** stealth, which sets the realistic ceiling (see `03e`).
->
-> **Scrub the in-code "Camoufox" mentions too.** The connector still carries stale Camoufox wording in its own docstrings — the module header tier list (`webcrawler_connector.py:10–12`) and `_crawl_with_stealthy`'s docstring (`:343` "StealthyFetcher (Camoufox)"). Fix these in the 03a refactor so the code matches reality (patchright-Chromium).
-
-Extraction is Trafilatura HTML→markdown in `_build_result()` (lines 371–469). Every Scrapling tier passes `proxy=get_proxy_url()` (lines 287, 329, 359). `crawl_url()` returns a tuple `(result_dict | None, error | None)`; `result_dict` has `content` / `metadata` / `crawler_type`.
-
-### The two call sites
-
-- **Type-1 indexer (billable path):** `app/tasks/connector_indexers/webcrawler_indexer.py` reads `FIRECRAWL_API_KEY` from connector config (line 115), builds `WebCrawlerConnector(firecrawl_api_key=api_key)` (line 139), and calls `crawler.crawl_url(url)` per URL (line 297). It already tracks `documents_indexed` / `documents_updated` / `documents_skipped` / `documents_failed` / `duplicate_content_count` (lines 167–171), but **none of these is a clean "crawl succeeded" count** — duplicates and unchanged docs are bucketed as skipped even though a successful fetch happened.
-- **Chat scrape tool (ad-hoc):** `main_agent/tools/scrape_webpage.py` (line 232) and `subagents/builtins/research/tools/scrape_webpage.py` (line 226) build `WebCrawlerConnector(firecrawl_api_key=...)` and call `crawl_url(url, formats=["markdown"])`. The `formats` arg is Firecrawl-only (declared at `crawl_url` signature line 59; docstring line 72: "only for Firecrawl").
-
-### Firecrawl threading (the cross-cutting surface)
-
-Firecrawl's API key is plumbed end-to-end and must be removed everywhere:
-
-| Layer | File:line | What to remove |
-|-------|-----------|----------------|
-| Crawler | `app/connectors/webcrawler_connector.py` | `firecrawl` import (22), `firecrawl_api_key`/`use_firecrawl` ctor (36–46), `set_api_key()` (48–56), tier-1 block (89–100), `_crawl_with_firecrawl()` (223–272), `formats` param + Firecrawl mentions in docstrings |
-| Dependency | `pyproject.toml:45` | `"firecrawl-py>=4.9.0"` (+ regenerate `uv.lock`). Scrapling stays (`pyproject.toml:91` → `"scrapling[fetchers]>=0.4.9"`) |
-| Connector config | `app/utils/validators.py` | `FIRECRAWL_API_KEY` from `WEBCRAWLER_CONNECTOR.optional`/`validators` (573–580); delete `validate_firecrawl_api_key_format()` (472–478) |
-| Indexer | `app/tasks/connector_indexers/webcrawler_indexer.py` | `api_key = connector.config.get("FIRECRAWL_API_KEY")` (115), `use_firecrawl` log field (135), pass-through to ctor (139) |
-| Chat setup | `app/tasks/chat/streaming/flows/shared/pre_stream_setup.py` | `setup_connector_and_firecrawl()` returns `firecrawl_api_key` (1–30) — collapse to connector-service-only |
-| Chat orchestrators | `new_chat/orchestrator.py` (86, 378, 412, 665), `resume_chat/orchestrator.py` (65, 317, 347, 483) | `firecrawl_api_key` threading |
-| Automations | `automations/actions/builtin/agent_task/dependencies.py` (19, 34, 85, 103), `.../invoke.py` (174) | `firecrawl_api_key` dep field |
-| Main agent | `main_agent/runtime/factory.py` (71, 142), `main_agent/tools/registry.py` (36) | `firecrawl_api_key` param/wiring |
-| Scrape tools | `main_agent/tools/scrape_webpage.py` (170, 232), `research/tools/scrape_webpage.py` (164, 226), `research/tools/index.py` (28) | `firecrawl_api_key` factory arg + ctor arg |
-| Tests | `tests/unit/automations/actions/builtin/agent_task/test_dependencies.py` (44, 81) | `firecrawl_api_key == "fc-key"` assertion + fake |
-
-> **Not a global env var.** `FIRECRAWL_API_KEY` is **not** in `.env.example` and there is no `Config.FIRECRAWL_API_KEY`. It lives only inside each WebCrawler connector's `config` JSON — the validator reads it from that dict (`validators.py:474`). So removal is code-only; **no env/docs change**. Existing `WEBCRAWLER_CONNECTOR` rows may still carry a now-dead `FIRECRAWL_API_KEY` key in `config` — harmless (it's simply ignored), optionally scrubbed by a tiny data migration if we want clean rows.
->
-> **Signature change, not just deletions.** Removing `firecrawl_api_key` mutates the `RuntimeDeps`-style dataclass (`agent_task/dependencies.py:34`) and the agent runtime factory/tool factories — a coordinated signature change across the chat + automations call graph, so land it atomically.
-
-### Runtime/deps already in place
-
-- `Dockerfile:112–115` runs `RUN scrapling install`; the `scrapling[fetchers]` extra pulls playwright/patchright. **No new install step needed** once Firecrawl is gone. *(Accuracy fix: the Dockerfile comment says "patchright Chromium + Camoufox", but `scrapling install` in 0.4.9 only fetches Chromium — `references/Scrapling/scrapling/cli.py:122,131` runs `playwright install chromium` + `install-deps chromium`, no Camoufox. Drop the stale "+ Camoufox" wording from the comment when touching this file. `03e` may add `install-deps` extras for fonts/Xvfb.)*
-- Proxy is read via `app/utils/proxy/get_proxy_url()` (`__init__.py:13`), backed by the `PROXY_PROVIDER` registry (`config/__init__.py:983`). `03a` leaves this single-URL model untouched (rotation is `03b`).
-
-## Target design
-
-### Tier ladder (Scrapling-only)
-
-`crawl_url()` becomes a 3-tier ladder, preserving the existing thread-offload + `NotImplementedError` handling for the browser tiers (Windows `SelectorEventLoop` cannot spawn subprocesses — lines 134–141, 161–168):
-
-1. `AsyncFetcher.get(...)` — fast static HTTP. **TLS gap to close:** the current call passes `stealthy_headers=True` but **not** `impersonate` (`webcrawler_connector.py:284–289`), so its TLS ClientHello is curl_cffi's *default* JA3 — trivially bot-flagged and incoherent with the browser tiers' UA. Add an `impersonate="chrome"` profile here (Scrapling's static engine accepts it — `references/Scrapling/scrapling/engines/static.py:36–47`). Tracked as a lever in `03e §2b` and validated by `03f §S3`.
-2. `DynamicFetcher.fetch(...)` — full browser (via `asyncio.to_thread`).
-3. `StealthyFetcher.fetch(...)` — patchright-Chromium anti-bot, last resort. Enable Cloudflare solving here by passing **`solve_cloudflare=True`** — a documented `StealthyFetcher.fetch` kwarg ("Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response", `references/Scrapling/scrapling/fetchers/stealth_chrome.py:38`; it's a `StealthSession` TypedDict key passed via `**kwargs`). The current stealthy call (connector lines 354–360) passes `headless`/`network_idle`/`block_ads`/`proxy` but **not** `solve_cloudflare`, so this is a real behavior add. (Note: `solve_cloudflare` runs the full browser challenge loop, so it's correctly scoped to the last-resort tier only.)
-
-Trafilatura extraction (`_build_result`) and `format_to_structured_document()` are unchanged.
-
-### Explicit outcome model
-
-Replace the implicit `(dict|None, str|None)` contract with an explicit outcome so callers (indexer, chat tool, and `03c` metering) agree on what "success" means. Proposed:
-
-```python
-class CrawlOutcomeStatus(str, Enum):
- SUCCESS = "success" # a tier returned usable extracted content
- EMPTY = "empty" # fetched, but no usable content after ALL tiers
- FAILED = "failed" # invalid URL or every tier errored
-```
-
-`crawl_url()` returns a small dataclass `CrawlOutcome(status, result, error, tier)` — **commit to the dataclass** (not a tuple): `03c` keys billing off `status == SUCCESS`, and Phase 6's fetch-only path (`06-pipelines-exec.md`) consumes `outcome.status` / `outcome.result` / `outcome.error` as attributes, so a tuple form would break that consumer. The **billable success predicate is single-sourced**: `status == CrawlOutcomeStatus.SUCCESS`. Picking a dataclass (over a tuple) also leaves room for later subplans to **append fields without breaking callers** — `03d` adds `captcha_attempts` / `captcha_solved` for per-attempt billing, and `03e`'s block classifier can attach a `block_type`. (This is distinct from the indexer's positional return, which must stay 2-tuple — see the wrapper note below.)
-
-| Outcome | When | Billable (`03c`)? | Document status (indexer) |
-|---------|------|-------------------|---------------------------|
-| `SUCCESS` | a tier extracted usable content (`_build_result` returned a dict) | **Yes — 1 unit** | `ready` (or unchanged/duplicate, see note) |
-| `EMPTY` | every tier was reached but none produced usable extracted content (the static tier also treats HTTP ≥ 400 as a miss and falls through — connector lines 292–301; the browser tiers attempt extraction regardless of status) | No | `failed("No content extracted")` |
-| `FAILED` | invalid URL, or all tiers raised | No | `failed()` |
-
-**Billing-policy note for `03c`:** success is the *crawl* succeeding (we fetched + extracted), independent of downstream KB dedupe. The indexer currently marks unchanged content as `skipped` (`webcrawler_indexer.py:341–347`) and cross-connector duplicates as `failed` (`:350–369`) — those still represent a **successful crawl** and should bill. `03c` must count `CrawlOutcomeStatus.SUCCESS`, not `documents_indexed`. Flagging here; final call lives in `03c`.
-
-### Extensibility seam (the tier ladder is a strategy chain)
-
-The 3-tier ladder is the first instance of a deliberate **`FetchStrategy` seam**: an *ordered list of strategies*, each `(url, ctx) -> CrawlOutcome`, tried in order until one returns `SUCCESS`. `crawl_url()` owns the chain; **every caller depends only on `CrawlOutcome`, never on which strategy produced it** (the indexer, the chat tool, and `03c` metering already do — keep that invariant sacred). This is what lets the moat grow without rework:
-
-- `03d` (captcha) attaches by escalating to the StealthyFetcher strategy with a `page_action` token-injector — a *parameterization* of the last tier, not a new caller contract.
-- `03e` (stealth-hardening) tunes/adds strategies (humanize, headed/Xvfb, persistent profiles, fingerprint flags) **behind the same return type**.
-- A future **paid-unblocker tier** (deferred — see `03e`) is just one more strategy appended last, flippable by config; no caller changes.
-- Future **platform actors** (Phase 8) reuse the same fetch strategies under their own structured extractors.
-
-MVP scope here is only the 3 Scrapling tiers + the `CrawlOutcome` contract; the seam is a design constraint (keep tiers pluggable + callers outcome-only), **not** a call for a heavyweight Strategy framework now.
-
-### Success counter for the indexer
-
-Add an explicit `crawls_succeeded` counter in `index_crawled_urls()` incremented whenever `crawl_url` returns `SUCCESS` (right after line 297's call, before the dedupe/unchanged branches), and surface it in the task-success metadata (lines 455–466). `03c` meters against this **in-function** counter (it charges inside the indexer — the count does not need to escape via the return).
-
-> **Do NOT widen the positional return tuple.** The shared `_run_indexing_with_notifications` wrapper unpacks every indexer's return **by length** (`search_source_connectors_routes.py:1499–1507`: `if len(result) == 3: a,b,c = result else: a,b = result`) — a 3-tuple would mislabel `crawls_succeeded` as `documents_skipped`, a 4-tuple would raise `ValueError`. Keep the existing `(total_processed, error)` shape. **Phase 6** (`06-pipelines-exec.md`) later exposes `crawls_succeeded`/`documents_indexed`/`crawls_attempted` to the pipeline run engine via an optional `stats` **out-param** (plus `folder_id`/`urls`/`bill`), not via the return — so `03a` only needs the counter + metadata here.
-
-### Chat scrape tool
-
-Drop the Firecrawl-only `formats=["markdown"]` arg (markdown is already the Trafilatura default). The tool keeps returning its asset dict; map the new outcome onto the existing `error` / `content` shape (lines 235–269) with no behavioral change for the agent.
-
-## Work items
-
-1. **Rip out Firecrawl** across the surface table above (crawler, dep, validators, indexer, chat plumbing, automations, tests) — code-only, no env/docs change.
-2. **Refactor `crawl_url`** to the 3-tier Scrapling ladder + explicit `CrawlOutcome`; enable `solve_cloudflare` on the stealthy tier.
-3. **Add `crawls_succeeded`** counting in `webcrawler_indexer.py` + expose in **task metadata only** (03c bills off the in-function counter). **Do not change the positional return tuple** (the shared wrapper unpacks by length; Phase 6 adds a `stats` out-param).
-4. **Update both `scrape_webpage` tools** to drop `firecrawl_api_key` + `formats`.
-5. **Tests:** unit tests for `crawl_url` outcomes (mock Scrapling fetchers → SUCCESS/EMPTY/FAILED); update `test_dependencies.py`; assert the indexer's `crawls_succeeded` count.
-
-## Risks / trade-offs
-
-- **Loss of a managed fallback.** Firecrawl was a hosted last resort for hostile anti-bot sites. Mitigation: the in-house stack — `StealthyFetcher` + Cloudflare solving (this plan), `03b` proxy rotation, `03e` stealth-hardening (humanize/headed/profiles/fonts + block-classifier), `03d` captcha solving — plus a **deferred paid-unblocker tier** behind the seam for the hostile residual. Realistic ceiling: this defeats Cloudflare + the long tail of moderate anti-bot, but runtime-level (patchright) stealth does **not** reliably beat top-tier fingerprinting (DataDome/Kasada/reCAPTCHA-Enterprise) — that's the deferred paid tier's job (`03e`). Acceptable per the decisions log (single-framework intent + in-house moat).
-- **Browser tiers on dev/Windows.** `DynamicFetcher`/`StealthyFetcher` need subprocess support; the existing `to_thread` + `NotImplementedError` guards (lines 134–141, 161–168) are preserved so static-only crawling still works in `uvicorn --reload`.
-- **`uv.lock` churn.** Removing `firecrawl-py` requires a lockfile regen + image rebuild; no new runtime deps are added.
-
-## Out of scope (hand-offs)
-
-- Proxy provider expansion + rotation → `03b`.
-- Crawl credit metering on `CrawlOutcomeStatus.SUCCESS` → `03c`.
-- reCAPTCHA/hCaptcha solving via captcha-tools → `03d` (**now active**, sequenced after `03e`). Cloudflare Turnstile stays in-framework (Scrapling).
-- Stealth-hardening (humanize, headed/Xvfb, persistent profiles, fonts, geoip locale/tz, block-classifier + per-domain strategy memory) and the deferred paid-unblocker tier → `03e`.
-- Whether ad-hoc **chat** scrapes are billed (vs only pipeline crawls) → decided in `03c`.
diff --git a/plans/backend/03b-proxy-expansion.md b/plans/backend/03b-proxy-expansion.md
deleted file mode 100644
index 64de97ac8..000000000
--- a/plans/backend/03b-proxy-expansion.md
+++ /dev/null
@@ -1,130 +0,0 @@
-# Phase 3b — Proxy provider expansion + rotation
-
-> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> Depends on `03a-crawler-core.md` (Scrapling-only crawler). Siblings: `03c-crawl-billing.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md`.
-
-> **Implementation note.** Same convention as `03a`: Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Crucially, `03a` is **now IMPLEMENTED**: the crawler moved to **`app/proprietary/web_crawler/connector.py`** (non-Apache-2 boundary) with Firecrawl removed and `crawl_url` refactored — the three Scrapling tiers now pass `proxy=get_proxy_url()` at **267/309/342** (not the pre-refactor 287/329/359). Locate code by **symbol/grep** against the new path, not absolute lines.
-
-> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `6226012`).** `CustomProxyProvider` added (single URL or rotating pool via Scrapling `ProxyRotator`) and registered as `"custom"`; `Config.CUSTOM_PROXY_URL`/`CUSTOM_PROXY_URLS` + `.env.example` docs; `is_pool_backed` added to the `ProxyProvider` ABC (default `False`) + a zero-arg `is_pool_backed()` package helper. The crawler does a **bounded, one-per-tier** `is_proxy_error` rotation-retry gated on `is_pool_backed()` — single-endpoint providers (incl. server-side-rotating `anonymous_proxies`) no-op. The zero-arg getter contract is unchanged for all consumers (crawler/YouTube/chat tools). Tests: provider (single/pool/dedupe/empty/playwright), registry (custom/anonymous/unknown-fallback), crawler rotation (retry / no-retry / non-proxy) — **16 passing**.
-
-## Objective
-
-Let the WebURL Crawler (and every other proxy consumer) run behind **either** the existing `anonymous_proxies` gateway **or** a new BYO `CustomProxyProvider`, chosen by a **single, app-wide** `Config.PROXY_PROVIDER`. When the active provider is backed by a **pool of endpoints**, rotate across them client-side with a bounded retry. **No branded vendors and no per-connector/per-crawl provider divergence** (resolved decisions): the whole app uses one provider.
-
-The provider abstraction already exists and is clean; the real gaps are just (1) only `anonymous_proxies` is registered (no BYO option), and (2) there is **no client-side rotation/retry** when a provider exposes multiple endpoints. The "process-global, env-only" shape is exactly what we want for a single-provider app, so we keep it.
-
-## Current state (cited)
-
-### A clean but process-global abstraction
-
-`app/utils/proxy/` is already a "subclass + register" provider package:
-
-- `base.py` — `ProxyProvider` ABC: `get_proxy_url()` (canonical `http://user:pass@host:port` for Scrapling/curl_cffi), `get_playwright_proxy()`, `get_requests_proxies()` (built from the URL by default).
-- `registry.py` — `_PROVIDERS` dict keyed by provider `name`; `get_active_provider()` resolves `Config.PROXY_PROVIDER` and **caches a single instance process-wide** (`_active_provider`, lines 23, 26–44). Only `AnonymousProxiesProvider` is registered (lines 17–19).
-- `__init__.py` — zero-arg module helpers `get_proxy_url()` / `get_playwright_proxy()` / `get_requests_proxies()` + a `get_residential_proxy_url()` back-compat alias, all delegating to `get_active_provider()`.
-- `providers/anonymous_proxies.py` — the one vendor. Note it's a **server-side rotating gateway**: host is `rotating.dnsproxifier.com:PORT` and the rotation `type`/location are encoded into a base64 "password" dict (lines 23–51). So per-request IP rotation already happens **upstream**; the client sends one static endpoint.
-- `app/utils/proxy_config.py` — a thin back-compat shim re-exporting the package (kept for old import paths).
-
-Config knobs: `Config.PROXY_PROVIDER` (default `"anonymous_proxies"`) + `RESIDENTIAL_PROXY_{USERNAME,PASSWORD,HOSTNAME,LOCATION,TYPE}` (`config/__init__.py:983–992`; documented commented-out in `.env.example:300–309`).
-
-### Who consumes the proxy (blast radius)
-
-The zero-arg getters are used in **more than just the crawler**, so any signature change must stay backward-compatible:
-
-| Consumer | File:line | Uses |
-|----------|-----------|------|
-| WebURL crawler (all 3 Scrapling tiers) | `proprietary/web_crawler/connector.py` (267, 309, 342) | `get_proxy_url()` |
-| YouTube transcript route | `routes/youtube_routes.py` (78, 119) | `get_proxy_url()` |
-| YouTube processor (indexing) | `tasks/document_processors/youtube_processor.py` (223, 229) | `get_requests_proxies()`, `get_proxy_url()` |
-| Chat scrape tools (YouTube branch) | `main_agent/tools/scrape_webpage.py` (80, 93), `research/tools/scrape_webpage.py` (74, 87) | `get_requests_proxies()`, `get_proxy_url()` |
-
-**Implication:** the whole app shares **one** global provider, so every consumer keeps calling the zero-arg getters **unchanged**. The only behavior change is (a) *which* provider class those getters resolve to (selected by `Config.PROXY_PROVIDER`) and (b) optional client-side rotation when that provider is pool-backed. No caller passes a key; no signatures change.
-
-**Shape note (verified):** `get_playwright_proxy()` has **no consumers** anywhere — only the ABC/impl/exports define it. All three crawler tiers and YouTube's `AsyncFetcher.get` consume the **string** form (`get_proxy_url`); browser fetchers accept a string proxy (`references/Scrapling/.../stealth_chrome.py:47` — "it can be a string or a dictionary"). The dict form (`get_requests_proxies`) is consumed only by YouTube-transcript fetches (`youtube_processor.py:223`, plus the chat tools' YouTube branch `main_agent/tools/scrape_webpage.py:80` and `research/tools/scrape_webpage.py:74`) — none of which are crawler tiers. **Conclusion:** rotation only needs the **string** shape; the playwright dict can be ignored (or left as-is on providers) rather than threaded through rotation.
-
-### Scrapling's rotation primitive
-
-`ProxyRotator(proxies: list, strategy=cyclic_rotation)` is a thread-safe cycler over a **static list** of proxy URLs/dicts (`get_proxy()`, `proxy_rotation.py:88–92`) plus `is_proxy_error(exc)` (`:27–30`) that matches proxy failure strings (`net::err_proxy`, `connection refused`, …). It does **not** manage credentials or sessions — it just hands out the next endpoint. So it's only useful when we have a **pool of distinct endpoints**; it adds nothing for a single server-side-rotating gateway like `anonymous_proxies`.
-
-Both are **publicly importable**: `from scrapling.engines.toolbelt import ProxyRotator, is_proxy_error` (re-exported in `engines/toolbelt/__init__.py:1–3`). The thread-safe `Lock` (`:71,90`) makes `ProxyRotator` safe to call from the browser tiers that `03a` runs via `asyncio.to_thread`. Version is guaranteed: the pinned floor is `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`) and these APIs exist as of `0.4.9` (`scrapling/__init__.py:2`).
-
-## Selection scope (resolved)
-
-The user expects a **single proxy provider across the entire app**, so 03b's selection scope is simply the **global, env-configured provider** — no per-connector or per-crawl override is built now. That is both the fastest path and, with one thin seam, the most scalable later:
-
-- All resolution stays behind today's `get_active_provider()` (env-selected, process-cached). One provider, app-wide.
-- "Scalable in future": if per-capability / per-Tracker proxying is ever wanted (revamp Phase 4a/5+), it layers on as an **optional argument** to a resolver (`resolve_proxy(override=...)`) with **zero** changes to existing call sites — but it is explicitly **not** implemented in 03b (YAGNI for a single-provider app).
-
-## Target design
-
-### 1. Registry: register the BYO provider; keep single-provider selection
-
-The existing `get_active_provider()` (env-selected via `Config.PROXY_PROVIDER`, cached process-wide in `_active_provider` — `registry.py:23,26–44`) is exactly the single-provider model we want — keep it. The only change is to **register `CustomProxyProvider`** so `PROXY_PROVIDER` can select it:
-
-- `_PROVIDERS["anonymous_proxies"] = AnonymousProxiesProvider` (existing) + `_PROVIDERS["custom"] = CustomProxyProvider` (new).
-- `get_active_provider()` resolves the **one** active provider from env and caches it — **no keyed multi-provider coexistence** (`get_provider(key)` is intentionally NOT added; YAGNI for a single-provider app).
-- Unknown `PROXY_PROVIDER` → existing warn-and-fallback path (`registry.py:35–41`) is unchanged.
-
-### 2. Provider config source: env only
-
-Credentials come from **env**, full stop (single global provider):
-
-- `anonymous_proxies` → existing `RESIDENTIAL_PROXY_*`. Unchanged.
-- `custom` → new env knobs (one URL or a pool): `CUSTOM_PROXY_URLS` (comma-separated) and/or `CUSTOM_PROXY_URL`.
-
-No per-connector proxy keys, and **no** connector-config validator changes (that was the per-connector path, now dropped).
-
-### 3. New provider: `CustomProxyProvider` (BYO) — the only addition
-
-- Accepts a raw proxy URL **or** a list of URLs from env. Covers "bring your own proxy / our own pool" with no vendor-specific auth assumptions.
-- Implements just `get_proxy_url()`; the base derives `get_requests_proxies()` from it (`base.py:37–46`). When configured with a **pool**, `get_proxy_url()` returns the *next* endpoint from an internal `ProxyRotator` (see §4) — so rotation is transparent to every caller of the zero-arg getter.
-- **No branded vendors** (resolved): Webshare/BrightData/Smartproxy/etc. are not shipped. A user who wants any specific vendor points `CustomProxyProvider` at that vendor's endpoint(s) via `CUSTOM_PROXY_URLS`.
-
-### 4. Rotation + retry (only when the active provider is pool-backed)
-
-- If `CustomProxyProvider` is configured with **multiple** URLs, it wraps them in Scrapling's `ProxyRotator` (cyclic default, thread-safe `Lock` — `proxy_rotation.py:71,88–92`) and returns the next endpoint per `get_proxy_url()` call. Because `03a`'s browser tiers run under `asyncio.to_thread`, the rotator's lock keeps this safe.
-- The crawler adds a **bounded** client-side retry: on a tier failure where `is_proxy_error(exc)` is true (`proxy_rotation.py:27–30`), it re-reads `get_proxy_url()` (next endpoint) and retries **that tier once** before falling through. One rotation-retry per tier — no unbounded fan-out on billable crawls.
-- Single-endpoint providers (`anonymous_proxies`, already server-side rotating; or `custom` with one URL) return the same endpoint every call, so the retry is a harmless no-op for them.
-
-### 5. Crawler stays on the zero-arg getter (no injection needed)
-
-Because there is one global provider and rotation lives **inside** it, the crawler does **not** need a proxy injected per crawl:
-
-- All three Scrapling tiers keep calling `get_proxy_url()` (which yields the rotating value when pool-backed). The only crawler edit is the bounded `is_proxy_error` retry from §4.
-- **No change** to YouTube/chat-tool consumers — same zero-arg getters, same global provider.
-- Future seam (NOT built): if per-capability / per-Tracker proxying is ever needed, resolve a provider from the capability/Tracker config and pass its `get_proxy_url` into the crawl — an additive optional arg, no change to today's call sites.
-
-## Config / env changes
-
-- `config/__init__.py:983–992` + `.env.example:300–309`: add `CUSTOM_PROXY_URLS` (comma-separated pool) and/or `CUSTOM_PROXY_URL` for the `custom` provider; keep all existing `RESIDENTIAL_PROXY_*` working unchanged. `PROXY_PROVIDER` now accepts `"custom"` in addition to `"anonymous_proxies"`.
-- **No** connector-config validator changes (the per-connector proxy path is dropped — single global provider).
-
-## Work items
-
-1. **`CustomProxyProvider`**: BYO single-URL or pool-of-URLs provider (reads `CUSTOM_PROXY_URL(S)`); internal `ProxyRotator` when pool-backed; register as `"custom"` in `_PROVIDERS`.
-2. **Rotation retry**: add a bounded (one-per-tier) `is_proxy_error` retry to the crawler's Scrapling tiers; single-endpoint providers no-op.
-3. **Config + docs**: `CUSTOM_PROXY_URL(S)` in `Config` + `.env.example`; document `PROXY_PROVIDER="custom"`.
-4. **Tests**: `CustomProxyProvider` single-URL vs pool; `ProxyRotator` cyclic order; `get_active_provider()` resolves `"custom"` and still warn-falls-back on unknown; crawler rotates once on `is_proxy_error` then falls through; `anonymous_proxies` + YouTube path unchanged (same endpoint each call).
-
-## Risks / trade-offs
-
-- **Backward compatibility of the getters.** The zero-arg `get_proxy_url()` contract is consumed in 4+ places; the design keeps it **completely intact** — no signature changes, no per-caller keys. Only the cached active provider's *implementation* changes when `PROXY_PROVIDER="custom"`.
-- **Over-rotation cost.** Rotation-retry is bounded to one extra attempt per tier so a billable crawl can't silently multiply upstream proxy usage.
-- **Single provider is a deliberate constraint.** One global provider app-wide (resolved). Per-capability/per-Tracker/per-connector selection is intentionally deferred behind a no-op seam (§5); not built now.
-- **Pool rotation under `to_thread`.** `ProxyRotator`'s `Lock` makes the rotating `get_proxy_url()` safe to call from the browser tiers `03a` offloads via `asyncio.to_thread`.
-
-## Resolved decisions
-
-- **Branded vendors → NONE.** Ship `CustomProxyProvider` (BYO) only; no Webshare/BrightData/Smartproxy/etc. subclasses. A user who wants a specific vendor points `CUSTOM_PROXY_URLS` at it.
-- **Selection scope → single global provider, app-wide.** No per-connector/per-crawl override is built; resolution is env-only via `Config.PROXY_PROVIDER`. A future per-capability/per-Tracker override is left as a no-op seam (§5) so it stays "scalable + fast to add later."
-- **Client-side rotation → built, but only active for a pool-backed `CustomProxyProvider`.** `anonymous_proxies` (server-side rotating) and single-URL custom configs skip it automatically. Rotation lives inside the provider so it's transparent to all callers.
-
-## Out of scope (hand-offs)
-
-- Per-**capability** / per-Tracker / per-connector proxy selection → deferred (revamp Phase 4a/5 *if ever needed*); §5 leaves a no-op seam, nothing is wired now.
-- Branded-vendor provider subclasses → not planned (use `CustomProxyProvider`).
-- **Static / sticky-session proxies (future).** A later capability will add **static proxy** support — sticky IPs held for the duration of a session — most likely paired with **authenticated/account-based scraping** to bypass logged-in platforms (the deferred platform connectors: LinkedIn, Instagram, etc.). This is a *different axis* from the rotating pool here: rotation maximizes IP diversity, whereas account bypass needs IP **stability** so a session/cookie stays bound to one IP. It is additive to this design — a new `ProxyProvider` (or a "sticky" mode/flag on `CustomProxyProvider`) registered under a new `PROXY_PROVIDER` key, with no change to the zero-arg getter contract — and stays consistent with the single-provider model (the active provider would be the static one when that workflow is selected). Build it alongside the platform connectors, not in Phase 3.
-- Crawl credit metering (proxy cost is absorbed into the flat `$1 / 1000 successful` price, **not** metered separately) → `03c`.
-- Captcha solving → `03d`.
-- **Geoip fingerprint coherence** (matching browser `locale`/`timezone_id` to the proxy's exit geo) → `03e`. 03b only owns *selecting* the proxy; `03e` consumes the chosen endpoint's geo. The provider's `RESIDENTIAL_PROXY_LOCATION` (`config/__init__.py`) is one input; resolving the actual exit IP's geo is `03e`'s job.
-- **Surfacing the crawl's chosen endpoint into the strategy.** Both `03d` (IP-bound captcha solves) and `03e` (geoip + sticky reuse) need the **exact** endpoint a crawl used, *not* a fresh `get_proxy_url()` call (which rotates on a pool-backed `CustomProxyProvider`). The capture-once seam lives in `03a`'s `FetchStrategy` context; 03b only guarantees the rotating getter and notes the consumers.
diff --git a/plans/backend/03c-crawl-billing.md b/plans/backend/03c-crawl-billing.md
deleted file mode 100644
index ae3073934..000000000
--- a/plans/backend/03c-crawl-billing.md
+++ /dev/null
@@ -1,140 +0,0 @@
-# Phase 3c — Crawl credit billing ($1 / 1000 successful requests)
-
-> **Status: ✅ IMPLEMENTED (`ci_mvp` @ `17bdb0682`).** `WebCrawlCreditService` added (mirrors `EtlCreditService`: gate → `check_credits` → `charge_credits`, plus static `billing_enabled`/`successes_to_micros`). Price is fully config-driven (`WEB_CRAWL_MICROS_PER_SUCCESS`, default `1000` = $1/1000; `WEB_CRAWL_CREDIT_BILLING_ENABLED`, default FALSE) — no hardcoded rate. Indexer wires owner-resolution + pre-flight block + audit-then-charge; both `scrape_webpage` tools fold one success into the turn accumulator (`call_kind="web_crawl"`). 20 new unit tests pass (service, config-driven price, indexer wiring, chat fold); `connector_indexers` suite green (120). Functional e2e (3a/3b/3c) green via `scripts/e2e_phase3_crawl_billing.py` (live proxy egress flip, static-tier crawl, chat fold = 1000µ, indexer bills owner 2×1000µ not trigger). Captcha per-attempt unit (`03d`) seam left open. **No migration** (`web_crawl` rides the existing `TokenUsage.usage_type` String(50)).
-
-> **⚠️ Billing surfaces reframed (2026-06-30 — Architecture correction + revamp adoption, `00-umbrella-plan.md`).** The implemented `WebCrawlCreditService` **stands unchanged**, but *where* it's called shifts, and the **charge moves to the capability executor** ([`04-capabilities.md`](04-capabilities.md)). Pipelines are dropped and `WEBCRAWLER_CONNECTOR` is retired, so the **connector/`webcrawler_indexer`** billing branch (described throughout this doc) is now **vestigial** (dormant, not deleted). The correct model: every `web.*`/`maps.*` verb **charges once per billable success inside its executor**, so all callers meter uniformly — chat, ongoing automation, REST, MCP. This closes a **code-verified gap**: `run_agent_task` (automation `agent_task`) establishes **no turn accumulator**, so the shipped `scrape_webpage` fold (no-op when `get_current_accumulator()` is `None`) bills **nothing** under automations today. The interactive **chat turn accumulator becomes an optional presentation fold** (still shows the crawl line on a chat turn), **not** the charging mechanism. Read "connector/pipeline crawls" below as historical context.
-
-> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Siblings: `03b-proxy-expansion.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md` (now active). `03d` extends this service with a **separate per-attempt captcha unit** (see §"Captcha billing seam" + `03d §4`).
-
-## Objective
-
-Charge **$1 per 1000 successful crawl requests** = **1000 micro-USD per successful crawl**, drawn from the existing unified credit wallet, **off by default for self-hosted/OSS** installs. The rate is **fully config-driven** — a single env/`Config` knob (`WEB_CRAWL_MICROS_PER_SUCCESS`) with **no hardcoded price anywhere** — so it can be retuned to any future rate with just an env change + restart, never a code edit (same convention as `MICROS_PER_PAGE`). $1/1000 is only the *default*. Two surfaces crawl, so both are metered:
-
-- **Connector / pipeline crawls** (the `webcrawler_indexer` path) → billed to the **workspace owner** via a dedicated `WebCrawlCreditService` (check + charge).
-- **Ad-hoc chat scrapes** (`scrape_webpage` tools) → the crawl cost is **folded into the chat turn's existing bill** (added to the turn accumulator), so it settles with the turn's normal premium finalize (`start_turn()` + `credit_finalize`). No second wallet hit. (Implies a second gate: only charged when the turn itself is a premium/billed turn — see §3.)
-
-> **Migration impact: none.** This phase adds config flags + code only. `TokenUsage.usage_type` is already a free-form `String(50)`, so the new `web_crawl` value needs **no Alembic migration or schema change**. (Per the standing guidance, the migration-sensitive work lives in Phases 1 and 5+; 03c is purely additive.)
-
-## The key realization (don't use `billable_call`)
-
-My first instinct was to add a flat reserve/finalize path around `TokenQuotaService` because `billable_call` settles at the LLM token accumulator (`acc.total_cost_micros`, `billable_calls.py:367`) — which is 0 for a non-LLM crawl, so `billable_call` would finalize at nothing.
-
-But there's already a **purpose-built precedent for per-unit, non-LLM wallet billing**: `app/services/etl_credit_service.py` (`EtlCreditService`). It deliberately **skips** the reserve/finalize dance and does a simple **gate → pre-check → post-charge**:
-
-- `billing_enabled()` → `config.ETL_CREDIT_BILLING_ENABLED` (`etl_credit_service.py:44–46`); **every method is a no-op when disabled** — that's what keeps self-hosted "effectively free" (`:9–11`, `:59–60`, `:84–85`, `:115–116`).
-- `pages_to_micros(pages, multiplier)` → `pages * multiplier * config.MICROS_PER_PAGE` (`:48–51`).
-- `check_credits(user_id, estimated_pages)` → raises `InsufficientCreditsError` if `required > available` where `available = balance - reserved` (`:76–102`).
-- `charge_credits(user_id, pages)` → `user.credit_micros_balance -= cost; commit; maybe_trigger_auto_reload(...)` (`:104–138`).
-
-It's wired into the **sibling connector indexers** exactly the way we need: `google_drive_indexer.py` (`558,562,605`), `dropbox_indexer.py` (`426,504,593`), `local_folder_indexer.py` (`639,698,864`) — instantiate `EtlCreditService(session)`, `check_credits(...)` before processing, `charge_credits(...)` after.
-
-**So 03c mirrors `EtlCreditService` with a crawl-specific unit and flag — no `billable_call`, no reserve/finalize.**
-
-## Current state (cited)
-
-- **Crawled URLs are billed nothing today.** `webcrawler_indexer.py` does not import `EtlCreditService` or touch the wallet (verified — it's absent from the file). The crawler extracts content directly via Trafilatura and bypasses the ETL page pipeline, so there's **no existing ETL charge** to double up with.
-- **Wallet model** (`token_quota_service.py`): the balance lives on `User.credit_micros_balance` / `User.credit_micros_reserved`; spendable = `balance - reserved`. Direct debit is exactly what `EtlCreditService.charge_credits` does.
-- **Audit trail**: `record_token_usage(session, *, usage_type, search_space_id, user_id, cost_micros=…, call_details=…, message_id=None, …)` (`token_tracking_service.py:517–568`) inserts a `TokenUsage` row (best-effort; caller commits). `TokenUsage.usage_type` is a `String(50)` indexed column explicitly intended for non-chat usage with `message_id` NULL (`db.py:1068–1116`, comment at `:1072–1084`: "indexing, image generation, podcasts — which keep message_id NULL").
-- **Per-feature config knobs** are the convention: `MICROS_PER_PAGE` (`config/__init__.py:655`, default `1000`), `ETL_CREDIT_BILLING_ENABLED` (`:652–653`, default FALSE), `QUOTA_DEFAULT_IMAGE_RESERVE_MICROS` (`:729`), etc. `maybe_trigger_auto_reload` (`auto_reload_service`) is the shared low-balance top-up nudge.
-
-> **Implementation note.** Same convention as `03a`/`03b`: Phases 1–2 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name (e.g. the `record_token_usage(search_space_id=…)` kwarg is now `workspace_id=…`; `_resolve_agent_billing_for_search_space`→`_resolve_agent_billing_for_workspace`; `select SearchSpace.user_id`→`select Workspace.user_id`). Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename and `03a`'s `crawl_url`/`scrape_webpage` refactor shift them.
-
-## Target design
-
-### 1. `WebCrawlCreditService` (mirror of `EtlCreditService`)
-
-New `app/services/web_crawl_credit_service.py` — a dedicated service (independent flag + price; lowest risk, matches the per-feature-knob convention). The two `@staticmethod` helpers are the **shared primitives both surfaces reuse**:
-
-- `billing_enabled()` → `config.WEB_CRAWL_CREDIT_BILLING_ENABLED` (new flag, default FALSE). *(static — also called by the chat tool)*
-- `successes_to_micros(n)` → `n * config.WEB_CRAWL_MICROS_PER_SUCCESS` (new knob, default `1000` = $1/1000). *(static — also called by the chat tool)*
-- `check_credits(owner_user_id, estimated_urls)` → no-op when disabled; else raise `InsufficientCreditsError` (reuse the one from `etl_credit_service`) if `successes_to_micros(estimated_urls) > available`.
-- `charge_credits(owner_user_id, successes)` → no-op when disabled; else debit `credit_micros_balance -= successes_to_micros(successes)`, commit, `maybe_trigger_auto_reload`.
-
-A separate `CreditMeterService` generalization is deliberately deferred until a third per-unit biller appears — exposing the price/flag as statics already lets the chat surface share the math without coupling the two flows.
-
-**Captcha billing seam (for `03d`).** `03d` (now active) adds a **separate per-attempt** captcha unit to this same service — `captcha_solves_to_micros(n) = n * config.WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE`, gated by `WEB_CRAWL_CAPTCHA_BILLING_ENABLED` (both new knobs, default FALSE), with `usage_type="web_crawl_captcha"`. It is **independent** of crawl SUCCESS because solvers charge per *attempt* regardless of crawl outcome (the asymmetry in the §"Risks" proxy/captcha note). Build the `WebCrawlCreditService` statics so `03d` can add one more `*_to_micros` static + one charge path without restructuring — no need to implement the captcha methods in `03c`, just leave the shape open.
-
-### 2. Wiring in `webcrawler_indexer.py`
-
-1. **Resolve the owner.** Bill the **workspace owner** (the product concept), not the triggering `user_id`: `owner_user_id = (select SearchSpace.user_id where id == search_space_id)`. (Mirrors how `billable_calls._resolve_agent_billing_for_search_space` (func `:441`) reads `search_space.user_id` at `billable_calls.py:499` — but we only need the owner id, so a direct `select` is enough; no need to call that LLM-model-resolving helper.)
-2. **Pre-flight gate** (after URL parse, before the crawl loop — i.e. the indexer's second pass that actually fetches URLs; "phase 2 of the 2-phase indexer", not roadmap Phase 2): `await svc.check_credits(owner_user_id, len(urls))`. On `InsufficientCreditsError`, `log_task_failure` with a clear "out of crawl credit" message and return — don't crawl. Because **successes ≤ len(urls)**, this upper-bound check means the wallet can never go negative from crawls (unlike ETL, where actual pages can exceed the estimate).
-3. **Audit (add, don't commit)**: `record_token_usage(session, usage_type="web_crawl", search_space_id=search_space_id, user_id=owner_user_id, cost_micros=successes_to_micros(crawls_succeeded), call_details={"urls": len(urls), "successes": crawls_succeeded, "connector_id": connector_id}, message_id=None)` — `record_token_usage` only `session.add`s the row (`token_tracking_service.py:552`), it does not commit.
-4. **Charge actuals**: `await svc.charge_credits(owner_user_id, crawls_succeeded)`. `charge_credits` debits the balance and **commits** (mirroring `EtlCreditService.charge_credits`), which flushes the step-3 audit row **and** the balance debit in one transaction.
-
-Ordering matters: audit must be added **before** the charge, because the charge's commit is what persists both. Both run on the indexer's existing `session` (same as the ETL services), after the final per-URL document commit (`webcrawler_indexer.py:432`). If `crawls_succeeded == 0`, skip both (no-op).
-
-### 3. Chat scrape crawls (folded into the chat turn's bill)
-
-The chat `scrape_webpage` tools (`main_agent/tools/scrape_webpage.py:183`, plus the `research` sibling) crawl via `WebCrawlerConnector.crawl_url` (`main_agent/tools/scrape_webpage.py:232–233`) — the **same** call `03a` refactors. The chat hook therefore keys off `03a`'s `CrawlOutcomeStatus.SUCCESS` return (a hard dependency on 03a's new `crawl_url` signature).
-
-**How the chat turn actually bills** (this is *not* `billable_call`): the chat orchestrators create the turn accumulator with `start_turn()` (`new_chat/orchestrator.py:185`, `resume_chat/orchestrator.py:147`, `anonymous_chat_routes.py:360`); a premium turn reserves up front and, at the end, `finalize_credit` debits `accumulator.total_cost_micros` via `credit_finalize` (`shared/premium_quota.py:83–104`). So adding to the accumulator before finalize is debited with the turn:
-
-```python
-from app.services.token_tracking_service import get_current_accumulator
-from app.services.web_crawl_credit_service import WebCrawlCreditService
-
-# after a SUCCESS crawl in the scrape tool
-acc = get_current_accumulator() # token_tracking_service.py:212–213 (turn ContextVar)
-if acc is not None and WebCrawlCreditService.billing_enabled():
- acc.add( # :112–135 — appends a TokenCallRecord
- model="web_crawl",
- prompt_tokens=0, completion_tokens=0, total_tokens=0,
- cost_micros=WebCrawlCreditService.successes_to_micros(1),
- call_kind="web_crawl",
- )
-```
-
-`accumulator.total_cost_micros` sums every call's `cost_micros` (`:174–186`), so the crawl micros ride along into the premium finalize — "included in the already-billed chat." `call_kind="web_crawl"` keeps it distinguishable in `per_message_summary` (`:137–159`).
-
-**Why the ContextVar is reliably visible (not hand-waving):** the LiteLLM `TokenTrackingCallback` already populates *this same* `_turn_accumulator` from deep inside the graph's LLM calls — that's how chat billing works today. If the ContextVar set by `start_turn()` reaches the LLM callback, it reaches a `scrape_webpage` tool in the same graph. The tools are `async` and `await crawl_url` in-loop, so the context is intact (the only thing that would break propagation is offloading to a raw OS thread without `copy_context`, which this path doesn't do).
-
-Two gates, both required, for a chat scrape to actually cost money:
-- **`WEB_CRAWL_CREDIT_BILLING_ENABLED`** (self-hosted: off → adds nothing).
-- **The turn is premium.** `needs_credit_quota` = `agent_config.is_premium` (`premium_quota.py:43–44`). Free / BYOK / **anonymous** (`anonymous_chat_routes.py:360`, no wallet) turns never reserve/finalize → the `web_crawl` cost is recorded in the accumulator breakdown but **not debited**. This is intentional: if the chat turn isn't billed, neither is its scrape.
-
-Other notes:
-- **No per-scrape pre-block.** The turn's up-front reservation is an LLM-only estimate (`reserve_credit`/`estimate_call_reserve_micros`, `premium_quota.py:65`); crawl micros added afterward aren't reserved, but `credit_finalize` debits actuals regardless and may push the balance slightly negative — same allow-exceed posture as ETL. Pre-blocking applies only to the indexer batch path (§2).
-- **Accounting caveat.** Chat-scrape crawl spend lands on the turn's `usage_type="chat"` `TokenUsage` row (inside `cost_micros` / the `web_crawl` line of `model_breakdown`), **not** a `usage_type="web_crawl"` row. "Total crawl spend" analytics must sum the indexer's `web_crawl` rows **plus** the `web_crawl` call-kind inside chat rows.
-
-### 4. What counts (from `03a`)
-
-One unit per `CrawlOutcomeStatus.SUCCESS` — a URL that yielded usable extracted content — **regardless of internal fallback tiers** and **regardless of downstream KB dedupe** (unchanged/duplicate still crawled successfully). `EMPTY`/`FAILED` are free.
-
-## Config / env changes
-
-- `config/__init__.py` (next to `ETL_CREDIT_BILLING_ENABLED`/`MICROS_PER_PAGE`, `:649–655`):
- - `WEB_CRAWL_CREDIT_BILLING_ENABLED = os.getenv(..., "FALSE").upper() == "TRUE"`
- - `WEB_CRAWL_MICROS_PER_SUCCESS = int(os.getenv("WEB_CRAWL_MICROS_PER_SUCCESS", "1000"))` — **the single source of truth for crawl price.** No literal price may appear in the service or call sites; every charge/estimate reads this. Default `1000` = $1/1000 crawls.
-- **Retuning formula (document in both `config` comment + `.env.example`):** `WEB_CRAWL_MICROS_PER_SUCCESS = round(USD_per_crawl * 1_000_000)`, equivalently `round(USD_per_1000_crawls * 1_000)`. Examples: $1/1000 → `1000`; $2/1000 → `2000`; $0.50/1000 → `500`; $5/1000 → `5000`. Change the env value and restart — no code/migration.
-- `.env.example`: document both (commented), noting hosted = TRUE, self-hosted = FALSE, and the conversion formula above.
-
-## Work items
-
-1. **`WebCrawlCreditService`** mirroring `EtlCreditService` (gate + `check_credits` + `charge_credits` + static `successes_to_micros`/`billing_enabled`).
-2. **Config knobs** + `.env.example` docs.
-3. **Indexer wiring**: owner resolution, pre-flight `check_credits`, then at end-of-run (after `:432`) `record_token_usage(usage_type="web_crawl")` (add) **followed by** `charge_credits(crawls_succeeded)` (commits both); skip when `crawls_succeeded == 0`.
-4. **Chat scrape wiring**: in both `scrape_webpage` tools, on SUCCESS fold `successes_to_micros(1)` into the turn accumulator (§3).
-5. **Tests**: billing-disabled → both surfaces no-op (self-hosted); indexer enabled + sufficient → debits `successes * 1000` + one `web_crawl` `TokenUsage`; indexer enabled + insufficient → task fails pre-crawl, no debit; owner (not trigger user) billed; `EMPTY`/`FAILED` free; chat scrape on a **premium** turn → `acc.total_cost_micros` rises by `1000` per success and is debited at finalize; chat scrape on a **free/anonymous** turn → recorded in the accumulator but **not** debited.
-
-## Risks / trade-offs
-
-- **Celery retry re-billing.** A task retry re-crawls only the URLs not already `ready`/`pending`/`processing` (the 2-phase indexer skips those, `webcrawler_indexer.py:196–219,341–347`), so a retry charges only re-crawled successes. Acceptable for MVP; Phase 5's `PipelineRun` can persist `charged_micros` for stronger idempotency.
-- **Same URL across connectors.** Each connector that crawls it performs a real request → each successful crawl bills, even though the 2nd is KB-deduped. This is intended per the `03a` billing-policy note.
-- **Session coupling.** `charge_credits` commits the indexer session; placing it after the final doc commit avoids entangling a debit with mid-run document state.
-- **Gating asymmetry (intended).** The **indexer** path bills whenever `WEB_CRAWL_CREDIT_BILLING_ENABLED` is on and the owner has credit — no "premium tier" requirement (matches ETL, which gates only on its own flag). The **chat** path additionally requires the turn to be premium, because it piggybacks the turn's reserve/finalize machinery. Same price, two different gate sets — call this out in the credit-status UI copy (frontend phase).
-- **No cross-surface double charge.** A chat scrape calls `crawl_url` directly and never enters `webcrawler_indexer`, so a given crawl is billed by exactly one surface.
-- **Proxy cost.** Absorbed into the flat $1/1000 (margin), **not** metered separately (see `03b`).
-- **Captcha cost.** **Metered separately** as a per-attempt unit (`03d §4`, decided: option (a)) because the solver charges per attempt regardless of crawl success — it cannot ride the per-success flat rate. The `captcha_solves_to_micros` seam above is where it attaches.
-
-## Resolved decisions (this pass)
-
-- **Bill ad-hoc chat scrapes? → YES.** Chat scrape crawls are metered too, but folded into the chat turn's existing bill via the turn accumulator (§3) rather than a separate wallet debit.
-- **Dedicated `WebCrawlCreditService` vs generalized meter? → dedicated**, with `billing_enabled`/`successes_to_micros` exposed as statics so the chat surface shares the price/flag. Generalize later only if a third per-unit biller appears.
-- **Pre-flight strictness → pre-block** the indexer batch run on insufficient credit (`len(urls)` is a safe upper bound). Chat scrapes are governed by the turn's own reservation, not a per-scrape block.
-
-## Out of scope (hand-offs)
-
-- Per-**pipeline** run accounting / `charged_micros` persistence for idempotency → Phases 5–7. Specifically, **Phase 6 moves pipeline-run billing out of the indexer to the run level**: it adds a `bill: bool = True` param to this indexer wiring and calls `index_crawled_urls(bill=False)` from the pipeline engine, which then does its own `check_credits` + `charge_credits(crawls_succeeded)` and stamps `PipelineRun.charged_micros` (idempotency). So this phase's in-indexer billing keeps serving the **connector `/index` + periodic** paths; the "pipeline crawls" half of the objective is refined by `06`. Structure the §2 wiring so it's easy to gate behind that flag.
-- Captcha-solver upstream cost pass-through → `03d` (now active; separate per-attempt `web_crawl_captcha` unit via the seam above).
-- Surfacing crawl spend in the credit-status UI → frontend umbrella.
diff --git a/plans/backend/03d-captcha-solving.md b/plans/backend/03d-captcha-solving.md
deleted file mode 100644
index a6b6d6d34..000000000
--- a/plans/backend/03d-captcha-solving.md
+++ /dev/null
@@ -1,111 +0,0 @@
-# Phase 3d — Captcha solving (reCAPTCHA / hCaptcha / image)
-
-> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> **Status: ✅ IMPLEMENTED (`ci_mvp`, captcha solving + per-attempt billing).** `captchatools==1.5.0` (MIT, pure-Python, single `requests` dep) wired as the StealthyFetcher-tier `page_action`. Generic config in Apache-2 `app/utils/captcha/` (`CaptchaConfig` + `captcha_enabled()` = flag **and** key); bypass logic (detect/harvest/inject + process latch) in proprietary `app/proprietary/web_crawler/captcha.py`. `CrawlOutcome` gained `captcha_attempts`/`captcha_solved`; the stealth tier captures the proxy **once** and reuses it for the solver (IP-coherence). Billing: `WebCrawlCreditService.captcha_solves_to_micros` + `charge_captcha` + generic `check_balance`; indexer accumulates attempts (even on failed crawls), combined crawl+captcha pre-flight, per-attempt owner charge with `usage_type="web_crawl_captcha"`; both chat scrape tools fold attempts into the turn before the success/fail branch. **No migration.** Off by default (zero attempts, zero cost). 40+ new unit tests pass.
-> **Implementation deltas from the original draft (the draft predated the real 03a code):**
-> - **There is no `FetchStrategy` class/seam** in the shipped 03a crawler — tiers are plain methods (`_crawl_with_async_fetcher`/`_crawl_with_dynamic`/`_crawl_with_stealthy`) that call `get_proxy_url()` inline. The "surface the chosen endpoint" seam is implemented concretely by capturing `proxy = get_proxy_url()` **once** in `_crawl_with_stealthy_sync` and passing it to both `StealthyFetcher.fetch(proxy=…)` and the factory (no re-rotation).
-> - **Attempts reach `CrawlOutcome` via a per-call `captcha_state` dict** threaded `crawl_url → _crawl_with_stealthy(_sync)` and stamped onto every stealth terminal outcome (success/empty/failed). Per-call (not on `self`) → concurrency-safe.
-> - **Solver timeout uses a non-context-managed `ThreadPoolExecutor`** + `shutdown(wait=False, cancel_futures=True)`; a `with` block would join the worker and defeat the timeout.
-> - **Combined pre-flight** lives on the indexer using the new generic `check_balance(required_micros)` (crawl successes + worst-case `len(urls) × MAX_ATTEMPTS` captcha), not a captcha-specific check method. The captcha worst-case is only reserved when **both** `captcha_billing_enabled()` **and** `captcha_enabled()` (solving actually on) — with solving off, attempts can never happen, so reserving would wrongly block a run for captcha that will never run (regression-tested).
-> **Decision recap (this pass):** Cloudflare stays in-framework (free, `03a`); reCAPTCHA/hCaptcha/image use a **paid** solver via `captchatools`, **opt-in + off by default**, and are billed as a **separate per-attempt unit** (option (a) below). Logged-in/account-gated captcha flows are **out of scope** (no authenticated scraping in this MVP — see umbrella Phase 8 / `03b` static-proxy hand-off).
-
-## Why this is the last tier, not the first
-
-A solve is the **most expensive and least reliable** move available (paid per attempt, 10–60 s latency, <100% success, ToS-sensitive). So the crawler must exhaust the cheaper, in-house levers first; captcha solving only fires when everything else has already failed:
-
-1. `03e` humanization + fingerprint coherence (geoip locale/tz, persistent profile, headed/Xvfb) → **avoid** triggering a challenge at all.
-2. `03a` `solve_cloudflare=True` on StealthyFetcher → free Cloudflare Turnstile/Interstitial.
-3. **This subplan** → only the residual reCAPTCHA v2/v3 + hCaptcha + image challenges, and only when `CAPTCHA_SOLVING_ENABLED`.
-
-## Scope boundary
-
-| Challenge | Handled by | Where |
-| --- | --- | --- |
-| Cloudflare Turnstile / Interstitial | Scrapling `solve_cloudflare=True` | `03a`, StealthyFetcher tier |
-| reCAPTCHA v2/v3, hCaptcha, image | **`captchatools`** (this subplan) | StealthyFetcher `page_action` |
-| DataDome / Kasada / reCAPTCHA-**Enterprise** | **none in the free stack** | deferred paid-unblocker tier (`03e`) |
-
-## Grounding (libraries verified)
-
-- **`captchatools` is itself the provider registry.** `new_harvester(api_key, solving_site, sitekey, captcha_url, captcha_type="v2"|"v3"|"hcaptcha"|"image", invisible_captcha=…, min_score=…, action=…)` → `.get_token(proxy=…, proxy_type=…, user_agent=…, b64_img=…)` returns a **token string** (`references/Captcha-Tools/README.md:28–47`). `solving_site` ∈ {`capmonster`,`2captcha`,`anticaptcha`,`capsolver`,`captchaai`} (`:113–127`). Errors: `ErrNoBalance`, `ErrWrongAPIKey`, `ErrWrongSitekey`, `ErrIncorrectCapType`, `ErrNoHarvester` via `captchatools.exceptions` (`:136–155`). **Implication:** we do **not** rebuild a multi-provider class hierarchy (unlike `03b`'s proxy registry) — `captchatools` already dispatches across the 5 services. Our layer is thin: config resolution + page detection/injection glue + billing.
-- **`captchatools` only harvests a token; it does not inject it.** The caller must drop the token into the page (`g-recaptcha-response` / `h-captcha-response` textarea, or invoke the JS callback) and submit.
-- **Injection requires a live browser page**, so this only works on the **StealthyFetcher** tier. Scrapling exposes `page_action`: "a function that takes the `page` object, runs after navigation, and does the automation you need" (`references/Scrapling/scrapling/fetchers/stealth_chrome.py:30,82`; engine docstrings `_browsers/_stealth.py:50,191,326,466`). The HTTP (`AsyncFetcher`) and `DynamicFetcher` tiers cannot solve interactive captchas — a captcha hit there must **escalate to StealthyFetcher** (`FetchStrategy` seam from `03a`; the ladder already ends there).
-- **Ordering is already correct in Scrapling.** `solve_cloudflare` runs **before** `page_action` (sync `_stealth.py:253–254` then `:258–260`; async `:529–530` then `:534–536`). So Cloudflare is cleared first, *then* our injector runs — exactly the tier order we want. `page_action` exceptions are caught + logged by Scrapling (`:262` / `:538`), so our factory must also return a clear signal (it can't rely on raising to abort the fetch).
-
-## Target design
-
-### 1. Config layer (`app/utils/captcha/`, mirroring `app/utils/proxy/` from `03b`)
-
-- `CaptchaConfig = (enabled, solving_site, api_key, captcha_type_default, min_score, max_attempts, timeout_s)`, resolved from **env only** — one app-wide config, mirroring `03b`'s env-only single-provider model (`get_active_provider()`); **no per-connector config** (that path was dropped in `03b`).
-- Env knobs (next to the proxy + crawl-billing knobs in `config/__init__.py`):
- - `CAPTCHA_SOLVING_ENABLED` (default `FALSE`) — off ⇒ zero solve attempts and zero solver cost.
- - `CAPTCHA_SOLVER_PROVIDER` (e.g. `capsolver`), `CAPTCHA_SOLVER_API_KEY`.
- - `CAPTCHA_MAX_ATTEMPTS_PER_URL` (default `1`), `CAPTCHA_SOLVE_TIMEOUT_S` (default `120`).
-- A `captcha_enabled()` static (mirrors `WebCrawlCreditService.billing_enabled()`) so callers gate cheaply before constructing anything.
-
-### 2. Detection + injection `page_action` factory
-
-Builds the **sync** callable passed to `StealthyFetcher.fetch(..., page_action=…)` (03a runs StealthyFetcher via `asyncio.to_thread`, so the sync engine path `_stealth.py:258–262` applies — the factory returns a plain `def(page): ...`, not a coroutine):
-
-1. **Detect** the challenge + sitekey in the DOM:
- - reCAPTCHA v2: `.g-recaptcha[data-sitekey]` (or iframe `src*="recaptcha"`).
- - hCaptcha: `.h-captcha[data-sitekey]` (or iframe `src*="hcaptcha"`).
- - reCAPTCHA v3: presence of `grecaptcha.execute` + a known `action` (config/site-mapped).
- - If no sitekey found → **no-op return** (nothing to solve; let `wait_selector`/extraction proceed).
-2. **Bind the proxy/UA** (the IP-coherence caveat below): harvest with the **exact endpoint used for this crawl tier** —
- `new_harvester(solving_site, api_key, sitekey, captcha_url=page.url, captcha_type=…, min_score=…, action=…).get_token(proxy=, proxy_type="HTTP", user_agent=)` (`README.md:44–46,107–110`).
-3. **Inject + submit**: set the response token into `g-recaptcha-response` / `h-captcha-response` (make textarea visible if needed), dispatch `input`/`change`, invoke the JS callback when present (`grecaptcha.getResponse` flows / `___grecaptcha_cfg` callbacks), then submit the form and `page.wait_for_load_state`. For v3 (no widget) the token is consumed by the site's own `execute()` flow.
-4. **Signal outcome** via a mutable closure cell (since Scrapling swallows `page_action` exceptions at `:262` **and discards its return value** at `:260`/`:536`): record `solved: bool` + `attempts: int` into a captured `dict` the crawler reads after `fetch()` returns, so billing (§4) and the retry cap can act on it. **This `page_action`+closure-cell helper is shared with `03f`'s test harness** (which uses the same mechanism to read JS-object verdicts like CreepJS `window.Fingerprint`) — factor it once.
-
-### 3. Crawler escalation (uses the `03a` `FetchStrategy` seam)
-
-- Only the **StealthyFetcher** strategy attempts solving. A captcha detected on a lower tier (HTTP/DynamicFetcher) returns non-`SUCCESS`, and the ladder already escalates to StealthyFetcher (`03a`), which this time is constructed with the captcha `page_action` when `captcha_enabled()`.
-- **Attempt cap:** at most `CAPTCHA_MAX_ATTEMPTS_PER_URL` solves per URL so one hostile page can't burn unbounded solver credit. The closure cell's `attempts` enforces it.
-- No new caller *contract*: the strategy still returns `CrawlOutcome` (`03a` invariant). Captcha is a *parameterization* of the last tier, not a new return shape.
-- **But billing needs the attempt count to escape the closure cell.** §4 charges per attempt on **both** the indexer and chat paths, so the count can't stay buried inside `page_action`'s closure — the crawler must copy it onto the outcome. Add **`captcha_attempts: int = 0`** (and `captcha_solved: bool = False`) as **dataclass fields on `CrawlOutcome`** (`03a`). This is safe: `03a`'s "don't widen the return" rule is about the *indexer's* positional `(total_processed, error)` tuple unpacked by length, **not** about `crawl_url`'s dataclass — adding fields to a dataclass breaks no consumer. The indexer sums `outcome.captcha_attempts`; the chat tool reads it off the same outcome.
-
-### 4. Billing — separate per-attempt unit (option (a), DECIDED)
-
-`03c` bills per **successful crawl** (`CrawlOutcomeStatus.SUCCESS`) and absorbs proxy cost into the flat $1/1000. **Captcha can't ride that model**: the solver charges **per attempt regardless of whether the crawl ultimately succeeds**, so a failed solve = real upstream cost with no billable crawl success. We therefore meter solves **independently**:
-
-- New static on `WebCrawlCreditService` (defined in `03c`, knobs added here): `captcha_solves_to_micros(n) = n * config.WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE`.
-- Each **attempt** (not each success) is charged when `CAPTCHA_SOLVING_ENABLED` **and** `WEB_CRAWL_CAPTCHA_BILLING_ENABLED`:
- - **Indexer/pipeline path:** after the crawl loop, debit `captcha_solves_to_micros(total_attempts)` — where `total_attempts = Σ outcome.captcha_attempts` over the batch — from the **workspace owner** (same owner resolution as `03c §2`) and `record_token_usage(usage_type="web_crawl_captcha", …, cost_micros=…, call_details={"attempts": n, "solved": k})`. Added before the charge commit, same one-transaction pattern as `03c`.
- - **Chat-scrape path:** fold `captcha_solves_to_micros(outcome.captcha_attempts)` into the turn accumulator with `call_kind="web_crawl_captcha"` (same mechanism as `03c §3`), so it settles with the premium turn; non-premium/anonymous turns record-but-don't-debit.
-- **No pre-block on solves** (unlike the crawl batch): attempts are bounded by `CAPTCHA_MAX_ATTEMPTS_PER_URL × len(urls)`, an upper bound the indexer can optionally pre-check against the wallet if we want symmetry with `03c §2` (recommended: pre-check the combined crawl + worst-case captcha estimate so a run can't strand mid-batch on solver insolvency).
-- New env (next to `03c`'s knobs): `WEB_CRAWL_CAPTCHA_BILLING_ENABLED` (default `FALSE`), `WEB_CRAWL_CAPTCHA_MICROS_PER_SOLVE` (default e.g. `3000` = $3/1000 attempts; type-dependent, set with margin over the solver's per-attempt price). **No DB migration** — `web_crawl_captcha` is just another free-form `TokenUsage.usage_type` value (`db.py` `usage_type String(50)`).
-
-### 5. Dependency + Docker
-
-- Add `captchatools` to `pyproject.toml` (build-time only). It's a thin HTTP client for the solver services — no browser binaries.
-- No `Dockerfile` change beyond the dependency (browser already installed by `03a`/`03e`).
-
-## Error handling (must not loop or silently burn credit)
-
-- `ErrNoBalance` / `ErrWrongAPIKey` (`README.md:134,136–143`) → **stop solving for the rest of the run** (set a process/run flag), surface a clear "captcha solver out of balance / misconfigured" message via `log_task_failure`, and fall through to a non-`SUCCESS` `CrawlOutcome`. The README explicitly warns balance-exhausted loops can get the **IP temporarily banned** (`:132–134`) — never retry on `ErrNoBalance`.
-- `ErrWrongSitekey` / `ErrIncorrectCapType` → log + skip this URL's solve (likely detection bug, not transient); count the attempt for billing only if the solver actually charged.
-- Timeout (`CAPTCHA_SOLVE_TIMEOUT_S`) → abort the solve, count one attempt, return non-`SUCCESS`.
-
-## Risks / considerations
-
-- **Solver-tier only.** No captcha solving on the HTTP/DynamicFetcher tiers; must escalate to StealthyFetcher (slower, browser-backed).
-- **Latency & flakiness.** Solves take 10–60 s and aren't guaranteed; the `CAPTCHA_MAX_ATTEMPTS_PER_URL` + `CAPTCHA_SOLVE_TIMEOUT_S` caps keep a single URL from burning unbounded solver credit.
-- **Proxy coherence (rotating-pool caveat).** Many captchas IP-bind the token, so the solver must egress from the **same IP** as the crawl. Under `03b`'s single app-wide provider this is automatic for single-endpoint providers (`anonymous_proxies`, single-URL `custom`). But a **pool-backed `CustomProxyProvider`** returns the *next* endpoint on each zero-arg `get_proxy_url()` call — so the `page_action` must **reuse the endpoint actually used for this crawl tier** (the crawler captures it once and passes it into the factory), NOT call `get_proxy_url()` again (which would rotate to a different IP). This is the same "surface the chosen endpoint" seam `03e`/`03d` both rely on — build it once in `03a`'s strategy context.
-- **Enterprise captchas are out of reach.** reCAPTCHA **Enterprise**, DataDome, and Kasada use behavioral + device signals the token-injection model doesn't satisfy reliably; those route to the deferred paid-unblocker tier (`03e`), not here.
-- **Policy / ToS.** Automated captcha solving may violate a target site's terms; gate behind the explicit `CAPTCHA_SOLVING_ENABLED` flag and treat as an opt-in, owner-acknowledged capability. Public data only (no logged-in bypass).
-
-## Work items
-
-1. **`app/utils/captcha/`**: `CaptchaConfig` + env resolution + `captcha_enabled()` static.
-2. **`page_action` factory**: detection (v2/v3/hcaptcha/image sitekey), harvest via `captchatools` with bound proxy/UA, inject+submit, outcome closure cell.
-3. **Crawler wiring**: construct StealthyFetcher with the factory when `captcha_enabled()`; enforce per-URL attempt cap; surface the crawl's chosen proxy endpoint into the factory.
-4. **Billing**: `WebCrawlCreditService.captcha_solves_to_micros` + the two knobs; debit per-attempt on the indexer path (owner) and fold into the turn on the chat path; `record_token_usage(usage_type="web_crawl_captcha")`.
-5. **Config + docs**: all env knobs in `Config` + `.env.example` (commented, hosted=TRUE / self-hosted=FALSE), plus the `captchatools` dependency.
-6. **Tests**: solving disabled → zero attempts, zero solver cost (self-hosted); v2 detected → harvest+inject+submit path invoked with the **crawl's** proxy endpoint (not a re-rotated one); `ErrNoBalance` → stops solving + no retry loop; attempt cap honored; billing enabled → per-attempt debit even when the crawl ultimately fails; chat path folds captcha micros into a premium turn and record-only on anonymous; `web_crawl_captcha` `TokenUsage` rows written.
-
-## Out of scope
-
-- Anything owned by `03a`/`03b`/`03c`/`03e`.
-- Cloudflare Turnstile (already handled by Scrapling in `03a`).
-- Logged-in / account-gated captcha flows (no authenticated scraping this MVP → umbrella Phase 8 + `03b` static-proxy hand-off).
-- Enterprise/behavioral anti-bot (DataDome/Kasada/reCAPTCHA-Enterprise) → deferred paid-unblocker tier (`03e`).
diff --git a/plans/backend/03e-stealth-hardening.md b/plans/backend/03e-stealth-hardening.md
deleted file mode 100644
index 0d16efd6b..000000000
--- a/plans/backend/03e-stealth-hardening.md
+++ /dev/null
@@ -1,167 +0,0 @@
-# Phase 3e — Stealth hardening (the in-house "undetectable" moat)
-
-> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> Depends on `03a` (Scrapling StealthyFetcher tier + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built).
-
-> **Architecture reconciliation (read before building).** Earlier drafts of this plan referenced a `FetchStrategy` seam / strategy-object contract `(url, ctx) -> CrawlOutcome`. **That seam does not exist in shipped code.** `03a`–`03d` shipped a *hardcoded 3-tier ladder* inside `WebCrawlerConnector.crawl_url` (`app/proprietary/web_crawler/connector.py`): plain methods `_crawl_with_async_fetcher` / `_crawl_with_dynamic` / `_crawl_with_stealthy(_sync)` wrapped by `_run_tier_with_proxy_retry`, every tier returning the shared `CrawlOutcome`. `03e` therefore implements levers **not** as strategy classes but as: (1) a **centralized per-tier kwargs builder** (`app/proprietary/web_crawler/stealth.py` — **proprietary**, since it's bypass-specific tuning; the generic block classifier stays Apache-2 in `app/utils/crawl/`) that turns config → `StealthyFetcher`/`AsyncFetcher` kwargs, imported by both the connector and `03f`'s harness so there's no test-vs-prod drift; (2) small **helpers around the existing ladder** (block classifier, strategy memory); and (3) the deferred paid tier as a config-gated **extra ladder step**, not a `FetchStrategy`. The captcha precedent holds: generic config/plumbing → `app/utils/`; actual bypass logic (WebGL spoof JS, humanize choreography) → `app/proprietary/web_crawler/`.
-
-> **Implementation note.** Same convention as `03a`–`03d`: Phases 1–2 are **SHIPPED** (live code says `workspace_id`/`Workspace` — substitute for any old `search_space_*`/`SearchSpace` citations and grep the new name); citations also predate `03a`'s `crawl_url` refactor; locate code by **symbol/grep**, not absolute lines. Scrapling references point at the on-disk (gitignored) `references/Scrapling/` checkout pinned to `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`).
-
-> **Build status.** Sliced for risk (see "Build slicing" below). **Slice A (SHIPPED in this pass):** centralized stealth kwargs builder + fingerprint flags + geoip-from-proxy-location + block classifier (additive `CrawlOutcome.block_type`) + fonts/Xvfb packages in the image. **Slice B / C (deferred):** headed-Xvfb runtime path, humanization/WebGL `init_script` assets, persistent profiles, strategy memory, paid-unblocker stub. Effectiveness of every lever is graded in `03f` (manual), not unit-tested here.
->
-> **First baseline evidence (`03f`, 2026-06-30 — headless, rotating residential, captcha OFF, Slice-A only with `CRAWL_GEOIP_MATCH_ENABLED=false`).** Suite S = **6 PASS / 4 FAIL** (full table in `03f`). The 4 fails map cleanly onto this plan's deferred levers and confirm the predicted ceiling — they are **not** Slice-A regressions:
-> - **iphey "Unreliable"** → the predicted geoip-incoherence tell (browser tz/locale vs proxy exit geo). It's now the live regression test for **§1 geoip coherence** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate next.
-> - **CreepJS `hasHeadlessWorkerUA: true`** (headless 33%) → the Web **Worker** `navigator.userAgent` still leaks `HeadlessChrome` (main-thread UA is clean). A concrete, plausibly-fixable **Slice-B** target (worker UA override via `init_script`), beyond today's Slice-A flags.
-> - **FingerprintJS Pro "tampering detected, access denied"** → the documented **commercial/device-fingerprint wall** (§2c WebGL/GPU gap). Needs Slice B/C or the §8 paid tier; expected to stay red on the free stack.
-> - **incolumitas** → a single legacy `fpscanner WEBDRIVER` check (all modern checks pass); lowest priority. Its IP classifier also flip-flopped `is_datacenter` true/false across rotations, exposing that the `anonymous_proxies` pool mixes datacenter+residential exits (a proxy-provider variable, not a code issue).
-
-## Objective
-
-Push the Universal WebURL Crawler as far toward "undetectable" as the **free / in-house stack** allows, so we hold a scraping moat for the next ~4–6 months **without** a third-party unblocker (ZenRows/ScrapFly/Bright Data) or a source-patched browser (CloakBrowser — rejected on licensing). Everything here is **runtime/config-level** stealth layered on Scrapling's patchright-Chromium (`03a` engine note): geoip fingerprint coherence, persistent profiles, headed execution, real fonts, and behavioral humanization — plus a **block classifier** + **per-domain strategy memory** so the ladder learns and the cheap tiers get skipped once a domain's working strategy is known. The paid-unblocker tier is defined here as a **deferred config-gated ladder step** (the explicit escape hatch for when in-house maintenance gets too costly), not built now.
-
-## The realistic ceiling (be honest with downstream devs)
-
-Scrapling already ships strong **runtime** stealth by default: `DEFAULT_ARGS + STEALTH_ARGS` (incl. `--disable-blink-features=AutomationControlled`), a persistent context, and `navigator.webdriver` masking via **patchright**. Be precise about what patchright actually is: it patches the **Playwright driver** to remove *automation leaks* — it never calls `Runtime.enable`, disables `Console.enable`, and reaches closed shadow roots (verified: [patchright-python](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python), [Scrappey](https://scrappey.com/qa/web-scraping-apis/what-is-patchright)). It hides **"I'm automated," not "this is a different machine."** It does **not** spoof canvas, WebGL, audio, screen, or GPU. With `03b` residential proxies + this plan's coherence/humanization on top, the crawler reliably handles **Cloudflare** (via `03a` `solve_cloudflare`) and the long tail of **moderate** anti-bot — which is most CI targets.
-
-**The device-fingerprint wall (the WebGL/GPU gap — see §2c).** Our worker has no GPU, so Chromium's WebGL renderer reports `Google SwiftShader` / `Mesa llvmpipe` — a string anti-bot vendors treat as ">95% bot" ([ipasis](https://ipasis.com/blog/webgl-fingerprinting-bot-detection), [Scrappey](https://scrappey.com/qa/anti-bot/what-is-webgl-fingerprinting)). Scrapling exposes only `allow_webgl` (on/off — and *off* is itself a tell), **no renderer spoof**; CloakBrowser fixes this at the C++ level (`--fingerprint-gpu-renderer`) and we cannot. Concretely, **without a real GPU or a source-patched binary we cannot**:
-
-1. **Pass pure device-fingerprint bot checks** — FingerprintJS bot-detection, CreepJS / BrowserScan "is this a real device." CloakBrowser passes these (BrowserScan 4/4); our SwiftShader renderer fails them.
-2. **Reach human-level reCAPTCHA v3 *scores*** (~0.7–0.9) **on our own browser** — we land ~0.1–0.5. (Note: `03d`'s **paid solver sidesteps this for token challenges** — the token is minted on the vendor's real-browser infra, so its score isn't ours. The gap only bites where the *site itself* reads our live v3 score.)
-3. **Max out the device-fingerprint component** of DataDome / Kasada / Akamai — one of several reasons we plateau below managed APIs.
-
-**Numbers (2026 benchmarks — use these in any user-facing copy).** Free stack **+ residential**: Basic (CF) ~99%; Medium (CF Pro / PerimeterX) ~90%; **Enterprise (DataDome / Kasada / Akamai) ~60–75% initially, decaying toward ~60% within 24h** as their ML clusters the fingerprint (managed APIs hold ~89%) ([scrapewise DataDome 2026](https://scrapewise.ai/blogs/bypass-datadome-web-scraping-2026)). So the honest line is **"partial and decaying, not SLA-grade,"** — neither "can't beat" nor "beats everything." For CI targets the top-tier defenses are the exception, so the in-house moat is a sound 4–6 month bet, with the deferred paid tier (§8) as the **evidence-driven** escape hatch (`03f`'s scorecard is the trigger). **Do not promise "beats everything."**
-
-## Levers (wired via the centralized kwargs builder + ladder helpers — callers stay outcome-only)
-
-### 1. Geoip fingerprint coherence (match the browser to the proxy exit IP)
-
-A residential IP in Berlin behind an `en-US`/`America/New_York` browser is an instant tell. Make the fingerprint cohere with the proxy's exit geo:
-
-- **Primary source (Slice A, no network call): the configured `RESIDENTIAL_PROXY_LOCATION` (`03b`, `config:1048`).** Map that string (best-effort: ISO-3166 alpha-2 like `us`/`de`, or a common country name) → a representative `(locale, timezone_id)`. Coarse country granularity is fine (per "Risks": wrong-but-coherent beats default-mismatched); unknown/empty → skip (leave Scrapling's system default). **No exit-IP resolution** — avoids a per-crawl network round-trip + failure mode. (A `geoip2`/MaxmindLite exit-IP path is a later refinement, not MVP.)
-- Pass the matched values into StealthyFetcher via the kwargs builder: `locale=` (drives `navigator.language` + `Accept-Language`) and `timezone_id=` (confirmed real kwargs on `StealthyFetcher.fetch`). These flow into the Playwright context.
-- The stealthy tier **already captures the chosen proxy endpoint once** (`_crawl_with_stealthy_sync`: `proxy = get_proxy_url()`, the same value handed to `03d`'s solver for IP-coherence) — reuse that captured `proxy`/location, **don't** re-call `get_proxy_url()` (which rotates on a pool-backed provider, `03b`).
-- **Caveat (be honest):** Scrapling applies `locale`/`timezone_id` via Playwright **CDP emulation**, which advanced systems can flag *as* emulation — CloakBrowser sets these as *binary flags* precisely to avoid "detectable CDP emulation" (`references/CloakBrowser/README.md:309`). Still strictly better than a mismatched default (wrong-but-coherent beats `UTC`+`en-US` behind a Berlin IP), but it's a weaker variant than a source-patched browser, not parity.
-
-### 2. Fingerprint flags Scrapling already exposes (turn them on)
-
-- `hide_canvas` — Scrapling adds **random** noise to canvas ops (verified in the installed `StealthyFetcher.fetch` docstring). **Not "free safe":** random-per-call noise is itself a signal on FingerprintJS/CreepJS-class detectors — an *unstable* canvas hash is as loud as `navigator.webdriver=true` ([dev.to](https://dev.to/tanwydd/how-your-canvas-fingerprint-gets-you-caught-and-why-random-noise-makes-it-worse-3ba1)); CloakBrowser's own FPJS fix is `--fingerprint-noise=false`. **Default OFF; gate per-domain and validate in `03f`.** Harmless against Cloudflare/moderate, potentially harmful against device-fingerprint graders. (Canvas stealth that actually *helps* must be **deterministic per persistent profile**, not random — out of scope here.)
-- `block_webrtc=True` — forces WebRTC to respect the proxy, preventing the **real local IP leak** that unmasks proxied browsers (`:41`). **This one is genuinely cheap + safe — keep default TRUE.**
-- `dns_over_https` — Scrapling can route DNS via Cloudflare DoH to stop the **DNS leak** that unmasks proxied browsers (same anti-leak class as `block_webrtc`; confirmed kwarg on `StealthyFetcher.fetch`). **Trade-off: it adds a DNS round-trip, a (small) latency cost — so per the "no speed regression" constraint it ships *off by default* (`CRAWL_DNS_OVER_HTTPS`, default FALSE), pre-wired for operators who prefer leak-safety over the marginal latency.** (Idea sourced from the Camoufox-based FlareSolverr alternatives `references/Byparr-main` / `references/trawl-dev`, which treat DNS/WebRTC leak coherence as first-class.)
-- `google_search=True` (default) — sets a Google referer so the first hit looks like organic arrival (`:45`); override per-need via `extra_headers` (`:46`).
-- `additional_args=` — last-priority Playwright context overrides for anything not surfaced as a first-class param (`:51`).
-
-### 2b. HTTP-tier TLS fingerprint (the AsyncFetcher tier — `impersonate`)
-
-The cheap static tier (`03a` tier 1) is the **first** thing every crawl hits. `03a` **already ships `impersonate="chrome"`** on it (`app/proprietary/web_crawler/connector.py`, the `AsyncFetcher.get` call) — Scrapling's static engine selects a matching curl_cffi browser profile (`references/Scrapling/scrapling/engines/static.py:36–47`), so the HTTP tier's **JA3/JA4/HTTP-2** already matches a real Chrome and coheres with the browser tiers' UA. `03e`'s remaining work here is **not** to add it, but to (a) fold the `impersonate` profile into the centralized per-tier kwargs builder (below) so it stays the single source of truth, and (b) keep the chosen profile coherent with the proxy exit / UA. `03f §S3` validates the shipped parity against `tls.peet.ws`.
-
-### 2c. WebGL / GPU renderer (the biggest free-stack gap — partial mitigation only)
-
-This is the "device-fingerprint wall" from the ceiling section. On a GPU-less worker the WebGL renderer string is `Google SwiftShader` / `ANGLE (Mesa, llvmpipe…)` — an instant tell on any detector that reads `WEBGL_debug_renderer_info`. Scrapling gives us **no renderer knob** (only `allow_webgl` on/off, and *off* is a tell). Three options, increasing strength + cost:
-
-1. **DIY JS spoof (optional, OFF by default — Slice B).** An `init_script` overrides `getParameter` for `UNMASKED_VENDOR_WEBGL` / `UNMASKED_RENDERER_WEBGL` to a believable, **platform-coherent** value (e.g. an Intel/ANGLE renderer that matches the spoofed Windows UA), held **consistent per persistent profile** (§3). **Impl note:** Scrapling's `init_script` is **an absolute path to a `.js` file**, *not* an inline string (confirmed in `StealthyFetcher.fetch` docstring) — so this ships as a **bundled `.js` asset** in `app/proprietary/web_crawler/` (next to `03d`'s captcha logic) passed by absolute path; the kwargs builder resolves the path when `CRAWL_WEBGL_SPOOF_ENABLED`. **Must** also patch `Function.prototype.toString` so the override still returns `[native code]` — otherwise *the spoof itself is the signal* (exactly the `toString()` probe Kasada catalogs, [web-scraping-guide](https://web-scraping-guide.com/)). This defeats **string-based** WebGL checks (the common case) but **not** render-output pixel-hash or GPU-perf probes (FingerprintJS-grade) — a software rasterizer's actual draw output still differs and can't be faked in JS. It is the brittle config-level approach patchright/CloakBrowser explicitly warn rots; treat it as a **cheap, test-gated win, not the moat**.
-2. **Real GPU passthrough (hosted workers only).** `--device /dev/dri` + ANGLE/EGL so WebGL reports a *genuine* renderer with *real* render output, defeating even output/perf probes. Best technical fix; costs GPU instances + image complexity ([livekit/egress GPU notes](https://stackoverflow.com/questions/78985698)). A hosted-tier escalation, off for self-hosted.
-3. **Defer to the paid unblocker (§8).** Route the residual device-fingerprint-gated domains (flagged by the §7 classifier as in-house-unreachable) to the external unblocker. The pragmatic moat-bounding fallback.
-
-**Decision for this MVP:** ship option 1 as an opt-in flag (`CRAWL_WEBGL_SPOOF_ENABLED`, default FALSE), name option 2 as the hosted escalation, and lean on §8 for the rest. Quantify all three in `03f`.
-
-### 3. Persistent per-domain profiles (look like a returning human)
-
-Scrapling defaults to a **temporary** user-data dir (fresh = suspicious). Use `user_data_dir=` (confirmed real kwarg; `launch_persistent_context`) to keep a **persistent profile per domain** (or per domain+proxy-geo), so cookies/localStorage/site-trust carry across crawls and the browser presents as a returning visitor rather than a brand-new incognito session. Store profiles under a configured dir (`shared_tmp`-style volume so API + worker share them).
-
-- **Concurrency hazard (Slice C design item — why this is deferred, not a one-liner):** Chromium **hard-locks** a `user_data_dir`; two crawls touching the same profile dir at once crash (`SingletonLock`). Our crawler runs concurrently (multiple connectors/chats across workers), so a naive per-domain dir collides. The fix needs one of: (a) an async lock keyed by profile dir so same-domain crawls serialize (adds latency, can deadlock under load), (b) per-`(domain, worker, pid)` dirs + a periodic prune (loses some cross-run trust, no contention), or (c) persistent only when the strategy memory (§7) says a domain *needs* it, temp otherwise. Decide in Slice C alongside §7; **Slice A keeps Scrapling's temp default.**
-
-### 4. Headed execution under Xvfb (defeat headless tells)
-
-`headless` defaults to hidden (`stealth_chrome.py:19/71`). Many WAFs flag headless Chromium. Run **headful** (`headless=False`) inside a virtual framebuffer (**Xvfb**) in the Docker worker so the browser is "visible" to itself but needs no real display. Gate behind a config flag (off by default for self-hosted, on for hosted workers that have Xvfb installed).
-
-### 5. Real fonts (canvas/emoji hash realism)
-
-A minimal container has almost no fonts, making canvas/emoji fingerprint hashes obviously synthetic. Install real font packages so font-enumeration + canvas/emoji hashes resemble a real desktop. Use the set CloakBrowser ships as proven against Kasada/Akamai emoji-canvas hashing (`README.md:737–739`): `fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic` (CJK), `fonts-wqy-zenhei` (CJK), `fonts-tlwg-loma-otf` — plus `fonts-dejavu`/`fonts-liberation` for Latin coverage. `Dockerfile` change only (`apt-get install fonts-*`).
-
-### 6. Behavioral humanization (DIY — Scrapling has no `humanize` for the Chromium engine)
-
-Unlike Camoufox, the patchright-Chromium StealthyFetcher exposes **no built-in `humanize`** (verified: zero matches in the StealthyFetcher param set). So humanization is custom, injected via the existing hooks:
-
-- `page_action=` (runs after navigation) — randomized mouse moves/curves, scrolls, hover-before-click, and small think-time delays before extraction. This is the **same hook `03d` uses** for token injection — `_crawl_with_stealthy_sync` currently builds exactly **one** `page_action` (the captcha injector), so Slice B must **compose** two callables (humanize → optional captcha solve) into the single `page_action` kwarg, not overwrite it.
-- `init_script=` (an **absolute path to a `.js` file** executed on page creation — *not* inline JS, see §2c) — early shims for any residual JS tells not covered by patchright (and the §2c WebGL spoof, if enabled).
-- Tunable `wait`/`network_idle` (`:29,27`) so dwell time isn't robotically constant.
-- **Concrete rules (from CloakBrowser's reCAPTCHA notes, `README.md:1280–1314` — encode these):** use native `time.sleep`, **never** `page.wait_for_timeout()` (it emits CDP traffic detectors flag); prefer `page.type()` over `fill()` for any input (real keystrokes vs. value-set); dwell **15s+** before triggering score-based checks; space repeat hits to score-gated domains **30s+**; keep a **stable identity per domain** (persistent profile §3 + consistent spoofed values §2c) so a score-based system sees a *returning* device, not a fresh one each visit.
-
-### 7. Block classifier + per-domain strategy memory (the "learning ladder")
-
-Two small in-house pieces make the ladder smart instead of brute-force:
-
-- **Block classifier (Slice A — additive, pure, unit-testable).** Inspect the fetched page (`status` + body markers; cookies/headers are not threaded through `_build_result` today, so MVP is **status + body-marker based**) and label the outcome: `OK` / `CLOUDFLARE` / `CAPTCHA_RECAPTCHA` / `CAPTCHA_HCAPTCHA` / `DATADOME` / `KASADA` / `RATE_LIMITED` / `EMPTY` / `UNKNOWN`. **Concrete markers — adopt the set proven in `references/trawl-dev/packages/tiers/src/detect.ts` (a near-identical classifier):** Cloudflare/DDoS-Guard → `cf-mitigated`-style title strings `"just a moment"` / `"checking your browser"` / `"enable javascript and cookies to continue"` / `"verify you are human"`, DOM ids `challenge-running` / `cf-challenge-running` / `turnstile-wrapper`, and `ddos-guard.net`; Turnstile → `class="cf-turnstile"` / `challenges.cloudflare.com/turnstile`; hCaptcha → `class="h-captcha"` / `hcaptcha.com/1/api`; reCAPTCHA → `class="g-recaptcha"` / `google.com/recaptcha` / `recaptcha.net`; DataDome → `datadome` script/cookie; **status `202`/`403`/`429` as a bot-gate** (`202` is an IMDb-style pre-gate, per `detect.ts:isBlocked`). **Invariant: the classifier is purely *additive* — it attaches `CrawlOutcome.block_type` and feeds telemetry/routing; it MUST NOT change *when* `SUCCESS` is returned, because `03c` bills on `status == SUCCESS` (`CrawlOutcome` docstring).** Wiring mirrors `03d`'s proven `captcha_state` pattern: a per-call `block_state` dict mutated in `_build_result` (which has `raw_html` + `status`) and stamped onto every `CrawlOutcome` return path in `crawl_url`.
-- **Per-domain strategy memory (Slice C — needs a new dependency + control-flow change).** Cache the **strategy that last succeeded per domain** (Redis key `crawl:strategy:{domain}` with TTL). Next crawl starts at the known-good tier, skipping cheaper tiers that always fail there. **Why deferred:** `WebCrawlerConnector()` is currently constructed with **no args** and is Redis/DB-free; this requires injecting a cache client (best-effort, no-op when absent) **and** letting `crawl_url` skip ladder tiers — a control-flow change to the tuned ladder. No DB migration (Redis, best-effort, self-healing on miss), but it is a deliberate design item, not a flag flip.
-- **Solved-session cache (Slice C refinement — speed-positive, the strongest version of the memory above).** Beyond remembering *which tier* won, cache the **solved session itself** — the `cf_clearance`/`__cf_bm` cookies + the UA that cleared the challenge — in Redis (`crawl:session:{domain}` with TTL), and **replay** it on the next crawl via Scrapling's `cookies=` input, so a returning hit **skips the expensive solve entirely** (the pattern `references/trawl-dev` Tier 2 uses for its ~500ms repeat path: `browser/src/session.ts` `session:{domain}` save/load/invalidate; `tiers/src/tier2.ts` replays cookies, and on a re-challenge marks `session-expired` → invalidate → escalate). **Net effect is faster, not slower** (it removes a solve), so it fits the "no speed regression" bar. **Caveat:** `cf_clearance` is **IP+UA-bound**, so it pays off most behind **sticky/static proxies** (the `03b` static-proxy future) and the cache key should include proxy-geo; under rotating residential the replayed cookie may not match the new exit IP (best-effort, self-healing via the `session-expired` invalidate).
-
-## 8. Deferred: paid-unblocker tier (the escape hatch, NOT built now)
-
-The moat strategy is explicit: **maintain in-house bypass for ~4–6 months, then move hostile targets to a paid unblocker if demand/maintenance justifies it.** That switch is **evidence-driven, not a guess** — `03f`'s manual scorecard quantifies the free-stack ceiling over time and is the documented trigger for flipping this tier. Pre-wire the seam so the switch is a config flip, not a refactor:
-
-- Define (but do not implement) a paid-unblocker tier — a `_crawl_with_paid_unblocker(url) -> dict | None` method returning the same shape as the existing tiers, appended **last** in `crawl_url`'s ladder (after the stealthy tier) and active only when an env flag + API key are set. (Same plain-method shape as today's tiers — there is no `FetchStrategy` contract to satisfy.)
-- It would call an external unblocker (ZenRows/ScrapFly/Bright Data Web Unlocker) for the residual `DATADOME`/`KASADA`/`reCAPTCHA-Enterprise` domains the block classifier flags as unreachable in-house.
-- **Its own billing** (cost-plus pass-through, decided at build time) — separate from `03c`'s flat crawl unit and `03d`'s per-solve unit, because unblocker pricing is per-request and provider-specific.
-- Until built, those domains simply return non-`SUCCESS` (free under `03c`). This keeps the umbrella's "WebURL Crawler is the moat" honest while bounding our maintenance risk.
-
-## Config / env changes
-
-Add (all default OFF / conservative; next to the `03b`/`03c` knobs in `config/__init__.py` + `.env.example`):
-
-- `CRAWL_GEOIP_MATCH_ENABLED` (default FALSE; Slice A — maps `RESIDENTIAL_PROXY_LOCATION` → `locale`/`timezone_id`, no exit-IP lookup).
-- `CRAWL_BLOCK_WEBRTC` (default TRUE — cheap, safe; Slice A), `CRAWL_HIDE_CANVAS` (default **FALSE** — random canvas noise can itself be a tell; opt-in + `03f`-validated, see §2; Slice A), and `CRAWL_GOOGLE_SEARCH_REFERER` (default TRUE — Scrapling's `google_search`; Slice A).
-- `CRAWL_DNS_OVER_HTTPS` (default **FALSE** — anti DNS-leak, but adds a DNS round-trip; default-off to honor the "no speed regression" bar, pre-wired for leak-safety-first operators; §2; Slice A).
-- `CRAWL_WEBGL_SPOOF_ENABLED` (default FALSE) + optional `CRAWL_WEBGL_VENDOR` / `CRAWL_WEBGL_RENDERER` override strings (§2c; `toString`-safe JS spoof, defeats string-checks only).
-- `CRAWL_PERSISTENT_PROFILES_DIR` (unset → Scrapling's temp default; set → per-domain profiles).
-- `CRAWL_HEADED_XVFB_ENABLED` (default FALSE; requires Xvfb in the image).
-- `CRAWL_HUMANIZE_ENABLED` (default TRUE) + dwell/jitter bounds.
-- `CRAWL_STRATEGY_MEMORY_TTL_S` (default e.g. `86400`; 0 → disabled).
-- `CRAWL_PAID_UNBLOCKER_ENABLED` (default FALSE) + provider/key (deferred tier).
-
-## Docker changes (`surfsense_backend/Dockerfile`)
-
-- Install **Xvfb** + the proven font set (`fonts-noto-color-emoji`, `fonts-unifont`, `fonts-ipafont-gothic`, `fonts-wqy-zenhei`, `fonts-tlwg-loma-otf`, `fonts-dejavu`, `fonts-liberation`; §5) in the worker image. (Note from `03a`: also drop the stale "+ Camoufox" comment near `:112`; `scrapling install` only fetches Chromium.)
-- Headed runs need the browser launched under `xvfb-run` (or an Xvfb display in the worker entrypoint), gated by `CRAWL_HEADED_XVFB_ENABLED`.
-
-## Build slicing (risk-staged; `03f` validates effectiveness)
-
-The levers split by risk/testability. **Slice A is built now** (prod-safe, config-gated, defaults preserve today's behavior, wiring + classifier unit-tested). **B/C are deferred** (need infra or architectural design; mostly only verifiable in `03f`).
-
-- **Slice A (SHIPPED):** centralized kwargs builder → fingerprint flags (`block_webrtc` ON, `hide_canvas` OFF, `google_search` ON, `dns_over_https` OFF); geoip coherence from `RESIDENTIAL_PROXY_LOCATION`; fonts + Xvfb **packages** in the Dockerfile; **additive** block classifier (`CrawlOutcome.block_type`, classified in `_build_result` **and** the static tier's `>=400` early-return so the first/cheapest-tier bot-gate isn't lost). **Licensing split:** the stealth kwargs builder is **proprietary** (`app/proprietary/web_crawler/stealth.py` — bypass tuning); the block classifier stays **Apache-2** (`app/utils/crawl/classifier.py` — generic passive telemetry from public markers). **`extra_headers`/`additional_args` were intentionally NOT wired** — they're last-priority override hatches with no clean env representation (dict-in-env) and no concrete need yet; add them when a real use appears rather than speculative config (YAGNI).
-- **Slice B (next):** headed-Xvfb **runtime path** (`headless=False` + entrypoint), humanization `page_action` (composed with `03d`'s injector), WebGL-spoof `init_script` `.js` asset (proprietary).
-- **Slice C (defer):** per-domain persistent profiles (concurrency design, §3), per-domain strategy memory + tier-skipping (`crawl_url` control-flow change + cache injection, §7), **solved-session cache** (`cf_clearance` replay via `cookies=`, §7 — speed-positive, best with sticky proxies), **warm browser pool** (a hosted-scale latency win — `references/trawl-dev` `browser/src/pool.ts` keeps N warm instances with an acquire-timeout→429; Scrapling's `StealthySession` already has page pooling), paid-unblocker config stub (§8). GPU passthrough stays infra/hosted-only (§2c).
-
-## Work items
-
-**Slice A (this pass):**
-1. **Config + `.env.example`** — add the Slice-A knobs (below); geoip reuses `RESIDENTIAL_PROXY_LOCATION`.
-2. **`app/proprietary/web_crawler/stealth.py`** (**proprietary** — bypass tuning) — `StealthConfig` + `get_stealth_config()` snapshot; coarse `RESIDENTIAL_PROXY_LOCATION → (locale, timezone_id)` map; `build_stealthy_kwargs(cfg)` returning the config-derived `StealthyFetcher` kwargs. Single source of truth imported by the connector **and** `03f`'s harness (no test-vs-prod drift). The AsyncFetcher tier **already carries `impersonate="chrome"`** (shipped `03a`, §2b) — leave it; document it as the static-tier coherence anchor.
-3. **`app/utils/crawl/classifier.py`** (Apache-2) — `BlockType` enum + `classify_block(status, html) -> BlockType` (pure, status + body-marker based).
-4. **Connector wiring** — merge `build_stealthy_kwargs(...)` into `_crawl_with_stealthy_sync`'s kwargs (existing `headless`/`network_idle`/`block_ads`/`solve_cloudflare`/`proxy`/captcha `page_action` preserved; lever keys never collide); add `block_type` to `CrawlOutcome`; thread a per-call `block_state` dict (classify in `_build_result` **and** the static tier's `>=400` early-return path, stamp on every `crawl_url` return) — **additive only, SUCCESS predicate unchanged**.
-5. **Docker** — Xvfb + the proven font set in the worker image; drop the stale "+ Camoufox" comment.
-6. **Tests + lints** — unit-test the builder (geoip map, flag wiring, snapshot) + classifier (each marker); update connector tests for `block_type`; run suite.
-
-**Slice B / C (deferred — tracked above):** WebGL-spoof `init_script`, humanization composer, headed-Xvfb runtime, persistent profiles, strategy memory + tier-skip, paid-unblocker stub, richer `(domain, block_type, winning_strategy, attempts, latency)` instrumentation.
-
-## Risks / trade-offs
-
-- **Arms race / maintenance.** Fingerprint bypasses rot as WAFs update; this is exactly the cost the deferred paid tier (§8) hedges. Instrumentation (work item 9) plus **`03f`'s scorecard** are what tell us when in-house upkeep stops being worth it.
-- **WebGL/GPU is a structural gap, not a bug (§2c).** The free stack cannot present a real consumer GPU; the JS spoof defeats string checks but not render-output/perf probes, and is itself brittle (rots on Chrome updates, must stay `toString`-clean). Plan for device-fingerprint-grade targets to need GPU-passthrough workers or the §8 paid tier — don't expect parity with CloakBrowser here.
-- **Headed/Xvfb cost.** Headful browsers use more CPU/RAM than headless; gate per-flag and only escalate to headed when the block classifier says cheaper tiers fail for a domain (per-domain memory keeps it from being the default).
-- **Profile growth.** Persistent profiles accumulate disk; add a size/TTL cap and periodic prune.
-- **Geoip accuracy.** A coarse country→locale map is fine; over-fitting per-city tz isn't worth it. Wrong-but-coherent beats default-mismatched.
-- **No silver bullet.** Reiterate the ceiling (§"realistic ceiling") in any user-facing copy: the crawler is "best-effort undetectable," not guaranteed.
-
-## Prior art evaluated — why not Camoufox (decision: skip for now)
-
-`references/Byparr-main` and `references/trawl-dev` are FlareSolverr alternatives built on **Camoufox** — an **MPL-2.0**, open-source, **C++/Juggler-patched Firefox** that natively closes our biggest gaps: hardware-accurate **WebGL** renderer/vendor spoofing (bundled `webgl_data.db`, §2c), **`geoip=True`** locale/tz/geo coherence from the proxy IP (§1), **`humanize=True`** mouse, plus font/WebRTC/audio/screen spoofing — all at the engine level (no detectable JS shims). It is effectively the CloakBrowser-class capability *without* CloakBrowser's licensing problem.
-
-**Decision (this MVP): do not adopt Camoufox.** Rationale: Scrapling **dropped** Camoufox in favor of patchright-Chromium for its `StealthyFetcher` ([scrapling stealthy docs](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html); confirmed in source — `references/Scrapling/scrapling/engines/_browsers/_stealth.py` imports `patchright`), so wiring Camoufox would mean a **separate proprietary fetch tier** (camoufox+playwright, outside Scrapling), a Firefox engine (a few Chrome-only targets won't fit), a heavy extra binary, and tracking an actively-but-experimentally-developed project. **Revisit trigger:** if `03f`'s scorecard shows the patchright tier's device-fingerprint ceiling (§2c) is blocking real CI targets, a Camoufox tier is the prime free escalation to evaluate **before** the §8 paid unblocker.
-
-## Out of scope (hand-offs)
-
-- Cloudflare solving (`03a`), reCAPTCHA/hCaptcha solving + its per-solve billing (`03d`), proxy rotation (`03b`), flat crawl billing (`03c`). **Forward note for `03d` (free solver fallback):** the Camoufox-based references solve reCAPTCHA-v2 / Turnstile **free** in-browser (audio-STT — `references/trawl-dev/packages/tiers/src/solvers/{stt,turnstile}.ts`) instead of a paid solver. `03d` shipped with paid `captchatools` as primary; a free STT path is a possible *fallback-before-paid* later, but audio STT is brittle/Google-rate-limited, so paid stays primary (and this doesn't change `03e`).
-- **Logged-in / account-based bypass** (sticky/static proxies + credential management) — deferred to the platform-actor work (umbrella Phase 8 + `03b` static-proxy hand-off). Public data only this MVP.
-- Building the paid-unblocker provider integration — deferred (§8 leaves only the seam + flag).
-- **Measuring** undetectability (the scorecard that grades these levers) → `03f` (manual harness). This plan *builds* the levers; `03f` *tests* them.
-- Platform-specific structured extractors (Google Maps, LinkedIn public, …) — these sit **on top** of this hardened fetch core as Phase-8 actors; this plan only delivers the core they depend on.
diff --git a/plans/backend/03f-undetectability-testing.md b/plans/backend/03f-undetectability-testing.md
deleted file mode 100644
index dc5af1380..000000000
--- a/plans/backend/03f-undetectability-testing.md
+++ /dev/null
@@ -1,155 +0,0 @@
-# Phase 3f — Undetectability & extraction test harness (manual scorecard)
-
-> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
-> **Status: ✅ IMPLEMENTED + first baseline run (`ci_mvp`, 2026-06-30).** Harness lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (package: `core.py` scorecard/closure-cell helper, `suite_stealth.py` Suite S, `suite_extraction.py` Suite E, `__main__.py` CLI, `README.md` runbook) — it's the moat's measurement tool, so it carries the `app/proprietary/LICENSE`, not Apache-2. Run: `python -m app.proprietary.web_crawler.testbench --suite all [--proxy URL] [--headed] [--no-screenshots]`. Suite S builds the StealthyFetcher tier from the **shipped** `build_stealthy_kwargs(get_stealth_config())` (no drift); Suite E drives the real `crawl_url` against ToS-safe sandboxes with deterministic assertions. Writes timestamped `scorecard-*.json`/`.md` + diffs the last run; the **whole `results/` tree is gitignored** (run-local: scorecards, `latest.json`, screenshots, dumps). Captcha forced OFF (unaided score). **Not** pytest-collected (live internet + proxy). **Manual-only** — no CI/automated gating (resolved decision); it's a dev-run scorecard, not a build gate. **First baseline: see "First baseline results" below.**
->
-> **Post-implementation notes (reconciled to shipped code):** (1) the `03a`/`03e` "`FetchStrategy` seam" never shipped — levers are a centralized kwargs builder, reflected below; (2) **every detection site is now auto-graded from its real DOM verdict** — the initial "INFO + screenshot, read manually" fallback was replaced after a first run by per-site parsers written against actual DOM dumps (reCAPTCHA-v3 reads the server `"score"` JSON, CreepJS the headless-% + boolean spoof tells incl. `hasHeadlessWorkerUA`, incolumitas the fpscanner FAIL keys + `is_datacenter`, fingerprint-scan the `Bot Risk Score`, FingerprintJS the block message, BrowserScan the Normal/Abnormal count, iphey the masthead verdict, Cloudflare the bypass line). `INFO` is now reserved for purely informational rows (TLS JA3/JA4, exit IP) + the manual browserleaks links; screenshots are still captured as a backstop. Each site is one entry in `suite_stealth.py`, so tightening a parser is a one-function change. Depends on `03a` (the Scrapling tier ladder + `CrawlOutcome`), `03b` (proxy provider), `03e` (the stealth levers being tested), and reuses the `page_action` + closure-cell mechanism shared with `03d`.
-
-> **Convention note.** This is **dev/operator tooling**, not a product code path, so it's untouched by the Phase 1–2 rename (no `search_space_id`/`workspace_id` concern). Citations to the gitignored reference checkouts (`references/CloakBrowser/`, `references/Scrapling/`) are pinned to what's on disk; locate code by **symbol/grep** if lines drift.
-
-## Objective
-
-A **manual, repeatable scorecard** that answers one question: *how undetectable (and how correct) is our Universal WebURL Crawler right now?* It drives the real Scrapling tiers against the industry's standard detection + sandbox sites, parses each site's verdict, and prints a pass/fail + numeric scorecard. Its real job is to **quantify the free-stack ceiling** (`03e`) over time so we know — with evidence — when fingerprint maintenance stops being worth it and we should flip the deferred paid-unblocker tier.
-
-It is explicitly **two axes, two suites** (kept separate so each scales independently):
-
-- **Suite S — Stealth / anti-bot** (is the crawler detected?): browser-tier fingerprint/bot tests, HTTP/TLS-tier fingerprint, proxy/leak verification.
-- **Suite E — Extraction correctness** (does the crawler get the content right?): scraping sandboxes for the HTTP vs JS (DynamicFetcher) tiers + Trafilatura quality.
-
-## How CloakBrowser tests (the pattern we copy)
-
-CloakBrowser's suite is a **driven-browser verdict-scraper**, not unit tests (`references/CloakBrowser/examples/stealth_test.py`, run via `bin/cloaktest`). The repeated shape:
-
-1. Launch a real page with **proxy + geoip**, toggleable **headed/headless** (`stealth_test.py:231` `launch(headless=…, proxy=…, geoip=True)`).
-2. `page.goto(site, wait_until="networkidle")` then **sleep** — scores compute async (Castle.js ~20 s `fingerprint_scan_test.py:31`; CreepJS ~30 s `:96`; reCAPTCHA polls up to 30 s `stealth_test.py:156–164`).
-3. `page.evaluate(js)` to **parse the verdict** from the rendered DOM (sannysoft table `stealth_test.py:32–45`) or from internal JS objects (`window.Fingerprint.headless` `fingerprint_scan_test.py:117–127`).
-4. Apply an explicit **pass threshold** + screenshot (`stealth_test.py:169–217`).
-
-Their shipped bars (we adopt as **aspirational targets**, see Scorecard): sannysoft **0 fails**; `bot.incolumitas` ≤ `{WEBDRIVER, connectionRTT}` known-FPs; `browserscan` **0 Abnormal**; `deviceandbrowserinfo` `isBot=false`; FingerprintJS demo **not blocked**; reCAPTCHA v3 **≥0.7** (they hit 0.9); CreepJS **headless ≤30% / stealth ≤30%** (`fingerprint_scan_test.py:112–114,166–171`).
-
-## The bridge to our Scrapling crawler
-
-We are **not** a single browser; we're the `03a` tier ladder. The harness therefore tests **per tier**, and extracts verdicts two ways:
-
-- **DOM/JSON-rendered verdicts** → just `StealthyFetcher.fetch(url, …)` (or `Fetcher.get` for JSON endpoints) and parse the **returned post-JS page** with Scrapling's selector (`load_dom` is on by default — `references/Scrapling/scrapling/fetchers/stealth_chrome.py:43`). Covers sannysoft, incolumitas, deviceandbrowserinfo, the reCAPTCHA score text, and every JSON endpoint (`tls.peet.ws/api/all`, `httpbin/headers`).
-- **Internal JS-object verdicts** → a **`page_action`** that runs `page.evaluate()` and stashes the result into a **closure cell**, because Scrapling **discards `page_action`'s return value** (sync `_stealth.py:260`, async `:536`). **This is the exact same `page_action`+closure-cell plumbing as `03d`'s token injector** — building the harness de-risks `03d` (and vice-versa); factor it once. *(As-built: CreepJS/fingerprint-scan turned out to render their verdicts into the visible DOM, so their parsers read the post-JS page text directly; the closure-cell `page_action` path is retained as the mechanism for any future `window.*`-only verdict + for the screenshot backstop.)*
-
-## Suite S — Stealth / anti-bot
-
-### S1. Browser tier (StealthyFetcher — the "undetectable" tier)
-
-All rows are **auto-graded** from the post-JS DOM text (no manual screenshot read); the `Extraction` column names the actual marker each parser keys on.
-
-| Site | Signal | Extraction | Aspirational bar |
-|---|---|---|---|
-| `bot.sannysoft.com` | webdriver/chrome/plugins/UA leaks | `class="failed"` cell count | 0 fails |
-| `bot.incolumitas.com` | 30+ checks incl. behavioral + IP class | `"":"FAIL"` keys, `is_datacenter` | 0 fpscanner FAIL |
-| `browserscan.net/bot-detection` | WebDriver/CDP/Navigator | `Normal`/`Abnormal` count | 0 Abnormal |
-| `deviceandbrowserinfo.com/are_you_a_bot` | fingerprint + behavioral | `"isBot":` JSON | `isBot=false` |
-| `abrahamjuliot.github.io/creepjs` | fingerprint **consistency/lies**, headless% | `N% headless` + boolean tells (`hasHeadlessWorkerUA`…) | headless ≤30%, no tells |
-| `fingerprint-scan.com` | Castle.js bot-risk + headless signals | `Bot Risk Score: N/100` | risk < 50 |
-| `demo.fingerprint.com/web-scraping` | **behavioral block** (FingerprintJS Pro Smart Signals) | block message (`access denied` / `tampering detected`) | not blocked |
-| `recaptcha-demo.appspot.com/...v3-request-scores.php` | Google server-verified human score | server verify `"score":` JSON | score ≥0.7 |
-| `www.scrapingcourse.com/cloudflare-challenge` | **Cloudflare challenge** (only row exercising `solve_cloudflare`) | `you bypassed the cloudflare challenge` line | bypassed |
-| `iphey.com` | cross-layer fingerprint+IP+geo coherence | masthead `Your Digital Identity Looks ` (async, ~25 s settle) | `Trustworthy` |
-
-### S2. Per-property fingerprint detail (manual/debug — validates `03e` levers directly)
-
-- `browserleaks.com/canvas`, `/webgl`, `/fonts` — confirms `03e` `hide_canvas` + font packages.
-- `browserleaks.com/webrtc` + DNS leak — confirms `03e` `block_webrtc` (no real-IP leak through the proxy).
-
-### S3. HTTP/TLS tier (AsyncFetcher / `Fetcher` — curl_cffi impersonation)
-
-Our HTTP tier is `curl_cffi`-based and impersonates a real Chrome TLS stack (`references/Scrapling/scrapling/fetchers/requests.py:29`; `engines/static.py:6–9,36–47`) **only when `impersonate=` is set** — which `03a` **now ships** (`app/proprietary/web_crawler/connector.py`, the `AsyncFetcher.get` call passes `impersonate="chrome"`). This row therefore **validates the shipped parity** rather than driving a fix: confirm the static tier's JA3/JA4 matches a real Chrome (the `03e §2b` lever). If you ever need a before/after, temporarily drop `impersonate=` to reproduce the curl_cffi-default (red) baseline.
-
-- `tls.peet.ws/api/all` (+ `/api/clean`) — JSON **JA3/JA4/Akamai-HTTP2/PeetPrint**; diff against a real-Chrome baseline.
-- `httpbin.co/headers` (or httpbingo) — header set/order/UA sanity.
-
-**Recommendation (resolved): TLS parity is a first-class *axis* but an *informational threshold*, not a hard gate.** We record JA3/JA4 and flag drift from the Chrome baseline, but don't "fail" on it — curl_cffi impersonation is strong yet JA-hashes shift across versions, and this is a manual scorecard anyway. Treat a *mismatch* as a tuning signal (pick a closer `impersonate` profile), not a regression.
-
-### S4. Proxy / leak verification
-
-- `httpbin.org/ip` — exit IP == the proxy endpoint actually used (capture-once seam from `03b`/`03e`, not a re-rotated `get_proxy_url()`).
-- WebRTC/DNS (S2) — no real-IP leak.
-
-## Suite E — Extraction correctness (separate axis)
-
-Purpose-built, ToS-safe sandboxes — validates the HTTP vs **DynamicFetcher (JS)** tiers and Trafilatura output, independent of stealth:
-
-- `books.toscrape.com` — static catalog + pagination (HTTP tier; baseline extraction).
-- `quotes.toscrape.com` — has **`/js`** (JS-rendered), **`/js-delayed`** (`?delay=`), **scroll** (infinite), **login** (CSRF) variants → exercises the DynamicFetcher JS tier + `wait_selector`/`network_idle`.
-- `scrapethissite.com` — mixed structures for extraction robustness.
-
-Assertion style: known expected values (e.g. first book title, quote count per page) so extraction regressions are caught deterministically.
-
-## First baseline results (2026-06-30, headless, rotating residential proxy)
-
-First real run of the free stack (patchright-Chromium + curl_cffi `impersonate="chrome"` + `anonymous_proxies` rotating residential), captcha solving OFF, Slice-A levers only (`CRAWL_GEOIP_MATCH_ENABLED=false`). **Suite S: 6 PASS / 4 FAIL across the 10 detection sites** (the other 6 rows are informational: TLS, exit IP, 4 browserleaks manual links).
-
-| Site | Verdict | Observed |
-|---|---|---|
-| sannysoft | ✅ PASS | 0 failed cells |
-| deviceandbrowserinfo | ✅ PASS | `isBot=false` (incl. `isPlaywright=false`, `isAutomatedWithCDP=false`) |
-| reCAPTCHA v3 | ✅ PASS | server score **0.9** |
-| BrowserScan | ✅ PASS | Test Results: Normal, 0 Abnormal (WebDriver/CDP/Navigator) |
-| fingerprint-scan | ✅ PASS | Bot Risk **35/100** (site flags >50) |
-| cloudflare_challenge | ✅ PASS | turnstile **solved** → "you bypassed the Cloudflare challenge" |
-| **CreepJS** | ❌ FAIL | headless **33%**, **`hasHeadlessWorkerUA: true`** (worker UA leaks `HeadlessChrome`) |
-| **incolumitas** | ❌ FAIL | only legacy `fpscanner WEBDRIVER: FAIL`; all modern tests OK |
-| **FingerprintJS Pro** | ❌ FAIL | "anti-detect tampering detected, access denied" |
-| **iphey** | ❌ FAIL | verdict **Unreliable** |
-
-**Reading the failures (maps directly to the moat roadmap):**
-- **FingerprintJS Pro** = the commercial bar (the documented free-stack ceiling). Needs WebGL/GPU + deeper patches (`03e` Slice B/C) or the deferred paid-unblocker (`03e §8`). Hardest.
-- **CreepJS `hasHeadlessWorkerUA`** = the **Web Worker** `navigator.userAgent` still reports `HeadlessChrome` (main-thread UA is clean). Known patchright leak, **plausibly fixable** (worker UA override) → concrete `03e` Slice-B candidate.
-- **iphey "Unreliable"** = almost certainly **geoip incoherence** (browser tz/locale default `America/Los_Angeles` vs the rotating proxy's exit geo, because `CRAWL_GEOIP_MATCH_ENABLED` is default-off in Slice A). iphey is now a **live regression test for the `03e` geoip-coherence lever** — flipping `CRAWL_GEOIP_MATCH_ENABLED` is the expected fix to validate.
-- **incolumitas** = a single 2017-era `fpscanner WEBDRIVER` check that even some real browsers trip; modern checks (webdriverPresent, SELENIUM_DRIVER, HEADCHR_*, CDP) all OK. Lowest priority.
-
-**Proxy-quality note:** incolumitas's IP classifier returned `is_datacenter: true` on one rotation and `false` on another — the `anonymous_proxies` rotating pool **mixes datacenter and residential exits**, a real undetectability variable independent of our code (input to future proxy-provider evaluation).
-
-## Scorecard & thresholds (resolved)
-
-- **Adopt CloakBrowser's bars as aspirational targets** (above), but the harness's primary output is **our actual measured numbers recorded as the baseline** — a `scorecard.md`/JSON snapshot per run (date, tier, proxy on/off, headed/headless, per-site result), written to the **gitignored** `results/` tree. Subsequent runs diff against the last on-disk baseline so we see drift (ours improving, or a WAF tightening).
-- Each row reports: site, tier, verdict, numeric (where applicable), PASS/FAIL vs aspirational bar, and screenshot path.
-- A run is summarized as `passed/total` per suite (like `stealth_test.py:314–317`), **never blocking** anything.
-
-## Harness design
-
-- Lives under the **proprietary boundary** at `surfsense_backend/app/proprietary/web_crawler/testbench/` (moat measurement tooling — not Apache-2; not collected by the normal pytest run, since it hits the live internet + needs proxies). It is moved here as a coherent package because it can't be cleanly *partially* moved (a proprietary Suite S would otherwise back-import generic scaffolding from `scripts/`, a forbidden app→scripts direction). A thin CLI mirrors `bin/cloaktest`: `python -m app.proprietary.web_crawler.testbench [--proxy URL] [--headed] [--suite S|E|all] [--no-screenshots]`.
-- **Reuse split (important — `crawl_url` is *not* a drop-in here):**
- - **Suite E** drives the **real `crawl_url`** end-to-end — extraction correctness *is* the production path (auto-ladder + Trafilatura markdown is exactly what we want to assert).
- - **Suite S** does **not** use `crawl_url`. Two reasons: (1) `crawl_url` auto-ladders and stops at the first `SUCCESS`, so a detection site might be answered by the cheap HTTP tier when we mean to grade the **StealthyFetcher** tier; (2) `crawl_url` returns Trafilatura **markdown** (`_build_result`), but verdict parsing needs the **raw post-JS DOM** (and `window.*` objects need a live page). So Suite S drives the **individual Scrapling fetchers directly**, per tier.
- - **Avoid test-vs-prod drift:** Suite S must construct the StealthyFetcher tier from the **same centralized stealth-config builder** `03e` shipped — `build_stealthy_kwargs(get_stealth_config())` in `app/proprietary/web_crawler/stealth.py` (the single source of truth for `block_webrtc`/`hide_canvas`/`google_search`/`dns_over_https` + geoip `locale`/`timezone_id`) — **not** a hand-rolled kwargs set, otherwise the scorecard grades a browser we don't ship. The static (HTTP) tier's `impersonate="chrome"` anchor stays hardcoded in the connector (`03a` §2b); `real_chrome`/headed/persistent-profile are Slice B/C and **not** yet in the builder, so the harness exercises today's Slice-A browser.
-- Outputs: console summary + screenshots + the `scorecard` snapshot.
-- Runs with the app-wide proxy provider (`03b`) and `03e` levers on, so the scorecard reflects production fetch behavior — **except captcha solving, which Suite S forces OFF** (`CAPTCHA_SOLVING_ENABLED=false`): we measure the *unaided* stealth/score, and it avoids the `03d` injector firing paid solves against the reCAPTCHA-demo / FingerprintJS rows. Captcha solving is exercised separately (functional `03d` test), not in the stealth scorecard.
-
-## The ceiling-decision loop (why this matters)
-
-The scorecard is the evidence base for the moat strategy: re-run it (a) on a cadence and (b) whenever crawls start failing in the wild. When the **hard** rows (FingerprintJS demo, reCAPTCHA-v3 ≥0.7, and any DataDome/Kasada targets) degrade and `03e` levers can't recover them, that's the documented trigger to flip the **deferred paid-unblocker tier** (`03e §8`). Without this harness, that decision is guesswork.
-
-## Config / dependencies
-
-- **No new production dependencies** — reuses Scrapling (already pinned) + the crawler. Optionally a tiny dev helper for the scorecard diff (or just stdlib `json`).
-- Needs **residential proxy creds** (the `03b` env) to be meaningful; document a "no-proxy" mode that still runs but is expected to fail the harder rows (datacenter IP).
-- A results dir — **entirely gitignored** (run-local scorecards, `latest.json`, screenshots, dumps); kept tracked only via its `.gitignore`.
-
-## Work items
-
-1. **Shared `page_action`+closure-cell helper** (factored with `03d`) for JS-object verdict extraction.
-2. **Suite S runners** (S1 browser, S3 TLS, S4 proxy-leak) with per-site parse + aspirational thresholds; S2 as manual debug links.
-3. **Suite E runners** against the toscrape/scrapethissite sandboxes with deterministic assertions.
-4. **CLI + scorecard** (snapshot writer + diff vs last baseline) mirroring `bin/cloaktest` ergonomics.
-5. **Docs**: a short runbook (how to run, read the scorecard, interpret the ceiling trigger).
-
-## Risks / trade-offs
-
-- **Flaky/rate-limited/changing sites.** Detection sites move DOM and tighten over time; the harness must tolerate parse misses (report `ERROR`, not crash — `stealth_test.py:298–300`) and is **manual** precisely so flakiness never blocks development.
-- **Proxy required for realism.** Without residential egress the hard rows fail by design (datacenter IP); document this so a red scorecard isn't misread.
-- **ToS.** These are public detection/sandbox sites intended for this purpose; keep volume low and don't hammer.
-- **Not a guarantee.** Passing sannysoft/CreepJS ≠ beating DataDome/Kasada; the scorecard's value is *trend + ceiling visibility*, not a green checkmark.
-
-## Out of scope (hand-offs)
-
-- **CI automation** — deferred (resolved: manual-only now). If revisited, it needs tolerant thresholds + proxy creds + nightly cadence; the scorecard JSON is designed to make that easy later.
-- The levers themselves (`03e`), captcha solving (`03d`), proxy provider (`03b`), billing (`03c`) — this plan only **measures** them.
-- The deferred paid-unblocker integration (`03e §8`) — the harness defines its *trigger*, not its build.
diff --git a/plans/backend/04-capabilities.md b/plans/backend/04-capabilities.md
deleted file mode 100644
index 0e47fe155..000000000
--- a/plans/backend/04-capabilities.md
+++ /dev/null
@@ -1,89 +0,0 @@
-# Phase 4 — Capabilities (the scraper APIs)
-
-> Build first; every other phase calls this registry. Sibling: `05-access.md` (the doors).
-> Reuses Phases 3a/3b/3c (shipped): `WebCrawlerConnector.crawl_url -> CrawlOutcome`, the proxy/stealth/
-> captcha tiers, and the `WebCrawlCreditService` billing seam. Locate code by symbol/grep.
-
-## Objective
-
-Expose the Acquisition engine as a small set of typed, stateless scraper verbs (`web.*`, `maps.*`) that
-return cleaned, AI-ready structured data. One capability = one function you call and get data back from.
-
-## Verb set (MVP)
-
-Two namespaces: `web.*` (generic crawler) and `maps.*` (Google Maps actor). Future platforms slot in as
-their own namespace.
-
-| Verb | Input → Output (cleaned) | Executes over | Billing unit |
-|------|--------------------------|---------------|--------------|
-| `web.scrape(urls[])` | `[{url, status, content, metadata}]` | loop `crawl_url` | per success |
-| `web.discover(query, top_k)` | `[{url, title, snippet, provider}]` | search providers | per search / free |
-| `maps.search(query, location)` | `[place]` | Maps actor | per place |
-| `maps.place(place_id\|url)` | `place` | Maps actor | 1 |
-| `maps.reviews(place)` | `[review]` (paged) | Maps actor | per page |
-
-`maps.*` returns typed structured objects (`{name, rating, review_count, hours, price_level, …}`).
-`web.scrape` takes a URL array.
-
-## Execution
-
-Each verb is a direct async call that returns its cleaned result. Slow verbs (large `web.scrape` arrays,
-`maps.reviews` paging) are bounded/streamed at the endpoint.
-
-## Registry
-
-One entry per verb, and the single source of truth the doors (`05`) and the agent (`07`) read from:
-
-```
-Capability {
- name # dotted, e.g. "web.scrape", "maps.search"
- input_schema # Pydantic
- output_schema # Pydantic (cleaned, AI-ready)
- executor # async fn; wraps Acquisition / Maps actor; returns data directly
- billing_unit # how the billing service charges this call
-}
-```
-
-Adding a verb once lights it up on every door.
-
-## Billing
-
-A verb declares a `billing_unit`; charging is delegated to the billing service (`03c` first provider).
-
-- `web.scrape` → per `SUCCESS` (`web_crawl` unit).
-- `maps.*` → new per-place / per-page unit registered with the billing service.
-- captcha attempts → existing per-attempt `web_crawl_captcha` unit (`03d`).
-- `web.discover` → per-search unit (or free).
-
-## Location
-
-New Apache-2 package `app/capabilities/` (registry, schemas, executors) — open-source and self-hostable.
-Executors import the proprietary Acquisition engine and Maps extractor (`app/proprietary/…`).
-
-## Work items
-
-1. Registry package: `Capability` dataclass + registry, imported by `05` and `07`.
-2. `web.scrape` executor: loop `crawl_url` over the URL array → per-URL cleaned rows; `web_crawl` unit.
-3. `web.discover` executor: wrap the search-provider core (SearXNG/Linkup/Baidu, env-keyed); its unit.
-4. `maps.*` contracts: input/output schemas + executor stubs against the Maps actor.
-5. Output normalization: each executor returns AI-ready structured output.
-6. Billing seam: honor `billing_unit` via the billing service.
-
-## Tests
-
-- `web.scrape([a,b,c])` returns one cleaned row per URL inline; partial failures don't fail the batch.
-- Each `SUCCESS` bills one `web_crawl` unit; `EMPTY`/`FAILED` free; disabling the flag no-ops.
-- A door and the agent hit the same executor for a verb.
-- `web.discover` returns `{url,title,snippet,provider}`; self-disables when no provider key is set.
-- `maps.*` returns typed structured objects against fixtures.
-
-## Out of scope
-
-- Doors → `05`. Keep-watching mode → `06`. Agent → `07`.
-- Google Maps actor build (incl. sourcing legality) — net-new, separate effort; `maps.*` are contracts.
-- Additional platform namespaces; recursive site crawl; a generic `web.extract` verb.
-
-## Open questions
-
-- `web.discover` metered vs free.
-- How large `web.scrape` / `maps.reviews` responses are bounded/streamed.
diff --git a/plans/backend/05-access.md b/plans/backend/05-access.md
deleted file mode 100644
index 1da602d16..000000000
--- a/plans/backend/05-access.md
+++ /dev/null
@@ -1,85 +0,0 @@
-# Phase 5 — Access / Surfaces (REST · MCP · chat doors)
-
-> Build after `04`. Together `04 + 05` ship the scraper-API product.
-> Reuses shipped infra: Identity/Tenancy, API keys, the chat agent + tool registry, the streaming layer,
-> Metering (`03c`), and Rocicorp Zero reactive sync. Locate code by symbol/grep.
-
-## Objective
-
-Expose the `04` capability registry through three doors — REST + API keys, an MCP server, and chat tools —
-each generated from the one registry so the I/O contract can't drift. Every door is the same thin adapter
-with no business logic.
-
-## Adapter shape (identical on every door)
-
-```
-parse input → validate against verb.input_schema → authn/authz → meter-gate (03c)
- → call the SAME executor (04) → serialize verb.output_schema → return cleaned data
-```
-
-The executor returns directly (`04`).
-
-## The three doors
-
-| Door | Who | Auth | Status |
-|------|-----|------|--------|
-| REST + API keys | external developers | existing API-key infra | public day one |
-| MCP server | external agents (Cursor/ChatGPT/Claude) | OAuth 2.1 or bearer (at implementation) | fast-follow |
-| Chat tools | the in-app agent (`07`) | existing session + workspace | seed exists (`scrape_webpage`) |
-
-Routes are generated, so public REST on day one costs nothing extra.
-
-## Chat surface
-
-The in-app product is the conversation: the user describes a need in plain language and the agent (`07`)
-picks and fills verbs. Raw typed verbs are exposed on REST/MCP for devs and external agents. The chat
-door classifies intent and routes:
-
-```
-"find / compare / what is / pull / right now" → ONE-SHOT → call verbs, answer now
-"watch / track / notify me / keep an eye / over time" → KEEP-WATCHING → hand to the ongoing mode (06)
-```
-
-## MCP — two directions
-
-- **Serve MCP** (new door): our capabilities as a remote MCP server — Streamable-HTTP `/mcp`, stateless,
- bounded/paginated outputs, read-only verbs (least privilege).
-- **Consume MCP** (reuse): the BYO-`MCP_CONNECTOR` — the user's own external MCP tools inside our chat
- agent. Fix its routing-map gap here.
-
-## Result delivery
-
-A chat tool call returns its data inline in the turn. Asynchronous results (e.g. the `06` ongoing mode
-writing between turns) reach the client via write-then-sync / stream: the worker writes rows and the
-client receives them over the existing SSE stream or a Zero-published table.
-
-## Work items
-
-1. Door generator: from the `04` registry emit (a) REST routes + models, (b) MCP tool schemas + handlers,
- (c) chat tool defs + handlers.
-2. REST surface: public routes + API-key auth (reuse) + `03c` meter-gate.
-3. Chat tools: generalize `scrape_webpage` into the registry-backed set (direct-return).
-4. Intent routing: classify one-shot vs keep-watching; hand keep-watching to `06`.
-5. MCP server (fast-follow): Streamable-HTTP `/mcp`, least-privilege; auth depth at implementation.
-6. Consume-MCP fix: repair the BYO-`MCP_CONNECTOR` routing-map gap.
-
-## Tests
-
-- A verb added to the registry appears on REST + MCP + chat with identical I/O.
-- REST without a valid key → 401; an over-budget call → blocked by the `03c` gate; a success charges once.
-- A chat scrape returns cleaned data inline in the same turn.
-- "compare X and Y" → one-shot; "watch X weekly" → routed to `06`.
-- A configured BYO MCP tool is reachable by the chat agent.
-
-## Out of scope
-
-- Verbs, executors, billing units → `04`.
-- Keep-watching mechanism + its delivery channel → `06`.
-- The CI subagent + its prompt → `07`.
-- Legacy branded connectors (backward-compat hygiene, separate task).
-
-## Open questions
-
-- MCP auth depth (decide at implementation).
-- Public REST rate-limiting / abuse posture.
-- Async delivery channel for `06` outputs: SSE stream vs Zero-published messages table (settled with `06`).
diff --git a/plans/backend/06-ongoing-automation.md b/plans/backend/06-ongoing-automation.md
deleted file mode 100644
index 8006b4a22..000000000
--- a/plans/backend/06-ongoing-automation.md
+++ /dev/null
@@ -1,65 +0,0 @@
-# Phase 6 — Ongoing Automation (chat-native "keep watching")
-
-> Depends on `04` (the verbs it re-invokes), `05` (the agent tools), and `07` (the `scraping` subagent).
-> Reuses the existing chat + automations machinery; adds no parallel engine.
-
-## Objective
-
-Support "keep watching": a persistent chat where the agent periodically re-invokes scraper verbs and drops
-results into the session. "What changed" is the agent reading its own prior turns from the durable
-checkpoint — no diff store.
-
-## Mechanism
-
-A **chat watch is an `Automation` bound to the current chat** — nothing more. No dedicated thread, no
-thread "kind", no schema change. Start it and the chat gains a `schedule` trigger that re-posts the
-question on a cadence; stop it and the automation is deleted and the chat is a normal chat again. The
-chat's own checkpoint is the memory.
-
-- **`chat_message` action** — params `{ thread_id, message }`. Its handler drains
- `stream_new_chat(user_query=message, chat_id=thread_id, …)` under `AuthContext.system(creator,
- source="automation")` against the **current chat**. Auto-approve (system auth; CI verbs are read-only
- and don't interrupt).
-- **Durable memory + delivery** — `stream_new_chat` persists messages to `new_chat_messages` (already
- Zero-synced to the UI) and advances the shared Postgres checkpointer keyed by `chat_id`. A scheduled run
- has no SSE client; it runs server-side and is delivered via the persisted rows. "What changed" is the
- agent reading the chat's own prior turns.
-- **Worker-safe checkpointer** — the shared `AsyncPostgresSaver` pool binds connections to the loop that
- opened them, but Celery uses a fresh loop per task (`PoolTimeout`). Dispose the checkpointer pool per
- task in `run_async_celery_task`, mirroring `_dispose_shared_db_engine`, so a worker can use the *durable*
- checkpointer (not `InMemorySaver`) that "what changed" requires.
-- **`start_watch`** — a `scraping` subagent tool that binds a watch to the *current* chat: it distills
- the recurring question + cadence and creates the automation (`schedule` + `chat_message(thread_id =
- current chat)`).
-- **"Is this chat watched?"** — derived: an active automation with a `chat_message` action targeting the
- chat. No stored flag.
-- **Controls** — run-now = trigger a run; stop = delete the automation (chat reverts to normal).
-- **Concurrency** — the checkpointer is single-writer per thread; a tick skips if the prior turn on that
- chat is still running (DB `ai_responding` flag).
-
-## Work items
-
-1. Durable checkpointer in workers: `_dispose_shared_checkpointer_pool` in `run_async_celery_task`
- (before + after), mirroring the SQLAlchemy engine dispose. **[done]**
-2. `chat_message` action: params + factory + handler (drains `stream_new_chat`); concurrency guard. **[done]**
-3. Watch service: create (bind `schedule` + `chat_message` automation to a chat) / stop (delete) /
- find-for-thread (is-watched) / run-now. **[done]**
-4. `start_watch` tool on the `scraping` subagent (+ prompt line); binds to the current chat. **[done]**
-5. Controls — chat tools (`stop_watch`, `refresh_watch`) + REST (`GET /automations/watches`,
- `POST /automations/{id}/run`; stop = `DELETE /automations/{id}`). **[done]**
-
-## Tests
-
-- `run_async_celery_task` disposes the checkpointer pool before and after a task. **[done]**
-- `chat_message` drains a turn on the given thread; skips when one is in-flight. **[done]**
-- Watch service: create binds a `schedule` + `chat_message` automation to the chat; stop deletes it;
- find-for-thread filters by plan; run-now launches the schedule trigger. **[done]**
-- `start_watch` / `stop_watch` / `refresh_watch` tools act on the current chat from tool context. **[done]**
-- Watch routes registered on the automations router. **[done]**
-- Integration (running stack): two scheduled runs on one chat — run 2 sees run 1 in checkpoint history.
-
-## Deferred / out of scope
-
-- Zero delivery-cost optimization (signal-column + REST fetch vs full-content sync) — app-wide, separate.
-- Server-side turns surviving browser navigation for *interactive* chat ("zombie streaming").
-- `start_watch` cadence UX refinements. Verbs/doors → `04`/`05`; agent playbook → `07`.
diff --git a/plans/backend/07-orchestration.md b/plans/backend/07-orchestration.md
deleted file mode 100644
index 50f3e6d9a..000000000
--- a/plans/backend/07-orchestration.md
+++ /dev/null
@@ -1,76 +0,0 @@
-# Phase 7 — Orchestration / Conversation (the CI-expert subagent)
-
-> **Status (2026-07-02): shipped for the web verbs.** The `scraping` subagent, its tools, prompt, and
-> router blurb are live; `web.discover`/`web.scrape` compose and the "keep watching" handoff (`06`) is
-> wired. `maps.*` composition is deferred until the Maps actor exists.
-> Sits atop `04`/`05` (and `06` for the ongoing mode). The multi-agent chat runtime (deepagents, subagent
-> dispatch, streaming, citation middleware) is shipped. Net-new here = one builtin subagent + its tools +
-> its prompt. Locate code by symbol/grep.
-
-## Objective
-
-Ship the `scraping` subagent — a builtin CI/scraper-expert that delivers the product through
-plain conversation: understand intent, compose scraper verbs (`04`), answer in plain language, and hand
-standing needs to the ongoing "keep watching" mode (`06`). It reasons over chat history to report what's
-new vs prior tool outputs already in context.
-
-## The subagent
-
-Builtin `subagents/builtins/scraping/`, peer to `research`/`deliverables`, packed via
-`pack_subagent`. Files follow the existing pattern: `agent.py` (`build_subagent`), `tools/index.py`
-(`NAME · RULESET · load_tools`), `description.md` (router one-liner), `system_prompt.md` (the playbook).
-The main agent delegates CI-/scraping-flavored requests to it via `description.md`.
-
-## The playbook (`system_prompt.md`)
-
-1. **Intent routing** — one-shot ("compare/find/what is") → compose verbs & answer; standing concern
- ("watch/track/notify when/weekly") → hand to `06`; ambiguous → ask one question ("just once, or keep
- watching?").
-2. **Verb composition** — chains `web.discover → web.scrape` and `maps.search → maps.place → maps.reviews`;
- infer URLs/queries/locations from context.
-3. **"What changed"** — re-invoke the verb and compare against prior tool outputs in the chat history,
- summarizing what's new.
-
-## The toolset (`load_tools`)
-
-| Tool | Wraps | Billing |
-|------|-------|---------|
-| capability verbs — `web.scrape`, `web.discover` shipped; `maps.search`/`maps.place`/`maps.reviews` deferred (Maps actor) | `04` executors (direct-return) | `03c` owner-wallet charge via `charge_capability` (same path as REST/MCP) |
-| `start_watch` / `stop_watch` / `refresh_watch` | bind a watch to the current chat via `06` | per re-invocation |
-
-Verbs reach the subagent through the chat door generator `access/chat.py::build_capability_tools` (`05`),
-which turns registry verbs (`04`) into tools with owner-wallet billing. The `scraping` subagent owns these
-CI-flavored calls; the main agent's `web_search`/`scrape_webpage` stay untouched.
-
-## Boundaries
-
-- The subagent drives the `04` verbs via tools; fetch/bypass/clean logic stays in Acquisition + `04`,
- callable headless.
-- Humans get the agent; REST/MCP callers call `04` verbs directly, skipping this subagent.
-
-## Work items
-
-1. **[done]** Subagent scaffold: `subagents/builtins/scraping/` (`agent.py` / `tools/index.py` /
- `description.md` / `system_prompt.md`), packed via `pack_subagent`.
-2. **[done]** CI playbook prompt: intent routing (block on ambiguous cadence), verb chains, "what changed" reasoning.
-3. **[done]** Chat door generator: `access/chat.py::build_capability_tools` turns registry verbs (`04`) into tools (`05`).
-4. **[done]** `start_watch` / `stop_watch` / `refresh_watch` seam to `06`, bound to the current chat.
-5. **[done]** Router registration: `description.md` for main-agent delegation.
-
-## Tests
-
-- A CI-/scraping-flavored request routes to `scraping`; a non-CI request does not.
-- "compare X and Y" composes verbs and answers in plain language, persisting nothing.
-- "reviews for the top 3 coffee shops near me" chains `maps.search → maps.place → maps.reviews`.
-- A follow-up re-invokes the verb and reports new items vs prior in-context outputs.
-- "watch this weekly" routes to the ongoing mode, not a one-shot answer.
-
-## Out of scope
-
-- Verbs, doors → `04`/`05`. Ongoing/periodic mechanism → `06`.
-- Richer multi-step playbooks, proactive suggestions, cross-topic synthesis.
-- Frontend CI surfaces → frontend umbrella.
-
-## Open questions
-
-- None outstanding for the web verbs. `maps.*` composition reopens when the Maps actor lands.