chore: synced plans

plans/backend/01-rename-db.md

@@ -72,17 +72,20 @@ These reference `search_space_id` by COLUMN but the index/constraint NAME does n
 ### Zero publication (single source of truth)
 
 - [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py) is "the single source of truth for `zero_publication`" (module docstring, lines 1-11). Publication changes "should update `ZERO_PUBLICATION` and call `apply_publication()` from a migration" (lines 3-5).
-- `search_space_id` appears in the published column lists:
+- `search_space_id` appears in **four** published column lists (a recent `main` merge added the `automations`/`new_chat_threads` entries — previously only `documents`/`podcasts`):
   - `DOCUMENT_COLS` line 31.
-  - `PODCAST_COLS` line 66.
-- These are the only two published tables with column lists that include the column (`ZERO_PUBLICATION` map, lines 71-82; other entries are `None` = all columns and auto-follow). `documents`/`podcasts` columns must be updated to `workspace_id`, then `apply_publication(op.get_bind())` re-run.
+  - `AUTOMATION_COLS` line 57 (`["id", "search_space_id"]`).
+  - `NEW_CHAT_THREAD_COLS` line 62 (`["id", "search_space_id"]`).
+  - `PODCAST_COLS` line 76.
+  - (`AUTOMATION_RUN_COLS` lines 44-53 does **not** contain the column — `automation_runs` is scoped via `automation_id`, so it auto-follows and needs no edit.)
+- These are the only **four** published tables with column lists that include the column (`ZERO_PUBLICATION` map, lines 81-94; other entries are `None` = all columns and auto-follow). All four (`documents`/`automations`/`new_chat_threads`/`podcasts`) columns must be updated to `workspace_id`, then `apply_publication(op.get_bind())` re-run.
 - Reconcile pattern to copy: [surfsense_backend/alembic/versions/155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) (calls `apply_publication(op.get_bind())`, no-op downgrade).
 
 ## Migration design
 
 ### Conventions (from the repo)
 
-- Revision ids are plain integers; the new migration chains after the **then-current head** — verify with `alembic heads` at implementation time, since the head advances as other PRs merge. Today the head is `166` ([surfsense_backend/alembic/versions/166_add_pat_and_api_access.py](surfsense_backend/alembic/versions/166_add_pat_and_api_access.py)), so the new file would be `167_rename_searchspace_to_workspace.py` with `revision="167"`, `down_revision="166"`. (Note: Phase 4b and Phase 5 each add a migration too; whichever lands first takes the next integer — chain by actual head, not by these illustrative numbers.)
+- Revision ids are plain integers; the new migration chains after the **then-current head** — verify with `alembic heads` at implementation time, since the head advances as other PRs merge. As of the latest `main` sync the head is `169` ([surfsense_backend/alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py](surfsense_backend/alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py); chain `166→167→168→169`), so the new file would be `170_rename_searchspace_to_workspace.py` with `revision="170"`, `down_revision="169"`. (Note: Phase 4b and Phase 5 each add a migration too; whichever lands first takes the next integer — chain by actual head, not by these illustrative numbers.)
 - Migrations run one-per-transaction under an advisory lock — [surfsense_backend/alembic/env.py](surfsense_backend/alembic/env.py) lines 77, 80-95. So all renames below land atomically.
 - Raw `op.execute("ALTER TABLE ...")` is the house style (e.g. [165_add_chunk_position.py](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 39-49).
 
@@ -107,19 +110,19 @@ These reference `search_space_id` by COLUMN but the index/constraint NAME does n
      - `ALTER INDEX IF EXISTS idx_documents_search_space_updated RENAME TO idx_documents_workspace_updated;`
      - `ALTER INDEX IF EXISTS ix_external_chat_bindings_search_space_state RENAME TO ix_external_chat_bindings_workspace_state;`
 4. Zero publication. DECISION: we will recreate the Zero replication (reset the zero-cache replica) as part of this deploy, so consumer recovery is handled out-of-band and finding 3 is de-risked. Follow the repo's established publication patterns — do NOT use raw `DROP`/`CREATE PUBLICATION` (forbidden by [116_create_zero_publication.py](surfsense_backend/alembic/versions/116_create_zero_publication.py) lines 8-17; reintroduces bug #1355). Sequence inside the migration:
-   - (Safe ordering) Neutralize the column-list dependency surgically so the RENAME is unconditionally permitted: `ALTER PUBLICATION zero_publication DROP TABLE documents, podcasts;`. This removes ONLY their column-list dependency and is far safer than re-emitting the whole member set via `SET TABLE` (which is a full replacement and would drop any table accidentally omitted from the hand-written list, including the quoted `"user"` table). `DROP TABLE` from a publication is within the blessed ALTER-PUBLICATION family — it is NOT the forbidden raw `DROP/CREATE PUBLICATION`.
+   - (Safe ordering) Neutralize the column-list dependency surgically so the RENAME is unconditionally permitted: `ALTER PUBLICATION zero_publication DROP TABLE documents, automations, new_chat_threads, podcasts;` (all **four** column-list tables). This removes ONLY their column-list dependency and is far safer than re-emitting the whole member set via `SET TABLE` (which is a full replacement and would drop any table accidentally omitted from the hand-written list, including the quoted `"user"` table). `DROP TABLE` from a publication is within the blessed ALTER-PUBLICATION family — it is NOT the forbidden raw `DROP/CREATE PUBLICATION`.
    - Rename the columns (steps 1-2).
-   - Update canonical `DOCUMENT_COLS`/`PODCAST_COLS` to `workspace_id` in [zero_publication.py](surfsense_backend/app/zero_publication.py) (31, 66), then call `apply_publication(op.get_bind())` (blessed plain `ALTER ... SET TABLE`, as used by [155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py)) to re-add `documents`/`podcasts` (dropped above) with the narrowed `workspace_id` column lists and reconcile the full member set in one shot.
+   - Update canonical `DOCUMENT_COLS`/`AUTOMATION_COLS`/`NEW_CHAT_THREAD_COLS`/`PODCAST_COLS` to `workspace_id` in [zero_publication.py](surfsense_backend/app/zero_publication.py) (31, 57, 62, 76), then call `apply_publication(op.get_bind())` (blessed plain `ALTER ... SET TABLE`, as used by [155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py)) to re-add all four dropped tables with the narrowed `workspace_id` column lists and reconcile the full member set in one shot.
    - Reference template for forcing a schema-change event (only if event triggers are unavailable): the `COMMENT ON PUBLICATION` bookend trio in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) lines 104-137. With event triggers installed (current setup), `apply_publication` is sufficient.
    - Replica recreate (operational, out-of-band): reset the `surfsense-zero-cache` replica (delete the volume / rely on `ZERO_AUTO_RESET=true`) so zero-cache does a fresh initial sync from the corrected publication. This is the "recreate replication" step and the primary consumer-recovery mechanism.
-   - INTERLOCK FOOTGUN (review finding 2): if canonical `DOCUMENT_COLS`/`PODCAST_COLS` are NOT updated to `workspace_id` before `apply_publication` runs, `_format_table_entry` ([zero_publication.py](surfsense_backend/app/zero_publication.py) lines 136-137) silently DROPS `documents`/`podcasts` from the publication (no error). Gate with `--verify` (see Verification).
+   - INTERLOCK FOOTGUN (review finding 2): if any of the canonical `DOCUMENT_COLS`/`AUTOMATION_COLS`/`NEW_CHAT_THREAD_COLS`/`PODCAST_COLS` are NOT updated to `workspace_id` before `apply_publication` runs, `_format_table_entry` ([zero_publication.py](surfsense_backend/app/zero_publication.py) lines 148-149) silently DROPS the mismatched table(s) from the publication (no error). Gate with `--verify` (see Verification).
 5. No DDL needed for the `ck_connections_scope_owner` CHECK: the physical expression auto-follows the column rename (Postgres tracks the column by attnum). The model's `CheckConstraint(...)` text is updated separately as an ORM edit (see ORM edits) so create_all/autogenerate stay consistent.
 
 ### downgrade()
 
 Provide a full reverse (rename `workspaces` -> `searchspaces`, columns back, constraints/indexes back, then restore the OLD publication shape). Note the repo sometimes uses no-op downgrades for publication-only migrations ([155](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) lines 22-23), but a structural rename should be reversible.
 
-CRITICAL (review finding 7 — do NOT call `apply_publication()` in downgrade): `apply_publication()` reads the LIVE canonical in `zero_publication.py`, which after this change is `workspace_id`. In a downgrade the column is back to `search_space_id`, so `apply_publication` would find `workspace_id` missing and silently DROP `documents`/`podcasts` from the publication — the finding-2 footgun, self-inflicted. Instead, the downgrade must restore the old shape with HARDCODED `search_space_id` column lists via a plain `ALTER PUBLICATION ... SET TABLE`, mirroring the constants-style approach in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) (which embeds literal `DOCUMENT_COLS` rather than importing the live module). Use the same DROP-then-SET neutralize sequence in reverse.
+CRITICAL (review finding 7 — do NOT call `apply_publication()` in downgrade): `apply_publication()` reads the LIVE canonical in `zero_publication.py`, which after this change is `workspace_id`. In a downgrade the column is back to `search_space_id`, so `apply_publication` would find `workspace_id` missing and silently DROP the four column-list tables (`documents`/`automations`/`new_chat_threads`/`podcasts`) from the publication — the finding-2 footgun, self-inflicted. Instead, the downgrade must restore the old shape with HARDCODED `search_space_id` column lists (for all four tables) via a plain `ALTER PUBLICATION ... SET TABLE`, mirroring the constants-style approach in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) (which embeds literal `DOCUMENT_COLS` rather than importing the live module). Use the same DROP-then-SET neutralize sequence in reverse.
 
 COHERENCE (review finding 6): even with a correct downgrade SQL, `downgrade()` only produces a consistent RUNNING system if the PRIOR code revision is redeployed alongside it — under the shim the model maps the attribute to physical `workspace_id`, which no longer exists after a pure DB downgrade. Document rollback = revert code + schema together as one operation.
 
@@ -132,12 +135,12 @@ These are required so the ORM matches the renamed physical schema; they are NOT
    - Every `ForeignKey("searchspaces.id", ...)` -> `ForeignKey("workspaces.id", ...)`; `ForeignKey("search_space_roles.id"...)` -> `workspace_roles.id`; `ForeignKey("search_space_invites.id"...)` -> `workspace_invites.id`.
    - Apply the explicit-name shim on each `search_space_id` column: `Column("workspace_id", ...)`; and `owner_search_space_id = Column("owner_workspace_id", ...)`.
    - Update ONLY the `name=` kwarg of constraints/indexes (1831, 2032, 2069, and the Index at 978-979). CRITICAL (review finding 1): do NOT change the inner column-reference strings (`"search_space_id"` at 1827, 2030, 2068, 2674, 979) — under the shim the column's `.key` is still `search_space_id`, and declarative `__table_args__` resolves these strings against `Table.c` by `.key`. Changing them to `"workspace_id"` raises a config-time error and the app won't boot.
-   - Update the runtime raw index DDL strings + names in `_INDEX_DEFINITIONS` (2820-2830): both the index name and the physical column (`search_space_id` -> `workspace_id`). These must match the migration's renamed indexes and ship in the same release; `setup_indexes()` ([db.py](surfsense_backend/app/db.py) line 2855) re-creates them under the new name on next boot when `DB_BOOTSTRAP_ON_STARTUP` is true.
+   - Update the runtime raw index DDL strings + names in `_INDEX_DEFINITIONS` (~2821-2831; +1 line vs the pre-merge cite after `main`'s `RefreshToken` edit — locate by grep): both the index name and the physical column (`search_space_id` -> `workspace_id`). These must match the migration's renamed indexes and ship in the same release; `setup_indexes()` ([db.py](surfsense_backend/app/db.py) ~line 2856) re-creates them under the new name on next boot when `DB_BOOTSTRAP_ON_STARTUP` is true.
    - Update `ck_connections_scope_owner` text (1579-1582) `search_space_id` -> `workspace_id`.
    - Update `ix_external_chat_bindings_search_space_state` name + the surrounding Index (978-979).
 2. Module models — `__tablename__` unaffected, but apply the column shim + FK target string:
    - [file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) (32), [automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) (27), [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) (50, plus the index at 36-41), [podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) (71).
-3. [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py): `DOCUMENT_COLS` (31) and `PODCAST_COLS` (66) `search_space_id` -> `workspace_id`. (The `_expected_columns` special-casing on lines 106 keys off table names `documents`/`podcasts`, not the column, so no change there.)
+3. [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py): `DOCUMENT_COLS` (31), `AUTOMATION_COLS` (57), `NEW_CHAT_THREAD_COLS` (62), and `PODCAST_COLS` (76) `search_space_id` -> `workspace_id`. (The `_expected_columns` special-casing on line 118 keys off table names `{"documents","user","podcasts"}`, not the column, so no change there — and `automations`/`new_chat_threads` aren't in that `_0_version` allowlist anyway.)
 4. Runtime raw-SQL audit (defensive): grep `surfsense_backend/app` for any `text(...)` / hardcoded `"searchspaces"` or `search_space_id` strings that execute at runtime (not Python attribute access) and fix them here, since the shim only covers ORM attribute access. Expectation is few; the ~250-file footprint is overwhelmingly attribute access handled by the shim.
 
 ## Confirmed decisions
@@ -154,7 +157,7 @@ The Alembic migration, the `db.py` edits (shim + constraint `name=` + `_INDEX_DE
 
 ## Verification & rollout
 
-- HARD GATE: `python -m app.zero_publication --verify` (CLI at [zero_publication.py](surfsense_backend/app/zero_publication.py) lines 253-264) must report verified, in CI and immediately post-migrate. This is the guard against the interlock footgun (finding 2).
+- HARD GATE: `python -m app.zero_publication --verify` (CLI at [zero_publication.py](surfsense_backend/app/zero_publication.py) ~lines 266-269; `verify_publication` defined at ~203) must report verified, in CI and immediately post-migrate. This is the guard against the interlock footgun (finding 2).
 - Boot the API + a Celery worker; run a smoke chat + a document upload to exercise `documents.workspace_id` and the publication; confirm Zero clients still sync `documents`/`podcasts`.
 - AUTOGENERATE DRIFT CHECK (strong, cheap): after the change, `alembic revision --autogenerate` must produce an EMPTY diff. A non-empty diff means the ORM and migrated DB disagree — i.e. a missed `ForeignKey("searchspaces.id")` string, a constraint `name=` mismatch, or a forgotten `Column("workspace_id", ...)` shim. This single check catches most subtle misses.
 - Confirm `alembic upgrade head` then `alembic downgrade -1` then `upgrade head` round-trips on a staging copy. NOTE: this validates SQL reversibility only — it is NOT an app-health check, because new code cannot run against the downgraded schema (finding 6). The downgraded intermediate state must show `documents`/`podcasts` still present in the publication with `search_space_id` lists (proves the hardcoded downgrade publication shape from finding 7 works).

plans/backend/02-rename-backend.md

@@ -94,7 +94,7 @@ These do not move with a symbol rename; each is decided here.
 6. Notification dedup/operation IDs embedding `{search_space_id}` (e.g. `doc_..._{search_space_id}_...`, `insufficient_credits_{search_space_id}_...`) and frontend deep-link strings like `/dashboard/{search_space_id}/buy-more`. DECISION: the ID is a numeric value, not the literal word — leave format strings as-is functionally; the embedded VALUE is unchanged. The `/dashboard/{id}/...` deep link points at a FRONTEND route still named `[search_space_id]` (deferred umbrella) — KEEP it until the frontend segment renames, else links 404.
 7. Storage path builders — `documents/{search_space_id}/...` ([file_storage/keys.py](surfsense_backend/app/file_storage/keys.py) 20-26) and `podcasts/{search_space_id}/...` ([podcasts/storage.py](surfsense_backend/app/podcasts/storage.py) 22-25). The path segment is the numeric ID; the literal word `search_space` is NOT in stored object keys. DECISION: rename the param only; NO blob migration needed; existing objects keep resolving.
 8. `SearchSourceConnector` — contains "search" but is the connectors table, a different concept (the word "search" here is unrelated to `SearchSpace`). OUT OF SCOPE: this rename does NOT touch it, and Phase 4 does not rename the class either — Phase 4 adds a Type-1/Type-2 taxonomy via a static `connector_type`→(category, availability) registry (no new column) and KEEPS `is_indexable` (`db.py:1868`). Leave `SearchSourceConnector` as-is.
-9. Historical Alembic migrations (`surfsense_backend/alembic/versions/*`) — ~20 files embed `searchspaces` / `search_space_id` as raw-SQL string literals (e.g. `23_associate_connectors_with_search_spaces.py`, `41_backfill_rbac_for_existing_searchspaces.py`, `40_move_llm_preferences_to_searchspace.py`). DECISION: NEVER rewrite these. They are an immutable replay log that intentionally references the schema as it existed at that revision; rewriting them corrupts history and breaks a clean `alembic upgrade` from zero. Verified safe: no migration imports the ORM classes (only `from app.db import Base` in `env.py` / `0_initial_schema.py`), so the class rename does not touch them. Phase 1's rename migration (the next integer after the live head — `167` at time of writing, but chain by actual `alembic heads`) is the single transition point; migrations after it use the new names. The scripted rename scope is `app/` + `tests/` ONLY.
+9. Historical Alembic migrations (`surfsense_backend/alembic/versions/*`) — ~20 files embed `searchspaces` / `search_space_id` as raw-SQL string literals (e.g. `23_associate_connectors_with_search_spaces.py`, `41_backfill_rbac_for_existing_searchspaces.py`, `40_move_llm_preferences_to_searchspace.py`). DECISION: NEVER rewrite these. They are an immutable replay log that intentionally references the schema as it existed at that revision; rewriting them corrupts history and breaks a clean `alembic upgrade` from zero. Verified safe: no migration imports the ORM classes (only `from app.db import Base` in `env.py` / `0_initial_schema.py`), so the class rename does not touch them. Phase 1's rename migration (the next integer after the live head — `170` at time of writing, since `main` advanced the head to `169`, but chain by actual `alembic heads`) is the single transition point; migrations after it use the new names. The scripted rename scope is `app/` + `tests/` ONLY.
 10. LangGraph persisted state channel key `search_space_id` — `input_state = {..., "search_space_id": search_space_id, ...}` ([tasks/chat/streaming/flows/new_chat/input_state.py](surfsense_backend/app/tasks/chat/streaming/flows/new_chat/input_state.py) 131) is a CHECKPOINTED state channel. The resume_chat flow reads persisted state (`agent.aget_state({"configurable": {"thread_id": ...}})`, [flows/resume_chat/resume_routing.py](surfsense_backend/app/tasks/chat/streaming/flows/resume_chat/resume_routing.py) 50) and HITL interrupts (`surfsense_resume_value`, `HumanReview`) can sit pending across a deploy. DECISION: rename the channel key to `workspace_id` for consistency, and ACCEPT that any chat thread paused at a HITL interrupt BEFORE the cutover must be restarted after (the old checkpoint carries `search_space_id`, the new graph reads `workspace_id`). Operationally: drain/resolve pending interrupts before deploy if practical. (Alternative — keep the channel key as a carve-out — rejected: leaves a lone `search_space_id` in otherwise-renamed agent state for a transient, conversation-scoped value. The `configurable` keys are `thread_id` / `surfsense_resume_value`, not `search_space_id`, so only this state channel is affected.)
 11. User-facing default literal — `users.py` seeds the default workspace `name="My Search Space"` (line 158, persisted + shown to users) and logs "Created default search space" (207). DECISION: rename the default name to "My Workspace" (backend-created seed value, not caught by a symbol rename); log strings are cosmetic. Also update the Redis-key assertion in `tests/unit/services/test_ai_sort_task_dedupe.py:14` when literal 3 is renamed.
 

plans/backend/04a-connector-category.md

@@ -42,15 +42,15 @@ Create routes hardcode `is_indexable` per type: `True` for Notion (`notion_add_c
 
 ### The chat subagent maps + the routing gap
 
-- `constants.py` — `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (`:5–24`, connector_type → subagent name; **note GitHub/BookStack/Elasticsearch have NO entry** — they are index-only, no chat subagent) and `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`:26–44`, subagent → required tokens; `deliverables`/`knowledge_base` require `frozenset()` = always built). The required tokens are a **mix of connector types and doc types** (e.g. `notion`→`NOTION_CONNECTOR`, but `dropbox`→`DROPBOX_FILE`, `google_drive`→`GOOGLE_DRIVE_FILE`, `onedrive`→`ONEDRIVE_FILE`).
-- Subagent exclusion: `subagents/registry.py` `get_subagents_to_exclude(available_connectors)` (`:136–152`) excludes a subagent when its required tokens aren't in the available set; `build_subagents(..., exclude=...)` (`:182–220`) skips excluded names and calls each builder with `mcp_tools=mcp.get(name)`. `SUBAGENT_BUILDERS_BY_NAME` (`:92–112`) has **no `mcp` builder** today.
-- **CRITICAL — `available_connectors` is NOT raw connector types.** `factory.py:102–105` computes `connector_types = get_available_connectors(...)` then `available_connectors = map_connectors_to_searchable_types(connector_types)`. This **shared, already-mapped** list of searchable-type tokens feeds *multiple* consumers: `get_subagents_to_exclude` (`factory.py:253`, `main_agent/middleware/stack.py:204`), the KB pre-search middleware (`shared/middleware/knowledge_search.py:_resolve_search_types:314–322` does `types.update(available_connectors)`), `web_search`'s live-connector filter, and the deliverables/`search_knowledge_base` tools. **Filtering this list would break legacy KB-doc searchability and web search** — see Target §4.
+- `multi_agent_chat/constants.py` (the chat-package root `constants.py`, **not** under `subagents/`) — `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (`:5–24`, connector_type → subagent name; **note GitHub/BookStack/Elasticsearch have NO entry** — they are index-only, no chat subagent) and `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`:26–44`, subagent → required tokens; `deliverables`/`knowledge_base` require `frozenset()` = always built). The required tokens are a **mix of connector types and doc types** (e.g. `notion`→`NOTION_CONNECTOR`, but `dropbox`→`DROPBOX_FILE`, `google_drive`→`GOOGLE_DRIVE_FILE`, `onedrive`→`ONEDRIVE_FILE`).
+- Subagent exclusion: `subagents/registry.py` `get_subagents_to_exclude(available_connectors)` (`:136–152`) excludes a builder in **two** cases: (a) it is **absent from** `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP` (`required_tokens is None`, `:145–147`) → excluded; (b) it has non-empty required tokens that don't intersect the available set (`:150–151`) → excluded. An **empty** `frozenset()` (deliverables/knowledge_base) is always kept (`:148–149`). `build_subagents(..., exclude=...)` (`:182–220`) additionally hard-skips `memory`/`research` (`:195`) and the names in `exclude`, then calls each builder with `mcp_tools=mcp.get(name)` (`:209`). `SUBAGENT_BUILDERS_BY_NAME` (`:92–112`) has **no `mcp` builder** today. **Consequence for §5:** adding an `mcp` builder *without* a `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP["mcp"]` entry would hit case (a) and exclude it forever — so §5's map entry is mandatory, not optional.
+- **CRITICAL — `available_connectors` is NOT raw connector types.** `factory.py:102–105` computes `connector_types = get_available_connectors(...)` then `available_connectors = map_connectors_to_searchable_types(connector_types)`. This **shared, already-mapped** list of searchable-type tokens feeds *multiple* consumers: `get_subagents_to_exclude` (`factory.py:253`, `main_agent/middleware/stack.py:202`), the on-demand KB-search tool (`subagents/builtins/knowledge_base/tools/search_knowledge_base.py:_search_types:53–63` does `types.update(available_connectors)` at `:61–62` — this is the post-`main`-merge replacement for the now-deleted `shared/middleware/knowledge_search.py`, which used to do the same in `_resolve_search_types`), `web_search`'s live-connector filter (`shared/tools/web_search.py`), and the `deliverables` `report` tool. **Filtering this list would break legacy KB-doc searchability and web search** — see Target §4.
 - `map_connectors_to_searchable_types` (`connector_searchable_types.py:65–100`) maps each configured connector type to a searchable token; **`MCP_CONNECTOR` is absent from `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`)**, so MCP connectors never produce a token in `available_connectors`.
 - **The bug:** `subagents/mcp_tools/index.py` `partition_mcp_tools_by_connector` (`:55–99`) routes each MCP tool via `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS.get(connector_type)`; `MCP_CONNECTOR` has **no entry**, so it hits the `connector_agent is None` branch and is **skipped with a warning** (`:88–95`). Generic user MCP servers therefore contribute zero tools today. (The bucket key is the agent name, e.g. `"mcp"`, consumed by `build_subagents` via `mcp.get("mcp")`.)
 
 ### Query-time searchable mapping (keeps legacy docs visible)
 
-`main_agent/runtime/connector_searchable_types.py` `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`) maps connector types → searchable doc-types for KB pre-search. The MIGRATING indexers' entries (NOTION→`NOTION_CONNECTOR`, etc.) **stay** so already-indexed docs remain searchable. (WebCrawler→`CRAWLED_URL`, file connectors→`*_FILE` stay too.)
+`main_agent/runtime/connector_searchable_types.py` `_CONNECTOR_TYPE_TO_SEARCHABLE` (`:22–54`, unchanged by the `main` merge) maps connector types → searchable doc-types that scope the KB-search tool (via `available_connectors` → `search_knowledge_base.py:_search_types`). The MIGRATING indexers' entries (NOTION→`NOTION_CONNECTOR`, etc.) **stay** so already-indexed docs remain searchable. (WebCrawler→`CRAWLED_URL`, file connectors→`*_FILE` stay too.)
 
 ## Target design
 
@@ -90,18 +90,18 @@ In the generic create handler `create_search_source_connector` (`search_source_c
 
 Guard `index_connector_content` (`:717`) up front: if `not is_indexable_type(connector_type)`, return a 4xx/no-op ("indexing disabled — migrating to MCP"). This neutralizes the Notion/GitHub/Confluence/BookStack/Elasticsearch branches without deleting their (now-dead) code paths.
 
-**Periodic path (must also be gated).** Create-gating blocks *new* MIGRATING rows, but **existing** MIGRATING connectors with `periodic_indexing_enabled=True` already have a `next_scheduled_at` and would keep firing via the meta-scheduler task that polls `next_scheduled_at` every minute (`periodic_scheduler.py:41–42`; first-run trigger `create_periodic_schedule:30–58`). The robust single chokepoint is to **gate the Celery index tasks at entry** by `is_indexable_type(connector_type)` — this covers the manual `/index` route, the first-run trigger, and the recurring meta-scheduler pass at once (a no-op early-return for MIGRATING types). The schema validator "periodic only if indexable" (`schemas/search_source_connector.py:42–45`) is unchanged and unaffected (it keys off the per-row `is_indexable` bool, not the registry).
+**Periodic path (must also be gated).** Create-gating blocks *new* MIGRATING rows, but **existing** MIGRATING connectors with `periodic_indexing_enabled=True` already have a `next_scheduled_at` and would keep firing via the meta-scheduler Beat task that polls `next_scheduled_at` every minute: `tasks/celery_tasks/schedule_checker_task.py` (due query `next_scheduled_at <= now` `:36`, dispatch loop `:88`, `task = task_map.get(connector.connector_type)` then `.delay(...)` `:118`). The first-run trigger is `create_periodic_schedule` (`utils/periodic_scheduler.py:30`, `task.delay(...)` `:97`). **Both paths re-dispatch through the same per-type Celery tasks** (`index_crawled_urls_task`, `index_notion_pages_task`, …), so the robust single chokepoint is to **gate the Celery index tasks at entry** by `is_indexable_type(connector_type)` — this covers the manual `/index` route, the first-run trigger, and the recurring meta-scheduler pass at once (a no-op early-return for MIGRATING types). (Aside: `schedule_checker_task` already auto-disables a `LIVE_CONNECTOR_TYPES` set at `:74-77` — an existing precedent for turning periodic off per type.) The schema validator "periodic only if indexable" (`schemas/search_source_connector.py:42–45`) is unchanged and unaffected (it keys off the per-row `is_indexable` bool, not the registry).
 
 Note: **update** (`PUT /search-source-connectors/{id}`, `search_source_connectors_routes.py:371`) is intentionally **left allowed** for MIGRATING rows so existing users can still edit/disable them; only create + index are gated.
 
 ### 4. Subagent gating (turns off branded chat tools) — by NAME, not by token
 
-> **Do NOT filter `available_connectors`.** It is a shared, already-mapped searchable-type list (see Current State). Stripping MIGRATING entries from it would (a) remove `NOTION_CONNECTOR`/etc. from KB pre-search → **break the very legacy-doc searchability this plan promises**, and (b) remove the live-search tokens → break `web_search` before 04b. The token list must stay intact.
+> **Do NOT filter `available_connectors`.** It is a shared, already-mapped searchable-type list (see Current State). Stripping MIGRATING entries from it would (a) remove `NOTION_CONNECTOR`/etc. from the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`) → **break the very legacy-doc searchability this plan promises**, and (b) remove the live-search tokens → break `web_search` before 04b. The token list must stay intact.
 
 Instead, **exclude the deprecated subagents by NAME**, derived from the registry. Extend `get_subagents_to_exclude` (`registry.py:136`) so that, in addition to its current token check, it also excludes any subagent whose mapped connector type(s) are all non-`AVAILABLE`:
 
 - Build the reverse of `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (subagent_name → set of connector types). For each `SUBAGENT_BUILDERS_BY_NAME` entry, if it has connector type(s) and **every** one is non-`AVAILABLE` in the registry, add it to the excluded set unconditionally (regardless of tokens).
-- This is the single chokepoint used by both `factory.py:253` and `stack.py:204`, and feeds `main_prompt_registry_subagent_lines(exclude)`, so the deprecated specialists disappear from both the build and the prompt.
+- This is the single chokepoint used by both `factory.py:253` and `stack.py:202`, and feeds `main_prompt_registry_subagent_lines(exclude)`, so the deprecated specialists disappear from both the build and the prompt.
 
 Net effect for MVP:
 
@@ -112,9 +112,9 @@ Net effect for MVP:
 
 Two changes are needed — the routing map AND surfacing the token, or the subagent will still never build:
 
-1. **Routing map** (`constants.py`): add `"MCP_CONNECTOR": "mcp"` to `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (so `partition_mcp_tools_by_connector` stops hitting the `connector_agent is None` skip at `mcp_tools/index.py:88–95`), and `"mcp": frozenset({"MCP_CONNECTOR"})` to `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP`.
+1. **Routing map** (`multi_agent_chat/constants.py`): add `"MCP_CONNECTOR": "mcp"` to `CONNECTOR_TYPE_TO_CONNECTOR_AGENT_MAPS` (so `partition_mcp_tools_by_connector` stops hitting the `connector_agent is None` skip at `mcp_tools/index.py:88–95`), and `"mcp": frozenset({"MCP_CONNECTOR"})` to `SUBAGENT_TO_REQUIRED_CONNECTOR_MAP`.
 2. **Builder**: add a generic `mcp` entry to `SUBAGENT_BUILDERS_BY_NAME` (`registry.py:92`); `build_subagents` will call it with `mcp_tools=mcp.get("mcp")` (the bucket key from step 1's map value).
-3. **Surface the token** (the gap step 4's predecessor missed): `MCP_CONNECTOR` is **not** in `_CONNECTOR_TYPE_TO_SEARCHABLE`, so `available_connectors` never contains it and the `mcp` subagent's required token `{MCP_CONNECTOR}` would never intersect → it would be excluded forever. Fix by adding `"MCP_CONNECTOR": "MCP_CONNECTOR"` to `_CONNECTOR_TYPE_TO_SEARCHABLE` (`connector_searchable_types.py:22`). Side effect: `MCP_CONNECTOR` joins the KB pre-search doc-type set (`_resolve_search_types`), where it matches zero indexed docs — harmless no-op. (Alternative if the no-op is undesirable: special-case `mcp` in `get_subagents_to_exclude` to include it whenever any configured connector is an `AVAILABLE` `MCP_TOOL`, passing the raw `connector_types` alongside. The searchable-map entry is simpler and preferred.)
+3. **Surface the token** (the gap step 4's predecessor missed): `MCP_CONNECTOR` is **not** in `_CONNECTOR_TYPE_TO_SEARCHABLE`, so `available_connectors` never contains it and the `mcp` subagent's required token `{MCP_CONNECTOR}` would never intersect → it would be excluded forever. Fix by adding `"MCP_CONNECTOR": "MCP_CONNECTOR"` to `_CONNECTOR_TYPE_TO_SEARCHABLE` (`connector_searchable_types.py:22`). Side effect: `MCP_CONNECTOR` joins the KB-search tool's doc-type scope (`search_knowledge_base.py:_search_types`), where it matches zero indexed docs — harmless no-op. (Alternative if the no-op is undesirable: special-case `mcp` in `get_subagents_to_exclude` to include it whenever any configured connector is an `AVAILABLE` `MCP_TOOL`, passing the raw `connector_types` alongside. The searchable-map entry is simpler and preferred.)
 
 ### 6. Keep `is_indexable` semantics intact
 
@@ -127,7 +127,7 @@ No change to the column or its validator. `is_indexable` continues to gate the r
 3. **Create gating** in the generic create handler (`:172`) + every per-service add route (reject non-`AVAILABLE`).
 4. **Index gating** at the Celery index-task entry (`is_indexable_type`) — covers manual `/index`, first-run trigger, and meta-scheduler.
 5. **Subagent gating by name**: extend `get_subagents_to_exclude` to also exclude subagents whose mapped connector types are all non-`AVAILABLE` (do **not** filter the shared `available_connectors` token list).
-6. **MCP routing-gap fix**: `constants.py` map entries + generic `mcp` builder in `SUBAGENT_BUILDERS_BY_NAME` + `MCP_CONNECTOR` token in `_CONNECTOR_TYPE_TO_SEARCHABLE` (so the subagent actually builds).
+6. **MCP routing-gap fix**: `multi_agent_chat/constants.py` map entries + generic `mcp` builder in `SUBAGENT_BUILDERS_BY_NAME` + `MCP_CONNECTOR` token in `_CONNECTOR_TYPE_TO_SEARCHABLE` (so the subagent actually builds).
 7. **Tests** (below).
 
 ## Tests
@@ -136,9 +136,9 @@ No change to the column or its validator. `is_indexable` continues to gate the r
 - **Create gating**: creating any `MIGRATING`/`DISABLED` type → 4xx; `WEBCRAWLER`/file/`MCP_CONNECTOR` → OK; MCP multi-instance still works.
 - **Index gating**: index task entry for Notion/GitHub/Confluence/BookStack/Elasticsearch → no-op; WebCrawler/GDrive/OneDrive/Dropbox → runs. Include a case for an **existing** MIGRATING row with `periodic_indexing_enabled=True` + due `next_scheduled_at` → meta-scheduler triggers a no-op (does not re-index).
 - **Subagent gating by name**: with a (legacy) Notion + Slack connector configured, `get_subagents_to_exclude` excludes `notion`/`slack`; `knowledge_base`/`deliverables` always built; a configured `google_drive`/`dropbox`/`onedrive` (AVAILABLE) is **not** excluded (still token-gated).
-- **No collateral damage to the token list**: gating subagents must NOT change `available_connectors` — assert the KB pre-search still resolves `NOTION_CONNECTOR` doc type for the legacy connector (guards against the "filter the shared list" regression).
+- **No collateral damage to the token list**: gating subagents must NOT change `available_connectors` — assert the KB-search tool still resolves the `NOTION_CONNECTOR` doc type (via `_search_types`) for the legacy connector (guards against the "filter the shared list" regression).
 - **MCP gap (two-part)**: (a) a configured `MCP_CONNECTOR` produces a `"mcp"` token in `map_connectors_to_searchable_types`; (b) its tools land in the `mcp` bucket and the `mcp` subagent is actually built (regression for both `mcp_tools/index.py:88–95` and the missing-token exclusion).
-- **Legacy searchability**: a search space with an existing (now-MIGRATING) Notion connector + indexed docs still returns those docs via KB pre-search (`connector_searchable_types` Notion entry unchanged).
+- **Legacy searchability**: a search space with an existing (now-MIGRATING) Notion connector + indexed docs still returns those docs via the KB-search tool (`connector_searchable_types` Notion entry unchanged).
 
 ## Risks / trade-offs
 
@@ -146,7 +146,7 @@ No change to the column or its validator. `is_indexable` continues to gate the r
 - **Dead code paths left in place.** The Notion/GitHub/Confluence/BookStack/Elasticsearch `/index` branches + indexers remain but are gated off (not deleted) to keep the diff small and reversible; delete when the real MCP migration lands.
 - **No column = not SQL/Zero-filterable.** Frontend (deferred) filters via the API's computed fields. If Zero needs server-side filtering later, promoting the registry to a denormalized column is additive.
 - **Registry/`is_indexable` dual-gate.** Two truths (type-level allowlist + per-row bool) must agree; the totality test + index-gating test cover the seam.
-- **Shared `available_connectors` is a footgun.** It feeds subagent-exclusion, KB pre-search, web_search, and deliverables simultaneously. Subagent gating is therefore done by *name* (registry-driven), never by mutating that list; the "no collateral damage" test pins this. This also removes any 04a→04b sequencing hazard (04a no longer touches the live-search tokens that 04b later retires).
+- **Shared `available_connectors` is a footgun.** It feeds subagent-exclusion, the KB-search tool's doc-type scope, web_search, and the deliverables `report` tool simultaneously. Subagent gating is therefore done by *name* (registry-driven), never by mutating that list; the "no collateral damage" test pins this. This also removes any 04a→04b sequencing hazard (04a no longer touches the live-search tokens that 04b later retires).
 - **File-source specialists kept.** `google_drive`/`dropbox`/`onedrive` stay as AVAILABLE chat specialists (decision in §4). If product wants pure-ingestion file connectors, that's an additive registry flag later.
 
 ## Out of scope (hand-offs)

plans/backend/04b-source-discovery.md

@@ -25,10 +25,12 @@ Resolved decisions driving this:
 
 ### The web_search tool already has a platform/per-workspace split
 
-`web_search` tool factory `create_web_search_tool(search_space_id, available_connectors)`:
-- **Research subagent**: `subagents/builtins/research/tools/web_search.py` — `_LIVE_SEARCH_CONNECTORS = {TAVILY_API, LINKUP_API, BAIDU_SEARCH_API}` (`:15–19`), `_LIVE_CONNECTOR_SPECS` → `ConnectorService.search_*` (`:21–25`). At call time it fans out **in parallel** to platform SearXNG (`web_search_service.is_available()` / `.search()`, `:166–185`) **plus** each active per-workspace live connector (`:187–199`), then dedupes by URL (`:213–221`).
-- **Shared (single-agent) variant**: `agents/chat/shared/tools/web_search.py` (mirror — apply the same rewire).
-- Platform SearXNG service: `app/services/web_search_service.py` — env-gated by `config.SEARXNG_DEFAULT_HOST` (`is_available():125–127`), in-process circuit breaker + Redis result cache; returns the `(result_obj, documents)` shape the tool consumes.
+> **Post-`main`-merge note.** A `main` sync **consolidated web_search to a single tool**: the old research-subagent copy `subagents/builtins/research/tools/web_search.py` was **deleted**, and there is now **one** factory at `app/agents/chat/shared/tools/web_search.py`. The research subagent now imports it (`subagents/builtins/research/tools/index.py:10,24–26`). So everywhere below that said "both variants / mirror" is now **one file**. The same merge also retired the XML result blob in favor of `[n]`-citation rendering (see §3).
+
+`web_search` tool factory `create_web_search_tool(search_space_id, available_connectors)` — the single canonical tool at `app/agents/chat/shared/tools/web_search.py`:
+- `_LIVE_SEARCH_CONNECTORS = {TAVILY_API, LINKUP_API, BAIDU_SEARCH_API}` (`:41–45`), `_LIVE_CONNECTOR_SPECS` → `ConnectorService.search_*` (`:47–51`), per-connector dispatch `_search_live_connector` (`:115–151`). The factory picks the active live connectors by intersecting `available_connectors` with `_LIVE_SEARCH_CONNECTORS` (`:163–167`).
+- At call time it fans out **in parallel** to platform SearXNG (`web_search_service.is_available()` / `.search()`, `:204–214`) **plus** each active per-workspace live connector (`:216–228`), then dedupes by URL (`:242–250`).
+- Platform SearXNG service: `app/services/web_search_service.py` — env-gated by `config.SEARXNG_DEFAULT_HOST`, in-process circuit breaker + Redis result cache; returns the `(result_obj, documents)` shape the tool consumes.
 
 ### Config
 
@@ -41,7 +43,7 @@ Resolved decisions driving this:
 Add platform env knobs alongside `SEARXNG_DEFAULT_HOST` (`config/__init__.py:~558`):
 
 - `LINKUP_API_KEY = os.getenv("LINKUP_API_KEY")`
-- `BAIDU_SEARCH_API_KEY = os.getenv("BAIDU_SEARCH_API_KEY")` (+ any host/region knobs `search_baidu` needs — port from its current `config` reads).
+- `BAIDU_SEARCH_API_KEY = os.getenv("BAIDU_SEARCH_API_KEY")` (+ the optional knobs `search_baidu` reads from connector `config` today — `BAIDU_MODEL` / `BAIDU_SEARCH_SOURCE` / `BAIDU_ENABLE_DEEP_SEARCH`, per `validators.py:507-514`; the required key is `BAIDU_API_KEY` in the per-workspace config, relocated to this platform env var — port from its current `config` reads).
 - (SearXNG unchanged.) Document all three in `.env.example` as optional; each provider self-disables when its key/host is unset (mirrors `web_search_service.is_available()`).
 
 No Tavily/Serper env (removed).
@@ -56,18 +58,18 @@ Introduce `app/services/web_search_service.py`-level functions (or a small `sear
 
 This removes per-workspace coupling: web search + discovery no longer depend on any connector row.
 
-### 3. Rewire the chat `web_search` tool (both variants)
+### 3. Rewire the chat `web_search` tool (single tool)
 
-- Drop the `_LIVE_SEARCH_CONNECTORS` / `_LIVE_CONNECTOR_SPECS` / `_search_live_connector` mechanism in **both** variants — they are identical in shape: `subagents/builtins/research/tools/web_search.py:15–25,144–151,187–199` and `shared/tools/web_search.py:15–25,~151–153,…`. Instead, fan out to the **platform providers** that are `is_available()` (SearXNG + Linkup + Baidu), all keyless from the caller's view.
-- Keep the existing parallel-gather + URL-dedupe (`:204–221`) and XML formatting (`:46–93`) unchanged.
-- **Call-site churn — keep the `available_connectors` parameter in the factory signature but stop using it for provider selection** (providers are now env-derived). This avoids touching every caller: `main_agent/tools/registry.py:42,48–50`, `subagents/builtins/research/tools/index.py:24–26`, `subagents/builtins/deliverables/tools/index.py:44` (+ `report.py:805`), and `anonymous_chat/agent.py:127` (passes `available_connectors=None` already). `search_space_id` likewise stays for logging only. (A later cleanup can remove the now-dead params.)
+- Drop the `_LIVE_SEARCH_CONNECTORS` / `_LIVE_CONNECTOR_SPECS` / `_search_live_connector` mechanism in the **single** tool `shared/tools/web_search.py` (`:41–51`, `:115–151`, and the factory's live-connector intersection `:163–167`, plus the per-workspace fan-out `:216–228`). Instead, fan out to the **platform providers** that are `is_available()` (SearXNG + Linkup + Baidu), all keyless from the caller's view.
+- Keep the existing parallel-gather + URL-dedupe (`:233–250`). **Note:** the `main` merge replaced the old XML result blob with `[n]`-citation rendering — the tool now builds `RenderableDocument`s (`_to_renderable_web_documents:66–112`) and returns them via `render_web_results` + `load_registry`, persisting the `citation_registry` on state (`:252–281`). Leave that render/registry path **unchanged**; only the *provider sourcing* (SearXNG+Linkup+Baidu, env-keyed) changes.
+- **Call-site churn — keep the `available_connectors` parameter in the factory signature but stop using it for provider selection** (providers are now env-derived). This avoids touching every caller: `main_agent/tools/registry.py:24,39`, `subagents/builtins/research/tools/index.py:10,24–26`, and `anonymous_chat/agent.py:127` (passes `available_connectors=None` already). **`deliverables` is no longer a web_search caller** (its KB/web tooling was removed in the `main` merge) — drop it from the churn list. `search_space_id` likewise stays for logging only. (A later cleanup can remove the now-dead params.)
 - Net behaviour: web search works platform-wide whenever any provider env key is set; it no longer requires the workspace to have a search connector configured (so anonymous chat gains web search too).
 
 ### 4. Source-discovery endpoint (the new capability)
 
 New route (e.g. `routes/source_discovery_routes.py`, mounted under `/api/v1/workspaces/{workspace_id}/...` to match the renamed surface): `POST .../source-discovery` taking `{ query/topic, top_k }` and returning a **ranked list of candidate URLs** (url, title, snippet, provider) for the user to add to the WebURL Crawler / a pipeline.
 
-- Implementation = call the platform providers (§2) in parallel, dedupe by URL (reuse the tool's dedupe), and return URL-centric results (not the chat XML blob).
+- Implementation = call the platform providers (§2) in parallel, dedupe by URL (reuse the tool's dedupe), and return URL-centric results (plain `{url,title,snippet,provider}` — **not** the chat tool's `[n]`-cited `render_web_results` output, which is for in-conversation citation, not URL suggestion).
 - Auth: standard workspace access check (same dependency as other workspace routes).
 - This is **backend-only**; the UX (a "find sources" affordance when configuring a crawler/pipeline) is deferred to the frontend umbrella.
 - Optional thin reuse: the chat `web_search` `_web_search_impl` and this endpoint can share a `discover_urls(query, top_k) -> list[UrlCandidate]` core.
@@ -75,7 +77,7 @@ New route (e.g. `routes/source_discovery_routes.py`, mounted under `/api/v1/work
 ### 5. Drop the 5 connector types
 
 - **Remove the enum members** `SERPER_API`/`TAVILY_API`/`SEARXNG_API`/`LINKUP_API`/`BAIDU_SEARCH_API` from `SearchSourceConnectorType` (`db.py:86–90`).
-- **Remove their connector code paths**: `ConnectorService.search_tavily` (`:481`, delete) and the per-workspace `search_searxng` (`:587`)/`search_baidu` (`:614`)/`search_linkup` (`:1968`) (replace with the platform service §2; delete the connector-config variants that read `connector.config[...]_API_KEY`, e.g. `:510`, `:2000`). Remove the three search entries from `connector_searchable_types.py:24–26`. Remove the `*_API` entries from the connector-config validation dict (`utils/validators.py:~491–506`, incl. `TAVILY_API`/`LINKUP_API`/`SEARXNG_API`/`BAIDU_SEARCH_API`/`SERPER_API`) and `04a`'s `HIDDEN` registry entries for these types. **Grep-guard:** no remaining references to the 5 enum names or `search_tavily` anywhere.
+- **Remove their connector code paths**: `ConnectorService.search_tavily` (`:481`, delete) and the per-workspace `search_searxng` (`:587`)/`search_baidu` (`:614`)/`search_linkup` (`:1968`) (replace with the platform service §2; delete the connector-config variants that read `connector.config[...]_API_KEY`, e.g. `:510`, `:2000`). Remove the three search entries from `connector_searchable_types.py:24–26`. Remove the `*_API` entries from the connector-config validation dict (`utils/validators.py:490–514`, the 5 keys in `connector_rules`: `SERPER_API:490`/`TAVILY_API:491`/`SEARXNG_API:492-505`/`LINKUP_API:506`/`BAIDU_SEARCH_API:507-514`) and `04a`'s `HIDDEN` registry entries for these types. **Grep-guard:** no remaining references to the 5 enum names or `search_tavily` anywhere.
 - **Migration (this phase is NOT migration-free):** existing connector rows of these 5 types must be handled before/with the enum change:
   - Alembic migration: `DELETE FROM search_source_connectors WHERE connector_type IN ('SERPER_API','TAVILY_API','SEARXNG_API','LINKUP_API','BAIDU_SEARCH_API');` (FKs cascade; these rows only held search API keys, now relocated to env). Self-hosted operators re-add Linkup/Baidu keys via env — call out in the migration docstring + `.env.example`.
   - **PG enum handling:** dropping a value from a Postgres enum type requires a type recreate. Lowest-risk option: after deleting the rows, leave the now-unused labels in the PG enum type (harmless orphans) and just remove them from the Python `StrEnum` so no new rows can use them. If a clean type is wanted, do the standard "create new enum → `ALTER COLUMN ... TYPE` with a `USING` cast → drop old" dance in the migration (heavier; only if desired). Recommend the orphan-label approach for MVP.
@@ -85,7 +87,7 @@ New route (e.g. `routes/source_discovery_routes.py`, mounted under `/api/v1/work
 
 1. **Config**: `LINKUP_API_KEY`, `BAIDU_SEARCH_API_KEY` (+ Baidu host/region) in `Config` + `.env.example`; keep `SEARXNG_DEFAULT_HOST`.
 2. **Platform providers**: port Linkup/Baidu search to env-keyed platform functions sharing SearXNG's `(result_obj, documents)` shape; delete Tavily.
-3. **Rewire `web_search`** (research + shared variants) to fan out to platform providers; drop `available_connectors`/live-connector plumbing; keep dedupe + formatting.
+3. **Rewire `web_search`** (the single shared tool `shared/tools/web_search.py`) to fan out to platform providers; drop `available_connectors`/live-connector plumbing; keep dedupe + the `[n]`-citation render path.
 4. **Source-discovery endpoint**: `POST .../source-discovery` → ranked URL candidates; shared `discover_urls()` core.
 5. **Drop the 5 enum values** + remove `search_*` connector methods, `connector_searchable_types` search entries, validators, and `04a` HIDDEN entries.
 6. **Migration**: delete the 5 connector types' rows; handle the PG enum (orphan-label approach); docstring notes the env re-keying.

plans/backend/05-pipelines-model.md

@@ -54,7 +54,7 @@ The cleanest existing analogue to model against.
 
 ### Zero publication mechanics
 
-- `zero_publication.py`: `ZERO_PUBLICATION` map (`:71–82`) is the single source of truth; `None` ⇒ publish full row (e.g. `folders`, `search_source_connectors`), a list ⇒ column subset (e.g. `AUTOMATION_RUN_COLS` `:44–53`). `apply_publication(conn)` (`:151`) reconciles via `ALTER PUBLICATION ... SET TABLE`; a migration just calls it (template: `159_publish_podcasts_to_zero.py:21–22`). `_format_table_entry` omits a table until it physically exists with all its canonical columns (`:113–140`), so the migration **must create the tables before** calling `apply_publication`. The `_0_version` allowlist (`{"documents","user","podcasts"}`, `:106`) applies **only to column-list tables** — irrelevant here since we publish full-row.
+- `zero_publication.py`: `ZERO_PUBLICATION` map (`:81–94`; line numbers post-`main`-merge, which added `automations`/`new_chat_threads` entries) is the single source of truth; `None` ⇒ publish full row (e.g. `folders`, `search_source_connectors`), a list ⇒ column subset (e.g. `AUTOMATION_RUN_COLS` `:44–53`). `apply_publication(conn)` (`:163`) reconciles via `ALTER PUBLICATION ... SET TABLE`; a migration just calls it (template: `159_publish_podcasts_to_zero.py:21–22`). `_format_table_entry` omits a table until it physically exists with all its canonical columns (`:125–152`), so the migration **must create the tables before** calling `apply_publication`. The `_0_version` allowlist (`{"documents","user","podcasts"}`, `:118`) applies **only to column-list tables** — irrelevant here since we publish full-row.
 
 ### The meta-scheduler we coexist with (Phase-6 reuse target)
 
@@ -63,7 +63,7 @@ The cleanest existing analogue to model against.
 
 ### Alembic head
 
-Today's head is **`166`** (`alembic/versions/166_add_pat_and_api_access.py`; sequential integer-prefixed files). By the time Phase 5 is implemented, Phase 1 (rename) and Phase 4b (search-enum drop) have each added a migration ahead of `166`. **Set `down_revision` to the then-current head** — verify with `alembic heads`; do not hardcode `166`.
+As of the latest `main` sync the head is **`169`** (`alembic/versions/169_migrate_google_oauth_account_ids_to_sub.py`; chain `166→167→168→169`, sequential integer-prefixed files). By the time Phase 5 is implemented, Phase 1 (rename) and Phase 4b (search-enum drop) have each added a migration ahead of `169`. **Set `down_revision` to the then-current head** — verify with `alembic heads`; do not hardcode a number.
 
 ## Target design
 
@@ -241,14 +241,14 @@ class PipelineRunList(BaseModel):
 
 ### 6. Zero publication (`zero_publication.py`)
 
-Add to `ZERO_PUBLICATION` (`:71–82`), both **full-row**:
+Add to `ZERO_PUBLICATION` (`:81–94`), both **full-row**:
 
 ```python
     "pipelines": None,
     "pipeline_runs": None,
 ```
 
-Full-row (like `folders`/`search_source_connectors`) — no `_0_version` allowlist edit needed, no column-drift migrations later. (If a bulky column is ever added and must be excluded, switch that table to an explicit COLS list and handle the `_0_version` seam at `:106` then.) `verify_publication` will then expect both tables; the migration's `apply_publication` call (§4) reconciles them.
+Full-row (like `folders`/`search_source_connectors`) — no `_0_version` allowlist edit needed, no column-drift migrations later. (If a bulky column is ever added and must be excluded, switch that table to an explicit COLS list and handle the `_0_version` seam at `:118` then.) `verify_publication` will then expect both tables; the migration's `apply_publication` call (§4) reconciles them.
 
 > **Client sync is not live until the frontend lands (safe).** Publishing backend-side only makes the rows *available* to Zero; nothing syncs to a client until the (deferred) frontend Zero schema + permissions include these tables. Publishing now is harmless and matches the existing pattern (e.g. `automation_runs` was published by migration before/independent of its UI). This entry exists so the chat-agent run-history context (Phase 6) and a later Pipelines UI get push for free.