From 9d9d8014f9299c411ec251672594ba73224ae4d7 Mon Sep 17 00:00:00 2001 From: "DESKTOP-RTLN3BA\\$punk" Date: Wed, 24 Jun 2026 08:15:54 -0700 Subject: [PATCH] init: plans --- plans/backend/00-umbrella-plan.md | 155 +++++++++++++++++++++++++ plans/backend/01-rename-db.md | 179 +++++++++++++++++++++++++++++ plans/backend/02-rename-backend.md | 133 +++++++++++++++++++++ 3 files changed, 467 insertions(+) create mode 100644 plans/backend/00-umbrella-plan.md create mode 100644 plans/backend/01-rename-db.md create mode 100644 plans/backend/02-rename-backend.md diff --git a/plans/backend/00-umbrella-plan.md b/plans/backend/00-umbrella-plan.md new file mode 100644 index 000000000..774be4a92 --- /dev/null +++ b/plans/backend/00-umbrella-plan.md @@ -0,0 +1,155 @@ +# CI Pivot MVP — Umbrella Plan + +> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`content_research/pivot/`). + +This is the high-level roadmap. It is sequenced to match the agreed order: rename first, then connector restructure, then Pipelines. + +> SCOPE: This umbrella currently covers the BACKEND only (`surfsense_backend`). Frontend (`surfsense_web`) and client apps (desktop, Obsidian, browser extension) will get their own umbrella/subplans LATER, once the backend is fully working as expected. Frontend-facing decisions (URL segment, TS types, i18n copy) are recorded below where relevant but are out of scope for the active phases. + +## Positioning + +"NotebookLM for Competitive Intelligence" — each WorkSpace acts as a workspace for setting up competitive-intelligence-optimised notebooks. + +## Target architecture + +```mermaid +flowchart TD + WS[WorkSpace] --> CONN[Connectors] + WS --> PIPE[Pipelines] + WS --> KB[(Knowledge Base: documents + chunks)] + CONN --> T1[Type 1: Data Sources - pull] + CONN --> T2[Type 2: MCP Tools - act] + T1 --> WEB[Universal WebURL Crawler - functional] + T1 --> PLAT[Platform connectors - coming soon] + T1 --> UP[File Upload] + PIPE --> RUN[PipelineRun history] + PIPE -->|"save_to_kb + destination folder"| KB + RUN -->|"manual or cron"| T1 + T2 --> CHAT[Chat / Automations] + CHAT --> DELIV[Deliverables: audio/video/report/image] + RUN -->|"read-only context"| CHAT +``` + +## Decisions locked + +- Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps. +- Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`. +- Connectors get a `category` discriminator: `DATA_SOURCE` (Type 1) vs `MCP_TOOL` (Type 2). Type 1 keeps only file/cloud data sources (WebURL crawler, Google Drive, OneDrive, Dropbox, YouTube, file uploads) plus deferred platform connectors. Everything else moves to MCP. Artifacts stay in the existing `deliverables` agent system (not routed through MCP). +- Web search APIs (Tavily, SearXNG, Linkup, Baidu, Serper) are repurposed as a SOURCE-DISCOVERY helper: they suggest URLs the user can add to the Universal WebURL Crawler when setting up pipelines (they are not a standalone connector type and do not index data). +- Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP. +- MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are disabled — they migrate to Type-2. +- `Pipeline` and `PipelineRun` are new first-class tables. A Pipeline references a connector + config + schedule + KB destination. File upload creates/uses a pipeline and registers a run; uploads always save to KB. +- The chat agent gets read-only access to pipeline run history (pipelines + their recent runs/status) as context, so it can reason about what was fetched, when, and whether runs succeeded — even for data not saved to the KB. +- Deferred (post-MVP): platform scraper implementations, public pay-as-you-go API for Type-1 connectors, public MCP server exposing the KB. + +## Platform connector research list (deferred build, MVP = "coming soon") + +- LinkedIn — people profiles (discovery by keyword/company), company info, job listings. +- Amazon — product (ASIN), search (keyword), pricing; reviews secondary. +- Google — Web Search (organic SERP), AI Overviews, Maps/Local (discover by location). +- Instagram — profiles first, then posts; discover profiles by username/keyword. +- Zillow / Redfin — full property listings (discover by search URL/filters); Zillow price history. +- Walmart — product, search; zipcode-localized pricing premium variant. +- eBay — search by keyword/category; price-comparison/resale feeds. +- Crunchbase — company info, search by keyword (B2B lead-gen / investor research). +- TikTok / YouTube — profiles/channels, posts/videos; discover by keyword/hashtag; TikTok Shop. +- Indeed / Glassdoor — job listings (discover by keyword in location), company reviews. + +## Backend phases (active — this umbrella) + +### Phase 1 — Rename foundation (DB) [`subplan: 01-rename-db.md`] + +- Alembic migration: rename `searchspaces` -> `workspaces`; rename `search_space_id` -> `workspace_id` on ~20 child tables; rename RBAC tables and their FKs; rename indexes/constraints (`uq_searchspace_*`, `idx_documents_search_space_id`, etc.); update Rocicorp Zero publication column lists (backend-owned `publication` definition; frontend Zero schema rename happens in the later frontend umbrella). +- Decide transition strategy: hard cutover (simplest for MVP) vs temporary API aliases for clients. +- Key files: `surfsense_backend/app/db.py`, `surfsense_backend/alembic/versions/` (new migration). + +### Phase 2 — Rename backend (code + API) [`subplan: 02-rename-backend.md`] + +- Rename models/schemas/services/routes/agents/tasks identifiers: `SearchSpace*` -> `Workspace*`, `search_space_id` -> `workspace_id`. +- Consolidate API to `/workspaces` and fix the `/searchspaces` vs `/search-spaces` inconsistency. +- High-touch files: `routes/search_spaces_routes.py`, `routes/rbac_routes.py`, `utils/rbac.py` (`check_search_space_access`), `schemas/search_space.py`, plus `search_space_id` threading through agents/Redis keys/storage paths (`documents/{id}/...`). + +### Phase 3 — Connector two-type restructure (backend) [`subplan: 03-connector-two-type-backend.md`] + +- Add `category` (`DATA_SOURCE` / `MCP_TOOL`) to `SearchSourceConnector` (replaces ad hoc `is_indexable`): `db.py` enum/model, schema, Alembic migration + data backfill that tags existing rows. +- Adjust backend routing/indexing so only Type-1 keeps the `/index` + Celery path; Type-2 resolves via MCP tools. +- Add a backend source-discovery endpoint for the WebURL Crawler (reuses existing web-search services); UI surfacing is deferred to the frontend umbrella. +- Frontend connector UI restructure is DEFERRED. + +**Type 1 — Data Sources (pull -> feed pipelines/KB).** Keep only: + +- Universal WebURL Crawler (functional for MVP; from current `WEBCRAWLER_CONNECTOR`). Gets a source-discovery assist powered by the web search APIs (see below) to suggest URLs for pipelines. +- Google Drive (native + Composio) — `google_drive_indexer.py`. +- OneDrive — `onedrive_indexer.py`. +- Dropbox — `dropbox_indexer.py`. +- YouTube — promote from frontend-only/document handling to a real Type-1 connector (extra work: no backend connector today). +- File uploads. +- Platform connectors (coming soon, not built): LinkedIn, Amazon, Google, Instagram, Zillow/Redfin, Walmart, eBay, Crunchbase, TikTok, Indeed/Glassdoor. + +**Type 2 — MCP Tools (act in chat/automations).** Migrate existing connectors to MCP (all audited services have an MCP available): + +- Notion, GitHub, Confluence, Slack, Teams, Linear, Jira, ClickUp, Airtable, Discord, Gmail, Google Calendar. (Linear/Jira/ClickUp/Slack/Airtable already store MCP server URL + OAuth in `config`.) +- BookStack (community MCP), Elasticsearch (official Elastic Agent Builder MCP, 9.2+), Luma (community MCP) — confirmed MCP available, migrate rather than disable. +- Fix known gap: `MCP_CONNECTOR` is missing from the subagent routing map (`constants.py`) — generic MCP tools get discovered but skipped. + +**Web search APIs — repurposed (not a connector type):** + +- Tavily, SearXNG, Linkup, Baidu, Serper become a source-discovery helper for the Universal WebURL Crawler: given a topic/competitor, suggest candidate URLs the user can add to a pipeline. Reuses the existing web-search services; backend endpoint here, UX deferred to frontend umbrella. + +**Disabled for MVP:** + +- Obsidian (plugin push) and Circleback (meeting webhook) — disabled for the pivot MVP. + +### Phase 4 — Pipelines data model [`subplan: 04-pipelines-model.md`] + +- New tables: `pipelines` (workspace_id, user_id, connector_id, name, config JSON, schedule/cron, `save_to_kb` bool, `destination_folder_id` nullable, enabled, next_scheduled_at) and `pipeline_runs` (pipeline_id, status, trigger = manual/scheduled/upload, timestamps, doc counts, error, optional raw-result blob ref). +- Models + Pydantic schemas + Alembic migration + backend Zero publication entry. +- Pipelines API routes: CRUD + manual run trigger + list runs. + +### Phase 5 — Pipeline execution + scheduling [`subplan: 05-pipelines-exec.md`] + +- Run engine: pipeline run -> invoke connector fetch (WebURL crawler for MVP) -> if `save_to_kb`, route through `IndexingPipelineService` into the destination folder -> write `PipelineRun` record. +- Scheduling: reuse the Celery Beat meta-scheduler pattern (`schedule_checker_task.py`, `periodic_scheduler.py`) for cron + manual triggers. +- When `save_to_kb` is off, persist the raw fetch result on the run (blob via `file_storage`) so it is retrievable without indexing. +- Chat agent context: expose pipeline run history to the `multi_agent_chat` agent (read-only) — via a tool (e.g. `list_pipelines` / `get_pipeline_runs`) and/or a context middleware injection (similar to `KnowledgeTreeMiddleware`). Scope strictly to the active workspace. Gives the agent awareness of recent runs, statuses, schedules, and last-fetched timestamps. + +### Phase 6 — File upload as a pipeline + KB-save-secondary [`subplan: 06-upload-pipeline-kb.md`] + +- Wire file upload (`documents_routes.py` fileupload flow) to create/use an "Uploads" pipeline and register a `PipelineRun`; uploads always `save_to_kb = true`. +- Generalize KB saving to be opt-in for non-upload pipelines via `save_to_kb` + destination folder. + +## Deferred — Frontend & client phases (separate umbrella, planned LATER) + +These are recorded for continuity but are NOT planned in this umbrella. They start once the backend phases above are working. + +- Frontend rename + i18n: route segment `[search_space_id]` -> `[workspace_id]`, `search-space-settings/` -> `workspace-settings/`, TS types, api services, Jotai atoms, components, cache keys, and "Workspace" copy across 5 locales (`messages/{en,zh,es,pt,hi}.json`), plus frontend Zero schema rename. +- Satellite/client apps + docs rename: `surfsense_desktop`, `surfsense_obsidian`, `surfsense_browser_extension`, `surfsense_evals`, README/docs. +- Connector two-type UI: restructure `connector-popup` and `connector-constants.ts` into the two labeled types. +- Pipelines UI + positioning: Pipelines section (list/create/configure/run-history/manual run), WebURL source-discovery UX, file-upload-as-pipeline surfacing, "coming soon" platform cards, "NotebookLM for Competitive Intelligence" copy. + +## Open items to confirm during subplanning + +- ~~Rename transition: hard cutover vs temporary API aliases~~ RESOLVED: HARD CUTOVER (see resolved log + 02-rename-backend.md). The frontend is rebuilt against the corrected backend in its own umbrella; backend is verified via tests/OpenAPI, not the old UI. +- Whether existing connector periodic-indexing config is migrated into Pipelines or coexists during MVP. +- Chat agent run-history access: tool vs middleware injection vs both (default: tool). +- Type-2 MCP migration depth: actually re-point native connectors (Notion/GitHub/Gmail/etc.) to MCP servers now, vs keep their existing native integration and just re-tag them under the MCP-Tools category for MVP. + +## Resolved decisions log + +- Web search APIs (Tavily/SearXNG/Linkup/Baidu/Serper): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines); not a standalone connector type. +- Obsidian + Circleback: disabled for MVP. +- MCP-availability audit: BookStack, Elasticsearch, Luma all have MCP available -> migrate to Type-2, none disabled. +- Rename transition policy: HARD CUTOVER of the external API (paths + JSON field names) in Phase 2 — no backward-compat aliases. Rationale: the frontend is (re)built against the corrected backend later, so there is no old client to keep alive; backend correctness is verified via the test suite + OpenAPI rather than the existing UI. + +## Subplan index (backend) + +| Phase | Subplan file | Status | +|-------|--------------|--------| +| 1 | `01-rename-db.md` | drafted | +| 2 | `02-rename-backend.md` | drafted | +| 3 | `03-connector-two-type-backend.md` | not started | +| 4 | `04-pipelines-model.md` | not started | +| 5 | `05-pipelines-exec.md` | not started | +| 6 | `06-upload-pipeline-kb.md` | not started | + +Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases"). diff --git a/plans/backend/01-rename-db.md b/plans/backend/01-rename-db.md new file mode 100644 index 000000000..0ea093374 --- /dev/null +++ b/plans/backend/01-rename-db.md @@ -0,0 +1,179 @@ +# Subplan 01 — Rename foundation (DB) + +Part of [00-umbrella-plan.md](00-umbrella-plan.md), Phase 1. Backend only (`surfsense_backend`). + +## Goal + +Physically rename the `SearchSpace` schema to `WorkSpace` in PostgreSQL via a single Alembic migration, and update the ORM's physical mapping + the Zero publication so the app keeps booting and running — WITHOUT yet touching the ~250 backend files that reference the `search_space_id` Python attribute (that symbolic rename is Phase 2). + +Canonical target names: + +- Table `searchspaces` -> `workspaces` +- Tables `search_space_roles` / `search_space_memberships` / `search_space_invites` -> `workspace_roles` / `workspace_memberships` / `workspace_invites` +- Column `search_space_id` -> `workspace_id` (all child tables); `external_chat_accounts.owner_search_space_id` -> `owner_workspace_id` +- Named constraints/indexes containing `searchspace` -> `workspace` (see inventory) + +## The central coupling decision (read first) + +A physical column rename instantly breaks every line of code that reads the old name. Today that is large: + +- `search_space_id` / `searchspaces` appear in ~250 backend files outside `db.py` (Grep over `surfsense_backend/app`, excluding `db.py`). Almost all are Python ORM attribute reads like `Document.search_space_id` or `.search_space_id`. + +To keep Phase 1 self-contained and low-risk, decouple the PHYSICAL rename from the SYMBOLIC rename using an explicit column-name mapping shim: + +- Keep the Python attribute name `search_space_id`, but bind it to the new physical column: + - Today: `search_space_id = Column(Integer, ForeignKey("searchspaces.id", ondelete="CASCADE"), ...)` ([surfsense_backend/app/db.py](surfsense_backend/app/db.py) line 1382 etc.) + - Phase 1: `search_space_id = Column("workspace_id", Integer, ForeignKey("workspaces.id", ondelete="CASCADE"), ...)` +- Result: the physical DB is fully renamed, the ORM maps the unchanged Python attribute to the new column, and the ~250 callers keep working untouched. +- Phase 2 then renames the Python attribute (`search_space_id` -> `workspace_id`), the class (`SearchSpace` -> `Workspace`), schemas, routes/API, and drops the explicit `"workspace_id"` first-arg shim. + +CONFIRMED: ship the shim approach. (Alternative big-bang rename rejected as too risky.) + +## Current-state inventory (all citations from code) + +### The table itself + +- `class SearchSpace(... __tablename__ = "searchspaces")` — [surfsense_backend/app/db.py](surfsense_backend/app/db.py) line 1688. + +### Child tables with a `search_space_id` FK to `searchspaces.id` + +All declared `ForeignKey("searchspaces.id", ondelete="CASCADE")` unless noted. Locations in [surfsense_backend/app/db.py](surfsense_backend/app/db.py): + +- `new_chat_threads` (605), `external_chat_bindings` (897), `token_usage` (1113), `folders` (1318), `documents` (1382), `video_presentations` (1506), `reports` (1533), `connections` (1561, nullable), `image_generations` (1672), `search_source_connectors` (1878), `logs` (1905), `search_space_roles` (2045), `search_space_memberships` (2076), `search_space_invites` (2118), `prompts` (2176), `agent_action_log` (2495), `document_revisions` (2566), `folder_revisions` (2607), `agent_permission_rules` (2643). +- `external_chat_accounts.owner_search_space_id` (785-786, nullable) — note the distinct column name. + +In other model modules: + +- `document_files` — [surfsense_backend/app/file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) line 32. +- `automations` — [surfsense_backend/app/automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) line 27. +- `notifications` — [surfsense_backend/app/notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) line 50. +- `podcasts` — [surfsense_backend/app/podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) line 71. + +### Named constraints / indexes that embed the old name + +- `uq_searchspace_user_connector_type_name` on `search_source_connectors` — [db.py](surfsense_backend/app/db.py) line 1831. +- `uq_searchspace_role_name` on `search_space_roles` — line 2032. +- `uq_user_searchspace_membership` on `search_space_memberships` — line 2069. +- `idx_documents_search_space_id` and `idx_documents_search_space_updated` (raw `CREATE INDEX CONCURRENTLY` DDL in a module-level list) — lines 2821-2830. +- `ix_external_chat_bindings_search_space_state` — line 978-979. + +These reference `search_space_id` by COLUMN but the index/constraint NAME does not embed "searchspace" (definitions auto-follow a column rename; names can stay): + +- `ix_notifications_user_space_created` — [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) line 36-41. +- `uq_agent_permission_rules_scope` — [db.py](surfsense_backend/app/db.py) line 2680. + +### CHECK constraint referencing the column AND a scope literal + +- `ck_connections_scope_owner` on `connections` — [db.py](surfsense_backend/app/db.py) lines 1578-1584: + - `"(scope = 'GLOBAL' AND search_space_id IS NULL ...) OR (scope = 'SEARCH_SPACE' AND search_space_id IS NOT NULL ...) OR (scope = 'USER' ...)"`. + - The physical CHECK expression auto-follows a column rename (Postgres tracks by attnum), but the model's `CheckConstraint(...)` text must be updated to `workspace_id` for create_all/autogenerate consistency. + - The scope string literal `'SEARCH_SPACE'` is a `ConnectionScope` value, not a column — see Confirmed decisions. + +### Zero publication (single source of truth) + +- [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py) is "the single source of truth for `zero_publication`" (module docstring, lines 1-11). Publication changes "should update `ZERO_PUBLICATION` and call `apply_publication()` from a migration" (lines 3-5). +- `search_space_id` appears in the published column lists: + - `DOCUMENT_COLS` line 31. + - `PODCAST_COLS` line 66. +- These are the only two published tables with column lists that include the column (`ZERO_PUBLICATION` map, lines 71-82; other entries are `None` = all columns and auto-follow). `documents`/`podcasts` columns must be updated to `workspace_id`, then `apply_publication(op.get_bind())` re-run. +- Reconcile pattern to copy: [surfsense_backend/alembic/versions/155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) (calls `apply_publication(op.get_bind())`, no-op downgrade). + +## Migration design + +### Conventions (from the repo) + +- Revision ids are plain integers; head is `165` (`revision="165"`, `down_revision="164"`) — [surfsense_backend/alembic/versions/165_add_chunk_position.py](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 25-26. New file: `166_rename_searchspace_to_workspace.py` with `revision="166"`, `down_revision="165"`. +- Migrations run one-per-transaction under an advisory lock — [surfsense_backend/alembic/env.py](surfsense_backend/alembic/env.py) lines 77, 80-95. So all renames below land atomically. +- Raw `op.execute("ALTER TABLE ...")` is the house style (e.g. [165_add_chunk_position.py](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 39-49). + +### upgrade() steps (order matters) + +1. Rename tables: + - `ALTER TABLE searchspaces RENAME TO workspaces;` + - `ALTER TABLE search_space_roles RENAME TO workspace_roles;` + - `ALTER TABLE search_space_memberships RENAME TO workspace_memberships;` + - `ALTER TABLE search_space_invites RENAME TO workspace_invites;` + - FKs and PKs follow the table automatically (referenced by OID). Sequence `searchspaces_id_seq` and `*_pkey` keep old names — CONFIRMED: also `ALTER SEQUENCE searchspaces_id_seq RENAME TO workspaces_id_seq;` and `ALTER INDEX searchspaces_pkey RENAME TO workspaces_pkey;` (plus the RBAC tables' `*_id_seq` / `*_pkey`) for a clean final schema. +2. Rename columns on every child table (use `ALTER TABLE RENAME COLUMN search_space_id TO workspace_id;` for the full list in the inventory; plus `external_chat_accounts.owner_search_space_id -> owner_workspace_id`). + - Metadata-only, fast even on large tables (same property the repo relies on in [165](surfsense_backend/alembic/versions/165_add_chunk_position.py) lines 36-41). + - ORDERING NOTE: the RBAC tables were renamed in step 1, so their column rename must target the NEW table names (`workspace_roles` / `workspace_memberships` / `workspace_invites`), not the old ones. +3. Rename the named constraints/indexes. Use defensive/idempotent forms because some objects are created at runtime, not by migrations (see review finding 4): + - Constraints (no `IF EXISTS` for `RENAME CONSTRAINT`; wrap each in a guarded `DO` block that checks `pg_constraint`): + - `uq_searchspace_user_connector_type_name` -> `uq_workspace_user_connector_type_name` (on `search_source_connectors`). + - `uq_searchspace_role_name` -> `uq_workspace_role_name` (on `workspace_roles`). + - `uq_user_searchspace_membership` -> `uq_user_workspace_membership` (on `workspace_memberships`). + - Indexes (these two are created by the runtime `setup_indexes()` startup routine, NOT a migration, so they may or may not exist depending on `DB_BOOTSTRAP_ON_STARTUP`; use `IF EXISTS`): + - `ALTER INDEX IF EXISTS idx_documents_search_space_id RENAME TO idx_documents_workspace_id;` + - `ALTER INDEX IF EXISTS idx_documents_search_space_updated RENAME TO idx_documents_workspace_updated;` + - `ALTER INDEX IF EXISTS ix_external_chat_bindings_search_space_state RENAME TO ix_external_chat_bindings_workspace_state;` +4. Zero publication. DECISION: we will recreate the Zero replication (reset the zero-cache replica) as part of this deploy, so consumer recovery is handled out-of-band and finding 3 is de-risked. Follow the repo's established publication patterns — do NOT use raw `DROP`/`CREATE PUBLICATION` (forbidden by [116_create_zero_publication.py](surfsense_backend/alembic/versions/116_create_zero_publication.py) lines 8-17; reintroduces bug #1355). Sequence inside the migration: + - (Safe ordering) Neutralize the column-list dependency surgically so the RENAME is unconditionally permitted: `ALTER PUBLICATION zero_publication DROP TABLE documents, podcasts;`. This removes ONLY their column-list dependency and is far safer than re-emitting the whole member set via `SET TABLE` (which is a full replacement and would drop any table accidentally omitted from the hand-written list, including the quoted `"user"` table). `DROP TABLE` from a publication is within the blessed ALTER-PUBLICATION family — it is NOT the forbidden raw `DROP/CREATE PUBLICATION`. + - Rename the columns (steps 1-2). + - Update canonical `DOCUMENT_COLS`/`PODCAST_COLS` to `workspace_id` in [zero_publication.py](surfsense_backend/app/zero_publication.py) (31, 66), then call `apply_publication(op.get_bind())` (blessed plain `ALTER ... SET TABLE`, as used by [155_reconcile_zero_publication.py](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py)) to re-add `documents`/`podcasts` (dropped above) with the narrowed `workspace_id` column lists and reconcile the full member set in one shot. + - Reference template for forcing a schema-change event (only if event triggers are unavailable): the `COMMENT ON PUBLICATION` bookend trio in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) lines 104-137. With event triggers installed (current setup), `apply_publication` is sufficient. + - Replica recreate (operational, out-of-band): reset the `surfsense-zero-cache` replica (delete the volume / rely on `ZERO_AUTO_RESET=true`) so zero-cache does a fresh initial sync from the corrected publication. This is the "recreate replication" step and the primary consumer-recovery mechanism. + - INTERLOCK FOOTGUN (review finding 2): if canonical `DOCUMENT_COLS`/`PODCAST_COLS` are NOT updated to `workspace_id` before `apply_publication` runs, `_format_table_entry` ([zero_publication.py](surfsense_backend/app/zero_publication.py) lines 136-137) silently DROPS `documents`/`podcasts` from the publication (no error). Gate with `--verify` (see Verification). +5. No DDL needed for the `ck_connections_scope_owner` CHECK: the physical expression auto-follows the column rename (Postgres tracks the column by attnum). The model's `CheckConstraint(...)` text is updated separately as an ORM edit (see ORM edits) so create_all/autogenerate stay consistent. + +### downgrade() + +Provide a full reverse (rename `workspaces` -> `searchspaces`, columns back, constraints/indexes back, then restore the OLD publication shape). Note the repo sometimes uses no-op downgrades for publication-only migrations ([155](surfsense_backend/alembic/versions/155_reconcile_zero_publication.py) lines 22-23), but a structural rename should be reversible. + +CRITICAL (review finding 7 — do NOT call `apply_publication()` in downgrade): `apply_publication()` reads the LIVE canonical in `zero_publication.py`, which after this change is `workspace_id`. In a downgrade the column is back to `search_space_id`, so `apply_publication` would find `workspace_id` missing and silently DROP `documents`/`podcasts` from the publication — the finding-2 footgun, self-inflicted. Instead, the downgrade must restore the old shape with HARDCODED `search_space_id` column lists via a plain `ALTER PUBLICATION ... SET TABLE`, mirroring the constants-style approach in [143_force_zero_publication_resync.py](surfsense_backend/alembic/versions/143_force_zero_publication_resync.py) (which embeds literal `DOCUMENT_COLS` rather than importing the live module). Use the same DROP-then-SET neutralize sequence in reverse. + +COHERENCE (review finding 6): even with a correct downgrade SQL, `downgrade()` only produces a consistent RUNNING system if the PRIOR code revision is redeployed alongside it — under the shim the model maps the attribute to physical `workspace_id`, which no longer exists after a pure DB downgrade. Document rollback = revert code + schema together as one operation. + +## ORM / source edits in this phase (mapping only) + +These are required so the ORM matches the renamed physical schema; they are NOT the broad symbolic rename. Explicitly OUT of scope for Phase 1: relationship attribute names (`User.search_spaces`, `back_populates="search_space"`, etc.) and class names (`SearchSpace`, `SearchSpaceRole`, ...). They stay as-is here and are renamed in Phase 2 — leaving them untouched is what keeps the ~250 callers green. + +1. [surfsense_backend/app/db.py](surfsense_backend/app/db.py): + - `__tablename__` for `SearchSpace` (1688), `SearchSpaceRole` (2027), `SearchSpaceMembership` (2064), `SearchSpaceInvite` (2113) -> new table names. + - Every `ForeignKey("searchspaces.id", ...)` -> `ForeignKey("workspaces.id", ...)`; `ForeignKey("search_space_roles.id"...)` -> `workspace_roles.id`; `ForeignKey("search_space_invites.id"...)` -> `workspace_invites.id`. + - Apply the explicit-name shim on each `search_space_id` column: `Column("workspace_id", ...)`; and `owner_search_space_id = Column("owner_workspace_id", ...)`. + - Update ONLY the `name=` kwarg of constraints/indexes (1831, 2032, 2069, and the Index at 978-979). CRITICAL (review finding 1): do NOT change the inner column-reference strings (`"search_space_id"` at 1827, 2030, 2068, 2674, 979) — under the shim the column's `.key` is still `search_space_id`, and declarative `__table_args__` resolves these strings against `Table.c` by `.key`. Changing them to `"workspace_id"` raises a config-time error and the app won't boot. + - Update the runtime raw index DDL strings + names in `_INDEX_DEFINITIONS` (2820-2830): both the index name and the physical column (`search_space_id` -> `workspace_id`). These must match the migration's renamed indexes and ship in the same release; `setup_indexes()` ([db.py](surfsense_backend/app/db.py) line 2855) re-creates them under the new name on next boot when `DB_BOOTSTRAP_ON_STARTUP` is true. + - Update `ck_connections_scope_owner` text (1579-1582) `search_space_id` -> `workspace_id`. + - Update `ix_external_chat_bindings_search_space_state` name + the surrounding Index (978-979). +2. Module models — `__tablename__` unaffected, but apply the column shim + FK target string: + - [file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) (32), [automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) (27), [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) (50, plus the index at 36-41), [podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) (71). +3. [surfsense_backend/app/zero_publication.py](surfsense_backend/app/zero_publication.py): `DOCUMENT_COLS` (31) and `PODCAST_COLS` (66) `search_space_id` -> `workspace_id`. (The `_expected_columns` special-casing on lines 106 keys off table names `documents`/`podcasts`, not the column, so no change there.) +4. Runtime raw-SQL audit (defensive): grep `surfsense_backend/app` for any `text(...)` / hardcoded `"searchspaces"` or `search_space_id` strings that execute at runtime (not Python attribute access) and fix them here, since the shim only covers ORM attribute access. Expectation is few; the ~250-file footprint is overwhelmingly attribute access handled by the shim. + +## Confirmed decisions + +- Strategy: SHIM approach (physical DB rename now; Python attribute kept via `Column("workspace_id", ...)`; symbolic attribute/class/API rename deferred to Phase 2). +- Scope literal `'SEARCH_SPACE'` (the `ConnectionScope` value in `ck_connections_scope_owner`): KEEP the string value as-is; only fix the `search_space_id` column reference in the CHECK. No enum change, no data UPDATE. +- Cosmetic names: RENAME auto-named sequences/PK indexes (`searchspaces_id_seq` -> `workspaces_id_seq`, `searchspaces_pkey` -> `workspaces_pkey`, and the RBAC tables' `*_id_seq`/`*_pkey`) for a clean final schema. +- Transition: HARD CUTOVER at the DB layer (no backward-compat old column/view). Client apps hit the API (Phase 2), not the DB, so the DB rename does not affect them. (Phase 2 RESOLVED: hard cutover, no API aliases — see [02-rename-backend.md](02-rename-backend.md).) +- Zero: RECREATE THE REPLICATION (reset the zero-cache replica / `ZERO_AUTO_RESET`) on deploy for consumer recovery. Publication mutated only via the blessed `apply_publication` / `ALTER ... SET TABLE` path (never raw DROP/CREATE PUBLICATION, per migration 116). See migration step 4. + +## Release coupling (review finding 5) + +The Alembic migration, the `db.py` edits (shim + constraint `name=` + `_INDEX_DEFINITIONS` + `ck_connections_scope_owner` text), the 4 module-model edits, and `zero_publication.py` MUST ship as ONE atomic release. There is no safe intermediate state where some are updated and others are not. + +## Verification & rollout + +- HARD GATE: `python -m app.zero_publication --verify` (CLI at [zero_publication.py](surfsense_backend/app/zero_publication.py) lines 253-264) must report verified, in CI and immediately post-migrate. This is the guard against the interlock footgun (finding 2). +- Boot the API + a Celery worker; run a smoke chat + a document upload to exercise `documents.workspace_id` and the publication; confirm Zero clients still sync `documents`/`podcasts`. +- AUTOGENERATE DRIFT CHECK (strong, cheap): after the change, `alembic revision --autogenerate` must produce an EMPTY diff. A non-empty diff means the ORM and migrated DB disagree — i.e. a missed `ForeignKey("searchspaces.id")` string, a constraint `name=` mismatch, or a forgotten `Column("workspace_id", ...)` shim. This single check catches most subtle misses. +- Confirm `alembic upgrade head` then `alembic downgrade -1` then `upgrade head` round-trips on a staging copy. NOTE: this validates SQL reversibility only — it is NOT an app-health check, because new code cannot run against the downgraded schema (finding 6). The downgraded intermediate state must show `documents`/`podcasts` still present in the publication with `search_space_id` lists (proves the hardcoded downgrade publication shape from finding 7 works). +- Run the full backend test suite; the shim keeps attribute-based tests green — watch specifically for fixtures/raw SQL that hardcode the table name `searchspaces`. +- Pre-flight check: confirm no DB views/materialized views depend on the renamed objects (they auto-follow a rename, but verify so a later `pg_dump` diff holds no surprises). +- Raw-SQL audit done-check: grep `surfsense_backend/app` for `text(...)` or string literals containing `searchspaces` / `search_space_id` that execute at runtime; current audit found none beyond ORM attribute/dict-key usage (which the shim covers). + +## Risks (with review findings) + +- RENAME COLUMN under an active Zero publication column list (finding 3) — de-risked: step 4 neutralizes the column-list dependency (publish documents/podcasts as ALL columns) before the rename, and the zero-cache replica is recreated on deploy, so consumers re-sync cleanly. +- Silent publication drop if canonical col lists aren't updated (finding 2) — mitigated by the `--verify` hard gate. +- Zero consumers recover via the planned replica recreate (reset `surfsense-zero-cache` / `ZERO_AUTO_RESET=true`), giving a fresh initial sync from the corrected publication; no raw DROP/CREATE PUBLICATION (bug #1355). +- Constraint/index column-ref strings accidentally changed under the shim (finding 1) — config-time boot failure; mitigated by the explicit "only change name=" instruction. +- Runtime-created indexes may not exist at migrate time (finding 4) — mitigated by `IF EXISTS` renames + guarded `DO` blocks for constraints. +- Missing a `ForeignKey("searchspaces.id")` string leaves SQLAlchemy metadata pointing at a non-existent table — fails fast at mapper config (`NoReferencedTableError`); the inventory list above is the checklist. +- Partial/non-atomic release (finding 5) or code-less downgrade (finding 6) — mitigated by shipping all edits in one release and treating rollback as code+schema together. +- `downgrade()` calling `apply_publication()` against the live (new) canonical (finding 7) — would silently re-drop `documents`/`podcasts`; mitigated by hardcoding the old `search_space_id` publication shape in `downgrade()` (143-style), never importing the live module. + +## Out of scope (later phases) + +- Phase 2: rename Python attribute `search_space_id` -> `workspace_id`, class `SearchSpace` -> `Workspace`, Pydantic schemas, API routes/paths (`/searchspaces` + `/search-spaces` -> `/workspaces`), Redis keys, storage path segments; drop the explicit `Column("workspace_id", ...)` shim. +- Frontend Zero schema, route segment, i18n — deferred frontend umbrella. diff --git a/plans/backend/02-rename-backend.md b/plans/backend/02-rename-backend.md new file mode 100644 index 000000000..12feac44f --- /dev/null +++ b/plans/backend/02-rename-backend.md @@ -0,0 +1,133 @@ +# Subplan 02 — Rename backend (code + API) + +Part of [00-umbrella-plan.md](00-umbrella-plan.md), Phase 2. Backend only (`surfsense_backend`). + +## Goal + +Remove the Phase 1 shim and complete the SYMBOLIC rename `SearchSpace -> Workspace` / `search_space_id -> workspace_id` across the ~150 backend files, then consolidate the three live URL spellings (`/searchspaces`, `/search-spaces`, `/search-space`) onto a single canonical `/workspaces`. After this phase the physical DB (Phase 1) and the Python/API surface speak the same name. + +Precondition: [01-rename-db.md](01-rename-db.md) is merged and live. Phase 1 left the Python attribute as `search_space_id` mapped to the physical column via `Column("workspace_id", ...)`; Phase 2 flips the attribute itself and drops that explicit-name shim. + +## Surface area (cited; counts are current-tree, will shift after Phase 1) + +Measured over `surfsense_backend/app` (Grep, case-insensitive `search.?space`): + +- `search_space` — ~150 files, ~2,800 line matches (dominated by ORM attribute access). +- `SearchSpace` (class/relationship target) — ~38 files, ~450 matches. +- Heaviest files: [services/connector_service.py](surfsense_backend/app/services/connector_service.py) (158), [routes/search_source_connectors_routes.py](surfsense_backend/app/routes/search_source_connectors_routes.py) (138), [routes/rbac_routes.py](surfsense_backend/app/routes/rbac_routes.py) (142), [routes/model_connections_routes.py](surfsense_backend/app/routes/model_connections_routes.py) (132), [routes/new_chat_routes.py](surfsense_backend/app/routes/new_chat_routes.py) (133), [routes/documents_routes.py](surfsense_backend/app/routes/documents_routes.py) (121), [db.py](surfsense_backend/app/db.py) (119). +- Plus ~120 test files under `surfsense_backend/tests` mirror these patterns (rename in the same release). + +This is mostly mechanical (an automated symbol rename handles the bulk), but a fixed set of STRING-LITERAL contracts and EXTERNAL contracts are NOT safe to blind-replace — they get explicit decisions in this plan (see "String-literal & external contracts"). + +## Transition policy: HARD CUTOVER (confirmed) + +The umbrella defers all frontend/client work, and the frontend will be (re)built against the corrected backend in its own umbrella rather than kept alive in lockstep. So this phase HARD-CUTS the external API — no backward-compat aliases. + +- URL paths: rename `/api/v1/searchspaces...`, `/api/v1/search-spaces/...`, `/api/v1/...search-space...` outright to the canonical `/api/v1/workspaces...` ([routes/search_spaces_routes.py](surfsense_backend/app/routes/search_spaces_routes.py) 73-373; mount at [app.py](surfsense_backend/app/app.py) 993, `crud_router` under `/api/v1`). +- JSON field names: rename `search_space_id` -> `workspace_id` in request/response bodies outright (e.g. [schemas/new_chat.py](surfsense_backend/app/schemas/new_chat.py), [schemas/rbac_schemas.py](surfsense_backend/app/schemas/rbac_schemas.py), [schemas/documents.py](surfsense_backend/app/schemas/documents.py)). No `populate_by_name` alias plumbing. + +CONSEQUENCE (accepted): the existing deployed frontend breaks against the renamed API until its umbrella lands. Backend correctness is therefore verified INDEPENDENTLY of the old UI — via the test suite, OpenAPI/`/docs`, and direct API calls — not by clicking through the current frontend. (Alias/dual-serve was considered and rejected: it adds plumbing to keep a frontend alive that is being redesigned anyway.) + +## Rename waves (order matters) + +### Wave A — ORM core in [db.py](surfsense_backend/app/db.py) + +1. Classes: `SearchSpace` (1688), `SearchSpaceRole` (2021/2027), `SearchSpaceMembership` (2058/2064), `SearchSpaceInvite` (2107/2113) -> `Workspace`, `WorkspaceRole`, `WorkspaceMembership`, `WorkspaceInvite`. +2. Drop the Phase 1 shim on every column: `search_space_id = Column("workspace_id", Integer, ForeignKey("workspaces.id", ...))` -> `workspace_id = Column(Integer, ForeignKey("workspaces.id", ...))` (the explicit `"workspace_id"` first arg is now redundant because attribute name == column name). Same for `owner_search_space_id` -> `owner_workspace_id` (785-787). +3. INVERSE OF PHASE-1 FINDING 1: now that the attribute `.key` becomes `workspace_id`, the `__table_args__` inner column-reference strings MUST flip to `"workspace_id"` to match — `UniqueConstraint("search_space_id", ...)` -> `UniqueConstraint("workspace_id", ...)` at 1826-1832 (`uq_workspace_user_connector_type_name`), 2029-2033 (`uq_workspace_role_name`), 2066-2070 (`uq_user_workspace_membership`), and the `Index(... "search_space_id" ...)` at 978-979. (Phase 1 deliberately left these as `search_space_id`; Phase 2 completes them. Getting this wrong = config-time boot failure, same failure mode as Phase 1 finding 1, just mirrored.) +4. Relationships (~35 pairs): rename the attribute names and both ends of `back_populates`, and the target-class strings: + - Child side: `search_space = relationship("SearchSpace", back_populates="...")` -> `workspace = relationship("Workspace", back_populates="...")` (e.g. Folder 1340, Document 1421, Connection 948). + - Hub side on `Workspace`: `back_populates="search_space"` -> `"workspace"` for folders (1726-1728), documents, threads, podcasts, video_presentations, reports, image_generations, logs, notifications, connectors, connections, automations, roles, memberships, invites. + - User side (both auth branches ~2205 and ~2337): `search_spaces = relationship("SearchSpace", ...)` -> `workspaces = relationship("Workspace", ...)`; `search_space_memberships` -> `workspace_memberships`; invites likewise. + - External chat: `owner_search_space = relationship("SearchSpace", foreign_keys=[owner_search_space_id])` (819-820) -> `owner_workspace = relationship("Workspace", foreign_keys=[owner_workspace_id])`. +5. CHECK text on `connections`: `ck_connections_scope_owner` (1578-1581) `search_space_id` -> `workspace_id` in the SQL string. (Phase 1 already did this if shipped; re-confirm. The scope literal `'SEARCH_SPACE'` stays — see enum carve-out.) +6. Runtime index DDL in `_INDEX_DEFINITIONS` (2820-2830): already renamed to `workspace_id`/`idx_documents_workspace_id` in Phase 1 — confirm no `search_space` remains. + +### Wave B — satellite ORM modules (mirror Wave A; drop shim, flip strings) + +- [automations/persistence/models/automation.py](surfsense_backend/app/automations/persistence/models/automation.py) (27-29 column, 68 relationship). +- [file_storage/persistence/models.py](surfsense_backend/app/file_storage/persistence/models.py) (32-34). +- [notifications/persistence/models.py](surfsense_backend/app/notifications/persistence/models.py) (50-52 column, 72 relationship, plus the index `ix_notifications_user_space_created` 36-41 — definition auto-follows; rename only if you want the name clean). +- [podcasts/persistence/models.py](surfsense_backend/app/podcasts/persistence/models.py) (71-72). + +### Wave C — Pydantic schemas + +- Dedicated module [schemas/search_space.py](surfsense_backend/app/schemas/search_space.py): classes `SearchSpaceBase/Create/Update/Read/WithStats` -> `Workspace*`. Rename file to `schemas/workspace.py`. Update re-exports in [schemas/__init__.py](surfsense_backend/app/schemas/__init__.py) (106-111, and `UserSearchSpaceAccess` ~87). +- Field `search_space_id` -> `workspace_id` across: new_chat, rbac_schemas (+ `search_space_name`, `UserSearchSpaceAccess`), documents, folders, image_generation, model_connections, search_source_connector, logs, stripe, prompts, chat_comments, video_presentations, reports, obsidian_plugin, podcasts/api/schemas, automations schemas, notifications/api/schemas. +- Under hard cutover the serialized JSON key changes outright to `workspace_id` (no alias). Update fixtures/contract tests accordingly. + +### Wave D — services / utils / tasks / agents / gateway / event_bus (the mechanical bulk) + +Pure attribute/param/function symbol rename `search_space_id -> workspace_id`, `search_space -> workspace`, plus function renames: +- RBAC helpers [utils/rbac.py](surfsense_backend/app/utils/rbac.py): `check_search_space_access` (129), `is_search_space_owner` (160), `get_search_space_with_access_check` (180) -> `*_workspace_*`. +- Validator [utils/validators.py](surfsense_backend/app/utils/validators.py): `validate_search_space_id` (16). +- Retrievers [retriever/documents_hybrid_search.py](surfsense_backend/app/retriever/documents_hybrid_search.py), [retriever/chunks_hybrid_search.py](surfsense_backend/app/retriever/chunks_hybrid_search.py) (~26 each, filter clauses). +- Connector service [services/connector_service.py](surfsense_backend/app/services/connector_service.py) (ctor param + `self.search_space_id`, + the in-process caches keyed by id — see literals). +- Indexers under `app/tasks/connector_indexers/*`, `indexing_pipeline/*`, `services/*/kb_sync_service.py`. +- Agents: dep-dict KEYS like `"search_space_id": d["search_space_id"]` (e.g. [agents/.../subagents/connectors/google_drive/tools/index.py](surfsense_backend/app/agents/chat/multi_agent_chat/subagents/connectors/google_drive/tools/index.py) 28; `main_agent/runtime/factory.py` 137); class `SearchSpaceSkillsBackend` (`.../main_agent/skills/backends.py` 184). +- Gateway: [gateway/inbox_processor.py](surfsense_backend/app/gateway/inbox_processor.py), [gateway/agent_invoke.py](surfsense_backend/app/gateway/agent_invoke.py), [gateway/auth_invariant.py](surfsense_backend/app/gateway/auth_invariant.py). +- Event bus: `Event.search_space_id` field ([event_bus/event.py](surfsense_backend/app/event_bus/event.py) 36), `publish(... search_space_id=...)` ([event_bus/bus.py](surfsense_backend/app/event_bus/bus.py) 43-49), payload-dict key in [event_bus/events/document_entered_folder.py](surfsense_backend/app/event_bus/events/document_entered_folder.py) 74-76 — see literals decision. + +### Wave E — routes: rename handlers + consolidate the three URL spellings + +Canonical: `/workspaces` (umbrella decision). Current spellings to retire: +- `/searchspaces` (no hyphen): [search_spaces_routes.py](surfsense_backend/app/routes/search_spaces_routes.py) (73-373), [rbac_routes.py](surfsense_backend/app/routes/rbac_routes.py), [agent_permissions_route.py](surfsense_backend/app/routes/agent_permissions_route.py), [team_memory_routes.py](surfsense_backend/app/routes/team_memory_routes.py). +- `/search-spaces` (hyphen, plural): [editor_routes.py](surfsense_backend/app/routes/editor_routes.py), [notes_routes.py](surfsense_backend/app/routes/notes_routes.py), [export_routes.py](surfsense_backend/app/routes/export_routes.py), [model_connections_routes.py](surfsense_backend/app/routes/model_connections_routes.py) (`/model-roles`). +- `/search-space` (hyphen, singular): [logs_routes.py](surfsense_backend/app/routes/logs_routes.py) 252, [gateway_webhook_routes.py](surfsense_backend/app/routes/gateway_webhook_routes.py) 992/1022, webhook path `{search_space_id}` in [circleback_webhook_route.py](surfsense_backend/app/routes/circleback_webhook_route.py) 215/315. +- Rename the route files (`search_spaces_routes.py` -> `workspaces_routes.py`) and handler fns (`create_search_space` etc.) and the import/include in [routes/__init__.py](surfsense_backend/app/routes/__init__.py) (63, 73). + +Mechanism: routers use `APIRouter()` with the path fully spelled in each decorator (no per-router prefix), mounted under `/api/v1` at [app.py](surfsense_backend/app/app.py) 993. So consolidation = edit each decorator string to `/workspaces/{workspace_id}/...` outright (hard cutover; no alias routers). + +### Wave F — tests + fixtures + +Rename `surfsense_backend/tests/**`; watch for fixtures that hardcode the table name `searchspaces`/JSON key `search_space_id` (those depend on the transition policy for API tests). + +## String-literal & external contracts (explicit decisions — NOT blind-replace) + +These do not move with a symbol rename; each is decided here. + +1. Enum VALUES `'SEARCH_SPACE'` — `ConnectionScope.SEARCH_SPACE` ([db.py](surfsense_backend/app/db.py) 204-207, stored in `connections.scope` via `SQLAlchemyEnum`, line 1558) and `ChatVisibility.SEARCH_SPACE` (510-520, stored in `new_chat_threads.visibility`, 596-597). DECISION: KEEP the enum value strings as-is. They are persisted in Postgres and exposed in JSON; renaming the value needs a data migration + PG enum-type alter for zero benefit. This matches Phase 1's decision to keep the `'SEARCH_SPACE'` CHECK literal. (Optionally rename only the Python member to `WORKSPACE` while keeping `= "SEARCH_SPACE"` — deferred; not worth the churn for MVP.) +2. Celery task `name=` strings — `"delete_search_space_background"` ([tasks/celery_tasks/document_tasks.py](surfsense_backend/app/tasks/celery_tasks/document_tasks.py) 206), `"ai_sort_search_space"` (1546). These are the WIRE NAME between producer and worker; tasks dispatch via `.delay()`/`send_task` (e.g. event trigger `send_task(TASK_NAME, ...)` in [automations/triggers/builtin/event/source.py](surfsense_backend/app/automations/triggers/builtin/event/source.py) 19). DECISION: KEEP the `name=` strings unchanged; freely rename the Python function symbols (`ai_sort_search_space_task` etc.). A rolling deploy with renamed wire names would orphan in-flight messages. (If a cosmetic rename is wanted later, do it with a dual-register + queue-drain, out of scope here.) +3. Redis key literals — `surfsense:spawn_paused:{search_space_id}` (`tasks/.../spawn_paused.py`) and `ai_sort:search_space:{search_space_id}:lock` ([document_tasks.py](surfsense_backend/app/tasks/celery_tasks/document_tasks.py) 1542). DECISION: rename literals to `workspace` (these hold short-lived locks / an ops toggle). Accept that any in-flight lock / paused-flag resets at deploy (locks are seconds-long; paused-flag is an ops action). UPDATE the runbook in `.env.example` (the `redis-cli SET surfsense:spawn_paused:` doc, ~503-504). +4. Event payload key `search_space_id` — `Event` is `model_dump`ed and sent to Celery for automation event triggers, and trigger filters read the key. DECISION: rename the field to `workspace_id` (internal, and triggers are re-evaluated continuously); accept transient loss of any event enqueued across the deploy boundary (fire-and-forget). Drain the event queue during the maintenance window to be safe. +5. OpenTelemetry attribute `search_space.id` + metric label ([observability/otel.py](surfsense_backend/app/observability/otel.py) 263-264/305-306, [observability/metrics.py](surfsense_backend/app/observability/metrics.py) 537-542). DECISION: KEEP the OTel/metric KEY `search_space.id` for now (dashboards/alerts depend on it); rename only the Python params. Schedule the observability-key rename as a separate, announced change. (Carve-out to avoid silently breaking alerting.) +6. Notification dedup/operation IDs embedding `{search_space_id}` (e.g. `doc_..._{search_space_id}_...`, `insufficient_credits_{search_space_id}_...`) and frontend deep-link strings like `/dashboard/{search_space_id}/buy-more`. DECISION: the ID is a numeric value, not the literal word — leave format strings as-is functionally; the embedded VALUE is unchanged. The `/dashboard/{id}/...` deep link points at a FRONTEND route still named `[search_space_id]` (deferred umbrella) — KEEP it until the frontend segment renames, else links 404. +7. Storage path builders — `documents/{search_space_id}/...` ([file_storage/keys.py](surfsense_backend/app/file_storage/keys.py) 20-26) and `podcasts/{search_space_id}/...` ([podcasts/storage.py](surfsense_backend/app/podcasts/storage.py) 22-25). The path segment is the numeric ID; the literal word `search_space` is NOT in stored object keys. DECISION: rename the param only; NO blob migration needed; existing objects keep resolving. +8. `SearchSourceConnector` — contains "search" but is the connectors table, a different concept. OUT OF SCOPE (its rename is Phase 3's `category` work, not this rename). +9. Historical Alembic migrations (`surfsense_backend/alembic/versions/*`) — ~20 files embed `searchspaces` / `search_space_id` as raw-SQL string literals (e.g. `23_associate_connectors_with_search_spaces.py`, `41_backfill_rbac_for_existing_searchspaces.py`, `40_move_llm_preferences_to_searchspace.py`). DECISION: NEVER rewrite these. They are an immutable replay log that intentionally references the schema as it existed at that revision; rewriting them corrupts history and breaks a clean `alembic upgrade` from zero. Verified safe: no migration imports the ORM classes (only `from app.db import Base` in `env.py` / `0_initial_schema.py`), so the class rename does not touch them. Phase 1's migration 166 is the single transition point; migrations >166 use the new names. The scripted rename scope is `app/` + `tests/` ONLY. +10. LangGraph persisted state channel key `search_space_id` — `input_state = {..., "search_space_id": search_space_id, ...}` ([tasks/chat/streaming/flows/new_chat/input_state.py](surfsense_backend/app/tasks/chat/streaming/flows/new_chat/input_state.py) 131) is a CHECKPOINTED state channel. The resume_chat flow reads persisted state (`agent.aget_state({"configurable": {"thread_id": ...}})`, [flows/resume_chat/resume_routing.py](surfsense_backend/app/tasks/chat/streaming/flows/resume_chat/resume_routing.py) 50) and HITL interrupts (`surfsense_resume_value`, `HumanReview`) can sit pending across a deploy. DECISION: rename the channel key to `workspace_id` for consistency, and ACCEPT that any chat thread paused at a HITL interrupt BEFORE the cutover must be restarted after (the old checkpoint carries `search_space_id`, the new graph reads `workspace_id`). Operationally: drain/resolve pending interrupts before deploy if practical. (Alternative — keep the channel key as a carve-out — rejected: leaves a lone `search_space_id` in otherwise-renamed agent state for a transient, conversation-scoped value. The `configurable` keys are `thread_id` / `surfsense_resume_value`, not `search_space_id`, so only this state channel is affected.) +11. User-facing default literal — `users.py` seeds the default workspace `name="My Search Space"` (line 158, persisted + shown to users) and logs "Created default search space" (207). DECISION: rename the default name to "My Workspace" (backend-created seed value, not caught by a symbol rename); log strings are cosmetic. Also update the Redis-key assertion in `tests/unit/services/test_ai_sort_task_dedupe.py:14` when literal 3 is renamed. + +## Execution approach + +- Automate the safe bulk: a scripted symbol rename (IDE rename / `ast-grep`-style) for `search_space_id -> workspace_id`, `search_space -> workspace`, `SearchSpace -> Workspace`. SCOPE STRICTLY to `app/` + `tests/` — NEVER `alembic/versions/` (carve-out 9: immutable replay log). EXCLUDE the other carve-outs above (enum values, Celery `name=`, OTel keys, frontend deep-links, default-name literal). Run per-wave, not all at once, so review is tractable. +- Then hand-apply the carve-outs and the routing consolidation. +- One atomic release (same coupling rule as Phase 1): ORM + schemas + routes + literals ship together; there is no half-renamed steady state. + +## Verification & rollout + +- `alembic revision --autogenerate` must produce an EMPTY diff: confirms the renamed ORM (classes/attrs/constraint inner-strings) still matches the Phase 1 physical schema. A diff means a missed attribute flip or a stale `__table_args__` string. +- Boot API + Celery worker; verify via OpenAPI/`/docs` + direct API calls (NOT the old frontend): create workspace, chat, document upload, an RBAC invite/membership op, an automation event trigger, and a podcast — exercising the renamed relationships (`workspace`, `workspaces`), tasks, and event payloads. +- Hard-cutover contract check: confirm `/workspaces...` resolves and the three legacy spellings now 404; confirm bodies use `workspace_id` only. +- Full backend test suite green; grep `surfsense_backend/app` for residual `search_space`/`SearchSpace` and confirm every remaining hit is an intentional carve-out (enum value, Celery `name=`, OTel key, frontend deep-link). +- Confirm `alembic/versions/*` was NOT modified by the rename (git diff shows zero changes under that path) — guard against carve-out 9. +- Resume check (carve-out 10): start a chat, trigger a HITL interrupt, deploy/rename, then verify NEW interrupts resume on `workspace_id`; accept that pre-cutover pending interrupts are restarted. Ideally drain pending interrupts before deploy. +- Observability check: `search_space.id` OTel attribute still emitted (carve-out 5) so dashboards keep working. + +## Risks + +- Stale `__table_args__` inner string after attribute flip (Wave A.3) — config-time boot failure; mirror of Phase 1 finding 1. Mitigated by the autogenerate empty-diff gate + boot smoke test. +- Relationship rename misses one `back_populates` end — mapper-config error at startup; the pair list in Wave A.4 is the checklist. +- Hard cutover breaks the existing frontend in the interim (accepted) — mitigated by verifying the backend via tests/OpenAPI/API calls, and by (re)building the frontend against the new contract in its umbrella. Do not rely on the old UI to smoke-test during this window. +- Renaming a Celery `name=` or the event payload key without a drain — orphaned/dropped in-flight messages; mitigated by the carve-out (keep task names) + queue-drain for events. +- Enum value migration scope creep — explicitly OUT (carve-out 1); keep `'SEARCH_SPACE'` values. +- Scripted rename corrupting historical Alembic migrations (carve-out 9) — would break `alembic upgrade` from zero / replay history; mitigated by scoping the rename to `app/`+`tests/` and the git-diff guard on `alembic/versions/`. +- Renaming the persisted LangGraph state channel key (carve-out 10) — breaks resume of threads paused at a HITL interrupt across the cutover; mitigated by draining interrupts pre-deploy and accepting restart of any stragglers (state is conversation-scoped/ephemeral). +- Sheer size (~150 app + ~120 test files) — mitigated by per-wave scripted rename + the empty-diff and residual-grep gates. + +## Out of scope (later phases / umbrella) + +- Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella. +- Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred. +- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 3 ([03-connector-two-type-backend.md](03-connector-two-type-backend.md)). +- Enum VALUE migration and observability-key rename — deliberately deferred announced changes.