chore: synced plans

This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-06-29 18:26:29 -07:00
parent 96b18af607
commit 34ab23f5d8
10 changed files with 24 additions and 20 deletions

View file

@ -59,12 +59,16 @@ flowchart TD
### Phase 1 — Rename foundation (DB) [`subplan: 01-rename-db.md`]
> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Migration `170` (chains `169`, current head) does the physical rename + Zero-publication reconcile. Shipped atomically with Phase 2. As-built record + re-runnable verification live in `01-rename-db.md`. **Deploy caveat:** zero-cache replica reset (`ZERO_AUTO_RESET`) required; from-scratch `alembic upgrade head` stays pre-existing-broken (rev 23 conflict — separate baseline-squash task), only the `169→170` path is verified.
- Alembic migration: rename `searchspaces` -> `workspaces`; rename `search_space_id` -> `workspace_id` on ~20 child tables; rename RBAC tables and their FKs; rename indexes/constraints (`uq_searchspace_*`, `idx_documents_search_space_id`, etc.); update Rocicorp Zero publication column lists (backend-owned `publication` definition; frontend Zero schema rename happens in the later frontend umbrella).
- Decide transition strategy: hard cutover (simplest for MVP) vs temporary API aliases for clients.
- Key files: `surfsense_backend/app/db.py`, `surfsense_backend/alembic/versions/` (new migration).
### Phase 2 — Rename backend (code + API) [`subplan: 02-rename-backend.md`]
> **✅ SHIPPED** (2026-06-27) · branch `feat/rename-searchspace-to-workspace` · PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546) (merged to `ci_mvp` via [#1562](https://github.com/MODSetter/SurfSense/pull/1562)). Symbolic rename across `app/` + `tests/` (Phase-1 shim dropped), API consolidated to `/workspaces` (legacy `/searchspaces` · `/search-spaces` · `/search-space` all retired/404). Verified (ground-truth `git grep`, 2026-06-29): every residual `search_space`/`SEARCH_SPACE` is a documented carve-out — enum values (`ConnectionScope`/`ChatVisibility.SEARCH_SPACE`), the `'SEARCH_SPACE'` CHECK literal (now paired with `workspace_id`), Celery wire names (`delete_search_space_background`, `ai_sort_search_space`), OTel key `search_space.id`, and the `SEARCH_SPACE_FORBIDDEN` error code; `alembic/versions/` untouched except `168`+`170`. Suite: `3016 passed, 1 skipped`. As-built record in `02-rename-backend.md`. **Clients are intentionally broken** until the frontend/satellite umbrellas land (hard cutover).
- Rename models/schemas/services/routes/agents/tasks identifiers: `SearchSpace*` -> `Workspace*`, `search_space_id` -> `workspace_id`.
- Consolidate API to `/workspaces` and fix the `/searchspaces` vs `/search-spaces` inconsistency.
- High-touch files: `routes/search_spaces_routes.py`, `routes/rbac_routes.py`, `utils/rbac.py` (`check_search_space_access`), `schemas/search_space.py`, plus `search_space_id` threading through agents/Redis keys/storage paths (`documents/{id}/...`).
@ -168,8 +172,8 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
| Phase | Subplan file | Status |
|-------|--------------|--------|
| 1 | `01-rename-db.md` | drafted |
| 2 | `02-rename-backend.md` | drafted |
| 1 | `01-rename-db.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562); migration `170`) |
| 2 | `02-rename-backend.md` | **SHIPPED** (2026-06-27, PR [#1546](https://github.com/MODSetter/SurfSense/pull/1546)/[#1562](https://github.com/MODSetter/SurfSense/pull/1562)) |
| 3 | `03a-crawler-core.md` | drafted |
| 3 | `03b-proxy-expansion.md` | drafted |
| 3 | `03c-crawl-billing.md` | drafted |

View file

@ -3,7 +3,7 @@
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Sibling subplans: `03b-proxy-expansion.md`, `03c-crawl-billing.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md`, `03f-undetectability-testing.md` (manual scorecard).
> **Implementation note (applies to all Phase-3 plans).** Phase 3 lands **after** Phases 12, which rename `SearchSpace``Workspace` and `search_space_id``workspace_id` everywhere. Citations below use **today's** names so they stay greppable against current code; when implementing, the live code will already say `workspace_*` — map accordingly. Apply every edit by **symbol/grep** (e.g. `firecrawl_api_key`, `crawl_url`, `FIRECRAWL_API_KEY`), **not** by the absolute line numbers cited here — Phase 2's rename (and `03a`'s own Firecrawl removal) shift line numbers.
> **Implementation note (applies to all Phase-3 plans).** Phases 12 are **SHIPPED** (2026-06-27) — `SearchSpace``Workspace` and `search_space_id``workspace_id` are renamed everywhere, so the **live code already says `workspace_*`**. Where citations below use the **old** `search_space_*`/`SearchSpace` names (written pre-rename), substitute the `workspace_*` equivalent and **grep the new name** (grepping the old name now returns nothing). Apply every edit by **symbol/grep** (e.g. `firecrawl_api_key`, `crawl_url`, `FIRECRAWL_API_KEY`), **not** by the absolute line numbers cited here — the rename (and `03a`'s own Firecrawl removal) shifted them.
## Objective

View file

@ -3,7 +3,7 @@
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Depends on `03a-crawler-core.md` (Scrapling-only crawler). Siblings: `03c-crawl-billing.md`, `03e-stealth-hardening.md`, `03d-captcha-solving.md`.
> **Implementation note.** Same convention as `03a`: citations use **today's** names (`search_space_id`/`SearchSpace`) — map to `workspace_id`/`Workspace` post Phases 12. Crucially, the `webcrawler_connector.py` line numbers below (e.g. the three tiers at 287/329/359) and the `scrape_webpage.py` lines **predate `03a`'s Firecrawl removal + `crawl_url` refactor**, so they will have moved by the time `03b` is implemented. Locate code by **symbol/grep**, not absolute lines.
> **Implementation note.** Same convention as `03a`: Phases 12 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Crucially, the `webcrawler_connector.py` line numbers below (e.g. the three tiers at 287/329/359) and the `scrape_webpage.py` lines **predate `03a`'s Firecrawl removal + `crawl_url` refactor**, so they will have moved by the time `03b` is implemented. Locate code by **symbol/grep**, not absolute lines.
## Objective

View file

@ -34,7 +34,7 @@ It's wired into the **sibling connector indexers** exactly the way we need: `goo
- **Audit trail**: `record_token_usage(session, *, usage_type, search_space_id, user_id, cost_micros=…, call_details=…, message_id=None, …)` (`token_tracking_service.py:517568`) inserts a `TokenUsage` row (best-effort; caller commits). `TokenUsage.usage_type` is a `String(50)` indexed column explicitly intended for non-chat usage with `message_id` NULL (`db.py:10681116`, comment at `:10721084`: "indexing, image generation, podcasts — which keep message_id NULL").
- **Per-feature config knobs** are the convention: `MICROS_PER_PAGE` (`config/__init__.py:655`, default `1000`), `ETL_CREDIT_BILLING_ENABLED` (`:652653`, default FALSE), `QUOTA_DEFAULT_IMAGE_RESERVE_MICROS` (`:729`), etc. `maybe_trigger_auto_reload` (`auto_reload_service`) is the shared low-balance top-up nudge.
> **Implementation note.** Same convention as `03a`/`03b`: citations use **today's** names (`search_space_id`/`SearchSpace`) so they stay greppable against current code; post Phases 12 the live code says `workspace_id`/`Workspace` — map accordingly (e.g. the `record_token_usage(search_space_id=…)` kwarg becomes `workspace_id=…`). Locate code by **symbol/grep**, not the absolute line numbers cited here, since Phase 2's rename and `03a`'s `crawl_url`/`scrape_webpage` refactor shift them.
> **Implementation note.** Same convention as `03a`/`03b`: Phases 12 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name (e.g. the `record_token_usage(search_space_id=…)` kwarg is now `workspace_id=…`; `_resolve_agent_billing_for_search_space``_resolve_agent_billing_for_workspace`; `select SearchSpace.user_id``select Workspace.user_id`). Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename and `03a`'s `crawl_url`/`scrape_webpage` refactor shift them.
## Target design

View file

@ -3,7 +3,7 @@
> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Depends on `03a` (Scrapling StealthyFetcher tier + `FetchStrategy` seam + `CrawlOutcome`) and `03b` (app-wide proxy provider). Precedes `03d` (captcha) in the escalation order — hardening **avoids** challenges; captcha solving is the paid last resort for the ones we can't avoid. Touches `03c` only via the deferred paid-unblocker tier (its own billing, decided if/when built).
> **Implementation note.** Same convention as `03a``03d`: citations use **today's** names (`search_space_id`/`SearchSpace`) and predate `03a`'s `crawl_url` refactor; locate code by **symbol/grep**, not absolute lines. Scrapling references point at the on-disk (gitignored) `references/Scrapling/` checkout pinned to `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`).
> **Implementation note.** Same convention as `03a``03d`: Phases 12 are **SHIPPED** (live code says `workspace_id`/`Workspace` — substitute for any old `search_space_*`/`SearchSpace` citations and grep the new name); citations also predate `03a`'s `crawl_url` refactor; locate code by **symbol/grep**, not absolute lines. Scrapling references point at the on-disk (gitignored) `references/Scrapling/` checkout pinned to `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`).
## Objective

View file

@ -3,7 +3,7 @@
> Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`.
> Sibling: `04b-source-discovery.md` (web-search repurposing). Precondition: Phases 12 (rename) live.
> **Implementation note.** Citations use **today's** names (`search_space_id`/`SearchSpace`); post Phases 12 the live code says `workspace_id`/`Workspace` — map accordingly. Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename shifts them.
> **Implementation note.** Phases 12 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here, since the rename shifted them.
## Objective

View file

@ -3,7 +3,7 @@
> Part of **Phase 4 — Connector two-type restructure (backend)**. See `00-umbrella-plan.md`.
> Sibling: `04a-connector-category.md`. Best sequenced **after** `04a` (taxonomy in place). Precondition: Phases 12 (rename) live.
> **Implementation note.** Citations use **today's** names (`search_space_id`/`SearchSpace`); post Phases 12 the live code says `workspace_id`/`Workspace` — map accordingly, and locate code by **symbol/grep**, not the absolute line numbers cited.
> **Implementation note.** Phases 12 are **SHIPPED**, so the live code already says `workspace_id`/`Workspace` — substitute for the old `search_space_*`/`SearchSpace` names in citations below and grep the new name; locate code by **symbol/grep**, not the absolute line numbers cited.
## Objective

View file

@ -3,7 +3,7 @@
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 5).
> Precondition: Phases 12 (rename) live, Phase 4a (connector registry) live. Siblings ahead: `06-pipelines-exec.md` (run engine + scheduling), `07-upload-pipeline-kb.md` (uploads-as-pipeline).
> **Implementation note (post-rename).** This phase adds **brand-new** tables, created *after* the rename, so they use the canonical names natively: table `workspaces`, column `workspace_id`, ORM class `Workspace`. Existing code cited below still shows the **pre-rename** names (`searchspaces`/`search_space_id`/`SearchSpace`) because those citations were taken against today's tree — Phases 12 rename them; map accordingly. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phase 3/4 migrations shift them).
> **Implementation note (post-rename).** This phase adds **brand-new** tables, created *after* the rename, so they use the canonical names natively: table `workspaces`, column `workspace_id`, ORM class `Workspace`. Phases 12 are now **SHIPPED**, so the live code uses the `workspace_*` names. Existing code cited below that still shows the **pre-rename** names (`searchspaces`/`search_space_id`/`SearchSpace`) was captured pre-rename — substitute the `workspace_*` equivalent and grep the new name. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phase 3/4 migrations shift them).
## Objective
@ -125,7 +125,7 @@ Add the inverse `pipelines = relationship("Pipeline", back_populates=...)` on **
> **Ownership decision (review).** Earlier draft used connector-style `user_id NOT NULL CASCADE`. Corrected to `created_by_id … SET NULL` because pipelines are **workspace-shared** content (closer to `Folder`/`Automation` than to the per-user `SearchSourceConnector`): removing a member must not delete the workspace's pipelines or erase run-history audit. For *connector-backed* pipelines this also avoids surprising double-cascade (the connector's own `user_id CASCADE` would already drop them on member deletion via `connector_id`); for the Phase-7 Uploads pipeline (`connector_id` NULL) it's the only thing keeping the pipeline alive after its creator leaves.
> **FK target naming:** `workspaces` / `workspace_id` assume Phases 12 are merged (the table is `workspaces` by then). If sequencing slips and Phase 5 lands before the rename, the FK target is `searchspaces` and the column `search_space_id` — but per the umbrella the rename is strictly first, so author against `workspaces`.
> **FK target naming:** Phases 12 are **SHIPPED**, so the table is `workspaces` and the column `workspace_id` — author the FK against `workspaces` / `workspace_id`. (The earlier "if sequencing slips before the rename" caveat is now moot.)
### 3. `pipeline_runs` table (`db.py`)

View file

@ -3,7 +3,7 @@
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 6).
> Precondition: Phase 5 (`05-pipelines-model.md`) live — `pipelines` / `pipeline_runs` tables, schemas, Zero, CRUD + `/run` (currently enqueues a **stub** `run_pipeline(run_id)`). Depends on `03a-crawler-core.md` (`crawl_url` SUCCESS signal + `crawls_succeeded` counter) and `03c-crawl-billing.md` (`WebCrawlCreditService`). Sibling ahead: `07-upload-pipeline-kb.md` (uploads-as-pipeline).
> **Implementation note (post-rename).** Citations use **today's** identifiers (`search_space_id` / `SearchSpace` / `searchspaces`). Phases 12 rename them to `workspace_id` / `Workspace` / `workspaces`; map accordingly. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 35 shift them).
> **Implementation note (post-rename).** Phases 12 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace` / `workspaces`. Citations below that still show `search_space_id` / `SearchSpace` / `searchspaces` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated; e.g. the `get_pipeline_runs` tool's dep key is `workspace_id`, matching the renamed deps dict). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 35 shift them).
## Objective
@ -28,7 +28,7 @@ This is backend-only; the Pipelines UI is deferred to the frontend umbrella.
| Scheduler | New `pipeline_schedule_select` Beat task modeled on `automations/triggers/builtin/schedule/selector.py` (cron, `FOR UPDATE SKIP LOCKED`, self-heal of NULL `next_scheduled_at`), **not** the simpler connector `frequency_minutes` checker — because Phase 5 chose `schedule_cron`. Reuses the existing `croniter` util (`automations/triggers/builtin/schedule/cron.py`). |
| Schedule timezone | Cron needs a timezone (`compute_next_fire_at(cron, timezone, …)`). Phase 5's model has only `schedule_cron`. **Add `schedule_timezone VARCHAR NOT NULL DEFAULT 'UTC'` to `pipelines`** (small additive amendment to `05` — see "Required 05 amendment"). Matches how automations store cron **and** timezone (`schedule/selector.py:86-89`). |
| Concurrency | Per-pipeline Redis lock + (when connector-backed) the **existing connector lock** `utils/indexing_locks.py` so a pipeline run and any residual connector index can't double-crawl the same connector. |
| Chat context | A main-agent **tool** `get_pipeline_runs` (registry pattern, `main_agent/tools/registry.py:85-102`), opening its own `shielded_async_session()` (like `KnowledgeTreeMiddleware`) and scoping to the build-time `search_space_id`. Always-on middleware injection is **deferred** (token cost; tool is on-demand). |
| Chat context | A main-agent **tool** `get_pipeline_runs` (registry pattern, `main_agent/tools/registry.py:85-102`), opening its own `shielded_async_session()` (like `KnowledgeTreeMiddleware`) and scoping to the build-time `workspace_id`. Always-on middleware injection is **deferred** (token cost; tool is on-demand). |
| Code location | New `app/pipelines/` package: `engine.py` (`execute_pipeline_run`), `tasks.py` (Celery `run_pipeline` + `pipeline_schedule_select`), `scheduler.py` (the tick), `storage.py` (blob key). The Phase-5 stub `run_pipeline` is **replaced** by the real task here; the `/run` route + scheduler enqueue it. (ORM stays in `db.py` per Phase 5; only the *engine* is a package, mirroring `app/automations/`.) |
## Current state (cited)
@ -255,7 +255,7 @@ When a pipeline that **wraps a connector** is created or enabled (Phase 5 routes
A read-only tool registered in the main-agent registry (umbrella default = tool):
```python
def create_get_pipeline_runs_tool(*, search_space_id: int) -> BaseTool:
def create_get_pipeline_runs_tool(*, workspace_id: int) -> BaseTool:
@tool
async def get_pipeline_runs(pipeline_id: int | None = None, limit: int = 20) -> str:
"""Recent pipeline run history for THIS workspace: name, status, trigger,
@ -263,7 +263,7 @@ def create_get_pipeline_runs_tool(*, search_space_id: int) -> BaseTool:
async with shielded_async_session() as session: # knowledge_tree/middleware.py:199 pattern
q = (select(PipelineRun, Pipeline.name, Pipeline.schedule_cron)
.join(Pipeline, PipelineRun.pipeline_id == Pipeline.id)
.where(Pipeline.workspace_id == search_space_id))
.where(Pipeline.workspace_id == workspace_id))
if pipeline_id is not None:
q = q.where(Pipeline.id == pipeline_id)
rows = (await session.execute(
@ -272,9 +272,9 @@ def create_get_pipeline_runs_tool(*, search_space_id: int) -> BaseTool:
return get_pipeline_runs
```
- **Registration**: add `"get_pipeline_runs": (_build_get_pipeline_runs_tool, ("search_space_id",))` to `_MAIN_AGENT_TOOL_FACTORIES` (`registry.py:85-102`); add the name to `main_agent/tools/index`; add display metadata to `shared/tools/catalog`.
- **Workspace scope is structural** — the tool only ever sees its build-time `search_space_id`; no cross-workspace leakage (same guarantee `KnowledgeTreeMiddleware` relies on).
- **Anonymous / no-workspace turns**: exclude the tool when there's no `search_space_id` (anonymous chat has no workspace/pipelines), the same way `KnowledgeTreeMiddleware` is CLOUD-only (`knowledge_tree/middleware.py:130-131`). The registry factory requires `search_space_id`, so just don't enable it in the anonymous tool set.
- **Registration**: add `"get_pipeline_runs": (_build_get_pipeline_runs_tool, ("workspace_id",))` to `_MAIN_AGENT_TOOL_FACTORIES` (`registry.py:85-102`) — the dep key is `workspace_id`, matching the post-rename deps dict; add the name to `main_agent/tools/index`; add display metadata to `shared/tools/catalog`.
- **Workspace scope is structural** — the tool only ever sees its build-time `workspace_id`; no cross-workspace leakage (same guarantee `KnowledgeTreeMiddleware` relies on).
- **Anonymous / no-workspace turns**: exclude the tool when there's no `workspace_id` (anonymous chat has no workspace/pipelines), the same way `KnowledgeTreeMiddleware` is CLOUD-only (`knowledge_tree/middleware.py:130-131`). The registry factory requires `workspace_id`, so just don't enable it in the anonymous tool set.
- The tool reads `pipelines`/`pipeline_runs` directly (no Zero dependency), so it works server-side immediately even though client Zero sync is dormant until the frontend lands (Phase 5 note).
- **Deferred**: an always-on `<pipeline_activity>` middleware injection (à la `KnowledgeTreeMiddleware`) — adds per-turn token cost; revisit if the agent under-uses the tool.

View file

@ -3,7 +3,7 @@
> Part of the **CI Pivot MVP**. See `00-umbrella-plan.md` (Phase 7) — the final backend phase.
> Precondition: Phase 5 (`05-pipelines-model.md`) live (`pipelines`/`pipeline_runs` tables, `PipelineRunTrigger.UPLOAD`, `connector_id` nullable) and Phase 6 (`06-pipelines-exec.md`) live (run engine, which **explicitly fails `connector_id IS NULL` runs**). No sibling ahead — this closes the backend umbrella.
> **Implementation note (post-rename).** Citations use **today's** identifiers (`search_space_id` / `SearchSpace`). Phases 12 rename them to `workspace_id` / `Workspace`; map accordingly. Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 36 shift them).
> **Implementation note (post-rename).** Phases 12 are **SHIPPED**, so the live code already uses `workspace_id` / `Workspace`. Citations below that still show `search_space_id` / `SearchSpace` are pre-rename — substitute the `workspace_*` equivalent and grep the new name. **New code in this plan uses the canonical `workspace_*` names** (snippets below updated). Locate code by **symbol/grep**, not the absolute line numbers cited here (the rename + Phases 36 shift them).
## Objective
@ -99,7 +99,7 @@ After Phase 2 dispatch (`documents_routes.py:307-316`), before building the resp
```python
if files_to_process: # only when something was actually queued
await record_upload_run(session, workspace_id=search_space_id,
await record_upload_run(session, workspace_id=workspace_id,
created_by_id=str(user.id),
documents_indexed=len(files_to_process))
await session.commit()
@ -113,7 +113,7 @@ if files_to_process: # only when something was actu
Folder-upload dispatches one batch task, so record the run at request time with the file count (`documents_routes.py` before the `return` at `:1743`):
```python
await record_upload_run(session, workspace_id=search_space_id,
await record_upload_run(session, workspace_id=workspace_id,
created_by_id=str(user.id), documents_indexed=len(files))
await session.commit() # best-effort / logged, same as §3
```