feat: plans till phase 3

plans/backend/00-umbrella-plan.md

@@ -1,6 +1,6 @@
 # CI Pivot MVP — Umbrella Plan
 
-> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`content_research/pivot/`).
+> Master roadmap for the Competitive Intelligence pivot. Each phase becomes its own subplan saved in this folder (`plans/backend/`).
 
 This is the high-level roadmap. It is sequenced to match the agreed order: rename first, then connector restructure, then Pipelines.
 
@@ -35,7 +35,7 @@ flowchart TD
 - Full rename SearchSpace -> WorkSpace across DB, API, URLs, code, satellite apps.
 - Canonical names (proposed defaults): DB table `workspaces`, column `workspace_id`, RBAC tables `workspace_roles` / `workspace_memberships` / `workspace_invites`, API base `/workspaces` (consolidating today's `/searchspaces` vs `/search-spaces` split), URL segment `[workspace_id]`, settings folder `workspace-settings`, TS type `Workspace`.
 - Connectors get a `category` discriminator: `DATA_SOURCE` (Type 1) vs `MCP_TOOL` (Type 2). Type 1 keeps only file/cloud data sources (WebURL crawler, Google Drive, OneDrive, Dropbox, YouTube, file uploads) plus deferred platform connectors. Everything else moves to MCP. Artifacts stay in the existing `deliverables` agent system (not routed through MCP).
-- Web search APIs (Tavily, SearXNG, Linkup, Baidu, Serper) are repurposed as a SOURCE-DISCOVERY helper: they suggest URLs the user can add to the Universal WebURL Crawler when setting up pipelines (they are not a standalone connector type and do not index data).
+- Web search APIs (SearXNG, Linkup, Baidu) are repurposed as a SOURCE-DISCOVERY helper: they suggest URLs the user can add to the Universal WebURL Crawler when setting up pipelines (they are not a standalone connector type and do not index data). NOTE: Tavily and Serper are being REMOVED from the search infra and are not part of this set.
 - Obsidian and Circleback (push/webhook sources) are DISABLED for the MVP.
 - MCP-availability audit complete: BookStack (community MCP servers), Elasticsearch (official Elastic Agent Builder MCP), and Luma (community MCP servers) all have MCP available, so none are disabled — they migrate to Type-2.
 - `Pipeline` and `PipelineRun` are new first-class tables. A Pipeline references a connector + config + schedule + KB destination. File upload creates/uses a pipeline and registers a run; uploads always save to KB.
@@ -69,7 +69,16 @@ flowchart TD
 - Consolidate API to `/workspaces` and fix the `/searchspaces` vs `/search-spaces` inconsistency.
 - High-touch files: `routes/search_spaces_routes.py`, `routes/rbac_routes.py`, `utils/rbac.py` (`check_search_space_access`), `schemas/search_space.py`, plus `search_space_id` threading through agents/Redis keys/storage paths (`documents/{id}/...`).
 
-### Phase 3 — Connector two-type restructure (backend) [`subplan: 03-connector-two-type-backend.md`]
+### Phase 3 — WebURL Crawler & Crawl Billing (backend) [`subplans: 03a–03d`]
+
+The Universal WebURL Crawler is the flagship Type-1 data source (the moat). This phase hardens it on a single framework (Scrapling), generalizes proxy support, introduces pay-as-you-go crawl credits, and (deferred) adds opt-in captcha solving. It is broken into focused subplans:
+
+- **`03a-crawler-core.md`** — Standardize the fetch layer on Scrapling. **Remove Firecrawl entirely** (no other frameworks). Define crisp per-URL success/empty/failure semantics, keep Trafilatura extraction, and expose a single billable "successful crawl" signal (one unit per URL that yields usable content, regardless of how many internal fallback tiers ran).
+- **`03b-proxy-expansion.md`** — Add a BYO `CustomProxyProvider` (the only new provider — **no branded vendors**) alongside `anonymous_proxies`, selectable via a **single, app-wide** `Config.PROXY_PROVIDER`. Add bounded client-side rotation+retry via Scrapling's `ProxyRotator`/`is_proxy_error` **only** when the active provider is pool-backed (`CUSTOM_PROXY_URLS`); single-endpoint providers (incl. `anonymous_proxies`) stay the default and no-op the retry. **No per-connector/per-crawl selection** (one provider app-wide); a per-pipeline override is left as a no-op seam for Phase 5/6.
+- **`03c-crawl-billing.md`** — Charge crawl credits at **$1 / 1000 successful requests = 1000 micro-USD per successful crawl**, drawn from the existing credit wallet (`credit_micros_balance`), gated by a new `WEB_CRAWL_CREDIT_BILLING_ENABLED` flag (off for self-hosted). Two surfaces: **connector/pipeline crawls** billed to the **workspace owner** via a dedicated `WebCrawlCreditService` (mirrors `EtlCreditService`'s gate → `check_credits` → `charge_credits`, **not** `billable_call`); **chat scrapes** fold their crawl cost into the chat turn's existing bill (turn accumulator). No DB migration (uses the existing free-form `web_crawl` usage_type).
+- **`03d-captcha-solving.md`** *(DEFERRED — sequenced last, non-MVP-blocking)* — Covers the captcha types Scrapling does **not** (reCAPTCHA v2/v3, hCaptcha, image) via `captchatools`. `captchatools` is **itself** the provider registry (`new_harvester(solving_site=…)` across capmonster/2captcha/anticaptcha/capsolver/captchaai), so we do **not** rebuild a provider hierarchy — our layer is thin: config resolution + a StealthyFetcher `page_action` that detects the sitekey, harvests a token, and injects it. Scrapling already handles Cloudflare Turnstile (`03a`). Flags the **billing asymmetry** (solvers charge per *attempt*, `03c` bills per *success*) for resolution at build time. Requires a paid solver account.
+
+### Phase 4 — Connector two-type restructure (backend) [`subplan: 04-connector-two-type-backend.md`]
 
 - Add `category` (`DATA_SOURCE` / `MCP_TOOL`) to `SearchSourceConnector` (replaces ad hoc `is_indexable`): `db.py` enum/model, schema, Alembic migration + data backfill that tags existing rows.
 - Adjust backend routing/indexing so only Type-1 keeps the `/index` + Celery path; Type-2 resolves via MCP tools.
@@ -94,26 +103,27 @@ flowchart TD
 
 **Web search APIs — repurposed (not a connector type):**
 
-- Tavily, SearXNG, Linkup, Baidu, Serper become a source-discovery helper for the Universal WebURL Crawler: given a topic/competitor, suggest candidate URLs the user can add to a pipeline. Reuses the existing web-search services; backend endpoint here, UX deferred to frontend umbrella.
+- SearXNG, Linkup, Baidu become a source-discovery helper for the Universal WebURL Crawler: given a topic/competitor, suggest candidate URLs the user can add to a pipeline. Reuses the existing web-search services; backend endpoint here, UX deferred to frontend umbrella. (Tavily and Serper are removed from the search infra — see resolved log.)
 
 **Disabled for MVP:**
 
 - Obsidian (plugin push) and Circleback (meeting webhook) — disabled for the pivot MVP.
 
-### Phase 4 — Pipelines data model [`subplan: 04-pipelines-model.md`]
+### Phase 5 — Pipelines data model [`subplan: 05-pipelines-model.md`]
 
 - New tables: `pipelines` (workspace_id, user_id, connector_id, name, config JSON, schedule/cron, `save_to_kb` bool, `destination_folder_id` nullable, enabled, next_scheduled_at) and `pipeline_runs` (pipeline_id, status, trigger = manual/scheduled/upload, timestamps, doc counts, error, optional raw-result blob ref).
 - Models + Pydantic schemas + Alembic migration + backend Zero publication entry.
 - Pipelines API routes: CRUD + manual run trigger + list runs.
 
-### Phase 5 — Pipeline execution + scheduling [`subplan: 05-pipelines-exec.md`]
+### Phase 6 — Pipeline execution + scheduling [`subplan: 06-pipelines-exec.md`]
 
 - Run engine: pipeline run -> invoke connector fetch (WebURL crawler for MVP) -> if `save_to_kb`, route through `IndexingPipelineService` into the destination folder -> write `PipelineRun` record.
+- **Crawl billing wiring (carry-over from `03c`):** `03c` meters crawls inside `webcrawler_indexer`. A pipeline run that crawls but has `save_to_kb=false` must NOT bypass billing — wire the pipeline fetch through the same `WebCrawlCreditService` (pre-check + charge on `crawls_succeeded`) regardless of the KB-save branch, ideally recording `charged_micros` on the `PipelineRun` for idempotency. Otherwise non-KB pipeline crawls are free by accident.
 - Scheduling: reuse the Celery Beat meta-scheduler pattern (`schedule_checker_task.py`, `periodic_scheduler.py`) for cron + manual triggers.
 - When `save_to_kb` is off, persist the raw fetch result on the run (blob via `file_storage`) so it is retrievable without indexing.
 - Chat agent context: expose pipeline run history to the `multi_agent_chat` agent (read-only) — via a tool (e.g. `list_pipelines` / `get_pipeline_runs`) and/or a context middleware injection (similar to `KnowledgeTreeMiddleware`). Scope strictly to the active workspace. Gives the agent awareness of recent runs, statuses, schedules, and last-fetched timestamps.
 
-### Phase 6 — File upload as a pipeline + KB-save-secondary [`subplan: 06-upload-pipeline-kb.md`]
+### Phase 7 — File upload as a pipeline + KB-save-secondary [`subplan: 07-upload-pipeline-kb.md`]
 
 - Wire file upload (`documents_routes.py` fileupload flow) to create/use an "Uploads" pipeline and register a `PipelineRun`; uploads always `save_to_kb = true`.
 - Generalize KB saving to be opt-in for non-upload pipelines via `save_to_kb` + destination folder.
@@ -136,10 +146,16 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
 
 ## Resolved decisions log
 
-- Web search APIs (Tavily/SearXNG/Linkup/Baidu/Serper): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines); not a standalone connector type.
+- Web search APIs (SearXNG/Linkup/Baidu): repurposed as source-discovery helper for the WebURL Crawler (suggest URLs for pipelines); not a standalone connector type.
+- Tavily and Serper: REMOVED from the search infra. They are dropped as search providers entirely (not repurposed). Phase 4's source-discovery endpoint must build only on the remaining providers (SearXNG, Linkup, Baidu).
 - Obsidian + Circleback: disabled for MVP.
 - MCP-availability audit: BookStack, Elasticsearch, Luma all have MCP available -> migrate to Type-2, none disabled.
 - Rename transition policy: HARD CUTOVER of the external API (paths + JSON field names) in Phase 2 — no backward-compat aliases. Rationale: the frontend is (re)built against the corrected backend later, so there is no old client to keep alive; backend correctness is verified via the test suite + OpenAPI rather than the existing UI.
+- WebURL Crawler framework: STANDARDIZE on Scrapling; **remove Firecrawl entirely** (no other scraping frameworks now or planned). Scrapling's `StealthyFetcher` handles Cloudflare; captcha-tools (deferred) covers the rest.
+- Crawl billing: reuse the existing credit wallet (`credit_micros_balance`) with a new `web_crawl` usage_type. Price: **$1 / 1000 successful requests** (1000 micro-USD per success). Connector/pipeline crawls bill the **workspace owner**; chat scrapes fold their crawl cost into the already-billed chat turn. Gated by `WEB_CRAWL_CREDIT_BILLING_ENABLED` (off for self-hosted); no DB migration required.
+- Billable unit: one unit per URL that returns usable extracted content, regardless of how many internal fallback tiers were attempted (not per HTTP fetch, not per URL-processed).
+- Captcha solving (captcha-tools): DEFERRED to the last Phase-3 subplan (`03d`); non-MVP-blocking.
+- Roadmap: WebURL Crawler & Crawl Billing inserted as the new Phase 3; connector two-type → Phase 4; pipelines → Phases 5/6/7.
 
 ## Subplan index (backend)
 
@@ -147,9 +163,13 @@ These are recorded for continuity but are NOT planned in this umbrella. They sta
 |-------|--------------|--------|
 | 1 | `01-rename-db.md` | drafted |
 | 2 | `02-rename-backend.md` | drafted |
-| 3 | `03-connector-two-type-backend.md` | not started |
-| 4 | `04-pipelines-model.md` | not started |
-| 5 | `05-pipelines-exec.md` | not started |
-| 6 | `06-upload-pipeline-kb.md` | not started |
+| 3 | `03a-crawler-core.md` | drafted |
+| 3 | `03b-proxy-expansion.md` | drafted |
+| 3 | `03c-crawl-billing.md` | drafted |
+| 3 | `03d-captcha-solving.md` | drafted (deferred — last) |
+| 4 | `04-connector-two-type-backend.md` | not started |
+| 5 | `05-pipelines-model.md` | not started |
+| 6 | `06-pipelines-exec.md` | not started |
+| 7 | `07-upload-pipeline-kb.md` | not started |
 
 Frontend & client subplans will be added under a separate umbrella later (see "Deferred — Frontend & client phases").

plans/backend/02-rename-backend.md

@@ -93,7 +93,7 @@ These do not move with a symbol rename; each is decided here.
 5. OpenTelemetry attribute `search_space.id` + metric label ([observability/otel.py](surfsense_backend/app/observability/otel.py) 263-264/305-306, [observability/metrics.py](surfsense_backend/app/observability/metrics.py) 537-542). DECISION: KEEP the OTel/metric KEY `search_space.id` for now (dashboards/alerts depend on it); rename only the Python params. Schedule the observability-key rename as a separate, announced change. (Carve-out to avoid silently breaking alerting.)
 6. Notification dedup/operation IDs embedding `{search_space_id}` (e.g. `doc_..._{search_space_id}_...`, `insufficient_credits_{search_space_id}_...`) and frontend deep-link strings like `/dashboard/{search_space_id}/buy-more`. DECISION: the ID is a numeric value, not the literal word — leave format strings as-is functionally; the embedded VALUE is unchanged. The `/dashboard/{id}/...` deep link points at a FRONTEND route still named `[search_space_id]` (deferred umbrella) — KEEP it until the frontend segment renames, else links 404.
 7. Storage path builders — `documents/{search_space_id}/...` ([file_storage/keys.py](surfsense_backend/app/file_storage/keys.py) 20-26) and `podcasts/{search_space_id}/...` ([podcasts/storage.py](surfsense_backend/app/podcasts/storage.py) 22-25). The path segment is the numeric ID; the literal word `search_space` is NOT in stored object keys. DECISION: rename the param only; NO blob migration needed; existing objects keep resolving.
-8. `SearchSourceConnector` — contains "search" but is the connectors table, a different concept. OUT OF SCOPE (its rename is Phase 3's `category` work, not this rename).
+8. `SearchSourceConnector` — contains "search" but is the connectors table, a different concept (the word "search" here is unrelated to `SearchSpace`). OUT OF SCOPE: this rename does NOT touch it, and Phase 4 does not rename the class either — Phase 4 only adds the `category` discriminator column (and supersedes the ad hoc `is_indexable` boolean, `db.py:1868`). Leave `SearchSourceConnector` as-is.
 9. Historical Alembic migrations (`surfsense_backend/alembic/versions/*`) — ~20 files embed `searchspaces` / `search_space_id` as raw-SQL string literals (e.g. `23_associate_connectors_with_search_spaces.py`, `41_backfill_rbac_for_existing_searchspaces.py`, `40_move_llm_preferences_to_searchspace.py`). DECISION: NEVER rewrite these. They are an immutable replay log that intentionally references the schema as it existed at that revision; rewriting them corrupts history and breaks a clean `alembic upgrade` from zero. Verified safe: no migration imports the ORM classes (only `from app.db import Base` in `env.py` / `0_initial_schema.py`), so the class rename does not touch them. Phase 1's migration 166 is the single transition point; migrations >166 use the new names. The scripted rename scope is `app/` + `tests/` ONLY.
 10. LangGraph persisted state channel key `search_space_id` — `input_state = {..., "search_space_id": search_space_id, ...}` ([tasks/chat/streaming/flows/new_chat/input_state.py](surfsense_backend/app/tasks/chat/streaming/flows/new_chat/input_state.py) 131) is a CHECKPOINTED state channel. The resume_chat flow reads persisted state (`agent.aget_state({"configurable": {"thread_id": ...}})`, [flows/resume_chat/resume_routing.py](surfsense_backend/app/tasks/chat/streaming/flows/resume_chat/resume_routing.py) 50) and HITL interrupts (`surfsense_resume_value`, `HumanReview`) can sit pending across a deploy. DECISION: rename the channel key to `workspace_id` for consistency, and ACCEPT that any chat thread paused at a HITL interrupt BEFORE the cutover must be restarted after (the old checkpoint carries `search_space_id`, the new graph reads `workspace_id`). Operationally: drain/resolve pending interrupts before deploy if practical. (Alternative — keep the channel key as a carve-out — rejected: leaves a lone `search_space_id` in otherwise-renamed agent state for a transient, conversation-scoped value. The `configurable` keys are `thread_id` / `surfsense_resume_value`, not `search_space_id`, so only this state channel is affected.)
 11. User-facing default literal — `users.py` seeds the default workspace `name="My Search Space"` (line 158, persisted + shown to users) and logs "Created default search space" (207). DECISION: rename the default name to "My Workspace" (backend-created seed value, not caught by a symbol rename); log strings are cosmetic. Also update the Redis-key assertion in `tests/unit/services/test_ai_sort_task_dedupe.py:14` when literal 3 is renamed.
@@ -129,5 +129,5 @@ These do not move with a symbol rename; each is decided here.
 
 - Frontend route segment `[search_space_id] -> [workspace_id]`, `search-space-settings/`, TS types, atoms, i18n, frontend Zero schema — deferred frontend umbrella.
 - Satellite apps (desktop, Obsidian, browser extension, evals) + docs — deferred.
-- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 3 ([03-connector-two-type-backend.md](03-connector-two-type-backend.md)).
+- Connector `category` discriminator and `SearchSourceConnector` handling — Phase 4 ([04-connector-two-type-backend.md](04-connector-two-type-backend.md)).
 - Enum VALUE migration and observability-key rename — deliberately deferred announced changes.

plans/backend/03a-crawler-core.md

@@ -0,0 +1,123 @@
+# Phase 3a — WebURL Crawler core (Scrapling-only) + success semantics
+
+> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
+> Sibling subplans: `03b-proxy-expansion.md`, `03c-crawl-billing.md`, `03d-captcha-solving.md` (deferred).
+
+> **Implementation note (applies to all Phase-3 plans).** Phase 3 lands **after** Phases 1–2, which rename `SearchSpace`→`Workspace` and `search_space_id`→`workspace_id` everywhere. Citations below use **today's** names so they stay greppable against current code; when implementing, the live code will already say `workspace_*` — map accordingly. Apply every edit by **symbol/grep** (e.g. `firecrawl_api_key`, `crawl_url`, `FIRECRAWL_API_KEY`), **not** by the absolute line numbers cited here — Phase 2's rename (and `03a`'s own Firecrawl removal) shift line numbers.
+
+## Objective
+
+Make the Universal WebURL Crawler a **single-framework (Scrapling) component** with deterministic per-URL outcome semantics and a clean, explicit **"successful crawl" signal** that Phase 3c bills on.
+
+Two hard requirements from the decisions log:
+
+1. **Remove Firecrawl entirely.** No other scraping framework now or planned. Scrapling's `StealthyFetcher` can bypass Cloudflare Turnstile when invoked with `solve_cloudflare=True` (see tier design below); captcha-tools (`03d`, deferred) covers the rest.
+2. **One billable unit = one URL that yields usable extracted content**, regardless of how many internal fallback tiers ran. `03a` must expose that signal; `03c` meters it.
+
+This subplan does NOT touch proxy rotation (→ `03b`), credit metering (→ `03c`), or captcha (→ `03d`).
+
+## Current state (cited)
+
+### The crawler
+
+`app/connectors/webcrawler_connector.py` — `WebCrawlerConnector.crawl_url()` is a 4-tier fallback ladder:
+
+1. **Firecrawl** (premium, if `firecrawl_api_key` set) — `_crawl_with_firecrawl()` (lines 89–100, 223–272), imports `from firecrawl import AsyncFirecrawlApp` (line 22).
+2. Scrapling `AsyncFetcher` (static HTTP, curl_cffi) — `_crawl_with_async_fetcher()` (lines 274–310).
+3. Scrapling `DynamicFetcher` (browser, run in a thread) — `_crawl_with_dynamic()` (lines 312–339).
+4. Scrapling `StealthyFetcher` (Camoufox anti-bot, run in a thread) — `_crawl_with_stealthy()` (lines 341–369).
+
+Extraction is Trafilatura HTML→markdown in `_build_result()` (lines 371–469). Every Scrapling tier passes `proxy=get_proxy_url()` (lines 287, 329, 359). `crawl_url()` returns a tuple `(result_dict | None, error | None)`; `result_dict` has `content` / `metadata` / `crawler_type`.
+
+### The two call sites
+
+- **Type-1 indexer (billable path):** `app/tasks/connector_indexers/webcrawler_indexer.py` reads `FIRECRAWL_API_KEY` from connector config (line 115), builds `WebCrawlerConnector(firecrawl_api_key=api_key)` (line 139), and calls `crawler.crawl_url(url)` per URL (line 297). It already tracks `documents_indexed` / `documents_updated` / `documents_skipped` / `documents_failed` / `duplicate_content_count` (lines 167–171), but **none of these is a clean "crawl succeeded" count** — duplicates and unchanged docs are bucketed as skipped even though a successful fetch happened.
+- **Chat scrape tool (ad-hoc):** `main_agent/tools/scrape_webpage.py` (line 232) and `subagents/builtins/research/tools/scrape_webpage.py` (line 226) build `WebCrawlerConnector(firecrawl_api_key=...)` and call `crawl_url(url, formats=["markdown"])`. The `formats` arg is Firecrawl-only (declared at `crawl_url` signature line 59; docstring line 72: "only for Firecrawl").
+
+### Firecrawl threading (the cross-cutting surface)
+
+Firecrawl's API key is plumbed end-to-end and must be removed everywhere:
+
+| Layer | File:line | What to remove |
+|-------|-----------|----------------|
+| Crawler | `app/connectors/webcrawler_connector.py` | `firecrawl` import (22), `firecrawl_api_key`/`use_firecrawl` ctor (36–46), `set_api_key()` (48–56), tier-1 block (89–100), `_crawl_with_firecrawl()` (223–272), `formats` param + Firecrawl mentions in docstrings |
+| Dependency | `pyproject.toml:45` | `"firecrawl-py>=4.9.0"` (+ regenerate `uv.lock`). Scrapling stays (`pyproject.toml:91` → `"scrapling[fetchers]>=0.4.9"`) |
+| Connector config | `app/utils/validators.py` | `FIRECRAWL_API_KEY` from `WEBCRAWLER_CONNECTOR.optional`/`validators` (573–580); delete `validate_firecrawl_api_key_format()` (472–478) |
+| Indexer | `app/tasks/connector_indexers/webcrawler_indexer.py` | `api_key = connector.config.get("FIRECRAWL_API_KEY")` (115), `use_firecrawl` log field (135), pass-through to ctor (139) |
+| Chat setup | `app/tasks/chat/streaming/flows/shared/pre_stream_setup.py` | `setup_connector_and_firecrawl()` returns `firecrawl_api_key` (1–30) — collapse to connector-service-only |
+| Chat orchestrators | `new_chat/orchestrator.py` (86, 378, 412, 665), `resume_chat/orchestrator.py` (65, 317, 347, 483) | `firecrawl_api_key` threading |
+| Automations | `automations/actions/builtin/agent_task/dependencies.py` (19, 34, 85, 103), `.../invoke.py` (174) | `firecrawl_api_key` dep field |
+| Main agent | `main_agent/runtime/factory.py` (71, 142), `main_agent/tools/registry.py` (36) | `firecrawl_api_key` param/wiring |
+| Scrape tools | `main_agent/tools/scrape_webpage.py` (170, 232), `research/tools/scrape_webpage.py` (164, 226), `research/tools/index.py` (28) | `firecrawl_api_key` factory arg + ctor arg |
+| Tests | `tests/unit/automations/actions/builtin/agent_task/test_dependencies.py` (44, 81) | `firecrawl_api_key == "fc-key"` assertion + fake |
+
+> **Not a global env var.** `FIRECRAWL_API_KEY` is **not** in `.env.example` and there is no `Config.FIRECRAWL_API_KEY`. It lives only inside each WebCrawler connector's `config` JSON — the validator reads it from that dict (`validators.py:474`). So removal is code-only; **no env/docs change**. Existing `WEBCRAWLER_CONNECTOR` rows may still carry a now-dead `FIRECRAWL_API_KEY` key in `config` — harmless (it's simply ignored), optionally scrubbed by a tiny data migration if we want clean rows.
+>
+> **Signature change, not just deletions.** Removing `firecrawl_api_key` mutates the `RuntimeDeps`-style dataclass (`agent_task/dependencies.py:34`) and the agent runtime factory/tool factories — a coordinated signature change across the chat + automations call graph, so land it atomically.
+
+### Runtime/deps already in place
+
+- `Dockerfile:112–115` runs `RUN scrapling install` to fetch patchright Chromium + Camoufox; the `scrapling[fetchers]` extra pulls playwright/patchright. **No new install step needed** once Firecrawl is gone.
+- Proxy is read via `app/utils/proxy/get_proxy_url()` (`__init__.py:13`), backed by the `PROXY_PROVIDER` registry (`config/__init__.py:983`). `03a` leaves this single-URL model untouched (rotation is `03b`).
+
+## Target design
+
+### Tier ladder (Scrapling-only)
+
+`crawl_url()` becomes a 3-tier ladder, preserving the existing thread-offload + `NotImplementedError` handling for the browser tiers (Windows `SelectorEventLoop` cannot spawn subprocesses — lines 134–141, 161–168):
+
+1. `AsyncFetcher.get(...)` — fast static HTTP.
+2. `DynamicFetcher.fetch(...)` — full browser (via `asyncio.to_thread`).
+3. `StealthyFetcher.fetch(...)` — Camoufox anti-bot, last resort. Enable Cloudflare solving here by passing **`solve_cloudflare=True`** — a documented `StealthyFetcher.fetch` kwarg ("Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response", `references/Scrapling/scrapling/fetchers/stealth_chrome.py:38`; it's a `StealthSession` TypedDict key passed via `**kwargs`). The current stealthy call (connector lines 354–360) passes `headless`/`network_idle`/`block_ads`/`proxy` but **not** `solve_cloudflare`, so this is a real behavior add. (Note: `solve_cloudflare` runs the full browser challenge loop, so it's correctly scoped to the last-resort tier only.)
+
+Trafilatura extraction (`_build_result`) and `format_to_structured_document()` are unchanged.
+
+### Explicit outcome model
+
+Replace the implicit `(dict|None, str|None)` contract with an explicit outcome so callers (indexer, chat tool, and `03c` metering) agree on what "success" means. Proposed:
+
+```python
+class CrawlOutcomeStatus(str, Enum):
+    SUCCESS = "success"   # a tier returned usable extracted content
+    EMPTY   = "empty"     # fetched, but no usable content after ALL tiers
+    FAILED  = "failed"    # invalid URL or every tier errored
+```
+
+`crawl_url()` returns a small dataclass `CrawlOutcome(status, result, error, tier)` (or, to minimize churn, keep the tuple and add a third element). Either way the **billable success predicate is single-sourced**: `status == SUCCESS`.
+
+| Outcome | When | Billable (`03c`)? | Document status (indexer) |
+|---------|------|-------------------|---------------------------|
+| `SUCCESS` | a tier extracted usable content (`_build_result` returned a dict) | **Yes — 1 unit** | `ready` (or unchanged/duplicate, see note) |
+| `EMPTY` | every tier was reached but none produced usable extracted content (the static tier also treats HTTP ≥ 400 as a miss and falls through — connector lines 292–301; the browser tiers attempt extraction regardless of status) | No | `failed("No content extracted")` |
+| `FAILED` | invalid URL, or all tiers raised | No | `failed(<error>)` |
+
+**Billing-policy note for `03c`:** success is the *crawl* succeeding (we fetched + extracted), independent of downstream KB dedupe. The indexer currently marks unchanged content as `skipped` (`webcrawler_indexer.py:341–347`) and cross-connector duplicates as `failed` (`:350–369`) — those still represent a **successful crawl** and should bill. `03c` must count `CrawlOutcomeStatus.SUCCESS`, not `documents_indexed`. Flagging here; final call lives in `03c`.
+
+### Success counter for the indexer
+
+Add an explicit `crawls_succeeded` counter in `index_crawled_urls()` incremented whenever `crawl_url` returns `SUCCESS` (right after line 297's call, before the dedupe/unchanged branches), and surface it in the task-success metadata (lines 455–466) and return value. This is the hand-off point `03c` meters against.
+
+### Chat scrape tool
+
+Drop the Firecrawl-only `formats=["markdown"]` arg (markdown is already the Trafilatura default). The tool keeps returning its asset dict; map the new outcome onto the existing `error` / `content` shape (lines 235–269) with no behavioral change for the agent.
+
+## Work items
+
+1. **Rip out Firecrawl** across the surface table above (crawler, dep, validators, indexer, chat plumbing, automations, tests) — code-only, no env/docs change.
+2. **Refactor `crawl_url`** to the 3-tier Scrapling ladder + explicit `CrawlOutcome`; enable `solve_cloudflare` on the stealthy tier.
+3. **Add `crawls_succeeded`** counting in `webcrawler_indexer.py` + expose in result/metadata.
+4. **Update both `scrape_webpage` tools** to drop `firecrawl_api_key` + `formats`.
+5. **Tests:** unit tests for `crawl_url` outcomes (mock Scrapling fetchers → SUCCESS/EMPTY/FAILED); update `test_dependencies.py`; assert the indexer's `crawls_succeeded` count.
+
+## Risks / trade-offs
+
+- **Loss of a managed fallback.** Firecrawl was a hosted last resort for hostile anti-bot sites. Mitigation: `StealthyFetcher` + Cloudflare solving now, `03b` proxy rotation, `03d` captcha solving. Acceptable per the decisions log (single-framework intent).
+- **Browser tiers on dev/Windows.** `DynamicFetcher`/`StealthyFetcher` need subprocess support; the existing `to_thread` + `NotImplementedError` guards (lines 134–141, 161–168) are preserved so static-only crawling still works in `uvicorn --reload`.
+- **`uv.lock` churn.** Removing `firecrawl-py` requires a lockfile regen + image rebuild; no new runtime deps are added.
+
+## Out of scope (hand-offs)
+
+- Proxy provider expansion + rotation → `03b`.
+- Crawl credit metering on `CrawlOutcomeStatus.SUCCESS` → `03c`.
+- reCAPTCHA/hCaptcha solving via captcha-tools → `03d` (deferred). Cloudflare Turnstile stays in-framework (Scrapling).
+- Whether ad-hoc **chat** scrapes are billed (vs only pipeline crawls) → decided in `03c`.

plans/backend/03b-proxy-expansion.md

@@ -0,0 +1,126 @@
+# Phase 3b — Proxy provider expansion + rotation
+
+> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
+> Depends on `03a-crawler-core.md` (Scrapling-only crawler). Siblings: `03c-crawl-billing.md`, `03d-captcha-solving.md` (deferred).
+
+> **Implementation note.** Same convention as `03a`: citations use **today's** names (`search_space_id`/`SearchSpace`) — map to `workspace_id`/`Workspace` post Phases 1–2. Crucially, the `webcrawler_connector.py` line numbers below (e.g. the three tiers at 287/329/359) and the `scrape_webpage.py` lines **predate `03a`'s Firecrawl removal + `crawl_url` refactor**, so they will have moved by the time `03b` is implemented. Locate code by **symbol/grep**, not absolute lines.
+
+## Objective
+
+Let the WebURL Crawler (and every other proxy consumer) run behind **either** the existing `anonymous_proxies` gateway **or** a new BYO `CustomProxyProvider`, chosen by a **single, app-wide** `Config.PROXY_PROVIDER`. When the active provider is backed by a **pool of endpoints**, rotate across them client-side with a bounded retry. **No branded vendors and no per-connector/per-crawl provider divergence** (resolved decisions): the whole app uses one provider.
+
+The provider abstraction already exists and is clean; the real gaps are just (1) only `anonymous_proxies` is registered (no BYO option), and (2) there is **no client-side rotation/retry** when a provider exposes multiple endpoints. The "process-global, env-only" shape is exactly what we want for a single-provider app, so we keep it.
+
+## Current state (cited)
+
+### A clean but process-global abstraction
+
+`app/utils/proxy/` is already a "subclass + register" provider package:
+
+- `base.py` — `ProxyProvider` ABC: `get_proxy_url()` (canonical `http://user:pass@host:port` for Scrapling/curl_cffi), `get_playwright_proxy()`, `get_requests_proxies()` (built from the URL by default).
+- `registry.py` — `_PROVIDERS` dict keyed by provider `name`; `get_active_provider()` resolves `Config.PROXY_PROVIDER` and **caches a single instance process-wide** (`_active_provider`, lines 23, 26–44). Only `AnonymousProxiesProvider` is registered (lines 17–19).
+- `__init__.py` — zero-arg module helpers `get_proxy_url()` / `get_playwright_proxy()` / `get_requests_proxies()` + a `get_residential_proxy_url()` back-compat alias, all delegating to `get_active_provider()`.
+- `providers/anonymous_proxies.py` — the one vendor. Note it's a **server-side rotating gateway**: host is `rotating.dnsproxifier.com:PORT` and the rotation `type`/location are encoded into a base64 "password" dict (lines 23–51). So per-request IP rotation already happens **upstream**; the client sends one static endpoint.
+- `app/utils/proxy_config.py` — a thin back-compat shim re-exporting the package (kept for old import paths).
+
+Config knobs: `Config.PROXY_PROVIDER` (default `"anonymous_proxies"`) + `RESIDENTIAL_PROXY_{USERNAME,PASSWORD,HOSTNAME,LOCATION,TYPE}` (`config/__init__.py:983–992`; documented commented-out in `.env.example:300–309`).
+
+### Who consumes the proxy (blast radius)
+
+The zero-arg getters are used in **more than just the crawler**, so any signature change must stay backward-compatible:
+
+| Consumer | File:line | Uses |
+|----------|-----------|------|
+| WebURL crawler (all 3 Scrapling tiers) | `connectors/webcrawler_connector.py` (287, 329, 359) | `get_proxy_url()` |
+| YouTube transcript route | `routes/youtube_routes.py` (78, 119) | `get_proxy_url()` |
+| YouTube processor (indexing) | `tasks/document_processors/youtube_processor.py` (223, 229) | `get_requests_proxies()`, `get_proxy_url()` |
+| Chat scrape tools (YouTube branch) | `main_agent/tools/scrape_webpage.py` (80, 93), `research/tools/scrape_webpage.py` (74, 87) | `get_requests_proxies()`, `get_proxy_url()` |
+
+**Implication:** the whole app shares **one** global provider, so every consumer keeps calling the zero-arg getters **unchanged**. The only behavior change is (a) *which* provider class those getters resolve to (selected by `Config.PROXY_PROVIDER`) and (b) optional client-side rotation when that provider is pool-backed. No caller passes a key; no signatures change.
+
+**Shape note (verified):** `get_playwright_proxy()` has **no consumers** anywhere — only the ABC/impl/exports define it. All three crawler tiers and YouTube's `AsyncFetcher.get` consume the **string** form (`get_proxy_url`); browser fetchers accept a string proxy (`references/Scrapling/.../stealth_chrome.py:47` — "it can be a string or a dictionary"). The dict form (`get_requests_proxies`) is consumed only by YouTube-transcript fetches (`youtube_processor.py:223`, plus the chat tools' YouTube branch `main_agent/tools/scrape_webpage.py:80` and `research/tools/scrape_webpage.py:74`) — none of which are crawler tiers. **Conclusion:** rotation only needs the **string** shape; the playwright dict can be ignored (or left as-is on providers) rather than threaded through rotation.
+
+### Scrapling's rotation primitive
+
+`ProxyRotator(proxies: list, strategy=cyclic_rotation)` is a thread-safe cycler over a **static list** of proxy URLs/dicts (`get_proxy()`, `proxy_rotation.py:88–92`) plus `is_proxy_error(exc)` (`:27–30`) that matches proxy failure strings (`net::err_proxy`, `connection refused`, …). It does **not** manage credentials or sessions — it just hands out the next endpoint. So it's only useful when we have a **pool of distinct endpoints**; it adds nothing for a single server-side-rotating gateway like `anonymous_proxies`.
+
+Both are **publicly importable**: `from scrapling.engines.toolbelt import ProxyRotator, is_proxy_error` (re-exported in `engines/toolbelt/__init__.py:1–3`). The thread-safe `Lock` (`:71,90`) makes `ProxyRotator` safe to call from the browser tiers that `03a` runs via `asyncio.to_thread`. Version is guaranteed: the pinned floor is `scrapling[fetchers]>=0.4.9` (`pyproject.toml:91`) and these APIs exist as of `0.4.9` (`scrapling/__init__.py:2`).
+
+## Selection scope (resolved)
+
+The user expects a **single proxy provider across the entire app**, so 03b's selection scope is simply the **global, env-configured provider** — no per-connector or per-crawl override is built now. That is both the fastest path and, with one thin seam, the most scalable later:
+
+- All resolution stays behind today's `get_active_provider()` (env-selected, process-cached). One provider, app-wide.
+- "Scalable in future": if per-pipeline proxying is ever wanted (Phase 5+), it layers on as an **optional argument** to a resolver (`resolve_proxy(override=pipeline.config)`) with **zero** changes to existing call sites — but it is explicitly **not** implemented in 03b (YAGNI for a single-provider app).
+
+## Target design
+
+### 1. Registry: register the BYO provider; keep single-provider selection
+
+The existing `get_active_provider()` (env-selected via `Config.PROXY_PROVIDER`, cached process-wide in `_active_provider` — `registry.py:23,26–44`) is exactly the single-provider model we want — keep it. The only change is to **register `CustomProxyProvider`** so `PROXY_PROVIDER` can select it:
+
+- `_PROVIDERS["anonymous_proxies"] = AnonymousProxiesProvider` (existing) + `_PROVIDERS["custom"] = CustomProxyProvider` (new).
+- `get_active_provider()` resolves the **one** active provider from env and caches it — **no keyed multi-provider coexistence** (`get_provider(key)` is intentionally NOT added; YAGNI for a single-provider app).
+- Unknown `PROXY_PROVIDER` → existing warn-and-fallback path (`registry.py:35–41`) is unchanged.
+
+### 2. Provider config source: env only
+
+Credentials come from **env**, full stop (single global provider):
+
+- `anonymous_proxies` → existing `RESIDENTIAL_PROXY_*`. Unchanged.
+- `custom` → new env knobs (one URL or a pool): `CUSTOM_PROXY_URLS` (comma-separated) and/or `CUSTOM_PROXY_URL`.
+
+No per-connector proxy keys, and **no** connector-config validator changes (that was the per-connector path, now dropped).
+
+### 3. New provider: `CustomProxyProvider` (BYO) — the only addition
+
+- Accepts a raw proxy URL **or** a list of URLs from env. Covers "bring your own proxy / our own pool" with no vendor-specific auth assumptions.
+- Implements just `get_proxy_url()`; the base derives `get_requests_proxies()` from it (`base.py:37–46`). When configured with a **pool**, `get_proxy_url()` returns the *next* endpoint from an internal `ProxyRotator` (see §4) — so rotation is transparent to every caller of the zero-arg getter.
+- **No branded vendors** (resolved): Webshare/BrightData/Smartproxy/etc. are not shipped. A user who wants any specific vendor points `CustomProxyProvider` at that vendor's endpoint(s) via `CUSTOM_PROXY_URLS`.
+
+### 4. Rotation + retry (only when the active provider is pool-backed)
+
+- If `CustomProxyProvider` is configured with **multiple** URLs, it wraps them in Scrapling's `ProxyRotator` (cyclic default, thread-safe `Lock` — `proxy_rotation.py:71,88–92`) and returns the next endpoint per `get_proxy_url()` call. Because `03a`'s browser tiers run under `asyncio.to_thread`, the rotator's lock keeps this safe.
+- The crawler adds a **bounded** client-side retry: on a tier failure where `is_proxy_error(exc)` is true (`proxy_rotation.py:27–30`), it re-reads `get_proxy_url()` (next endpoint) and retries **that tier once** before falling through. One rotation-retry per tier — no unbounded fan-out on billable crawls.
+- Single-endpoint providers (`anonymous_proxies`, already server-side rotating; or `custom` with one URL) return the same endpoint every call, so the retry is a harmless no-op for them.
+
+### 5. Crawler stays on the zero-arg getter (no injection needed)
+
+Because there is one global provider and rotation lives **inside** it, the crawler does **not** need a proxy injected per crawl:
+
+- All three Scrapling tiers keep calling `get_proxy_url()` (which yields the rotating value when pool-backed). The only crawler edit is the bounded `is_proxy_error` retry from §4.
+- **No change** to YouTube/chat-tool consumers — same zero-arg getters, same global provider.
+- Future seam (NOT built): if per-pipeline proxying is ever needed, resolve a provider from `pipeline.config` and pass its `get_proxy_url` into the crawl — an additive optional arg, no change to today's call sites.
+
+## Config / env changes
+
+- `config/__init__.py:983–992` + `.env.example:300–309`: add `CUSTOM_PROXY_URLS` (comma-separated pool) and/or `CUSTOM_PROXY_URL` for the `custom` provider; keep all existing `RESIDENTIAL_PROXY_*` working unchanged. `PROXY_PROVIDER` now accepts `"custom"` in addition to `"anonymous_proxies"`.
+- **No** connector-config validator changes (the per-connector proxy path is dropped — single global provider).
+
+## Work items
+
+1. **`CustomProxyProvider`**: BYO single-URL or pool-of-URLs provider (reads `CUSTOM_PROXY_URL(S)`); internal `ProxyRotator` when pool-backed; register as `"custom"` in `_PROVIDERS`.
+2. **Rotation retry**: add a bounded (one-per-tier) `is_proxy_error` retry to the crawler's Scrapling tiers; single-endpoint providers no-op.
+3. **Config + docs**: `CUSTOM_PROXY_URL(S)` in `Config` + `.env.example`; document `PROXY_PROVIDER="custom"`.
+4. **Tests**: `CustomProxyProvider` single-URL vs pool; `ProxyRotator` cyclic order; `get_active_provider()` resolves `"custom"` and still warn-falls-back on unknown; crawler rotates once on `is_proxy_error` then falls through; `anonymous_proxies` + YouTube path unchanged (same endpoint each call).
+
+## Risks / trade-offs
+
+- **Backward compatibility of the getters.** The zero-arg `get_proxy_url()` contract is consumed in 4+ places; the design keeps it **completely intact** — no signature changes, no per-caller keys. Only the cached active provider's *implementation* changes when `PROXY_PROVIDER="custom"`.
+- **Over-rotation cost.** Rotation-retry is bounded to one extra attempt per tier so a billable crawl can't silently multiply upstream proxy usage.
+- **Single provider is a deliberate constraint.** One global provider app-wide (resolved). Per-pipeline/per-connector selection is intentionally deferred behind a no-op seam (§5); not built now.
+- **Pool rotation under `to_thread`.** `ProxyRotator`'s `Lock` makes the rotating `get_proxy_url()` safe to call from the browser tiers `03a` offloads via `asyncio.to_thread`.
+
+## Resolved decisions
+
+- **Branded vendors → NONE.** Ship `CustomProxyProvider` (BYO) only; no Webshare/BrightData/Smartproxy/etc. subclasses. A user who wants a specific vendor points `CUSTOM_PROXY_URLS` at it.
+- **Selection scope → single global provider, app-wide.** No per-connector/per-crawl override is built; resolution is env-only via `Config.PROXY_PROVIDER`. A future per-pipeline override is left as a no-op seam (§5) so it stays "scalable + fast to add later."
+- **Client-side rotation → built, but only active for a pool-backed `CustomProxyProvider`.** `anonymous_proxies` (server-side rotating) and single-URL custom configs skip it automatically. Rotation lives inside the provider so it's transparent to all callers.
+
+## Out of scope (hand-offs)
+
+- Per-**pipeline** / per-connector proxy selection → deferred (Phases 5–7 *if ever needed*); §5 leaves a no-op seam, nothing is wired now.
+- Branded-vendor provider subclasses → not planned (use `CustomProxyProvider`).
+- **Static / sticky-session proxies (future).** A later capability will add **static proxy** support — sticky IPs held for the duration of a session — most likely paired with **authenticated/account-based scraping** to bypass logged-in platforms (the deferred platform connectors: LinkedIn, Instagram, etc.). This is a *different axis* from the rotating pool here: rotation maximizes IP diversity, whereas account bypass needs IP **stability** so a session/cookie stays bound to one IP. It is additive to this design — a new `ProxyProvider` (or a "sticky" mode/flag on `CustomProxyProvider`) registered under a new `PROXY_PROVIDER` key, with no change to the zero-arg getter contract — and stays consistent with the single-provider model (the active provider would be the static one when that workflow is selected). Build it alongside the platform connectors, not in Phase 3.
+- Crawl credit metering (proxy cost is absorbed into the flat `$1 / 1000 successful` price, **not** metered separately) → `03c`.
+- Captcha solving → `03d` (deferred).

plans/backend/03c-crawl-billing.md

@@ -0,0 +1,132 @@
+# Phase 3c — Crawl credit billing ($1 / 1000 successful requests)
+
+> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
+> Depends on `03a-crawler-core.md` (the `CrawlOutcomeStatus.SUCCESS` signal + `crawls_succeeded` counter). Sibling: `03b-proxy-expansion.md`. `03d` is deferred.
+
+## Objective
+
+Charge **$1 per 1000 successful crawl requests** = **1000 micro-USD per successful crawl**, drawn from the existing unified credit wallet, **off by default for self-hosted/OSS** installs. Two surfaces crawl, so both are metered:
+
+- **Connector / pipeline crawls** (the `webcrawler_indexer` path) → billed to the **workspace owner** via a dedicated `WebCrawlCreditService` (check + charge).
+- **Ad-hoc chat scrapes** (`scrape_webpage` tools) → the crawl cost is **folded into the chat turn's existing bill** (added to the turn accumulator), so it settles with the turn's normal premium finalize (`start_turn()` + `credit_finalize`). No second wallet hit. (Implies a second gate: only charged when the turn itself is a premium/billed turn — see §3.)
+
+> **Migration impact: none.** This phase adds config flags + code only. `TokenUsage.usage_type` is already a free-form `String(50)`, so the new `web_crawl` value needs **no Alembic migration or schema change**. (Per the standing guidance, the migration-sensitive work lives in Phases 1 and 5+; 03c is purely additive.)
+
+## The key realization (don't use `billable_call`)
+
+My first instinct was to add a flat reserve/finalize path around `TokenQuotaService` because `billable_call` settles at the LLM token accumulator (`acc.total_cost_micros`, `billable_calls.py:367`) — which is 0 for a non-LLM crawl, so `billable_call` would finalize at nothing.
+
+But there's already a **purpose-built precedent for per-unit, non-LLM wallet billing**: `app/services/etl_credit_service.py` (`EtlCreditService`). It deliberately **skips** the reserve/finalize dance and does a simple **gate → pre-check → post-charge**:
+
+- `billing_enabled()` → `config.ETL_CREDIT_BILLING_ENABLED` (`etl_credit_service.py:44–46`); **every method is a no-op when disabled** — that's what keeps self-hosted "effectively free" (`:9–11`, `:59–60`, `:84–85`, `:115–116`).
+- `pages_to_micros(pages, multiplier)` → `pages * multiplier * config.MICROS_PER_PAGE` (`:48–51`).
+- `check_credits(user_id, estimated_pages)` → raises `InsufficientCreditsError` if `required > available` where `available = balance - reserved` (`:76–102`).
+- `charge_credits(user_id, pages)` → `user.credit_micros_balance -= cost; commit; maybe_trigger_auto_reload(...)` (`:104–138`).
+
+It's wired into the **sibling connector indexers** exactly the way we need: `google_drive_indexer.py` (`558,562,605`), `dropbox_indexer.py` (`426,504,593`), `local_folder_indexer.py` (`639,698,864`) — instantiate `EtlCreditService(session)`, `check_credits(...)` before processing, `charge_credits(...)` after.
+
+**So 03c mirrors `EtlCreditService` with a crawl-specific unit and flag — no `billable_call`, no reserve/finalize.**
+
+## Current state (cited)
+
+- **Crawled URLs are billed nothing today.** `webcrawler_indexer.py` does not import `EtlCreditService` or touch the wallet (verified — it's absent from the file). The crawler extracts content directly via Trafilatura and bypasses the ETL page pipeline, so there's **no existing ETL charge** to double up with.
+- **Wallet model** (`token_quota_service.py`): the balance lives on `User.credit_micros_balance` / `User.credit_micros_reserved`; spendable = `balance - reserved`. Direct debit is exactly what `EtlCreditService.charge_credits` does.
+- **Audit trail**: `record_token_usage(session, *, usage_type, search_space_id, user_id, cost_micros=…, call_details=…, message_id=None, …)` (`token_tracking_service.py:517–568`) inserts a `TokenUsage` row (best-effort; caller commits). `TokenUsage.usage_type` is a `String(50)` indexed column explicitly intended for non-chat usage with `message_id` NULL (`db.py:1068–1116`, comment at `:1072–1084`: "indexing, image generation, podcasts — which keep message_id NULL").
+- **Per-feature config knobs** are the convention: `MICROS_PER_PAGE` (`config/__init__.py:655`, default `1000`), `ETL_CREDIT_BILLING_ENABLED` (`:652–653`, default FALSE), `QUOTA_DEFAULT_IMAGE_RESERVE_MICROS` (`:729`), etc. `maybe_trigger_auto_reload` (`auto_reload_service`) is the shared low-balance top-up nudge.
+
+> **Implementation note.** Same convention as `03a`/`03b`: citations use **today's** names (`search_space_id`/`SearchSpace`) so they stay greppable against current code; post Phases 1–2 the live code says `workspace_id`/`Workspace` — map accordingly (e.g. the `record_token_usage(search_space_id=…)` kwarg becomes `workspace_id=…`). Locate code by **symbol/grep**, not the absolute line numbers cited here, since Phase 2's rename and `03a`'s `crawl_url`/`scrape_webpage` refactor shift them.
+
+## Target design
+
+### 1. `WebCrawlCreditService` (mirror of `EtlCreditService`)
+
+New `app/services/web_crawl_credit_service.py` — a dedicated service (independent flag + price; lowest risk, matches the per-feature-knob convention). The two `@staticmethod` helpers are the **shared primitives both surfaces reuse**:
+
+- `billing_enabled()` → `config.WEB_CRAWL_CREDIT_BILLING_ENABLED` (new flag, default FALSE). *(static — also called by the chat tool)*
+- `successes_to_micros(n)` → `n * config.WEB_CRAWL_MICROS_PER_SUCCESS` (new knob, default `1000` = $1/1000). *(static — also called by the chat tool)*
+- `check_credits(owner_user_id, estimated_urls)` → no-op when disabled; else raise `InsufficientCreditsError` (reuse the one from `etl_credit_service`) if `successes_to_micros(estimated_urls) > available`.
+- `charge_credits(owner_user_id, successes)` → no-op when disabled; else debit `credit_micros_balance -= successes_to_micros(successes)`, commit, `maybe_trigger_auto_reload`.
+
+A separate `CreditMeterService` generalization is deliberately deferred until a third per-unit biller appears — exposing the price/flag as statics already lets the chat surface share the math without coupling the two flows.
+
+### 2. Wiring in `webcrawler_indexer.py`
+
+1. **Resolve the owner.** Bill the **workspace owner** (the product concept), not the triggering `user_id`: `owner_user_id = (select SearchSpace.user_id where id == search_space_id)`. (Mirrors how `billable_calls._resolve_agent_billing_for_search_space` (func `:441`) reads `search_space.user_id` at `billable_calls.py:499` — but we only need the owner id, so a direct `select` is enough; no need to call that LLM-model-resolving helper.)
+2. **Pre-flight gate** (after URL parse, before the crawl loop — i.e. the indexer's second pass that actually fetches URLs; "phase 2 of the 2-phase indexer", not roadmap Phase 2): `await svc.check_credits(owner_user_id, len(urls))`. On `InsufficientCreditsError`, `log_task_failure` with a clear "out of crawl credit" message and return — don't crawl. Because **successes ≤ len(urls)**, this upper-bound check means the wallet can never go negative from crawls (unlike ETL, where actual pages can exceed the estimate).
+3. **Audit (add, don't commit)**: `record_token_usage(session, usage_type="web_crawl", search_space_id=search_space_id, user_id=owner_user_id, cost_micros=successes_to_micros(crawls_succeeded), call_details={"urls": len(urls), "successes": crawls_succeeded, "connector_id": connector_id}, message_id=None)` — `record_token_usage` only `session.add`s the row (`token_tracking_service.py:552`), it does not commit.
+4. **Charge actuals**: `await svc.charge_credits(owner_user_id, crawls_succeeded)`. `charge_credits` debits the balance and **commits** (mirroring `EtlCreditService.charge_credits`), which flushes the step-3 audit row **and** the balance debit in one transaction.
+
+Ordering matters: audit must be added **before** the charge, because the charge's commit is what persists both. Both run on the indexer's existing `session` (same as the ETL services), after the final per-URL document commit (`webcrawler_indexer.py:432`). If `crawls_succeeded == 0`, skip both (no-op).
+
+### 3. Chat scrape crawls (folded into the chat turn's bill)
+
+The chat `scrape_webpage` tools (`main_agent/tools/scrape_webpage.py:183`, plus the `research` sibling) crawl via `WebCrawlerConnector.crawl_url` (`main_agent/tools/scrape_webpage.py:232–233`) — the **same** call `03a` refactors. The chat hook therefore keys off `03a`'s `CrawlOutcomeStatus.SUCCESS` return (a hard dependency on 03a's new `crawl_url` signature).
+
+**How the chat turn actually bills** (this is *not* `billable_call`): the chat orchestrators create the turn accumulator with `start_turn()` (`new_chat/orchestrator.py:185`, `resume_chat/orchestrator.py:147`, `anonymous_chat_routes.py:360`); a premium turn reserves up front and, at the end, `finalize_credit` debits `accumulator.total_cost_micros` via `credit_finalize` (`shared/premium_quota.py:83–104`). So adding to the accumulator before finalize is debited with the turn:
+
+```python
+from app.services.token_tracking_service import get_current_accumulator
+from app.services.web_crawl_credit_service import WebCrawlCreditService
+
+# after a SUCCESS crawl in the scrape tool
+acc = get_current_accumulator()              # token_tracking_service.py:212–213 (turn ContextVar)
+if acc is not None and WebCrawlCreditService.billing_enabled():
+    acc.add(                                 # :112–135 — appends a TokenCallRecord
+        model="web_crawl",
+        prompt_tokens=0, completion_tokens=0, total_tokens=0,
+        cost_micros=WebCrawlCreditService.successes_to_micros(1),
+        call_kind="web_crawl",
+    )
+```
+
+`accumulator.total_cost_micros` sums every call's `cost_micros` (`:174–186`), so the crawl micros ride along into the premium finalize — "included in the already-billed chat." `call_kind="web_crawl"` keeps it distinguishable in `per_message_summary` (`:137–159`).
+
+**Why the ContextVar is reliably visible (not hand-waving):** the LiteLLM `TokenTrackingCallback` already populates *this same* `_turn_accumulator` from deep inside the graph's LLM calls — that's how chat billing works today. If the ContextVar set by `start_turn()` reaches the LLM callback, it reaches a `scrape_webpage` tool in the same graph. The tools are `async` and `await crawl_url` in-loop, so the context is intact (the only thing that would break propagation is offloading to a raw OS thread without `copy_context`, which this path doesn't do).
+
+Two gates, both required, for a chat scrape to actually cost money:
+- **`WEB_CRAWL_CREDIT_BILLING_ENABLED`** (self-hosted: off → adds nothing).
+- **The turn is premium.** `needs_credit_quota` = `agent_config.is_premium` (`premium_quota.py:43–44`). Free / BYOK / **anonymous** (`anonymous_chat_routes.py:360`, no wallet) turns never reserve/finalize → the `web_crawl` cost is recorded in the accumulator breakdown but **not debited**. This is intentional: if the chat turn isn't billed, neither is its scrape.
+
+Other notes:
+- **No per-scrape pre-block.** The turn's up-front reservation is an LLM-only estimate (`reserve_credit`/`estimate_call_reserve_micros`, `premium_quota.py:65`); crawl micros added afterward aren't reserved, but `credit_finalize` debits actuals regardless and may push the balance slightly negative — same allow-exceed posture as ETL. Pre-blocking applies only to the indexer batch path (§2).
+- **Accounting caveat.** Chat-scrape crawl spend lands on the turn's `usage_type="chat"` `TokenUsage` row (inside `cost_micros` / the `web_crawl` line of `model_breakdown`), **not** a `usage_type="web_crawl"` row. "Total crawl spend" analytics must sum the indexer's `web_crawl` rows **plus** the `web_crawl` call-kind inside chat rows.
+
+### 4. What counts (from `03a`)
+
+One unit per `CrawlOutcomeStatus.SUCCESS` — a URL that yielded usable extracted content — **regardless of internal fallback tiers** and **regardless of downstream KB dedupe** (unchanged/duplicate still crawled successfully). `EMPTY`/`FAILED` are free.
+
+## Config / env changes
+
+- `config/__init__.py` (next to `ETL_CREDIT_BILLING_ENABLED`/`MICROS_PER_PAGE`, `:649–655`):
+  - `WEB_CRAWL_CREDIT_BILLING_ENABLED = os.getenv(..., "FALSE").upper() == "TRUE"`
+  - `WEB_CRAWL_MICROS_PER_SUCCESS = int(os.getenv("WEB_CRAWL_MICROS_PER_SUCCESS", "1000"))`
+- `.env.example`: document both (commented), noting hosted = TRUE, self-hosted = FALSE.
+
+## Work items
+
+1. **`WebCrawlCreditService`** mirroring `EtlCreditService` (gate + `check_credits` + `charge_credits` + static `successes_to_micros`/`billing_enabled`).
+2. **Config knobs** + `.env.example` docs.
+3. **Indexer wiring**: owner resolution, pre-flight `check_credits`, then at end-of-run (after `:432`) `record_token_usage(usage_type="web_crawl")` (add) **followed by** `charge_credits(crawls_succeeded)` (commits both); skip when `crawls_succeeded == 0`.
+4. **Chat scrape wiring**: in both `scrape_webpage` tools, on SUCCESS fold `successes_to_micros(1)` into the turn accumulator (§3).
+5. **Tests**: billing-disabled → both surfaces no-op (self-hosted); indexer enabled + sufficient → debits `successes * 1000` + one `web_crawl` `TokenUsage`; indexer enabled + insufficient → task fails pre-crawl, no debit; owner (not trigger user) billed; `EMPTY`/`FAILED` free; chat scrape on a **premium** turn → `acc.total_cost_micros` rises by `1000` per success and is debited at finalize; chat scrape on a **free/anonymous** turn → recorded in the accumulator but **not** debited.
+
+## Risks / trade-offs
+
+- **Celery retry re-billing.** A task retry re-crawls only the URLs not already `ready`/`pending`/`processing` (the 2-phase indexer skips those, `webcrawler_indexer.py:196–219,341–347`), so a retry charges only re-crawled successes. Acceptable for MVP; Phase 5's `PipelineRun` can persist `charged_micros` for stronger idempotency.
+- **Same URL across connectors.** Each connector that crawls it performs a real request → each successful crawl bills, even though the 2nd is KB-deduped. This is intended per the `03a` billing-policy note.
+- **Session coupling.** `charge_credits` commits the indexer session; placing it after the final doc commit avoids entangling a debit with mid-run document state.
+- **Gating asymmetry (intended).** The **indexer** path bills whenever `WEB_CRAWL_CREDIT_BILLING_ENABLED` is on and the owner has credit — no "premium tier" requirement (matches ETL, which gates only on its own flag). The **chat** path additionally requires the turn to be premium, because it piggybacks the turn's reserve/finalize machinery. Same price, two different gate sets — call this out in the credit-status UI copy (frontend phase).
+- **No cross-surface double charge.** A chat scrape calls `crawl_url` directly and never enters `webcrawler_indexer`, so a given crawl is billed by exactly one surface.
+- **Proxy/captcha cost.** Absorbed into the flat $1/1000 (margin), **not** metered separately (see `03b`, and `03d` for captcha which has its own per-solve upstream cost — revisit there).
+
+## Resolved decisions (this pass)
+
+- **Bill ad-hoc chat scrapes? → YES.** Chat scrape crawls are metered too, but folded into the chat turn's existing bill via the turn accumulator (§3) rather than a separate wallet debit.
+- **Dedicated `WebCrawlCreditService` vs generalized meter? → dedicated**, with `billing_enabled`/`successes_to_micros` exposed as statics so the chat surface shares the price/flag. Generalize later only if a third per-unit biller appears.
+- **Pre-flight strictness → pre-block** the indexer batch run on insufficient credit (`len(urls)` is a safe upper bound). Chat scrapes are governed by the turn's own reservation, not a per-scrape block.
+
+## Out of scope (hand-offs)
+
+- Per-**pipeline** run accounting / `charged_micros` persistence for idempotency → Phases 5–7.
+- Captcha-solver upstream cost pass-through → `03d` (deferred).
+- Surfacing crawl spend in the credit-status UI → frontend umbrella.

plans/backend/03d-captcha-solving.md

@@ -0,0 +1,58 @@
+# Phase 3d — Captcha solving (DEFERRED — sequenced last, non-MVP-blocking)
+
+> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
+> **Status: deferred.** Build only after `03a`/`03b`/`03c` ship and there's a real need. Depends on `03a` (the StealthyFetcher tier) and the per-crawl proxy from `03b`. Touches the crawl billing model from `03c`.
+
+## Why deferred
+
+Scrapling already solves **Cloudflare Turnstile/Interstitial** via `solve_cloudflare=True` on the StealthyFetcher tier (`03a`; `references/Scrapling/scrapling/fetchers/stealth_chrome.py:38,90`). That covers the most common interstitial. The remaining captcha types (reCAPTCHA v2/v3, hCaptcha, image) need a **paid third-party solver**, add latency (10–60s/solve), have <100% success, and carry target-ToS/legal considerations. None of that is MVP-blocking, so it's sequenced last.
+
+## Scope boundary
+
+| Challenge | Handled by | Where |
+| --- | --- | --- |
+| Cloudflare Turnstile / Interstitial | Scrapling `solve_cloudflare=True` | `03a`, StealthyFetcher tier |
+| reCAPTCHA v2/v3, hCaptcha, image | **`captchatools`** (this subplan) | StealthyFetcher `page_action` |
+
+## Grounding (libraries verified)
+
+- **`captchatools` is itself the provider registry.** `new_harvester(api_key, solving_site, sitekey, captcha_url, captcha_type="v2"|"v3"|"hcaptcha"|"image", ...)` → `.get_token(proxy=…, proxy_type=…, user_agent=…, b64_img=…)` returns a **token string** (`references/Captcha-Tools/README.md:28–47`). `solving_site` ∈ {`capmonster`,`2captcha`,`anticaptcha`,`capsolver`,`captchaai`} (`:113–127`). Errors: `ErrNoBalance`, `ErrWrongAPIKey`, `ErrWrongSitekey`, … via `captchatools.exceptions` (`:136–155`). **Implication:** we do **not** rebuild a multi-provider class hierarchy (unlike `03b`'s proxy registry) — `captchatools` already dispatches across the 5 services. Our layer is thin: config resolution + page detection/injection glue.
+- **`captchatools` only harvests a token; it does not inject it.** The caller must drop the token into the page (`g-recaptcha-response` / `h-captcha-response` textarea, or invoke the JS callback) and submit.
+- **Injection requires a browser page**, so this only works on the **StealthyFetcher** tier. Scrapling exposes `page_action`: "a function that takes the `page` object, runs after navigation, and does the automation you need" (`stealth_chrome.py:30,82`). AsyncFetcher (HTTP) and DynamicFetcher cannot solve interactive captchas — a captcha hit there must escalate to StealthyFetcher.
+
+## Sketch (when built)
+
+1. **Config layer** (`app/utils/captcha/`, mirroring `app/utils/proxy/`'s config resolution from `03b`):
+   - `CaptchaConfig` = `(enabled, solving_site, api_key)`, resolved from env defaults or per-connector config (same `resolve_*` pattern as proxy in `03b`).
+   - Env: `CAPTCHA_SOLVING_ENABLED` (default FALSE), `CAPTCHA_SOLVER_PROVIDER`, `CAPTCHA_SOLVER_API_KEY`. Off by default → zero captcha attempts (and zero solver cost).
+2. **Detection + injection `page_action` factory** — builds a callback passed to `StealthyFetcher.async_fetch(..., page_action=…)`:
+   - Detect sitekey in DOM (`.g-recaptcha[data-sitekey]`, `.h-captcha[data-sitekey]`, reCAPTCHA-v3 via `grecaptcha.execute`).
+   - Harvest: `new_harvester(solving_site, api_key, sitekey, captcha_url=page.url, captcha_type=…).get_token(proxy=<the 03b per-crawl proxy>, user_agent=<page UA>)`.
+   - Inject token + dispatch events / invoke callback; submit; wait for navigation.
+3. **Crawler escalation**: only the StealthyFetcher tier attempts solving; a captcha detected on a lower tier escalates to StealthyFetcher (the ladder already ends there per `03a`).
+4. **Dependency**: add `captchatools` to `pyproject.toml` (build-time only).
+
+## The billing asymmetry (the hard part — decide at build time)
+
+`03c` bills the workspace owner **per successful crawl** (`CrawlOutcomeStatus.SUCCESS`), and absorbs proxy cost into the flat $1/1000. **Captcha is different**: the solver charges **per attempt** (~$1–3 / 1000, type-dependent) **regardless of whether the crawl ultimately succeeds**. So a failed solve = real upstream cost with **no billable success** — proxy's absorb-it model doesn't transfer cleanly.
+
+Options (resolve when this is actually built):
+- **(a) Separate per-solve charge** — meter each solve attempt as its own unit (e.g. `web_crawl_captcha` usage_type, its own `*_MICROS_PER_*` knob), independent of crawl SUCCESS. Most cost-honest; bills even on failed solves (matching the upstream charge).
+- **(b) Higher crawl price when captcha enabled** — absorb into a fatter flat rate; simplest UX, but cross-subsidizes failed solves and easy-vs-hard pages unevenly.
+- **(c) Cost-plus pass-through** — meter the solver's reported cost × margin.
+
+Recommendation leaning **(a)** (separate per-attempt unit) because the upstream cost is per-attempt and significant, but defer the final call.
+
+## Risks / considerations
+
+- **Solver-tier only.** No captcha solving on the HTTP/DynamicFetcher tiers; must escalate to StealthyFetcher (slower, browser-backed).
+- **Latency & flakiness.** Solves take 10–60s and aren't guaranteed; tune timeouts and a max-attempts cap so a single URL can't burn unbounded solver credit.
+- **Solver-account balance.** `ErrNoBalance`/`ErrWrongAPIKey` (`README.md:136–155`) must surface clearly and disable solving rather than loop.
+- **Proxy coherence.** Pass the **same** per-crawl proxy (`03b`) to `get_token(proxy=…)` so the solve happens from the same IP as the crawl (some captchas IP-bind the token).
+- **Policy / ToS.** Automated captcha solving may violate a target site's terms; gate behind the explicit `CAPTCHA_SOLVING_ENABLED` flag and treat as an opt-in, owner-acknowledged capability.
+
+## Out of scope
+
+- Anything in `03a`/`03b`/`03c`.
+- Cloudflare Turnstile (already handled by Scrapling in `03a`).
+- Final captcha billing model — chosen at build time (see options above).