SurfSense/plans/backend/03a-crawler-core.md

# Phase 3a — WebURL Crawler core (Scrapling-only) + success semantics

> Part of **Phase 3 — WebURL Crawler & Crawl Billing**. See `00-umbrella-plan.md`.
> Sibling subplans: `03b-proxy-expansion.md`, `03c-crawl-billing.md`, `03d-captcha-solving.md` (deferred).

> **Implementation note (applies to all Phase-3 plans).** Phase 3 lands **after** Phases 1–2, which rename `SearchSpace`→`Workspace` and `search_space_id`→`workspace_id` everywhere. Citations below use **today's** names so they stay greppable against current code; when implementing, the live code will already say `workspace_*` — map accordingly. Apply every edit by **symbol/grep** (e.g. `firecrawl_api_key`, `crawl_url`, `FIRECRAWL_API_KEY`), **not** by the absolute line numbers cited here — Phase 2's rename (and `03a`'s own Firecrawl removal) shift line numbers.

## Objective

Make the Universal WebURL Crawler a **single-framework (Scrapling) component** with deterministic per-URL outcome semantics and a clean, explicit **"successful crawl" signal** that Phase 3c bills on.

Two hard requirements from the decisions log:

1. **Remove Firecrawl entirely.** No other scraping framework now or planned. Scrapling's `StealthyFetcher` can bypass Cloudflare Turnstile when invoked with `solve_cloudflare=True` (see tier design below); captcha-tools (`03d`, deferred) covers the rest.
2. **One billable unit = one URL that yields usable extracted content**, regardless of how many internal fallback tiers ran. `03a` must expose that signal; `03c` meters it.

This subplan does NOT touch proxy rotation (→ `03b`), credit metering (→ `03c`), or captcha (→ `03d`).

## Current state (cited)

### The crawler

`app/connectors/webcrawler_connector.py` — `WebCrawlerConnector.crawl_url()` is a 4-tier fallback ladder:

1. **Firecrawl** (premium, if `firecrawl_api_key` set) — `_crawl_with_firecrawl()` (lines 89–100, 223–272), imports `from firecrawl import AsyncFirecrawlApp` (line 22).
2. Scrapling `AsyncFetcher` (static HTTP, curl_cffi) — `_crawl_with_async_fetcher()` (lines 274–310).
3. Scrapling `DynamicFetcher` (browser, run in a thread) — `_crawl_with_dynamic()` (lines 312–339).
4. Scrapling `StealthyFetcher` (Camoufox anti-bot, run in a thread) — `_crawl_with_stealthy()` (lines 341–369).

Extraction is Trafilatura HTML→markdown in `_build_result()` (lines 371–469). Every Scrapling tier passes `proxy=get_proxy_url()` (lines 287, 329, 359). `crawl_url()` returns a tuple `(result_dict | None, error | None)`; `result_dict` has `content` / `metadata` / `crawler_type`.

### The two call sites

- **Type-1 indexer (billable path):** `app/tasks/connector_indexers/webcrawler_indexer.py` reads `FIRECRAWL_API_KEY` from connector config (line 115), builds `WebCrawlerConnector(firecrawl_api_key=api_key)` (line 139), and calls `crawler.crawl_url(url)` per URL (line 297). It already tracks `documents_indexed` / `documents_updated` / `documents_skipped` / `documents_failed` / `duplicate_content_count` (lines 167–171), but **none of these is a clean "crawl succeeded" count** — duplicates and unchanged docs are bucketed as skipped even though a successful fetch happened.
- **Chat scrape tool (ad-hoc):** `main_agent/tools/scrape_webpage.py` (line 232) and `subagents/builtins/research/tools/scrape_webpage.py` (line 226) build `WebCrawlerConnector(firecrawl_api_key=...)` and call `crawl_url(url, formats=["markdown"])`. The `formats` arg is Firecrawl-only (declared at `crawl_url` signature line 59; docstring line 72: "only for Firecrawl").

### Firecrawl threading (the cross-cutting surface)

Firecrawl's API key is plumbed end-to-end and must be removed everywhere:

| Layer | File:line | What to remove |
|-------|-----------|----------------|
| Crawler | `app/connectors/webcrawler_connector.py` | `firecrawl` import (22), `firecrawl_api_key`/`use_firecrawl` ctor (36–46), `set_api_key()` (48–56), tier-1 block (89–100), `_crawl_with_firecrawl()` (223–272), `formats` param + Firecrawl mentions in docstrings |
| Dependency | `pyproject.toml:45` | `"firecrawl-py>=4.9.0"` (+ regenerate `uv.lock`). Scrapling stays (`pyproject.toml:91` → `"scrapling[fetchers]>=0.4.9"`) |
| Connector config | `app/utils/validators.py` | `FIRECRAWL_API_KEY` from `WEBCRAWLER_CONNECTOR.optional`/`validators` (573–580); delete `validate_firecrawl_api_key_format()` (472–478) |
| Indexer | `app/tasks/connector_indexers/webcrawler_indexer.py` | `api_key = connector.config.get("FIRECRAWL_API_KEY")` (115), `use_firecrawl` log field (135), pass-through to ctor (139) |
| Chat setup | `app/tasks/chat/streaming/flows/shared/pre_stream_setup.py` | `setup_connector_and_firecrawl()` returns `firecrawl_api_key` (1–30) — collapse to connector-service-only |
| Chat orchestrators | `new_chat/orchestrator.py` (86, 378, 412, 665), `resume_chat/orchestrator.py` (65, 317, 347, 483) | `firecrawl_api_key` threading |
| Automations | `automations/actions/builtin/agent_task/dependencies.py` (19, 34, 85, 103), `.../invoke.py` (174) | `firecrawl_api_key` dep field |
| Main agent | `main_agent/runtime/factory.py` (71, 142), `main_agent/tools/registry.py` (36) | `firecrawl_api_key` param/wiring |
| Scrape tools | `main_agent/tools/scrape_webpage.py` (170, 232), `research/tools/scrape_webpage.py` (164, 226), `research/tools/index.py` (28) | `firecrawl_api_key` factory arg + ctor arg |
| Tests | `tests/unit/automations/actions/builtin/agent_task/test_dependencies.py` (44, 81) | `firecrawl_api_key == "fc-key"` assertion + fake |

> **Not a global env var.** `FIRECRAWL_API_KEY` is **not** in `.env.example` and there is no `Config.FIRECRAWL_API_KEY`. It lives only inside each WebCrawler connector's `config` JSON — the validator reads it from that dict (`validators.py:474`). So removal is code-only; **no env/docs change**. Existing `WEBCRAWLER_CONNECTOR` rows may still carry a now-dead `FIRECRAWL_API_KEY` key in `config` — harmless (it's simply ignored), optionally scrubbed by a tiny data migration if we want clean rows.
>
> **Signature change, not just deletions.** Removing `firecrawl_api_key` mutates the `RuntimeDeps`-style dataclass (`agent_task/dependencies.py:34`) and the agent runtime factory/tool factories — a coordinated signature change across the chat + automations call graph, so land it atomically.

### Runtime/deps already in place

- `Dockerfile:112–115` runs `RUN scrapling install` to fetch patchright Chromium + Camoufox; the `scrapling[fetchers]` extra pulls playwright/patchright. **No new install step needed** once Firecrawl is gone.
- Proxy is read via `app/utils/proxy/get_proxy_url()` (`__init__.py:13`), backed by the `PROXY_PROVIDER` registry (`config/__init__.py:983`). `03a` leaves this single-URL model untouched (rotation is `03b`).

## Target design

### Tier ladder (Scrapling-only)

`crawl_url()` becomes a 3-tier ladder, preserving the existing thread-offload + `NotImplementedError` handling for the browser tiers (Windows `SelectorEventLoop` cannot spawn subprocesses — lines 134–141, 161–168):

1. `AsyncFetcher.get(...)` — fast static HTTP.
2. `DynamicFetcher.fetch(...)` — full browser (via `asyncio.to_thread`).
3. `StealthyFetcher.fetch(...)` — Camoufox anti-bot, last resort. Enable Cloudflare solving here by passing **`solve_cloudflare=True`** — a documented `StealthyFetcher.fetch` kwarg ("Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response", `references/Scrapling/scrapling/fetchers/stealth_chrome.py:38`; it's a `StealthSession` TypedDict key passed via `**kwargs`). The current stealthy call (connector lines 354–360) passes `headless`/`network_idle`/`block_ads`/`proxy` but **not** `solve_cloudflare`, so this is a real behavior add. (Note: `solve_cloudflare` runs the full browser challenge loop, so it's correctly scoped to the last-resort tier only.)

Trafilatura extraction (`_build_result`) and `format_to_structured_document()` are unchanged.

### Explicit outcome model

Replace the implicit `(dict|None, str|None)` contract with an explicit outcome so callers (indexer, chat tool, and `03c` metering) agree on what "success" means. Proposed:

```python
class CrawlOutcomeStatus(str, Enum):
    SUCCESS = "success"   # a tier returned usable extracted content
    EMPTY   = "empty"     # fetched, but no usable content after ALL tiers
    FAILED  = "failed"    # invalid URL or every tier errored
```

`crawl_url()` returns a small dataclass `CrawlOutcome(status, result, error, tier)` (or, to minimize churn, keep the tuple and add a third element). Either way the **billable success predicate is single-sourced**: `status == SUCCESS`.

| Outcome | When | Billable (`03c`)? | Document status (indexer) |
|---------|------|-------------------|---------------------------|
| `SUCCESS` | a tier extracted usable content (`_build_result` returned a dict) | **Yes — 1 unit** | `ready` (or unchanged/duplicate, see note) |
| `EMPTY` | every tier was reached but none produced usable extracted content (the static tier also treats HTTP ≥ 400 as a miss and falls through — connector lines 292–301; the browser tiers attempt extraction regardless of status) | No | `failed("No content extracted")` |
| `FAILED` | invalid URL, or all tiers raised | No | `failed(<error>)` |

**Billing-policy note for `03c`:** success is the *crawl* succeeding (we fetched + extracted), independent of downstream KB dedupe. The indexer currently marks unchanged content as `skipped` (`webcrawler_indexer.py:341–347`) and cross-connector duplicates as `failed` (`:350–369`) — those still represent a **successful crawl** and should bill. `03c` must count `CrawlOutcomeStatus.SUCCESS`, not `documents_indexed`. Flagging here; final call lives in `03c`.

### Success counter for the indexer

Add an explicit `crawls_succeeded` counter in `index_crawled_urls()` incremented whenever `crawl_url` returns `SUCCESS` (right after line 297's call, before the dedupe/unchanged branches), and surface it in the task-success metadata (lines 455–466) and return value. This is the hand-off point `03c` meters against.

### Chat scrape tool

Drop the Firecrawl-only `formats=["markdown"]` arg (markdown is already the Trafilatura default). The tool keeps returning its asset dict; map the new outcome onto the existing `error` / `content` shape (lines 235–269) with no behavioral change for the agent.

## Work items

1. **Rip out Firecrawl** across the surface table above (crawler, dep, validators, indexer, chat plumbing, automations, tests) — code-only, no env/docs change.
2. **Refactor `crawl_url`** to the 3-tier Scrapling ladder + explicit `CrawlOutcome`; enable `solve_cloudflare` on the stealthy tier.
3. **Add `crawls_succeeded`** counting in `webcrawler_indexer.py` + expose in result/metadata.
4. **Update both `scrape_webpage` tools** to drop `firecrawl_api_key` + `formats`.
5. **Tests:** unit tests for `crawl_url` outcomes (mock Scrapling fetchers → SUCCESS/EMPTY/FAILED); update `test_dependencies.py`; assert the indexer's `crawls_succeeded` count.

## Risks / trade-offs

- **Loss of a managed fallback.** Firecrawl was a hosted last resort for hostile anti-bot sites. Mitigation: `StealthyFetcher` + Cloudflare solving now, `03b` proxy rotation, `03d` captcha solving. Acceptable per the decisions log (single-framework intent).
- **Browser tiers on dev/Windows.** `DynamicFetcher`/`StealthyFetcher` need subprocess support; the existing `to_thread` + `NotImplementedError` guards (lines 134–141, 161–168) are preserved so static-only crawling still works in `uvicorn --reload`.
- **`uv.lock` churn.** Removing `firecrawl-py` requires a lockfile regen + image rebuild; no new runtime deps are added.

## Out of scope (hand-offs)

- Proxy provider expansion + rotation → `03b`.
- Crawl credit metering on `CrawlOutcomeStatus.SUCCESS` → `03c`.
- reCAPTCHA/hCaptcha solving via captcha-tools → `03d` (deferred). Cloudflare Turnstile stays in-framework (Scrapling).
- Whether ad-hoc **chat** scrapes are billed (vs only pipeline crawls) → decided in `03c`.