feat(fetch): Fetcher trait so vertical extractors work under any HTTP backend

Adds `webclaw_fetch::Fetcher` trait. All 28 vertical extractors now
take `client: &dyn Fetcher` instead of `client: &FetchClient` directly.
Backwards-compatible: FetchClient implements Fetcher, blanket impls
cover `&T` and `Arc<T>`, so existing CLI / MCP / self-hosted-server
callers keep working unchanged.

Motivation: the production API server (api.webclaw.io) must not do
in-process TLS fingerprinting; it delegates all HTTP to the Go
tls-sidecar. Before this trait, exposing /v1/scrape/{vertical} on
production would have required importing wreq into the server's
dep graph, violating the CLAUDE.md rule. Now production can provide
its own TlsSidecarFetcher implementation and pass it to the same
dispatcher the OSS server uses.

Changes:
- New `crates/webclaw-fetch/src/fetcher.rs` defining the trait plus
  blanket impls for `&T` and `Arc<T>`.
- `FetchClient` gains a tiny impl block in client.rs that forwards to
  its existing public methods.
- All 28 extractor signatures migrated from `&FetchClient` to
  `&dyn Fetcher` (sed-driven bulk rewrite, no semantic change).
- `cloud::smart_fetch` and `cloud::smart_fetch_html` take `&dyn Fetcher`.
- `extractors::dispatch_by_url` and `extractors::dispatch_by_name`
  take `&dyn Fetcher`.
- `async-trait 0.1` added to webclaw-fetch deps (Rust 1.75+ has
  native async-fn-in-trait but dyn dispatch still needs async_trait).
- Version bumped to 0.5.1, CHANGELOG updated.

Tests: 215 passing in webclaw-fetch (no new tests needed — the existing
extractor tests exercise the trait methods transparently).
Clippy: clean workspace-wide.
This commit is contained in:
Valerio 2026-04-22 21:17:50 +02:00
parent aaa5103504
commit 058493bc8f
37 changed files with 241 additions and 73 deletions

15
Cargo.lock generated
View file

@ -3199,7 +3199,7 @@ dependencies = [
[[package]]
name = "webclaw-cli"
version = "0.5.0"
version = "0.5.1"
dependencies = [
"clap",
"dotenvy",
@ -3220,7 +3220,7 @@ dependencies = [
[[package]]
name = "webclaw-core"
version = "0.5.0"
version = "0.5.1"
dependencies = [
"ego-tree",
"once_cell",
@ -3238,8 +3238,9 @@ dependencies = [
[[package]]
name = "webclaw-fetch"
version = "0.5.0"
version = "0.5.1"
dependencies = [
"async-trait",
"bytes",
"calamine",
"http",
@ -3262,7 +3263,7 @@ dependencies = [
[[package]]
name = "webclaw-llm"
version = "0.5.0"
version = "0.5.1"
dependencies = [
"async-trait",
"reqwest",
@ -3275,7 +3276,7 @@ dependencies = [
[[package]]
name = "webclaw-mcp"
version = "0.5.0"
version = "0.5.1"
dependencies = [
"dirs",
"dotenvy",
@ -3295,7 +3296,7 @@ dependencies = [
[[package]]
name = "webclaw-pdf"
version = "0.5.0"
version = "0.5.1"
dependencies = [
"pdf-extract",
"thiserror",
@ -3304,7 +3305,7 @@ dependencies = [
[[package]]
name = "webclaw-server"
version = "0.5.0"
version = "0.5.1"
dependencies = [
"anyhow",
"axum",