3 commits

Author	SHA1	Message	Date
Valerio	058493bc8f	feat(fetch): Fetcher trait so vertical extractors work under any HTTP backend ... Adds `webclaw_fetch::Fetcher` trait. All 28 vertical extractors now take `client: &dyn Fetcher` instead of `client: &FetchClient` directly. Backwards-compatible: FetchClient implements Fetcher, blanket impls cover `&T` and `Arc<T>`, so existing CLI / MCP / self-hosted-server callers keep working unchanged. Motivation: the production API server (api.webclaw.io) must not do in-process TLS fingerprinting; it delegates all HTTP to the Go tls-sidecar. Before this trait, exposing /v1/scrape/{vertical} on production would have required importing wreq into the server's dep graph, violating the CLAUDE.md rule. Now production can provide its own TlsSidecarFetcher implementation and pass it to the same dispatcher the OSS server uses. Changes: - New `crates/webclaw-fetch/src/fetcher.rs` defining the trait plus blanket impls for `&T` and `Arc<T>`. - `FetchClient` gains a tiny impl block in client.rs that forwards to its existing public methods. - All 28 extractor signatures migrated from `&FetchClient` to `&dyn Fetcher` (sed-driven bulk rewrite, no semantic change). - `cloud::smart_fetch` and `cloud::smart_fetch_html` take `&dyn Fetcher`. - `extractors::dispatch_by_url` and `extractors::dispatch_by_name` take `&dyn Fetcher`. - `async-trait 0.1` added to webclaw-fetch deps (Rust 1.75+ has native async-fn-in-trait but dyn dispatch still needs async_trait). - Version bumped to 0.5.1, CHANGELOG updated. Tests: 215 passing in webclaw-fetch (no new tests needed — the existing extractor tests exercise the trait methods transparently). Clippy: clean workspace-wide.	2026-04-22 21:17:50 +02:00
Valerio	7f5eb93b65	feat(extractors): wave 6b, etsy_listing + HTML fallbacks for substack/youtube ... Adds etsy_listing and hardens two existing extractors with HTML fallbacks so transient API failures still return useful data. New: - etsy_listing: /listing/{id}(/slug) with Schema.org Product JSON-LD + OG fallback. Antibot-gated, routes through cloud::smart_fetch_html like amazon_product and ebay_listing. Auto-dispatched (etsy host is unique). Hardened: - substack_post: when /api/v1/posts/{slug} returns non-200 (rate limit, 403 on hardened custom domains, 5xx), fall back to HTML fetch and parse OG tags + Article JSON-LD. Response shape is stable across both paths, with a `data_source` field of "api" or "html_fallback". - youtube_video: when ytInitialPlayerResponse is missing (EU-consent interstitial, age-gated, some live pre-shows), fall back to OG tags for title/description/thumbnail. `data_source` now "player_response" or "og_fallback". Tests: 91 passing in webclaw-fetch (9 new), clippy clean.	2026-04-22 16:44:51 +02:00
Valerio	8cc727c2f2	feat(extractors): wave 6a, 5 easy verticals (27 total) ... Adds 5 structured extractors that hit public APIs with stable shapes: - github_issue: /repos/{o}/{r}/issues/{n} (rejects PRs, points to github_pr) - shopify_collection: /collections/{handle}.json + products.json - woocommerce_product: /wp-json/wc/store/v1/products?slug={slug} - substack_post: /api/v1/posts/{slug} (works on custom domains too) - youtube_video: ytInitialPlayerResponse blob from /watch HTML Auto-dispatched: github_issue, youtube_video (unique hosts and stable URL shapes). Explicit-call: shopify_collection, woocommerce_product, substack_post (URL shapes overlap with non-target sites). Tests: 82 total passing in webclaw-fetch (12 new), clippy clean.	2026-04-22 16:33:35 +02:00