webclaw/crates/webclaw-server
Valerio 86182ef28a fix(server): switch default browser profile to Firefox
Reddit blocks wreq's Chrome 145 BoringSSL fingerprint at the JA3/JA4
TLS layer even though our HTTP headers correctly impersonate Chrome.
Curl from the same machine with the same Chrome User-Agent string
returns 200 from Reddit's .json endpoint; webclaw with the Chrome
profile returns 403. The detector clearly fingerprints below the
header layer.

Tested all six vertical extractors with the Firefox profile:
reddit, hackernews, github_repo, pypi, npm, huggingface_model all
return correct typed JSON. Firefox is a strict improvement on the
Chrome default for sites with active TLS-level bot detection, with
no regressions on the API-flavored sites that were already working.

Real fix is per-extractor preferred profile, but the structural
change to allow per-call profile selection in FetchClient is a
larger refactor. Flipping the global default is a one-line change
that ships the unblock now and lets users hit the new
/v1/scrape/{vertical} routes against Reddit immediately.
2026-04-22 14:11:55 +02:00
..
src fix(server): switch default browser profile to Firefox 2026-04-22 14:11:55 +02:00
Cargo.toml feat(server): add OSS webclaw-server REST API binary (closes #29) 2026-04-22 12:25:11 +02:00