apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-06-23 02:48:06 +02:00
Code Issues Projects Releases Packages Wiki Activity

fix(mcp): vertical_scrape uses Firefox profile, not default Chrome
Some checks are pending
CI / Test (push) Waiting to run
CI / Lint (push) Waiting to run
CI / Docs (push) Waiting to run

Browse source 
Reddit's .json API rejects the wreq-Chrome TLS fingerprint with a
403 even from residential IPs. Their block list includes known
browser-emulation library fingerprints. wreq-Firefox passes. The
CLI `vertical` subcommand already forced Firefox; MCP
`vertical_scrape` was still falling back to the long-lived
`self.fetch_client` which defaults to Chrome, so reddit failed
on MCP and nobody noticed because the earlier test runs all had
an API key set that masked the issue.

Switched vertical_scrape to reuse `self.firefox_or_build()` which
gives us the cached Firefox client (same pattern the scrape tool
uses when the caller requests `browser: firefox`). Firefox is
strictly-safer-than-Chrome for every vertical in the catalog, so
making it the hard default for `vertical_scrape` is the right call.

Verified end-to-end from a clean shell with no WEBCLAW_API_KEY:
- MCP reddit: 679ms, post/author/6 comments correct
- MCP instagram_profile: 1157ms, 18471 followers

No change to the `scrape` tool -- it keeps the user-selectable
browser param.

Bumps version to 0.5.3.
This commit is contained in:
Valerio 2026-04-22 23:18:11 +02:00
parent 0daa2fec1a
commit 4bf11d902f
3 changed files with 23 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

14
Cargo.lock generated
View file

@ -3199,7 +3199,7 @@ dependencies = [
[[package]]
name = "webclaw-cli"
version = "0.5.2"
version = "0.5.3"
dependencies = [
"clap",
"dotenvy",
@ -3220,7 +3220,7 @@ dependencies = [
[[package]]
name = "webclaw-core"
version = "0.5.2"
version = "0.5.3"
dependencies = [
"ego-tree",
"once_cell",
@ -3238,7 +3238,7 @@ dependencies = [
[[package]]
name = "webclaw-fetch"
version = "0.5.2"
version = "0.5.3"
dependencies = [
"async-trait",
"bytes",
@ -3263,7 +3263,7 @@ dependencies = [
[[package]]
name = "webclaw-llm"
version = "0.5.2"
version = "0.5.3"
dependencies = [
"async-trait",
"reqwest",
@ -3276,7 +3276,7 @@ dependencies = [
[[package]]
name = "webclaw-mcp"
version = "0.5.2"
version = "0.5.3"
dependencies = [
"dirs",
"dotenvy",
@ -3296,7 +3296,7 @@ dependencies = [
[[package]]
name = "webclaw-pdf"
version = "0.5.2"
version = "0.5.3"
dependencies = [
"pdf-extract",
"thiserror",
@ -3305,7 +3305,7 @@ dependencies = [
[[package]]
name = "webclaw-server"
version = "0.5.2"
version = "0.5.3"
dependencies = [
"anyhow",
"axum",