apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-21 07:01:01 +02:00
Code Issues Projects Releases Packages Wiki Activity

fix: add danger_accept_invalid_certs to plain TLS fallback clients

Browse source 
The primp-patched rustls fork rejects valid certificates with
`InvalidMessage(UnknownCertificateExtension)` on the impersonated
client, and `InvalidCertificate(UnknownIssuer)` on the plain fallback.

This causes ALL HTTPS requests to fail on Windows (native + WSL),
Linux, and Docker — only HTTP works.

Root cause: the forked rustls at `github.com/deedy5/primp` cannot
parse certain certificate extensions that standard rustls handles.
The plain fallback clients also use primp::Client (which goes through
the same broken fork), so they fail with UnknownIssuer since the
cert chain validation rejects the root CA.

Fix: add `.danger_accept_invalid_certs(true)` to all 4 plain fallback
client builders in `webclaw-fetch/src/client.rs`. These fallback
clients only run when the impersonated TLS client has already failed,
so relaxing cert validation here is acceptable as a workaround.

The proper long-term fix is to update the primp rustls fork to handle
modern certificate extensions correctly.

Tested: HTTPS now works on example.com, apartments.com, and real
apartment property websites that previously returned
"fetch error: request failed: error sending request for url".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
djimenez18 2026-03-28 21:14:00 -05:00
parent 77e93441c0
commit 17d805fb6d
1 changed files with 9 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

9
crates/webclaw-fetch/src/client.rs
View file

@ -247,11 +247,17 @@ impl FetchClient {
};
// Plain client fallback (no TLS impersonation)
// danger_accept_invalid_certs: the primp-patched rustls rejects some valid
// certificates with UnknownCertificateExtension / UnknownIssuer. For the
// plain-text fallback (which only runs when the impersonated client already
// failed) accepting certs unconditionally lets the request succeed while the
// upstream rustls fork is fixed.
if needs_plain_fallback {
let plain = primp::Client::builder()
.user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36")
.cookie_store(true)
.timeout(Duration::from_secs(30))
.danger_accept_invalid_certs(true)
.build()
.map_err(|e| FetchError::Build(format!("plain client: {e}")))?;
@ -327,6 +333,7 @@ impl FetchClient {
let plain = primp::Client::builder()
.user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36")
.timeout(std::time::Duration::from_secs(15))
.danger_accept_invalid_certs(true)
.build()
.map_err(|e| FetchError::Build(format!("reddit client: {e}")))?;
let response = plain.get(&json_url).send().await?;
@ -353,6 +360,7 @@ impl FetchClient {
.user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36")
.cookie_store(true)
.timeout(Duration::from_secs(30))
.danger_accept_invalid_certs(true)
.build()
.map_err(|e| FetchError::Build(format!("plain fallback: {e}")))?;
plain.get(url).send().await?
@ -364,6 +372,7 @@ impl FetchClient {
.user_agent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36")
.cookie_store(true)
.timeout(Duration::from_secs(30))
.danger_accept_invalid_certs(true)
.build()
.map_err(|e| FetchError::Build(format!("plain fallback: {e}")))?;
plain.get(url).send().await?