apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-06-07 22:15:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat: replace custom TLS stack with wreq (BoringSSL), bump v0.3.3

Browse source 
Migrated webclaw-fetch from webclaw-tls (patched rustls/h2/hyper/reqwest)
to wreq by @0x676e67. wreq uses BoringSSL for TLS and the http2 crate
for HTTP/2 fingerprinting — battle-tested with 60+ browser profiles.

This removes all 5 [patch.crates-io] entries that consumers previously
needed. Browser profiles (Chrome 145, Firefox 135, Safari 18, Edge 145)
are now built directly on wreq's Emulation API with correct TLS options,
HTTP/2 SETTINGS ordering, pseudo-header order, and header wire order.

84% pass rate across 1000 real sites. 384 unit tests green.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Valerio 2026-04-01 18:04:55 +02:00
parent 0d0da265ab
commit aaf51eddef
10 changed files with 754 additions and 343 deletions
Download patch file Download diff file Expand all files Collapse all files

4
crates/webclaw-fetch/src/document.rs
View file

@ -34,7 +34,7 @@ impl std::fmt::Display for DocType {
/// Detect document type from response headers or URL extension.
/// Returns `None` for non-document responses (HTML, PDF, etc.).
pub fn is_document_content_type(headers: &webclaw_http::HeaderMap, url: &str) -> Option<DocType> {
pub fn is_document_content_type(headers: &http::HeaderMap, url: &str) -> Option<DocType> {
// Check Content-Type header first
if let Some(ct) = headers.get("content-type").and_then(|v| v.to_str().ok()) {
let mime = ct.split(';').next().unwrap_or("").trim();
@ -474,7 +474,7 @@ fn strip_markdown_formatting(markdown: &str) -> String {
#[cfg(test)]
mod tests {
use super::*;
use webclaw_http::HeaderMap;
use http::HeaderMap;
fn headers_with(name: &str, value: &str) -> HeaderMap {
let mut h = HeaderMap::new();