mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-07 22:15:12 +02:00
aaf51eddef
Migrated webclaw-fetch from webclaw-tls (patched rustls/h2/hyper/reqwest) to wreq by @0x676e67. wreq uses BoringSSL for TLS and the http2 crate for HTTP/2 fingerprinting — battle-tested with 60+ browser profiles. This removes all 5 [patch.crates-io] entries that consumers previously needed. Browser profiles (Chrome 145, Firefox 135, Safari 18, Edge 145) are now built directly on wreq's Emulation API with correct TLS options, HTTP/2 SETTINGS ordering, pseudo-header order, and header wire order. 84% pass rate across 1000 real sites. 384 unit tests green. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
22 lines
754 B
Rust
22 lines
754 B
Rust
//! webclaw-fetch: HTTP client layer with browser TLS fingerprint impersonation.
|
|
//! Uses wreq (BoringSSL) for browser-grade TLS + HTTP/2 fingerprinting.
|
|
//! Automatically detects PDF responses and delegates to webclaw-pdf.
|
|
pub mod browser;
|
|
pub mod client;
|
|
pub mod crawler;
|
|
pub mod document;
|
|
pub mod error;
|
|
pub mod linkedin;
|
|
pub mod proxy;
|
|
pub mod reddit;
|
|
pub mod sitemap;
|
|
pub mod tls;
|
|
|
|
pub use browser::BrowserProfile;
|
|
pub use client::{BatchExtractResult, BatchResult, FetchClient, FetchConfig, FetchResult};
|
|
pub use crawler::{CrawlConfig, CrawlResult, CrawlState, Crawler, PageResult};
|
|
pub use error::FetchError;
|
|
pub use proxy::{parse_proxy_file, parse_proxy_line};
|
|
pub use sitemap::SitemapEntry;
|
|
pub use http::HeaderMap;
|
|
pub use webclaw_pdf::PdfMode;
|