apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-06-06 22:05:13 +02:00
Code Issues Projects Releases Packages Wiki Activity

perf: reduce fetch timeout to 12s and retries to 2

Browse source 
Stress testing showed 33% of proxies are dead, causing 30s+ timeouts
per request with 3 retries (worst case 94s). Reducing timeout from 30s
to 12s and retries from 3 to 2 brings worst case to 25s. Combined with
disabling 509 dead proxies from the pool, this should significantly
improve response times under load.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Valerio 2026-04-10 17:18:57 +02:00
parent 5ea646a332
commit 954aabe3e8
3 changed files with 11 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

5
crates/webclaw-fetch/src/client.rs
View file

@ -44,7 +44,7 @@ impl Default for FetchConfig {
browser: BrowserProfile::Chrome,
proxy: None,
proxy_pool: Vec::new(),
timeout: Duration::from_secs(30),
timeout: Duration::from_secs(12),
follow_redirects: true,
max_redirects: 10,
headers: HashMap::from([("Accept-Language".to_string(), "en-US,en;q=0.9".to_string())]),
@ -207,13 +207,12 @@ impl FetchClient {
/// Fetch a URL and return the raw HTML + response metadata.
///
/// Automatically retries on transient failures (network errors, 5xx, 429)
/// with exponential backoff: 0s, 1s, 3s (3 attempts total).
/// with exponential backoff: 0s, 1s (2 attempts total).
#[instrument(skip(self), fields(url = %url))]
pub async fn fetch(&self, url: &str) -> Result<FetchResult, FetchError> {
let delays = [
Duration::ZERO,
Duration::from_secs(1),
Duration::from_secs(3),
];
let mut last_err = None;