mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-13 23:15:13 +02:00
fe6e9b5d28
Webclaw's default -t timeout is 30s; slow sites previously sat
silently with no feedback. Now during a fetch, every 10s of elapsed
time webclaw writes one line to stderr:
# webclaw: still fetching <URL> (Ns)
Fetches completing in under 10s emit nothing (the timer never fires).
Stdout output is untouched - pure feedback signal on stderr.
No timeout change. No new flags. Default behavior is augmented at
stderr only.
Implemented via tokio::select! between the fetch future and a
tokio::time::interval. Latency cost: a single tokio task spawn
and a 10s tick - microseconds on the fast path.
10 new tests in webclaw-fetch::progress::tests (none ignored; the
slow-future test uses a 50ms test interval to keep cargo test fast).
Workspace total 710 -> 720.
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| extractors | ||
| browser.rs | ||
| client.rs | ||
| cloud.rs | ||
| crawler.rs | ||
| document.rs | ||
| error.rs | ||
| fetcher.rs | ||
| lib.rs | ||
| linkedin.rs | ||
| locale.rs | ||
| progress.rs | ||
| proxy.rs | ||
| reddit.rs | ||
| sitemap.rs | ||
| tls.rs | ||
| url_security.rs | ||