apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-06-25 03:08:06 +02:00
Code Issues Projects Releases Packages Wiki Activity

polish(fetch,mcp): robots parser + firefox client cache + Acquire ordering (P3) (#23)
Some checks are pending
CI / Test (push) Waiting to run
CI / Lint (push) Waiting to run
CI / Docs (push) Waiting to run

Browse source 
Three P3 items from the 2026-04-16 audit. Bump to 0.3.17.

webclaw-fetch/sitemap.rs: parse_robots_txt used trimmed[..8] slice
plus eq_ignore_ascii_case for the directive test. That was fragile:
"Sitemap :" (space before colon) fell through silently, inline
"# ..." comments leaked into the URL, and a line with no URL at all
returned an empty string. Rewritten to split on the first colon,
match any-case "sitemap" as the directive name, strip comments, and
require `://` in the value. +7 unit tests cover case variants,
space-before-colon, comments, empty values, non-URL values, and
non-sitemap directives.

webclaw-fetch/crawler.rs: is_cancelled uses Ordering::Acquire
instead of Relaxed. Behaviourally equivalent on current hardware for
single-word atomic loads, but the explicit ordering documents intent
for readers + compilers.

webclaw-mcp/server.rs: add lazy OnceLock cache for the Firefox
FetchClient. Tool calls that repeatedly request the firefox profile
without cookies used to build a fresh reqwest pool + TLS stack per
call. Chrome (default) already used the long-lived field; Random is
per-call by design; cookie-bearing requests still build ad-hoc since
the cookie header is part of the client shape.

Tests: 85 webclaw-fetch (was 78, +7 new sitemap), 272 webclaw-core,
43 webclaw-llm, 11 CLI — all green. Clippy clean across workspace.

Refs: docs/AUDIT-2026-04-16.md P3 section

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Valerio 2026-04-16 20:21:32 +02:00 • committed by GitHub
parent d69c50a31d
commit 095ae5d4b1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 153 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

12
Cargo.lock generated
View file

@ -3102,7 +3102,7 @@ dependencies = [
[[package]]
name = "webclaw-cli"
version = "0.3.16"
version = "0.3.17"
dependencies = [
"clap",
"dotenvy",
@ -3123,7 +3123,7 @@ dependencies = [
[[package]]
name = "webclaw-core"
version = "0.3.16"
version = "0.3.17"
dependencies = [
"ego-tree",
"once_cell",
@ -3141,7 +3141,7 @@ dependencies = [
[[package]]
name = "webclaw-fetch"
version = "0.3.16"
version = "0.3.17"
dependencies = [
"bytes",
"calamine",
@ -3163,7 +3163,7 @@ dependencies = [
[[package]]
name = "webclaw-llm"
version = "0.3.16"
version = "0.3.17"
dependencies = [
"async-trait",
"reqwest",
@ -3176,7 +3176,7 @@ dependencies = [
[[package]]
name = "webclaw-mcp"
version = "0.3.16"
version = "0.3.17"
dependencies = [
"dirs",
"dotenvy",
@ -3197,7 +3197,7 @@ dependencies = [
[[package]]
name = "webclaw-pdf"
version = "0.3.16"
version = "0.3.17"
dependencies = [
"pdf-extract",
"thiserror",