mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-07-11 05:22:10 +02:00
feat(fetch,llm): DoS hardening via response caps + glob validation (P2)
Response body caps: - webclaw-fetch::Response::from_wreq now rejects bodies over 50 MB. Checks Content-Length up front (before the allocation) and the actual .bytes() length after (belt-and-braces against lying upstreams). Previously the HTML -> markdown conversion downstream could allocate multiple String copies per page; a 100 MB page would OOM the process. - webclaw-llm providers (anthropic/openai/ollama) share a new response_json_capped helper with a 5 MB cap. Protects against a malicious or runaway provider response exhausting memory. Crawler frontier cap: after each BFS depth level the frontier is truncated to max(max_pages * 10, 100) entries, keeping the most recently discovered links. Dense pages (tag clouds, search results) used to push the frontier into the tens of thousands even after max_pages halted new fetches. Glob pattern validation: user-supplied include_patterns / exclude_patterns are rejected at Crawler::new if they contain more than 4 `**` wildcards or exceed 1024 chars. The backtracking matcher degrades exponentially on deeply-nested `**` against long paths. Cleanup: - Removed blanket #![allow(dead_code)] from webclaw-cli/src/main.rs; no warnings surfaced, the suppression was obsolete. - core/.gitignore: replaced overbroad *.json with specific local- artifact patterns (previous rule would have swallowed package.json, components.json, .smithery/*.json). Tests: +4 validate_glob tests. Full workspace test: 283 passed (webclaw-core + webclaw-fetch + webclaw-llm). Version: 0.3.15 -> 0.3.16 CHANGELOG updated. Refs: docs/AUDIT-2026-04-16.md (P2 section) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
7773c8af2a
commit
98c30365a2
13 changed files with 756 additions and 13 deletions
7
.gitignore
vendored
7
.gitignore
vendored
|
|
@ -1,6 +1,11 @@
|
|||
target/
|
||||
.DS_Store
|
||||
.env
|
||||
.env.*
|
||||
proxies.txt
|
||||
.claude/skills/
|
||||
*.json
|
||||
# Scratch / local artifacts (previously covered by overbroad `*.json`,
|
||||
# which would have also swallowed package.json, components.json,
|
||||
# .smithery/*.json if they were ever modified).
|
||||
*.local.json
|
||||
local-test-results.json
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue