webclaw/crates
Valerio a4c351d5ae feat: add fallback sitemap paths for broader discovery
Try /sitemap_index.xml, /wp-sitemap.xml, and /sitemap/sitemap-index.xml
after the standard /sitemap.xml. WordPress 5.5+ and many CMS platforms
use non-standard paths that were previously missed. Paths found via
robots.txt are deduplicated to avoid double-fetching.

Bump to 0.3.11.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 18:22:57 +02:00
..
webclaw-cli feat: CLI --research flag + MCP cloud fallback + structured research output 2026-04-03 14:04:04 +02:00
webclaw-core fix: resolve clippy warnings from #14 (collapsible_if, manual_inspect) 2026-04-04 15:28:59 +02:00
webclaw-fetch feat: add fallback sitemap paths for broader discovery 2026-04-10 18:22:57 +02:00
webclaw-llm Initial release: webclaw v0.1.0 — web content extraction for LLMs 2026-03-23 18:31:11 +01:00
webclaw-mcp fix: MCP research saves to file, returns compact response 2026-04-03 16:05:45 +02:00
webclaw-pdf Initial release: webclaw v0.1.0 — web content extraction for LLMs 2026-03-23 18:31:11 +01:00