apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-02 04:08:08 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat: add fallback sitemap paths for broader discovery

Browse source 
Try /sitemap_index.xml, /wp-sitemap.xml, and /sitemap/sitemap-index.xml
after the standard /sitemap.xml. WordPress 5.5+ and many CMS platforms
use non-standard paths that were previously missed. Paths found via
robots.txt are deduplicated to avoid double-fetching.

Bump to 0.3.11.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Valerio 2026-04-10 18:22:57 +02:00
parent 25b6282d5f
commit a4c351d5ae
4 changed files with 39 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Cargo.toml
View file

@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/*"]
[workspace.package]
version = "0.3.10"
version = "0.3.11"
edition = "2024"
license = "AGPL-3.0"
repository = "https://github.com/0xMassi/webclaw"