mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-05-29 20:45:12 +02:00
feat: add allow_subdomains and allow_external_links to CrawlConfig
Crawls are same-origin by default. Enable allow_subdomains to follow sibling/child subdomains (blog.example.com from example.com), or allow_external_links for full cross-origin crawling. Root domain extraction uses a heuristic that handles two-part TLDs (co.uk, com.au). Includes 5 unit tests for root_domain(). Bump to 0.3.12. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
a4c351d5ae
commit
050b2ef463
7 changed files with 109 additions and 17 deletions
|
|
@ -1218,6 +1218,8 @@ async fn run_crawl(cli: &Cli) -> Result<(), String> {
|
|||
exclude_patterns,
|
||||
progress_tx: Some(progress_tx),
|
||||
cancel_flag: Some(Arc::clone(&cancel_flag)),
|
||||
allow_subdomains: false,
|
||||
allow_external_links: false,
|
||||
};
|
||||
|
||||
// Load resume state if --crawl-state file exists
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue