apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-23 07:21:02 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(core): JS-hub page detector + --prefer-articles flag

Browse source 
Detects ESPN-style hub pages (espn.com/nba/, /nfl/, /mlb/, /nhl/, /soccer/)
where the rendered markup has nav-only content with no article bodies —
chrome retry doesn't help because the data genuinely isn't in the markup.
Heuristic: word_count < 500 AND link_count >= 5 against the extracted output.

--prefer-articles: when set, a hub-classified page returns the extracted
link list (reusing the M1 --mode summary machinery) instead of the sparse
body. On non-hub pages, behavior is unchanged.

stderr hint: always emitted on hub detection so the caller knows to drill
/story/_/id/<id>/ URLs from a citation list.

False-positive resistance verified: BBC News /world (link-heavy aggregator,
1500+ words body) and n1info.rs (widget-heavy but content-rich) both
classify as non-hub and emit full extraction.

9 new tests in webclaw-core (317 -> 326).
This commit is contained in:
devnen 2026-05-23 18:55:17 +02:00
parent 339f41bb7c
commit 31a8f6150f
4 changed files with 383 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

2
crates/webclaw-core/src/llm/mod.rs
View file

@ -6,11 +6,13 @@
/// to a deduplicated section at the end.
mod body;
mod cleanup;
mod hub_detect;
mod images;
mod links;
mod metadata;
mod output_size;
pub use hub_detect::{classify as classify_hub, HubClassification};
pub use output_size::{
to_json_summary, to_json_toc, to_llm_summary, to_llm_toc, truncate_json_with_wrapper,
truncate_with_footer,