mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-10 22:45:13 +02:00
74bac87435
Pages like Express.co.uk live blogs nest 200+ DOM levels deep, overflowing the default 1 MB main-thread stack on Windows during recursive markdown conversion. Two-layer fix: 1. markdown.rs: add depth parameter to node_to_md/children_to_md/inline_text with MAX_DOM_DEPTH=200 guard — falls back to plain text collection at limit 2. lib.rs: wrap extract_with_options in a worker thread with 8 MB stack so html5ever parsing and extraction both have room on deeply nested pages Tested with Express.co.uk live blog (previously crashed, now extracts 2000+ lines of clean markdown) and drudgereport.com (still works correctly). |
||
|---|---|---|
| .. | ||
| llm | ||
| brand.rs | ||
| data_island.rs | ||
| diff.rs | ||
| domain.rs | ||
| error.rs | ||
| extractor.rs | ||
| js_eval.rs | ||
| lib.rs | ||
| markdown.rs | ||
| metadata.rs | ||
| noise.rs | ||
| structured_data.rs | ||
| types.rs | ||
| youtube.rs | ||