fix: prevent stack overflow on deeply nested HTML pages

Browse source

Pages like Express.co.uk live blogs nest 200+ DOM levels deep, overflowing
the default 1 MB main-thread stack on Windows during recursive markdown
conversion.

Two-layer fix:

1. markdown.rs: add depth parameter to node_to_md/children_to_md/inline_text
   with MAX_DOM_DEPTH=200 guard — falls back to plain text collection at limit

2. lib.rs: wrap extract_with_options in a worker thread with 8 MB stack so
   html5ever parsing and extraction both have room on deeply nested pages

Tested with Express.co.uk live blog (previously crashed, now extracts 2000+
lines of clean markdown) and drudgereport.com (still works correctly).

...

This commit is contained in:

devnen 2026-04-03 23:45:19 +02:00

parent 95a6681b02

commit 74bac87435

3 changed files with 129 additions and 28 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

15

crates/webclaw-core/testdata/express_test.html vendored Normal file

View file

File diff suppressed because one or more lines are too long