`search_from = abs_pos + 1` landed mid-char when a rejected match
started on a multi-byte UTF-8 character, panicking on the next
`markdown[search_from..]` slice. Advance by `needle.len()` instead —
always a valid char boundary, and skips the whole rejected match
instead of re-scanning inside it.
Repro: webclaw https://bruler.ru/about_brand -f json
Before: panic "byte index 782 is not a char boundary; it is inside 'Ч'"
After: extracts 2.3KB of clean Cyrillic markdown with 7 sections
Two regression tests cover multi-byte rejected matches and
all-rejected cycles in Cyrillic text.
Closes#16
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Version bump for layout table, stack overflow, and noise filter fixes
contributed by @devnen. Also fixes cargo fmt issues that caused CI lint
failure on the merge commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two related fixes for content being stripped by the noise filter:
1. Remove <form> from unconditional noise tags. ASP.NET and similar
frameworks wrap entire pages in a <form> tag — these are not input
forms. Forms with >500 chars of text are now treated as content
wrappers, not noise.
2. Add safety valve for class/ID noise matching. When malformed HTML
leaves a noise container unclosed (e.g., <div class="header"> missing
its </div>), the HTML5 parser makes all subsequent siblings into
children of that container. A header/nav/footer with >5000 chars of
text is almost certainly a broken wrapper absorbing real content —
exempt it from noise filtering.
