mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-05-16 18:35:12 +02:00
70c67f2ed6
Two related fixes for content being stripped by the noise filter: 1. Remove <form> from unconditional noise tags. ASP.NET and similar frameworks wrap entire pages in a <form> tag — these are not input forms. Forms with >500 chars of text are now treated as content wrappers, not noise. 2. Add safety valve for class/ID noise matching. When malformed HTML leaves a noise container unclosed (e.g., <div class="header"> missing its </div>), the HTML5 parser makes all subsequent siblings into children of that container. A header/nav/footer with >5000 chars of text is almost certainly a broken wrapper absorbing real content — exempt it from noise filtering. |
||
|---|---|---|
| .. | ||
| llm | ||
| brand.rs | ||
| data_island.rs | ||
| diff.rs | ||
| domain.rs | ||
| error.rs | ||
| extractor.rs | ||
| js_eval.rs | ||
| lib.rs | ||
| markdown.rs | ||
| metadata.rs | ||
| noise.rs | ||
| structured_data.rs | ||
| types.rs | ||
| youtube.rs | ||