mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-07-06 04:32:11 +02:00
217bfe088b
Reddit blocked unauthenticated `.json` access, so the previous extractor returned block pages or timed out on every thread. Switch to parsing old.reddit.com's server-rendered HTML, which needs no API key or JS. Fetch layer: - Rewrite every Reddit host to old.reddit.com before fetching; drop all `.json` URL handling and the JSON response parser. Extraction (webclaw-core::reddit): - New HTML parser producing a typed post + nested comment tree. - Comments nest structurally (.comment > .child > .sitetable > .comment); old.reddit omits a usable depth attribute, so the tree is walked recursively. Bodies live in .entry > form > .usertext-body > .md. - Post metadata: title, author, subreddit, score, comment count (data-comments-count), self-vs-link (self class / self.* domain), flair, self-text body. - Comment scores read the .score.unvoted title (the displayed value, not the ±1 vote-state siblings); hidden scores are None, not 0. - Deleted comments are kept in place so their replies aren't orphaned; "load more comments" stubs are skipped. Markdown output: - Reply nesting via blockquote depth (avoids 4-space indentation turning text and code fences into broken indented-code blocks). - Links keep their target as [text](url); root-relative reddit links resolve against old.reddit.com. Nested lists indent correctly. - A recognised but unparseable /comments/ page returns no content rather than falling through to generic extraction of Reddit chrome. Tests: regression suite runs against real old.reddit.com fixtures (testdata/reddit/), the ground truth that surfaced the parsing and markdown bugs synthetic HTML had hidden. Fixtures are excluded from the published crate.
66 lines
2.1 KiB
Rust
66 lines
2.1 KiB
Rust
//! Reddit structured extractor — parses old.reddit.com HTML.
|
|
//!
|
|
//! Fetches old.reddit.com (stable server-rendered HTML, no JS required)
|
|
//! and delegates parsing to `webclaw_core::reddit`. Returns a typed JSON
|
|
//! value with `{ url, post, comments }` structure.
|
|
|
|
use serde_json::Value;
|
|
|
|
use super::ExtractorInfo;
|
|
use crate::error::FetchError;
|
|
use crate::fetcher::Fetcher;
|
|
|
|
pub const INFO: ExtractorInfo = ExtractorInfo {
|
|
name: "reddit",
|
|
label: "Reddit thread",
|
|
description: "Returns post + nested comment tree with scores, authors, and timestamps.",
|
|
url_patterns: &[
|
|
"https://www.reddit.com/r/*/comments/*",
|
|
"https://reddit.com/r/*/comments/*",
|
|
"https://old.reddit.com/r/*/comments/*",
|
|
],
|
|
};
|
|
|
|
pub fn matches(url: &str) -> bool {
|
|
webclaw_core::reddit::is_reddit_url(url) && url.contains("/comments/")
|
|
}
|
|
|
|
pub async fn extract(client: &dyn Fetcher, url: &str) -> Result<Value, FetchError> {
|
|
let fetch_url = crate::reddit::to_old_reddit_url(url);
|
|
let resp = client.fetch(&fetch_url).await?;
|
|
if resp.status != 200 {
|
|
return Err(FetchError::Build(format!(
|
|
"reddit: unexpected status {}",
|
|
resp.status
|
|
)));
|
|
}
|
|
|
|
let thread = webclaw_core::reddit::try_extract_thread(&resp.html, url).ok_or_else(|| {
|
|
FetchError::BodyDecode(
|
|
"reddit: page structure not recognised — is this a thread URL?".into(),
|
|
)
|
|
})?;
|
|
|
|
serde_json::to_value(&thread)
|
|
.map_err(|e| FetchError::BodyDecode(format!("reddit: serialisation error: {e}")))
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn matches_thread_urls() {
|
|
assert!(matches(
|
|
"https://www.reddit.com/r/rust/comments/abc123/some_title/"
|
|
));
|
|
assert!(matches("https://old.reddit.com/r/rust/comments/abc123/x/"));
|
|
assert!(matches("https://reddit.com/r/rust/comments/abc/x"));
|
|
}
|
|
|
|
#[test]
|
|
fn rejects_listing_and_non_reddit() {
|
|
assert!(!matches("https://www.reddit.com/r/rust"));
|
|
assert!(!matches("https://example.com/r/rust/comments/abc/x"));
|
|
}
|
|
}
|