2026-03-29 16:40:10 +02:00
|
|
|
//! webclaw-fetch: HTTP client layer with browser TLS fingerprint impersonation.
|
2026-04-01 18:04:55 +02:00
|
|
|
//! Uses wreq (BoringSSL) for browser-grade TLS + HTTP/2 fingerprinting.
|
2026-03-29 16:40:10 +02:00
|
|
|
//! Automatically detects PDF responses and delegates to webclaw-pdf.
|
2026-03-23 18:31:11 +01:00
|
|
|
pub mod browser;
|
|
|
|
|
pub mod client;
|
|
|
|
|
pub mod crawler;
|
2026-03-26 15:28:23 +01:00
|
|
|
pub mod document;
|
2026-03-23 18:31:11 +01:00
|
|
|
pub mod error;
|
feat(extractors): add vertical extractors module + first 6 verticals
New extractors module returns site-specific typed JSON instead of
generic markdown. Each extractor:
- declares a URL pattern via matches()
- fetches from the site's official JSON API where one exists
- returns a typed serde_json::Value with documented field names
- exposes an INFO struct that powers the /v1/extractors catalog
First 6 verticals shipped, all hitting public JSON APIs (no HTML
scraping, zero antibot risk):
- reddit → www.reddit.com/*/.json
- hackernews → hn.algolia.com/api/v1/items/{id} (full thread in one call)
- github_repo → api.github.com/repos/{owner}/{repo}
- pypi → pypi.org/pypi/{name}/json
- npm → registry.npmjs.org/{name} + downloads/point/last-week
- huggingface_model → huggingface.co/api/models/{owner}/{name}
Server-side routes added:
- POST /v1/scrape/{vertical} explicit per-vertical extraction
- GET /v1/extractors catalog (name, label, description, url_patterns)
The dispatcher validates that URL matches the requested vertical
before running, so users get "URL doesn't match the X extractor"
instead of opaque parse failures inside the extractor.
17 unit tests cover URL matching + path parsing for each vertical.
Live tests against canonical URLs (rust-lang/rust, requests pypi,
react npm, whisper-large-v3 hf, item 8863 hn, an r/micro_saas post)
all return correct typed JSON in 100-300ms. Sample sizes: github
863B, npm 700B, pypi 1.7KB, hf 3.2KB, hn 38KB (full comment tree).
Marketing positioning: Firecrawl charges 5 credits per /extract call
and you write the schema. Webclaw returns the same JSON in 1 credit
per /scrape/{vertical} call with hand-written deterministic
extractors per site.
2026-04-22 14:11:43 +02:00
|
|
|
pub mod extractors;
|
2026-03-29 16:54:35 +02:00
|
|
|
pub mod linkedin;
|
2026-03-23 18:31:11 +01:00
|
|
|
pub mod proxy;
|
2026-03-29 16:54:35 +02:00
|
|
|
pub mod reddit;
|
2026-03-23 18:31:11 +01:00
|
|
|
pub mod sitemap;
|
2026-04-01 18:04:55 +02:00
|
|
|
pub mod tls;
|
2026-03-23 18:31:11 +01:00
|
|
|
|
|
|
|
|
pub use browser::BrowserProfile;
|
|
|
|
|
pub use client::{BatchExtractResult, BatchResult, FetchClient, FetchConfig, FetchResult};
|
2026-03-25 21:38:28 +01:00
|
|
|
pub use crawler::{CrawlConfig, CrawlResult, CrawlState, Crawler, PageResult};
|
2026-03-23 18:31:11 +01:00
|
|
|
pub use error::FetchError;
|
2026-04-01 18:25:40 +02:00
|
|
|
pub use http::HeaderMap;
|
2026-03-23 18:31:11 +01:00
|
|
|
pub use proxy::{parse_proxy_file, parse_proxy_line};
|
|
|
|
|
pub use sitemap::SitemapEntry;
|
|
|
|
|
pub use webclaw_pdf::PdfMode;
|