mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-07 22:15:12 +02:00
86182ef28a
Reddit blocks wreq's Chrome 145 BoringSSL fingerprint at the JA3/JA4
TLS layer even though our HTTP headers correctly impersonate Chrome.
Curl from the same machine with the same Chrome User-Agent string
returns 200 from Reddit's .json endpoint; webclaw with the Chrome
profile returns 403. The detector clearly fingerprints below the
header layer.
Tested all six vertical extractors with the Firefox profile:
reddit, hackernews, github_repo, pypi, npm, huggingface_model all
return correct typed JSON. Firefox is a strict improvement on the
Chrome default for sites with active TLS-level bot detection, with
no regressions on the API-flavored sites that were already working.
Real fix is per-extractor preferred profile, but the structural
change to allow per-call profile selection in FetchClient is a
larger refactor. Flipping the global default is a one-line change
that ships the unblock now and lets users hit the new
/v1/scrape/{vertical} routes against Reddit immediately.
49 lines
1.5 KiB
Rust
49 lines
1.5 KiB
Rust
//! Shared application state. Cheap to clone via Arc; held by the axum
|
|
//! Router for the life of the process.
|
|
|
|
use std::sync::Arc;
|
|
use webclaw_fetch::{BrowserProfile, FetchClient, FetchConfig};
|
|
|
|
/// Single-process state shared across all request handlers.
|
|
#[derive(Clone)]
|
|
pub struct AppState {
|
|
inner: Arc<Inner>,
|
|
}
|
|
|
|
struct Inner {
|
|
/// Wrapped in `Arc` because `fetch_and_extract_batch_with_options`
|
|
/// (used by the /v1/batch handler) takes `self: &Arc<Self>` so it
|
|
/// can clone the client into spawned tasks. The single-call handlers
|
|
/// auto-deref `&Arc<FetchClient>` -> `&FetchClient`, so this costs
|
|
/// them nothing.
|
|
pub fetch: Arc<FetchClient>,
|
|
pub api_key: Option<String>,
|
|
}
|
|
|
|
impl AppState {
|
|
/// Build the application state. The fetch client is constructed once
|
|
/// and shared across requests so connection pools + browser profile
|
|
/// state don't churn per request.
|
|
pub fn new(api_key: Option<String>) -> anyhow::Result<Self> {
|
|
let config = FetchConfig {
|
|
browser: BrowserProfile::Firefox,
|
|
..FetchConfig::default()
|
|
};
|
|
let fetch = FetchClient::new(config)
|
|
.map_err(|e| anyhow::anyhow!("failed to build fetch client: {e}"))?;
|
|
Ok(Self {
|
|
inner: Arc::new(Inner {
|
|
fetch: Arc::new(fetch),
|
|
api_key,
|
|
}),
|
|
})
|
|
}
|
|
|
|
pub fn fetch(&self) -> &Arc<FetchClient> {
|
|
&self.inner.fetch
|
|
}
|
|
|
|
pub fn api_key(&self) -> Option<&str> {
|
|
self.inner.api_key.as_deref()
|
|
}
|
|
}
|