mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-07 22:15:12 +02:00
feat(server): add OSS webclaw-server REST API binary (closes #29)
Self-hosters hitting docs/self-hosting were promised three binaries
but the OSS Docker image only shipped two. webclaw-server lived in
the closed-source hosted-platform repo, which couldn't be opened. This
adds a minimal axum REST API in the OSS repo so self-hosting actually
works without pretending to ship the cloud platform.
Crate at crates/webclaw-server/. Stateless, no database, no job queue,
single binary. Endpoints: GET /health, POST /v1/{scrape, crawl, map,
batch, extract, summarize, diff, brand}. JSON shapes mirror
api.webclaw.io for the endpoints OSS can support, so swapping between
self-hosted and hosted is a base-URL change.
Auth: optional bearer token via WEBCLAW_API_KEY / --api-key. Comparison
is constant-time (subtle::ConstantTimeEq). Open mode (no key) is
allowed and binds 127.0.0.1 by default; the Docker image flips
WEBCLAW_HOST=0.0.0.0 so the container is reachable out of the box.
Hard caps to keep naive callers from OOMing the process: crawl capped
at 500 pages synchronously, batch capped at 100 URLs / 20 concurrent.
For unbounded crawls or anti-bot bypass the docs point users at the
hosted API.
Dockerfile + Dockerfile.ci updated to copy webclaw-server into
/usr/local/bin and EXPOSE 3000. Workspace version bumped to 0.4.0
(new public binary).
This commit is contained in:
parent
b4bfff120e
commit
2ba682adf3
20 changed files with 1116 additions and 11 deletions
49
crates/webclaw-server/src/state.rs
Normal file
49
crates/webclaw-server/src/state.rs
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
//! Shared application state. Cheap to clone via Arc; held by the axum
|
||||
//! Router for the life of the process.
|
||||
|
||||
use std::sync::Arc;
|
||||
use webclaw_fetch::{BrowserProfile, FetchClient, FetchConfig};
|
||||
|
||||
/// Single-process state shared across all request handlers.
|
||||
#[derive(Clone)]
|
||||
pub struct AppState {
|
||||
inner: Arc<Inner>,
|
||||
}
|
||||
|
||||
struct Inner {
|
||||
/// Wrapped in `Arc` because `fetch_and_extract_batch_with_options`
|
||||
/// (used by the /v1/batch handler) takes `self: &Arc<Self>` so it
|
||||
/// can clone the client into spawned tasks. The single-call handlers
|
||||
/// auto-deref `&Arc<FetchClient>` -> `&FetchClient`, so this costs
|
||||
/// them nothing.
|
||||
pub fetch: Arc<FetchClient>,
|
||||
pub api_key: Option<String>,
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
/// Build the application state. The fetch client is constructed once
|
||||
/// and shared across requests so connection pools + browser profile
|
||||
/// state don't churn per request.
|
||||
pub fn new(api_key: Option<String>) -> anyhow::Result<Self> {
|
||||
let config = FetchConfig {
|
||||
browser: BrowserProfile::Chrome,
|
||||
..FetchConfig::default()
|
||||
};
|
||||
let fetch = FetchClient::new(config)
|
||||
.map_err(|e| anyhow::anyhow!("failed to build fetch client: {e}"))?;
|
||||
Ok(Self {
|
||||
inner: Arc::new(Inner {
|
||||
fetch: Arc::new(fetch),
|
||||
api_key,
|
||||
}),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn fetch(&self) -> &Arc<FetchClient> {
|
||||
&self.inner.fetch
|
||||
}
|
||||
|
||||
pub fn api_key(&self) -> Option<&str> {
|
||||
self.inner.api_key.as_deref()
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue