feat(server): add OSS webclaw-server REST API binary (closes #29)
Self-hosters hitting docs/self-hosting were promised three binaries
but the OSS Docker image only shipped two. webclaw-server lived in
the closed-source hosted-platform repo, which couldn't be opened. This
adds a minimal axum REST API in the OSS repo so self-hosting actually
works without pretending to ship the cloud platform.
Crate at crates/webclaw-server/. Stateless, no database, no job queue,
single binary. Endpoints: GET /health, POST /v1/{scrape, crawl, map,
batch, extract, summarize, diff, brand}. JSON shapes mirror
api.webclaw.io for the endpoints OSS can support, so swapping between
self-hosted and hosted is a base-URL change.
Auth: optional bearer token via WEBCLAW_API_KEY / --api-key. Comparison
is constant-time (subtle::ConstantTimeEq). Open mode (no key) is
allowed and binds 127.0.0.1 by default; the Docker image flips
WEBCLAW_HOST=0.0.0.0 so the container is reachable out of the box.
Hard caps to keep naive callers from OOMing the process: crawl capped
at 500 pages synchronously, batch capped at 100 URLs / 20 concurrent.
For unbounded crawls or anti-bot bypass the docs point users at the
hosted API.
Dockerfile + Dockerfile.ci updated to copy webclaw-server into
/usr/local/bin and EXPOSE 3000. Workspace version bumped to 0.4.0
(new public binary).
2026-04-22 12:25:11 +02:00
|
|
|
//! API error type. Maps internal errors to HTTP status codes + JSON.
|
|
|
|
|
|
|
|
|
|
use axum::{
|
|
|
|
|
Json,
|
|
|
|
|
http::StatusCode,
|
|
|
|
|
response::{IntoResponse, Response},
|
|
|
|
|
};
|
|
|
|
|
use serde_json::json;
|
|
|
|
|
use thiserror::Error;
|
|
|
|
|
|
|
|
|
|
/// Public-facing API error. Always serializes as `{ "error": "..." }`.
|
|
|
|
|
/// Keep messages user-actionable; internal details belong in tracing logs.
|
|
|
|
|
///
|
|
|
|
|
/// `Unauthorized` / `NotFound` / `Internal` are kept on the enum as
|
|
|
|
|
/// stable variants for handlers that don't exist yet (planned: per-key
|
|
|
|
|
/// rate-limit responses, dynamic route 404s). Marking them dead-code-OK
|
|
|
|
|
/// is preferable to inventing them later in three places.
|
|
|
|
|
#[allow(dead_code)]
|
|
|
|
|
#[derive(Debug, Error)]
|
|
|
|
|
pub enum ApiError {
|
|
|
|
|
#[error("{0}")]
|
|
|
|
|
BadRequest(String),
|
|
|
|
|
|
|
|
|
|
#[error("unauthorized")]
|
|
|
|
|
Unauthorized,
|
|
|
|
|
|
|
|
|
|
#[error("not found")]
|
|
|
|
|
NotFound,
|
|
|
|
|
|
|
|
|
|
#[error("upstream fetch failed: {0}")]
|
|
|
|
|
Fetch(String),
|
|
|
|
|
|
|
|
|
|
#[error("extraction failed: {0}")]
|
|
|
|
|
Extract(String),
|
|
|
|
|
|
|
|
|
|
#[error("LLM provider error: {0}")]
|
|
|
|
|
Llm(String),
|
|
|
|
|
|
|
|
|
|
#[error("internal: {0}")]
|
|
|
|
|
Internal(String),
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl ApiError {
|
|
|
|
|
pub fn bad_request(msg: impl Into<String>) -> Self {
|
|
|
|
|
Self::BadRequest(msg.into())
|
|
|
|
|
}
|
|
|
|
|
#[allow(dead_code)]
|
|
|
|
|
pub fn internal(msg: impl Into<String>) -> Self {
|
|
|
|
|
Self::Internal(msg.into())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn status(&self) -> StatusCode {
|
|
|
|
|
match self {
|
|
|
|
|
Self::BadRequest(_) => StatusCode::BAD_REQUEST,
|
|
|
|
|
Self::Unauthorized => StatusCode::UNAUTHORIZED,
|
|
|
|
|
Self::NotFound => StatusCode::NOT_FOUND,
|
|
|
|
|
Self::Fetch(_) => StatusCode::BAD_GATEWAY,
|
|
|
|
|
Self::Extract(_) | Self::Llm(_) => StatusCode::UNPROCESSABLE_ENTITY,
|
|
|
|
|
Self::Internal(_) => StatusCode::INTERNAL_SERVER_ERROR,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl IntoResponse for ApiError {
|
|
|
|
|
fn into_response(self) -> Response {
|
|
|
|
|
let body = Json(json!({ "error": self.to_string() }));
|
|
|
|
|
(self.status(), body).into_response()
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl From<webclaw_fetch::FetchError> for ApiError {
|
|
|
|
|
fn from(e: webclaw_fetch::FetchError) -> Self {
|
2026-05-04 11:50:57 +02:00
|
|
|
match e {
|
|
|
|
|
webclaw_fetch::FetchError::InvalidUrl(msg) => {
|
|
|
|
|
Self::BadRequest(format!("invalid url: {msg}"))
|
|
|
|
|
}
|
|
|
|
|
other => Self::Fetch(other.to_string()),
|
|
|
|
|
}
|
feat(server): add OSS webclaw-server REST API binary (closes #29)
Self-hosters hitting docs/self-hosting were promised three binaries
but the OSS Docker image only shipped two. webclaw-server lived in
the closed-source hosted-platform repo, which couldn't be opened. This
adds a minimal axum REST API in the OSS repo so self-hosting actually
works without pretending to ship the cloud platform.
Crate at crates/webclaw-server/. Stateless, no database, no job queue,
single binary. Endpoints: GET /health, POST /v1/{scrape, crawl, map,
batch, extract, summarize, diff, brand}. JSON shapes mirror
api.webclaw.io for the endpoints OSS can support, so swapping between
self-hosted and hosted is a base-URL change.
Auth: optional bearer token via WEBCLAW_API_KEY / --api-key. Comparison
is constant-time (subtle::ConstantTimeEq). Open mode (no key) is
allowed and binds 127.0.0.1 by default; the Docker image flips
WEBCLAW_HOST=0.0.0.0 so the container is reachable out of the box.
Hard caps to keep naive callers from OOMing the process: crawl capped
at 500 pages synchronously, batch capped at 100 URLs / 20 concurrent.
For unbounded crawls or anti-bot bypass the docs point users at the
hosted API.
Dockerfile + Dockerfile.ci updated to copy webclaw-server into
/usr/local/bin and EXPOSE 3000. Workspace version bumped to 0.4.0
(new public binary).
2026-04-22 12:25:11 +02:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl From<webclaw_core::ExtractError> for ApiError {
|
|
|
|
|
fn from(e: webclaw_core::ExtractError) -> Self {
|
|
|
|
|
Self::Extract(e.to_string())
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl From<webclaw_llm::LlmError> for ApiError {
|
|
|
|
|
fn from(e: webclaw_llm::LlmError) -> Self {
|
|
|
|
|
Self::Llm(e.to_string())
|
|
|
|
|
}
|
|
|
|
|
}
|