apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-04-25 00:06:21 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

feat(server): add OSS webclaw-server REST API binary (closes #29)

Browse source 
Self-hosters hitting docs/self-hosting were promised three binaries
but the OSS Docker image only shipped two. webclaw-server lived in
the closed-source hosted-platform repo, which couldn't be opened. This
adds a minimal axum REST API in the OSS repo so self-hosting actually
works without pretending to ship the cloud platform.

Crate at crates/webclaw-server/. Stateless, no database, no job queue,
single binary. Endpoints: GET /health, POST /v1/{scrape, crawl, map,
batch, extract, summarize, diff, brand}. JSON shapes mirror
api.webclaw.io for the endpoints OSS can support, so swapping between
self-hosted and hosted is a base-URL change.

Auth: optional bearer token via WEBCLAW_API_KEY / --api-key. Comparison
is constant-time (subtle::ConstantTimeEq). Open mode (no key) is
allowed and binds 127.0.0.1 by default; the Docker image flips
WEBCLAW_HOST=0.0.0.0 so the container is reachable out of the box.

Hard caps to keep naive callers from OOMing the process: crawl capped
at 500 pages synchronously, batch capped at 100 URLs / 20 concurrent.
For unbounded crawls or anti-bot bypass the docs point users at the
hosted API.

Dockerfile + Dockerfile.ci updated to copy webclaw-server into
/usr/local/bin and EXPOSE 3000. Workspace version bumped to 0.4.0
(new public binary).
This commit is contained in:
Valerio 2026-04-22 12:25:11 +02:00
parent b4bfff120e
commit 2ba682adf3
20 changed files with 1116 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

15
CLAUDE.md
View file

@ -20,9 +20,11 @@ webclaw/
webclaw-pdf/ # PDF text extraction via pdf-extract
webclaw-mcp/ # MCP server (Model Context Protocol) for AI agents
webclaw-cli/ # CLI binary
webclaw-server/ # Minimal axum REST API (self-hosting; OSS counterpart
# of api.webclaw.io, without anti-bot / JS / jobs / auth)
```
Two binaries: `webclaw` (CLI), `webclaw-mcp` (MCP server).
Three binaries: `webclaw` (CLI), `webclaw-mcp` (MCP server), `webclaw-server` (REST API for self-hosting).
### Core Modules (`webclaw-core`)
- `extractor.rs` — Readability-style scoring: text density, semantic tags, link density penalty
@ -60,6 +62,17 @@ Two binaries: `webclaw` (CLI), `webclaw-mcp` (MCP server).
- Works with Claude Desktop, Claude Code, and any MCP client
- Uses `rmcp` crate (official Rust MCP SDK)
### REST API Server (`webclaw-server`)
- Axum 0.8, stateless, no database, no job queue
- 8 POST routes + /health, JSON shapes mirror api.webclaw.io where the
capability exists in OSS
- Constant-time bearer-token auth via `subtle::ConstantTimeEq` when
`--api-key` / `WEBCLAW_API_KEY` is set; otherwise open mode
- Hard caps: crawl ≤ 500 pages, batch ≤ 100 URLs, 20 concurrent
- Does NOT include: anti-bot bypass, JS rendering, async jobs,
multi-tenant auth, billing, proxy rotation, search/research/watch/
agent-scrape. Those live behind api.webclaw.io and are closed-source.
## Hard Rules
- **Core has ZERO network dependencies** — takes `&str` HTML, returns structured output. Keep it WASM-compatible.

130
Cargo.lock generated
View file

@ -182,6 +182,70 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
[[package]]
name = "axum"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90"
dependencies = [
"axum-core",
"axum-macros",
"bytes",
"form_urlencoded",
"futures-util",
"http",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"itoa",
"matchit",
"memchr",
"mime",
"percent-encoding",
"pin-project-lite",
"serde_core",
"serde_json",
"serde_path_to_error",
"serde_urlencoded",
"sync_wrapper",
"tokio",
"tower",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "axum-core"
version = "0.5.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1"
dependencies = [
"bytes",
"futures-core",
"http",
"http-body",
"http-body-util",
"mime",
"pin-project-lite",
"sync_wrapper",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "axum-macros"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7aa268c23bfbbd2c4363b9cd302a4f504fb2a9dfe7e3451d66f35dd392e20aca"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "base64"
version = "0.22.1"
@ -1132,6 +1196,12 @@ version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
[[package]]
name = "httpdate"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
[[package]]
name = "hyper"
version = "1.9.0"
@ -1145,6 +1215,7 @@ dependencies = [
"http",
"http-body",
"httparse",
"httpdate",
"itoa",
"pin-project-lite",
"smallvec",
@ -1559,6 +1630,12 @@ dependencies = [
"regex-automata",
]
[[package]]
name = "matchit"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3"
[[package]]
name = "md-5"
version = "0.10.6"
@ -1575,6 +1652,12 @@ version = "2.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
[[package]]
name = "mime"
version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
[[package]]
name = "minimal-lexical"
version = "0.2.1"
@ -2403,6 +2486,17 @@ dependencies = [
"zmij",
]
[[package]]
name = "serde_path_to_error"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457"
dependencies = [
"itoa",
"serde",
"serde_core",
]
[[package]]
name = "serde_urlencoded"
version = "0.7.1"
@ -2757,6 +2851,7 @@ dependencies = [
"tokio",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
@ -2780,6 +2875,7 @@ dependencies = [
"tower",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
@ -2800,6 +2896,7 @@ version = "0.1.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
dependencies = [
"log",
"pin-project-lite",
"tracing-attributes",
"tracing-core",
@ -3102,7 +3199,7 @@ dependencies = [
[[package]]
name = "webclaw-cli"
version = "0.3.19"
version = "0.4.0"
dependencies = [
"clap",
"dotenvy",
@ -3123,7 +3220,7 @@ dependencies = [
[[package]]
name = "webclaw-core"
version = "0.3.19"
version = "0.4.0"
dependencies = [
"ego-tree",
"once_cell",
@ -3141,7 +3238,7 @@ dependencies = [
[[package]]
name = "webclaw-fetch"
version = "0.3.19"
version = "0.4.0"
dependencies = [
"bytes",
"calamine",
@ -3163,7 +3260,7 @@ dependencies = [
[[package]]
name = "webclaw-llm"
version = "0.3.19"
version = "0.4.0"
dependencies = [
"async-trait",
"reqwest",
@ -3176,7 +3273,7 @@ dependencies = [
[[package]]
name = "webclaw-mcp"
version = "0.3.19"
version = "0.4.0"
dependencies = [
"dirs",
"dotenvy",
@ -3197,13 +3294,34 @@ dependencies = [
[[package]]
name = "webclaw-pdf"
version = "0.3.19"
version = "0.4.0"
dependencies = [
"pdf-extract",
"thiserror",
"tracing",
]
[[package]]
name = "webclaw-server"
version = "0.4.0"
dependencies = [
"anyhow",
"axum",
"clap",
"serde",
"serde_json",
"subtle",
"thiserror",
"tokio",
"tower-http",
"tracing",
"tracing-subscriber",
"webclaw-core",
"webclaw-fetch",
"webclaw-llm",
"webclaw-pdf",
]
[[package]]
name = "webpki-root-certs"
version = "1.0.6"

2
Cargo.toml
View file

@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/*"]
[workspace.package]
version = "0.3.19"
version = "0.4.0"
edition = "2024"
license = "AGPL-3.0"
repository = "https://github.com/0xMassi/webclaw"

28
Dockerfile
View file

@ -1,5 +1,12 @@
# webclaw — Multi-stage Docker build
# Produces 2 binaries: webclaw (CLI) and webclaw-mcp (MCP server)
# Produces 3 binaries:
# webclaw — CLI (single-shot extraction, crawl, MCP-less use)
# webclaw-mcp — MCP server (stdio, for AI agents)
# webclaw-server — minimal REST API for self-hosting (OSS, stateless)
#
# NOTE: this is NOT the hosted API at api.webclaw.io — the cloud service
# adds anti-bot bypass, JS rendering, multi-tenant auth and async jobs
# that are intentionally not open-source. See docs/self-hosting.
# ---------------------------------------------------------------------------
# Stage 1: Build all binaries in release mode
@ -25,6 +32,7 @@ COPY crates/webclaw-llm/Cargo.toml crates/webclaw-llm/Cargo.toml
COPY crates/webclaw-pdf/Cargo.toml crates/webclaw-pdf/Cargo.toml
COPY crates/webclaw-mcp/Cargo.toml crates/webclaw-mcp/Cargo.toml
COPY crates/webclaw-cli/Cargo.toml crates/webclaw-cli/Cargo.toml
COPY crates/webclaw-server/Cargo.toml crates/webclaw-server/Cargo.toml
# Copy .cargo config if present (optional build flags)
COPY .cargo .cargo
@ -35,7 +43,8 @@ RUN mkdir -p crates/webclaw-core/src && echo "" > crates/webclaw-core/src/lib.rs
&& mkdir -p crates/webclaw-llm/src && echo "" > crates/webclaw-llm/src/lib.rs \
&& mkdir -p crates/webclaw-pdf/src && echo "" > crates/webclaw-pdf/src/lib.rs \
&& mkdir -p crates/webclaw-mcp/src && echo "fn main() {}" > crates/webclaw-mcp/src/main.rs \
&& mkdir -p crates/webclaw-cli/src && echo "fn main() {}" > crates/webclaw-cli/src/main.rs
&& mkdir -p crates/webclaw-cli/src && echo "fn main() {}" > crates/webclaw-cli/src/main.rs \
&& mkdir -p crates/webclaw-server/src && echo "fn main() {}" > crates/webclaw-server/src/main.rs
# Pre-build dependencies (this layer is cached until Cargo.toml/lock changes)
RUN cargo build --release 2>/dev/null || true
@ -54,9 +63,22 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Copy both binaries
# Copy all three binaries
COPY --from=builder /build/target/release/webclaw /usr/local/bin/webclaw
COPY --from=builder /build/target/release/webclaw-mcp /usr/local/bin/webclaw-mcp
COPY --from=builder /build/target/release/webclaw-server /usr/local/bin/webclaw-server
# Default port the REST API listens on when you run `webclaw-server` inside
# the container. Override with -e WEBCLAW_PORT=... or --port. Published only
# as documentation; callers still need `-p 3000:3000` on `docker run`.
EXPOSE 3000
# Container default: bind all interfaces so `-p 3000:3000` works. The binary
# itself defaults to 127.0.0.1 (safe for `cargo run` on a laptop); inside
# Docker that would make the server unreachable, so we flip it here.
# Override with -e WEBCLAW_HOST=127.0.0.1 if you front this with another
# process in the same container.
ENV WEBCLAW_HOST=0.0.0.0
# Entrypoint shim: forwards webclaw args/URL to the binary, but exec's other
# commands directly so this image can be used as a FROM base with custom CMD.

9
Dockerfile.ci
View file

@ -12,6 +12,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ARG BINARY_DIR
COPY ${BINARY_DIR}/webclaw /usr/local/bin/webclaw
COPY ${BINARY_DIR}/webclaw-mcp /usr/local/bin/webclaw-mcp
COPY ${BINARY_DIR}/webclaw-server /usr/local/bin/webclaw-server
# Default REST API port when running `webclaw-server` inside the container.
EXPOSE 3000
# Container default: bind all interfaces so `-p 3000:3000` works. The
# binary itself defaults to 127.0.0.1; flipping here keeps the CLI safe on
# a laptop but makes the container reachable out of the box.
ENV WEBCLAW_HOST=0.0.0.0
# Entrypoint shim: forwards webclaw args/URL to the binary, but exec's other
# commands directly so this image can be used as a FROM base with custom CMD.

29
crates/webclaw-server/Cargo.toml Normal file
View file

@ -0,0 +1,29 @@
[package]
name = "webclaw-server"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
description = "Minimal REST API server for self-hosting webclaw extraction. Wraps the OSS extraction crates with HTTP endpoints. NOT the production hosted API at api.webclaw.io — this is a stateless, single-binary reference server for local + self-hosted deployments."
[[bin]]
name = "webclaw-server"
path = "src/main.rs"
[dependencies]
webclaw-core = { workspace = true }
webclaw-fetch = { workspace = true }
webclaw-llm = { workspace = true }
webclaw-pdf = { workspace = true }
axum = { version = "0.8", features = ["macros"] }
tokio = { workspace = true }
tower-http = { version = "0.6", features = ["trace", "cors"] }
clap = { workspace = true, features = ["derive", "env"] }
serde = { workspace = true }
serde_json = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true, features = ["env-filter"] }
anyhow = "1"
thiserror = { workspace = true }
subtle = "2.6"

48
crates/webclaw-server/src/auth.rs Normal file
View file

@ -0,0 +1,48 @@
//! Optional bearer-token middleware.
//!
//! When the server is started without `--api-key`, every request is allowed
//! through (server runs in "open" mode — appropriate for `localhost`-only
//! deployments). When a key is configured, every `/v1/*` request must
//! present `Authorization: Bearer <key>` and the comparison is constant-
//! time to avoid timing-leaking the key.
use axum::{
extract::{Request, State},
http::StatusCode,
middleware::Next,
response::Response,
};
use subtle::ConstantTimeEq;
use crate::state::AppState;
/// Axum middleware. Mount with `axum::middleware::from_fn_with_state`.
pub async fn require_bearer(
State(state): State<AppState>,
request: Request,
next: Next,
) -> Result<Response, StatusCode> {
let Some(expected) = state.api_key() else {
// Open mode — no key configured. Allow everything.
return Ok(next.run(request).await);
};
let Some(header) = request
.headers()
.get("authorization")
.and_then(|v| v.to_str().ok())
else {
return Err(StatusCode::UNAUTHORIZED);
};
let presented = header
.strip_prefix("Bearer ")
.or_else(|| header.strip_prefix("bearer "))
.ok_or(StatusCode::UNAUTHORIZED)?;
if presented.as_bytes().ct_eq(expected.as_bytes()).into() {
Ok(next.run(request).await)
} else {
Err(StatusCode::UNAUTHORIZED)
}
}

87
crates/webclaw-server/src/error.rs Normal file
View file

@ -0,0 +1,87 @@
//! API error type. Maps internal errors to HTTP status codes + JSON.
use axum::{
Json,
http::StatusCode,
response::{IntoResponse, Response},
};
use serde_json::json;
use thiserror::Error;
/// Public-facing API error. Always serializes as `{ "error": "..." }`.
/// Keep messages user-actionable; internal details belong in tracing logs.
///
/// `Unauthorized` / `NotFound` / `Internal` are kept on the enum as
/// stable variants for handlers that don't exist yet (planned: per-key
/// rate-limit responses, dynamic route 404s). Marking them dead-code-OK
/// is preferable to inventing them later in three places.
#[allow(dead_code)]
#[derive(Debug, Error)]
pub enum ApiError {
#[error("{0}")]
BadRequest(String),
#[error("unauthorized")]
Unauthorized,
#[error("not found")]
NotFound,
#[error("upstream fetch failed: {0}")]
Fetch(String),
#[error("extraction failed: {0}")]
Extract(String),
#[error("LLM provider error: {0}")]
Llm(String),
#[error("internal: {0}")]
Internal(String),
}
impl ApiError {
pub fn bad_request(msg: impl Into<String>) -> Self {
Self::BadRequest(msg.into())
}
#[allow(dead_code)]
pub fn internal(msg: impl Into<String>) -> Self {
Self::Internal(msg.into())
}
fn status(&self) -> StatusCode {
match self {
Self::BadRequest(_) => StatusCode::BAD_REQUEST,
Self::Unauthorized => StatusCode::UNAUTHORIZED,
Self::NotFound => StatusCode::NOT_FOUND,
Self::Fetch(_) => StatusCode::BAD_GATEWAY,
Self::Extract(_) | Self::Llm(_) => StatusCode::UNPROCESSABLE_ENTITY,
Self::Internal(_) => StatusCode::INTERNAL_SERVER_ERROR,
}
}
}
impl IntoResponse for ApiError {
fn into_response(self) -> Response {
let body = Json(json!({ "error": self.to_string() }));
(self.status(), body).into_response()
}
}
impl From<webclaw_fetch::FetchError> for ApiError {
fn from(e: webclaw_fetch::FetchError) -> Self {
Self::Fetch(e.to_string())
}
}
impl From<webclaw_core::ExtractError> for ApiError {
fn from(e: webclaw_core::ExtractError) -> Self {
Self::Extract(e.to_string())
}
}
impl From<webclaw_llm::LlmError> for ApiError {
fn from(e: webclaw_llm::LlmError) -> Self {
Self::Llm(e.to_string())
}
}

118
crates/webclaw-server/src/main.rs Normal file
View file

@ -0,0 +1,118 @@
//! webclaw-server — minimal REST API for self-hosting webclaw extraction.
//!
//! This is the OSS reference server. It is intentionally small:
//! single binary, stateless, no database, no job queue. It wraps the
//! same extraction crates the CLI and MCP server use, exposed over
//! HTTP with JSON shapes that mirror the hosted API at
//! api.webclaw.io where the underlying capability exists in OSS.
//!
//! Hosted-only features (anti-bot bypass, JS rendering, async crawl
//! jobs, multi-tenant auth, billing) are *not* implemented here and
//! never will be — they're closed-source. See the docs for the full
//! "what self-hosting gives you vs. what the cloud gives you" matrix.
mod auth;
mod error;
mod routes;
mod state;
use std::net::{IpAddr, SocketAddr};
use std::time::Duration;
use axum::{
Router,
middleware::from_fn_with_state,
routing::{get, post},
};
use clap::Parser;
use tower_http::cors::{Any, CorsLayer};
use tower_http::trace::TraceLayer;
use tracing::info;
use tracing_subscriber::{EnvFilter, fmt};
use crate::state::AppState;
#[derive(Parser, Debug)]
#[command(
name = "webclaw-server",
version,
about = "Minimal self-hosted REST API for webclaw extraction.",
long_about = "Stateless single-binary REST API. Wraps the OSS extraction \
crates over HTTP. For the full hosted platform (anti-bot, \
JS render, async jobs, multi-tenant), use api.webclaw.io."
)]
struct Args {
/// Port to listen on. Env: WEBCLAW_PORT.
#[arg(short, long, env = "WEBCLAW_PORT", default_value_t = 3000)]
port: u16,
/// Host to bind to. Env: WEBCLAW_HOST.
/// Default `127.0.0.1` keeps the server local-only; set to
/// `0.0.0.0` to expose on all interfaces (only do this with
/// `--api-key` set or behind a reverse proxy that adds auth).
#[arg(long, env = "WEBCLAW_HOST", default_value = "127.0.0.1")]
host: IpAddr,
/// Optional bearer token. Env: WEBCLAW_API_KEY. When set, every
/// `/v1/*` request must present `Authorization: Bearer <key>`.
/// When unset, the server runs in open mode (no auth) — only
/// safe on a local-bound interface or behind another auth layer.
#[arg(long, env = "WEBCLAW_API_KEY")]
api_key: Option<String>,
/// Tracing filter. Env: RUST_LOG.
#[arg(long, env = "RUST_LOG", default_value = "info,webclaw_server=info")]
log: String,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let args = Args::parse();
fmt()
.with_env_filter(EnvFilter::try_new(&args.log).unwrap_or_else(|_| EnvFilter::new("info")))
.with_target(false)
.compact()
.init();
let state = AppState::new(args.api_key.clone())?;
let v1 = Router::new()
.route("/scrape", post(routes::scrape::scrape))
.route("/crawl", post(routes::crawl::crawl))
.route("/map", post(routes::map::map))
.route("/batch", post(routes::batch::batch))
.route("/extract", post(routes::extract::extract))
.route("/summarize", post(routes::summarize::summarize_route))
.route("/diff", post(routes::diff::diff_route))
.route("/brand", post(routes::brand::brand))
.layer(from_fn_with_state(state.clone(), auth::require_bearer));
let app = Router::new()
.route("/health", get(routes::health::health))
.nest("/v1", v1)
.layer(
// Permissive CORS — same posture as a self-hosted dev tool.
// Tighten in front with a reverse proxy if you expose this
// publicly.
CorsLayer::new()
.allow_origin(Any)
.allow_methods(Any)
.allow_headers(Any)
.max_age(Duration::from_secs(3600)),
)
.layer(TraceLayer::new_for_http())
.with_state(state);
let addr = SocketAddr::from((args.host, args.port));
let listener = tokio::net::TcpListener::bind(addr).await?;
let auth_status = if args.api_key.is_some() {
"bearer auth required"
} else {
"open mode (no auth)"
};
info!(%addr, mode = auth_status, "webclaw-server listening");
axum::serve(listener, app).await?;
Ok(())
}

85
crates/webclaw-server/src/routes/batch.rs Normal file
View file

@ -0,0 +1,85 @@
//! POST /v1/batch — fetch + extract many URLs in parallel.
//!
//! `concurrency` is hard-capped at 20 to avoid hammering targets and
//! to bound memory growth for naive callers. For larger batches use
//! the hosted API.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_core::ExtractionOptions;
use crate::{error::ApiError, state::AppState};
const HARD_MAX_URLS: usize = 100;
const HARD_MAX_CONCURRENCY: usize = 20;
#[derive(Debug, Deserialize, Default)]
#[serde(default)]
pub struct BatchRequest {
pub urls: Vec<String>,
pub concurrency: Option<usize>,
pub include_selectors: Vec<String>,
pub exclude_selectors: Vec<String>,
pub only_main_content: bool,
}
pub async fn batch(
State(state): State<AppState>,
Json(req): Json<BatchRequest>,
) -> Result<Json<Value>, ApiError> {
if req.urls.is_empty() {
return Err(ApiError::bad_request("`urls` is required"));
}
if req.urls.len() > HARD_MAX_URLS {
return Err(ApiError::bad_request(format!(
"too many urls: {} (max {HARD_MAX_URLS})",
req.urls.len()
)));
}
let concurrency = req.concurrency.unwrap_or(5).clamp(1, HARD_MAX_CONCURRENCY);
let options = ExtractionOptions {
include_selectors: req.include_selectors,
exclude_selectors: req.exclude_selectors,
only_main_content: req.only_main_content,
include_raw_html: false,
};
let url_refs: Vec<&str> = req.urls.iter().map(|s| s.as_str()).collect();
let results = state
.fetch()
.fetch_and_extract_batch_with_options(&url_refs, concurrency, &options)
.await;
let mut ok = 0usize;
let mut errors = 0usize;
let mut out: Vec<Value> = Vec::with_capacity(results.len());
for r in results {
match r.result {
Ok(extraction) => {
ok += 1;
out.push(json!({
"url": r.url,
"metadata": extraction.metadata,
"markdown": extraction.content.markdown,
}));
}
Err(e) => {
errors += 1;
out.push(json!({
"url": r.url,
"error": e.to_string(),
}));
}
}
}
Ok(Json(json!({
"total": out.len(),
"completed": ok,
"errors": errors,
"results": out,
})))
}

32
crates/webclaw-server/src/routes/brand.rs Normal file
View file

@ -0,0 +1,32 @@
//! POST /v1/brand — extract brand identity (colors, fonts, logo) from a page.
//!
//! Pure DOM/CSS analysis — no LLM, no network beyond the page fetch itself.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_core::brand::extract_brand;
use crate::{error::ApiError, state::AppState};
#[derive(Debug, Deserialize)]
pub struct BrandRequest {
pub url: String,
}
pub async fn brand(
State(state): State<AppState>,
Json(req): Json<BrandRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let fetched = state.fetch().fetch(&req.url).await?;
let brand = extract_brand(&fetched.html, Some(&fetched.url));
Ok(Json(json!({
"url": req.url,
"brand": brand,
})))
}

85
crates/webclaw-server/src/routes/crawl.rs Normal file
View file

@ -0,0 +1,85 @@
//! POST /v1/crawl — synchronous BFS crawl.
//!
//! NOTE: this server is stateless — there is no job queue. Crawls run
//! inline and return when complete. `max_pages` is hard-capped at 500
//! to avoid OOM on naive callers. For large crawls + async jobs, use
//! the hosted API at api.webclaw.io.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use std::time::Duration;
use webclaw_fetch::{CrawlConfig, Crawler, FetchConfig};
use crate::{error::ApiError, state::AppState};
const HARD_MAX_PAGES: usize = 500;
#[derive(Debug, Deserialize, Default)]
#[serde(default)]
pub struct CrawlRequest {
pub url: String,
pub max_depth: Option<usize>,
pub max_pages: Option<usize>,
pub use_sitemap: bool,
pub concurrency: Option<usize>,
pub allow_subdomains: bool,
pub allow_external_links: bool,
pub include_patterns: Vec<String>,
pub exclude_patterns: Vec<String>,
}
pub async fn crawl(
State(_state): State<AppState>,
Json(req): Json<CrawlRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let max_pages = req.max_pages.unwrap_or(50).min(HARD_MAX_PAGES);
let max_depth = req.max_depth.unwrap_or(3);
let concurrency = req.concurrency.unwrap_or(5).min(20);
let config = CrawlConfig {
fetch: FetchConfig::default(),
max_depth,
max_pages,
concurrency,
delay: Duration::from_millis(200),
path_prefix: None,
use_sitemap: req.use_sitemap,
include_patterns: req.include_patterns,
exclude_patterns: req.exclude_patterns,
allow_subdomains: req.allow_subdomains,
allow_external_links: req.allow_external_links,
progress_tx: None,
cancel_flag: None,
};
let crawler = Crawler::new(&req.url, config).map_err(ApiError::from)?;
let result = crawler.crawl(&req.url, None).await;
let pages: Vec<Value> = result
.pages
.iter()
.map(|p| {
json!({
"url": p.url,
"depth": p.depth,
"metadata": p.extraction.as_ref().map(|e| &e.metadata),
"markdown": p.extraction.as_ref().map(|e| e.content.markdown.as_str()).unwrap_or(""),
"error": p.error,
})
})
.collect();
Ok(Json(json!({
"url": req.url,
"status": "completed",
"total": result.total,
"completed": result.ok,
"errors": result.errors,
"elapsed_secs": result.elapsed_secs,
"pages": pages,
})))
}

92
crates/webclaw-server/src/routes/diff.rs Normal file
View file

@ -0,0 +1,92 @@
//! POST /v1/diff — compare current page content against a prior snapshot.
//!
//! Caller passes either a full prior `ExtractionResult` or the minimal
//! `{ markdown, metadata }` shape used by the hosted API. We re-fetch
//! the URL, extract, and run `webclaw_core::diff::diff` over the pair.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_core::{Content, ExtractionResult, Metadata, diff::diff};
use crate::{error::ApiError, state::AppState};
#[derive(Debug, Deserialize)]
pub struct DiffRequest {
pub url: String,
pub previous: PreviousSnapshot,
}
/// Either a full prior extraction, or the minimal `{ markdown, metadata }`
/// shape returned by /v1/scrape. Untagged so callers can send whichever
/// they have on hand.
#[derive(Debug, Deserialize)]
#[serde(untagged)]
pub enum PreviousSnapshot {
Full(ExtractionResult),
Minimal {
#[serde(default)]
markdown: String,
#[serde(default)]
metadata: Option<Metadata>,
},
}
impl PreviousSnapshot {
fn into_extraction(self) -> ExtractionResult {
match self {
Self::Full(r) => r,
Self::Minimal { markdown, metadata } => ExtractionResult {
metadata: metadata.unwrap_or_else(empty_metadata),
content: Content {
markdown,
plain_text: String::new(),
links: Vec::new(),
images: Vec::new(),
code_blocks: Vec::new(),
raw_html: None,
},
domain_data: None,
structured_data: Vec::new(),
},
}
}
}
fn empty_metadata() -> Metadata {
Metadata {
title: None,
description: None,
author: None,
published_date: None,
language: None,
url: None,
site_name: None,
image: None,
favicon: None,
word_count: 0,
}
}
pub async fn diff_route(
State(state): State<AppState>,
Json(req): Json<DiffRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let current = state.fetch().fetch_and_extract(&req.url).await?;
let previous = req.previous.into_extraction();
let result = diff(&previous, &current);
Ok(Json(json!({
"url": req.url,
"status": result.status,
"diff": result.text_diff,
"metadata_changes": result.metadata_changes,
"links_added": result.links_added,
"links_removed": result.links_removed,
"word_count_delta": result.word_count_delta,
})))
}

81
crates/webclaw-server/src/routes/extract.rs Normal file
View file

@ -0,0 +1,81 @@
//! POST /v1/extract — LLM-powered structured extraction.
//!
//! Two modes:
//! * `schema` — JSON Schema describing what to extract.
//! * `prompt` — natural-language instructions.
//!
//! At least one must be provided. The provider chain is built per
//! request from env (Ollama -> OpenAI -> Anthropic). Self-hosters
//! get the same fallback behaviour as the CLI.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_llm::{ProviderChain, extract::extract_json, extract::extract_with_prompt};
use crate::{error::ApiError, state::AppState};
#[derive(Debug, Deserialize, Default)]
#[serde(default)]
pub struct ExtractRequest {
pub url: String,
pub schema: Option<Value>,
pub prompt: Option<String>,
/// Optional override of the provider model name (e.g. `gpt-4o-mini`).
pub model: Option<String>,
}
pub async fn extract(
State(state): State<AppState>,
Json(req): Json<ExtractRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let has_schema = req.schema.is_some();
let has_prompt = req
.prompt
.as_deref()
.map(|p| !p.trim().is_empty())
.unwrap_or(false);
if !has_schema && !has_prompt {
return Err(ApiError::bad_request(
"either `schema` or `prompt` is required",
));
}
// Fetch + extract first so we feed the LLM clean markdown instead of
// raw HTML. Cheaper tokens, better signal.
let extraction = state.fetch().fetch_and_extract(&req.url).await?;
let content = if extraction.content.markdown.trim().is_empty() {
extraction.content.plain_text.clone()
} else {
extraction.content.markdown.clone()
};
if content.trim().is_empty() {
return Err(ApiError::Extract(
"no extractable content on page".to_string(),
));
}
let chain = ProviderChain::default().await;
if chain.is_empty() {
return Err(ApiError::Llm(
"no LLM providers configured (set OLLAMA_HOST, OPENAI_API_KEY, or ANTHROPIC_API_KEY)"
.to_string(),
));
}
let model = req.model.as_deref();
let data = if let Some(schema) = req.schema.as_ref() {
extract_json(&content, schema, &chain, model).await?
} else {
let prompt = req.prompt.as_deref().unwrap_or_default();
extract_with_prompt(&content, prompt, &chain, model).await?
};
Ok(Json(json!({
"url": req.url,
"data": data,
})))
}

10
crates/webclaw-server/src/routes/health.rs Normal file
View file

@ -0,0 +1,10 @@
use axum::Json;
use serde_json::{Value, json};
pub async fn health() -> Json<Value> {
Json(json!({
"status": "ok",
"version": env!("CARGO_PKG_VERSION"),
"service": "webclaw-server",
}))
}

49
crates/webclaw-server/src/routes/map.rs Normal file
View file

@ -0,0 +1,49 @@
//! POST /v1/map — discover URLs from a site's sitemaps.
//!
//! Walks robots.txt + common sitemap paths, recursively resolves
//! `<sitemapindex>` files, and returns the deduplicated list of URLs.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_fetch::sitemap;
use crate::{error::ApiError, state::AppState};
#[derive(Debug, Deserialize)]
pub struct MapRequest {
pub url: String,
/// When true, return the full SitemapEntry objects (with lastmod,
/// priority, changefreq). Defaults to false → bare URL strings,
/// matching the hosted-API shape.
#[serde(default)]
pub include_metadata: bool,
}
pub async fn map(
State(state): State<AppState>,
Json(req): Json<MapRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let entries = sitemap::discover(state.fetch(), &req.url).await?;
let body = if req.include_metadata {
json!({
"url": req.url,
"count": entries.len(),
"urls": entries,
})
} else {
let urls: Vec<&str> = entries.iter().map(|e| e.url.as_str()).collect();
json!({
"url": req.url,
"count": urls.len(),
"urls": urls,
})
};
Ok(Json(body))
}

18
crates/webclaw-server/src/routes/mod.rs Normal file
View file

@ -0,0 +1,18 @@
//! HTTP route handlers.
//!
//! The OSS server exposes a deliberately small surface that mirrors the
//! hosted-API JSON shapes where the underlying capability exists in the
//! OSS crates. Endpoints that depend on private infrastructure
//! (anti-bot bypass with stealth Chrome, JS rendering at scale,
//! per-user auth, billing, async job queues, agent loops) are
//! intentionally not implemented here. Use api.webclaw.io for those.
pub mod batch;
pub mod brand;
pub mod crawl;
pub mod diff;
pub mod extract;
pub mod health;
pub mod map;
pub mod scrape;
pub mod summarize;

108
crates/webclaw-server/src/routes/scrape.rs Normal file
View file

@ -0,0 +1,108 @@
//! POST /v1/scrape — fetch a URL, run extraction, return the requested
//! formats. JSON shape mirrors the hosted-API response where possible so
//! migrating from self-hosted → cloud is a config change, not a code one.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_core::{ExtractionOptions, llm::to_llm_text};
use crate::{error::ApiError, state::AppState};
#[derive(Debug, Deserialize, Default)]
#[serde(default)]
pub struct ScrapeRequest {
pub url: String,
/// Output formats. Allowed: "markdown", "text", "llm", "json", "html".
/// Defaults to ["markdown"]. Accepts a single string ("format")
/// or an array ("formats") for hosted-API compatibility.
#[serde(alias = "format")]
pub formats: ScrapeFormats,
pub include_selectors: Vec<String>,
pub exclude_selectors: Vec<String>,
pub only_main_content: bool,
}
#[derive(Debug, Deserialize)]
#[serde(untagged)]
pub enum ScrapeFormats {
One(String),
Many(Vec<String>),
}
impl Default for ScrapeFormats {
fn default() -> Self {
Self::Many(vec!["markdown".into()])
}
}
impl ScrapeFormats {
fn as_vec(&self) -> Vec<String> {
match self {
Self::One(s) => vec![s.clone()],
Self::Many(v) => v.clone(),
}
}
}
pub async fn scrape(
State(state): State<AppState>,
Json(req): Json<ScrapeRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let formats = req.formats.as_vec();
let options = ExtractionOptions {
include_selectors: req.include_selectors,
exclude_selectors: req.exclude_selectors,
only_main_content: req.only_main_content,
include_raw_html: formats.iter().any(|f| f == "html"),
};
let extraction = state
.fetch()
.fetch_and_extract_with_options(&req.url, &options)
.await?;
let mut body = json!({
"url": extraction.metadata.url.clone().unwrap_or_else(|| req.url.clone()),
"metadata": extraction.metadata,
});
let obj = body.as_object_mut().expect("json::object");
for f in &formats {
match f.as_str() {
"markdown" => {
obj.insert("markdown".into(), json!(extraction.content.markdown));
}
"text" => {
obj.insert("text".into(), json!(extraction.content.plain_text));
}
"llm" => {
let llm = to_llm_text(&extraction, extraction.metadata.url.as_deref());
obj.insert("llm".into(), json!(llm));
}
"html" => {
if let Some(raw) = &extraction.content.raw_html {
obj.insert("html".into(), json!(raw));
}
}
"json" => {
obj.insert("json".into(), json!(extraction));
}
other => {
return Err(ApiError::bad_request(format!(
"unknown format: '{other}' (allowed: markdown, text, llm, html, json)"
)));
}
}
}
if !extraction.structured_data.is_empty() {
obj.insert("structured_data".into(), json!(extraction.structured_data));
}
Ok(Json(body))
}

52
crates/webclaw-server/src/routes/summarize.rs Normal file
View file

@ -0,0 +1,52 @@
//! POST /v1/summarize — LLM-powered page summary.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_llm::{ProviderChain, summarize::summarize};
use crate::{error::ApiError, state::AppState};
#[derive(Debug, Deserialize, Default)]
#[serde(default)]
pub struct SummarizeRequest {
pub url: String,
pub max_sentences: Option<usize>,
pub model: Option<String>,
}
pub async fn summarize_route(
State(state): State<AppState>,
Json(req): Json<SummarizeRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let extraction = state.fetch().fetch_and_extract(&req.url).await?;
let content = if extraction.content.markdown.trim().is_empty() {
extraction.content.plain_text.clone()
} else {
extraction.content.markdown.clone()
};
if content.trim().is_empty() {
return Err(ApiError::Extract(
"no extractable content on page".to_string(),
));
}
let chain = ProviderChain::default().await;
if chain.is_empty() {
return Err(ApiError::Llm(
"no LLM providers configured (set OLLAMA_HOST, OPENAI_API_KEY, or ANTHROPIC_API_KEY)"
.to_string(),
));
}
let summary = summarize(&content, req.max_sentences, &chain, req.model.as_deref()).await?;
Ok(Json(json!({
"url": req.url,
"summary": summary,
})))
}

49
crates/webclaw-server/src/state.rs Normal file
View file

@ -0,0 +1,49 @@
//! Shared application state. Cheap to clone via Arc; held by the axum
//! Router for the life of the process.
use std::sync::Arc;
use webclaw_fetch::{BrowserProfile, FetchClient, FetchConfig};
/// Single-process state shared across all request handlers.
#[derive(Clone)]
pub struct AppState {
inner: Arc<Inner>,
}
struct Inner {
/// Wrapped in `Arc` because `fetch_and_extract_batch_with_options`
/// (used by the /v1/batch handler) takes `self: &Arc<Self>` so it
/// can clone the client into spawned tasks. The single-call handlers
/// auto-deref `&Arc<FetchClient>` -> `&FetchClient`, so this costs
/// them nothing.
pub fetch: Arc<FetchClient>,
pub api_key: Option<String>,
}
impl AppState {
/// Build the application state. The fetch client is constructed once
/// and shared across requests so connection pools + browser profile
/// state don't churn per request.
pub fn new(api_key: Option<String>) -> anyhow::Result<Self> {
let config = FetchConfig {
browser: BrowserProfile::Chrome,
..FetchConfig::default()
};
let fetch = FetchClient::new(config)
.map_err(|e| anyhow::anyhow!("failed to build fetch client: {e}"))?;
Ok(Self {
inner: Arc::new(Inner {
fetch: Arc::new(fetch),
api_key,
}),
})
}
pub fn fetch(&self) -> &Arc<FetchClient> {
&self.inner.fetch
}
pub fn api_key(&self) -> Option<&str> {
self.inner.api_key.as_deref()
}
}