webclaw/crates/webclaw-server/src/routes/summarize.rs
Valerio 2ba682adf3 feat(server): add OSS webclaw-server REST API binary (closes #29)
Self-hosters hitting docs/self-hosting were promised three binaries
but the OSS Docker image only shipped two. webclaw-server lived in
the closed-source hosted-platform repo, which couldn't be opened. This
adds a minimal axum REST API in the OSS repo so self-hosting actually
works without pretending to ship the cloud platform.

Crate at crates/webclaw-server/. Stateless, no database, no job queue,
single binary. Endpoints: GET /health, POST /v1/{scrape, crawl, map,
batch, extract, summarize, diff, brand}. JSON shapes mirror
api.webclaw.io for the endpoints OSS can support, so swapping between
self-hosted and hosted is a base-URL change.

Auth: optional bearer token via WEBCLAW_API_KEY / --api-key. Comparison
is constant-time (subtle::ConstantTimeEq). Open mode (no key) is
allowed and binds 127.0.0.1 by default; the Docker image flips
WEBCLAW_HOST=0.0.0.0 so the container is reachable out of the box.

Hard caps to keep naive callers from OOMing the process: crawl capped
at 500 pages synchronously, batch capped at 100 URLs / 20 concurrent.
For unbounded crawls or anti-bot bypass the docs point users at the
hosted API.

Dockerfile + Dockerfile.ci updated to copy webclaw-server into
/usr/local/bin and EXPOSE 3000. Workspace version bumped to 0.4.0
(new public binary).
2026-04-22 12:25:11 +02:00

52 lines
1.5 KiB
Rust

//! POST /v1/summarize — LLM-powered page summary.
use axum::{Json, extract::State};
use serde::Deserialize;
use serde_json::{Value, json};
use webclaw_llm::{ProviderChain, summarize::summarize};
use crate::{error::ApiError, state::AppState};
#[derive(Debug, Deserialize, Default)]
#[serde(default)]
pub struct SummarizeRequest {
pub url: String,
pub max_sentences: Option<usize>,
pub model: Option<String>,
}
pub async fn summarize_route(
State(state): State<AppState>,
Json(req): Json<SummarizeRequest>,
) -> Result<Json<Value>, ApiError> {
if req.url.trim().is_empty() {
return Err(ApiError::bad_request("`url` is required"));
}
let extraction = state.fetch().fetch_and_extract(&req.url).await?;
let content = if extraction.content.markdown.trim().is_empty() {
extraction.content.plain_text.clone()
} else {
extraction.content.markdown.clone()
};
if content.trim().is_empty() {
return Err(ApiError::Extract(
"no extractable content on page".to_string(),
));
}
let chain = ProviderChain::default().await;
if chain.is_empty() {
return Err(ApiError::Llm(
"no LLM providers configured (set OLLAMA_HOST, OPENAI_API_KEY, or ANTHROPIC_API_KEY)"
.to_string(),
));
}
let summary = summarize(&content, req.max_sentences, &chain, req.model.as_deref()).await?;
Ok(Json(json!({
"url": req.url,
"summary": summary,
})))
}