apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-06-08 22:25:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(server): add OSS webclaw-server REST API binary (closes #29)

Browse source 
Self-hosters hitting docs/self-hosting were promised three binaries
but the OSS Docker image only shipped two. webclaw-server lived in
the closed-source hosted-platform repo, which couldn't be opened. This
adds a minimal axum REST API in the OSS repo so self-hosting actually
works without pretending to ship the cloud platform.

Crate at crates/webclaw-server/. Stateless, no database, no job queue,
single binary. Endpoints: GET /health, POST /v1/{scrape, crawl, map,
batch, extract, summarize, diff, brand}. JSON shapes mirror
api.webclaw.io for the endpoints OSS can support, so swapping between
self-hosted and hosted is a base-URL change.

Auth: optional bearer token via WEBCLAW_API_KEY / --api-key. Comparison
is constant-time (subtle::ConstantTimeEq). Open mode (no key) is
allowed and binds 127.0.0.1 by default; the Docker image flips
WEBCLAW_HOST=0.0.0.0 so the container is reachable out of the box.

Hard caps to keep naive callers from OOMing the process: crawl capped
at 500 pages synchronously, batch capped at 100 URLs / 20 concurrent.
For unbounded crawls or anti-bot bypass the docs point users at the
hosted API.

Dockerfile + Dockerfile.ci updated to copy webclaw-server into
/usr/local/bin and EXPOSE 3000. Workspace version bumped to 0.4.0
(new public binary).
This commit is contained in:
Valerio 2026-04-22 12:25:11 +02:00
parent b4bfff120e
commit 2ba682adf3
20 changed files with 1116 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

28
Dockerfile
View file

@ -1,5 +1,12 @@
# webclaw — Multi-stage Docker build
# Produces 2 binaries: webclaw (CLI) and webclaw-mcp (MCP server)
# Produces 3 binaries:
# webclaw — CLI (single-shot extraction, crawl, MCP-less use)
# webclaw-mcp — MCP server (stdio, for AI agents)
# webclaw-server — minimal REST API for self-hosting (OSS, stateless)
#
# NOTE: this is NOT the hosted API at api.webclaw.io — the cloud service
# adds anti-bot bypass, JS rendering, multi-tenant auth and async jobs
# that are intentionally not open-source. See docs/self-hosting.
# ---------------------------------------------------------------------------
# Stage 1: Build all binaries in release mode
@ -25,6 +32,7 @@ COPY crates/webclaw-llm/Cargo.toml crates/webclaw-llm/Cargo.toml
COPY crates/webclaw-pdf/Cargo.toml crates/webclaw-pdf/Cargo.toml
COPY crates/webclaw-mcp/Cargo.toml crates/webclaw-mcp/Cargo.toml
COPY crates/webclaw-cli/Cargo.toml crates/webclaw-cli/Cargo.toml
COPY crates/webclaw-server/Cargo.toml crates/webclaw-server/Cargo.toml
# Copy .cargo config if present (optional build flags)
COPY .cargo .cargo
@ -35,7 +43,8 @@ RUN mkdir -p crates/webclaw-core/src && echo "" > crates/webclaw-core/src/lib.rs
&& mkdir -p crates/webclaw-llm/src && echo "" > crates/webclaw-llm/src/lib.rs \
&& mkdir -p crates/webclaw-pdf/src && echo "" > crates/webclaw-pdf/src/lib.rs \
&& mkdir -p crates/webclaw-mcp/src && echo "fn main() {}" > crates/webclaw-mcp/src/main.rs \
&& mkdir -p crates/webclaw-cli/src && echo "fn main() {}" > crates/webclaw-cli/src/main.rs
&& mkdir -p crates/webclaw-cli/src && echo "fn main() {}" > crates/webclaw-cli/src/main.rs \
&& mkdir -p crates/webclaw-server/src && echo "fn main() {}" > crates/webclaw-server/src/main.rs
# Pre-build dependencies (this layer is cached until Cargo.toml/lock changes)
RUN cargo build --release 2>/dev/null || true
@ -54,9 +63,22 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Copy both binaries
# Copy all three binaries
COPY --from=builder /build/target/release/webclaw /usr/local/bin/webclaw
COPY --from=builder /build/target/release/webclaw-mcp /usr/local/bin/webclaw-mcp
COPY --from=builder /build/target/release/webclaw-server /usr/local/bin/webclaw-server
# Default port the REST API listens on when you run `webclaw-server` inside
# the container. Override with -e WEBCLAW_PORT=... or --port. Published only
# as documentation; callers still need `-p 3000:3000` on `docker run`.
EXPOSE 3000
# Container default: bind all interfaces so `-p 3000:3000` works. The binary
# itself defaults to 127.0.0.1 (safe for `cargo run` on a laptop); inside
# Docker that would make the server unreachable, so we flip it here.
# Override with -e WEBCLAW_HOST=127.0.0.1 if you front this with another
# process in the same container.
ENV WEBCLAW_HOST=0.0.0.0
# Entrypoint shim: forwards webclaw args/URL to the binary, but exec's other
# commands directly so this image can be used as a FROM base with custom CMD.