Published to the MCP registry as io.github.0xMassi/webclaw@0.6.17. Description
no longer names a competitor (public-scrubbing posture); packages[].version
stays 0.6.16 (the live @webclaw/mcp on npm — the registry validates it exists).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
npm pack was bundling packages/create-webclaw/.mcpregistry_*_token (the
mcp-publisher credential files) into the tarball because the package had no
'files' field and npm did not honor the repo-root .gitignore. Restrict the
published package to index.mjs + README.md (matches @webclaw/mcp).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aligns with the industry standard: Firecrawl, the official MCP reference servers,
Exa, Brave, Apify, Bright Data, etc. all install their MCP via a single
`npx -y <pkg>` config line — none use a create-* scaffolder for the MCP itself
(create-* is a project-bootstrap convention). create-webclaw now auto-detects AI
tools and writes the canonical `npx @webclaw/mcp` config into each, byte-identical
to webclaw.io/docs and the MCP registries, so a scaffolded config can never
diverge from a hand-written one.
- buildMcpEntry + all writers emit {command:"npx", args:["-y","@webclaw/mcp"]}
across every format (JSON, Continue array, OpenCode array, Codex TOML)
- remove the entire binary-download machinery (getTarget/download/extract/cargo
fallback, ~250 lines) — the @webclaw/mcp launcher handles that now
- README: reflect the new behavior + show the one-block manual config
- version 0.1.6 -> 0.1.7
Verified: correct npx config (with/without API key) across all formats;
idempotent re-runs; zero binary-download code remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The published bin was named `webclaw-mcp`, which npx shadows with any existing
`webclaw-mcp` on PATH (Homebrew/cargo/create-webclaw) — so `npx @webclaw/mcp`
silently ran a stale binary instead of the launcher, for exactly our existing
CLI users. Rename the bin to the unique `webclaw-mcp-launcher`; `npx
@webclaw/mcp` still resolves the single bin. Verified end-to-end: with a stale
webclaw-mcp on PATH, npx now runs the launcher and returns 14 tools.
- bin: webclaw-mcp -> webclaw-mcp-launcher; version 0.6.15 -> 0.6.16
- mark webclaw-mcp.mjs executable
- release.yml: version the launcher INDEPENDENTLY of the core release (a
launcher-only fix must ship without a core bump) — publish the next patch
above the current npm version and pin RELEASE_TAG to the release tag
- server.json: sync to 0.6.16
Published + deprecated 0.6.15 (bin collision); 0.6.16 is the first good release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- README: rewrite to match the main webclaw README's voice/positioning
(clean markdown/JSON/LLM-ready context + anti-bot angle), accurate 14-tool
local/cloud split, consistent links. This is the package's npm landing page.
- version @webclaw/mcp in lockstep with the core release (0.6.15) so
@webclaw/mcp@X.Y.Z always installs webclaw vX.Y.Z; sync the MCP-registry
server.json version to match.
- release.yml: add an `npm` job (needs: release) that pins the launcher's
RELEASE_TAG to the tag and publishes @webclaw/mcp on stable tags. Requires
an NPM_TOKEN repo secret with publish rights on the @webclaw scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`npx -y @webclaw/mcp` downloads the prebuilt webclaw-mcp binary (verified
against the release SHA256SUMS), caches it, and runs it as an MCP stdio server.
This makes webclaw introspectable in MCP registries (Glama/Smithery/MCP
registry) and installable in any MCP client with no Rust build.
- packages/webclaw-mcp: zero-dep launcher (.mjs) + package.json + README
- README/CLAUDE/examples/smithery: mcpServers config -> npx -y @webclaw/mcp
- registry: point the MCP-registry entry (server.json) at @webclaw/mcp, the
real stdio server, instead of create-webclaw (a scaffolder); drop
create-webclaw's mcpName claim so ownership isn't duplicated
- CHANGELOG: unreleased entry
create-webclaw stays as the auto-detect installer. .mcp.json is intentionally
left pointing at the local install path until @webclaw/mcp is published to npm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
README said "10 MCP Tools" and omitted list_extractors, vertical_scrape,
lead, lead_batch; the server now has 14. Drop the hard count, list all
14, and add lead/lead_batch (require WEBCLAW_API_KEY). Correct the wrong
config paths (OpenCode -> ~/.config/opencode/opencode.json, Codex ->
~/.codex/config.toml TOML, Antigravity ~/.config variant), fix the
offline-tool count, and drop the stale TLS-library link. server.json tool
count 12 -> 14. No installer logic or version change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`npx create-webclaw` never used the prebuilt binary on any platform and
silently fell back to `cargo install`, which fails with "'cargo' is not
recognized" / "cargo: not found" unless Rust is installed. Four bugs:
1. Asset name mismatch: getAssetName() hardcoded `webclaw-mcp-<target>`,
but release assets are `webclaw-<tag>-<target>` (versioned, no `mcp-`
infix). The `find()` always returned undefined, so the prebuilt path
was never taken — on every OS, not just Windows. Now the asset name is
built from the release tag_name + a platform→target map.
2. `unzip` is absent on Windows. The `.zip` branch now uses PowerShell
`Expand-Archive` (ships with Windows 10/11) and keeps `unzip` only for
the non-Windows case.
3. The prebuilt failure was swallowed by a bare `catch {}`, hiding the
real cause (a 403 is almost always a GitHub API rate limit). The error
is now surfaced, with a rate-limit hint + GITHUB_TOKEN support on the
api.github.com request (token dropped on CDN redirects).
4. (missed by the report's own suggested fix) Archives extract into a
`webclaw-<tag>-<target>/` subdirectory holding three binaries, so the
old `chmod(BINARY_PATH)` hit a nonexistent path. webclaw-mcp is now
lifted out of that subdir to BINARY_PATH and the rest is cleaned up.
BINARY_NAME/BINARY_PATH also gain the `.exe` suffix on Windows so the
written MCP config points at a real file.
Tested in Docker (no Windows machine available):
- Linux amd64 + arm64 on Debian trixie: full flow installs the binary and
it answers a real MCP initialize handshake (serverInfo webclaw-mcp
0.6.13, 12 tools).
- Windows .zip path validated against the real release zip: Expand-Archive
equivalent extraction, nested `.exe` resolved + lifted, PE header `MZ`.
Executing the .exe needs Windows (the reporter confirmed that on Win11).
- Bug 3: with the GitHub API blocked, the new build prints the real reason
instead of "No pre-built binary found".
Closes#71
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(fetch,llm): DoS hardening via response caps + glob validation (P2)
Response body caps:
- webclaw-fetch::Response::from_wreq now rejects bodies over 50 MB. Checks
Content-Length up front (before the allocation) and the actual
.bytes() length after (belt-and-braces against lying upstreams).
Previously the HTML -> markdown conversion downstream could allocate
multiple String copies per page; a 100 MB page would OOM the process.
- webclaw-llm providers (anthropic/openai/ollama) share a new
response_json_capped helper with a 5 MB cap. Protects against a
malicious or runaway provider response exhausting memory.
Crawler frontier cap: after each BFS depth level the frontier is
truncated to max(max_pages * 10, 100) entries, keeping the most
recently discovered links. Dense pages (tag clouds, search results)
used to push the frontier into the tens of thousands even after
max_pages halted new fetches.
Glob pattern validation: user-supplied include_patterns /
exclude_patterns are rejected at Crawler::new if they contain more
than 4 `**` wildcards or exceed 1024 chars. The backtracking matcher
degrades exponentially on deeply-nested `**` against long paths.
Cleanup:
- Removed blanket #![allow(dead_code)] from webclaw-cli/src/main.rs;
no warnings surfaced, the suppression was obsolete.
- core/.gitignore: replaced overbroad *.json with specific local-
artifact patterns (previous rule would have swallowed package.json,
components.json, .smithery/*.json).
Tests: +4 validate_glob tests. Full workspace test: 283 passed
(webclaw-core + webclaw-fetch + webclaw-llm).
Version: 0.3.15 -> 0.3.16
CHANGELOG updated.
Refs: docs/AUDIT-2026-04-16.md (P2 section)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: gitignore CLI research dumps, drop accidentally-tracked file
research-*.json output from `webclaw ... --research ...` got silently
swept into git by the relaxed *.json gitignore in the preceding commit.
The old blanket *.json rule was hiding both this legitimate scratch
file AND packages/create-webclaw/server.json (MCP registry config that
we DO want tracked).
Removes the research dump from git and adds a narrower research-*.json
ignore pattern so future CLI output doesn't get re-tracked by accident.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Crawls are same-origin by default. Enable allow_subdomains to follow
sibling/child subdomains (blog.example.com from example.com), or
allow_external_links for full cross-origin crawling.
Root domain extraction uses a heuristic that handles two-part TLDs
(co.uk, com.au). Includes 5 unit tests for root_domain().
Bump to 0.3.12.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
