apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-24 07:31:01 +02:00
Code Issues Projects Releases Packages Wiki Activity
webclaw/packages/create-webclaw
History
Valerio 8d3483cbe9 chore(registry): reword server.json (drop competitor name), bump to 0.6.17 
Published to the MCP registry as io.github.0xMassi/webclaw@0.6.17. Description
no longer names a competitor (public-scrubbing posture); packages[].version
stays 0.6.16 (the live @webclaw/mcp on npm — the registry validates it exists).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 13:25:29 +02:00
..
index.mjs refactor(create-webclaw): write the npx @webclaw/mcp config, drop binary download 2026-07-22 11:34:01 +02:00
package.json fix(create-webclaw): add files allowlist so npm never ships secrets 2026-07-22 12:08:32 +02:00
README.md refactor(create-webclaw): write the npx @webclaw/mcp config, drop binary download 2026-07-22 11:34:01 +02:00
server.json chore(registry): reword server.json (drop competitor name), bump to 0.6.17 2026-07-22 13:25:29 +02:00

README.md

webclaw

One command to give your AI agent reliable web access.
No headless browser. No Puppeteer. No 403s.

npm installs Stars License

Quick Start

npx create-webclaw

That's it. Auto-detects your AI tools and writes the npx @webclaw/mcp config into each — nothing to install.

Works with Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, OpenCode, Codex CLI, and Antigravity.

The Problem

Your AI agent calls fetch() and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.

When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.

The Fix

webclaw impersonates the latest Chrome at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.

Then it extracts just the content — clean markdown, 67% fewer tokens.

                     Raw HTML                          webclaw
┌──────────────────────────────────┐    ┌──────────────────────────────────┐
│ <div class="ad-wrapper">         │    │ # Breaking: AI Breakthrough      │
│ <nav class="global-nav">         │    │                                  │
│ <script>window.__NEXT_DATA__     │    │ Researchers achieved 94%         │
│ ={...8KB of JSON...}</script>    │    │ accuracy on cross-domain         │
│ <div class="social-share">       │    │ reasoning benchmarks.            │
│ <!-- 142,847 characters -->      │    │                                  │
│                                  │    │ ## Key Findings                  │
│         4,820 tokens             │    │         1,590 tokens             │
└──────────────────────────────────┘    └──────────────────────────────────┘

What It Does

npx create-webclaw
  1. Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
  2. Asks for your API key (optional — works locally without one)
  3. Writes the npx @webclaw/mcp config into each detected tool

The server itself runs via @webclaw/mcp — npx fetches it on first launch and caches it. create-webclaw is just the convenience that writes that config for you.

Prefer to configure by hand?

Add this one block to your client's mcpServers config — it's identical to what create-webclaw writes:

{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"]
    }
  }
}

MCP Tools

After setup, your AI agent has access to these 14 tools:

Tool What it does API key needed?
scrape Extract content from any URL No
crawl Recursively crawl a website No
search Web search + parallel scrape Yes (Serper)
map Discover URLs from sitemaps No
batch Extract multiple URLs in parallel No
extract LLM-powered structured extraction Yes
summarize Content summarization Yes
diff Track content changes No
brand Extract brand identity No
research Deep multi-page research Yes
list_extractors List the built-in vertical extractors No
vertical_scrape Scrape with a built-in vertical extractor No
lead Enrich a company URL into an outreach-ready lead (leadership + socials, tech, pricing, public emails) Yes (webclaw)
lead_batch Enrich up to 25 company URLs at once Yes (webclaw)

8 of the 14 tools work fully offline — no API key, no cloud, no tracking.

Supported Tools

Tool Config location
Claude Desktop ~/Library/Application Support/Claude/claude_desktop_config.json
Claude Code ~/.claude.json
Cursor .cursor/mcp.json
Windsurf ~/.codeium/windsurf/mcp_config.json
VS Code (Continue) ~/.continue/config.json
OpenCode ~/.config/opencode/opencode.json (or ./opencode.json)
Codex CLI ~/.codex/config.toml
Antigravity ~/.config/antigravity/mcp.json (or ~/.antigravity/mcp.json)

Sites That Work

webclaw gets through where default fetch() gets blocked:

Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with 99% success rate.

Alternative Install Methods

Homebrew

brew tap 0xMassi/webclaw && brew install webclaw

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

Prebuilt Binaries

Download from GitHub Releases for macOS (arm64, x86_64) and Linux (x86_64, aarch64).

Links

License

AGPL-3.0