apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-24 07:31:01 +02:00
Code Issues Projects Releases Packages Wiki Activity
webclaw/packages/create-webclaw/README.md
Valerio d387669c13 refactor(create-webclaw): write the npx @webclaw/mcp config, drop binary download 
Aligns with the industry standard: Firecrawl, the official MCP reference servers,
Exa, Brave, Apify, Bright Data, etc. all install their MCP via a single
`npx -y <pkg>` config line — none use a create-* scaffolder for the MCP itself
(create-* is a project-bootstrap convention). create-webclaw now auto-detects AI
tools and writes the canonical `npx @webclaw/mcp` config into each, byte-identical
to webclaw.io/docs and the MCP registries, so a scaffolded config can never
diverge from a hand-written one.

- buildMcpEntry + all writers emit {command:"npx", args:["-y","@webclaw/mcp"]}
  across every format (JSON, Continue array, OpenCode array, Codex TOML)
- remove the entire binary-download machinery (getTarget/download/extract/cargo
  fallback, ~250 lines) — the @webclaw/mcp launcher handles that now
- README: reflect the new behavior + show the one-block manual config
- version 0.1.6 -> 0.1.7

Verified: correct npx config (with/without API key) across all formats;
idempotent re-runs; zero binary-download code remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:34:01 +02:00

6.3 KiB
Raw Blame History

webclaw

One command to give your AI agent reliable web access.
No headless browser. No Puppeteer. No 403s.

npm installs Stars License

Quick Start

npx create-webclaw

That's it. Auto-detects your AI tools and writes the npx @webclaw/mcp config into each — nothing to install.

Works with Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, OpenCode, Codex CLI, and Antigravity.

The Problem

Your AI agent calls fetch() and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.

When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.

The Fix

webclaw impersonates the latest Chrome at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.

Then it extracts just the content — clean markdown, 67% fewer tokens.

                     Raw HTML                          webclaw
┌──────────────────────────────────┐    ┌──────────────────────────────────┐
│ <div class="ad-wrapper">         │    │ # Breaking: AI Breakthrough      │
│ <nav class="global-nav">         │    │                                  │
│ <script>window.__NEXT_DATA__     │    │ Researchers achieved 94%         │
│ ={...8KB of JSON...}</script>    │    │ accuracy on cross-domain         │
│ <div class="social-share">       │    │ reasoning benchmarks.            │
│ <!-- 142,847 characters -->      │    │                                  │
│                                  │    │ ## Key Findings                  │
│         4,820 tokens             │    │         1,590 tokens             │
└──────────────────────────────────┘    └──────────────────────────────────┘

What It Does

npx create-webclaw
  1. Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
  2. Asks for your API key (optional — works locally without one)
  3. Writes the npx @webclaw/mcp config into each detected tool

The server itself runs via @webclaw/mcp — npx fetches it on first launch and caches it. create-webclaw is just the convenience that writes that config for you.

Prefer to configure by hand?

Add this one block to your client's mcpServers config — it's identical to what create-webclaw writes:

{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"]
    }
  }
}

MCP Tools

After setup, your AI agent has access to these 14 tools:

Tool What it does API key needed?
scrape Extract content from any URL No
crawl Recursively crawl a website No
search Web search + parallel scrape Yes (Serper)
map Discover URLs from sitemaps No
batch Extract multiple URLs in parallel No
extract LLM-powered structured extraction Yes
summarize Content summarization Yes
diff Track content changes No
brand Extract brand identity No
research Deep multi-page research Yes
list_extractors List the built-in vertical extractors No
vertical_scrape Scrape with a built-in vertical extractor No
lead Enrich a company URL into an outreach-ready lead (leadership + socials, tech, pricing, public emails) Yes (webclaw)
lead_batch Enrich up to 25 company URLs at once Yes (webclaw)

8 of the 14 tools work fully offline — no API key, no cloud, no tracking.

Supported Tools

Tool Config location
Claude Desktop ~/Library/Application Support/Claude/claude_desktop_config.json
Claude Code ~/.claude.json
Cursor .cursor/mcp.json
Windsurf ~/.codeium/windsurf/mcp_config.json
VS Code (Continue) ~/.continue/config.json
OpenCode ~/.config/opencode/opencode.json (or ./opencode.json)
Codex CLI ~/.codex/config.toml
Antigravity ~/.config/antigravity/mcp.json (or ~/.antigravity/mcp.json)

Sites That Work

webclaw gets through where default fetch() gets blocked:

Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with 99% success rate.

Alternative Install Methods

Homebrew

brew tap 0xMassi/webclaw && brew install webclaw

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

Prebuilt Binaries

Download from GitHub Releases for macOS (arm64, x86_64) and Linux (x86_64, aarch64).

Links

License

AGPL-3.0