apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-21 07:01:01 +02:00
Code Issues Projects Releases Packages Wiki Activity
webclaw/packages/create-webclaw
History
Valerio 4673fc9b9a create-webclaw: fix README tool list + config paths (14 tools) 
README said "10 MCP Tools" and omitted list_extractors, vertical_scrape,
lead, lead_batch; the server now has 14. Drop the hard count, list all
14, and add lead/lead_batch (require WEBCLAW_API_KEY). Correct the wrong
config paths (OpenCode -> ~/.config/opencode/opencode.json, Codex ->
~/.codex/config.toml TOML, Antigravity ~/.config variant), fix the
offline-tool count, and drop the stale TLS-library link. server.json tool
count 12 -> 14. No installer logic or version change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 18:50:35 +02:00
..
index.mjs fix(create-webclaw): repair binary install on Windows (and all platforms) 2026-06-27 11:58:14 +02:00
package.json fix(create-webclaw): repair binary install on Windows (and all platforms) 2026-06-27 11:58:14 +02:00
README.md create-webclaw: fix README tool list + config paths (14 tools) 2026-07-18 18:50:35 +02:00
server.json create-webclaw: fix README tool list + config paths (14 tools) 2026-07-18 18:50:35 +02:00

README.md

webclaw

One command to give your AI agent reliable web access.
No headless browser. No Puppeteer. No 403s.

npm installs Stars License

Quick Start

npx create-webclaw

That's it. Auto-detects your AI tools, downloads the MCP server, configures everything.

Works with Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, OpenCode, Codex CLI, and Antigravity.

The Problem

Your AI agent calls fetch() and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.

When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.

The Fix

webclaw impersonates the latest Chrome at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.

Then it extracts just the content — clean markdown, 67% fewer tokens.

                     Raw HTML                          webclaw
┌──────────────────────────────────┐    ┌──────────────────────────────────┐
│ <div class="ad-wrapper">         │    │ # Breaking: AI Breakthrough      │
│ <nav class="global-nav">         │    │                                  │
│ <script>window.__NEXT_DATA__     │    │ Researchers achieved 94%         │
│ ={...8KB of JSON...}</script>    │    │ accuracy on cross-domain         │
│ <div class="social-share">       │    │ reasoning benchmarks.            │
│ <!-- 142,847 characters -->      │    │                                  │
│                                  │    │ ## Key Findings                  │
│         4,820 tokens             │    │         1,590 tokens             │
└──────────────────────────────────┘    └──────────────────────────────────┘

What It Does

npx create-webclaw
  1. Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
  2. Downloads the webclaw-mcp binary for your platform (macOS arm64/x86, Linux x86/arm64)
  3. Asks for your API key (optional — works locally without one)
  4. Writes the MCP config for each detected tool

MCP Tools

After setup, your AI agent has access to these 14 tools:

Tool What it does API key needed?
scrape Extract content from any URL No
crawl Recursively crawl a website No
search Web search + parallel scrape Yes (Serper)
map Discover URLs from sitemaps No
batch Extract multiple URLs in parallel No
extract LLM-powered structured extraction Yes
summarize Content summarization Yes
diff Track content changes No
brand Extract brand identity No
research Deep multi-page research Yes
list_extractors List the built-in vertical extractors No
vertical_scrape Scrape with a built-in vertical extractor No
lead Enrich a company URL into an outreach-ready lead (leadership + socials, tech, pricing, public emails) Yes (webclaw)
lead_batch Enrich up to 25 company URLs at once Yes (webclaw)

8 of the 14 tools work fully offline — no API key, no cloud, no tracking.

Supported Tools

Tool Config location
Claude Desktop ~/Library/Application Support/Claude/claude_desktop_config.json
Claude Code ~/.claude.json
Cursor .cursor/mcp.json
Windsurf ~/.codeium/windsurf/mcp_config.json
VS Code (Continue) ~/.continue/config.json
OpenCode ~/.config/opencode/opencode.json (or ./opencode.json)
Codex CLI ~/.codex/config.toml
Antigravity ~/.config/antigravity/mcp.json (or ~/.antigravity/mcp.json)

Sites That Work

webclaw gets through where default fetch() gets blocked:

Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with 99% success rate.

Alternative Install Methods

Homebrew

brew tap 0xMassi/webclaw && brew install webclaw

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

Prebuilt Binaries

Download from GitHub Releases for macOS (arm64, x86_64) and Linux (x86_64, aarch64).

Links

License

AGPL-3.0