webclaw

mirror of https://github.com/0xMassi/webclaw.git synced 2026-04-25 00:06:21 +02:00

Valerio 4e81c3430d docs: update npm package license to AGPL-3.0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>		2026-04-02 11:33:43 +02:00
..
index.mjs	Initial release: webclaw v0.1.0 — web content extraction for LLMs	2026-03-23 18:31:11 +01:00
package.json	docs: update npm package license to AGPL-3.0	2026-04-02 11:33:43 +02:00
README.md	docs: update npm package license to AGPL-3.0	2026-04-02 11:33:43 +02:00

README.md

One command to give your AI agent reliable web access.
_{No headless browser. No Puppeteer. No 403s.}

Quick Start

npx create-webclaw

That's it. Auto-detects your AI tools, downloads the MCP server, configures everything.

Works with Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, OpenCode, Codex CLI, and Antigravity.

The Problem

Your AI agent calls fetch() and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.

When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.

The Fix

webclaw impersonates Chrome 146 at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.

Then it extracts just the content — clean markdown, 67% fewer tokens.

                     Raw HTML                          webclaw
┌──────────────────────────────────┐    ┌──────────────────────────────────┐
│ <div class="ad-wrapper">         │    │ # Breaking: AI Breakthrough      │
│ <nav class="global-nav">         │    │                                  │
│ <script>window.__NEXT_DATA__     │    │ Researchers achieved 94%         │
│ ={...8KB of JSON...}</script>    │    │ accuracy on cross-domain         │
│ <div class="social-share">       │    │ reasoning benchmarks.            │
│ <!-- 142,847 characters -->      │    │                                  │
│                                  │    │ ## Key Findings                  │
│         4,820 tokens             │    │         1,590 tokens             │
└──────────────────────────────────┘    └──────────────────────────────────┘

What It Does

npx create-webclaw

Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
Downloads the webclaw-mcp binary for your platform (macOS arm64/x86, Linux x86/arm64)
Asks for your API key (optional — works locally without one)
Writes the MCP config for each detected tool

10 MCP Tools

After setup, your AI agent has access to:

Tool	What it does	API key needed?
scrape	Extract content from any URL	No
crawl	Recursively crawl a website	No
search	Web search + parallel scrape	Yes (Serper)
map	Discover URLs from sitemaps	No
batch	Extract multiple URLs in parallel	No
extract	LLM-powered structured extraction	Yes
summarize	Content summarization	Yes
diff	Track content changes	No
brand	Extract brand identity	No
research	Deep multi-page research	Yes

8 of 10 tools work fully offline. No API key, no cloud, no tracking.

Supported Tools

Tool	Config location
Claude Desktop	`~/Library/Application Support/Claude/claude_desktop_config.json`
Claude Code	`~/.claude.json`
Cursor	`.cursor/mcp.json`
Windsurf	`~/.codeium/windsurf/mcp_config.json`
VS Code (Continue)	`~/.continue/config.json`
OpenCode	`~/.opencode/config.json`
Codex CLI	`~/.codex/config.json`
Antigravity	`~/.antigravity/mcp.json`

Sites That Work

webclaw gets through where default fetch() gets blocked:

Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with 99% success rate.

Alternative Install Methods

Homebrew

brew tap 0xMassi/webclaw && brew install webclaw

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

Prebuilt Binaries

Download from GitHub Releases for macOS (arm64, x86_64) and Linux (x86_64, aarch64).

License

AGPL-3.0