mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-07-24 07:31:01 +02:00
d387669c13
Aligns with the industry standard: Firecrawl, the official MCP reference servers,
Exa, Brave, Apify, Bright Data, etc. all install their MCP via a single
`npx -y <pkg>` config line — none use a create-* scaffolder for the MCP itself
(create-* is a project-bootstrap convention). create-webclaw now auto-detects AI
tools and writes the canonical `npx @webclaw/mcp` config into each, byte-identical
to webclaw.io/docs and the MCP registries, so a scaffolded config can never
diverge from a hand-written one.
- buildMcpEntry + all writers emit {command:"npx", args:["-y","@webclaw/mcp"]}
across every format (JSON, Continue array, OpenCode array, Codex TOML)
- remove the entire binary-download machinery (getTarget/download/extract/cargo
fallback, ~250 lines) — the @webclaw/mcp launcher handles that now
- README: reflect the new behavior + show the one-block manual config
- version 0.1.6 -> 0.1.7
Verified: correct npx config (with/without API key) across all formats;
idempotent re-runs; zero binary-download code remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
168 lines
6.3 KiB
Markdown
168 lines
6.3 KiB
Markdown
<p align="center">
|
|
<a href="https://webclaw.io">
|
|
<picture>
|
|
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/0xMassi/webclaw/main/.github/banner.png" />
|
|
<img src="https://raw.githubusercontent.com/0xMassi/webclaw/main/.github/banner.png" alt="webclaw" width="700" />
|
|
</picture>
|
|
</a>
|
|
</p>
|
|
|
|
<h3 align="center">
|
|
One command to give your AI agent reliable web access.<br/>
|
|
<sub>No headless browser. No Puppeteer. No 403s.</sub>
|
|
</h3>
|
|
|
|
<p align="center">
|
|
<a href="https://www.npmjs.com/package/create-webclaw"><img src="https://img.shields.io/npm/dt/create-webclaw?style=for-the-badge&logo=npm&logoColor=white&label=Installs&color=CB3837" alt="npm installs" /></a>
|
|
<a href="https://github.com/0xMassi/webclaw"><img src="https://img.shields.io/github/stars/0xMassi/webclaw?style=for-the-badge&logo=github&logoColor=white&label=Stars&color=181717" alt="Stars" /></a>
|
|
<a href="https://github.com/0xMassi/webclaw/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-10B981?style=for-the-badge" alt="License" /></a>
|
|
</p>
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
npx create-webclaw
|
|
```
|
|
|
|
That's it. Auto-detects your AI tools and writes the `npx @webclaw/mcp` config into each — nothing to install.
|
|
|
|
Works with **Claude Desktop**, **Claude Code**, **Cursor**, **Windsurf**, **VS Code**, **OpenCode**, **Codex CLI**, and **Antigravity**.
|
|
|
|
---
|
|
|
|
## The Problem
|
|
|
|
Your AI agent calls `fetch()` and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.
|
|
|
|
When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.
|
|
|
|
## The Fix
|
|
|
|
webclaw impersonates the latest Chrome at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.
|
|
|
|
Then it extracts just the content — clean markdown, 67% fewer tokens.
|
|
|
|
```
|
|
Raw HTML webclaw
|
|
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
|
|
│ <div class="ad-wrapper"> │ │ # Breaking: AI Breakthrough │
|
|
│ <nav class="global-nav"> │ │ │
|
|
│ <script>window.__NEXT_DATA__ │ │ Researchers achieved 94% │
|
|
│ ={...8KB of JSON...}</script> │ │ accuracy on cross-domain │
|
|
│ <div class="social-share"> │ │ reasoning benchmarks. │
|
|
│ <!-- 142,847 characters --> │ │ │
|
|
│ │ │ ## Key Findings │
|
|
│ 4,820 tokens │ │ 1,590 tokens │
|
|
└──────────────────────────────────┘ └──────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## What It Does
|
|
|
|
```bash
|
|
npx create-webclaw
|
|
```
|
|
|
|
1. Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
|
|
2. Asks for your API key (optional — **works locally without one**)
|
|
3. Writes the `npx @webclaw/mcp` config into each detected tool
|
|
|
|
The server itself runs via [`@webclaw/mcp`](https://www.npmjs.com/package/@webclaw/mcp) — `npx` fetches it on first launch and caches it. `create-webclaw` is just the convenience that writes that config for you.
|
|
|
|
### Prefer to configure by hand?
|
|
|
|
Add this one block to your client's `mcpServers` config — it's identical to what `create-webclaw` writes:
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"webclaw": {
|
|
"command": "npx",
|
|
"args": ["-y", "@webclaw/mcp"]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## MCP Tools
|
|
|
|
After setup, your AI agent has access to these 14 tools:
|
|
|
|
| Tool | What it does | API key needed? |
|
|
|------|-------------|-----------------|
|
|
| **scrape** | Extract content from any URL | No |
|
|
| **crawl** | Recursively crawl a website | No |
|
|
| **search** | Web search + parallel scrape | Yes (Serper) |
|
|
| **map** | Discover URLs from sitemaps | No |
|
|
| **batch** | Extract multiple URLs in parallel | No |
|
|
| **extract** | LLM-powered structured extraction | Yes |
|
|
| **summarize** | Content summarization | Yes |
|
|
| **diff** | Track content changes | No |
|
|
| **brand** | Extract brand identity | No |
|
|
| **research** | Deep multi-page research | Yes |
|
|
| **list_extractors** | List the built-in vertical extractors | No |
|
|
| **vertical_scrape** | Scrape with a built-in vertical extractor | No |
|
|
| **lead** | Enrich a company URL into an outreach-ready lead (leadership + socials, tech, pricing, public emails) | Yes (webclaw) |
|
|
| **lead_batch** | Enrich up to 25 company URLs at once | Yes (webclaw) |
|
|
|
|
**8 of the 14 tools work fully offline** — no API key, no cloud, no tracking.
|
|
|
|
## Supported Tools
|
|
|
|
| Tool | Config location |
|
|
|------|----------------|
|
|
| Claude Desktop | `~/Library/Application Support/Claude/claude_desktop_config.json` |
|
|
| Claude Code | `~/.claude.json` |
|
|
| Cursor | `.cursor/mcp.json` |
|
|
| Windsurf | `~/.codeium/windsurf/mcp_config.json` |
|
|
| VS Code (Continue) | `~/.continue/config.json` |
|
|
| OpenCode | `~/.config/opencode/opencode.json` (or `./opencode.json`) |
|
|
| Codex CLI | `~/.codex/config.toml` |
|
|
| Antigravity | `~/.config/antigravity/mcp.json` (or `~/.antigravity/mcp.json`) |
|
|
|
|
## Sites That Work
|
|
|
|
webclaw gets through where default `fetch()` gets blocked:
|
|
|
|
Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with **99% success rate**.
|
|
|
|
## Alternative Install Methods
|
|
|
|
### Homebrew
|
|
|
|
```bash
|
|
brew tap 0xMassi/webclaw && brew install webclaw
|
|
```
|
|
|
|
### Docker
|
|
|
|
```bash
|
|
docker run --rm ghcr.io/0xmassi/webclaw https://example.com
|
|
```
|
|
|
|
### Cargo
|
|
|
|
```bash
|
|
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli
|
|
```
|
|
|
|
### Prebuilt Binaries
|
|
|
|
Download from [GitHub Releases](https://github.com/0xMassi/webclaw/releases) for macOS (arm64, x86_64) and Linux (x86_64, aarch64).
|
|
|
|
---
|
|
|
|
## Links
|
|
|
|
- [Website](https://webclaw.io)
|
|
- [Documentation](https://webclaw.io/docs)
|
|
- [GitHub](https://github.com/0xMassi/webclaw)
|
|
- [Discord](https://discord.gg/KDfd48EpnW)
|
|
- [Status](https://status.webclaw.io)
|
|
|
|
## License
|
|
|
|
AGPL-3.0
|