mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-04-25 00:06:21 +02:00
docs: update npm package license to AGPL-3.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Valerio
parent
c43da982c3
commit
4e81c3430d
2 changed files with 142 additions and 23 deletions
|
|
@ -1,21 +1,96 @@
|
|||
# create-webclaw
|
||||
<p align="center">
|
||||
<a href="https://webclaw.io">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/0xMassi/webclaw/main/.github/banner.png" />
|
||||
<img src="https://raw.githubusercontent.com/0xMassi/webclaw/main/.github/banner.png" alt="webclaw" width="700" />
|
||||
</picture>
|
||||
</a>
|
||||
</p>
|
||||
|
||||
Set up [webclaw](https://webclaw.io) MCP server for AI agents in one command.
|
||||
<h3 align="center">
|
||||
One command to give your AI agent reliable web access.<br/>
|
||||
<sub>No headless browser. No Puppeteer. No 403s.</sub>
|
||||
</h3>
|
||||
|
||||
## Usage
|
||||
<p align="center">
|
||||
<a href="https://www.npmjs.com/package/create-webclaw"><img src="https://img.shields.io/npm/dt/create-webclaw?style=for-the-badge&logo=npm&logoColor=white&label=Installs&color=CB3837" alt="npm installs" /></a>
|
||||
<a href="https://github.com/0xMassi/webclaw"><img src="https://img.shields.io/github/stars/0xMassi/webclaw?style=for-the-badge&logo=github&logoColor=white&label=Stars&color=181717" alt="Stars" /></a>
|
||||
<a href="https://github.com/0xMassi/webclaw/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-AGPL--3.0-10B981?style=for-the-badge" alt="License" /></a>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
npx create-webclaw
|
||||
```
|
||||
|
||||
## What it does
|
||||
That's it. Auto-detects your AI tools, downloads the MCP server, configures everything.
|
||||
|
||||
1. Detects installed AI tools (Claude Desktop, Claude Code, Cursor, Windsurf, VS Code + Continue)
|
||||
2. Downloads the `webclaw-mcp` binary for your platform
|
||||
3. Asks for your API key (optional — works locally without one)
|
||||
4. Configures MCP in each detected tool
|
||||
Works with **Claude Desktop**, **Claude Code**, **Cursor**, **Windsurf**, **VS Code**, **OpenCode**, **Codex CLI**, and **Antigravity**.
|
||||
|
||||
## Supported tools
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
Your AI agent calls `fetch()` and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.
|
||||
|
||||
When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.
|
||||
|
||||
## The Fix
|
||||
|
||||
webclaw impersonates Chrome 146 at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.
|
||||
|
||||
Then it extracts just the content — clean markdown, 67% fewer tokens.
|
||||
|
||||
```
|
||||
Raw HTML webclaw
|
||||
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
|
||||
│ <div class="ad-wrapper"> │ │ # Breaking: AI Breakthrough │
|
||||
│ <nav class="global-nav"> │ │ │
|
||||
│ <script>window.__NEXT_DATA__ │ │ Researchers achieved 94% │
|
||||
│ ={...8KB of JSON...}</script> │ │ accuracy on cross-domain │
|
||||
│ <div class="social-share"> │ │ reasoning benchmarks. │
|
||||
│ <!-- 142,847 characters --> │ │ │
|
||||
│ │ │ ## Key Findings │
|
||||
│ 4,820 tokens │ │ 1,590 tokens │
|
||||
└──────────────────────────────────┘ └──────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What It Does
|
||||
|
||||
```bash
|
||||
npx create-webclaw
|
||||
```
|
||||
|
||||
1. Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
|
||||
2. Downloads the `webclaw-mcp` binary for your platform (macOS arm64/x86, Linux x86/arm64)
|
||||
3. Asks for your API key (optional — **works locally without one**)
|
||||
4. Writes the MCP config for each detected tool
|
||||
|
||||
## 10 MCP Tools
|
||||
|
||||
After setup, your AI agent has access to:
|
||||
|
||||
| Tool | What it does | API key needed? |
|
||||
|------|-------------|-----------------|
|
||||
| **scrape** | Extract content from any URL | No |
|
||||
| **crawl** | Recursively crawl a website | No |
|
||||
| **search** | Web search + parallel scrape | Yes (Serper) |
|
||||
| **map** | Discover URLs from sitemaps | No |
|
||||
| **batch** | Extract multiple URLs in parallel | No |
|
||||
| **extract** | LLM-powered structured extraction | Yes |
|
||||
| **summarize** | Content summarization | Yes |
|
||||
| **diff** | Track content changes | No |
|
||||
| **brand** | Extract brand identity | No |
|
||||
| **research** | Deep multi-page research | Yes |
|
||||
|
||||
**8 of 10 tools work fully offline.** No API key, no cloud, no tracking.
|
||||
|
||||
## Supported Tools
|
||||
|
||||
| Tool | Config location |
|
||||
|------|----------------|
|
||||
|
|
@ -24,17 +99,51 @@ npx create-webclaw
|
|||
| Cursor | `.cursor/mcp.json` |
|
||||
| Windsurf | `~/.codeium/windsurf/mcp_config.json` |
|
||||
| VS Code (Continue) | `~/.continue/config.json` |
|
||||
| OpenCode | `~/.opencode/config.json` |
|
||||
| Codex CLI | `~/.codex/config.json` |
|
||||
| Antigravity | `~/.antigravity/mcp.json` |
|
||||
|
||||
## MCP tools provided
|
||||
## Sites That Work
|
||||
|
||||
After setup, your AI agent has access to:
|
||||
webclaw gets through where default `fetch()` gets blocked:
|
||||
|
||||
- **scrape** — extract content from any URL
|
||||
- **crawl** — recursively crawl a website
|
||||
- **search** — web search + parallel scrape
|
||||
- **map** — discover URLs from sitemaps
|
||||
- **batch** — extract multiple URLs in parallel
|
||||
- **extract** — LLM-powered structured extraction
|
||||
- **summarize** — content summarization
|
||||
- **diff** — track content changes
|
||||
- **brand** — extract brand identity
|
||||
Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with **99% success rate**.
|
||||
|
||||
## Alternative Install Methods
|
||||
|
||||
### Homebrew
|
||||
|
||||
```bash
|
||||
brew tap 0xMassi/webclaw && brew install webclaw
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker run --rm ghcr.io/0xmassi/webclaw https://example.com
|
||||
```
|
||||
|
||||
### Cargo
|
||||
|
||||
```bash
|
||||
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli
|
||||
```
|
||||
|
||||
### Prebuilt Binaries
|
||||
|
||||
Download from [GitHub Releases](https://github.com/0xMassi/webclaw/releases) for macOS (arm64, x86_64) and Linux (x86_64, aarch64).
|
||||
|
||||
---
|
||||
|
||||
## Links
|
||||
|
||||
- [Website](https://webclaw.io)
|
||||
- [Documentation](https://webclaw.io/docs)
|
||||
- [GitHub](https://github.com/0xMassi/webclaw)
|
||||
- [TLS Library](https://github.com/0xMassi/webclaw-tls)
|
||||
- [Discord](https://discord.gg/KDfd48EpnW)
|
||||
- [Status](https://status.webclaw.io)
|
||||
|
||||
## License
|
||||
|
||||
AGPL-3.0
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
{
|
||||
"name": "create-webclaw",
|
||||
"version": "0.1.0",
|
||||
"version": "0.1.3",
|
||||
"mcpName": "io.github.0xMassi/webclaw",
|
||||
"description": "Set up webclaw MCP server for AI agents (Claude, Cursor, Windsurf, OpenCode, Codex, Antigravity)",
|
||||
"bin": {
|
||||
"create-webclaw": "./index.mjs"
|
||||
|
|
@ -9,18 +10,27 @@
|
|||
"keywords": [
|
||||
"webclaw",
|
||||
"mcp",
|
||||
"mcp-server",
|
||||
"ai",
|
||||
"ai-agent",
|
||||
"scraping",
|
||||
"web-scraping",
|
||||
"scraper",
|
||||
"crawler",
|
||||
"extract",
|
||||
"markdown",
|
||||
"llm",
|
||||
"claude",
|
||||
"cursor",
|
||||
"windsurf",
|
||||
"opencode",
|
||||
"codex",
|
||||
"antigravity",
|
||||
"web-scraping"
|
||||
"tls-fingerprint",
|
||||
"cloudflare-bypass"
|
||||
],
|
||||
"author": "webclaw",
|
||||
"license": "MIT",
|
||||
"license": "AGPL-3.0",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/0xMassi/webclaw"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue