apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-24 07:31:01 +02:00
Code Issues Projects Releases Packages Wiki Activity
webclaw/examples/mcp-web-scraping/README.md
Valerio 0678f0ba14 feat(mcp): add @webclaw/mcp zero-install launcher as the canonical MCP command 
`npx -y @webclaw/mcp` downloads the prebuilt webclaw-mcp binary (verified
against the release SHA256SUMS), caches it, and runs it as an MCP stdio server.
This makes webclaw introspectable in MCP registries (Glama/Smithery/MCP
registry) and installable in any MCP client with no Rust build.

- packages/webclaw-mcp: zero-dep launcher (.mjs) + package.json + README
- README/CLAUDE/examples/smithery: mcpServers config -> npx -y @webclaw/mcp
- registry: point the MCP-registry entry (server.json) at @webclaw/mcp, the
  real stdio server, instead of create-webclaw (a scaffolder); drop
  create-webclaw's mcpName claim so ownership isn't duplicated
- CHANGELOG: unreleased entry

create-webclaw stays as the auto-detect installer. .mcp.json is intentionally
left pointing at the local install path until @webclaw/mcp is published to npm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 18:14:30 +02:00

1.1 KiB
Raw Blame History

MCP Web Scraping

Use webclaw as a local MCP server so Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, or another MCP client can fetch clean web context.

Install

npx create-webclaw

The installer detects supported MCP clients and can write the config for you.

Manual Config

{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"],
      "env": {
        "WEBCLAW_API_KEY": "wc_your_key"
      }
    }
  }
}

WEBCLAW_API_KEY is optional for local extraction. Add it when you want cloud fallback for protected sites, JS rendering, hosted search, or hosted research.

Example Prompts

Scrape https://docs.rs/tokio and summarize the parts about task spawning.

Crawl https://docs.example.com up to depth 2 and return the pages most relevant to authentication.

Extract the pricing tiers from https://example.com/pricing as JSON with fields name, price, limits, and features.

The MCP server exposes tools for scrape, crawl, map, batch, extract, summarize, diff, brand, research, search, and vertical extractors.