apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-07-10 05:12:10 +02:00
Code Issues Projects Releases Packages Wiki Activity

docs: scrub public SKILL.md of competitor/vendor wording and over-claims

Browse source 
The public SKILL.md named a competitor ("Beats Firecrawl") and protection
vendors (Cloudflare, DataDome, AWS WAF), described bypass mechanics, and
claimed the OSS path handles protected/JS pages automatically — which is
only true when WEBCLAW_API_KEY is set.

- Remove competitor and vendor names; describe capabilities generically
  (bot-protected / JavaScript-rendered pages).
- Drop the antibot response-shape block and the bot-bypass research
  example; use a neutral research example instead.
- Correct the over-claims: cloud fallback for protected/JS pages happens
  only when WEBCLAW_API_KEY is set; otherwise a clear message is returned.
- Apply the same edits to the skill/ copy so both stay identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Valerio 2026-07-02 12:35:30 +02:00
parent a869b0af69
commit 7ae422f3bc
2 changed files with 44 additions and 60 deletions
Download patch file Download diff file Expand all files Collapse all files

52
SKILL.md
View file

@ -1,6 +1,6 @@
--- ---
name: webclaw name: webclaw
description: Web extraction engine with antibot bypass. Scrape, crawl, extract, summarize, search, map, diff, monitor, research, and analyze any URL — including Cloudflare-protected sites. Use when you need reliable web content, the built-in web_fetch fails, or you need structured data extraction from web pages. description: Web extraction engine for LLMs and agents. Scrape, crawl, extract, summarize, search, map, diff, monitor, research, and analyze any URL into clean Markdown, text, or JSON, including pages that block bots or render with JavaScript. Use when you need reliable web content, the built-in web_fetch fails, or you need structured data extraction from web pages.
homepage: https://webclaw.io homepage: https://webclaw.io
user-invocable: true user-invocable: true
metadata: {"openclaw":{"emoji":"🦀","requires":{"env":["WEBCLAW_API_KEY"]},"primaryEnv":"WEBCLAW_API_KEY","homepage":"https://webclaw.io","install":[{"id":"npx","kind":"node","bins":["webclaw-mcp"],"label":"npx create-webclaw"}]}} metadata: {"openclaw":{"emoji":"🦀","requires":{"env":["WEBCLAW_API_KEY"]},"primaryEnv":"WEBCLAW_API_KEY","homepage":"https://webclaw.io","install":[{"id":"npx","kind":"node","bins":["webclaw-mcp"],"label":"npx create-webclaw"}]}}
@ -8,12 +8,12 @@ metadata: {"openclaw":{"emoji":"🦀","requires":{"env":["WEBCLAW_API_KEY"]},"pr
# webclaw # webclaw
High-quality web extraction with automatic antibot bypass. Beats Firecrawl on extraction quality and handles Cloudflare, DataDome, and JS-rendered pages automatically. Web extraction tuned for LLM and RAG pipelines. Bot-protected and JavaScript-rendered pages are handled for you when `WEBCLAW_API_KEY` is set; without it, those pages return a clear message explaining how to enable it.
## When to use this skill ## When to use this skill
- **Always** when you need to fetch web content and want reliable results - **Always** when you need to fetch web content and want reliable results
- When `web_fetch` returns empty/blocked content (403, Cloudflare challenges) - When `web_fetch` returns empty or blocked content (403s, bot challenges)
- When you need structured data extraction (pricing tables, product info) - When you need structured data extraction (pricing tables, product info)
- When you need to crawl an entire site or discover all URLs - When you need to crawl an entire site or discover all URLs
- When you need LLM-optimized content (cleaner than raw markdown) - When you need LLM-optimized content (cleaner than raw markdown)
@ -80,15 +80,7 @@ curl -X POST https://api.webclaw.io/v1/scrape \
- `llm` — optimized for LLM consumption: includes page title, URL, and cleaned content with link references. Best for feeding to AI models. - `llm` — optimized for LLM consumption: includes page title, URL, and cleaned content with link references. Best for feeding to AI models.
- `json` — full extraction result with all metadata - `json` — full extraction result with all metadata
**When antibot bypass activates** (automatic, no extra config): Protected or JavaScript-rendered pages are handled through the cloud fallback when `WEBCLAW_API_KEY` is set (see Smart Fetch Architecture below).
```json
{
"antibot": {
"bypass": true,
"elapsed_ms": 3200
}
}
```
### 2. Crawl — scrape an entire website ### 2. Crawl — scrape an entire website
@ -394,7 +386,7 @@ curl -X POST https://api.webclaw.io/v1/research \
-H "Authorization: Bearer $WEBCLAW_API_KEY" \ -H "Authorization: Bearer $WEBCLAW_API_KEY" \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"query": "How does Cloudflare Turnstile work and what are its known bypass methods?", "query": "Compare the top managed PostgreSQL providers on pricing, backups, and regions",
"max_iterations": 5, "max_iterations": 5,
"max_sources": 10, "max_sources": 10,
"topic": "security", "topic": "security",
@ -425,16 +417,16 @@ Response when complete:
{ {
"id": "res-abc-123", "id": "res-abc-123",
"status": "completed", "status": "completed",
"query": "How does Cloudflare Turnstile work and what are its known bypass methods?", "query": "Compare the top managed PostgreSQL providers on pricing, backups, and regions",
"report": "# Cloudflare Turnstile Analysis\n\n## Overview\nCloudflare Turnstile is a CAPTCHA replacement that...\n\n## How It Works\n...\n\n## Known Bypass Methods\n...", "report": "# Managed PostgreSQL Providers\n\n## Overview\n...\n\n## Pricing\n...\n\n## Backups and regions\n...",
"sources": [ "sources": [
{ "url": "https://developers.cloudflare.com/turnstile/", "title": "Turnstile Documentation" }, { "url": "https://neon.tech/pricing", "title": "Neon Pricing" },
{ "url": "https://blog.cloudflare.com/turnstile-ga/", "title": "Turnstile GA Announcement" } { "url": "https://supabase.com/pricing", "title": "Supabase Pricing" }
], ],
"findings": [ "findings": [
"Turnstile uses browser environment signals and proof-of-work challenges", "Entry tiers cluster around $19-25/mo for a small production instance",
"Managed mode auto-selects challenge difficulty based on visitor risk score", "Point-in-time recovery windows range from 7 to 30 days by tier",
"Known bypass approaches include instrumented browser automation" "Region coverage is widest on the largest providers"
], ],
"iterations": 5, "iterations": 5,
"elapsed_ms": 34200 "elapsed_ms": 34200
@ -600,7 +592,7 @@ curl -X DELETE https://api.webclaw.io/v1/watch/watch-abc-123 \
- **Batch over individual scrapes** when fetching multiple URLs — it's faster and more efficient. - **Batch over individual scrapes** when fetching multiple URLs — it's faster and more efficient.
- **Use `map` before `crawl`** to discover the site structure first, then crawl specific sections. - **Use `map` before `crawl`** to discover the site structure first, then crawl specific sections.
- **Use `extract` with a JSON schema** for reliable structured output (e.g., pricing tables, product specs, contact info). - **Use `extract` with a JSON schema** for reliable structured output (e.g., pricing tables, product specs, contact info).
- **Antibot bypass is automatic** — no extra configuration needed. Works on Cloudflare, DataDome, AWS WAF, and JS-rendered SPAs. - **Protected pages are handled for you** when `WEBCLAW_API_KEY` is set. Bot-protected sites and JavaScript-rendered SPAs fall back to the cloud engine; without a key they return a clear setup message.
- **Use `search` with `scrape: true`** to get full page content for each search result in one call instead of searching then scraping separately. - **Use `search` with `scrape: true`** to get full page content for each search result in one call instead of searching then scraping separately.
- **Use `research` for complex questions** that need multiple sources — it handles the search-read-synthesize loop automatically. Enable `deep: true` for thorough analysis. - **Use `research` for complex questions** that need multiple sources — it handles the search-read-synthesize loop automatically. Enable `deep: true` for thorough analysis.
- **Use `agent-scrape` for interactive pages** where data is behind filters, pagination, or form submissions that a simple scrape cannot reach. - **Use `agent-scrape` for interactive pages** where data is behind filters, pagination, or form submissions that a simple scrape cannot reach.
@ -608,23 +600,23 @@ curl -X DELETE https://api.webclaw.io/v1/watch/watch-abc-123 \
## Smart Fetch Architecture ## Smart Fetch Architecture
The webclaw MCP server uses a **local-first** approach: The webclaw MCP server uses a local-first approach:
1. **Local fetch** — fast, free, no API credits used (~80% of sites) 1. **Local fetch**: fast, free, no API credits used (most sites)
2. **Cloud API fallback** — automatic when bot protection or JS rendering is detected 2. **Cloud fallback**: used when a page is bot-protected or JavaScript-rendered, and only when `WEBCLAW_API_KEY` is set
This means: This means:
- Most scrapes cost zero credits (local extraction) - Most scrapes run locally and cost zero credits
- Cloudflare, DataDome, AWS WAF sites automatically fall back to the cloud API - Bot-protected sites fall back to the cloud engine when `WEBCLAW_API_KEY` is set
- JS-rendered SPAs (React, Next.js, Vue) also fall back automatically - JavaScript-rendered SPAs (React, Next.js, Vue) fall back the same way
- Set `WEBCLAW_API_KEY` to enable cloud fallback - Without a key, those pages return a clear message explaining how to enable the fallback
## vs web_fetch ## vs web_fetch
| | webclaw | web_fetch | | | webclaw | web_fetch |
|---|---------|-----------| |---|---------|-----------|
| Cloudflare bypass | Automatic (cloud fallback) | Fails (403) | | Bot-protected sites | Cloud fallback with `WEBCLAW_API_KEY` | Fails (403) |
| JS-rendered pages | Automatic fallback | Readability only | | JavaScript-rendered pages | Cloud fallback with a key | Readability only |
| Output quality | 20-step optimization pipeline | Basic HTML parsing | | Output quality | 20-step optimization pipeline | Basic HTML parsing |
| Structured extraction | LLM-powered, schema-based | None | | Structured extraction | LLM-powered, schema-based | None |
| Crawling | Full site crawl with sitemap | Single page only | | Crawling | Full site crawl with sitemap | Single page only |

52
skill/SKILL.md
View file

@ -1,6 +1,6 @@
--- ---
name: webclaw name: webclaw
description: Web extraction engine with antibot bypass. Scrape, crawl, extract, summarize, search, map, diff, monitor, research, and analyze any URL — including Cloudflare-protected sites. Use when you need reliable web content, the built-in web_fetch fails, or you need structured data extraction from web pages. description: Web extraction engine for LLMs and agents. Scrape, crawl, extract, summarize, search, map, diff, monitor, research, and analyze any URL into clean Markdown, text, or JSON, including pages that block bots or render with JavaScript. Use when you need reliable web content, the built-in web_fetch fails, or you need structured data extraction from web pages.
homepage: https://webclaw.io homepage: https://webclaw.io
user-invocable: true user-invocable: true
metadata: {"openclaw":{"emoji":"🦀","requires":{"env":["WEBCLAW_API_KEY"]},"primaryEnv":"WEBCLAW_API_KEY","homepage":"https://webclaw.io","install":[{"id":"npx","kind":"node","bins":["webclaw-mcp"],"label":"npx create-webclaw"}]}} metadata: {"openclaw":{"emoji":"🦀","requires":{"env":["WEBCLAW_API_KEY"]},"primaryEnv":"WEBCLAW_API_KEY","homepage":"https://webclaw.io","install":[{"id":"npx","kind":"node","bins":["webclaw-mcp"],"label":"npx create-webclaw"}]}}
@ -8,12 +8,12 @@ metadata: {"openclaw":{"emoji":"🦀","requires":{"env":["WEBCLAW_API_KEY"]},"pr
# webclaw # webclaw
High-quality web extraction with automatic antibot bypass. Beats Firecrawl on extraction quality and handles Cloudflare, DataDome, and JS-rendered pages automatically. Web extraction tuned for LLM and RAG pipelines. Bot-protected and JavaScript-rendered pages are handled for you when `WEBCLAW_API_KEY` is set; without it, those pages return a clear message explaining how to enable it.
## When to use this skill ## When to use this skill
- **Always** when you need to fetch web content and want reliable results - **Always** when you need to fetch web content and want reliable results
- When `web_fetch` returns empty/blocked content (403, Cloudflare challenges) - When `web_fetch` returns empty or blocked content (403s, bot challenges)
- When you need structured data extraction (pricing tables, product info) - When you need structured data extraction (pricing tables, product info)
- When you need to crawl an entire site or discover all URLs - When you need to crawl an entire site or discover all URLs
- When you need LLM-optimized content (cleaner than raw markdown) - When you need LLM-optimized content (cleaner than raw markdown)
@ -80,15 +80,7 @@ curl -X POST https://api.webclaw.io/v1/scrape \
- `llm` — optimized for LLM consumption: includes page title, URL, and cleaned content with link references. Best for feeding to AI models. - `llm` — optimized for LLM consumption: includes page title, URL, and cleaned content with link references. Best for feeding to AI models.
- `json` — full extraction result with all metadata - `json` — full extraction result with all metadata
**When antibot bypass activates** (automatic, no extra config): Protected or JavaScript-rendered pages are handled through the cloud fallback when `WEBCLAW_API_KEY` is set (see Smart Fetch Architecture below).
```json
{
"antibot": {
"bypass": true,
"elapsed_ms": 3200
}
}
```
### 2. Crawl — scrape an entire website ### 2. Crawl — scrape an entire website
@ -394,7 +386,7 @@ curl -X POST https://api.webclaw.io/v1/research \
-H "Authorization: Bearer $WEBCLAW_API_KEY" \ -H "Authorization: Bearer $WEBCLAW_API_KEY" \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"query": "How does Cloudflare Turnstile work and what are its known bypass methods?", "query": "Compare the top managed PostgreSQL providers on pricing, backups, and regions",
"max_iterations": 5, "max_iterations": 5,
"max_sources": 10, "max_sources": 10,
"topic": "security", "topic": "security",
@ -425,16 +417,16 @@ Response when complete:
{ {
"id": "res-abc-123", "id": "res-abc-123",
"status": "completed", "status": "completed",
"query": "How does Cloudflare Turnstile work and what are its known bypass methods?", "query": "Compare the top managed PostgreSQL providers on pricing, backups, and regions",
"report": "# Cloudflare Turnstile Analysis\n\n## Overview\nCloudflare Turnstile is a CAPTCHA replacement that...\n\n## How It Works\n...\n\n## Known Bypass Methods\n...", "report": "# Managed PostgreSQL Providers\n\n## Overview\n...\n\n## Pricing\n...\n\n## Backups and regions\n...",
"sources": [ "sources": [
{ "url": "https://developers.cloudflare.com/turnstile/", "title": "Turnstile Documentation" }, { "url": "https://neon.tech/pricing", "title": "Neon Pricing" },
{ "url": "https://blog.cloudflare.com/turnstile-ga/", "title": "Turnstile GA Announcement" } { "url": "https://supabase.com/pricing", "title": "Supabase Pricing" }
], ],
"findings": [ "findings": [
"Turnstile uses browser environment signals and proof-of-work challenges", "Entry tiers cluster around $19-25/mo for a small production instance",
"Managed mode auto-selects challenge difficulty based on visitor risk score", "Point-in-time recovery windows range from 7 to 30 days by tier",
"Known bypass approaches include instrumented browser automation" "Region coverage is widest on the largest providers"
], ],
"iterations": 5, "iterations": 5,
"elapsed_ms": 34200 "elapsed_ms": 34200
@ -600,7 +592,7 @@ curl -X DELETE https://api.webclaw.io/v1/watch/watch-abc-123 \
- **Batch over individual scrapes** when fetching multiple URLs — it's faster and more efficient. - **Batch over individual scrapes** when fetching multiple URLs — it's faster and more efficient.
- **Use `map` before `crawl`** to discover the site structure first, then crawl specific sections. - **Use `map` before `crawl`** to discover the site structure first, then crawl specific sections.
- **Use `extract` with a JSON schema** for reliable structured output (e.g., pricing tables, product specs, contact info). - **Use `extract` with a JSON schema** for reliable structured output (e.g., pricing tables, product specs, contact info).
- **Antibot bypass is automatic** — no extra configuration needed. Works on Cloudflare, DataDome, AWS WAF, and JS-rendered SPAs. - **Protected pages are handled for you** when `WEBCLAW_API_KEY` is set. Bot-protected sites and JavaScript-rendered SPAs fall back to the cloud engine; without a key they return a clear setup message.
- **Use `search` with `scrape: true`** to get full page content for each search result in one call instead of searching then scraping separately. - **Use `search` with `scrape: true`** to get full page content for each search result in one call instead of searching then scraping separately.
- **Use `research` for complex questions** that need multiple sources — it handles the search-read-synthesize loop automatically. Enable `deep: true` for thorough analysis. - **Use `research` for complex questions** that need multiple sources — it handles the search-read-synthesize loop automatically. Enable `deep: true` for thorough analysis.
- **Use `agent-scrape` for interactive pages** where data is behind filters, pagination, or form submissions that a simple scrape cannot reach. - **Use `agent-scrape` for interactive pages** where data is behind filters, pagination, or form submissions that a simple scrape cannot reach.
@ -608,23 +600,23 @@ curl -X DELETE https://api.webclaw.io/v1/watch/watch-abc-123 \
## Smart Fetch Architecture ## Smart Fetch Architecture
The webclaw MCP server uses a **local-first** approach: The webclaw MCP server uses a local-first approach:
1. **Local fetch** — fast, free, no API credits used (~80% of sites) 1. **Local fetch**: fast, free, no API credits used (most sites)
2. **Cloud API fallback** — automatic when bot protection or JS rendering is detected 2. **Cloud fallback**: used when a page is bot-protected or JavaScript-rendered, and only when `WEBCLAW_API_KEY` is set
This means: This means:
- Most scrapes cost zero credits (local extraction) - Most scrapes run locally and cost zero credits
- Cloudflare, DataDome, AWS WAF sites automatically fall back to the cloud API - Bot-protected sites fall back to the cloud engine when `WEBCLAW_API_KEY` is set
- JS-rendered SPAs (React, Next.js, Vue) also fall back automatically - JavaScript-rendered SPAs (React, Next.js, Vue) fall back the same way
- Set `WEBCLAW_API_KEY` to enable cloud fallback - Without a key, those pages return a clear message explaining how to enable the fallback
## vs web_fetch ## vs web_fetch
| | webclaw | web_fetch | | | webclaw | web_fetch |
|---|---------|-----------| |---|---------|-----------|
| Cloudflare bypass | Automatic (cloud fallback) | Fails (403) | | Bot-protected sites | Cloud fallback with `WEBCLAW_API_KEY` | Fails (403) |
| JS-rendered pages | Automatic fallback | Readability only | | JavaScript-rendered pages | Cloud fallback with a key | Readability only |
| Output quality | 20-step optimization pipeline | Basic HTML parsing | | Output quality | 20-step optimization pipeline | Basic HTML parsing |
| Structured extraction | LLM-powered, schema-based | None | | Structured extraction | LLM-powered, schema-based | None |
| Crawling | Full site crawl with sitemap | Single page only | | Crawling | Full site crawl with sitemap | Single page only |