mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-17 23:55:13 +02:00
Merge pull request #67 from 0xMassi/docs/document-search-map-perf
docs(claude-md): document search, map, and perf
This commit is contained in:
Valerio
GitHub
commit
3caca67cd1
1 changed files with 56 additions and 19 deletions
75
CLAUDE.md
75
CLAUDE.md
|
|
@ -15,6 +15,7 @@ webclaw/
|
|||
# + proxy pool rotation (per-request)
|
||||
# + PDF content-type detection
|
||||
# + document parsing (DOCX, XLSX, CSV)
|
||||
# + layered URL discovery (map) + Serper web search (BYO key)
|
||||
webclaw-llm/ # LLM provider chain (Ollama -> OpenAI -> Anthropic)
|
||||
# + JSON schema extraction, prompt extraction, summarization
|
||||
webclaw-pdf/ # PDF text extraction via pdf-extract
|
||||
|
|
@ -30,25 +31,34 @@ Three binaries: `webclaw` (CLI), `webclaw-mcp` (MCP server), `webclaw-server` (R
|
|||
- `extractor.rs` — Readability-style scoring: text density, semantic tags, link density penalty
|
||||
- `noise.rs` — Shared noise filter: tags, ARIA roles, class/ID patterns. Tailwind-safe.
|
||||
- `data_island.rs` — JSON data island extraction for React SPAs, Next.js, Contentful CMS
|
||||
- `structured_data.rs` — JSON-LD, Next.js `__NEXT_DATA__`, and SvelteKit data-island extraction
|
||||
- `js_eval.rs` — QuickJS sandbox (rquickjs) that runs inline `<script>` tags to recover JS-assigned blobs (`window.__PRELOADED_STATE__`, Next.js `self.__next_f`) the static path can't see. Behind the default `quickjs` feature, gated `cfg(not(target_arch = "wasm32"))` — rquickjs links a C lib and won't build for wasm. Never ungate it (see Hard Rules). Runtime-gated for speed: the VM is skipped entirely when the page has no JS-candidate markers (`has_js_candidate_data`), and it reuses the already-parsed document instead of re-parsing.
|
||||
- `endpoints.rs` — API surface discovery: REST paths, GraphQL, and WebSocket endpoints mined from inline scripts + JS bundle text (regex over string literals, DoS-bounded). Pure: caller passes raw text.
|
||||
- `markdown.rs` — HTML to markdown with URL resolution, asset collection
|
||||
- `llm.rs` — 9-step LLM optimization pipeline (image strip, emphasis strip, link dedup, stat merge, whitespace collapse)
|
||||
- `llm/` — directory module (`mod` + `body`/`cleanup`/`images`/`links`/`metadata`): 9-step LLM optimization pipeline (image strip, emphasis strip, link dedup, stat merge, whitespace collapse)
|
||||
- `domain.rs` — Domain detection from URL patterns + DOM heuristics
|
||||
- `metadata.rs` — OG, Twitter Card, standard meta tag extraction
|
||||
- `types.rs` — Core data structures (ExtractionResult, Metadata, Content)
|
||||
- `filter.rs` — CSS selector include/exclude filtering (ExtractionOptions)
|
||||
- `types.rs` — Core data structures (ExtractionResult, Metadata, Content, plus ExtractionOptions for include/exclude CSS selectors — applied in `extractor.rs`; there is no `filter.rs`)
|
||||
- `diff.rs` — Content change tracking engine (snapshot diffing)
|
||||
- `brand.rs` — Brand identity extraction from DOM structure and CSS
|
||||
- `reddit.rs` — old.reddit.com thread vertical extractor (parses server-rendered HTML directly; no JS/API key). Test fixtures under `testdata/reddit/*.html` are `exclude`d from the published crate (Cargo.toml).
|
||||
- `youtube.rs` — `ytInitialPlayerResponse` parser, structured markdown for `youtube.com/watch` URLs (title, channel, views, published, duration, description). Produces the legacy markdown shape — for transcripts and a structured `YoutubeData` block see the production server's `youtube_transcript.rs` short-circuit (yt-dlp via proxy pool).
|
||||
|
||||
### Fetch Modules (`webclaw-fetch`)
|
||||
- `client.rs` — FetchClient with wreq BoringSSL TLS impersonation; implements the public `Fetcher` trait so callers (including server adapters) can swap in alternative implementations
|
||||
- `browser.rs` — Browser profiles: Chrome (142/136/133/131), Firefox (144/135/133/128)
|
||||
- `client.rs` — `FetchClient` with wreq BoringSSL TLS impersonation; also implements batch (`BatchResult`/`BatchExtractResult` — there is no `batch.rs`). Implements the public `Fetcher` trait so callers (incl. server adapters) can swap implementations.
|
||||
- `fetcher.rs` — the public `Fetcher` trait (`Send + Sync`). Vertical extractors take `&dyn Fetcher`, not `&FetchClient`.
|
||||
- `browser.rs` — `BrowserProfile`/`BrowserVariant` enums only (Chrome, ChromeMacos, Firefox, Safari, SafariIos26, Edge). No version numbers live here.
|
||||
- `tls.rs` — the real fingerprint builder: per-variant wreq `Emulation` (cipher/sigalg/curve lists, TLS extension order, HTTP/2 SETTINGS, header wire-order). Browser versions are set HERE: Chrome 145, Firefox 135, Edge 145, Safari 18.3.1, Safari iOS 26. SafariIos26 composes on top of `wreq_util::Profile::SafariIos26`. SSRF-safe redirect policy lives here too.
|
||||
- `extractors/` — ~28 vertical site extractors (Amazon, eBay, GitHub, Instagram, LinkedIn, Reddit, YouTube, npm, PyPI, HuggingFace, ...); `extractors/mod.rs` is the dispatch table. All reach the network through `&dyn Fetcher`. `extractors/og.rs` is the shared single-pass Open Graph (`og:*`) meta parser the verticals use (`raw()` vs `unescaped()`).
|
||||
- `crawler.rs` — BFS same-origin crawler with configurable depth/concurrency/delay
|
||||
- `sitemap.rs` — Sitemap discovery and parsing (sitemap.xml, robots.txt)
|
||||
- `batch.rs` — Multi-URL concurrent extraction
|
||||
- `sitemap.rs` — Sitemap discovery and parsing (sitemap.xml, robots.txt; gzip `.xml.gz` supported via `decode_sitemap_body`, sitemap-index recursion)
|
||||
- `map.rs` — layered URL discovery (`discover_urls` / `MapOptions`): sitemaps first, then a bounded same-origin crawl fallback when the sitemap is thin, harvesting links from fetched pages + the unfetched frontier (deduped against the sitemap set)
|
||||
- `search.rs` — web search via Serper.dev with the caller's own key (`search` / `SearchOptions` / `SearchResult`; pure `parse_serper_organic`). Plain wreq client (JSON API, no fingerprinting); optional bounded concurrent fetch+extract of result pages. Powers the CLI `search` subcommand, the MCP `search` tool, and the OSS server `POST /v1/search`.
|
||||
- `proxy.rs` — Proxy pool with per-request rotation
|
||||
- `document.rs` — Document parsing: DOCX, XLSX, CSV auto-detection and extraction
|
||||
- `search.rs` — Web search via Serper.dev with parallel result scraping
|
||||
- `cloud.rs` — `CloudClient` for hosted antibot escalation, exposed via `Fetcher::cloud()`
|
||||
- `locale.rs` — Accept-Language by TLD (`accept_language_for_tld` / `_for_url`)
|
||||
- `url_security.rs` — SSRF guards + SSRF-safe redirect policy
|
||||
|
||||
### LLM Modules (`webclaw-llm`)
|
||||
- Provider chain: Ollama (local-first) -> OpenAI -> Anthropic
|
||||
|
|
@ -59,26 +69,31 @@ Three binaries: `webclaw` (CLI), `webclaw-mcp` (MCP server), `webclaw-server` (R
|
|||
|
||||
### MCP Server (`webclaw-mcp`)
|
||||
- Model Context Protocol server over stdio transport
|
||||
- 8 tools: scrape, crawl, map, batch, extract, summarize, diff, brand
|
||||
- 12 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, research, search, list_extractors, vertical_scrape. `search` is local-first via the caller's `SERPER_API_KEY` (falls back to the hosted API when unset); `research` uses the hosted deep-research API. The rest run locally.
|
||||
- Works with Claude Desktop, Claude Code, and any MCP client
|
||||
- Uses `rmcp` crate (official Rust MCP SDK)
|
||||
|
||||
### REST API Server (`webclaw-server`)
|
||||
- Axum 0.8, stateless, no database, no job queue
|
||||
- 8 POST routes + /health, JSON shapes mirror api.webclaw.io where the
|
||||
capability exists in OSS
|
||||
- 10 POST routes (incl. `POST /v1/scrape/{vertical}` and `POST /v1/search`) +
|
||||
`GET /v1/extractors` + `GET /health`. JSON shapes mirror api.webclaw.io
|
||||
where the capability exists in OSS. The vertical surface
|
||||
(`routes/structured.rs`) mirrors the MCP `list_extractors` /
|
||||
`vertical_scrape` tools. `POST /v1/search` is gated on `SERPER_API_KEY`
|
||||
(returns 501 when unset).
|
||||
- Constant-time bearer-token auth via `subtle::ConstantTimeEq` when
|
||||
`--api-key` / `WEBCLAW_API_KEY` is set; otherwise open mode
|
||||
- Hard caps: crawl ≤ 500 pages, batch ≤ 100 URLs, 20 concurrent
|
||||
- Does NOT include: anti-bot bypass, JS rendering, async jobs,
|
||||
multi-tenant auth, billing, proxy rotation, search/research/watch/
|
||||
multi-tenant auth, billing, proxy rotation, research/watch/
|
||||
agent-scrape. Those live behind api.webclaw.io and are closed-source.
|
||||
(Web search IS available here as a bring-your-own-Serper-key path.)
|
||||
|
||||
## Hard Rules
|
||||
|
||||
- **Core has ZERO network dependencies** — takes `&str` HTML, returns structured output. Keep it WASM-compatible.
|
||||
- **webclaw-fetch uses wreq 6.x** (BoringSSL). No `[patch.crates-io]` forks needed; wreq handles TLS internally.
|
||||
- **No special RUSTFLAGS** — `.cargo/config.toml` is currently empty of build flags. Don't add any.
|
||||
- **Core has ZERO network dependencies** — takes `&str` HTML, returns structured output. Keep it WASM-compatible. The `quickjs` feature (default ON) pulls in rquickjs, which links a C lib and can't target wasm32; it's gated `cfg(not(target_arch = "wasm32"))` in `lib.rs`. CI compiles webclaw-core for wasm32 both with AND without default features — never ungate that.
|
||||
- **webclaw-fetch pins wreq exactly**: `wreq = "=6.0.0-rc.29"` + `wreq-util = "=3.0.0-rc.12"` (BoringSSL). The `=` pin is deliberate — these are release candidates with no semver stability between rc.N builds. No `[patch.crates-io]` forks needed; wreq handles TLS internally.
|
||||
- **No build flags in `.cargo/config.toml`** (it is comments-only) — don't add any locally. BUT CI (`.github/workflows/ci.yml`, `deps.yml`) DOES export `RUSTFLAGS: "--cfg reqwest_unstable"` for the wreq path; don't remove it from CI.
|
||||
- **webclaw-llm uses plain reqwest**. LLM APIs don't need TLS fingerprinting, so no wreq dep.
|
||||
- **Vertical extractors take `&dyn Fetcher`**, not `&FetchClient`. This lets the production server plug in a `ProductionFetcher` that adds domain_hints routing and antibot escalation on top of the same wreq client.
|
||||
- **qwen3 thinking tags** (`<think>`) are stripped at both provider and consumer levels.
|
||||
|
|
@ -86,12 +101,28 @@ Three binaries: `webclaw` (CLI), `webclaw-mcp` (MCP server), `webclaw-server` (R
|
|||
## Build & Test
|
||||
|
||||
```bash
|
||||
cargo build --release # Both binaries
|
||||
cargo build --release # All three binaries (webclaw, webclaw-mcp, webclaw-server)
|
||||
cargo test --workspace # All tests
|
||||
cargo test -p webclaw-core # Core only
|
||||
cargo test -p webclaw-llm # LLM only
|
||||
```
|
||||
|
||||
CI (`.github/workflows/ci.yml`, with `RUSTFLAGS=--cfg reqwest_unstable`) runs four jobs — match them locally before pushing:
|
||||
- `cargo test --workspace`
|
||||
- `cargo fmt --check --all` + `cargo clippy --all -- -D warnings` (warnings fail CI)
|
||||
- `cargo check --target wasm32-unknown-unknown -p webclaw-core` **with and without** `--no-default-features` (guards the WASM-safe rule)
|
||||
- `cargo doc --no-deps --workspace`
|
||||
|
||||
## Repo Layout & Packaging
|
||||
|
||||
Workspace is version **0.6.13**, edition **2024**, license **AGPL-3.0** (matters for the public-OSS scrubbing rules). No crate declares `rust-version`, so MSRV is implicit — edition 2024 floors it at Rust 1.85+; CI pins `dtolnay/rust-toolchain@stable`.
|
||||
|
||||
Artifacts outside `crates/` that need separate attention:
|
||||
- `packages/create-webclaw/` — `npx create-webclaw` Node scaffolder that installs/configures the MCP server for AI agents (Claude, Cursor, Windsurf, ...). Versioned independently (own `package.json`) — bump it separately when MCP setup changes.
|
||||
- `smithery.yaml` + `glama.json` — MCP-registry manifests (Smithery stdio config spawning `webclaw-mcp` with optional `WEBCLAW_API_KEY`; Glama). Update when the MCP launch command or env changes.
|
||||
- `examples/` — runnable demos (cloudflare-diagnostics, firecrawl-compatible-api, html-to-markdown-rag, mcp-web-scraping, proxy-backed-crawling).
|
||||
- `Dockerfile` / `Dockerfile.ci` / `docker-compose.yml`, `benchmarks/` (`/benchmark` skill), `SKILL.md` + `skill/` (Claude Code skill).
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
|
|
@ -107,12 +138,18 @@ webclaw https://example.com --only-main-content
|
|||
webclaw url1 url2 url3 --proxy-file proxies.txt
|
||||
webclaw --urls-file urls.txt --concurrency 10
|
||||
|
||||
# Sitemap discovery
|
||||
# URL discovery (--map): sitemaps first, bounded crawl fallback when the sitemap is thin
|
||||
webclaw https://docs.example.com --map
|
||||
webclaw https://news.ycombinator.com --map --map-pages 150 --map-limit 500
|
||||
webclaw https://docs.example.com --map --no-map-crawl # sitemap-only (no crawl fallback)
|
||||
|
||||
# Crawling (with sitemap seeding)
|
||||
webclaw https://docs.example.com --crawl --depth 2 --max-pages 50 --sitemap
|
||||
|
||||
# Web search via Serper.dev (bring your own key: --serper-key or SERPER_API_KEY)
|
||||
webclaw search "rust async runtime" --num 5
|
||||
webclaw search "best web scraper" --scrape -f json # also fetch + extract result pages
|
||||
|
||||
# Change tracking
|
||||
webclaw https://example.com -f json > snap.json
|
||||
webclaw https://example.com --diff-with snap.json
|
||||
|
|
@ -140,8 +177,8 @@ cat page.html | webclaw --stdin
|
|||
|
||||
- Scoring minimum: 50 chars text length
|
||||
- Semantic bonus: +50 for `<article>`/`<main>`, +25 for content class/ID
|
||||
- Link density: >50% = 0.1x score, >30% = 0.5x
|
||||
- Data island fallback triggers when DOM word count < 30
|
||||
- Link density (generic divs): >50% = 0.1x score, >30% = 0.5x. Semantic nodes (article/main/role=main) get a milder curve: >70% = 0.3x, >50% = 0.5x (`extractor.rs`)
|
||||
- Data island fallback triggers when DOM word count < 500 (`SPARSE_THRESHOLD` in `data_island.rs`)
|
||||
- Eyebrow text max: 80 chars
|
||||
|
||||
## MCP Setup
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue