apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-06-07 22:15:12 +02:00
Code Issues Projects Releases Packages Wiki Activity

docs(contributing): describe in-process wreq TLS, drop stale patched-deps

Browse source 
The TLS layer moved to wreq (BoringSSL) in-process; there is no longer a
[patch.crates-io] section or a separate TLS fork. Update the architecture
tree and crate-boundary notes to match.
This commit is contained in:
Valerio 2026-06-04 17:56:24 +02:00
parent 58d274ffe9
commit 9a63c1a3ca
1 changed files with 9 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

20
CONTRIBUTING.md
View file

@ -91,18 +91,16 @@ Body is optional but encouraged for non-trivial changes.
``` ```
webclaw (this repo) webclaw (this repo)
├── crates/ └── crates/
│ ├── webclaw-core/ # Pure extraction engine (HTML → markdown/json/text) ├── webclaw-core/ # Pure extraction engine (HTML → markdown/json/text)
│ ├── webclaw-fetch/ # HTTP client + crawler + sitemap + batch ├── webclaw-fetch/ # HTTP client (wreq/BoringSSL) + crawler + sitemap + batch
│ ├── webclaw-llm/ # LLM provider chain (Ollama → OpenAI → Anthropic) ├── webclaw-llm/ # LLM provider chain (Ollama → OpenAI → Anthropic)
│ ├── webclaw-pdf/ # PDF text extraction ├── webclaw-pdf/ # PDF text extraction
│ ├── webclaw-cli/ # CLI binary ├── webclaw-cli/ # CLI binary
│ └── webclaw-mcp/ # MCP server binary └── webclaw-mcp/ # MCP server binary
│
└── [patch.crates-io] # Points to webclaw-tls for TLS fingerprinting
``` ```
TLS fingerprinting lives in a separate repo: [webclaw-tls](https://github.com/0xMassi/webclaw-tls). The `[patch.crates-io]` section in `Cargo.toml` overrides rustls, h2, hyper, hyper-util, and reqwest with our patched forks for browser-grade JA4 + HTTP/2 Akamai fingerprinting. TLS fingerprinting is handled in-process by [wreq](https://crates.io/crates/wreq) (BoringSSL), so `webclaw-fetch` impersonates real browser TLS directly. There are no `[patch.crates-io]` forks or external TLS dependencies.
## Crate Boundaries ## Crate Boundaries
@ -111,7 +109,7 @@ Changes that cross crate boundaries need extra care:
| Crate | Network? | Key constraint | | Crate | Network? | Key constraint |
|-------|----------|----------------| |-------|----------|----------------|
| webclaw-core | No | Zero network deps, WASM-safe | | webclaw-core | No | Zero network deps, WASM-safe |
| webclaw-fetch | Yes (webclaw-http) | Uses [webclaw-tls](https://github.com/0xMassi/webclaw-tls) for TLS fingerprinting | | webclaw-fetch | Yes (wreq) | Browser TLS impersonation via wreq (BoringSSL); no patched deps |
| webclaw-llm | Yes (reqwest) | Plain reqwest — LLM APIs don't need TLS fingerprinting | | webclaw-llm | Yes (reqwest) | Plain reqwest — LLM APIs don't need TLS fingerprinting |
| webclaw-pdf | No | Minimal, wraps pdf-extract | | webclaw-pdf | No | Minimal, wraps pdf-extract |
| webclaw-cli | Yes | Depends on all above | | webclaw-cli | Yes | Depends on all above |