mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-08 22:25:12 +02:00
b041f3cddd
Adds 8 more vertical extractors using public JSON APIs. All hit
deterministic endpoints with no antibot risk. Live tests pass
against canonical URLs for each.
AI / ML ecosystem (3):
- crates_io \u2192 crates.io/api/v1/crates/{name}
- huggingface_dataset \u2192 huggingface.co/api/datasets/{path} (handles both
legacy /datasets/{name} and canonical {owner}/{name})
- arxiv \u2192 export.arxiv.org/api/query (Atom XML parsed by quick-xml)
Code / version control (2):
- github_pr \u2192 api.github.com/repos/{owner}/{repo}/pulls/{number}
- github_release \u2192 api.github.com/repos/{owner}/{repo}/releases/tags/{tag}
Infrastructure (1):
- docker_hub \u2192 hub.docker.com/v2/repositories/{namespace}/{name}
(official-image shorthand /_/nginx normalized to library/nginx)
Community / publishing (2):
- dev_to \u2192 dev.to/api/articles/{username}/{slug}
- stackoverflow \u2192 api.stackexchange.com/2.3/questions/{id} + answers,
filter=withbody for rendered HTML, sort=votes for
consistent top-answers ordering
Live test results (real URLs):
- serde: 942M downloads, 838B response
- 'Attention Is All You Need': abstract + authors, 1.8KB
- nginx official: 12.9B pulls, 21k stars, 17KB
- openai/gsm8k: 822k downloads, 1.7KB
- rust-lang/rust#138000: merged by RalfJung, +3/-2, 1KB
- webclaw v0.4.0: 2.4KB
- a real dev.to article: 2.2KB body, 3.1KB total
- python yield Q&A: score 13133, 51 answers, 104KB
Catalog now exposes 14 extractors via GET /v1/extractors. Total
unit tests across the module: 34 passing. Clippy clean. Fmt clean.
Marketing positioning sharpens: 14 dedicated extractors, all
deterministic, all 1-credit-per-call. Firecrawl's /extract is
5 credits per call and you write the schema yourself.
|
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| Cargo.toml | ||