Bump the workspace to 0.6.16 (aligns with @webclaw/mcp@0.6.16, which
expects a matching binary release) and roll the changelog: Atlas Cloud
opt-in provider, the --urls-file single-row fix (#86), and the
@webclaw/mcp zero-install launcher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Atlas Cloud was merged as a provider type but wasn't reachable from the
binaries — nothing constructed it, so ATLASCLOUD_API_KEY had no effect.
Add it to ProviderChain::default() after Anthropic, gated on its key, so
it's only active when explicitly configured and never preempts an
already-configured provider. Also add an ATLASCLOUD_MODEL env fallback so
chain users (new(None, None, None)) can override the model, matching the
existing GEMINI_MODEL pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add Simplified Chinese README (README_zh-CN.md)
* chore: add README language switcher and FUNDING sponsor link
* docs: unlink Firecrawl anchor in zh README and align switcher link style
Independent review found the backfill ran AFTER the --diff-with and --brand
gates, so single-row --urls-file + --diff/--brand still hit 'no input'.
Move the collect_urls + backfill before all mode gates so a one-line file
behaves like a positional URL everywhere. --watch now reuses the shared
entries instead of re-collecting (avoids double-counting the backfilled URL).
Verified: --brand single-row no longer errors; 37 tests pass, clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per maintainer decision: expose Atlas Cloud as an explicit provider
(--llm-provider atlascloud / WEBCLAW_LLM_PROVIDER=atlascloud) but do NOT
auto-add it to the default ProviderChain when ATLASCLOUD_API_KEY is set.
Reverts the chain.rs integration and drops ATLASCLOUD_API_KEY from the
'no providers available' hint (it's not a default provider). The provider
itself, its registration, and the explicit CLI branch are kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A one-line --urls-file produced entries.len()==1, which the batch gates
(run_batch / run_batch_llm require >1) skipped, so control fell to the
single-scrape path — which reads cli.urls (empty) and errored 'no input
provided'. Two+ rows worked; exactly one did not.
Backfill the lone file URL into cli.urls before the gates so a one-line
file behaves identically to a positional URL (stdout, --output-dir, cloud
fallback, --raw-html, --diff, LLM paths). The comma/CSV url,filename parsing
is unchanged (verified irrelevant to this bug). Adds 3 network-free tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Claude Code / agent skill moved to its own repo (github.com/0xMassi/webclaw-skill,
skills/webclaw + skills/lead-enrichment). Core still git-tracked two byte-identical,
outdated copies (SKILL.md + skill/SKILL.md) — an OLD hosted-API/curl skill (key-required,
phantom Agent-Scrape/Watch tools) that also caused registries to auto-derive a stale
'/tree/main/skill' source link. Remove both and repoint CLAUDE.md at the canonical repo.
Also fix the stale workspace-version line (0.6.13 -> 0.6.15).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Published to the MCP registry as io.github.0xMassi/webclaw@0.6.17. Description
no longer names a competitor (public-scrubbing posture); packages[].version
stays 0.6.16 (the live @webclaw/mcp on npm — the registry validates it exists).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
npm pack was bundling packages/create-webclaw/.mcpregistry_*_token (the
mcp-publisher credential files) into the tarball because the package had no
'files' field and npm did not honor the repo-root .gitignore. Restrict the
published package to index.mjs + README.md (matches @webclaw/mcp).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The launcher is published (@webclaw/mcp@0.6.16), so the in-repo MCP config
now uses the canonical npx form instead of a local ~/.webclaw binary path.
Also ignore the mcp-publisher credential files so they can never be
committed from packages/create-webclaw/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aligns with the industry standard: Firecrawl, the official MCP reference servers,
Exa, Brave, Apify, Bright Data, etc. all install their MCP via a single
`npx -y <pkg>` config line — none use a create-* scaffolder for the MCP itself
(create-* is a project-bootstrap convention). create-webclaw now auto-detects AI
tools and writes the canonical `npx @webclaw/mcp` config into each, byte-identical
to webclaw.io/docs and the MCP registries, so a scaffolded config can never
diverge from a hand-written one.
- buildMcpEntry + all writers emit {command:"npx", args:["-y","@webclaw/mcp"]}
across every format (JSON, Continue array, OpenCode array, Codex TOML)
- remove the entire binary-download machinery (getTarget/download/extract/cargo
fallback, ~250 lines) — the @webclaw/mcp launcher handles that now
- README: reflect the new behavior + show the one-block manual config
- version 0.1.6 -> 0.1.7
Verified: correct npx config (with/without API key) across all formats;
idempotent re-runs; zero binary-download code remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The published bin was named `webclaw-mcp`, which npx shadows with any existing
`webclaw-mcp` on PATH (Homebrew/cargo/create-webclaw) — so `npx @webclaw/mcp`
silently ran a stale binary instead of the launcher, for exactly our existing
CLI users. Rename the bin to the unique `webclaw-mcp-launcher`; `npx
@webclaw/mcp` still resolves the single bin. Verified end-to-end: with a stale
webclaw-mcp on PATH, npx now runs the launcher and returns 14 tools.
- bin: webclaw-mcp -> webclaw-mcp-launcher; version 0.6.15 -> 0.6.16
- mark webclaw-mcp.mjs executable
- release.yml: version the launcher INDEPENDENTLY of the core release (a
launcher-only fix must ship without a core bump) — publish the next patch
above the current npm version and pin RELEASE_TAG to the release tag
- server.json: sync to 0.6.16
Published + deprecated 0.6.15 (bin collision); 0.6.16 is the first good release.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- README: rewrite to match the main webclaw README's voice/positioning
(clean markdown/JSON/LLM-ready context + anti-bot angle), accurate 14-tool
local/cloud split, consistent links. This is the package's npm landing page.
- version @webclaw/mcp in lockstep with the core release (0.6.15) so
@webclaw/mcp@X.Y.Z always installs webclaw vX.Y.Z; sync the MCP-registry
server.json version to match.
- release.yml: add an `npm` job (needs: release) that pins the launcher's
RELEASE_TAG to the tag and publishes @webclaw/mcp on stable tags. Requires
an NPM_TOKEN repo secret with publish rights on the @webclaw scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`npx -y @webclaw/mcp` downloads the prebuilt webclaw-mcp binary (verified
against the release SHA256SUMS), caches it, and runs it as an MCP stdio server.
This makes webclaw introspectable in MCP registries (Glama/Smithery/MCP
registry) and installable in any MCP client with no Rust build.
- packages/webclaw-mcp: zero-dep launcher (.mjs) + package.json + README
- README/CLAUDE/examples/smithery: mcpServers config -> npx -y @webclaw/mcp
- registry: point the MCP-registry entry (server.json) at @webclaw/mcp, the
real stdio server, instead of create-webclaw (a scaffolder); drop
create-webclaw's mcpName claim so ownership isn't duplicated
- CHANGELOG: unreleased entry
create-webclaw stays as the auto-detect installer. .mcp.json is intentionally
left pointing at the local install path until @webclaw/mcp is published to npm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
README said "10 MCP Tools" and omitted list_extractors, vertical_scrape,
lead, lead_batch; the server now has 14. Drop the hard count, list all
14, and add lead/lead_batch (require WEBCLAW_API_KEY). Correct the wrong
config paths (OpenCode -> ~/.config/opencode/opencode.json, Codex ->
~/.codex/config.toml TOML, Antigravity ~/.config variant), fix the
offline-tool count, and drop the stale TLS-library link. server.json tool
count 12 -> 14. No installer logic or version change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hosted /v1/lead (single) and /v1/lead/batch (async) endpoints had no
MCP surface, so Claude Desktop/Cursor/etc. couldn't enrich leads and the
skill advertised a `lead` tool that didn't exist. Add `lead` and
`lead_batch` mirroring the existing `research` tool: cloud-only (require
WEBCLAW_API_KEY), no in-process founder-recovery. `lead` proxies
POST /v1/lead; `lead_batch` starts POST /v1/lead/batch then polls
GET /v1/lead/batch/{id} to completion (up to 25 URLs, ~10 min cap).
Tool count 12 → 14 (get_info string + CLAUDE.md synced).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GitHub placements (README sponsor section + the proxy-backed crawling
guide) now carry utm_source=github so sponsorship clicks attribute to
the surface that drove them; the webclaw.io placements use their own
source tag.
Promote the one-command skills.sh install (npx skills add
0xMassi/webclaw-skill) alongside the MCP Server section. Notes the
keyless-by-default local path and the optional WEBCLAW_API_KEY upgrade.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The public SKILL.md named a competitor ("Beats Firecrawl") and protection
vendors (Cloudflare, DataDome, AWS WAF), described bypass mechanics, and
claimed the OSS path handles protected/JS pages automatically — which is
only true when WEBCLAW_API_KEY is set.
- Remove competitor and vendor names; describe capabilities generically
(bot-protected / JavaScript-rendered pages).
- Drop the antibot response-shape block and the bot-bypass research
example; use a neutral research example instead.
- Correct the over-claims: cloud fallback for protected/JS pages happens
only when WEBCLAW_API_KEY is set; otherwise a clear message is returned.
- Apply the same edits to the skill/ copy so both stay identical.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
clap prints the current value of an arg's `env` var in --help by default,
which leaks secrets (e.g. `[env: WEBCLAW_API_KEY=wc_...]`) for anyone who
has them set. Add `hide_env_values = true` to every arg backed by a
credential-bearing env var so --help shows the variable name only.
Covers the CLI (WEBCLAW_API_KEY, SERPER_API_KEY, WEBCLAW_PROXY,
WEBCLAW_PROXY_FILE, WEBCLAW_WEBHOOK_URL, WEBCLAW_LLM_BASE_URL) and the
server (WEBCLAW_API_KEY).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidate on shieldcn.dev, which already powers the stars badge, so the
README uses one chart/badge provider instead of also depending on
star-history.com. Keeps the responsive light/dark <picture> (mode=dark/light).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Distribution release — the extraction engine is unchanged from 0.6.13. Ships
the broader-Linux-compatibility work: gnu binaries are now built on glibc 2.35
(so they run on Debian 12 / Ubuntu 22.04), plus new static musl binaries that
run on any Linux (Alpine, Amazon Linux 2023, RHEL 9). Also carries the
create-webclaw Windows/all-platform install fix and the WEBCLAW_API_KEY
setup-script fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A v*-rc* tag triggered the same release pipeline as a stable tag, which
would publish a normal release, overwrite ghcr.io/0xmassi/webclaw:latest,
and repoint the Homebrew formula at the rc — shipping a prerelease to every
stable Docker/brew user. Guard all three on the SemVer prerelease hyphen:
- release: add --prerelease for hyphenated tags
- docker: still push :${tag} (testable rc image) but only move :latest for stable
- homebrew: skip entirely for prereleases
Lets us cut rc tags (e.g. to validate the new musl build job) without
touching stable users.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gnu Linux binaries are glibc-floored (2.35 after #74), so they still
won't run on Amazon Linux 2023 / RHEL 9 (glibc 2.34), Alpine, or anything
older. Add fully static musl builds that run on ANY Linux regardless of
glibc.
Adds x86_64-unknown-linux-musl and aarch64-unknown-linux-musl to the build
matrix, built with cargo-zigbuild (zig as the C/C++ cross-compiler for
BoringSSL). Build scripts (bindgen) run as the glibc host so libclang loads,
and the linked output is fully static. A native Alpine build can't do this —
its static build scripts can't dlopen libclang.
musl assets ship ALONGSIDE the gnu ones (gnu stays default; musl is the
runs-anywhere fallback). The release job globs *.tar.gz, so the new assets
are checksummed + uploaded automatically; the docker/homebrew jobs enumerate
gnu targets explicitly and are unaffected.
Validated in Docker: cargo-zigbuild produced a fully static aarch64-musl
webclaw-mcp (ldd: not a dynamic executable) that answered an MCP handshake on
Alpine, Debian 11 (glibc 2.31), Debian 12, Amazon Linux 2023 (2.34), and
Ubuntu 24.04 — everywhere, including where the gnu builds fail.
Closes#73
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Linux release binaries were built on ubuntu-latest (now Ubuntu 24.04,
glibc 2.39), so they required GLIBC_2.38 and failed to start on older LTS
distros (Debian 12 = 2.36, Ubuntu 22.04 = 2.35) with:
libc.so.6: version `GLIBC_2.38' not found (required by webclaw-mcp)
glibc is forward- but not backward-compatible, so the build host's glibc
sets the floor. Pin both Linux targets to ubuntu-22.04 (glibc 2.35); the
aarch64 cross toolchain tracks the runner distro, so this lowers both.
Note: 2.35 still won't cover Amazon Linux 2023 / RHEL 9 (2.34) — full
coverage needs a musl static build. Tracked in #73.
Refs #73
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`npx create-webclaw` never used the prebuilt binary on any platform and
silently fell back to `cargo install`, which fails with "'cargo' is not
recognized" / "cargo: not found" unless Rust is installed. Four bugs:
1. Asset name mismatch: getAssetName() hardcoded `webclaw-mcp-<target>`,
but release assets are `webclaw-<tag>-<target>` (versioned, no `mcp-`
infix). The `find()` always returned undefined, so the prebuilt path
was never taken — on every OS, not just Windows. Now the asset name is
built from the release tag_name + a platform→target map.
2. `unzip` is absent on Windows. The `.zip` branch now uses PowerShell
`Expand-Archive` (ships with Windows 10/11) and keeps `unzip` only for
the non-Windows case.
3. The prebuilt failure was swallowed by a bare `catch {}`, hiding the
real cause (a 403 is almost always a GitHub API rate limit). The error
is now surfaced, with a rate-limit hint + GITHUB_TOKEN support on the
api.github.com request (token dropped on CDN redirects).
4. (missed by the report's own suggested fix) Archives extract into a
`webclaw-<tag>-<target>/` subdirectory holding three binaries, so the
old `chmod(BINARY_PATH)` hit a nonexistent path. webclaw-mcp is now
lifted out of that subdir to BINARY_PATH and the rest is cleaned up.
BINARY_NAME/BINARY_PATH also gain the `.exe` suffix on Windows so the
written MCP config points at a real file.
Tested in Docker (no Windows machine available):
- Linux amd64 + arm64 on Debian trixie: full flow installs the binary and
it answers a real MCP initialize handshake (serverInfo webclaw-mcp
0.6.13, 12 tools).
- Windows .zip path validated against the real release zip: Expand-Archive
equivalent extraction, nested `.exe` resolved + lifted, PE header `MZ`.
Executing the .exe needs Windows (the reporter confirmed that on Win11).
- Bug 3: with the GitHub API blocked, the new build prints the real reason
instead of "No pre-built binary found".
Closes#71
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Quantum Proxies is no longer a sponsor. Remove their Studio Partners entry
from the README and delete the banner/logo assets.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- LLM provider chain is Ollama -> OpenAI -> Gemini -> Anthropic; Gemini
was added ahead of Anthropic (Google Cloud credits preferred) but the
docs still listed Ollama -> OpenAI -> Anthropic.
- Document the top-level webclaw-fetch verticals reddit.rs / linkedin.rs
(distinct from extractors/ and webclaw-core parsers) and progress.rs.
- Bump extractor count ~28 -> ~30 and call out the shared helpers
(og.rs, github_common.rs, jsonld_product.rs, ecommerce_product.rs).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
setup.sh and deploy/hetzner.sh emitted WEBCLAW_AUTH_KEY into the server's
.env, but webclaw-server reads WEBCLAW_API_KEY (env = "WEBCLAW_API_KEY").
The generated key was silently ignored — and since hetzner.sh binds
0.0.0.0, the server refused to start at all (it rejects a public bind
without WEBCLAW_API_KEY). Fix both .env writers, plus the hetzner help
line that told users to grep the wrong name and the env.example sample.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring core/CLAUDE.md current with the slices rescued this cycle, and fold
in earlier /init corrections that were never committed.
New capabilities documented:
- search: webclaw-fetch `search.rs` (Serper BYO-key) + the CLI `search`
subcommand + the OSS `POST /v1/search` route (gated on SERPER_API_KEY)
+ the now-local-first MCP `search` tool.
- map: webclaw-fetch `map.rs` (`discover_urls`/`MapOptions`, sitemap +
bounded crawl fallback), gzip sitemap support, and the new
`--map-pages`/`--no-map-crawl`/`--map-limit` CLI flags.
- perf: shared `extractors/og.rs` parser and the QuickJS runtime gate /
parsed-document reuse noted on `js_eval.rs`.
Corrections folded in: real browser fingerprint versions live in tls.rs
(not browser.rs), accurate module/route lists, Repo Layout section, and
removal of the now-false "search lives only in production" notes.
Bumped the stated workspace version to 0.6.13.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
