apunkt/webclaw
1
0
Fork 0
mirror of https://github.com/0xMassi/webclaw.git synced 2026-06-15 23:35:14 +02:00
Code Issues Projects Releases Packages Wiki Activity
webclaw/Cargo.toml

46 lines
1.5 KiB
TOML
Raw Permalink Normal View History

Initial release: webclaw v0.1.0 — web content extraction for LLMs CLI + MCP server for extracting clean, structured content from any URL. 6 Rust crates, 10 MCP tools, TLS fingerprinting, 5 output formats. MIT Licensed | https://webclaw.io
2026-03-23 18:31:11 +01:00
[workspace]
resolver = "2"
members = ["crates/*"]
[workspace.package]
chore(release): v0.6.5 Reddit extraction moves from the dead .json API to old.reddit.com HTML.
2026-06-04 17:36:02 +02:00
version = "0.6.5"
Initial release: webclaw v0.1.0 — web content extraction for LLMs CLI + MCP server for extracting clean, structured content from any URL. 6 Rust crates, 10 MCP tools, TLS fingerprinting, 5 output formats. MIT Licensed | https://webclaw.io
2026-03-23 18:31:11 +01:00
edition = "2024"
perf(core): hot-path extraction speedups + senior-grade hardening Extraction ~22% faster on the corpus benchmark with byte-identical output: - hoist recompiled CSS selectors in the markdown noise path - single-pass shared og() meta parsing across vertical extractors - output-safe QuickJS gating (skip the JS VM when no candidate data) + reuse the already-parsed document instead of re-parsing - wreq connect_timeout + connection-pool tuning; dedup the retry loop Reliability + correctness: - char-boundary-safe truncation of LLM error bodies (shared helper) - HTTP connect/read timeouts on all LLM provider clients - isolate pdf-extract behind catch_unwind + spawn_blocking - OSS server: crawl inherits the shared fetch profile; ProviderChain built once in AppState; request TimeoutLayer API / safety / docs: - #[non_exhaustive] on public enums + result structs (+ builders) - #![forbid(unsafe_code)] on pure crates, deny on llm - //! crate docs + doctests; scrub bypass/vendor/target specifics from public crate docs and comments Tooling: [profile.release] lto/codegen-units/strip, MSRV pin, deny.toml + cargo-deny CI, macOS test matrix. CLI main.rs split into focused modules.
2026-06-04 20:22:00 +02:00
rust-version = "1.85"
feat: SvelteKit data extraction + license change to AGPL-3.0 - Extract structured JSON from SvelteKit kit.start() data arrays - Convert JS object literals (unquoted keys) to valid JSON - Data appears in structured_data field (machine-readable) - License changed from MIT to AGPL-3.0 - Bump to v0.3.4 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 20:37:56 +02:00
license = "AGPL-3.0"
Initial release: webclaw v0.1.0 — web content extraction for LLMs CLI + MCP server for extracting clean, structured content from any URL. 6 Rust crates, 10 MCP tools, TLS fingerprinting, 5 output formats. MIT Licensed | https://webclaw.io
2026-03-23 18:31:11 +01:00
repository = "https://github.com/0xMassi/webclaw"
perf(core): hot-path extraction speedups + senior-grade hardening Extraction ~22% faster on the corpus benchmark with byte-identical output: - hoist recompiled CSS selectors in the markdown noise path - single-pass shared og() meta parsing across vertical extractors - output-safe QuickJS gating (skip the JS VM when no candidate data) + reuse the already-parsed document instead of re-parsing - wreq connect_timeout + connection-pool tuning; dedup the retry loop Reliability + correctness: - char-boundary-safe truncation of LLM error bodies (shared helper) - HTTP connect/read timeouts on all LLM provider clients - isolate pdf-extract behind catch_unwind + spawn_blocking - OSS server: crawl inherits the shared fetch profile; ProviderChain built once in AppState; request TimeoutLayer API / safety / docs: - #[non_exhaustive] on public enums + result structs (+ builders) - #![forbid(unsafe_code)] on pure crates, deny on llm - //! crate docs + doctests; scrub bypass/vendor/target specifics from public crate docs and comments Tooling: [profile.release] lto/codegen-units/strip, MSRV pin, deny.toml + cargo-deny CI, macOS test matrix. CLI main.rs split into focused modules.
2026-06-04 20:22:00 +02:00
# Hardened release profile: thin LTO + a single codegen unit enable
# cross-crate inlining on the hot extraction path and shrink the binaries,
# and stripping symbols trims the shipped artifact. We deliberately do NOT
# set `panic = "abort"`: webclaw-pdf relies on std::panic::catch_unwind to
# recover from panics inside the pdf-extract parser, and abort would turn
# those recoverable panics into hard process kills.
[profile.release]
lto = "thin"
codegen-units = 1
strip = true
# Conservative, high-value hardening lints applied workspace-wide. Crates
# opt in via `[lints] workspace = true`. Kept deliberately narrow so
# `clippy -D warnings` stays green — the goal is hardening, not a cleanup
# sweep that would break the build.
[workspace.lints.rust]
unsafe_op_in_unsafe_fn = "warn"
[workspace.lints.clippy]
mem_forget = "warn"
Initial release: webclaw v0.1.0 — web content extraction for LLMs CLI + MCP server for extracting clean, structured content from any URL. 6 Rust crates, 10 MCP tools, TLS fingerprinting, 5 output formats. MIT Licensed | https://webclaw.io
2026-03-23 18:31:11 +01:00
[workspace.dependencies]
webclaw-core = { path = "crates/webclaw-core" }
webclaw-fetch = { path = "crates/webclaw-fetch" }
webclaw-llm = { path = "crates/webclaw-llm" }
webclaw-pdf = { path = "crates/webclaw-pdf" }
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
thiserror = "2"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
clap = { version = "4", features = ["derive", "env"] }
dotenvy = "0.15"
Reference in a new issue Copy permalink