mirror of
https://github.com/0xMassi/webclaw.git
synced 2026-06-20 02:24:05 +02:00
feat(search): standalone web search via Serper.dev (bring-your-own-key)
Rescued from the stale perf/audit-fixes branch and ported cleanly onto current main. OSS surfaces can now search without the hosted webclaw API when the caller supplies their own Serper.dev key (free at serper.dev). - webclaw-fetch::search() — calls Serper.dev directly (plain wreq client; a JSON API needs no fingerprinting) and, with scrape=true, fetches + extracts the top result pages concurrently (bounded) via the caller's FetchClient. parse_serper_organic() is pure and unit-tested. - MCP `search` tool: local-first — uses SERPER_API_KEY when set, else falls back to the hosted webclaw API. Adds country/lang/scrape params. - OSS REST server: POST /v1/search, gated on SERPER_API_KEY (501 when unset, with a setup hint). Adds ApiError::NotImplemented. - CLI: `webclaw search <query> [--serper-key|SERPER_API_KEY] [--num] [--country] [--lang] [--scrape] [--format]`. No new dependencies (reuses futures-util already in the tree). Original work by the prior author on perf/audit-fixes; this re-applies only the search slice onto main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
0c6f323f51
commit
06f151c560
10 changed files with 622 additions and 7 deletions
|
|
@ -38,16 +38,24 @@ pub enum ApiError {
|
|||
|
||||
#[error("internal: {0}")]
|
||||
Internal(String),
|
||||
|
||||
#[error("{0}")]
|
||||
NotImplemented(String),
|
||||
}
|
||||
|
||||
impl ApiError {
|
||||
pub fn bad_request(msg: impl Into<String>) -> Self {
|
||||
Self::BadRequest(msg.into())
|
||||
}
|
||||
#[allow(dead_code)]
|
||||
pub fn internal(msg: impl Into<String>) -> Self {
|
||||
Self::Internal(msg.into())
|
||||
}
|
||||
/// 501 — a capability the operator hasn't configured (e.g. search
|
||||
/// without `SERPER_API_KEY`). Distinct from `BadRequest` (client's
|
||||
/// fault) and `Internal` (our fault): it's a deployment-config gap.
|
||||
pub fn not_implemented(msg: impl Into<String>) -> Self {
|
||||
Self::NotImplemented(msg.into())
|
||||
}
|
||||
|
||||
fn status(&self) -> StatusCode {
|
||||
match self {
|
||||
|
|
@ -57,6 +65,7 @@ impl ApiError {
|
|||
Self::Fetch(_) => StatusCode::BAD_GATEWAY,
|
||||
Self::Extract(_) | Self::Llm(_) => StatusCode::UNPROCESSABLE_ENTITY,
|
||||
Self::Internal(_) => StatusCode::INTERNAL_SERVER_ERROR,
|
||||
Self::NotImplemented(_) => StatusCode::NOT_IMPLEMENTED,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue