mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-10 22:32:16 +02:00
feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews. Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
DESKTOP-RTLN3BA\$punk
parent
b8285a0b72
commit
80927a2872
48 changed files with 724 additions and 766 deletions
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: kb-research
|
||||
description: Structured approach to finding and synthesizing information from the user's knowledge base
|
||||
allowed-tools: scrape_webpage, read_file, ls_tree, grep
|
||||
allowed-tools: read_file, ls_tree, grep
|
||||
---
|
||||
|
||||
# Knowledge-base research
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
name: meeting-prep
|
||||
description: Pull together briefing materials before a scheduled meeting
|
||||
allowed-tools: task, scrape_webpage, read_file
|
||||
allowed-tools: task, read_file
|
||||
---
|
||||
|
||||
# Meeting preparation
|
||||
|
|
|
|||
|
|
@ -7,8 +7,9 @@ CRITICAL — ground factual answers in what you actually receive this turn:
|
|||
- injected workspace context (see `<dynamic_context>`),
|
||||
- the user's connected apps via `task(mcp_discovery, ...)` (Slack, Jira,
|
||||
Notion, Gmail, Calendar, etc. — live data that is NOT in the knowledge base),
|
||||
- results from your other tool calls (`scrape_webpage`) or the Google Search
|
||||
specialist via `task(google_search, ...)`,
|
||||
- results from your specialist calls — the web crawler via
|
||||
`task(web_crawler, ...)` or the Google Search specialist via
|
||||
`task(google_search, ...)`,
|
||||
- or substantive summaries returned by a `task` specialist you invoked.
|
||||
|
||||
For questions about the user's own files and notes, dispatch
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ Structured reasoning:
|
|||
- For non-trivial work, `<thinking>` / short `<plan>` before tool calls is fine.
|
||||
|
||||
Professional objectivity:
|
||||
- Accuracy over flattery; verify with **scrape_webpage** or **task** (e.g. `task(google_search, …)` for public facts) when unsure — don’t invent connector access.
|
||||
- Accuracy over flattery; verify with **task** (e.g. `task(web_crawler, …)` to read a page, `task(google_search, …)` for public facts) when unsure — don’t invent connector access.
|
||||
|
||||
Task management:
|
||||
- For 3+ steps, use todo tooling; update statuses promptly.
|
||||
|
|
|
|||
|
|
@ -16,6 +16,6 @@ Output style:
|
|||
Tool calls:
|
||||
- Parallelise independent calls in one turn.
|
||||
- For SurfSense-product questions, point the user to https://www.surfsense.com/docs;
|
||||
use **task(google_search, …)** / **scrape_webpage** for fresh public facts; integrations and
|
||||
use **task(google_search, …)** / **task(web_crawler, …)** for fresh public facts; integrations and
|
||||
heavy workflows → **task**.
|
||||
</provider_hints>
|
||||
|
|
|
|||
|
|
@ -3,7 +3,6 @@ You have two execution channels. Pick the one that owns the work — never
|
|||
simulate one with the other.
|
||||
|
||||
### 1. Direct tools (you call them yourself)
|
||||
- `scrape_webpage` — fetch the body of a specific public URL.
|
||||
- `update_memory` — curate persistent memory (see `<memory_protocol>`).
|
||||
- `write_todos` — maintain a structured plan when the turn series spans
|
||||
multiple specialists or steps. Mark each item
|
||||
|
|
@ -18,10 +17,11 @@ https://www.surfsense.com/docs. There is no docs-search tool; give the link.
|
|||
overviews, a specialist's summary of a SERP) are pointers, not sources.
|
||||
When the answer lives on a page — a team roster, a portfolio or directory
|
||||
listing, a pricing table, docs — fetch the page before answering:
|
||||
- One or a few known URLs → `scrape_webpage` directly.
|
||||
- One or a few known URLs → `task(web_crawler, …)` with those URLs (it
|
||||
fetches only the seeds at `maxCrawlDepth=0`).
|
||||
- A site section or many pages (a whole team + portfolio, every pricing
|
||||
page of a list of companies, a paginated directory) →
|
||||
`task(web_crawler, …)` with the seed URLs.
|
||||
`task(web_crawler, …)` with the seed URLs and a higher depth.
|
||||
Never answer with "you can find it at <URL>" for public facts your tools
|
||||
can retrieve — retrieve them, then answer with the facts and cite the page.
|
||||
Large results are fine: extract and return them, don't ask permission for
|
||||
|
|
@ -35,7 +35,8 @@ web search returns only snippets that need a second pass. Use the Search
|
|||
specialist for entities without a storefront (online-only companies,
|
||||
software vendors, publications), for facts and current events, and to
|
||||
enrich places Maps already found. When a lead list needs both, run Maps
|
||||
discovery first, then scrape or search the found websites for contacts.
|
||||
discovery first, then crawl (`task(web_crawler, …)`) or search the found
|
||||
websites for contacts.
|
||||
|
||||
**Requested-N lists count distinct entities that fit the ask.** When the
|
||||
user asks for N leads/items/results, every entry must be a distinct
|
||||
|
|
@ -222,7 +223,8 @@ user: "Post the launch announcement to #general and let me know when it's up."
|
|||
task(mcp_discovery, "In Slack, post '<launch announcement text>' to
|
||||
#general. Return the message permalink.")
|
||||
Next turn (with the receipt's `verifiable_url` in hand):
|
||||
scrape_webpage(url=<verifiable_url from the receipt>)
|
||||
task(web_crawler, "Crawl <verifiable_url from the receipt> and confirm
|
||||
the post is live; return what you find.")
|
||||
→ confirm the post is live, then tell the user it's up with the URL.
|
||||
If the reply has NO Receipt with `status="success"`, treat it as a
|
||||
silent failure: surface the error verbatim, do not retry.
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
"""``scrape_webpage`` — description + few-shot examples."""
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
- `scrape_webpage` — Fetch and extract readable content from a single URL.
|
||||
- Use when the user wants the actual page body (article, table, dashboard
|
||||
snapshot), not just search snippets.
|
||||
- Try the tool when a URL is given or referenced; don't refuse without
|
||||
attempting unless the URL is clearly unsafe or invalid.
|
||||
- Public web only. For URLs behind a connector (Notion pages, Linear
|
||||
issues, Confluence, anything that needs auth), use `task` with the
|
||||
matching specialist instead.
|
||||
- Args: `url`, `max_length` (default 50000).
|
||||
- Returns title, metadata, and markdown-ish body. Summarise clearly and
|
||||
link back with `[label](url)`.
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
<example>
|
||||
user: "Check out https://dev.to/some-article"
|
||||
→ scrape_webpage(url="https://dev.to/some-article")
|
||||
(Respond with a structured analysis — key points, takeaways.)
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "Read this article and summarize it for me: https://example.com/blog/ai-trends"
|
||||
→ scrape_webpage(url="https://example.com/blog/ai-trends")
|
||||
(Thorough summary using headings and bullets.)
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: (after discussing https://example.com/stats) "Can you get the live data from that page?"
|
||||
→ scrape_webpage(url="https://example.com/stats")
|
||||
(Always attempt scraping first. Never refuse before trying.)
|
||||
</example>
|
||||
|
||||
<example>
|
||||
user: "https://example.com/blog/weekend-recipes"
|
||||
→ scrape_webpage(url="https://example.com/blog/weekend-recipes")
|
||||
(When a user sends just a URL with no instructions, scrape it and provide
|
||||
a concise summary.)
|
||||
</example>
|
||||
|
|
@ -43,9 +43,9 @@
|
|||
like podcasts/videos), the operation did not happen — treat as
|
||||
failure and surface that to the user verbatim, do not retry blindly.
|
||||
|
||||
2. **`scrape_webpage`** — when a Receipt carries a `verifiable_url`
|
||||
2. **`task(web_crawler, …)`** — when a Receipt carries a `verifiable_url`
|
||||
(Notion page URL, Slack permalink, Jira issue URL, Linear identifier
|
||||
URL, etc.), you can fetch that URL and confirm the operation
|
||||
URL, etc.), you can crawl that URL and confirm the operation
|
||||
externally. Use this for high-stakes mutations the user explicitly
|
||||
called out (e.g. "send the launch email to the whole team") or when
|
||||
the subagent's self-report contradicts what the user expected.
|
||||
|
|
@ -54,7 +54,7 @@
|
|||
|
||||
- `status="success"`: the mutation already committed in the backend.
|
||||
If a `verifiable_url` is present and the request was high-stakes,
|
||||
you may `scrape_webpage` it to externally confirm. Otherwise trust
|
||||
you may crawl it via `task(web_crawler, …)` to externally confirm. Otherwise trust
|
||||
the Receipt and tell the user it is done. Celery-backed deliverables
|
||||
(podcasts, video presentations) also land here — the subagent
|
||||
already waited for the worker to finish, so a `success` Receipt
|
||||
|
|
@ -67,6 +67,6 @@
|
|||
their backend before returning. If you ever do see a pending
|
||||
Receipt, tell the user the work has been **kicked off** (quote the
|
||||
`external_id` / `preview` so they can find it later), do not
|
||||
`scrape_webpage` it, and do not re-dispatch the same
|
||||
crawl it, and do not re-dispatch the same
|
||||
`task(...)` call hoping it will be done "this time".
|
||||
</verification>
|
||||
|
|
|
|||
|
|
@ -6,7 +6,6 @@ Connector integrations, MCP, deliverables, etc. are delegated via ``task`` subag
|
|||
from __future__ import annotations
|
||||
|
||||
MAIN_AGENT_SURFSENSE_TOOL_NAMES_ORDERED: tuple[str, ...] = (
|
||||
"scrape_webpage",
|
||||
"update_memory",
|
||||
"create_automation",
|
||||
)
|
||||
|
|
|
|||
|
|
@ -23,17 +23,12 @@ from langchain_core.tools import BaseTool
|
|||
|
||||
from app.db import ChatVisibility
|
||||
|
||||
from .scrape_webpage import create_scrape_webpage_tool
|
||||
from .update_memory import (
|
||||
create_update_memory_tool,
|
||||
create_update_team_memory_tool,
|
||||
)
|
||||
|
||||
|
||||
def _build_scrape_webpage_tool(_deps: dict[str, Any]) -> BaseTool:
|
||||
return create_scrape_webpage_tool()
|
||||
|
||||
|
||||
def _build_create_automation_tool(deps: dict[str, Any]) -> BaseTool:
|
||||
# Deferred import: the automation package is a sibling under ``main_agent``
|
||||
# and is only needed at build time, mirroring the shared registry's
|
||||
|
|
@ -63,12 +58,11 @@ def _build_update_memory_tool(deps: dict[str, Any]) -> BaseTool:
|
|||
|
||||
|
||||
# Ordered to match the historical main-agent binding order:
|
||||
# scrape_webpage, create_automation, update_memory.
|
||||
# create_automation, update_memory.
|
||||
# Each entry is ``(factory, required_dependency_names)``.
|
||||
_MAIN_AGENT_TOOL_FACTORIES: dict[
|
||||
str, tuple[Callable[[dict[str, Any]], BaseTool], tuple[str, ...]]
|
||||
] = {
|
||||
"scrape_webpage": (_build_scrape_webpage_tool, ()),
|
||||
"create_automation": (
|
||||
_build_create_automation_tool,
|
||||
("workspace_id", "user_id", "llm"),
|
||||
|
|
|
|||
|
|
@ -1,372 +0,0 @@
|
|||
"""
|
||||
Web scraping tool for the SurfSense agent.
|
||||
|
||||
This module provides a tool for scraping and extracting content from webpages
|
||||
using the existing WebCrawlerConnector. For YouTube URLs, it fetches the
|
||||
transcript directly via the YouTubeTranscriptApi instead of crawling the page.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from fake_useragent import UserAgent
|
||||
from langchain_core.tools import tool
|
||||
from requests import Session
|
||||
from scrapling.fetchers import AsyncFetcher
|
||||
from youtube_transcript_api import YouTubeTranscriptApi
|
||||
|
||||
from app.proprietary.platforms.youtube.url_resolver import get_youtube_video_id
|
||||
from app.proprietary.web_crawler import (
|
||||
CrawlOutcomeStatus,
|
||||
WebCrawlerConnector,
|
||||
)
|
||||
from app.utils.proxy import get_proxy_url, get_requests_proxies
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _bill_successful_scrape() -> None:
|
||||
"""Fold one successful crawl into the current chat turn's bill (Phase 3c).
|
||||
|
||||
The cost rides the turn accumulator and settles at the premium
|
||||
``finalize_credit`` step — no separate wallet hit. Free / BYOK / anonymous
|
||||
turns (which never reserve/finalize) still record the line in the
|
||||
breakdown but are never debited. No-op when crawl billing is disabled or
|
||||
there is no active turn (e.g. non-chat callers).
|
||||
"""
|
||||
from app.services.token_tracking_service import get_current_accumulator
|
||||
from app.services.web_crawl_credit_service import WebCrawlCreditService
|
||||
|
||||
if not WebCrawlCreditService.billing_enabled():
|
||||
return
|
||||
acc = get_current_accumulator()
|
||||
if acc is None:
|
||||
return
|
||||
acc.add(
|
||||
model="web_crawl",
|
||||
prompt_tokens=0,
|
||||
completion_tokens=0,
|
||||
total_tokens=0,
|
||||
cost_micros=WebCrawlCreditService.successes_to_micros(1),
|
||||
call_kind="web_crawl",
|
||||
)
|
||||
|
||||
|
||||
def _bill_captcha_attempts(outcome) -> None:
|
||||
"""Fold captcha solve attempts (Phase 3d) into the current chat turn's bill.
|
||||
|
||||
Per *attempt*, not per success: a solve that didn't rescue the crawl still
|
||||
cost real solver money, so this runs before the success/failure branch.
|
||||
Mirrors :func:`_bill_successful_scrape` (rides the turn accumulator, settles
|
||||
at ``finalize_credit``; record-only on free/anonymous turns). No-op when
|
||||
captcha billing is off, there were no attempts, or no turn is active.
|
||||
"""
|
||||
from app.services.token_tracking_service import get_current_accumulator
|
||||
from app.services.web_crawl_credit_service import WebCrawlCreditService
|
||||
|
||||
if not WebCrawlCreditService.captcha_billing_enabled():
|
||||
return
|
||||
attempts = getattr(outcome, "captcha_attempts", 0) or 0
|
||||
if attempts <= 0:
|
||||
return
|
||||
acc = get_current_accumulator()
|
||||
if acc is None:
|
||||
return
|
||||
acc.add(
|
||||
model="web_crawl_captcha",
|
||||
prompt_tokens=0,
|
||||
completion_tokens=0,
|
||||
total_tokens=0,
|
||||
cost_micros=WebCrawlCreditService.captcha_solves_to_micros(attempts),
|
||||
call_kind="web_crawl_captcha",
|
||||
)
|
||||
|
||||
|
||||
def extract_domain(url: str) -> str:
|
||||
"""Extract the domain from a URL."""
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
domain = parsed.netloc
|
||||
if domain.startswith("www."):
|
||||
domain = domain[4:]
|
||||
return domain
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def generate_scrape_id(url: str) -> str:
|
||||
"""Generate a unique ID for a scraped webpage."""
|
||||
hash_val = hashlib.md5(url.encode()).hexdigest()[:12]
|
||||
return f"scrape-{hash_val}"
|
||||
|
||||
|
||||
def truncate_content(content: str, max_length: int = 50000) -> tuple[str, bool]:
|
||||
"""
|
||||
Truncate content to a maximum length.
|
||||
|
||||
Returns:
|
||||
Tuple of (truncated_content, was_truncated)
|
||||
"""
|
||||
if len(content) <= max_length:
|
||||
return content, False
|
||||
|
||||
# Prefer truncating at a sentence/paragraph boundary.
|
||||
truncated = content[:max_length]
|
||||
last_period = truncated.rfind(".")
|
||||
last_newline = truncated.rfind("\n\n")
|
||||
|
||||
boundary = max(last_period, last_newline)
|
||||
if boundary > max_length * 0.8: # only if the boundary isn't too far back
|
||||
truncated = content[: boundary + 1]
|
||||
|
||||
return truncated + "\n\n[Content truncated...]", True
|
||||
|
||||
|
||||
async def _scrape_youtube_video(
|
||||
url: str, video_id: str, max_length: int
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Fetch YouTube video metadata and transcript via the YouTubeTranscriptApi.
|
||||
|
||||
Returns a result dict in the same shape as the regular scrape_webpage output.
|
||||
"""
|
||||
scrape_id = generate_scrape_id(url)
|
||||
domain = "youtube.com"
|
||||
|
||||
# --- Video metadata via oEmbed ---
|
||||
residential_proxies = get_requests_proxies()
|
||||
|
||||
params = {
|
||||
"format": "json",
|
||||
"url": f"https://www.youtube.com/watch?v={video_id}",
|
||||
}
|
||||
oembed_url = "https://www.youtube.com/oembed"
|
||||
|
||||
try:
|
||||
oembed_fetch_start = time.perf_counter()
|
||||
oembed_page = await AsyncFetcher.get(
|
||||
oembed_url,
|
||||
params=params,
|
||||
proxy=get_proxy_url(),
|
||||
stealthy_headers=True,
|
||||
)
|
||||
logger.info(
|
||||
"[scrape_webpage][perf] source=oembed video=%s status=%s fetch_ms=%.1f",
|
||||
video_id,
|
||||
getattr(oembed_page, "status", None),
|
||||
(time.perf_counter() - oembed_fetch_start) * 1000,
|
||||
)
|
||||
video_data = oembed_page.json()
|
||||
except Exception:
|
||||
video_data = {}
|
||||
|
||||
title = video_data.get("title", "YouTube Video")
|
||||
author = video_data.get("author_name", "Unknown")
|
||||
|
||||
# --- Transcript via YouTubeTranscriptApi ---
|
||||
try:
|
||||
transcript_fetch_start = time.perf_counter()
|
||||
ua = UserAgent()
|
||||
http_client = Session()
|
||||
http_client.headers.update({"User-Agent": ua.random})
|
||||
if residential_proxies:
|
||||
http_client.proxies.update(residential_proxies)
|
||||
ytt_api = YouTubeTranscriptApi(http_client=http_client)
|
||||
|
||||
# Pick the first transcript (video's primary language) rather than
|
||||
# defaulting to English.
|
||||
transcript_list = ytt_api.list(video_id)
|
||||
transcript = next(iter(transcript_list))
|
||||
captions = transcript.fetch()
|
||||
|
||||
logger.info(
|
||||
"[scrape_webpage][perf] source=transcript video=%s fetch_ms=%.1f",
|
||||
video_id,
|
||||
(time.perf_counter() - transcript_fetch_start) * 1000,
|
||||
)
|
||||
logger.info(
|
||||
f"[scrape_webpage] Fetched transcript for {video_id} "
|
||||
f"in {transcript.language} ({transcript.language_code})"
|
||||
)
|
||||
|
||||
transcript_segments = []
|
||||
for line in captions:
|
||||
start_time = line.start
|
||||
duration = line.duration
|
||||
text = line.text
|
||||
timestamp = f"[{start_time:.2f}s-{start_time + duration:.2f}s]"
|
||||
transcript_segments.append(f"{timestamp} {text}")
|
||||
transcript_text = "\n".join(transcript_segments)
|
||||
except Exception as e:
|
||||
logger.warning(f"[scrape_webpage] No transcript for video {video_id}: {e}")
|
||||
transcript_text = f"No captions available for this video. Error: {e!s}"
|
||||
|
||||
content = f"# {title}\n\n**Author:** {author}\n**Video ID:** {video_id}\n\n## Transcript\n\n{transcript_text}"
|
||||
|
||||
content, was_truncated = truncate_content(content, max_length)
|
||||
word_count = len(content.split())
|
||||
|
||||
description = f"YouTube video by {author}"
|
||||
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": title,
|
||||
"description": description,
|
||||
"content": content,
|
||||
"domain": domain,
|
||||
"word_count": word_count,
|
||||
"was_truncated": was_truncated,
|
||||
"crawler_type": "youtube_transcript",
|
||||
"author": author,
|
||||
}
|
||||
|
||||
|
||||
def create_scrape_webpage_tool():
|
||||
"""
|
||||
Factory function to create the scrape_webpage tool.
|
||||
|
||||
Returns:
|
||||
A configured tool function for scraping webpages.
|
||||
"""
|
||||
|
||||
@tool
|
||||
async def scrape_webpage(
|
||||
url: str,
|
||||
max_length: int = 50000,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Scrape and extract the main content from a webpage.
|
||||
|
||||
Use this tool when the user wants you to read, summarize, or answer
|
||||
questions about a specific webpage's content, or to pull a site's
|
||||
contact details (emails, phone numbers, social profiles) for lead or
|
||||
competitive-intelligence work. This tool actually fetches and reads the
|
||||
full page content; JS-rendered pages are loaded in a real browser and
|
||||
auto-scrolled, so lazy-loaded listings (directories, infinite-scroll
|
||||
feeds) are captured too. For YouTube video URLs it fetches the
|
||||
transcript directly instead of crawling the page.
|
||||
|
||||
For a single page this returns its content plus any contacts found. To
|
||||
sweep a whole site (e.g. hunt a contact/privacy page for an email), use
|
||||
the web.crawl capability with maxCrawlDepth > 0 instead.
|
||||
|
||||
Common triggers:
|
||||
- "Read this article and summarize it"
|
||||
- "What does this page say about X?"
|
||||
- "Summarize this blog post for me"
|
||||
- "Find the contact email / socials for this company"
|
||||
- "What's in this webpage?"
|
||||
|
||||
Args:
|
||||
url: The URL of the webpage to scrape (must be HTTP/HTTPS)
|
||||
max_length: Maximum content length to return (default: 50000 chars)
|
||||
|
||||
Returns:
|
||||
A dictionary containing:
|
||||
- id: Unique identifier for this scrape
|
||||
- assetId: The URL (for deduplication)
|
||||
- kind: "article" (type of content)
|
||||
- href: The URL to open when clicked
|
||||
- title: Page title
|
||||
- description: Brief description or excerpt
|
||||
- content: The extracted main content (markdown format)
|
||||
- domain: The domain name
|
||||
- word_count: Approximate word count
|
||||
- was_truncated: Whether content was truncated
|
||||
- contacts: {emails, phones, socials} harvested from the page
|
||||
- links: every link on the page as {url, text, rel, kind} where
|
||||
kind is internal/external/social/email/tel. Use the anchor text
|
||||
to tie a link to an entity (e.g. which person a LinkedIn URL
|
||||
belongs to) and to pick the next page to scrape.
|
||||
- error: Error message (if scraping failed)
|
||||
"""
|
||||
scrape_id = generate_scrape_id(url)
|
||||
domain = extract_domain(url)
|
||||
|
||||
if not url.startswith(("http://", "https://")):
|
||||
url = f"https://{url}"
|
||||
|
||||
try:
|
||||
# YouTube URLs use the transcript API instead of crawling.
|
||||
video_id = get_youtube_video_id(url)
|
||||
if video_id:
|
||||
return await _scrape_youtube_video(url, video_id, max_length)
|
||||
|
||||
connector = WebCrawlerConnector()
|
||||
outcome = await connector.crawl_url(url)
|
||||
|
||||
# 03d: bill any captcha attempts (even if the crawl ultimately failed).
|
||||
_bill_captcha_attempts(outcome)
|
||||
|
||||
if outcome.status is not CrawlOutcomeStatus.SUCCESS or not outcome.result:
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": domain or "Webpage",
|
||||
"domain": domain,
|
||||
"error": outcome.error or "No content returned from crawler",
|
||||
}
|
||||
|
||||
result = outcome.result
|
||||
_bill_successful_scrape()
|
||||
content = result.get("content", "")
|
||||
metadata = result.get("metadata", {})
|
||||
|
||||
title = metadata.get("title", "")
|
||||
if not title:
|
||||
title = domain or url.split("/")[-1] or "Webpage"
|
||||
|
||||
description = metadata.get("description", "")
|
||||
if not description and content:
|
||||
first_para = content.split("\n\n")[0] if content else ""
|
||||
description = (
|
||||
first_para[:300] + "..." if len(first_para) > 300 else first_para
|
||||
)
|
||||
|
||||
content, was_truncated = truncate_content(content, max_length)
|
||||
word_count = len(content.split())
|
||||
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": title,
|
||||
"description": description,
|
||||
"content": content,
|
||||
"domain": domain,
|
||||
"word_count": word_count,
|
||||
"was_truncated": was_truncated,
|
||||
"crawler_type": result.get("crawler_type", "unknown"),
|
||||
"author": metadata.get("author"),
|
||||
"date": metadata.get("date"),
|
||||
# Contact/social signals from raw HTML (footer/legal boilerplate
|
||||
# the markdown omits) — surfaced for lead/CI discovery.
|
||||
"contacts": result.get("contacts"),
|
||||
# Per-anchor inventory (url/text/rel/kind): anchor text is the
|
||||
# raw material for tying targets to entities.
|
||||
"links": result.get("link_records"),
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
error_message = str(e)
|
||||
logger.error(f"[scrape_webpage] Error scraping {url}: {error_message}")
|
||||
return {
|
||||
"id": scrape_id,
|
||||
"assetId": url,
|
||||
"kind": "article",
|
||||
"href": url,
|
||||
"title": domain or "Webpage",
|
||||
"domain": domain,
|
||||
"error": f"Failed to scrape: {error_message[:100]}",
|
||||
}
|
||||
|
||||
return scrape_webpage
|
||||
|
|
@ -105,7 +105,7 @@ class Receipt(TypedDict, total=False):
|
|||
``None`` only when the operation failed before the backend assigned one."""
|
||||
|
||||
verifiable_url: str | None
|
||||
"""URL the parent can pass to ``scrape_webpage`` to verify the
|
||||
"""URL the parent can crawl (via ``task(web_crawler, …)``) to verify the
|
||||
operation. ``None`` when no public URL exists (Gmail, KB, raw images
|
||||
stored in the DB)."""
|
||||
|
||||
|
|
|
|||
|
|
@ -64,10 +64,6 @@ TOOL_CATALOG: list[ToolMetadata] = [
|
|||
name="search_knowledge_base",
|
||||
description="Search the user's knowledge base with hybrid semantic + keyword retrieval",
|
||||
),
|
||||
ToolMetadata(
|
||||
name="scrape_webpage",
|
||||
description="Scrape and extract the main content from a webpage",
|
||||
),
|
||||
ToolMetadata(
|
||||
name="create_automation",
|
||||
description="Draft an automation from an NL intent; user approves the card; tool saves",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue