feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
|
|
|
"""``youtube.scrape`` capability registration (billed per video; see config
|
|
|
|
|
``YOUTUBE_MICROS_PER_VIDEO``)."""
|
2026-07-02 20:18:53 +02:00
|
|
|
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
|
|
|
from app.capabilities.core import BillingUnit, Capability, register_capability
|
2026-07-02 20:18:53 +02:00
|
|
|
from app.capabilities.youtube.scrape.executor import build_scrape_executor
|
|
|
|
|
from app.capabilities.youtube.scrape.schemas import ScrapeInput, ScrapeOutput
|
|
|
|
|
|
|
|
|
|
YOUTUBE_SCRAPE = Capability(
|
|
|
|
|
name="youtube.scrape",
|
|
|
|
|
description=(
|
2026-07-08 03:38:08 +05:30
|
|
|
"Scrape public YouTube videos, channels, playlists, and subtitles. Use "
|
|
|
|
|
"urls or search_queries."
|
2026-07-02 20:18:53 +02:00
|
|
|
),
|
|
|
|
|
input_schema=ScrapeInput,
|
|
|
|
|
output_schema=ScrapeOutput,
|
|
|
|
|
executor=build_scrape_executor(),
|
feat(billing): meter platform scrapers per item; consolidate web scraping onto web.crawl
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-05 17:08:01 -07:00
|
|
|
billing_unit=BillingUnit.YOUTUBE_VIDEO,
|
2026-07-08 03:38:08 +05:30
|
|
|
docs_url="/docs/connectors/native/youtube",
|
2026-07-02 20:18:53 +02:00
|
|
|
)
|
|
|
|
|
|
|
|
|
|
register_capability(YOUTUBE_SCRAPE)
|
