The run-reader set gained export_run (CSV export of a stored run), but this
manifest assertion still expected only read_run/search_run. Add export_run so
the agent-door test reflects the shipped readers.
The Explore feed (/api/explore/item_list) is a global trending-video feed
served to anonymous sessions, and it returns the same itemStruct shape as the
other listings — so the verb reuses parse_video, the listing flow, the
TikTokVideoItem output, and the per-video billing meter wholesale. Adds a
browser-capture marker + fetch_trending, a synthetic-target orchestrator entry,
and the tiktok.trending capability, surfaced on the chat subagent.
Comments load over a signed /api/comment/list XHR that TikTok serves to
anonymous sessions once the comments panel is opened (unlike profile-video and
general-search feeds), so this is a reliable verb. Given video URLs it returns
CommentItems (text, author, likes, reply counts; replies carry repliesToId),
deduped per video, capped, and degraded to an ErrorItem for empty/withheld
videos or a bad_url ErrorItem for non-video inputs.
Generalizes the browser capture over a pluggable interaction step so the
comments flow (open panel, scroll the panel to paginate) reuses the same
warm+capture scaffolding as listing/user-search. Billed per comment on a new
TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment
market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
Video/general search is login-walled for anonymous sessions, but the Users
tab (/api/search/user) returns public account records without a redirect, so
this exposes the one reliably-unblocked search path. A keyword yields
TikTokProfileItems (name, followers, bio, verification), deduped per query,
capped, and degraded to an ErrorItem when a query is empty/withheld.
Reuses the browser capture (generalized over XHR markers + extractor) and the
shared profile item shape. Billed per account on a new TIKTOK_USER meter
(TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
Profile and search feeds are trust-gated: an anonymous headless session
gets an empty item_list (profile) or no results XHR (search), while
hashtag feeds load. A zero-item listing now yields one honest ErrorItem
(errorCode="no_items") instead of vanishing silently, and ErrorItems are
excluded from billing so a blocked target is surfaced but never charged.
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Crawler engine: escalate thin JS-shell pages past static fetch, repair
currency-lossy extractions, emit categorized link records with anchor
provenance, and decode percent-encoded mailto:/tel: contacts; site crawls
reuse the connector ladder via Scrapling's spider engine with URL pattern
filters. Agent layer: read_run gains char_offset paging, search_run gains
match excerpts, new export_run turns stored runs into CSV workspace docs;
reddit search fair-shares the item budget across queries and dedupes
cross-query hits. Subagent prompts and routing teach crawl-after-search,
full-run coverage before summarizing, and executing own-tool next steps
instead of returning partial.
- Added `Run` model to track scraper invocations, including metadata such as status, input, and output.
- Implemented `ToolOutputSpill` model to store context-editing spills separately from user-facing logs.
- Updated middleware to handle spill placeholders and integrate with the new models.
- Enhanced REST API to record runs and expose run history through new endpoints.
- Adjusted tests to validate the new run logging functionality and ensure proper integration with existing capabilities.
- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users.
- Introduced `reddit.scrape` capability for fetching data using URLs and search queries.
- Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits.
- Created input/output models for the Reddit scraper to define request and response structures.
- Added documentation for the new Reddit scraping functionality and its usage.
- Integrated the Reddit subagent into the existing multi-agent chat framework.
Resolve conflicts against the new native google-maps actor + repo-wide
ruff-format pass:
- Keep legacy webcrawler KB indexer + its test deleted (modify/delete).
- test_validators: keep WEBCRAWLER case removed (validator gone).
- test_fetch_resilience: keep platforms.youtube import path (our reorg).
- Relocate google_maps actor + tests scrapers/ -> platforms/ to match the
reorg convention (youtube already there); rewrite imports + fixture paths.
- Add missing __init__.py across the capabilities/ test subtree so duplicate
test basenames get unique module paths under importlib mode.
Note: google_maps fixture-backed tests error on ci_mvp too (fixtures/*.json
never committed upstream) - pre-existing, out of scope here.
web.crawl scrapes a single URL (maxCrawlDepth=0) or spiders a whole site,
backed by the proprietary site_crawler engine. Rewires the scraping subagent
tools and capability tests onto the new verb.
