The profile feed (/api/post/item_list) returns an empty 200 to headless
sessions but serves data headful on the same proxy IP. Run fetch_item_list
headful and dismiss the login modal that blocks mid-scroll.
Brings the three newer verbs to parity with tiktok.scrape everywhere it was
wired: MCP tools (+ selfcheck manifest), the chat subagent prompt/description,
the playground catalog, the native docs page, and the SEO marketing page. Also
adds live e2e stages for comments, user search, and trending. Docs/FAQ now
state the real contract: profile metadata is reliable while its video list can
be withheld, and keyword video search is walled (use user search for accounts).
The Explore feed (/api/explore/item_list) is a global trending-video feed
served to anonymous sessions, and it returns the same itemStruct shape as the
other listings — so the verb reuses parse_video, the listing flow, the
TikTokVideoItem output, and the per-video billing meter wholesale. Adds a
browser-capture marker + fetch_trending, a synthetic-target orchestrator entry,
and the tiktok.trending capability, surfaced on the chat subagent.
Comments load over a signed /api/comment/list XHR that TikTok serves to
anonymous sessions once the comments panel is opened (unlike profile-video and
general-search feeds), so this is a reliable verb. Given video URLs it returns
CommentItems (text, author, likes, reply counts; replies carry repliesToId),
deduped per video, capped, and degraded to an ErrorItem for empty/withheld
videos or a bad_url ErrorItem for non-video inputs.
Generalizes the browser capture over a pluggable interaction step so the
comments flow (open panel, scroll the panel to paginate) reuses the same
warm+capture scaffolding as listing/user-search. Billed per comment on a new
TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment
market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
Video/general search is login-walled for anonymous sessions, but the Users
tab (/api/search/user) returns public account records without a redirect, so
this exposes the one reliably-unblocked search path. A keyword yields
TikTokProfileItems (name, followers, bio, verification), deduped per query,
capped, and degraded to an ErrorItem when a query is empty/withheld.
Reuses the browser capture (generalized over XHR markers + extractor) and the
shared profile item shape. Billed per account on a new TIKTOK_USER meter
(TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
A profile's account data (name, followers, bio, verification) lives in the
page's rehydration blob and loads over plain HTTP without a signed request,
so emit it first and always. The video listing needs a signed item_list XHR
that TikTok withholds from anonymous sessions, so it stays best-effort and
degrades to an ErrorItem. A blocked profile now yields its metadata instead
of only an ErrorItem.
Profile and search feeds are trust-gated: an anonymous headless session
gets an empty item_list (profile) or no results XHR (search), while
hashtag feeds load. A zero-item listing now yields one honest ErrorItem
(errorCode="no_items") instead of vanishing silently, and ErrorItems are
excluded from billing so a blocked target is surfaced but never charged.
- Introduced a `docs_url` field to the Google Maps reviews, scrape, YouTube comments, YouTube scrape, and web crawl capabilities for improved documentation access.
- Simplified descriptions for each capability to enhance clarity and user understanding.
- Introduced a `docs_url` field to the `Capability` and `CapabilitySummary` classes for better documentation access.
- Updated the `REDDIT_SCRAPE` capability to include a specific documentation link.
- Enhanced the PlaygroundRunner component to display the documentation link when available, improving user guidance.
- Introduced a semaphore to limit concurrent page renders, enhancing resource management.
- Updated session handling to defer browser closure until all in-flight renders are complete.
- Improved comments for clarity on the behavior of concurrent fetches and session lifecycle.
- Cleaned up imports in alembic environment and migration flow scripts for consistency.
- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
- Introduced a comprehensive specification for renaming `searchSpace` to `workspace` in `surfsense_web` and `surfsense_desktop`, ensuring all TypeScript identifiers, React props, and local data structures are updated.
- Implemented migration shims for persisted local state to prevent data loss during the transition.
- Updated observability metrics and IPC channels to reflect the new naming convention.
- Removed legacy `active-search-space` module and replaced it with `active-workspace` to maintain consistency.
- Ensured no behavioral changes or data loss for users during the renaming process.
- Deleted the `README.md` from the `app/proprietary` directory, which outlined licensing and usage rules for proprietary components. This removal reflects a shift in project documentation strategy and simplifies the codebase.
- Added the ensure_publication function to create and verify the zero_publication if it is missing, ensuring idempotency during database initialization.
- Integrated ensure_publication into the create_db_and_tables function to prevent zero-cache crash loops on startup.
- Introduced a self-check script to validate the ensure_publication functionality on a create_all-bootstrapped database.
- Updated various components to reflect the transition from search space to workspace, including adjustments in imports and routing paths.
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Learned from lead-generation e2e runs (medical EMR, AI study app):
- routing.md: requested-N lists must be distinct entities that fit the
asked segment - branches/locations of one parent count once, parent
website domain is an ownership signal, fill shortfalls by widening
discovery instead of padding; place-shaped discovery routes to the
Maps specialist, open-web/enrichment to Search.
- google_maps prompt: exclusion criteria are strict (a satellite of an
excluded chain is still that chain); shared parent domains mean one
organization judged against the parent.
- google_search prompt: pure place-discovery tasks return blocked with
a next_step pointing to the Maps specialist.
Co-authored-by: Cursor <cursoragent@cursor.com>
Crawler engine: escalate thin JS-shell pages past static fetch, repair
currency-lossy extractions, emit categorized link records with anchor
provenance, and decode percent-encoded mailto:/tel: contacts; site crawls
reuse the connector ladder via Scrapling's spider engine with URL pattern
filters. Agent layer: read_run gains char_offset paging, search_run gains
match excerpts, new export_run turns stored runs into CSV workspace docs;
reddit search fair-shares the item budget across queries and dedupes
cross-query hits. Subagent prompts and routing teach crawl-after-search,
full-run coverage before summarizing, and executing own-tool next steps
instead of returning partial.
- Added `Run` model to track scraper invocations, including metadata such as status, input, and output.
- Implemented `ToolOutputSpill` model to store context-editing spills separately from user-facing logs.
- Updated middleware to handle spill placeholders and integrate with the new models.
- Enhanced REST API to record runs and expose run history through new endpoints.
- Adjusted tests to validate the new run logging functionality and ensure proper integration with existing capabilities.
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
`mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.
Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
and linkup-sdk deps and their config/env vars.
Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
malformed editable-install distribution with no `Version` field raised
KeyError deep in importlib.metadata, and since it runs on every
record_subagent_invoke_duration call it was crashing every task()
delegation. Verified end-to-end against live GPT-5.4.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users.
- Introduced `reddit.scrape` capability for fetching data using URLs and search queries.
- Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits.
- Created input/output models for the Reddit scraper to define request and response structures.
- Added documentation for the new Reddit scraping functionality and its usage.
- Integrated the Reddit subagent into the existing multi-agent chat framework.
