- Deleted the `00-umbrella-plan.md`, `00b-pipeline-diagrams.md`, `01-rename-db.md`, `02-rename-backend.md`, `03a-crawler-core.md`, `03b-proxy-expansion.md`, and `03c-crawl-billing.md` files as they are no longer relevant to the current project structure.
- This cleanup reflects the architectural pivot and the shift towards a more streamlined documentation approach.
- Integrated Reddit link into the CommunityStrip component for enhanced community engagement.
- Updated the FooterNew component to include a Reddit section with the corresponding icon.
- Modified JSON-LD structured data to include the Reddit URL for improved SEO and social media presence.
- Introduced a new schema structure for API connectors, including request and response fields.
- Updated connector implementations for Google Maps, Google Search, Reddit, Web Crawl, and YouTube to include detailed schema information.
- Enhanced the ConnectorPage component to render request and response schemas dynamically.
- Added SchemaField interface to standardize schema field definitions across connectors.
- Updated the pricing page to include improved SEO metadata, including title, description, and keywords.
- Added structured data using JsonLd for better search engine visibility.
- Revised pricing plan descriptions and features for clarity and accuracy.
- Enhanced link behavior for external pricing links to open in a new tab with appropriate security attributes.
- Updated FAQ sections to reflect changes in credit and billing explanations.
- Introduced new skills in the skills-lock.json for animation vocabulary, emil design engineering, and review animations, enhancing the skill set available for users.
- Homepage rebuilt around competitive-intelligence positioning: connector grid,
how-it-works, use-case art, compare table, FAQ, and a scripted new-chat hero
demo (typewriter prompt, agent timeline, streamed answer, /login on send)
- Marketing pages for the Reddit/YouTube/Maps/SERP/Web Crawl scrape APIs at
/<slug>, a /connectors hub, and a bespoke /mcp-connector page
- Remove stale app/dashboard/[search_space_id] tree that broke the build
(canonical route is [workspace_id])
Co-authored-by: Cursor <cursoragent@cursor.com>
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Learned from lead-generation e2e runs (medical EMR, AI study app):
- routing.md: requested-N lists must be distinct entities that fit the
asked segment - branches/locations of one parent count once, parent
website domain is an ownership signal, fill shortfalls by widening
discovery instead of padding; place-shaped discovery routes to the
Maps specialist, open-web/enrichment to Search.
- google_maps prompt: exclusion criteria are strict (a satellite of an
excluded chain is still that chain); shared parent domains mean one
organization judged against the parent.
- google_search prompt: pure place-discovery tasks return blocked with
a next_step pointing to the Maps specialist.
Co-authored-by: Cursor <cursoragent@cursor.com>
Crawler engine: escalate thin JS-shell pages past static fetch, repair
currency-lossy extractions, emit categorized link records with anchor
provenance, and decode percent-encoded mailto:/tel: contacts; site crawls
reuse the connector ladder via Scrapling's spider engine with URL pattern
filters. Agent layer: read_run gains char_offset paging, search_run gains
match excerpts, new export_run turns stored runs into CSV workspace docs;
reddit search fair-shares the item budget across queries and dedupes
cross-query hits. Subagent prompts and routing teach crawl-after-search,
full-run coverage before summarizing, and executing own-tool next steps
instead of returning partial.
- Added `Run` model to track scraper invocations, including metadata such as status, input, and output.
- Implemented `ToolOutputSpill` model to store context-editing spills separately from user-facing logs.
- Updated middleware to handle spill placeholders and integrate with the new models.
- Enhanced REST API to record runs and expose run history through new endpoints.
- Adjusted tests to validate the new run logging functionality and ensure proper integration with existing capabilities.
MCP consolidation:
- Route all MCP-capable connectors (Slack, Jira, Linear, ClickUp, Airtable,
Notion, Confluence, interim Gmail/Calendar, custom MCP) through a single
`mcp_discovery` subagent. Drive/OneDrive/Dropbox stay native to enrich the KB.
- Deprecate Discord/Teams/Luma: no viable official MCP server.
Google-only web search:
- Remove the main-agent `web_search` tool and the SearXNG platform service;
all public web search now flows through the `google_search` subagent via task().
- Deprecate the Tavily/SearXNG/Linkup/Baidu search connectors (HTTP 410 on
create, "Deprecated" badge); guide heavy users to the custom MCP connector.
- Remove web search from anonymous chat (pure Q&A).
- Tear SearXNG out of docker compose + install scripts; drop tavily-python
and linkup-sdk deps and their config/env vars.
Fix:
- metrics._package_version() now swallows any metadata lookup failure. A
malformed editable-install distribution with no `Version` field raised
KeyError deep in importlib.metadata, and since it runs on every
record_subagent_invoke_duration call it was crashing every task()
delegation. Verified end-to-end against live GPT-5.4.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Added a new `reddit` subagent to scrape structured data from Reddit posts, comments, and users.
- Introduced `reddit.scrape` capability for fetching data using URLs and search queries.
- Implemented tools for scraping and parsing Reddit data, including handling pagination and rate limits.
- Created input/output models for the Reddit scraper to define request and response structures.
- Added documentation for the new Reddit scraping functionality and its usage.
- Integrated the Reddit subagent into the existing multi-agent chat framework.
- Updated proxy configuration in `.env.example` files to use `PROXY_URL` and `PROXY_URLS` instead of `CUSTOM_PROXY_URL` and `CUSTOM_PROXY_URLS`.
- Introduced `DataImpulseProvider` for proxy management, replacing the deprecated `AnonymousProxiesProvider`.
- Enhanced documentation to reflect changes in proxy setup and usage.
- Adjusted related code in the proxy registry and configuration files to support the new provider structure.
