Updated the fetch_json function to raise InstagramAccessBlockedError immediately upon detecting a login redirect (302 -> /accounts/login/). This change prevents unnecessary IP rotations when encountering endpoint-level access blocks, improving the efficiency of the scraper's handling of Instagram's login wall.
Introduced a new test to ensure that InstagramAccessBlockedError is properly propagated without causing deadlocks in the scraper's fan_out function. This regression test verifies that the error surfaces correctly when a blocked job is encountered, enhancing the resilience of the fetch process.
Updated the fan_out function to handle InstagramAccessBlockedError more gracefully. Instead of raising the error directly, it now puts the error into the results queue to prevent deadlocks. This change ensures that the consumer can handle access block scenarios without interrupting the processing of other jobs.
Added tests to handle scenarios where a 200 status response is redirected to a login page, ensuring proper rotation of IPs and raising errors when persistent login redirects occur. Updated the _FakeSession class to support login wall detection.
Added detection for Instagram's soft login wall, which returns a 200 status with login HTML. Implemented a new function to identify login redirects and adjusted the fetch logic to treat these cases similarly to 401/403 responses.
- Introduced a `docs_url` field to the Google Maps reviews, scrape, YouTube comments, YouTube scrape, and web crawl capabilities for improved documentation access.
- Simplified descriptions for each capability to enhance clarity and user understanding.
- Introduced a `docs_url` field to the `Capability` and `CapabilitySummary` classes for better documentation access.
- Updated the `REDDIT_SCRAPE` capability to include a specific documentation link.
- Enhanced the PlaygroundRunner component to display the documentation link when available, improving user guidance.
- Introduced a semaphore to limit concurrent page renders, enhancing resource management.
- Updated session handling to defer browser closure until all in-flight renders are complete.
- Improved comments for clarity on the behavior of concurrent fetches and session lifecycle.
- Cleaned up imports in alembic environment and migration flow scripts for consistency.
- Adjusted Google Maps and YouTube micro pricing in the .env.example file for better cost management.
- Introduced new environment variables for captcha solving and stealth browser hardening to improve scraping resilience.
- Removed outdated smoke test for scraper API endpoints to streamline testing.
- Enhanced anonymous chat agent's system prompt to clarify capabilities and suggest account creation for advanced features.
- Updated Reddit fetch logic to prioritize new session handling and improve resilience against IP-related issues.
- Added compacting functionality for scraper results to optimize data handling and presentation.
- Improved workspace and document management tools with clearer descriptions and enhanced functionality.
- Introduced new UI components for agent setup guidance in the web application.
- Introduced a comprehensive specification for renaming `searchSpace` to `workspace` in `surfsense_web` and `surfsense_desktop`, ensuring all TypeScript identifiers, React props, and local data structures are updated.
- Implemented migration shims for persisted local state to prevent data loss during the transition.
- Updated observability metrics and IPC channels to reflect the new naming convention.
- Removed legacy `active-search-space` module and replaced it with `active-workspace` to maintain consistency.
- Ensured no behavioral changes or data loss for users during the renaming process.
- Deleted the `README.md` from the `app/proprietary` directory, which outlined licensing and usage rules for proprietary components. This removal reflects a shift in project documentation strategy and simplifies the codebase.
- Added the ensure_publication function to create and verify the zero_publication if it is missing, ensuring idempotency during database initialization.
- Integrated ensure_publication into the create_db_and_tables function to prevent zero-cache crash loops on startup.
- Introduced a self-check script to validate the ensure_publication functionality on a create_all-bootstrapped database.
- Updated various components to reflect the transition from search space to workspace, including adjustments in imports and routing paths.
Add per-item, per-platform billing for the platform-native connectors (Reddit, Google Search, Google Maps places/reviews, YouTube videos/comments) through the capability gate/charge seam. Rates are config-driven with a shared wallet-credit module (wallet_credit) and a dedicated PlatformScrapeCreditService; agent and REST capability runs now record cost_micros. Google Maps scrape dual-meters places and attached reviews.
Remove the main-agent scrape_webpage tool now that the web.crawl capability covers single-page (maxCrawlDepth=0) and site crawling. The main agent now reaches crawling via task(web_crawler, ...). Update prompts, tool catalog, receipts, skills, proprietary docs, and tests; drop the obsolete chat-turn crawl fold path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Learned from lead-generation e2e runs (medical EMR, AI study app):
- routing.md: requested-N lists must be distinct entities that fit the
asked segment - branches/locations of one parent count once, parent
website domain is an ownership signal, fill shortfalls by widening
discovery instead of padding; place-shaped discovery routes to the
Maps specialist, open-web/enrichment to Search.
- google_maps prompt: exclusion criteria are strict (a satellite of an
excluded chain is still that chain); shared parent domains mean one
organization judged against the parent.
- google_search prompt: pure place-discovery tasks return blocked with
a next_step pointing to the Maps specialist.
Co-authored-by: Cursor <cursoragent@cursor.com>
Crawler engine: escalate thin JS-shell pages past static fetch, repair
currency-lossy extractions, emit categorized link records with anchor
provenance, and decode percent-encoded mailto:/tel: contacts; site crawls
reuse the connector ladder via Scrapling's spider engine with URL pattern
filters. Agent layer: read_run gains char_offset paging, search_run gains
match excerpts, new export_run turns stored runs into CSV workspace docs;
reddit search fair-shares the item budget across queries and dedupes
cross-query hits. Subagent prompts and routing teach crawl-after-search,
full-run coverage before summarizing, and executing own-tool next steps
instead of returning partial.
