- Added support for extracting likes, comments, username, timestamp, and caption from Open Graph meta tags.
- Implemented fallback mechanisms to ensure graceful degradation when expected data is missing.
- Updated unit tests to validate new parsing logic and ensure robustness against unrecognized formats.
Enhanced the hero section by including Instagram in the list of platforms monitored by SurfSense. Additionally, introduced a new use case for "Social sentiment mining" that details the capabilities of pulling public posts and analyzing audience reactions on Instagram.
Introduced new tests to validate the behavior of the Instagram scraper when discovering anonymous profiles and handling hashtag searches. The tests ensure that profile lookups succeed without requiring authentication, while hashtag searches correctly raise an error when access is blocked, enhancing the robustness of the scraper's functionality.
Enhanced the Instagram scraper to clarify the requirements for accessing user profiles and hashtags. Updated the error message for blocked access to provide detailed guidance on necessary credentials. Introduced a regex for validating Instagram usernames and refined the discovery function to handle profile queries directly, improving user experience and error handling in anonymous mode.
Enhanced the fetch_json function to immediately raise InstagramAccessBlockedError for login-gated endpoints, preventing unnecessary IP rotations. Introduced a new constant for authentication-walled paths to streamline the detection of access blocks, improving the scraper's efficiency in handling Instagram's restrictions.
Revised tests for handling login redirects to improve clarity and efficiency. The test for immediate failure on endpoint-level walls was updated to prevent unnecessary IP rotations. Additionally, new tests were added to ensure that partial results are returned when encountering blocked targets, enhancing the resilience of the scraper's batch processing.
Updated the fan_out function to allow partial results when encountering blocked targets. Instead of aborting the entire batch on a hard login wall, the function now tracks blocked statuses and raises InstagramAccessBlockedError only if all targets are blocked. This change improves the scraper's resilience and efficiency in handling Instagram's access restrictions.
Updated the fetch_json function to raise InstagramAccessBlockedError immediately upon detecting a login redirect (302 -> /accounts/login/). This change prevents unnecessary IP rotations when encountering endpoint-level access blocks, improving the efficiency of the scraper's handling of Instagram's login wall.
Introduced a new test to ensure that InstagramAccessBlockedError is properly propagated without causing deadlocks in the scraper's fan_out function. This regression test verifies that the error surfaces correctly when a blocked job is encountered, enhancing the resilience of the fetch process.
Updated the fan_out function to handle InstagramAccessBlockedError more gracefully. Instead of raising the error directly, it now puts the error into the results queue to prevent deadlocks. This change ensures that the consumer can handle access block scenarios without interrupting the processing of other jobs.
Added tests to handle scenarios where a 200 status response is redirected to a login page, ensuring proper rotation of IPs and raising errors when persistent login redirects occur. Updated the _FakeSession class to support login wall detection.
Added detection for Instagram's soft login wall, which returns a 200 status with login HTML. Implemented a new function to identify login redirects and adjusted the fetch logic to treat these cases similarly to 401/403 responses.
