Anish Sarkar
be26f00d20
refactor(instagram): enhance fetch logic for login wall detection
...
Updated the fetch_json function to raise InstagramAccessBlockedError immediately upon detecting a login redirect (302 -> /accounts/login/). This change prevents unnecessary IP rotations when encountering endpoint-level access blocks, improving the efficiency of the scraper's handling of Instagram's login wall.
2026-07-09 19:56:29 +05:30
Anish Sarkar
4af0b9dbbd
test(instagram): add regression test for InstagramAccessBlockedError handling
...
Introduced a new test to ensure that InstagramAccessBlockedError is properly propagated without causing deadlocks in the scraper's fan_out function. This regression test verifies that the error surfaces correctly when a blocked job is encountered, enhancing the resilience of the fetch process.
2026-07-09 19:09:06 +05:30
Anish Sarkar
e8f8eeab27
fix(instagram): improve error handling for Instagram access blocks in scraper
...
Updated the fan_out function to handle InstagramAccessBlockedError more gracefully. Instead of raising the error directly, it now puts the error into the results queue to prevent deadlocks. This change ensures that the consumer can handle access block scenarios without interrupting the processing of other jobs.
2026-07-09 19:08:18 +05:30
Anish Sarkar
5551ebbda7
chore(instagram): modify instagram svg
2026-07-09 19:08:00 +05:30
Anish Sarkar
6384409b1d
test(instagram): enhance fetch resilience tests for login redirects
...
Added tests to handle scenarios where a 200 status response is redirected to a login page, ensuring proper rotation of IPs and raising errors when persistent login redirects occur. Updated the _FakeSession class to support login wall detection.
2026-07-09 18:29:29 +05:30
Anish Sarkar
d800ca33a5
feat(instagram): handle soft login wall in fetch logic
...
Added detection for Instagram's soft login wall, which returns a 200 status with login HTML. Implemented a new function to identify login redirects and adjusted the fetch logic to treat these cases similarly to 401/403 responses.
2026-07-09 18:29:17 +05:30
Anish Sarkar
ba83d16cba
docs(web): add Instagram to MCP server tool docs
2026-07-09 16:07:50 +05:30
Anish Sarkar
f5aad487ac
docs(web): document Instagram native connector
2026-07-09 16:07:42 +05:30
Anish Sarkar
4e86710b26
feat(web): cross-link Instagram from sibling connectors
2026-07-09 16:07:34 +05:30
Anish Sarkar
fa2ac406f3
feat(web): add Instagram connector marketing page
2026-07-09 16:07:19 +05:30
Anish Sarkar
d00030d97c
feat(web): add Instagram playground platform and brand icon
2026-07-09 16:03:34 +05:30
Anish Sarkar
7309ca5ca0
test(instagram): add manual e2e scraper script
2026-07-09 16:03:29 +05:30
Anish Sarkar
e151346fbe
test(chat): include Instagram in subagent composition test
2026-07-09 16:03:23 +05:30
Anish Sarkar
458d223ef8
test(billing): cover Instagram item and comment billing
2026-07-09 16:02:54 +05:30
Anish Sarkar
2e66e714e2
test(instagram): add capability unit tests
2026-07-09 16:02:48 +05:30
Anish Sarkar
9da136070d
test(instagram): add platform-layer unit tests
2026-07-09 16:02:41 +05:30
Anish Sarkar
98eab8c4b5
feat(chat): add Instagram builtin subagent
2026-07-09 16:02:05 +05:30
Anish Sarkar
c5e3fd4006
feat(mcp): add Instagram scrape/comments/details tools
2026-07-09 16:01:55 +05:30
Anish Sarkar
929efe8152
feat(instagram): register instagram.* capability namespace
2026-07-09 16:01:49 +05:30
Anish Sarkar
7ea474be93
feat(instagram): add details capability verb
2026-07-09 16:01:38 +05:30
Anish Sarkar
38a26b1ccf
feat(instagram): add comments capability verb
2026-07-09 14:58:58 +05:30
Anish Sarkar
ff55537bce
feat(instagram): add scrape capability verb
2026-07-09 14:58:54 +05:30
Anish Sarkar
eccc142974
docs(instagram): add platform scraper README
2026-07-09 14:58:47 +05:30
Anish Sarkar
6270250f74
feat(instagram): add fan-out orchestrator with deterministic early-stop cleanup
2026-07-09 14:58:38 +05:30
Anish Sarkar
b1eff478fd
feat(instagram): add proxy-aware fetch with warm-up and rotate-on-block
2026-07-09 14:58:32 +05:30
Anish Sarkar
2d29001004
feat(instagram): add web-JSON parsers
2026-07-09 14:58:27 +05:30
Anish Sarkar
8dc8ad6a36
feat(instagram): add URL classifier and normalizer
2026-07-09 14:58:15 +05:30
Anish Sarkar
842a0e666d
feat(instagram): add platform input/output schemas
2026-07-09 14:58:11 +05:30
Anish Sarkar
51c2dfdfe6
feat(config): add Instagram per-item and per-comment scrape rates
2026-07-09 14:57:59 +05:30
Anish Sarkar
43660ee51b
feat(billing): map Instagram units to rate keys and display nouns
2026-07-09 14:57:53 +05:30
Anish Sarkar
f204db7bce
feat(billing): add INSTAGRAM_ITEM and INSTAGRAM_COMMENT units
2026-07-09 14:57:44 +05:30
Anish Sarkar
8486a4f914
feat(constants): add Instagram to SUBAGENT_TO_REQUIRED_CONNECTOR_MAP for enhanced multi-agent chat capabilities
2026-07-09 14:08:23 +05:30
Anish Sarkar
3bd864ed95
feat(env): add Instagram scraping configuration parameters to .env.example
2026-07-09 14:08:10 +05:30
Rohan Verma
3eb485ff31
Merge pull request #1575 from SyntaxSawdust/feature/1354-embedding-base-url
...
Support separate embedding base URL configuration
2026-07-08 17:11:42 -07:00
Rohan Verma
d2ed0bb382
Merge pull request #1570 from pelazas/fix/scheduler-task-map-bookstack
...
fix: dispatch periodic BookStack indexing from the scheduler task_map
2026-07-08 15:07:38 -07:00
DESKTOP-RTLN3BA\$punk
b8155c6c0e
Merge commit ' 1ae927a7db' into dev
2026-07-08 15:05:11 -07:00
CREDO23
ba375ea3ee
feat(tiktok): emit graceful ErrorItem for blocked/empty listings
...
Profile and search feeds are trust-gated: an anonymous headless session
gets an empty item_list (profile) or no results XHR (search), while
hashtag feeds load. A zero-item listing now yields one honest ErrorItem
(errorCode="no_items") instead of vanishing silently, and ErrorItems are
excluded from billing so a blocked target is surfaced but never charged.
2026-07-08 23:14:50 +02:00
CREDO23
f731d371ab
feat(tiktok): add TikTok SEO landing page and site-wide marketing parity
2026-07-08 20:36:46 +02:00
CREDO23
5f9472b122
feat(tiktok): advertise tiktok across agent prompts, MCP instructions, docs
2026-07-08 20:30:38 +02:00
CREDO23
ea6befdf59
docs(tiktok): native connector page + nav card
2026-07-08 20:21:29 +02:00
CREDO23
ed1c3a1f3d
feat(tiktok): expose tiktok.scrape on MCP, chat subagent, playground
2026-07-08 20:20:20 +02:00
CREDO23
2943d8b23c
feat(tiktok): tiktok.scrape capability + billing wire-up
2026-07-08 18:21:44 +02:00
CREDO23
9dd39faa50
test(tiktok): pin live fixtures, trim listing debug scaffolding
2026-07-08 17:53:29 +02:00
CREDO23
ea95a0529f
test(tiktok): live e2e smoke script
2026-07-08 17:14:13 +02:00
CREDO23
5a326f3818
fix(tiktok): warm browser listing + drop ctx-var-in-generator
2026-07-08 17:14:13 +02:00
CREDO23
ad8b73477a
feat(tiktok): browser-driven signed listings
2026-07-08 16:37:36 +02:00
CREDO23
daa0856c44
feat(tiktok): blob-first orchestrator + video flow
2026-07-08 16:07:07 +02:00
CREDO23
44b3e640d3
feat(tiktok): rotate-on-block fetch session
2026-07-08 15:51:17 +02:00
CREDO23
5688ab0678
feat(tiktok): scrape input schema
2026-07-08 15:51:17 +02:00
CREDO23
ef66063bfc
feat(tiktok): video normalizer and parser tests
2026-07-08 15:40:50 +02:00
