Anish Sarkar
e38ca19b18
Merge remote-tracking branch 'upstream/dev' into feat/instagram-scraper
2026-07-11 04:31:11 +05:30
Rohan Verma
ff695fa6d4
Merge pull request #1589 from CREDO23/feature/tiktok-scraper
...
[Feat] TikTok public data scraper (scrape, comments, user search, trending)
2026-07-10 15:38:15 -07:00
Anish Sarkar
ec3f47abf8
feat(web): surface carousel, tagged users, and location in connector marketing
2026-07-11 03:44:51 +05:30
Anish Sarkar
42f9f35ec7
docs(web): document new media fields and search_type aliasing
2026-07-11 03:44:45 +05:30
Anish Sarkar
819486ac46
docs(instagram): document PolarisMedia extraction and richer media fields
2026-07-11 03:44:38 +05:30
Anish Sarkar
75c3addd2b
test(instagram): cover PolarisMedia relay parsing and enriched media fields
2026-07-11 03:44:33 +05:30
Anish Sarkar
d3c65a37b1
refactor(instagram): drop unused slug field and mark share links unsupported
2026-07-11 03:44:28 +05:30
Anish Sarkar
4bc3d43b56
refactor(instagram): remove unused share-link redirect resolver
2026-07-11 03:44:23 +05:30
Anish Sarkar
452fe30aa4
refactor(instagram): drop login-walled comment fields and note search_type aliasing
2026-07-11 03:44:15 +05:30
Anish Sarkar
cb4adab852
feat(instagram): extract PolarisMedia relay JSON with carousel, tags, and location
2026-07-11 03:44:10 +05:30
Anish Sarkar
69bf748c2c
chore(.gitignore): ignore Instagram tagged test fixture
2026-07-11 03:25:39 +05:30
Anish Sarkar
b1fe739308
docs(instagram): update platform scraper README
2026-07-11 03:25:36 +05:30
Anish Sarkar
3e8f31152e
feat(web): update Instagram marketing copy
2026-07-11 03:25:34 +05:30
Anish Sarkar
b75ab81261
docs(web): refresh Instagram connector docs
2026-07-11 03:25:30 +05:30
Anish Sarkar
c001f4b16e
feat(subagent): update Instagram subagent system prompt
2026-07-11 03:25:25 +05:30
Anish Sarkar
ca7d23d33a
refactor(mcp): align Instagram scraper tool with capability changes
2026-07-11 03:25:23 +05:30
Anish Sarkar
27d22a9a2a
refactor(instagram): drop mentions from scrape capability
2026-07-11 03:25:20 +05:30
Anish Sarkar
5cac6612c3
refactor(instagram): update platform schemas and scraper for tagged media
2026-07-11 03:25:17 +05:30
Anish Sarkar
457be1871c
feat(instagram): improve parsing of Instagram media IDs and mentions
...
- Enhanced the regular expression for mentions to prevent trailing punctuation from being included in handles.
- Added support for extracting media IDs from deep-link meta tags in anonymous posts.
- Updated unit tests to validate the new media ID extraction and ensure proper handling of mentions.
2026-07-11 03:06:04 +05:30
Anish Sarkar
d41ccb7edd
feat(instagram): enhance Open Graph parsing for anonymous posts
...
- Added support for extracting likes, comments, username, timestamp, and caption from Open Graph meta tags.
- Implemented fallback mechanisms to ensure graceful degradation when expected data is missing.
- Updated unit tests to validate new parsing logic and ensure robustness against unrecognized formats.
2026-07-11 02:56:29 +05:30
Anish Sarkar
467f51fd5c
chore(.gitignore): add Instagram test fixtures for unit tests
2026-07-11 02:50:43 +05:30
Anish Sarkar
4bd3ec275c
delete(instagram): remove obsolete post fixture for Instagram tests
2026-07-11 02:50:26 +05:30
Anish Sarkar
949cc1b29f
chore(scripts): update Instagram e2e scraper script
2026-07-11 02:21:26 +05:30
Anish Sarkar
ba70ae1f45
test(instagram): update platform tests and add discovery fixtures
2026-07-11 02:21:14 +05:30
Anish Sarkar
82a63043f7
test(instagram): update capability tests for schema changes
2026-07-11 02:21:11 +05:30
Anish Sarkar
42d03fdbc7
feat(web): update Instagram marketing and playground catalog
2026-07-11 02:21:08 +05:30
Anish Sarkar
69a8433910
docs(web): refresh Instagram connector and MCP docs
2026-07-11 02:21:05 +05:30
Anish Sarkar
9e9dd8a124
feat(subagent): update Instagram subagent prompts and tools
2026-07-11 02:21:02 +05:30
Anish Sarkar
8fc86b7fa5
refactor(mcp): align Instagram scraper tool with new pipeline
2026-07-11 02:20:58 +05:30
Anish Sarkar
36bb4a1bea
docs(instagram): update platform scraper README
2026-07-11 02:20:54 +05:30
Anish Sarkar
4813dc96e3
refactor(instagram): streamline scraper, fetch, and parsing pipeline
2026-07-11 02:20:51 +05:30
Anish Sarkar
de1990f9f6
refactor(instagram): simplify scrape and details capability schemas
2026-07-11 02:20:46 +05:30
Anish Sarkar
8afa4c6fc6
refactor(instagram): remove unused comments capability
2026-07-11 02:20:39 +05:30
CREDO23
903502296e
fix(tiktok-marketing): stop advertising keyword search as a video source
...
The hero, intro, use case, API-reference fields, and two FAQs all listed search as a video path, contradicting the FAQ that admits keyword video search is login-walled. Reconcile every claim to one truth: videos via hashtag/profile/URL; keyword search returns no videos; keyword -> user_search for accounts.
2026-07-10 17:46:08 +02:00
CREDO23
6a0ff1495f
fix(tiktok-mcp): flag that search_queries returns no videos in the scrape tool
2026-07-10 17:40:04 +02:00
CREDO23
1911b78379
fix(tiktok-agent): drop keyword search from the routing description's video sources
2026-07-10 17:40:04 +02:00
CREDO23
cd82f861de
fix(tiktok-agent): stop steering the subagent to keyword search for videos
...
The playbook listed search_queries as a video source in three places, then contradicted itself. Replace with action-oriented guidance: videos via hashtags/URLs, accounts via user_search. Drop 'login-walled' (the agent has no login path).
2026-07-10 17:40:04 +02:00
CREDO23
9793751b44
docs(tiktok): trim stage 2 comment to the why
2026-07-10 17:28:47 +02:00
CREDO23
f74a73efdb
docs(tiktok): trim fetch_item_list retry docstring to intent
2026-07-10 17:28:47 +02:00
CREDO23
01bc3f8de3
docs(config): trim narration from xvfb + listing-retry comments
2026-07-10 17:28:47 +02:00
CREDO23
97e250be4b
docs(env): document TIKTOK_LISTING_MAX_ATTEMPTS
2026-07-10 17:25:48 +02:00
CREDO23
8f825b7b3c
test(tiktok): cover item_list retry-on-empty loop
2026-07-10 17:25:48 +02:00
CREDO23
c8fcf87685
fix(tiktok): retry empty item_list captures on a fresh exit IP
...
The profile feed is withheld from flagged IPs and the proxy rotates per request, so re-fetching an empty capture (up to TIKTOK_LISTING_MAX_ATTEMPTS) turns a bad first draw into a hit instead of an ErrorItem.
2026-07-10 17:25:48 +02:00
CREDO23
15fb50c83e
feat(tiktok): add TIKTOK_LISTING_MAX_ATTEMPTS knob
...
Bounds the browser-listing retry-on-empty; default 3, set 1 for a static IP.
2026-07-10 17:25:48 +02:00
CREDO23
493579913e
docs(tiktok): note stage 2 profile feed is headful only when flag is set
2026-07-10 17:17:53 +02:00
CREDO23
c29b7f4b6b
docs(env): document CRAWL_HEADED_XVFB_ENABLED
2026-07-10 17:17:53 +02:00
CREDO23
d3a6e2cd45
docs(docker): note entrypoint wires xvfb at runtime
...
The xvfb runtime path is now implemented, so drop the stale 'deferred to Slice B' note.
2026-07-10 17:17:53 +02:00
CREDO23
6400dc5f04
fix(tiktok): gate headful profile feed on CRAWL_HEADED_XVFB_ENABLED
...
fetch_item_list goes headful only when the flag promises a display; otherwise it stays headless so the browser launch never fails and the empty feed degrades to an ErrorItem. Verified under xvfb-run: real videos returned on a clean proxy IP, graceful degrade on a flagged one.
2026-07-10 17:17:45 +02:00
CREDO23
a551dd7553
feat(docker): start Xvfb when CRAWL_HEADED_XVFB_ENABLED is set
...
For every browser-running role (api/worker/all, not migrate) the entrypoint brings up a virtual display and exports DISPLAY, registering Xvfb for graceful shutdown. Off keeps browsers headless.
2026-07-10 17:17:36 +02:00
CREDO23
6784503414
feat(crawl): add CRAWL_HEADED_XVFB_ENABLED flag
...
Single switch promising an Xvfb display so the stealth browser can run headful (TikTok's profile feed is empty to headless Chromium). Defaults FALSE.
2026-07-10 17:17:27 +02:00
