Commit graph

20 commits

Author SHA1 Message Date
CREDO23
f74a73efdb docs(tiktok): trim fetch_item_list retry docstring to intent 2026-07-10 17:28:47 +02:00
CREDO23
c8fcf87685 fix(tiktok): retry empty item_list captures on a fresh exit IP
The profile feed is withheld from flagged IPs and the proxy rotates per request, so re-fetching an empty capture (up to TIKTOK_LISTING_MAX_ATTEMPTS) turns a bad first draw into a hit instead of an ErrorItem.
2026-07-10 17:25:48 +02:00
CREDO23
6400dc5f04 fix(tiktok): gate headful profile feed on CRAWL_HEADED_XVFB_ENABLED
fetch_item_list goes headful only when the flag promises a display; otherwise it stays headless so the browser launch never fails and the empty feed degrades to an ErrorItem. Verified under xvfb-run: real videos returned on a clean proxy IP, graceful degrade on a flagged one.
2026-07-10 17:17:45 +02:00
CREDO23
c679f2a3ef fix(tiktok): fetch profile video lists headful + dismiss login modal
The profile feed (/api/post/item_list) returns an empty 200 to headless
sessions but serves data headful on the same proxy IP. Run fetch_item_list
headful and dismiss the login modal that blocks mid-scroll.
2026-07-10 16:45:45 +02:00
CREDO23
67b5472b9f feat(tiktok): add tiktok.trending verb for the Explore feed
The Explore feed (/api/explore/item_list) is a global trending-video feed
served to anonymous sessions, and it returns the same itemStruct shape as the
other listings — so the verb reuses parse_video, the listing flow, the
TikTokVideoItem output, and the per-video billing meter wholesale. Adds a
browser-capture marker + fetch_trending, a synthetic-target orchestrator entry,
and the tiktok.trending capability, surfaced on the chat subagent.
2026-07-09 18:52:56 +02:00
CREDO23
7723b5b8b6 feat(tiktok): add tiktok.comments verb for public comment scraping
Comments load over a signed /api/comment/list XHR that TikTok serves to
anonymous sessions once the comments panel is opened (unlike profile-video and
general-search feeds), so this is a reliable verb. Given video URLs it returns
CommentItems (text, author, likes, reply counts; replies carry repliesToId),
deduped per video, capped, and degraded to an ErrorItem for empty/withheld
videos or a bad_url ErrorItem for non-video inputs.

Generalizes the browser capture over a pluggable interaction step so the
comments flow (open panel, scroll the panel to paginate) reuses the same
warm+capture scaffolding as listing/user-search. Billed per comment on a new
TIKTOK_COMMENT meter (TIKTOK_MICROS_PER_COMMENT, matching the per-comment
market), surfaced on the chat subagent alongside tiktok.scrape/user_search.
2026-07-09 18:35:26 +02:00
CREDO23
192b6dc31a feat(tiktok): add tiktok.user_search verb for account discovery
Video/general search is login-walled for anonymous sessions, but the Users
tab (/api/search/user) returns public account records without a redirect, so
this exposes the one reliably-unblocked search path. A keyword yields
TikTokProfileItems (name, followers, bio, verification), deduped per query,
capped, and degraded to an ErrorItem when a query is empty/withheld.

Reuses the browser capture (generalized over XHR markers + extractor) and the
shared profile item shape. Billed per account on a new TIKTOK_USER meter
(TIKTOK_MICROS_PER_USER), surfaced on the chat subagent alongside tiktok.scrape.
2026-07-09 18:00:40 +02:00
CREDO23
6652efd035 feat(tiktok): return profile metadata even when video listing is gated
A profile's account data (name, followers, bio, verification) lives in the
page's rehydration blob and loads over plain HTTP without a signed request,
so emit it first and always. The video listing needs a signed item_list XHR
that TikTok withholds from anonymous sessions, so it stays best-effort and
degrades to an ErrorItem. A blocked profile now yields its metadata instead
of only an ErrorItem.
2026-07-09 17:48:19 +02:00
CREDO23
ba375ea3ee feat(tiktok): emit graceful ErrorItem for blocked/empty listings
Profile and search feeds are trust-gated: an anonymous headless session
gets an empty item_list (profile) or no results XHR (search), while
hashtag feeds load. A zero-item listing now yields one honest ErrorItem
(errorCode="no_items") instead of vanishing silently, and ErrorItems are
excluded from billing so a blocked target is surfaced but never charged.
2026-07-08 23:14:50 +02:00
CREDO23
9dd39faa50 test(tiktok): pin live fixtures, trim listing debug scaffolding 2026-07-08 17:53:29 +02:00
CREDO23
5a326f3818 fix(tiktok): warm browser listing + drop ctx-var-in-generator 2026-07-08 17:14:13 +02:00
CREDO23
ad8b73477a feat(tiktok): browser-driven signed listings 2026-07-08 16:37:36 +02:00
CREDO23
daa0856c44 feat(tiktok): blob-first orchestrator + video flow 2026-07-08 16:07:07 +02:00
CREDO23
44b3e640d3 feat(tiktok): rotate-on-block fetch session 2026-07-08 15:51:17 +02:00
CREDO23
5688ab0678 feat(tiktok): scrape input schema 2026-07-08 15:51:17 +02:00
CREDO23
ef66063bfc feat(tiktok): video normalizer and parser tests 2026-07-08 15:40:50 +02:00
CREDO23
0bbcf99852 feat(tiktok): author normalizer 2026-07-08 15:40:38 +02:00
CREDO23
8840fd2c53 feat(tiktok): rehydration blob extractor 2026-07-08 15:40:21 +02:00
CREDO23
b2d387f994 feat(tiktok): URL target resolver 2026-07-08 15:40:07 +02:00
CREDO23
ef3f9f2e25 feat(tiktok): item output schemas 2026-07-08 15:39:58 +02:00