apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-16 23:01:06 +02:00
Code Issues Projects Releases Packages Wiki Activity

fix(tiktok): gate headful profile feed on CRAWL_HEADED_XVFB_ENABLED

Browse source 
fetch_item_list goes headful only when the flag promises a display; otherwise it stays headless so the browser launch never fails and the empty feed degrades to an ErrorItem. Verified under xvfb-run: real videos returned on a clean proxy IP, graceful degrade on a flagged one.
This commit is contained in:
CREDO23 2026-07-10 17:17:45 +02:00
parent a551dd7553
commit 6400dc5f04
1 changed files with 7 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

12
surfsense_backend/app/proprietary/platforms/tiktok/session/listing.py
View file

@ -11,9 +11,9 @@ The pure response-shape parsing lives in :func:`items_from_response`; this modul
is the untested browser-I/O glue (covered by the e2e smoke, not unit tests).
Needs a residential proxy; datacenter IPs get empty bodies and 429s. The profile
feed (``/api/post/item_list``) returns an empty 200 to headless sessions, so
:func:`fetch_item_list` runs headful. ponytail: headful needs a display — run it
under Xvfb on a headless server.
feed returns an empty 200 to headless sessions, so :func:`fetch_item_list` goes
headful only when ``CRAWL_HEADED_XVFB_ENABLED`` promises an Xvfb display — else it
stays headless and degrades to an ``ErrorItem`` instead of crashing on launch.
"""
from __future__ import annotations
@ -25,6 +25,7 @@ from typing import Any
from scrapling.fetchers import StealthyFetcher
from app.config import config
from app.proprietary.web_crawler.stealth import (
build_stealthy_kwargs,
get_stealth_config,
@ -250,7 +251,8 @@ def _fetch_sync(
async def fetch_item_list(page_url: str, target_count: int) -> list[dict[str, Any]]:
"""Return up to ``target_count`` itemStructs from a listing page's XHRs.
Runs headful: the profile feed returns an empty 200 to headless sessions.
Headful when ``CRAWL_HEADED_XVFB_ENABLED`` promises a display (the profile feed
is empty to headless sessions); headless otherwise so launch never fails.
"""
return await asyncio.to_thread(
_fetch_sync,
@ -259,7 +261,7 @@ async def fetch_item_list(page_url: str, target_count: int) -> list[dict[str, An
_ITEM_LIST_MARKERS,
items_from_response,
_scroll_page,
headless=False,
headless=not config.CRAWL_HEADED_XVFB_ENABLED,
)