apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-16 23:01:06 +02:00
Code Issues Projects Releases Packages Wiki Activity

docs(tiktok): note stage 2 profile feed is headful only when flag is set

Browse source
This commit is contained in:
CREDO23 2026-07-10 17:17:53 +02:00
parent c29b7f4b6b
commit 493579913e
1 changed files with 4 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

8
surfsense_backend/scripts/e2e_tiktok_scrape.py
View file

@ -7,8 +7,8 @@ Run from the backend directory:
What it exercises (everything REAL — live network, live proxy, live browser):
Stage 1 — proxy egress proof (informational).
Stage 2 — profile via the full pipeline: asserts real videos come back
(fetch_item_list runs headful), degrading to one ErrorItem if not.
Stage 2 — profile via the full pipeline: real videos when headful
(CRAWL_HEADED_XVFB_ENABLED), else one ErrorItem — never silent empty.
Stage 3 — blob video path over HTTP (URL taken from a captured hashtag struct).
Stage 4 — hashtag listing via the stealth browser (captures item_list XHRs).
Stage 5 — full scrape_tiktok() pipeline on a hashtag.
@ -106,8 +106,8 @@ async def stage_profile_listing() -> tuple[bool, list[dict[str, Any]]]:
_hr(f"STAGE 2 — profile listing graceful-degrade: @{_PROFILE}")
from app.proprietary.platforms.tiktok import TikTokScrapeInput, scrape_tiktok
# fetch_item_list runs headful, so we expect real videos; still accept an
# ErrorItem (never a silent empty) to keep the graceful-degradation contract.
# Headful (flag on) yields real videos; still accept an ErrorItem (never a
# silent empty) so the graceful-degradation contract holds either way.
items = await scrape_tiktok(
TikTokScrapeInput(profiles=[_PROFILE], resultsPerPage=_COUNT), limit=_COUNT
)