apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-08 22:22:17 +02:00
Code Issues Projects Releases Packages Wiki Activity

refactor(native-connector): relocate youtube scraper under app/proprietary + parallelize playlists

Browse source 
Move app/scrapers -> app/proprietary/scrapers/youtube to sit alongside the existing proprietary web_crawler/platforms namespace, updating all external imports (routes, tests, e2e script, README). Internal imports were relative so are unchanged.

Also parallelize playlist per-video resolution: page video ids sequentially, then resolve the heavy watch-page fetches concurrently via fan_out (~150 videos ~70s, down from a few minutes). Items stream in completion order; sort by the order field for playlist order.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-07-02 02:34:04 -07:00
parent 99378d7b63
commit 0445c646c6
16 changed files with 412 additions and 320 deletions
Download patch file Download diff file Expand all files Collapse all files

8
surfsense_backend/scripts/e2e_youtube_scraper.py
View file

@ -1,4 +1,4 @@
"""Manual functional e2e for the YouTube scraper (app/scrapers/youtube).
"""Manual functional e2e for the YouTube scraper (app/proprietary/scrapers/youtube).
Run from the backend directory:
cd surfsense_backend
@ -31,20 +31,20 @@ for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"):
load_dotenv(_candidate)
break
from app.scrapers.youtube import ( # noqa: E402
from app.proprietary.scrapers.youtube import ( # noqa: E402
YouTubeCommentsInput,
YouTubeScrapeInput,
scrape_comments,
scrape_youtube,
)
from app.scrapers.youtube.innertube import ( # noqa: E402
from app.proprietary.scrapers.youtube.innertube import ( # noqa: E402
INNERTUBE_PUBLIC_API_KEY,
INNERTUBE_SEARCH_URL,
build_innertube_payload,
fetch_html,
post_innertube,
)
from app.scrapers.youtube.parsers import ( # noqa: E402
from app.proprietary.scrapers.youtube.parsers import ( # noqa: E402
extract_yt_initial_data,
extract_yt_initial_player_response,
)