SurfSense/surfsense_backend/app/proprietary/platforms/youtube

YouTube Scraper

A platform-native YouTube scraper that is a drop-in clone of the Apify "YouTube Scraper" and "YouTube Comments Scraper" actors — same input surface, same output item shape. It talks to YouTube's internal InnerTube API plus the public watch/channel HTML, egresses through a residential proxy, and streams Apify-shaped dicts.

No API keys, no Apify account, no headless browser on the happy path.

---

Quick start

from app.proprietary.platforms.youtube import (
    YouTubeScrapeInput, scrape_youtube,
    YouTubeCommentsInput, scrape_comments,
)

# Videos — by search query and/or direct URLs (video/channel/playlist/hashtag/search)
videos = await scrape_youtube(
    YouTubeScrapeInput(searchQueries=["surfsense"], maxResults=50)
)
videos = await scrape_youtube(
    YouTubeScrapeInput(startUrls=[{"url": "https://www.youtube.com/@SomeChannel"}],
                       maxResults=20, downloadSubtitles=True)
)

# Comments — one output item per top-level comment AND per reply
comments = await scrape_comments(
    YouTubeCommentsInput(
        startUrls=[{"url": "https://www.youtube.com/watch?v=VIDEO_ID"}],
        maxComments=200, sortCommentsBy="TOP_COMMENTS",
    )
)

Both have a streaming twin — iter_youtube() / iter_comments() — that yields items as they arrive (unbounded, continuation-paged). scrape_* is just a collector with an optional limit guard.

The HTTP surface lives in app/routes/youtube_routes.py.

---

Module map

File	Responsibility
__init__.py	Public exports (entry points + schemas).
schemas.py	Pydantic input/output models mirroring the Apify camelCase spec. extra="allow" on outputs keeps the contract open.
scraper.py	Video orchestrator. Resolves URLs → per-flow async generators (_video_flow, _search_flow, _channel_flow, _playlist_flow), runs them through the fan_out worker pool.
comments.py	Comments orchestrator. Watch page → comments-section continuation → /next paging, with concurrent per-thread reply fetching.
innertube.py	The network seam. Proxy-only fetch (fetch_html, post_innertube), reusable sticky-IP sessions, reactive IP rotation, StealthyFetcher fallback, and the InnerTube payload builder.
parsers.py	Pure, I/O-free JSON/HTML traversal + normalization (find_all/find_first/dig, parse_video_page, parse_search_response, comment/continuation token extractors, parse_count, …).
url_resolver.py	Classify a URL into video / channel / playlist / hashtag / search and extract its id.
search_filters.py	Encode Apify search filters into YouTube's sp= base64 protobuf (sort/date/type/length/feature flags), composable.
subtitles.py	Subtitle download via youtube-transcript-api, shaped to Apify subtitles[].

Everything in parsers.py is deterministic and unit-tested offline; everything that touches the network is funneled through innertube.py.

---

How it fetches (the important part)

All network I/O goes through **fetch_html** (GET watch/channel pages) and **post_innertube** (POST InnerTube browse/search/next). Design rules:

1. Proxy-only egress. Every request goes through the residential proxy (app/utils/proxy.get_proxy_url). We never connect directly — a direct hit would expose and risk-block the server IP.
2. Session reuse = sticky IP. Within one flow (a continuation chain, or the jobs a worker pulls), a single keep-alive FetcherSession is reused. This roughly halves warm latency (~2.1s → ~1.0s) because only the first request pays the TCP+TLS handshake, and it pins one sticky exit IP instead of drawing a new (often slow) residential node per request.
3. Reactive IP rotation. A sticky IP is kept until it's actually blocked. On 403/429 or a connection error, the session rotates to a fresh IP and retries, up to _MAX_ROTATIONS (3). A probe of 120 sequential requests on one IP saw zero blocks, so rotation is reactive, not proactive.
4. Browser fallback. If all proxy attempts fail on an HTML page, fetch_html falls back to StealthyFetcher (headless, solve_cloudflare=True) in a worker thread. Optional — needs patchright browsers installed. Age-gated content requires login and is not bypassable.

The active session is bound to the current async task via a ContextVar (_current_session), so parsers and orchestrators never thread a session argument through every call — each concurrent flow transparently uses its own session/IP.

InnerTube payloads

build_innertube_payload(...) builds the WEB client context payload (I/O-free, unit-testable). Some endpoints reject a keyless POST; scraper._post retries once with the public web key (INNERTUBE_PUBLIC_API_KEY) when the keyless call returns nothing. hl=<lang> on a /next call returns the creator-localized title/description (the translation flow).

---

Concurrency model

Independent jobs — each startUrl, each searchQuery, each comment video — run concurrently through **fan_out**, a warm worker pool (_FANOUT_CONCURRENCY = 16):

• Each worker opens one proxy session and reuses it across the sequential jobs it pulls, so only the first job per worker pays the handshake.
• A bad job yields nothing rather than aborting the batch (per-job try/except). One dead URL / comments-disabled video never kills the run.
• Results stream out as each job finishes; within a flow, continuation paging stays sequential.
• If the consumer stops early (collector hits its limit), workers are cancelled and awaited so every session's finally closes — no leaked keep-alive connections.

Comment reply threads for a page are fetched concurrently on the same multiplexed session (asyncio.gather), capped at the remaining budget.

---

Data flow

• Video by URL → fetch watch HTML → parse_video_page (reads ytInitialData + ytInitialPlayerResponse) → optional subtitles + translation.
• Search → InnerTube /search (+ sp= filter protobuf) → paginate via continuation tokens up to maxResults.
• Channel → fetch the videos-tab seed once (reused for channel-wide metadata
  • the About panel via /browse), then page videos / shorts / streams tabs, each capped independently (maxResults / maxResultsShorts / maxResultStreams). sortVideosBy uses the sort chips; oldestPostDate cuts off newest-first.
• Playlist → /browse VL<id>, paged via the continuation token → resolve each video via the video flow.
• Hashtag → the dedicated hashtag page (/hashtag/<tag>), whose feed is videoRenderer lockups (parsed like search) — not a #tag search.
• Comments → watch HTML seeds the comments-section token → /next returns comment entities + per-thread reply tokens + the page token. maxComments counts every emitted item (comments + replies).

commentsCount

For the comments scraper, the authoritative total is read from the comments-section header (commentsHeaderRenderer.countText), not the watch-page HTML where it's lazy-loaded/absent. Known gap: the video scraper's VideoItem.commentsCount still comes from search/watch HTML and is often null — it would need an extra /next call to backfill (intentionally not done to keep the video path cheap).

---

API spec

Mirrors the Apify "YouTube Scraper" and "YouTube Comments Scraper" actors (camelCase, extra="allow"). Inputs use Pydantic defaults; every field is additive — unknown inputs are accepted, unsourced outputs come back as None/[] — so parity grows without breaking consumers. schemas.py is the source of truth.

Video scraper — input (YouTubeScrapeInput)

Field	Type / values	Default	Notes
searchQueries	string[]	[]	Discovery by query. Ignored when startUrls is set.
startUrls	[{ "url": string }]	[]	Direct URLs: video, channel, playlist, hashtag, search. Overrides searchQueries.
maxResults	int ≥ 0	0	Cap of regular videos per query and per channel. 0 = fetch none.
maxResultsShorts	int ≥ 0	0	Cap of Shorts per channel.
maxResultStreams	int ≥ 0	0	Cap of live/streams per channel.
downloadSubtitles	bool	false	Populate subtitles[].
subtitlesLanguage	string	"en"	Also drives the translation flow when non-en (see translatedTitle).
subtitlesFormat	srt	vtt	xml
preferAutoGeneratedSubtitles	bool	false
saveSubsToKVS	bool	false	Accepted for parity; no-op (Apify key-value-store concept).
sortingOrder	relevance	rating	date
dateFilter	hour	today	week
videoType	video	movie	null
lengthFilter	under4	between420	plus20
isHD hasSubtitles hasCC is3D isLive isBought is4K is360 hasLocation isHDR isVR180	bool	null	Search feature filters (encoded into sp=).
oldestPostDate	string (date)	null	Channel cutoff; day-accurate (relative times).
sortVideosBy	NEWEST	POPULAR	OLDEST

Video scraper — output (VideoItem)

Field	Type	Populated?
title id url viewCount date duration	str/int	yes
type	video	shorts
thumbnailUrl	str	yes
input fromYTUrl order	str/int	yes (provenance: source query/URL, origin URL, index)
text	str	yes (description)
descriptionLinks	[{ url, text }]	yes
hashtags	string[]	yes
likes commentsCount commentsTurnedOff	int/bool	partial (often null on the video path — see commentsCount note)
location	str	when present
collaborators	[{ name, username, url }]	when present
translatedTitle translatedText	str	when subtitlesLanguage != "en"
subtitles	[{ srtUrl, type, language, srt }]	when downloadSubtitles
isMembersOnly isPaidContent	bool	yes (default false)
isMonetized isAgeRestricted	bool	best-effort (null when unknown)
channelName channelUrl channelUsername channelId	str	yes
numberOfSubscribers channelTotalVideos channelTotalViews	int	channel/deep fields
channelDescription channelLocation channelJoinedDate	str	channel About panel
isChannelVerified channelBannerUrl channelAvatarUrl	bool/str	channel fields

Comments scraper — input (YouTubeCommentsInput)

Field	Type / values	Default	Notes
startUrls	[{ "url": string }]	[]	Video URLs only (non-video URLs skipped).
maxComments	int ≥ 1	1	Counts every emitted item (top-level comments and replies).
sortCommentsBy	TOP_COMMENTS	NEWEST_FIRST	"NEWEST_FIRST"
oldestCommentDate	string (date)	null	Forces newest-first and stops at the cutoff.

Comments scraper — output (CommentItem)

Field	Type	Notes
cid	str	Comment id.
comment	str	Text.
author	str
type	comment	reply
replyToCid	str	Parent cid (replies only).
replyCount	int	Replies under a top-level comment.
voteCount	int	Likes.
authorIsChannelOwner hasCreatorHeart	bool
publishedTimeText	str	Relative time ("2 days ago").
videoId pageUrl title	str	Source video.
commentsCount	int	Authoritative total from the comments header.

---

Configuration

• Proxy — required for real runs; configured via app/utils/proxy.py (residential rotating gateway env vars). With no proxy configured the fetchers fall back to one-shot direct AsyncFetcher calls (fine for local tests, not for production).
• Concurrency — scraper._FANOUT_CONCURRENCY (16). The gateway handled 64 parallel flows with zero failures in a ramp probe, so this leaves headroom.
• Rotation — innertube._BLOCK_STATUSES (403, 429) and _MAX_ROTATIONS (3).

---

Testing

• Offline unit tests (no network) — run these on every change:

  cd surfsense_backend
  .venv/Scripts/python.exe -m pytest tests/unit/scrapers/youtube/
  

  • test_parsers.py — parser/normalization + filter-protobuf + URL-resolver cases against hand-built and (if present) captured real fixtures.
  • test_fetch_resilience.py — deterministic rotate-on-block (429/error → rotate → 200, exhaustion, no-rotate on 404, stealthy fallback) and the fan_out no-session-leak-on-early-stop guarantee, all with stubbed sessions.
• Live functional harness — scripts/e2e_youtube_scraper.py (needs live network + optional proxy creds). Exercises video/search/channel/comments/ location/collaborators/translation end to end, and regenerates the offline fixtures into tests/unit/scrapers/youtube/fixtures/:

  .venv/Scripts/python.exe scripts/e2e_youtube_scraper.py
  

---

Extending it

• Add an output field → populate it in the relevant parsers.py function and add it to schemas.py. Because outputs are extra="allow", forgetting the schema line won't drop the value, but declaring it documents the contract.
• Add a URL kind → extend url_resolver.resolve_url + add a _*_flow in scraper.py and a branch in _dispatch.
• Add a search filter → add the field to YouTubeScrapeInput and encode it in search_filters.build_search_params (verify byte-for-byte against a real YouTube sp= token in the unit test).

Known ceilings (grep ponytail: in the source for the live list)

• Hashtag scraping returns a single feed page (~20-35 videos); YouTube exposes no continuation for the hashtag feed through this path. Upgrade path for more depth: fall back to the #tag search route.
• Playlist video ids are paged sequentially (each continuation depends on the last), then the per-video watch-page fetches run concurrently via fan_out (~150 videos ≈ 70s). Because resolution is fanned out, items stream back in completion order, not playlist order — sort by the order field to restore it.
• oldestPostDate / oldestCommentDate cutoffs are day-accurate at best (channel/list pages only expose coarse relative times like "2 years ago").
• Keyless-vs-keyed InnerTube retry does one extra request on the keyed path instead of remembering which worked.
• Video-path commentsCount (see above).