/")` → `parse_post`, which
+ reads the embedded mobile-v1 `PolarisMedia` JSON (full fidelity) and falls
+ back to Open Graph meta only if that blob is absent. Numeric-ID post URLs are
+ skipped (the page keys on the shortCode).
+3. `fetch_json` / `fetch_html` warm the session on first use, rotate the IP +
+ re-warm on 401/403, back off on 429, return `None` on 404, and raise
+ `InstagramAccessBlockedError` on a `/accounts/login/` redirect.
+4. Parsers map raw web JSON/HTML to flat dicts; the orchestrator stamps
+ `scrapedAt` and applies `resultsLimit` / `onlyPostsNewerThan` as request-time
+ policy.
+
+## Profile discovery (Google-backed)
+
+Instagram's native keyword search is login-walled, so `_discover` resolves a
+query that is a valid handle directly (`"messi"` → `instagram.com/messi/`) and
+routes any other query (e.g. `"national geographic"`) through
+`_discover_via_google`, which calls the `google_search` platform with
+`site:instagram.com`, classifies each organic URL with `resolve_url`, keeps the
+**profile** hits (discovery is profile-only), de-dupes, and caps at `searchLimit`.
+
+Caveats:
+
+- **Coupling**: Instagram depends on the `google_search` platform. The dependency
+ is one-directional and lives behind `_discover_via_google` so it stays testable.
+- **Quality**: results reflect Google's index/ranking of `instagram.com`, not
+ IG's own relevance. This is discovery, not search parity.
+
+## Observed limits & calibration caveats
+
+- Anonymous web JSON/HTML is rate-limited per IP; the sticky-session pool keeps
+ each IP's request rate modest but a hot pool will still hit login walls — that's
+ the `InstagramAccessBlockedError` path, not a bug.
+- `likesCount` is frequently withheld on anonymous responses (surfaces as `-1` or
+ absent upstream); treat it as best-effort.
+- **Single-post extraction** reads the mobile-v1 `PolarisMedia` object embedded in
+ the public `/p/` document (og-meta is a lossy fallback). If Instagram strips both
+ for a given post (private, taken down, or a login interstitial), `parse_post`
+ returns `None` — an honest empty, never a fabricated item. ponytail: the
+ embedded-blob shape can drift; a live probe that dumps the raw HTML pins it (see
+ Testing) and any change is contained to `_find_media` / `parse_post`.
+- The `$3.50 / 1k items` default meter assumes the proxy-bytes-per-item measured
+ on the reference targets; re-measure with the scale harness before high-volume use.
+
+## Testing
+
+- Offline unit tests: `tests/unit/platforms/instagram/` — `test_skeleton.py`
+ (schema + URL resolver), `test_parsers.py` (mapping incl. `parse_post`
+ relay-JSON/og shapes; fixture-pinned tests skip when the fixture is absent),
+ `test_discovery.py` (Google-backed profile discovery with a fake `scrape_serps`),
+ `test_fetch_resilience.py` (warm / rotate / backoff loop + fan-out with fake
+ sessions, no network), `test_budget.py` (fair-share caps + de-dup).
+- Stress / accuracy harness (live, needs network + residential proxy):
+ `scripts/stress/stress_instagram_scraper.py` — `--mode live-discovery` (profile
+ discovery accuracy), `--mode probe-post` (dumps a real anonymous `/p/` payload
+ to `fixtures/post.json` and shows what `parse_post` extracted), `--mode
+ probe-mentions` (settles that the tagged/`mentions` feed is login-walled), and
+ `--mode accuracy` (field coverage across the profile + single-post flows).
+
+```bash
+cd surfsense_backend
+uv run pytest tests/unit/platforms/instagram/
+# Live single-post probe: confirms /p/ is anonymously extractable + pins the shape
+uv run python scripts/stress/stress_instagram_scraper.py --mode probe-post \
+ --post https://www.instagram.com/p//
+```
+
+## TODO / out of scope (v1)
+
+- Deep feed pagination past the first web page of profile media (GraphQL cursor
+ doc-id).
+- Sticky-IP provider parity (same `__sid` caveat as the Reddit sibling).
diff --git a/surfsense_backend/app/proprietary/platforms/instagram/__init__.py b/surfsense_backend/app/proprietary/platforms/instagram/__init__.py
new file mode 100644
index 000000000..2302ca9fc
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/instagram/__init__.py
@@ -0,0 +1,18 @@
+"""Platform-native Instagram scraper (anonymous, no browser)."""
+
+from .fetch import InstagramAccessBlockedError
+from .schemas import (
+ InstagramMediaItem,
+ InstagramProfile,
+ InstagramScrapeInput,
+)
+from .scraper import iter_instagram, scrape_instagram
+
+__all__ = [
+ "InstagramAccessBlockedError",
+ "InstagramMediaItem",
+ "InstagramProfile",
+ "InstagramScrapeInput",
+ "iter_instagram",
+ "scrape_instagram",
+]
diff --git a/surfsense_backend/app/proprietary/platforms/instagram/fetch.py b/surfsense_backend/app/proprietary/platforms/instagram/fetch.py
new file mode 100644
index 000000000..ab94f81a0
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/instagram/fetch.py
@@ -0,0 +1,403 @@
+"""Proxy-aware fetch seam for the Instagram scraper (no browser).
+
+All network I/O flows through :func:`fetch_json` and always egresses through the
+residential proxy (a direct hit would expose and risk-block the server IP).
+
+Instagram's public web app exposes anonymous JSON endpoints that a logged-out
+browser calls, guarded by the ``X-IG-App-ID`` web app id and a warmed
+``csrftoken``/``mid`` cookie pair:
+
+ warm one anonymous session (plain GET to ``www.instagram.com/`` mints
+ ``csrftoken``/``mid``), then GET the ``api/v1/*/web_info`` /
+ ``web_profile_info`` endpoints through that same Chrome-impersonated,
+ sticky-IP session with the ``X-IG-App-ID`` header.
+
+This is a direct port of ``../reddit/fetch.py``'s rotate-on-block sticky-session
+pattern (``_RotatingSession`` + ``_current_session`` ContextVar +
+``open_proxy_holder``/``bind_proxy_holder``/``proxy_session``), with an
+Instagram-specific :func:`warm_session` and header set.
+
+Honest ceiling: anonymous Instagram access is the most hostile of our platforms.
+Login walls appear as 401/403 and rotate the exit IP; 429 backs off on the same
+IP. Observed per-IP/session limits are documented in ``README.md``; the safe
+``_FANOUT_CONCURRENCY`` is deliberately low. ponytail: the pacing/rotation
+constants are calibrated to residential exits and may need retuning per pool —
+watch for 401/403/429 log spam and adjust.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import random
+import time
+from collections.abc import Callable
+from contextlib import asynccontextmanager, suppress
+from contextvars import ContextVar
+from datetime import UTC, datetime
+from typing import Any
+from urllib.parse import urlencode
+
+from scrapling.fetchers import AsyncFetcher, FetcherSession
+
+from app.utils.proxy import get_proxy_url
+
+logger = logging.getLogger(__name__)
+
+
+class InstagramAccessBlockedError(RuntimeError):
+ """Raised when every rotated IP is refused anonymous access.
+
+ This is the Instagram login-wall branch: after warming and rotating, the
+ exit IPs still 401/403. We are anonymous-only and cannot log in, so instead
+ of silently returning nothing we surface it as a clear error (mirrors
+ reddit's ``RedditAccessBlockedError`` / google_maps' ``SignInRequiredError``).
+ The executor turns it into a 403 for REST callers.
+ """
+
+
+# Per-flow proxy session, set by ``bind_proxy_holder`` around one continuation
+# chain. Reusing one keep-alive connection pins a single sticky exit IP so the
+# warmed ``csrftoken``/``mid`` cookies (bound to that IP) stay valid across the
+# warm-up + every subsequent web-endpoint fetch. A ContextVar keeps each
+# concurrent fan-out flow on its own session/IP without threading a param
+# through every call.
+_current_session: ContextVar[_RotatingSession | None] = ContextVar(
+ "instagram_proxy_session", default=None
+)
+
+# 401/403 => this IP hit the login wall; rotate to a fresh one and re-warm.
+# 429 => rate limited; back off on the SAME IP (rotating wouldn't help and burns
+# the pool).
+_ROTATE_STATUSES = frozenset({401, 403})
+_BACKOFF_STATUS = 429
+_MAX_ROTATIONS = 3
+_MAX_BACKOFFS = 4
+_BACKOFF_BASE_S = 5.0
+
+# Instagram 429s hard on bursts. Pace each sticky session so a fast IP can't
+# burst past the per-IP threshold. ponytail: 1.5s is tuned to residential exits;
+# a pool with a stricter per-IP cap may need it raised — watch for 429 log spam.
+_MIN_INTERVAL_S = 1.5
+_PACE_JITTER_S = 0.5
+
+# A healthy fetch lands in ~1-2s; cap at 15s so a dead sticky IP costs one
+# bounded wait, then the timeout falls into the generic exception branch of
+# fetch_json and rotates to a fresh IP — same treatment as a 403.
+_REQUEST_TIMEOUT_S = 15.0
+
+# The anonymous web app id every logged-out instagram.com XHR carries. Without
+# it the api/v1/*/web_info and web_profile_info endpoints 403 outright.
+_IG_APP_ID = "936619743392459"
+_HEADERS = {
+ "Accept-Language": "en-US,en;q=0.9",
+ "X-IG-App-ID": _IG_APP_ID,
+ "X-Requested-With": "XMLHttpRequest",
+ "Referer": "https://www.instagram.com/",
+}
+
+# A plain GET to the home page mints the anonymous csrftoken/mid cookie pair.
+_WARM_URL = "https://www.instagram.com/"
+_BASE = "https://www.instagram.com"
+_CSRF_COOKIE = "csrftoken"
+
+# Soft login wall: instead of a 401/403, IG answers an api/v1/* request with a
+# 302 to /accounts/login/ that the impersonated client follows to a 200 login
+# page. The status is 200 but the body is login HTML, so this evades the
+# status-code rotate check — detect it via the response's final URL and treat
+# it exactly like a 403.
+_LOGIN_PATH = "/accounts/login"
+
+
+def now_iso() -> str:
+ """UTC timestamp in the millisecond ISO shape used by scraper output."""
+ return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
+
+
+def _response_cookie_names(page: Any) -> set[str]:
+ """Cookie names set by a response (best-effort across scrapling shapes)."""
+ cookies = getattr(page, "cookies", None)
+ if isinstance(cookies, dict):
+ return set(cookies.keys())
+ return set()
+
+
+def _parse_json(page: Any) -> Any | None:
+ """Parse a scrapling response body into JSON, or ``None``.
+
+ Prefers ``page.json()``; falls back to ``json.loads`` on the raw body when
+ the impersonated response hands back text.
+ """
+ fn = getattr(page, "json", None)
+ if callable(fn):
+ with suppress(Exception):
+ return fn()
+ for attr in ("body", "text"):
+ val = getattr(page, attr, None)
+ if isinstance(val, bytes):
+ val = val.decode("utf-8", "replace")
+ if isinstance(val, str) and val.strip():
+ with suppress(Exception):
+ return json.loads(val)
+ return None
+ return None
+
+
+def _page_text(page: Any) -> str | None:
+ """Best-effort HTML/text body of a scrapling response, or ``None``."""
+ for attr in ("text", "body", "content"):
+ val = getattr(page, attr, None)
+ if callable(val):
+ with suppress(Exception):
+ val = val()
+ if isinstance(val, bytes):
+ val = val.decode("utf-8", "replace")
+ if isinstance(val, str) and val.strip():
+ return val
+ return None
+
+
+def _is_login_redirect(page: Any) -> bool:
+ """True if IG redirected this request to the anonymous login wall.
+
+ A soft block: the final URL lands on ``/accounts/login/`` (served 200), so
+ the status check never fires. Best-effort — returns ``False`` when the
+ response exposes no URL.
+ """
+ final = getattr(page, "url", None)
+ return isinstance(final, str) and _LOGIN_PATH in final
+
+
+def _build_url(path: str, params: dict[str, Any] | None) -> str:
+ """Absolute URL for an instagram.com path (accepts already-absolute URLs)."""
+ base = path if path.startswith("http") else f"{_BASE}/{path.strip('/')}/"
+ if not params:
+ return base
+ qs = urlencode({k: v for k, v in params.items() if v is not None})
+ sep = "&" if "?" in base else "?"
+ return f"{base}{sep}{qs}" if qs else base
+
+
+class _RotatingSession:
+ """Owns one live ``FetcherSession`` (sticky IP) and can swap it for a fresh one.
+
+ ``rotate()`` closes the current keep-alive connection and opens a new one, so
+ the rotating gateway hands out a different residential exit IP. Because the
+ warmed cookies bind to the exit IP, ``rotate()`` also drops the warmed state
+ — the next fetch re-warms on the new IP. Used sequentially within a single
+ flow (never shared across concurrent tasks), so no locking is needed.
+ ``session`` is ``None`` only when no proxy is configured.
+ """
+
+ def __init__(self) -> None:
+ self._cm: Any | None = None
+ self.session: Any | None = None
+ self.rotations = 0
+ self.warmed = False
+ self._last_at = 0.0
+
+ async def _open(self) -> None:
+ proxy = get_proxy_url()
+ self.warmed = False
+ if proxy is None:
+ self._cm = self.session = None
+ return
+ self._cm = FetcherSession(
+ proxy=proxy,
+ stealthy_headers=True,
+ impersonate="chrome",
+ timeout=_REQUEST_TIMEOUT_S,
+ )
+ self.session = await self._cm.__aenter__()
+
+ async def close(self) -> None:
+ if self._cm is not None:
+ with suppress(Exception): # best-effort teardown
+ await self._cm.__aexit__(None, None, None)
+ self._cm = self.session = None
+
+ async def rotate(self) -> Any | None:
+ """Drop the current IP and connect through a fresh one. Returns new session."""
+ await self.close()
+ self.rotations += 1
+ await self._open()
+ logger.info("[instagram] rotated proxy session (rotation #%d)", self.rotations)
+ return self.session
+
+ async def pace(self) -> None:
+ """Sleep to hold this sticky IP under Instagram's per-IP rate threshold."""
+ wait = _MIN_INTERVAL_S - (time.monotonic() - self._last_at)
+ if wait > 0:
+ await asyncio.sleep(wait + random.uniform(0, _PACE_JITTER_S))
+ self._last_at = time.monotonic()
+
+
+async def open_proxy_holder() -> _RotatingSession:
+ """Open a warm rotate-on-block session holder (caller owns ``close()``)."""
+ holder = _RotatingSession()
+ await holder._open()
+ return holder
+
+
+@asynccontextmanager
+async def bind_proxy_holder(holder: _RotatingSession):
+ """Route this task's fetches through ``holder`` for the enclosed block.
+
+ Does NOT close the holder — enables pooling warm sessions across sequential
+ jobs so each job skips the proxy handshake AND the cookie warm-up.
+ """
+ token = _current_session.set(holder)
+ try:
+ yield holder
+ finally:
+ _current_session.reset(token)
+
+
+@asynccontextmanager
+async def proxy_session():
+ """Open one reused, rotate-on-block proxy session for a continuation chain."""
+ holder = await open_proxy_holder()
+ try:
+ async with bind_proxy_holder(holder):
+ yield holder
+ finally:
+ await holder.close()
+
+
+async def warm_session(session: Any) -> bool:
+ """Mint anonymous ``csrftoken``/``mid`` cookies on a freshly opened session.
+
+ Returns ``True`` when a ``csrftoken`` was issued (the session can now reach
+ the web endpoints), else ``False`` (caller rotates the IP and retries).
+
+ Takes an already-open ``session`` (never constructs one) so tests can drive
+ warm/rotate deterministically with a fake session, exactly like the reddit
+ sibling's fetch-resilience tests.
+ """
+ seen: set[str] = set()
+ with suppress(Exception):
+ page = await session.get(_WARM_URL, headers=_HEADERS)
+ seen |= _response_cookie_names(page)
+ return _CSRF_COOKIE in seen
+
+
+async def _get_page(session: Any, url: str) -> Any:
+ """GET through the warmed sticky session, or a one-shot proxied fetch."""
+ if session is not None:
+ return await session.get(url, headers=_HEADERS)
+ return await AsyncFetcher.get(
+ url,
+ headers=_HEADERS,
+ proxy=get_proxy_url(),
+ stealthy_headers=True,
+ timeout=_REQUEST_TIMEOUT_S,
+ )
+
+
+async def fetch_json(path: str, params: dict[str, Any] | None = None) -> Any | None:
+ """GET an Instagram web endpoint through a warmed session; parse JSON.
+
+ Returns parsed JSON (dict or list), or ``None`` on 404 / non-block failure.
+ See :func:`_fetch` for the warm/rotate/backoff resilience contract.
+ """
+ return await _fetch(path, params, _parse_json)
+
+
+async def fetch_html(path: str, params: dict[str, Any] | None = None) -> str | None:
+ """GET an Instagram web page through a warmed session; return its HTML text.
+
+ Same warm/rotate/backoff resilience as :func:`fetch_json` (a login-wall
+ redirect still raises :class:`InstagramAccessBlockedError`), but hands back
+ the raw HTML body for the pages that embed their data in the document
+ (``/p//`` embedded PolarisMedia JSON / og-meta) instead of a JSON
+ XHR endpoint.
+ """
+ return await _fetch(path, params, _page_text)
+
+
+async def _fetch(
+ path: str,
+ params: dict[str, Any] | None,
+ extract: Callable[[Any], Any | None],
+) -> Any | None:
+ """GET an Instagram web endpoint through a warmed HTTP session.
+
+ Applies ``extract`` to the 200 response (JSON parse or HTML text); returns
+ ``None`` on 404 / non-block failure. Warms cookies once per session; rotates
+ the residential IP and re-warms on 401/403; backs off on 429. Raises
+ :class:`InstagramAccessBlockedError` only when every rotated IP refuses
+ anonymous access (the login-wall branch, which we cannot satisfy).
+ """
+ holder = _current_session.get()
+ if holder is None:
+ # No bound session (e.g. a direct call outside fan_out): open a
+ # short-lived warmed session for this one fetch, then tear it down.
+ async with proxy_session():
+ return await _fetch(path, params, extract)
+
+ url = _build_url(path, params)
+ attempt = 0
+ backoffs = 0
+ while True:
+ session = holder.session
+ try:
+ if session is not None and not holder.warmed:
+ warmed_ok = await warm_session(session)
+ holder.warmed = True # attempted; don't re-warm this IP
+ if not warmed_ok:
+ if attempt < _MAX_ROTATIONS:
+ attempt += 1
+ await holder.rotate()
+ continue
+ raise InstagramAccessBlockedError(
+ f"could not warm session after {attempt} IP rotations for {path}"
+ )
+
+ await holder.pace()
+ page = await _get_page(session, url)
+ status = page.status
+
+ # Endpoint-level login wall (302 -> /accounts/login/, served as 200
+ # login HTML): fail fast, do NOT rotate. Unlike the per-IP 401/403
+ # below — which recovers on a fresh exit IP, so it still rotates —
+ # every rotated IP hits this same wall (observed live), so rotating
+ # only burns the pool and re-warms for an unwinnable block. Raising
+ # (vs returning None) keeps a blocked target distinguishable from an
+ # empty one; fan_out swallows it per-target for partial results.
+ if _is_login_redirect(page):
+ raise InstagramAccessBlockedError(
+ f"Instagram login wall on {path} (endpoint requires auth)"
+ )
+ if status == 200:
+ return extract(page)
+ if status == 404:
+ return None
+ if status == _BACKOFF_STATUS and backoffs < _MAX_BACKOFFS:
+ backoffs += 1
+ delay = _BACKOFF_BASE_S * (2 ** (backoffs - 1))
+ logger.warning("[instagram] 429 on %s; backing off %.1fs", path, delay)
+ await asyncio.sleep(delay + random.uniform(0, 1))
+ continue
+ if status in _ROTATE_STATUSES:
+ # Bare 401/403: a per-IP block that a fresh exit IP recovers, so
+ # rotate and re-warm. (The endpoint-level auth wall is caught by
+ # the login-redirect branch above and fails fast without rotating.)
+ if attempt < _MAX_ROTATIONS:
+ attempt += 1
+ await holder.rotate()
+ continue
+ raise InstagramAccessBlockedError(
+ f"Instagram refused {path} on {attempt} rotated IPs ({status})"
+ )
+ logger.warning("[instagram] GET %s returned %s", path, status)
+ return None
+ except InstagramAccessBlockedError:
+ raise
+ except Exception as e:
+ logger.warning("[instagram] GET %s failed: %s", path, e)
+ if attempt < _MAX_ROTATIONS:
+ attempt += 1
+ await holder.rotate()
+ continue
+ return None
diff --git a/surfsense_backend/app/proprietary/platforms/instagram/parsers.py b/surfsense_backend/app/proprietary/platforms/instagram/parsers.py
new file mode 100644
index 000000000..c76d32e5b
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/instagram/parsers.py
@@ -0,0 +1,567 @@
+"""Pure JSON -> item mapping for the Instagram scraper.
+
+Framework-agnostic and I/O-free so it can be unit-tested against captured
+fixtures. Every function takes a raw Instagram web-JSON node and returns a plain
+dict shaped like the public actor spec — no network, no proxy, no ``scrapedAt``
+stamp (the orchestrator adds provenance so these stay deterministic under
+fixture tests).
+
+Instagram's web JSON nests media under GraphQL-style ``edge_*`` containers
+(``edge_media_to_caption``, ``edge_liked_by``, ...) with ``taken_at_timestamp``
+epoch seconds. These parsers flatten that into the actor's camelCase item shape.
+Fields the anonymous endpoints don't expose are left unset (``None``/``[]``) so
+parity is additive.
+"""
+
+from __future__ import annotations
+
+import html
+import json
+import re
+from datetime import UTC, datetime
+from typing import Any
+
+_BASE = "https://www.instagram.com"
+_HASHTAG_RE = re.compile(r"#(\w+)")
+# Instagram handles are letters/digits/period/underscore but never start or end
+# with a period, so anchor both ends to alphanumerics/underscore — otherwise
+# trailing sentence punctuation ("@hulu.") leaks into the handle.
+_MENTION_RE = re.compile(r"@([A-Za-z0-9_](?:[A-Za-z0-9_.]*[A-Za-z0-9_])?)")
+_TYPE_MAP = {
+ "GraphImage": "Image",
+ "GraphVideo": "Video",
+ "GraphSidecar": "Sidecar",
+ "XDTGraphImage": "Image",
+ "XDTGraphVideo": "Video",
+ "XDTGraphSidecar": "Sidecar",
+}
+# Mobile v1 ``media_type``: 1 = image, 2 = video, 8 = carousel/sidecar. Used by
+# the single-post relay parser (the embedded PolarisMedia blob uses this int, not
+# the GraphQL ``__typename`` the profile feed uses).
+_MEDIA_TYPE = {1: "Image", 2: "Video", 8: "Sidecar"}
+
+
+def _int(value: Any) -> int | None:
+ """Coerce to int, or ``None`` (never coerces bools)."""
+ if isinstance(value, bool):
+ return None
+ if isinstance(value, int):
+ return value
+ if isinstance(value, float):
+ return int(value)
+ return None
+
+
+def _utc_from_sec(value: Any) -> str | None:
+ """Epoch seconds -> millisecond ISO string, or ``None``."""
+ if not isinstance(value, int | float) or isinstance(value, bool):
+ return None
+ dt = datetime.fromtimestamp(float(value), tz=UTC)
+ return dt.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
+
+
+def _edge_count(node: dict[str, Any], key: str) -> int | None:
+ """``node[key].count`` for a GraphQL ``edge_*`` container."""
+ container = node.get(key)
+ if isinstance(container, dict):
+ return _int(container.get("count"))
+ return None
+
+
+def _edges(container: Any) -> list[dict[str, Any]]:
+ """``container.edges[].node`` list for a GraphQL ``edge_*`` container."""
+ if not isinstance(container, dict):
+ return []
+ out = []
+ for edge in container.get("edges") or []:
+ node = edge.get("node") if isinstance(edge, dict) else None
+ if not isinstance(node, dict):
+ continue
+ out.append(node)
+ return out
+
+
+def _caption_text(node: dict[str, Any]) -> str | None:
+ """First caption edge's text (web feed) or a flat ``caption`` fallback."""
+ edges = _edges(node.get("edge_media_to_caption"))
+ if edges:
+ text = edges[0].get("text")
+ if isinstance(text, str):
+ return text
+ cap = node.get("caption")
+ if isinstance(cap, dict):
+ cap = cap.get("text")
+ if isinstance(cap, str):
+ return cap
+ return None
+
+
+def _likes_count(node: dict[str, Any]) -> int | None:
+ """Like count. ``-1`` (creator hid it) is passed through, never coerced."""
+ for key in ("edge_liked_by", "edge_media_preview_like"):
+ count = _edge_count(node, key)
+ if count is not None:
+ return count
+ return _int(node.get("like_count"))
+
+
+def _shortcode(node: dict[str, Any]) -> str | None:
+ code = node.get("shortcode") or node.get("code")
+ if isinstance(code, str):
+ return code
+ return None
+
+
+def _user_ref(user: Any) -> dict[str, Any] | None:
+ """A trimmed public-user dict (tagged users / coauthor producers), or None.
+
+ Normalizes the two anonymous dialects: the profile feed nests the handle
+ under ``edge_media_to_tagged_user...node.user`` / ``coauthor_producers`` while
+ the single-post relay blob uses ``usertags.in[].user`` — both carry the same
+ scalar user fields, so this trims them to one shape.
+ """
+ if not isinstance(user, dict):
+ return None
+ ref = {
+ "username": user.get("username"),
+ "fullName": user.get("full_name"),
+ "id": user.get("id") or user.get("pk"),
+ "isVerified": user.get("is_verified"),
+ "profilePicUrl": user.get("profile_pic_url"),
+ }
+ return ref if ref["username"] or ref["id"] else None
+
+
+def _iv2_url(iv2: Any) -> str | None:
+ """First candidate URL from a mobile ``image_versions2`` container, or None."""
+ if isinstance(iv2, dict):
+ cands = iv2.get("candidates")
+ if isinstance(cands, list) and cands and isinstance(cands[0], dict):
+ url = cands[0].get("url")
+ return url if isinstance(url, str) else None
+ return None
+
+
+def _location_ref(loc: Any) -> tuple[str | None, str | None]:
+ """``(name, id)`` from a location node, or ``(None, None)``."""
+ if isinstance(loc, dict):
+ lid = loc.get("id") or loc.get("pk")
+ return loc.get("name"), (str(lid) if lid is not None else None)
+ return None, None
+
+
+def _feed_child(node: dict[str, Any]) -> dict[str, Any]:
+ """Map a profile-feed ``edge_sidecar_to_children`` child to a childPost dict."""
+ dims = node.get("dimensions") if isinstance(node.get("dimensions"), dict) else {}
+ is_video = bool(node.get("is_video"))
+ return {
+ "id": node.get("id"),
+ "shortCode": node.get("shortcode"),
+ "type": "Video" if is_video else "Image",
+ "displayUrl": node.get("display_url"),
+ "videoUrl": node.get("video_url") if is_video else None,
+ "alt": node.get("accessibility_caption"),
+ "dimensionsHeight": _int(dims.get("height")),
+ "dimensionsWidth": _int(dims.get("width")),
+ }
+
+
+def _relay_child(node: dict[str, Any]) -> dict[str, Any]:
+ """Map a single-post relay ``carousel_media`` child to a childPost dict."""
+ mt = node.get("media_type")
+ vv = node.get("video_versions")
+ video_url = (
+ vv[0].get("url")
+ if isinstance(vv, list) and vv and isinstance(vv[0], dict)
+ else None
+ )
+ is_video = mt == 2 or bool(video_url)
+ return {
+ "id": node.get("id"),
+ "shortCode": node.get("code"),
+ "type": _MEDIA_TYPE.get(mt) or ("Video" if is_video else "Image"),
+ "displayUrl": _iv2_url(node.get("image_versions2")) or node.get("display_uri"),
+ "videoUrl": video_url,
+ "alt": node.get("accessibility_caption"),
+ "dimensionsHeight": _int(node.get("original_height")),
+ "dimensionsWidth": _int(node.get("original_width")),
+ }
+
+
+def parse_media(node: dict[str, Any]) -> dict[str, Any]:
+ """Map a raw timeline/feed media node to a flat media item dict."""
+ code = _shortcode(node)
+ caption = _caption_text(node)
+ typename = node.get("__typename")
+ owner = node.get("owner") if isinstance(node.get("owner"), dict) else {}
+ dims = node.get("dimensions") if isinstance(node.get("dimensions"), dict) else {}
+ is_video = bool(node.get("is_video"))
+ children = _edges(node.get("edge_sidecar_to_children"))
+ tagged = [
+ ref
+ for n in _edges(node.get("edge_media_to_tagged_user"))
+ if (ref := _user_ref(n.get("user")))
+ ]
+ coauthors = [
+ ref for c in (node.get("coauthor_producers") or []) if (ref := _user_ref(c))
+ ]
+ loc_name, loc_id = _location_ref(node.get("location"))
+ return {
+ "id": node.get("id"),
+ "type": _TYPE_MAP.get(typename) or ("Video" if is_video else "Image"),
+ "shortCode": code,
+ "caption": caption,
+ "hashtags": _HASHTAG_RE.findall(caption) if caption else [],
+ "mentions": _MENTION_RE.findall(caption) if caption else [],
+ "url": f"{_BASE}/p/{code}/" if code else None,
+ "commentsCount": _edge_count(node, "edge_media_to_comment")
+ or _int(node.get("comment_count")),
+ "dimensionsHeight": _int(dims.get("height")),
+ "dimensionsWidth": _int(dims.get("width")),
+ "displayUrl": node.get("display_url"),
+ "images": [c.get("display_url") for c in children if c.get("display_url")],
+ "childPosts": [_feed_child(c) for c in children],
+ "videoUrl": node.get("video_url") if is_video else None,
+ "alt": node.get("accessibility_caption"),
+ "likesCount": _likes_count(node),
+ "videoViewCount": _int(node.get("video_view_count")) if is_video else None,
+ "videoDuration": node.get("video_duration") if is_video else None,
+ "timestamp": _utc_from_sec(node.get("taken_at_timestamp")),
+ "ownerUsername": owner.get("username"),
+ "ownerId": owner.get("id") or node.get("owner_id"),
+ "ownerFullName": owner.get("full_name"),
+ "isPinned": bool(node.get("pinned_for_users")),
+ "productType": node.get("product_type"),
+ "paidPartnership": node.get("is_paid_partnership"),
+ "taggedUsers": tagged,
+ "coauthorProducers": coauthors,
+ "musicInfo": node.get("clips_music_attribution_info"),
+ "locationName": loc_name,
+ "locationId": loc_id,
+ "isCommentsDisabled": node.get("comments_disabled"),
+ }
+
+
+def parse_profile(user: dict[str, Any]) -> dict[str, Any]:
+ """Map a raw ``web_profile_info`` ``data.user`` to a flat profile item dict."""
+ username = user.get("username")
+ latest = [parse_media(n) for n in _edges(user.get("edge_owner_to_timeline_media"))]
+ related = [
+ ref for n in _edges(user.get("edge_related_profiles")) if (ref := _user_ref(n))
+ ]
+ return {
+ "detailKind": "profile",
+ "id": user.get("id"),
+ "username": username,
+ "url": f"{_BASE}/{username}/" if username else None,
+ "fullName": user.get("full_name"),
+ "biography": user.get("biography"),
+ "externalUrl": user.get("external_url"),
+ "followersCount": _edge_count(user, "edge_followed_by"),
+ "followsCount": _edge_count(user, "edge_follow"),
+ "postsCount": _edge_count(user, "edge_owner_to_timeline_media"),
+ "highlightReelCount": _int(user.get("highlight_reel_count")),
+ "igtvVideoCount": _edge_count(user, "edge_felix_video_timeline"),
+ "isBusinessAccount": user.get("is_business_account"),
+ "businessCategoryName": user.get("business_category_name"),
+ "private": user.get("is_private"),
+ "verified": user.get("is_verified"),
+ "profilePicUrl": user.get("profile_pic_url"),
+ "profilePicUrlHD": user.get("profile_pic_url_hd"),
+ "relatedProfiles": related,
+ "latestPosts": latest,
+ }
+
+
+# Anonymous single-post extraction (/p//, /reel//) --------
+#
+# Instagram serves logged-out visitors the post's full metadata inside the
+# document itself, not via a JSON XHR (the ``?__a=1`` API 404s / login-walls for
+# anonymous callers). Two anonymous surfaces carry it, in order of fidelity:
+# 1. An inline ``', re.DOTALL
+)
+_OG_RE = re.compile(r' on Instagram: ".
+_OG_TITLE_RE = re.compile(r"^(.+?)\s+on Instagram:\s*(.*)$", re.DOTALL)
+# The numeric media id (pk) rides in the App Link deep-link meta tags
+# (al:ios:url / al:android:url = "instagram://media?id=") on anonymous pages,
+# even though the og:* tags omit it.
+_MEDIA_ID_RE = re.compile(r"instagram://media\?id=(\d+)")
+
+
+def _og_date_to_iso(value: str) -> str | None:
+ """``"July 9, 2026"`` -> ``"2026-07-09"`` (date-only; og carries no time)."""
+ try:
+ return (
+ datetime.strptime(value, "%B %d, %Y").replace(tzinfo=UTC).date().isoformat()
+ )
+ except ValueError:
+ return None
+
+
+def _clean_caption(raw: str) -> str | None:
+ """HTML-unescape and unwrap the surrounding quotes off an og caption preview."""
+ return html.unescape(raw).strip().strip('"').strip() or None
+
+
+def _parse_og_meta(og: dict[str, str]) -> dict[str, Any]:
+ """Lift post fields out of Instagram's Open Graph tags (see module notes above).
+
+ Caption + full name come from ``og:title``; counts + username + date from
+ ``og:description``. Every field is optional and independently guarded, so a
+ shape we don't recognise yields a partial (or empty) dict instead of raising.
+ """
+ out: dict[str, Any] = {}
+ desc = og.get("description", "")
+ title = og.get("title", "")
+
+ counts = _OG_COUNTS_RE.search(desc)
+ if counts:
+ out["likes"] = _html_int(counts.group(1))
+ out["comments"] = _html_int(counts.group(2))
+
+ owner_date = _OG_OWNER_DATE_RE.search(desc)
+ if owner_date:
+ out["ownerUsername"] = owner_date.group(1).strip().lstrip("@") or None
+ out["timestamp"] = _og_date_to_iso(owner_date.group(2))
+
+ tm = _OG_TITLE_RE.match(title)
+ if tm:
+ out["ownerFullName"] = tm.group(1).strip() or None
+ out["caption"] = _clean_caption(tm.group(2))
+ elif owner_date:
+ # No usable og:title: fall back to the caption after og:description's
+ # date prefix — still clean (the counts/username/date are stripped).
+ out["caption"] = _clean_caption(desc[owner_date.end() :])
+ return out
+
+
+def _html_int(value: Any) -> int | None:
+ """Coerce a string/number (``"1,234"``) to int, or ``None``."""
+ if isinstance(value, bool):
+ return None
+ if isinstance(value, int | float):
+ return int(value)
+ if isinstance(value, str):
+ digits = value.replace(",", "").strip()
+ if digits.isdigit():
+ return int(digits)
+ return None
+
+
+def _og_tags(html: str) -> dict[str, str]:
+ """Map ``og:`` -> content for the post document."""
+ return {k.lower(): v for k, v in _OG_RE.findall(html)}
+
+
+def _find_media(root: Any, shortcode: str | None) -> dict[str, Any] | None:
+ """Depth-first search a JSON tree for the post's mobile-v1 media object.
+
+ Matches on ``code == shortcode`` (so a carousel *child* or a related post
+ can't be picked instead of the target) plus ``taken_at`` and an id, which
+ together uniquely identify the top-level ``PolarisMedia`` node.
+ """
+ stack = [root]
+ while stack:
+ cur = stack.pop()
+ if isinstance(cur, dict):
+ if (
+ cur.get("taken_at") is not None
+ and ("pk" in cur or "id" in cur)
+ and (shortcode is None or cur.get("code") == shortcode)
+ ):
+ return cur
+ stack.extend(cur.values())
+ elif isinstance(cur, list):
+ stack.extend(cur)
+ return None
+
+
+def _relay_media(html: str, shortcode: str | None) -> dict[str, Any] | None:
+ """Locate the embedded ``PolarisMedia`` object for this post, or ``None``.
+
+ The logged-out media payload is inlined as one of ~40 ``application/json``
+ script blocks. We only ``json.loads`` blocks that mention ``taken_at`` (and
+ the shortcode when known) so a single post fetch doesn't parse every blob.
+ """
+ for raw in _APP_JSON_RE.findall(html):
+ if "taken_at" not in raw:
+ continue
+ if shortcode and shortcode not in raw:
+ continue
+ try:
+ data = json.loads(raw)
+ except (ValueError, TypeError):
+ continue
+ media = _find_media(data, shortcode)
+ if media is not None:
+ return media
+ return None
+
+
+def _media_from_relay(
+ media: dict[str, Any], *, url: str, shortcode: str | None
+) -> dict[str, Any]:
+ """Map an embedded mobile-v1 ``PolarisMedia`` object to a flat media item.
+
+ Same output shape as :func:`parse_media` (so it flows through
+ ``InstagramMediaItem`` unchanged), sourced from the relay dialect
+ (``user``/``taken_at``/``usertags.in``/``carousel_media``/flat counts).
+ """
+ mt = media.get("media_type")
+ cap = media.get("caption")
+ caption = (
+ cap.get("text")
+ if isinstance(cap, dict)
+ else (cap if isinstance(cap, str) else None)
+ )
+ carousel = media.get("carousel_media")
+ carousel = (
+ [c for c in carousel if isinstance(c, dict)]
+ if isinstance(carousel, list)
+ else []
+ )
+ vv = media.get("video_versions")
+ video_url = (
+ vv[0].get("url")
+ if isinstance(vv, list) and vv and isinstance(vv[0], dict)
+ else None
+ )
+ is_video = mt == 2 or bool(video_url)
+ owner = media.get("user") if isinstance(media.get("user"), dict) else {}
+ tagged = [
+ ref
+ for t in ((media.get("usertags") or {}).get("in") or [])
+ if isinstance(t, dict) and (ref := _user_ref(t.get("user")))
+ ]
+ coauthors = [
+ ref for c in (media.get("coauthor_producers") or []) if (ref := _user_ref(c))
+ ]
+ loc_name, loc_id = _location_ref(media.get("location"))
+ # The relay ``id`` is ``POLARIS_``; strip the prefix so single-post ids
+ # match the numeric pk that og-fallback + the al:ios meta also yield.
+ ident = media.get("id")
+ if isinstance(ident, str) and ident.startswith("POLARIS_"):
+ ident = ident[len("POLARIS_") :]
+ pk = media.get("pk")
+ media_id = ident or (str(pk) if pk is not None else None)
+ return {
+ "id": media_id,
+ "type": _MEDIA_TYPE.get(mt) or ("Video" if is_video else "Image"),
+ "shortCode": media.get("code") or shortcode,
+ "caption": caption,
+ "hashtags": list(dict.fromkeys(_HASHTAG_RE.findall(caption)))
+ if caption
+ else [],
+ "mentions": list(dict.fromkeys(_MENTION_RE.findall(caption)))
+ if caption
+ else [],
+ "url": url,
+ "commentsCount": _int(media.get("comment_count")),
+ "dimensionsHeight": _int(media.get("original_height")),
+ "dimensionsWidth": _int(media.get("original_width")),
+ "displayUrl": _iv2_url(media.get("image_versions2"))
+ or media.get("display_uri"),
+ "images": [
+ u
+ for c in carousel
+ if (u := _iv2_url(c.get("image_versions2")) or c.get("display_uri"))
+ ],
+ "childPosts": [_relay_child(c) for c in carousel],
+ "videoUrl": video_url,
+ "alt": media.get("accessibility_caption"),
+ "likesCount": _int(media.get("like_count")),
+ "videoViewCount": _int(media.get("view_count") or media.get("play_count"))
+ if is_video
+ else None,
+ "videoDuration": media.get("video_duration") if is_video else None,
+ "timestamp": _utc_from_sec(media.get("taken_at")),
+ "ownerUsername": owner.get("username"),
+ "ownerId": owner.get("id") or owner.get("pk"),
+ "ownerFullName": owner.get("full_name"),
+ "productType": media.get("product_type"),
+ "taggedUsers": tagged,
+ "coauthorProducers": coauthors,
+ "locationName": loc_name,
+ "locationId": loc_id,
+ }
+
+
+def parse_post(
+ html: str | None, *, url: str, shortcode: str | None = None
+) -> dict[str, Any] | None:
+ """Map an anonymous ``/p//`` (or ``/reel/``) HTML page to a media dict.
+
+ Prefers the embedded mobile-v1 ``PolarisMedia`` relay JSON (full fidelity),
+ falling back to the lossy Open Graph meta tags only if that blob is absent.
+ Returns a dict shaped like :func:`parse_media` (so it flows through
+ ``InstagramMediaItem`` unchanged), or ``None`` when the document carries
+ neither surface (e.g. a login interstitial slipped past the fetch-layer
+ redirect check — the caller treats ``None`` as "nothing to emit", never a
+ silent success).
+ """
+ if not isinstance(html, str) or not html.strip():
+ return None
+
+ media = _relay_media(html, shortcode)
+ if media is not None:
+ return _media_from_relay(media, url=url, shortcode=shortcode)
+
+ # Fallback: no embedded relay blob -> Open Graph meta only.
+ og = _og_tags(html)
+ if not og:
+ return None
+ og_meta = _parse_og_meta(og)
+ caption = og_meta.get("caption")
+ video_url = og.get("video")
+ is_video = bool(video_url) or og.get("type") == "video.other"
+ id_match = _MEDIA_ID_RE.search(html)
+ return {
+ "id": id_match.group(1) if id_match else None,
+ "type": "Video" if is_video else "Image",
+ "shortCode": shortcode,
+ "caption": caption,
+ "hashtags": list(dict.fromkeys(_HASHTAG_RE.findall(caption)))
+ if caption
+ else [],
+ "mentions": list(dict.fromkeys(_MENTION_RE.findall(caption)))
+ if caption
+ else [],
+ "url": url,
+ "commentsCount": og_meta.get("comments"),
+ "displayUrl": og.get("image"),
+ "videoUrl": video_url if is_video else None,
+ "likesCount": og_meta.get("likes"),
+ "timestamp": og_meta.get("timestamp"),
+ "ownerUsername": og_meta.get("ownerUsername"),
+ "ownerFullName": og_meta.get("ownerFullName"),
+ }
diff --git a/surfsense_backend/app/proprietary/platforms/instagram/schemas.py b/surfsense_backend/app/proprietary/platforms/instagram/schemas.py
new file mode 100644
index 000000000..930a2acb9
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/instagram/schemas.py
@@ -0,0 +1,136 @@
+# ruff: noqa: N815
+"""Input/output models for the Instagram scraper.
+
+The models mirror the public Instagram scraper actor spec so the endpoint can be
+a drop-in: the input accepts the full documented surface, and every output field
+is emitted (``None``/``[]`` when the anonymous web endpoints cannot source it
+yet) so the contract expands additively — the same rule the Google Search and
+YouTube models follow.
+
+**Anonymous only.** There is deliberately **no** authentication field on the
+input (no username/password/token/cookie/``login*``) — the scraper holds only
+Instagram's anonymous web-session cookies (``csrftoken``/``mid``) and can never
+log in. Anything auth-shaped a caller sends lands in ``extra`` and is ignored.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+InstagramResultsType = Literal["posts", "details", "reels"]
+# Kept as distinct values for actor-spec parity, but under anonymous Google-backed
+# discovery ``user`` and ``profile`` are aliases: both resolve to profile targets
+# (IG's own hashtag/place/keyword search is login-walled).
+InstagramSearchType = Literal["profile", "user"]
+
+
+class InstagramScrapeInput(BaseModel):
+ """Instagram scraper input surface (anonymous, no auth fields).
+
+ Field names mirror the public actor spec verbatim. ``resultsLimit`` /
+ ``searchLimit`` are collector policy applied by :func:`scrape_instagram`,
+ NOT ceilings baked into the streaming flows. Fields the acquisition layer
+ doesn't source yet are still accepted via ``extra="allow"`` for parity.
+ """
+
+ model_config = ConfigDict(extra="allow")
+ resultsType: InstagramResultsType = "posts"
+ directUrls: list[str] = Field(default_factory=list)
+ resultsLimit: int | None = Field(default=None, ge=1)
+ onlyPostsNewerThan: str | None = None
+ search: str | None = None
+ searchType: InstagramSearchType = "profile"
+ searchLimit: int | None = Field(default=None, ge=1, le=250)
+ addParentData: bool = False
+ skipPinnedPosts: bool = False
+ addProfileStatistics: bool = False
+
+
+class _ItemBase(BaseModel):
+ """Common error / provenance fields carried on every output item.
+
+ Errors surface as item-level fields (never exceptions) so a partial run
+ still returns the items it could source, mirroring the actor's shape.
+ """
+
+ model_config = ConfigDict(extra="allow")
+ inputUrl: str | None = None
+ error: str | None = None
+ errorDescription: str | None = None
+ requestErrorMessages: list[str] = Field(default_factory=list)
+
+ def to_output(self) -> dict[str, Any]:
+ """Serialize to the flat output dict shape (keeps extras)."""
+ return self.model_dump(exclude_none=False)
+
+
+class InstagramMediaItem(_ItemBase):
+ """A post or reel. One flat schema per the actor FAQ.
+
+ ``firstComment``/``latestComments`` are intentionally absent: comment
+ *content* is login-walled (only the anonymous comment *count* is exposed, as
+ ``commentsCount``), so this scraper can never source them.
+ """
+
+ id: str | None = None
+ type: Literal["Image", "Video", "Sidecar"] | None = None
+ shortCode: str | None = None
+ caption: str | None = None
+ hashtags: list[str] = Field(default_factory=list)
+ mentions: list[str] = Field(default_factory=list)
+ url: str | None = None
+ commentsCount: int | None = None
+ dimensionsHeight: int | None = None
+ dimensionsWidth: int | None = None
+ displayUrl: str | None = None
+ images: list[str] = Field(default_factory=list)
+ videoUrl: str | None = None
+ alt: str | None = None
+ likesCount: int | None = None
+ videoViewCount: int | None = None
+ videoPlayCount: int | None = None
+ reshareCount: int | None = None
+ timestamp: str | None = None
+ childPosts: list[dict[str, Any]] = Field(default_factory=list)
+ ownerUsername: str | None = None
+ ownerId: str | None = None
+ ownerFullName: str | None = None
+ isPinned: bool | None = None
+ productType: str | None = None
+ videoDuration: float | None = None
+ paidPartnership: bool | None = None
+ taggedUsers: list[dict[str, Any]] = Field(default_factory=list)
+ musicInfo: dict[str, Any] | None = None
+ coauthorProducers: list[dict[str, Any]] = Field(default_factory=list)
+ locationName: str | None = None
+ locationId: str | None = None
+ isCommentsDisabled: bool | None = None
+ dataSource: dict[str, Any] | None = None
+
+
+class InstagramProfile(_ItemBase):
+ """A profile detail item (``detailKind = "profile"``)."""
+
+ detailKind: Literal["profile"] = "profile"
+ id: str | None = None
+ username: str | None = None
+ url: str | None = None
+ fullName: str | None = None
+ biography: str | None = None
+ externalUrl: str | None = None
+ followersCount: int | None = None
+ followsCount: int | None = None
+ postsCount: int | None = None
+ highlightReelCount: int | None = None
+ igtvVideoCount: int | None = None
+ isBusinessAccount: bool | None = None
+ businessCategoryName: str | None = None
+ private: bool | None = None
+ verified: bool | None = None
+ profilePicUrl: str | None = None
+ profilePicUrlHD: str | None = None
+ relatedProfiles: list[dict[str, Any]] = Field(default_factory=list)
+ latestPosts: list[dict[str, Any]] = Field(default_factory=list)
+ statistics: dict[str, Any] | None = None
diff --git a/surfsense_backend/app/proprietary/platforms/instagram/scraper.py b/surfsense_backend/app/proprietary/platforms/instagram/scraper.py
new file mode 100644
index 000000000..f248b5653
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/instagram/scraper.py
@@ -0,0 +1,429 @@
+"""Orchestrator for the Instagram scraper.
+
+The core is the async generator :func:`iter_instagram` (unbounded);
+:func:`scrape_instagram` is a thin collector with a caller-supplied ``limit``
+guard. Any cap is caller policy, never baked into flow logic.
+
+Independent targets (one per ``directUrl`` / discovered entity) fan out
+concurrently on a pool of warm sessions (sticky IPs); each target's own paging
+stays sequential. ``fan_out`` is ported from ``../reddit/scraper.py`` but bound
+to *this* module's proxy holders so every worker warms its own session once and
+reuses it.
+
+Anonymous-only. Every surface here is reachable without a login: profile web
+info, the media embedded in a profile page, single-post/reel pages, and
+Google-backed handle discovery. Login-walled surfaces (hashtag/place feeds,
+comment threads, IG's native keyword search) are deliberately absent.
+
+Flows are selected by ``resultsType``:
+- ``posts`` / ``reels`` -> media items (profile feed, or a single
+ ``/p/``/``/reel/`` page, or discovery search)
+- ``details`` -> profile metadata (by URL or discovery search)
+
+ponytail: deep feed pagination (past the first web page of media) needs the
+GraphQL cursor endpoint whose doc-id drifts; v1 emits the first page and stops.
+The upgrade path is a ``_paginate_feed`` helper in this file plus a doc-id in
+``fetch.py`` — contained to these two files, per the acquisition-seam rule.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import re
+from collections.abc import AsyncIterator
+from contextlib import aclosing
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+from app.proprietary.platforms.google_search.schemas import GoogleSearchScrapeInput
+from app.proprietary.platforms.google_search.scraper import scrape_serps
+
+from .fetch import (
+ InstagramAccessBlockedError,
+ bind_proxy_holder,
+ fetch_html,
+ fetch_json,
+ now_iso,
+ open_proxy_holder,
+)
+from .parsers import parse_media, parse_post, parse_profile
+from .schemas import InstagramScrapeInput
+from .url_resolver import ResolvedUrl, resolve_url
+
+logger = logging.getLogger(__name__)
+
+__all__ = [
+ "InstagramAccessBlockedError",
+ "iter_instagram",
+ "scrape_instagram",
+]
+
+# Independent jobs run concurrently on a pool of warm proxy sessions. Anonymous
+# Instagram is the most hostile platform, so this stays low to avoid burning the
+# residential pool with parallel login walls.
+_FANOUT_CONCURRENCY = 8
+
+_PROFILE_PATH = "api/v1/users/web_profile_info/"
+
+# Instagram usernames: 1-30 chars of letters/digits/period/underscore. Used to
+# treat a profile/user discovery query as a direct (anonymous) handle lookup.
+_HANDLE_RE = re.compile(r"[A-Za-z0-9._]{1,30}\Z")
+
+
+def _parse_newer_than(value: str | None) -> datetime | None:
+ """Parse ``onlyPostsNewerThan`` (ISO, YYYY-MM-DD, or relative) to UTC.
+
+ Relative forms: ``" "`` where unit is minute/hour/day/week/month/
+ year (singular or plural). Anything unparseable returns ``None`` (no filter).
+ """
+ if not value:
+ return None
+ text = value.strip().lower()
+ parts = text.split()
+ if len(parts) == 2 and parts[0].isdigit():
+ n = int(parts[0])
+ unit = parts[1].rstrip("s")
+ days = {
+ "minute": n / 1440,
+ "hour": n / 24,
+ "day": n,
+ "week": n * 7,
+ "month": n * 30,
+ "year": n * 365,
+ }.get(unit)
+ if days is None:
+ return None
+ return datetime.now(UTC) - timedelta(days=days)
+ try:
+ dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
+ if dt.tzinfo:
+ return dt
+ return dt.replace(tzinfo=UTC)
+ except ValueError:
+ return None
+
+
+def _is_after(timestamp: str | None, cutoff: datetime | None) -> bool:
+ """True if the item ``timestamp`` (ISO) is at/after the cutoff (or no cutoff)."""
+ if cutoff is None:
+ return True
+ if not timestamp:
+ return True
+ try:
+ dt = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
+ return dt >= cutoff
+ except ValueError:
+ return True
+
+
+async def fan_out(
+ jobs: list[AsyncIterator[dict[str, Any]]], *, concurrency: int = _FANOUT_CONCURRENCY
+) -> AsyncIterator[dict[str, Any]]:
+ """Stream items from independent async-iterator jobs via a warm worker pool.
+
+ Each worker opens ONE proxy session and reuses it across the sequential jobs
+ it pulls, so only the first job per worker pays the proxy handshake + the
+ cookie warm-up. Partial results (matches the reddit sibling): one blocked or
+ failed target yields nothing rather than aborting the batch — Instagram is
+ an aggregation, not an atomic transaction, so 4/5 good targets beat 0/5. But
+ if EVERY target was refused (zero items AND a hard block seen), the whole run
+ couldn't reach anonymous data, so we surface ``InstagramAccessBlockedError``
+ (-> 403) instead of a misleading empty success. Workers are cancelled and
+ their sessions closed if the consumer stops early.
+ """
+ if not jobs:
+ return
+ job_queue: asyncio.Queue[AsyncIterator[dict[str, Any]]] = asyncio.Queue()
+ for job in jobs:
+ job_queue.put_nowait(job)
+ results: asyncio.Queue[list[dict[str, Any]]] = asyncio.Queue()
+ blocked = False # set if any target hit a hard login/auth wall
+
+ async def worker() -> None:
+ nonlocal blocked
+ holder = None
+ try:
+ holder = await open_proxy_holder()
+ except Exception as e: # no session: jobs still run via one-shot fetches
+ logger.warning("[instagram] proxy session open failed: %s", e)
+ try:
+ while True:
+ try:
+ job = job_queue.get_nowait()
+ except asyncio.QueueEmpty:
+ return
+ items: list[dict[str, Any]] = []
+ try:
+ if holder is not None:
+ async with bind_proxy_holder(holder):
+ items = [item async for item in job]
+ else:
+ items = [item async for item in job]
+ except InstagramAccessBlockedError as e:
+ # Partial results: a blocked target must not kill the batch.
+ # Record it so a fully-blocked run can still surface the 403.
+ blocked = True
+ logger.warning("[instagram] target blocked: %s", e)
+ except Exception as e: # one bad target must not kill the run
+ logger.warning("[instagram] fan-out job failed: %s", e)
+ await results.put(items)
+ finally:
+ if holder is not None:
+ await holder.close()
+
+ tasks = [asyncio.create_task(worker()) for _ in range(min(concurrency, len(jobs)))]
+ emitted = 0
+ try:
+ for _ in range(len(jobs)):
+ for item in await results.get():
+ emitted += 1
+ yield item
+ finally:
+ for task in tasks:
+ if not task.done():
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ # Reached only on natural exhaustion (an early-stop close raises GeneratorExit
+ # inside the loop and skips this). Nothing came back AND a wall was hit ->
+ # the run was fully refused, so fail loud rather than return empty.
+ if emitted == 0 and blocked:
+ raise InstagramAccessBlockedError(
+ "Instagram refused anonymous access to every target"
+ )
+
+
+def _emit(partial: dict[str, Any], *, input_url: str | None) -> dict[str, Any]:
+ """Stamp provenance and serialize (parsers return plain dicts)."""
+ out = {**partial, "scrapedAt": now_iso()}
+ if input_url is not None:
+ out.setdefault("inputUrl", input_url)
+ return out
+
+
+async def _profile_user(username: str) -> dict[str, Any] | None:
+ """Fetch a profile's ``data.user`` node, or ``None``."""
+ data = await fetch_json(_PROFILE_PATH, {"username": username})
+ if isinstance(data, dict):
+ user = (
+ data.get("data", {}).get("user")
+ if isinstance(data.get("data"), dict)
+ else None
+ )
+ if isinstance(user, dict):
+ return user
+ return None
+ return None
+
+
+def _media_matches(item: dict[str, Any], result_type: str) -> bool:
+ """Filter a media item by feed type. ``reels`` keeps clips/videos only."""
+ if result_type == "reels":
+ return item.get("type") == "Video" or item.get("productType") == "clips"
+ return True
+
+
+async def _media_flow(
+ resolved: ResolvedUrl,
+ *,
+ input_model: InstagramScrapeInput,
+ cutoff: datetime | None,
+ per_target: int,
+) -> AsyncIterator[dict[str, Any]]:
+ """Emit media items for a profile feed, or a single ``/p/``/``/reel/`` page."""
+ from .parsers import _edges
+
+ result_type = input_model.resultsType
+ if resolved.kind == "profile":
+ user = await _profile_user(resolved.value)
+ if user is None:
+ return
+ nodes = _edges(user.get("edge_owner_to_timeline_media"))
+ emitted = 0
+ for node in nodes:
+ item = parse_media(node)
+ if input_model.skipPinnedPosts and item.get("isPinned"):
+ continue
+ if not _media_matches(item, result_type):
+ continue
+ if not _is_after(item.get("timestamp"), cutoff):
+ continue
+ yield _emit(item, input_url=resolved.url)
+ emitted += 1
+ if emitted >= per_target:
+ return
+ return
+ if resolved.kind in ("post", "reel"):
+ # Single-post extraction: the anonymous ``?__a=1`` JSON API 404s/login-
+ # walls, but the public /p// document embeds the mobile-v1
+ # PolarisMedia JSON (og-meta fallback), which parse_post reads. Numeric-ID
+ # URLs can't be keyed this way (the page needs the shortCode), so they're
+ # skipped upstream.
+ if resolved.numeric_post_id:
+ return
+ html = await fetch_html(f"p/{resolved.value}/")
+ item = parse_post(html, url=resolved.url, shortcode=resolved.value)
+ if item is None:
+ return
+ if not _media_matches(item, result_type):
+ return
+ if not _is_after(item.get("timestamp"), cutoff):
+ return
+ yield _emit(item, input_url=resolved.url)
+ return
+
+
+async def _details_flow(
+ resolved: ResolvedUrl, *, input_model: InstagramScrapeInput
+) -> AsyncIterator[dict[str, Any]]:
+ """Emit one profile detail item for a URL (anonymous web_profile_info)."""
+ if resolved.kind == "profile":
+ user = await _profile_user(resolved.value)
+ if user is not None:
+ yield _emit(parse_profile(user), input_url=resolved.url)
+
+
+def _kind_matches(resolved: ResolvedUrl, search_type: str) -> bool:
+ """True if a resolved IG URL is the kind the discovery query asked for.
+
+ Discovery is profile-only now (hashtag/place feeds are login-walled), so
+ every supported ``search_type`` maps to a profile target.
+ """
+ return resolved.kind == "profile"
+
+
+async def _discover_via_google(
+ query: str, *, search_type: str, limit: int
+) -> list[ResolvedUrl]:
+ """Discover IG profile targets via Google ``site:instagram.com`` (anonymous).
+
+ Instagram's own keyword search is login-walled, so we reuse the existing
+ ``google_search`` platform, classify each organic URL with ``resolve_url``,
+ keep the profile hits, de-dup, and cap at ``limit``.
+
+ Quality caveat: results reflect Google's index/ranking of instagram.com, not
+ IG's own relevance. This is discovery, not search parity (see README).
+ """
+ serps = await scrape_serps(
+ GoogleSearchScrapeInput(
+ queries=query, site="instagram.com", maxPagesPerQuery=1
+ ),
+ limit=1,
+ )
+ resolved: list[ResolvedUrl] = []
+ seen: set[tuple[str, str]] = set()
+ for serp in serps:
+ for org in serp.get("organicResults") or []:
+ url = org.get("url", "") if isinstance(org, dict) else ""
+ r = resolve_url(url)
+ if r is None or not _kind_matches(r, search_type):
+ continue
+ key = (r.kind, r.value)
+ if key in seen:
+ continue
+ seen.add(key)
+ resolved.append(r)
+ if len(resolved) >= limit:
+ return resolved
+ return resolved
+
+
+async def _discover(query: str, *, search_type: str, limit: int) -> list[ResolvedUrl]:
+ """Resolve a discovery query into profile targets - anonymously.
+
+ A query that is a valid handle resolves directly against the anonymous
+ profile endpoint ("messi" -> instagram.com/messi/). A non-handle query (e.g.
+ "national geographic") goes through Google ``site:instagram.com`` since IG's
+ native keyword search is login-walled.
+ """
+ handle = query.strip().lstrip("@")
+ if _HANDLE_RE.match(handle):
+ url = f"https://www.instagram.com/{handle}/"
+ return [ResolvedUrl("profile", handle, url)][:limit]
+ return await _discover_via_google(query, search_type=search_type, limit=limit)
+
+
+def _resolve_inputs(input_model: InstagramScrapeInput) -> list[ResolvedUrl]:
+ """Resolve ``directUrls`` (URLs take priority over ``search``)."""
+ resolved: list[ResolvedUrl] = []
+ for url in input_model.directUrls:
+ r = resolve_url(url)
+ if r is None:
+ logger.warning("[instagram] unrecognized URL: %s", url)
+ continue
+ resolved.append(r)
+ return resolved
+
+
+async def _targets(input_model: InstagramScrapeInput) -> list[ResolvedUrl]:
+ """The resolved targets for this run: direct URLs, else discovery search."""
+ if input_model.directUrls:
+ return _resolve_inputs(input_model)
+ if not input_model.search:
+ return []
+ limit = input_model.searchLimit or 10
+ queries = [q.strip() for q in input_model.search.split(",") if q.strip()]
+ targets: list[ResolvedUrl] = []
+ for query in queries:
+ targets.extend(
+ await _discover(query, search_type=input_model.searchType, limit=limit)
+ )
+ return targets
+
+
+async def iter_instagram(
+ input_model: InstagramScrapeInput,
+) -> AsyncIterator[dict[str, Any]]:
+ """Yield flat Instagram items. ``directUrls`` override ``search``.
+
+ Independent targets fan out concurrently; each target's paging stays
+ sequential. De-dupes media by ``id`` across targets.
+ """
+ targets = await _targets(input_model)
+ if not targets:
+ return
+ result_type = input_model.resultsType
+ cutoff = _parse_newer_than(input_model.onlyPostsNewerThan)
+ per_target = input_model.resultsLimit or 10
+
+ if result_type == "details":
+ jobs = [_details_flow(r, input_model=input_model) for r in targets]
+ async with aclosing(fan_out(jobs)) as stream:
+ async for item in stream:
+ yield item
+ return
+
+ # posts / reels -> media feeds, de-duped by id across targets.
+ jobs = [
+ _media_flow(r, input_model=input_model, cutoff=cutoff, per_target=per_target)
+ for r in targets
+ ]
+ seen: set[str] = set()
+ async with aclosing(fan_out(jobs)) as stream:
+ async for item in stream:
+ item_id = item.get("id")
+ if isinstance(item_id, str):
+ if item_id in seen:
+ continue
+ seen.add(item_id)
+ yield item
+
+
+async def scrape_instagram(
+ input_model: InstagramScrapeInput, *, limit: int | None = None
+) -> list[dict[str, Any]]:
+ """Collect :func:`iter_instagram` into a list, honoring an optional ``limit``.
+
+ ``limit`` is a request-time policy guard, NOT a ceiling in the streaming
+ core.
+ """
+ from app.capabilities.core.progress import emit_progress
+
+ results: list[dict[str, Any]] = []
+ async with aclosing(iter_instagram(input_model)) as stream:
+ async for item in stream:
+ results.append(item)
+ emit_progress("scraping", current=len(results), total=limit, unit="item")
+ if limit is not None and len(results) >= limit:
+ break
+ return results
diff --git a/surfsense_backend/app/proprietary/platforms/instagram/url_resolver.py b/surfsense_backend/app/proprietary/platforms/instagram/url_resolver.py
new file mode 100644
index 000000000..197a90276
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/instagram/url_resolver.py
@@ -0,0 +1,85 @@
+"""Classify and normalize an Instagram URL into a scrape job.
+
+Covers the anonymously-scrapable ``directUrls`` shapes: a profile, a post
+(``/p/``), and a reel (``/reel/``), plus bare profile IDs. Hashtag and place
+URLs are deliberately unsupported — their feeds are login-walled for anonymous
+callers (use Google-backed discovery + single-post extraction instead).
+Non-Instagram hosts resolve to ``None`` so the orchestrator can skip them.
+Mirrors the frozen ``ResolvedUrl`` dataclass shape of ``../reddit/url_resolver.py``.
+
+Normalization rules (from the reference spec):
+- ``_u/`` and ``/profilecard/`` segments are stripped.
+- Story URLs (``/stories//...``) reduce to the profile.
+- Numeric post-ID URLs cannot be single-post-extracted anonymously (the HTML
+ page keys on the shortCode), so they resolve with ``numeric_post_id`` set and
+ the media flow skips them.
+- ``share/`` links are unsupported (they need a network redirect to resolve to a
+ canonical post/profile URL); pass the resolved ``/p/`` or profile URL instead.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Literal
+from urllib.parse import urlparse
+
+ResolvedKind = Literal["profile", "post", "reel"]
+
+_INSTAGRAM_HOSTS = frozenset({"m.instagram.com", "www.instagram.com", "instagram.com"})
+_STRIP_SEGMENTS = frozenset({"_u", "profilecard"})
+_RESERVED = frozenset(
+ {"p", "s", "tv", "reel", "reels", "share", "explore", "stories", "accounts"}
+)
+
+
+@dataclass(frozen=True)
+class ResolvedUrl:
+ """A classified Instagram target: the kind, its key, and the source URL."""
+
+ kind: ResolvedKind
+ value: str
+ url: str
+ numeric_post_id: bool = False
+
+
+def _is_instagram_host(hostname: str | None) -> bool:
+ if not hostname:
+ return False
+ return hostname.lower() in _INSTAGRAM_HOSTS
+
+
+def _segments(url: str) -> list[str]:
+ parsed = urlparse(url)
+ if not _is_instagram_host(parsed.hostname):
+ return []
+ if not parsed.path:
+ return []
+ return [s for s in parsed.path.split("/") if s and s not in _STRIP_SEGMENTS]
+
+
+def resolve_url(url: str) -> ResolvedUrl | None:
+ """Classify an Instagram URL into a scrape job, or ``None`` if unrecognized.
+
+ A bare token with no ``http`` prefix and no ``/`` is treated as a profile ID
+ (the reference accepts bare profile handles alongside full URLs).
+ """
+ if "instagram.com" not in url.lower():
+ token = url.strip().lstrip("@")
+ if token and "/" not in token and "." not in token:
+ return ResolvedUrl("profile", token, f"https://www.instagram.com/{token}/")
+ segments = _segments(url)
+ if not segments:
+ return None
+ head = segments[0]
+ if head == "p" and len(segments) >= 2:
+ code = segments[1]
+ return ResolvedUrl("post", code, url, numeric_post_id=code.isdigit())
+ if head in ("reel", "reels") and len(segments) >= 2:
+ code = segments[1]
+ return ResolvedUrl("reel", code, url, numeric_post_id=code.isdigit())
+ if head == "stories" and len(segments) >= 2:
+ user = segments[1]
+ return ResolvedUrl("profile", user, f"https://www.instagram.com/{user}/")
+ if head not in _RESERVED:
+ return ResolvedUrl("profile", head, url)
+ return None
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/__init__.py b/surfsense_backend/app/proprietary/platforms/tiktok/__init__.py
new file mode 100644
index 000000000..91f716d1f
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/__init__.py
@@ -0,0 +1,35 @@
+"""Anonymous, blob-first TikTok scraper (public interface).
+
+The capability layer depends only on the names re-exported here: the input
+schema, the collector/generator, the video item shape, and the hard-block error.
+"""
+
+from __future__ import annotations
+
+from .orchestrator import (
+ iter_tiktok,
+ scrape_tiktok,
+ scrape_tiktok_comments,
+ scrape_tiktok_trending,
+ search_tiktok_users,
+)
+from .schemas import (
+ CommentItem,
+ TikTokProfileItem,
+ TikTokScrapeInput,
+ TikTokVideoItem,
+)
+from .session import TikTokAccessBlockedError
+
+__all__ = [
+ "CommentItem",
+ "TikTokAccessBlockedError",
+ "TikTokProfileItem",
+ "TikTokScrapeInput",
+ "TikTokVideoItem",
+ "iter_tiktok",
+ "scrape_tiktok",
+ "scrape_tiktok_comments",
+ "scrape_tiktok_trending",
+ "search_tiktok_users",
+]
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/__init__.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/__init__.py
new file mode 100644
index 000000000..3afd536f2
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/__init__.py
@@ -0,0 +1,25 @@
+"""Turn raw TikTok page/API payloads into normalized items."""
+
+from __future__ import annotations
+
+from .author import parse_author, parse_profile
+from .comments import comments_from_response, parse_comment
+from .hydration import extract_rehydration_data
+from .item_list import items_from_response
+from .scopes import user_info, video_item_struct
+from .user_search import parse_search_user, users_from_response
+from .video import parse_video
+
+__all__ = [
+ "comments_from_response",
+ "extract_rehydration_data",
+ "items_from_response",
+ "parse_author",
+ "parse_comment",
+ "parse_profile",
+ "parse_search_user",
+ "parse_video",
+ "user_info",
+ "users_from_response",
+ "video_item_struct",
+]
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/author.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/author.py
new file mode 100644
index 000000000..7dc1783e4
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/author.py
@@ -0,0 +1,38 @@
+"""Normalize TikTok author/profile payloads into an ``authorMeta`` dict."""
+
+from __future__ import annotations
+
+from typing import Any
+
+_PROFILE_URL = "https://www.tiktok.com/@{username}"
+
+
+def build_author_meta(author: dict[str, Any], stats: dict[str, Any]) -> dict[str, Any]:
+ """Map an author object + its stats to the ``authorMeta`` output shape."""
+ username = author.get("uniqueId")
+ return {
+ "id": author.get("id"),
+ "name": username,
+ "nickName": author.get("nickname"),
+ "profileUrl": _PROFILE_URL.format(username=username) if username else None,
+ "verified": author.get("verified"),
+ "signature": author.get("signature"),
+ "avatar": author.get("avatarLarger") or author.get("avatarMedium"),
+ "privateAccount": author.get("privateAccount"),
+ "fans": stats.get("followerCount"),
+ "following": stats.get("followingCount"),
+ "heart": stats.get("heartCount"),
+ "video": stats.get("videoCount"),
+ }
+
+
+def parse_author(user_info: dict[str, Any]) -> dict[str, Any]:
+ """Map a ``webapp.user-detail`` ``userInfo`` (``{user, stats}``) to authorMeta."""
+ return build_author_meta(user_info.get("user") or {}, user_info.get("stats") or {})
+
+
+def parse_profile(user_info: dict[str, Any]) -> dict[str, Any]:
+ """Map a ``userInfo`` to a standalone :class:`TikTokProfileItem` output dict."""
+ from ..schemas.items import TikTokProfileItem
+
+ return TikTokProfileItem(**parse_author(user_info)).to_output()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/comments.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/comments.py
new file mode 100644
index 000000000..8ca96cad3
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/comments.py
@@ -0,0 +1,55 @@
+"""Parse a captured ``/api/comment/list`` response into comment items.
+
+The comment API returns ``{"comments": [...]}`` where each entry uses the
+mobile-API snake_case shape (``cid``, ``digg_count``, ``reply_comment_total``,
+``create_time``, and a nested ``user`` with ``uid``/``unique_id``/``nickname``/
+``avatar_thumb``). ``reply_id != "0"`` marks a reply to a parent comment.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from .timestamps import epoch_to_iso
+
+
+def comments_from_response(body: Any) -> list[dict[str, Any]]:
+ """Return the raw comment records carried by one API response, or ``[]``."""
+ if not isinstance(body, dict):
+ return []
+ comments = body.get("comments")
+ if not isinstance(comments, list):
+ return []
+ return [c for c in comments if isinstance(c, dict)]
+
+
+def _avatar(user: dict[str, Any]) -> str | None:
+ thumb = user.get("avatar_thumb")
+ if isinstance(thumb, dict):
+ urls = thumb.get("url_list")
+ if isinstance(urls, list) and urls:
+ return urls[0]
+ return None
+
+
+def parse_comment(raw: dict[str, Any], video_url: str) -> dict[str, Any]:
+ """Map a raw comment record to a :class:`CommentItem` output dict."""
+ from ..schemas.items import CommentItem
+
+ user = raw.get("user") if isinstance(raw.get("user"), dict) else {}
+ reply_id = raw.get("reply_id")
+ create_time = raw.get("create_time")
+ return CommentItem(
+ id=raw.get("cid"),
+ text=raw.get("text"),
+ videoWebUrl=video_url,
+ diggCount=raw.get("digg_count"),
+ replyCommentTotal=raw.get("reply_comment_total"),
+ createTime=create_time,
+ createTimeISO=epoch_to_iso(create_time),
+ uid=user.get("uid"),
+ uniqueId=user.get("unique_id"),
+ nickName=user.get("nickname"),
+ avatar=_avatar(user),
+ repliesToId=reply_id if reply_id and reply_id != "0" else None,
+ ).to_output()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/hydration.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/hydration.py
new file mode 100644
index 000000000..4cc466849
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/hydration.py
@@ -0,0 +1,28 @@
+"""Extract the ``__UNIVERSAL_DATA_FOR_REHYDRATION__`` JSON embedded in page HTML.
+
+TikTok server-renders the first page of data into a single script tag; parsing
+it yields page-one items (video/profile/hashtag) without any signed API call.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Any
+
+_BLOB_RE = re.compile(
+ r']*id="__UNIVERSAL_DATA_FOR_REHYDRATION__"[^>]*>(.*?)',
+ re.DOTALL,
+)
+
+
+def extract_rehydration_data(html: str) -> dict[str, Any] | None:
+ """Return the parsed rehydration blob, or ``None`` if absent/unparseable."""
+ match = _BLOB_RE.search(html)
+ if not match:
+ return None
+ try:
+ data = json.loads(match.group(1))
+ except (json.JSONDecodeError, ValueError):
+ return None
+ return data if isinstance(data, dict) else None
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/item_list.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/item_list.py
new file mode 100644
index 000000000..7dddf891f
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/item_list.py
@@ -0,0 +1,30 @@
+"""Item structs from a captured ``item_list`` / search API response body.
+
+Profile and hashtag listings return ``{"itemList": [...]}``; search returns
+``{"data": [{"item": {...}}]}``. Both element shapes are the same itemStruct
+:func:`parse_video` already consumes.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def items_from_response(body: Any) -> list[dict[str, Any]]:
+ """Return the itemStructs carried by one API response, or ``[]``."""
+ if not isinstance(body, dict):
+ return []
+
+ item_list = body.get("itemList")
+ if isinstance(item_list, list):
+ return [i for i in item_list if isinstance(i, dict)]
+
+ data = body.get("data")
+ if isinstance(data, list):
+ return [
+ entry["item"]
+ for entry in data
+ if isinstance(entry, dict) and isinstance(entry.get("item"), dict)
+ ]
+
+ return []
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/scopes.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/scopes.py
new file mode 100644
index 000000000..a20a72b2b
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/scopes.py
@@ -0,0 +1,30 @@
+"""Navigate the rehydration blob to the scopes the flows consume."""
+
+from __future__ import annotations
+
+from typing import Any
+
+_DEFAULT = "__DEFAULT_SCOPE__"
+
+
+def _scope(data: dict[str, Any], name: str) -> dict[str, Any] | None:
+ scope = (data.get(_DEFAULT) or {}).get(name)
+ return scope if isinstance(scope, dict) else None
+
+
+def video_item_struct(data: dict[str, Any]) -> dict[str, Any] | None:
+ """The ``itemStruct`` of a video-detail page, or ``None``."""
+ scope = _scope(data, "webapp.video-detail")
+ if not scope:
+ return None
+ item = (scope.get("itemInfo") or {}).get("itemStruct")
+ return item if isinstance(item, dict) else None
+
+
+def user_info(data: dict[str, Any]) -> dict[str, Any] | None:
+ """The ``userInfo`` (``{user, stats}``) of a profile page, or ``None``."""
+ scope = _scope(data, "webapp.user-detail")
+ if not scope:
+ return None
+ info = scope.get("userInfo")
+ return info if isinstance(info, dict) else None
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/timestamps.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/timestamps.py
new file mode 100644
index 000000000..ed45baa8f
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/timestamps.py
@@ -0,0 +1,18 @@
+"""Millisecond-ISO timestamps matching the actor's output shape."""
+
+from __future__ import annotations
+
+from datetime import UTC, datetime
+
+
+def epoch_to_iso(seconds: int | str | None) -> str | None:
+ """Convert a Unix-seconds timestamp to ``YYYY-MM-DDTHH:MM:SS.000Z``."""
+ if not seconds:
+ return None
+ stamp = datetime.fromtimestamp(int(seconds), tz=UTC)
+ return stamp.strftime("%Y-%m-%dT%H:%M:%S.000Z")
+
+
+def now_iso() -> str:
+ """Current UTC time in the millisecond-ISO output shape."""
+ return datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/user_search.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/user_search.py
new file mode 100644
index 000000000..5ffc2cd4d
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/user_search.py
@@ -0,0 +1,56 @@
+"""Parse the ``/api/search/user`` response into profile items.
+
+User search returns ``{"user_list": [{"user_info": {...}}, ...]}`` where each
+``user_info`` uses the mobile-API snake_case shape (``uid``, ``unique_id``,
+``follower_count``, ``total_favorited``, ``avatar_thumb.url_list``) — distinct
+from the camelCase ``webapp.user-detail`` blob the profile flow reads, so it gets
+its own mapping into the shared :class:`TikTokProfileItem` output contract.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+_PROFILE_URL = "https://www.tiktok.com/@{username}"
+
+
+def users_from_response(body: Any) -> list[dict[str, Any]]:
+ """Return the ``user_info`` objects carried by one search response, or ``[]``."""
+ if not isinstance(body, dict):
+ return []
+ user_list = body.get("user_list")
+ if not isinstance(user_list, list):
+ return []
+ return [
+ entry["user_info"]
+ for entry in user_list
+ if isinstance(entry, dict) and isinstance(entry.get("user_info"), dict)
+ ]
+
+
+def _avatar(user_info: dict[str, Any]) -> str | None:
+ thumb = user_info.get("avatar_thumb")
+ if isinstance(thumb, dict):
+ urls = thumb.get("url_list")
+ if isinstance(urls, list) and urls:
+ return urls[0]
+ return None
+
+
+def parse_search_user(user_info: dict[str, Any]) -> dict[str, Any]:
+ """Map a search ``user_info`` to a :class:`TikTokProfileItem` output dict."""
+ from ..schemas.items import TikTokProfileItem
+
+ username = user_info.get("unique_id")
+ return TikTokProfileItem(
+ id=user_info.get("uid"),
+ name=username,
+ nickName=user_info.get("nickname"),
+ profileUrl=_PROFILE_URL.format(username=username) if username else None,
+ verified=bool(user_info.get("enterprise_verify_reason")),
+ signature=user_info.get("signature"),
+ avatar=_avatar(user_info),
+ fans=user_info.get("follower_count"),
+ heart=user_info.get("total_favorited"),
+ secUid=user_info.get("sec_uid"),
+ ).to_output()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/extraction/video.py b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/video.py
new file mode 100644
index 000000000..1bfe66452
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/extraction/video.py
@@ -0,0 +1,73 @@
+"""Normalize a raw TikTok ``itemStruct`` into a :class:`TikTokVideoItem` dict."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from ..schemas.items import TikTokVideoItem
+from .author import build_author_meta
+from .timestamps import epoch_to_iso
+
+_VIDEO_URL = "https://www.tiktok.com/@{username}/video/{video_id}"
+
+
+def _music_meta(music: dict[str, Any]) -> dict[str, Any]:
+ return {
+ "musicId": music.get("id"),
+ "musicName": music.get("title"),
+ "musicAuthor": music.get("authorName"),
+ "musicOriginal": music.get("original"),
+ "playUrl": music.get("playUrl"),
+ }
+
+
+def _video_meta(video: dict[str, Any]) -> dict[str, Any]:
+ return {
+ "height": video.get("height"),
+ "width": video.get("width"),
+ "duration": video.get("duration"),
+ "coverUrl": video.get("cover"),
+ "format": video.get("format"),
+ "definition": video.get("definition"),
+ }
+
+
+def _hashtags(challenges: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ return [
+ {"id": c.get("id"), "name": c.get("title")}
+ for c in challenges
+ if isinstance(c, dict)
+ ]
+
+
+def parse_video(item: dict[str, Any]) -> dict[str, Any]:
+ """Map an ``itemStruct`` to the flat output contract, filling known fields."""
+ author = item.get("author") or {}
+ author_stats = item.get("authorStats") or {}
+ stats = item.get("stats") or {}
+ username = author.get("uniqueId")
+ video_id = item.get("id")
+
+ web_url = (
+ _VIDEO_URL.format(username=username, video_id=video_id)
+ if username and video_id
+ else None
+ )
+ create_time = item.get("createTime")
+
+ return TikTokVideoItem(
+ id=video_id,
+ text=item.get("desc"),
+ createTime=create_time,
+ createTimeISO=epoch_to_iso(create_time),
+ authorMeta=build_author_meta(author, author_stats),
+ musicMeta=_music_meta(item.get("music") or {}),
+ videoMeta=_video_meta(item.get("video") or {}),
+ webVideoUrl=web_url,
+ diggCount=stats.get("diggCount"),
+ shareCount=stats.get("shareCount"),
+ playCount=stats.get("playCount"),
+ collectCount=stats.get("collectCount"),
+ commentCount=stats.get("commentCount"),
+ hashtags=_hashtags(item.get("challenges") or []),
+ ).to_output()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/flows/__init__.py b/surfsense_backend/app/proprietary/platforms/tiktok/flows/__init__.py
new file mode 100644
index 000000000..a7e38eba1
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/flows/__init__.py
@@ -0,0 +1,19 @@
+"""Per-target scrape flows: resolved target -> normalized items."""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator, Awaitable, Callable
+
+FetchFn = Callable[[str], Awaitable[str | None]]
+"""Fetch a page's HTML by URL (blob-first video flow)."""
+
+FetchListingFn = Callable[[str, int], Awaitable[list[dict]]]
+"""Load a listing page and return up to ``count`` captured itemStructs."""
+
+FetchUsersFn = Callable[[str, int], Awaitable[list[dict]]]
+"""Load a user-search page and return up to ``count`` captured ``user_info`` records."""
+
+FetchCommentsFn = Callable[[str, int], Awaitable[list[dict]]]
+"""Load a video page and return up to ``count`` captured raw comment records."""
+
+FlowResult = AsyncIterator[dict]
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/flows/comments.py b/surfsense_backend/app/proprietary/platforms/tiktok/flows/comments.py
new file mode 100644
index 000000000..28a7772a2
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/flows/comments.py
@@ -0,0 +1,52 @@
+"""Comments flow: a video URL -> its public comment thread.
+
+Comments load over a signed ``/api/comment/list`` XHR that TikTok *does* serve to
+anonymous sessions once the comments panel is opened (unlike profile-video/search
+feeds). Records are deduped by comment id, capped, and — when a video has none or
+withholds them — degraded to one ErrorItem, mirroring the listing flow.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from typing import Any
+
+from ..extraction import parse_comment
+from ..extraction.timestamps import now_iso
+from ..schemas import ErrorItem
+from ..targets.types import TikTokTarget
+from . import FetchCommentsFn
+
+_EMPTY_MESSAGE = (
+ "No comments returned. The video may have none, comments disabled, or TikTok "
+ "withheld them from anonymous access."
+)
+
+
+async def iter_comments(
+ target: TikTokTarget, *, cap: int, fetch_comments: FetchCommentsFn
+) -> AsyncIterator[dict[str, Any]]:
+ if cap <= 0:
+ return
+ seen: set[str] = set()
+ emitted = 0
+ for raw in await fetch_comments(target.url, cap):
+ out = parse_comment(raw, target.url)
+ cid = out.get("id")
+ if cid is not None:
+ if cid in seen:
+ continue
+ seen.add(cid)
+ out["scrapedAt"] = now_iso()
+ yield out
+ emitted += 1
+ if emitted >= cap:
+ return
+ if emitted == 0:
+ yield ErrorItem(
+ url=target.url,
+ input=target.value,
+ error=_EMPTY_MESSAGE,
+ errorCode="no_comments",
+ scrapedAt=now_iso(),
+ ).to_output()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/flows/listing.py b/surfsense_backend/app/proprietary/platforms/tiktok/flows/listing.py
new file mode 100644
index 000000000..3e930271f
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/flows/listing.py
@@ -0,0 +1,55 @@
+"""Listing flow shared by profile, hashtag, and search targets.
+
+The browser seam returns raw itemStructs captured from the signed ``item_list``
+XHRs; this maps each to the output contract, drops duplicate video ids, and
+stops at the per-target ``cap``.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from typing import Any
+
+from ..extraction import parse_video
+from ..extraction.timestamps import now_iso
+from ..schemas import ErrorItem
+from ..targets.types import TikTokTarget
+from . import FetchListingFn
+
+# Profile and search feeds are trust-gated: an anonymous headless session gets an
+# empty body (profile) or no results XHR (search), while hashtag feeds load. We
+# can't tell "genuinely empty" from "blocked" here, so a zero-item listing emits
+# one honest ErrorItem instead of vanishing silently.
+_EMPTY_LISTING_MESSAGE = (
+ "No videos returned. The target may be empty/private/removed, or TikTok "
+ "withheld this feed from anonymous access (common for profiles and search)."
+)
+
+
+async def iter_listing(
+ target: TikTokTarget, *, cap: int, fetch_listing: FetchListingFn
+) -> AsyncIterator[dict[str, Any]]:
+ if cap <= 0:
+ return
+ seen: set[str] = set()
+ emitted = 0
+ for item in await fetch_listing(target.url, cap):
+ out = parse_video(item)
+ video_id = out.get("id")
+ if video_id is not None:
+ if video_id in seen:
+ continue
+ seen.add(video_id)
+ out["scrapedAt"] = now_iso()
+ yield out
+ emitted += 1
+ if emitted >= cap:
+ return
+ if emitted == 0:
+ yield ErrorItem(
+ url=target.url,
+ input=target.value,
+ error=_EMPTY_LISTING_MESSAGE,
+ errorCode="no_items",
+ scrapedAt=now_iso(),
+ ).to_output()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/flows/profile.py b/surfsense_backend/app/proprietary/platforms/tiktok/flows/profile.py
new file mode 100644
index 000000000..feebd3c0d
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/flows/profile.py
@@ -0,0 +1,37 @@
+"""Profile flow: reliable blob metadata first, then the (gated) video listing.
+
+A profile's account data (name, followers, bio, verification) lives in the page's
+rehydration blob and loads over plain HTTP without a signed request, so we emit it
+first and always. The video listing needs a signed ``item_list`` XHR that TikTok
+withholds from anonymous sessions, so it is best-effort: it streams videos when it
+loads and degrades to an ErrorItem (via :func:`iter_listing`) when withheld. The
+metadata item therefore survives even when the videos are blocked.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from typing import Any
+
+from ..extraction import extract_rehydration_data, parse_profile, user_info
+from ..extraction.timestamps import now_iso
+from ..targets.types import TikTokTarget
+from . import FetchFn, FetchListingFn
+from .listing import iter_listing
+
+
+async def iter_profile(
+ target: TikTokTarget,
+ *,
+ cap: int,
+ fetch: FetchFn,
+ fetch_listing: FetchListingFn,
+) -> AsyncIterator[dict[str, Any]]:
+ html = await fetch(target.url)
+ info = user_info(extract_rehydration_data(html) or {}) if html else None
+ if info:
+ item = parse_profile(info)
+ item["scrapedAt"] = now_iso()
+ yield item
+ async for out in iter_listing(target, cap=cap, fetch_listing=fetch_listing):
+ yield out
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/flows/user_search.py b/surfsense_backend/app/proprietary/platforms/tiktok/flows/user_search.py
new file mode 100644
index 000000000..872212e80
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/flows/user_search.py
@@ -0,0 +1,54 @@
+"""User-search flow: keyword -> public account records.
+
+Unlike video/general search (login-walled for anonymous sessions), the Users tab
+hits ``/api/search/user`` and returns account records without a redirect. Each
+query's results are deduped by uid, capped, and — when a query returns nothing —
+degraded to one ErrorItem, mirroring the listing flow.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from typing import Any
+from urllib.parse import quote
+
+from ..extraction import parse_search_user
+from ..extraction.timestamps import now_iso
+from ..schemas import ErrorItem
+from . import FetchUsersFn
+
+_USER_SEARCH_URL = "https://www.tiktok.com/search/user?q={query}"
+_EMPTY_MESSAGE = (
+ "No accounts returned for this query. It may have no matches, or TikTok "
+ "withheld the results from anonymous access."
+)
+
+
+async def iter_user_search(
+ query: str, *, cap: int, fetch_users: FetchUsersFn
+) -> AsyncIterator[dict[str, Any]]:
+ if cap <= 0:
+ return
+ url = _USER_SEARCH_URL.format(query=quote(query))
+ seen: set[str] = set()
+ emitted = 0
+ for user_info in await fetch_users(url, cap):
+ out = parse_search_user(user_info)
+ uid = out.get("id")
+ if uid is not None:
+ if uid in seen:
+ continue
+ seen.add(uid)
+ out["scrapedAt"] = now_iso()
+ yield out
+ emitted += 1
+ if emitted >= cap:
+ return
+ if emitted == 0:
+ yield ErrorItem(
+ url=url,
+ input=query,
+ error=_EMPTY_MESSAGE,
+ errorCode="no_users",
+ scrapedAt=now_iso(),
+ ).to_output()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/flows/video.py b/surfsense_backend/app/proprietary/platforms/tiktok/flows/video.py
new file mode 100644
index 000000000..0cdefe1c4
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/flows/video.py
@@ -0,0 +1,28 @@
+"""Video-URL flow: fetch a post page, read its rehydration blob, emit one item."""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from typing import Any
+
+from ..extraction import extract_rehydration_data, parse_video, video_item_struct
+from ..extraction.timestamps import now_iso
+from ..targets.types import TikTokTarget
+from . import FetchFn
+
+
+async def iter_video(
+ target: TikTokTarget, *, fetch: FetchFn
+) -> AsyncIterator[dict[str, Any]]:
+ html = await fetch(target.url)
+ if not html:
+ return
+ data = extract_rehydration_data(html)
+ if not data:
+ return
+ item = video_item_struct(data)
+ if item is None:
+ return
+ out = parse_video(item)
+ out["scrapedAt"] = now_iso()
+ yield out
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/orchestrator.py b/surfsense_backend/app/proprietary/platforms/tiktok/orchestrator.py
new file mode 100644
index 000000000..a738233a6
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/orchestrator.py
@@ -0,0 +1,196 @@
+"""Resolve a :class:`TikTokScrapeInput` into targets and stream their items.
+
+Targets run sequentially on one warm sticky IP; ``limit`` is collector policy
+applied by :func:`scrape_tiktok`, never baked into a flow. Each kind routes to
+its flow via :func:`_dispatch`: video URLs read the rehydration blob over HTTP,
+listings capture signed item_list XHRs through the stealth browser.
+"""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from typing import Any
+
+from .extraction.timestamps import now_iso
+from .flows import FetchCommentsFn, FetchFn, FetchListingFn, FetchUsersFn
+from .flows.comments import iter_comments
+from .flows.listing import iter_listing
+from .flows.profile import iter_profile
+from .flows.user_search import iter_user_search
+from .flows.video import iter_video
+from .schemas import ErrorItem, TikTokScrapeInput
+from .session import (
+ fetch_comments,
+ fetch_html,
+ fetch_item_list,
+ fetch_trending,
+ fetch_user_search,
+)
+from .targets import resolve_target
+from .targets.types import TikTokTarget
+
+_PROFILE_URL = "https://www.tiktok.com/@{name}"
+_HASHTAG_URL = "https://www.tiktok.com/tag/{tag}"
+_EXPLORE_URL = "https://www.tiktok.com/explore"
+
+
+def _resolve_targets(input_model: TikTokScrapeInput) -> list[TikTokTarget]:
+ """Build the target list from the URL/profile/hashtag sources.
+
+ A raw ``tiktok.com/search?...`` URL passed explicitly in
+ ``startUrls``/``postURLs`` still resolves here and keeps its native listing
+ routing; there is no keyword-search shortcut.
+ """
+ targets: list[TikTokTarget] = []
+ for entry in input_model.startUrls:
+ resolved = resolve_target(entry.url)
+ if resolved is not None:
+ targets.append(resolved)
+ for url in input_model.postURLs:
+ resolved = resolve_target(url)
+ if resolved is not None:
+ targets.append(resolved)
+ for profile in input_model.profiles:
+ name = profile.lstrip("@")
+ targets.append(TikTokTarget("profile", name, _PROFILE_URL.format(name=name)))
+ for tag in input_model.hashtags:
+ targets.append(TikTokTarget("hashtag", tag, _HASHTAG_URL.format(tag=tag)))
+ return targets
+
+
+def _dispatch(
+ target: TikTokTarget,
+ *,
+ cap: int,
+ fetch: FetchFn,
+ fetch_listing: FetchListingFn,
+) -> AsyncIterator[dict[str, Any]]:
+ if target.kind == "video":
+ return iter_video(target, fetch=fetch)
+ if target.kind == "profile":
+ return iter_profile(target, cap=cap, fetch=fetch, fetch_listing=fetch_listing)
+ return iter_listing(target, cap=cap, fetch_listing=fetch_listing)
+
+
+async def iter_tiktok(
+ input_model: TikTokScrapeInput,
+ *,
+ fetch: FetchFn = fetch_html,
+ fetch_listing: FetchListingFn = fetch_item_list,
+) -> AsyncIterator[dict[str, Any]]:
+ """Yield normalized items for every resolved target, in order.
+
+ The video flow's ``fetch_html`` opens its own warmed proxy session per call
+ when none is bound; the listing flow drives its own browser. Neither binds a
+ ContextVar across these ``yield``s, so the generator stays context-safe.
+ """
+ cap = input_model.resultsPerPage
+ for target in _resolve_targets(input_model):
+ async for item in _dispatch(
+ target, cap=cap, fetch=fetch, fetch_listing=fetch_listing
+ ):
+ yield item
+
+
+async def scrape_tiktok(
+ input_model: TikTokScrapeInput,
+ *,
+ limit: int | None = None,
+ fetch: FetchFn = fetch_html,
+ fetch_listing: FetchListingFn = fetch_item_list,
+) -> list[dict[str, Any]]:
+ """Collect :func:`iter_tiktok` into a list, honoring an optional ``limit``."""
+ from app.capabilities.core.progress import emit_progress
+
+ results: list[dict[str, Any]] = []
+ async for item in iter_tiktok(
+ input_model, fetch=fetch, fetch_listing=fetch_listing
+ ):
+ results.append(item)
+ emit_progress("scraping", current=len(results), total=limit, unit="item")
+ if limit is not None and len(results) >= limit:
+ break
+ return results
+
+
+async def search_tiktok_users(
+ queries: list[str],
+ *,
+ per_query: int,
+ limit: int | None = None,
+ fetch_users: FetchUsersFn = fetch_user_search,
+) -> list[dict[str, Any]]:
+ """Collect user-search account records across queries, honoring ``limit``."""
+ from app.capabilities.core.progress import emit_progress
+
+ results: list[dict[str, Any]] = []
+ for query in queries:
+ async for item in iter_user_search(
+ query, cap=per_query, fetch_users=fetch_users
+ ):
+ results.append(item)
+ emit_progress("searching", current=len(results), total=limit, unit="item")
+ if limit is not None and len(results) >= limit:
+ return results
+ return results
+
+
+async def scrape_tiktok_trending(
+ *,
+ count: int,
+ fetch_trending_fn: FetchListingFn = fetch_trending,
+) -> list[dict[str, Any]]:
+ """Collect up to ``count`` trending videos from the Explore feed.
+
+ A single global feed, so it reuses the listing flow (parse/dedupe/cap/empty-
+ ErrorItem) over a synthetic target — no user input to resolve.
+ """
+ from app.capabilities.core.progress import emit_progress
+
+ target = TikTokTarget(kind="trending", value="explore", url=_EXPLORE_URL)
+ results: list[dict[str, Any]] = []
+ async for item in iter_listing(target, cap=count, fetch_listing=fetch_trending_fn):
+ results.append(item)
+ emit_progress("scraping", current=len(results), total=count, unit="item")
+ return results
+
+
+async def scrape_tiktok_comments(
+ video_urls: list[str],
+ *,
+ per_video: int,
+ limit: int | None = None,
+ fetch_comments_fn: FetchCommentsFn = fetch_comments,
+) -> list[dict[str, Any]]:
+ """Collect comments across video URLs, honoring ``limit``.
+
+ A non-video URL yields one ``bad_url`` ErrorItem (never a silent drop) so the
+ caller can tell "wrong input" from "video has no comments".
+ """
+ from app.capabilities.core.progress import emit_progress
+
+ results: list[dict[str, Any]] = []
+ for url in video_urls:
+ target = resolve_target(url)
+ if target is None or target.kind != "video":
+ results.append(
+ ErrorItem(
+ url=url,
+ input=url,
+ error="Not a TikTok video URL.",
+ errorCode="bad_url",
+ scrapedAt=now_iso(),
+ ).to_output()
+ )
+ emit_progress("scraping", current=len(results), total=limit, unit="item")
+ if limit is not None and len(results) >= limit:
+ return results
+ continue
+ async for item in iter_comments(
+ target, cap=per_video, fetch_comments=fetch_comments_fn
+ ):
+ results.append(item)
+ emit_progress("scraping", current=len(results), total=limit, unit="item")
+ if limit is not None and len(results) >= limit:
+ return results
+ return results
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/schemas/__init__.py b/surfsense_backend/app/proprietary/platforms/tiktok/schemas/__init__.py
new file mode 100644
index 000000000..a52400a58
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/schemas/__init__.py
@@ -0,0 +1,26 @@
+"""Apify-compatible input and output contracts for the TikTok scraper."""
+
+from __future__ import annotations
+
+from .input import StartUrl, TikTokScrapeInput
+from .items import (
+ AuthorMeta,
+ CommentItem,
+ ErrorItem,
+ MusicMeta,
+ TikTokProfileItem,
+ TikTokVideoItem,
+ VideoMeta,
+)
+
+__all__ = [
+ "AuthorMeta",
+ "CommentItem",
+ "ErrorItem",
+ "MusicMeta",
+ "StartUrl",
+ "TikTokProfileItem",
+ "TikTokScrapeInput",
+ "TikTokVideoItem",
+ "VideoMeta",
+]
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/schemas/input.py b/surfsense_backend/app/proprietary/platforms/tiktok/schemas/input.py
new file mode 100644
index 000000000..f710eed72
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/schemas/input.py
@@ -0,0 +1,59 @@
+# ruff: noqa: N815 - field names mirror the public camelCase TikTok/Apify API
+"""Input surface for the TikTok scraper, shaped to the Clockworks actor.
+
+Anonymous only: no auth-shaped field exists here. ``extra="allow"`` keeps the
+contract in parity with the actor for fields the scraper does not read.
+
+``resultsPerPage`` is a per-target count; the cross-target ceiling is caller
+policy applied by the collector, never baked into a flow.
+"""
+
+from __future__ import annotations
+
+from typing import Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+ProfileSorting = Literal["latest", "popular", "oldest"]
+ProfileSection = Literal["videos", "reposts"]
+SearchSection = Literal["", "/video", "/user"]
+
+
+class StartUrl(BaseModel):
+ """A single direct URL entry (``{"url": ...}``; extra keys ignored)."""
+
+ model_config = ConfigDict(extra="allow")
+
+ url: str
+
+
+class TikTokScrapeInput(BaseModel):
+ model_config = ConfigDict(extra="allow")
+
+ # Discovery
+ startUrls: list[StartUrl] = Field(default_factory=list)
+ hashtags: list[str] = Field(default_factory=list)
+ profiles: list[str] = Field(default_factory=list)
+ searchQueries: list[str] = Field(default_factory=list)
+ postURLs: list[str] = Field(default_factory=list)
+
+ # Per-target count
+ resultsPerPage: int = Field(default=1, ge=1)
+
+ # Profile options
+ profileScrapeSections: list[ProfileSection] = Field(
+ default_factory=lambda: ["videos"]
+ )
+ profileSorting: ProfileSorting = "latest"
+ excludePinnedPosts: bool = False
+
+ # Search options
+ searchSection: SearchSection = ""
+ maxProfilesPerQuery: int = Field(default=10, ge=1)
+
+ # Incremental filters (ISO date or relative " days" per the actor)
+ oldestPostDateUnified: str | None = None
+ newestPostDate: str | None = None
+
+ # Proxy
+ proxyCountryCode: str = "None"
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/schemas/items.py b/surfsense_backend/app/proprietary/platforms/tiktok/schemas/items.py
new file mode 100644
index 000000000..50a5c4293
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/schemas/items.py
@@ -0,0 +1,147 @@
+# ruff: noqa: N815 - field names mirror the public camelCase TikTok/Apify API
+"""Output items keyed to the Clockworks TikTok actor's shape.
+
+Every model is open (``extra="allow"``) and defaults unsourced fields to
+``None``/``[]`` so the MVP can populate a reliable subset and expand the
+contract additively without breaking consumers.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict, Field
+
+
+class AuthorMeta(BaseModel):
+ model_config = ConfigDict(extra="allow")
+
+ id: str | None = None
+ name: str | None = None
+ nickName: str | None = None
+ profileUrl: str | None = None
+ verified: bool | None = None
+ signature: str | None = None
+ avatar: str | None = None
+ privateAccount: bool | None = None
+ fans: int | None = None
+ following: int | None = None
+ heart: int | None = None
+ video: int | None = None
+
+
+class TikTokProfileItem(AuthorMeta):
+ """A profile's public metadata, read from the page's rehydration blob.
+
+ Emitted even when the video listing is withheld from anonymous access, so a
+ blocked profile still yields its account data (name, followers, bio,
+ verification) instead of only an ErrorItem. Distinguishable from a video item
+ by the absence of ``webVideoUrl``/``text``.
+ """
+
+ scrapedAt: str | None = None
+
+ def to_output(self) -> dict[str, Any]:
+ return self.model_dump(exclude_none=False)
+
+
+class MusicMeta(BaseModel):
+ model_config = ConfigDict(extra="allow")
+
+ musicId: str | None = None
+ musicName: str | None = None
+ musicAuthor: str | None = None
+ musicOriginal: bool | None = None
+ playUrl: str | None = None
+
+
+class VideoMeta(BaseModel):
+ model_config = ConfigDict(extra="allow")
+
+ height: int | None = None
+ width: int | None = None
+ duration: int | None = None
+ coverUrl: str | None = None
+ format: str | None = None
+ definition: str | None = None
+
+
+class TikTokVideoItem(BaseModel):
+ """A single scraped post (video or slideshow)."""
+
+ model_config = ConfigDict(extra="allow")
+
+ id: str | None = None
+ text: str | None = None
+ textLanguage: str | None = None
+ createTime: int | None = None
+ createTimeISO: str | None = None
+ isAd: bool | None = None
+
+ authorMeta: AuthorMeta = Field(default_factory=AuthorMeta)
+ musicMeta: MusicMeta = Field(default_factory=MusicMeta)
+ videoMeta: VideoMeta = Field(default_factory=VideoMeta)
+
+ webVideoUrl: str | None = None
+ mediaUrls: list[str] = Field(default_factory=list)
+
+ diggCount: int | None = None
+ shareCount: int | None = None
+ playCount: int | None = None
+ collectCount: int | None = None
+ commentCount: int | None = None
+
+ hashtags: list[dict[str, Any]] = Field(default_factory=list)
+ mentions: list[str] = Field(default_factory=list)
+
+ isSlideshow: bool | None = None
+ isPinned: bool | None = None
+ isSponsored: bool | None = None
+
+ scrapedAt: str | None = None
+
+ def to_output(self) -> dict[str, Any]:
+ return self.model_dump(exclude_none=False)
+
+
+class CommentItem(BaseModel):
+ """A single comment or reply under a post."""
+
+ model_config = ConfigDict(extra="allow")
+
+ id: str | None = None
+ text: str | None = None
+ videoWebUrl: str | None = None
+ diggCount: int | None = None
+ replyCommentTotal: int | None = None
+ createTime: int | None = None
+ createTimeISO: str | None = None
+
+ uid: str | None = None
+ uniqueId: str | None = None
+ nickName: str | None = None
+ avatar: str | None = None
+
+ repliesToId: str | None = None
+ scrapedAt: str | None = None
+
+ def to_output(self) -> dict[str, Any]:
+ return self.model_dump(exclude_none=False)
+
+
+class ErrorItem(BaseModel):
+ """Per-input failure, distinguished from normal items by ``errorCode``.
+
+ Mirrors the actor's convention so a private/deleted/empty target surfaces
+ as an item instead of silently vanishing from the results.
+ """
+
+ model_config = ConfigDict(extra="allow")
+
+ url: str | None = None
+ input: str | None = None
+ error: str | None = None
+ errorCode: str | None = None
+
+ def to_output(self) -> dict[str, Any]:
+ return self.model_dump(exclude_none=False)
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/session/__init__.py b/surfsense_backend/app/proprietary/platforms/tiktok/session/__init__.py
new file mode 100644
index 000000000..6fa1c1b45
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/session/__init__.py
@@ -0,0 +1,25 @@
+"""Cookie-warmed, rotate-on-block proxy session and page-fetch seam."""
+
+from __future__ import annotations
+
+from .client import fetch_html
+from .errors import TikTokAccessBlockedError
+from .listing import (
+ fetch_comments,
+ fetch_item_list,
+ fetch_trending,
+ fetch_user_search,
+)
+from .proxy import bind_proxy_holder, open_proxy_holder, proxy_session
+
+__all__ = [
+ "TikTokAccessBlockedError",
+ "bind_proxy_holder",
+ "fetch_comments",
+ "fetch_html",
+ "fetch_item_list",
+ "fetch_trending",
+ "fetch_user_search",
+ "open_proxy_holder",
+ "proxy_session",
+]
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/session/client.py b/surfsense_backend/app/proprietary/platforms/tiktok/session/client.py
new file mode 100644
index 000000000..e5be1e419
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/session/client.py
@@ -0,0 +1,130 @@
+"""GET TikTok page HTML through a cookie-warmed, sticky-IP proxy session.
+
+The warm-up mints TikTok's anonymous device cookie (``ttwid``) on the first
+homepage hit; the target page then server-renders its rehydration blob. Rotates
+the residential IP and re-warms on 403, backs off on 429, and raises
+:class:`TikTokAccessBlockedError` only when every rotated IP refuses access.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import random
+from contextlib import suppress
+from typing import Any
+
+from scrapling.fetchers import AsyncFetcher
+
+from app.utils.proxy import get_proxy_url
+
+from .errors import TikTokAccessBlockedError
+from .proxy import _REQUEST_TIMEOUT_S, _current_session, proxy_session
+
+logger = logging.getLogger(__name__)
+
+# 403 => IP blocked; rotate and re-warm. 429 => rate limited; back off same IP.
+_ROTATE_STATUS = 403
+_BACKOFF_STATUS = 429
+_MAX_ROTATIONS = 3
+_MAX_BACKOFFS = 4
+_BACKOFF_BASE_S = 5.0
+
+_HOME_URL = "https://www.tiktok.com/"
+_TTWID_COOKIE = "ttwid"
+_HEADERS = {"Accept-Language": "en-US,en;q=0.9"}
+
+
+def _response_cookie_names(page: Any) -> set[str]:
+ cookies = getattr(page, "cookies", None)
+ return set(cookies.keys()) if isinstance(cookies, dict) else set()
+
+
+def _page_html(page: Any) -> str | None:
+ for attr in ("text", "body"):
+ val = getattr(page, attr, None)
+ if isinstance(val, bytes):
+ val = val.decode("utf-8", "replace")
+ if isinstance(val, str) and val.strip():
+ return val
+ return None
+
+
+async def warm_session(session: Any) -> bool:
+ """Mint an anonymous ``ttwid`` cookie; ``True`` if the session can now fetch."""
+ with suppress(Exception):
+ page = await session.get(_HOME_URL, headers=_HEADERS)
+ if _TTWID_COOKIE in _response_cookie_names(page):
+ return True
+ return False
+
+
+async def _get_page(session: Any, url: str) -> Any:
+ if session is not None:
+ return await session.get(url, headers=_HEADERS)
+ return await AsyncFetcher.get(
+ url,
+ headers=_HEADERS,
+ proxy=get_proxy_url(),
+ stealthy_headers=True,
+ timeout=_REQUEST_TIMEOUT_S,
+ )
+
+
+async def fetch_html(url: str) -> str | None:
+ """Return page HTML, or ``None`` on 404 / non-block failure."""
+ holder = _current_session.get()
+ if holder is None:
+ async with proxy_session():
+ return await fetch_html(url)
+
+ attempt = 0
+ backoffs = 0
+ while True:
+ session = holder.session
+ try:
+ if session is not None and not holder.warmed:
+ warmed_ok = await warm_session(session)
+ holder.warmed = True
+ if not warmed_ok:
+ if attempt < _MAX_ROTATIONS:
+ attempt += 1
+ await holder.rotate()
+ continue
+ raise TikTokAccessBlockedError(
+ f"could not warm session after {attempt} IP rotations: {url}"
+ )
+
+ await holder.pace()
+ page = await _get_page(session, url)
+ status = page.status
+
+ if status == 200:
+ return _page_html(page)
+ if status == 404:
+ return None
+ if status == _BACKOFF_STATUS and backoffs < _MAX_BACKOFFS:
+ backoffs += 1
+ delay = _BACKOFF_BASE_S * (2 ** (backoffs - 1))
+ logger.warning("[tiktok] 429 on %s; backing off %.1fs", url, delay)
+ await asyncio.sleep(delay + random.uniform(0, 1))
+ continue
+ if status == _ROTATE_STATUS and attempt < _MAX_ROTATIONS:
+ attempt += 1
+ await holder.rotate()
+ continue
+ if status == _ROTATE_STATUS:
+ raise TikTokAccessBlockedError(
+ f"TikTok refused {url} on {attempt} rotated IPs (403)"
+ )
+ logger.warning("[tiktok] GET %s returned %s", url, status)
+ return None
+ except TikTokAccessBlockedError:
+ raise
+ except Exception as e:
+ logger.warning("[tiktok] GET %s failed: %s", url, e)
+ if attempt < _MAX_ROTATIONS:
+ attempt += 1
+ await holder.rotate()
+ continue
+ return None
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/session/errors.py b/surfsense_backend/app/proprietary/platforms/tiktok/session/errors.py
new file mode 100644
index 000000000..708cb0212
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/session/errors.py
@@ -0,0 +1,10 @@
+"""Fetch-seam errors surfaced to the capability layer."""
+
+from __future__ import annotations
+
+
+class TikTokAccessBlockedError(RuntimeError):
+ """Raised when every rotated IP is refused anonymous access.
+
+ Distinguishes a hard block from an empty result; the route maps it to 403.
+ """
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/session/listing.py b/surfsense_backend/app/proprietary/platforms/tiktok/session/listing.py
new file mode 100644
index 000000000..bc8af56ff
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/session/listing.py
@@ -0,0 +1,314 @@
+"""Browser-driven listing fetch: let TikTok sign its own ``item_list`` XHRs.
+
+Profile/hashtag/search listings need signed requests (``X-Gnarly``) whose
+algorithm rev's monthly and reads a browser canvas fingerprint. Rather than port
+and chase that signer, we load the page in the stealth browser we already run
+(patchright-Chromium, via the web-crawler tier) and capture the itemStruct JSON
+the page's own scripts fetch while scrolling. The browser is the client, so it
+signs correctly for whatever version TikTok ships.
+
+The pure response-shape parsing lives in :func:`items_from_response`; this module
+is the untested browser-I/O glue (covered by the e2e smoke, not unit tests).
+
+Needs a residential proxy; datacenter IPs get empty bodies and 429s. The profile
+feed returns an empty 200 to headless sessions, so :func:`fetch_item_list` goes
+headful only when ``CRAWL_HEADED_XVFB_ENABLED`` promises an Xvfb display — else it
+stays headless and degrades to an ``ErrorItem`` instead of crashing on launch.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from collections.abc import Callable
+from typing import Any
+
+from scrapling.fetchers import StealthyFetcher
+
+from app.config import config
+from app.proprietary.web_crawler.stealth import (
+ build_stealthy_kwargs,
+ get_stealth_config,
+)
+from app.utils.proxy import get_proxy_url
+
+from ..extraction import (
+ comments_from_response,
+ items_from_response,
+ users_from_response,
+)
+
+logger = logging.getLogger(__name__)
+
+ExtractFn = Callable[[Any], list[dict[str, Any]]]
+# Drives the page after navigation to trigger/paginate the target XHRs, filling
+# ``collected`` until it reaches ``target_count`` (or the interaction gives up).
+InteractFn = Callable[[Any, list[dict[str, Any]], int], None]
+
+# XHR paths that carry itemStructs for the three listing kinds.
+_ITEM_LIST_MARKERS = (
+ "/api/post/item_list",
+ "/api/challenge/item_list",
+ "/api/search/",
+)
+# The user-search XHR carries account records (user_list), not itemStructs.
+_USER_SEARCH_MARKERS = ("/api/search/user",)
+# The Explore feed's trending videos arrive as ordinary itemStructs.
+_EXPLORE_MARKERS = ("/api/explore/item_list",)
+# The comment feed fires only after the comments panel is opened.
+_COMMENT_MARKERS = ("/api/comment/list",)
+_COMMENT_ICON_SELECTORS = (
+ '[data-e2e="comment-icon"]',
+ '[data-e2e="browse-comment"]',
+)
+# The comment icon hydrates a beat after DOM-ready; wait for it before clicking.
+_COMMENT_ICON_WAIT_MS = 8000
+# First comment page lands shortly after the click — don't declare "empty" early.
+_COMMENT_FIRST_PAGE_MS = 3500
+_HOME_URL = "https://www.tiktok.com/"
+_MSTOKEN_COOKIE = "msToken"
+# Bounded scroll: a dead page can't loop forever, and a live one stops early
+# once enough items are captured.
+_SCROLL_MAX_ROUNDS = 20
+_SCROLL_SETTLE_MS = 1500
+# Warm-up poll for the anonymous msToken cookie the item_list API requires.
+_WARM_POLLS = 8
+_WARM_POLL_MS = 500
+
+
+def _has_mstoken(page: Any) -> bool:
+ try:
+ return any(c.get("name") == _MSTOKEN_COOKIE for c in page.context.cookies())
+ except Exception:
+ return False
+
+
+def _dismiss_login_modal(page: Any) -> None:
+ """Close the login modal that blocks scrolling; Escape as fallback.
+
+ Only the modal's own close button, never a generic dialog button (avoids
+ clicking "Log in").
+ """
+ try:
+ closed = page.evaluate(
+ """() => {
+ const btn = document.querySelector('[data-e2e="modal-close-inner-button"]');
+ if (btn) { btn.click(); return true; }
+ return false;
+ }"""
+ )
+ if not closed:
+ page.keyboard.press("Escape")
+ except Exception:
+ pass
+
+
+def _scroll_page(page: Any, collected: list[dict[str, Any]], target_count: int) -> None:
+ """Page down a listing feed until enough items are captured or it stops growing."""
+ last_height = 0
+ for _ in range(_SCROLL_MAX_ROUNDS):
+ if len(collected) >= target_count:
+ break
+ _dismiss_login_modal(page)
+ page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
+ page.wait_for_timeout(_SCROLL_SETTLE_MS)
+ height = page.evaluate("document.body.scrollHeight")
+ if not height or height <= last_height:
+ break
+ last_height = height
+
+
+def _open_comments(page: Any) -> None:
+ """Click the comment icon so the first ``/api/comment/list`` XHR fires.
+
+ The icon must be present and interactive first (the SPA hydrates it a beat
+ after DOM-ready), so we wait for it, then fall back to a JS click if the
+ normal click is intercepted (cookie banner / overlay).
+ """
+ for selector in _COMMENT_ICON_SELECTORS:
+ try:
+ page.wait_for_selector(selector, timeout=_COMMENT_ICON_WAIT_MS)
+ except Exception:
+ continue
+ try:
+ page.click(selector, timeout=_COMMENT_ICON_WAIT_MS)
+ return
+ except Exception:
+ try:
+ page.eval_on_selector(selector, "el => el.click()")
+ return
+ except Exception:
+ continue
+
+
+def _scroll_comments(
+ page: Any, collected: list[dict[str, Any]], target_count: int
+) -> None:
+ """Open the comments panel, then scroll its last comment into view to paginate.
+
+ Comment XHRs fire only after the panel is opened, and paging must scroll the
+ panel (not the page, which would advance the video feed), so we anchor on the
+ last ``comment-level-1`` element. ponytail: naive scroll-to-last paging,
+ bounded by ``_SCROLL_MAX_ROUNDS``; upgrade to container-height tracking if
+ deep threads under-fetch.
+ """
+ _open_comments(page)
+ # The panel's first page lands a beat after the click; give it room before
+ # we decide there are no comments to page through.
+ page.wait_for_timeout(_COMMENT_FIRST_PAGE_MS)
+ for _ in range(_SCROLL_MAX_ROUNDS):
+ if len(collected) >= target_count:
+ break
+ moved = page.evaluate(
+ """() => {
+ const items = document.querySelectorAll('[data-e2e="comment-level-1"]');
+ if (!items.length) return false;
+ items[items.length - 1].scrollIntoView({block: 'end'});
+ return true;
+ }"""
+ )
+ page.wait_for_timeout(_SCROLL_SETTLE_MS)
+ if not moved:
+ break
+
+
+def _build_page_action(
+ collected: list[dict[str, Any]],
+ url: str,
+ target_count: int,
+ markers: tuple[str, ...],
+ extract: ExtractFn,
+ interact: InteractFn,
+):
+ """A sync ``page_action`` that warms the session then captures matching XHRs.
+
+ A cold context returns an empty body, so we first mint the anonymous
+ ``msToken`` (homepage hit), then navigate to the target with the listener
+ already attached so page-one fires into it; ``interact`` pages the rest.
+ ``markers``/``extract`` select which XHRs to keep and how to unwrap them.
+ """
+
+ def _on_response(response: Any) -> None:
+ response_url = getattr(response, "url", "")
+ if not any(marker in response_url for marker in markers):
+ return
+ try:
+ body = response.json()
+ except Exception:
+ # An empty 200 (TikTok soft-block) or a body evicted before read.
+ return
+ collected.extend(extract(body))
+
+ def _warm(page: Any) -> None:
+ if _has_mstoken(page):
+ return
+ page.goto(_HOME_URL, wait_until="domcontentloaded")
+ for _ in range(_WARM_POLLS):
+ page.wait_for_timeout(_WARM_POLL_MS)
+ if _has_mstoken(page):
+ break
+
+ def page_action(page: Any) -> Any:
+ page.on("response", _on_response)
+ try:
+ _warm(page)
+ # Navigate (back) to the target with the listener attached and a
+ # token in hand, so the page-one XHR fires into the capture.
+ page.goto(url, wait_until="domcontentloaded")
+ page.wait_for_timeout(_SCROLL_SETTLE_MS)
+ interact(page, collected, target_count)
+ except Exception as exc:
+ logger.debug("[tiktok] capture interaction aborted: %s", exc)
+ return page
+
+ return page_action
+
+
+def _fetch_sync(
+ url: str,
+ target_count: int,
+ markers: tuple[str, ...],
+ extract: ExtractFn,
+ interact: InteractFn,
+ *,
+ headless: bool = True,
+) -> list[dict[str, Any]]:
+ collected: list[dict[str, Any]] = []
+ kwargs = build_stealthy_kwargs(get_stealth_config())
+ StealthyFetcher.fetch(
+ url,
+ headless=headless,
+ network_idle=False,
+ proxy=get_proxy_url(),
+ page_action=_build_page_action(
+ collected, url, target_count, markers, extract, interact
+ ),
+ **kwargs,
+ )
+ return collected[:target_count]
+
+
+async def fetch_item_list(page_url: str, target_count: int) -> list[dict[str, Any]]:
+ """Return up to ``target_count`` itemStructs from a listing page's XHRs.
+
+ Headful when ``CRAWL_HEADED_XVFB_ENABLED`` promises a display (the profile feed
+ is empty to headless sessions); headless otherwise so launch never fails.
+ Retries an empty draw up to ``TIKTOK_LISTING_MAX_ATTEMPTS`` for a fresh exit IP.
+ """
+ headless = not config.CRAWL_HEADED_XVFB_ENABLED
+ attempts = max(1, config.TIKTOK_LISTING_MAX_ATTEMPTS)
+ for attempt in range(1, attempts + 1):
+ items = await asyncio.to_thread(
+ _fetch_sync,
+ page_url,
+ target_count,
+ _ITEM_LIST_MARKERS,
+ items_from_response,
+ _scroll_page,
+ headless=headless,
+ )
+ if items or attempt == attempts:
+ return items
+ logger.info(
+ "[tiktok] empty item_list for %s (attempt %d/%d); retrying on a fresh exit IP",
+ page_url,
+ attempt,
+ attempts,
+ )
+ return []
+
+
+async def fetch_user_search(page_url: str, target_count: int) -> list[dict[str, Any]]:
+ """Return up to ``target_count`` ``user_info`` records from a user-search page."""
+ return await asyncio.to_thread(
+ _fetch_sync,
+ page_url,
+ target_count,
+ _USER_SEARCH_MARKERS,
+ users_from_response,
+ _scroll_page,
+ )
+
+
+async def fetch_comments(page_url: str, target_count: int) -> list[dict[str, Any]]:
+ """Return up to ``target_count`` raw comment records from a video page's XHRs."""
+ return await asyncio.to_thread(
+ _fetch_sync,
+ page_url,
+ target_count,
+ _COMMENT_MARKERS,
+ comments_from_response,
+ _scroll_comments,
+ )
+
+
+async def fetch_trending(page_url: str, target_count: int) -> list[dict[str, Any]]:
+ """Return up to ``target_count`` trending itemStructs from the Explore feed."""
+ return await asyncio.to_thread(
+ _fetch_sync,
+ page_url,
+ target_count,
+ _EXPLORE_MARKERS,
+ items_from_response,
+ _scroll_page,
+ )
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/session/proxy.py b/surfsense_backend/app/proprietary/platforms/tiktok/session/proxy.py
new file mode 100644
index 000000000..dca6c776b
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/session/proxy.py
@@ -0,0 +1,113 @@
+"""Rotate-on-block sticky proxy session, bound per-flow via a ContextVar.
+
+Reusing one keep-alive connection pins a single residential exit IP so the
+warmed cookie jar (``ttwid``/``msToken``, bound to that IP) stays valid across
+the warm-up and every subsequent fetch. Ported from the Reddit sibling; the
+TikTok-specific warm-up lives in :mod:`client`.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import random
+import time
+from contextlib import asynccontextmanager, suppress
+from contextvars import ContextVar
+from typing import Any
+
+from scrapling.fetchers import FetcherSession
+
+from app.utils.proxy import get_proxy_url
+
+logger = logging.getLogger(__name__)
+
+# Pace each sticky IP so a fast exit can't burst past TikTok's per-IP threshold.
+_MIN_INTERVAL_S = 0.5
+_PACE_JITTER_S = 0.25
+# A healthy fetch lands in ~1-2s; cap a dead IP at one bounded wait before it
+# falls through to a rotation.
+_REQUEST_TIMEOUT_S = 15.0
+
+_current_session: ContextVar[_RotatingSession | None] = ContextVar(
+ "tiktok_proxy_session", default=None
+)
+
+
+class _RotatingSession:
+ """Owns one live ``FetcherSession`` (sticky IP); ``rotate()`` swaps the IP.
+
+ Used sequentially within a single flow (never shared across concurrent
+ tasks), so no locking is needed. ``session`` is ``None`` only when no proxy
+ is configured.
+ """
+
+ def __init__(self) -> None:
+ self._cm: Any | None = None
+ self.session: Any | None = None
+ self.rotations = 0
+ self.warmed = False
+ self._last_at = 0.0
+
+ async def _open(self) -> None:
+ proxy = get_proxy_url()
+ self.warmed = False
+ if proxy is None:
+ self._cm = self.session = None
+ return
+ self._cm = FetcherSession(
+ proxy=proxy,
+ stealthy_headers=True,
+ impersonate="chrome",
+ timeout=_REQUEST_TIMEOUT_S,
+ )
+ self.session = await self._cm.__aenter__()
+
+ async def close(self) -> None:
+ if self._cm is not None:
+ with suppress(Exception):
+ await self._cm.__aexit__(None, None, None)
+ self._cm = self.session = None
+
+ async def rotate(self) -> Any | None:
+ """Drop the current IP and connect through a fresh one."""
+ await self.close()
+ self.rotations += 1
+ await self._open()
+ logger.info("[tiktok] rotated proxy session (rotation #%d)", self.rotations)
+ return self.session
+
+ async def pace(self) -> None:
+ """Sleep to hold this sticky IP under TikTok's per-IP rate threshold."""
+ wait = _MIN_INTERVAL_S - (time.monotonic() - self._last_at)
+ if wait > 0:
+ await asyncio.sleep(wait + random.uniform(0, _PACE_JITTER_S))
+ self._last_at = time.monotonic()
+
+
+async def open_proxy_holder() -> _RotatingSession:
+ """Open a warm rotate-on-block session holder (caller owns ``close()``)."""
+ holder = _RotatingSession()
+ await holder._open()
+ return holder
+
+
+@asynccontextmanager
+async def bind_proxy_holder(holder: _RotatingSession):
+ """Route this task's fetches through ``holder`` for the enclosed block."""
+ token = _current_session.set(holder)
+ try:
+ yield holder
+ finally:
+ _current_session.reset(token)
+
+
+@asynccontextmanager
+async def proxy_session():
+ """Open one reused, rotate-on-block proxy session for a continuation chain."""
+ holder = await open_proxy_holder()
+ try:
+ async with bind_proxy_holder(holder):
+ yield holder
+ finally:
+ await holder.close()
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/targets/__init__.py b/surfsense_backend/app/proprietary/platforms/tiktok/targets/__init__.py
new file mode 100644
index 000000000..ac7410154
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/targets/__init__.py
@@ -0,0 +1,8 @@
+"""TikTok URL classification into scrape targets."""
+
+from __future__ import annotations
+
+from .resolver import resolve_target
+from .types import TargetKind, TikTokTarget
+
+__all__ = ["TargetKind", "TikTokTarget", "resolve_target"]
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/targets/resolver.py b/surfsense_backend/app/proprietary/platforms/tiktok/targets/resolver.py
new file mode 100644
index 000000000..ffd15ba2a
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/targets/resolver.py
@@ -0,0 +1,50 @@
+"""Classify a TikTok URL into a :class:`TikTokTarget`, or ``None``."""
+
+from __future__ import annotations
+
+from urllib.parse import parse_qs, unquote, urlparse
+
+from .types import SearchSection, TikTokTarget
+
+_TIKTOK_HOSTS = frozenset({"tiktok.com", "www.tiktok.com", "m.tiktok.com"})
+_SEARCH_SECTIONS: frozenset[SearchSection] = frozenset({"video", "user"})
+
+
+def _is_tiktok_host(hostname: str | None) -> bool:
+ return bool(hostname) and hostname.lower() in _TIKTOK_HOSTS
+
+
+def resolve_target(url: str) -> TikTokTarget | None:
+ parsed = urlparse(url)
+ if not _is_tiktok_host(parsed.hostname):
+ return None
+
+ segments = [s for s in (parsed.path or "").split("/") if s]
+ if not segments:
+ return None
+
+ # Profile / video live under /@username[...].
+ if segments[0].startswith("@"):
+ username = segments[0][1:]
+ if not username:
+ return None
+ if len(segments) >= 3 and segments[1] == "video" and segments[2]:
+ return TikTokTarget("video", segments[2], url, username=username)
+ return TikTokTarget("profile", username, url)
+
+ if segments[0] == "tag" and len(segments) >= 2 and segments[1]:
+ return TikTokTarget("hashtag", unquote(segments[1]), url)
+
+ if segments[0] == "search":
+ query = parse_qs(parsed.query).get("q", [None])[0]
+ if not query:
+ return None
+ section = segments[1] if len(segments) >= 2 else None
+ return TikTokTarget(
+ "search",
+ unquote(query),
+ url,
+ section=section if section in _SEARCH_SECTIONS else None,
+ )
+
+ return None
diff --git a/surfsense_backend/app/proprietary/platforms/tiktok/targets/types.py b/surfsense_backend/app/proprietary/platforms/tiktok/targets/types.py
new file mode 100644
index 000000000..417d27741
--- /dev/null
+++ b/surfsense_backend/app/proprietary/platforms/tiktok/targets/types.py
@@ -0,0 +1,25 @@
+"""Scrape-target value object produced by URL classification."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Literal
+
+TargetKind = Literal["video", "profile", "hashtag", "search", "trending"]
+SearchSection = Literal["video", "user"]
+
+
+@dataclass(frozen=True, slots=True)
+class TikTokTarget:
+ """One classified scrape target.
+
+ ``value`` holds the kind-specific identifier: video id, username, hashtag
+ name, or search query. ``username`` is set for videos (needed to build the
+ canonical post URL). ``section`` narrows a search to videos or users.
+ """
+
+ kind: TargetKind
+ value: str
+ url: str
+ username: str | None = None
+ section: SearchSection | None = None
diff --git a/surfsense_backend/app/routes/__init__.py b/surfsense_backend/app/routes/__init__.py
index 1a5b182b8..85450d4b7 100644
--- a/surfsense_backend/app/routes/__init__.py
+++ b/surfsense_backend/app/routes/__init__.py
@@ -3,7 +3,9 @@ from fastapi import APIRouter, Depends
# Import verb namespaces for their registration side effects before the door builds.
import app.capabilities.google_maps
import app.capabilities.google_search
+import app.capabilities.instagram
import app.capabilities.reddit
+import app.capabilities.tiktok
import app.capabilities.web
import app.capabilities.youtube # noqa: F401
from app.automations.api import router as automations_router
diff --git a/surfsense_backend/app/services/model_connection_service.py b/surfsense_backend/app/services/model_connection_service.py
index cdfd1d725..54d0a3c3f 100644
--- a/surfsense_backend/app/services/model_connection_service.py
+++ b/surfsense_backend/app/services/model_connection_service.py
@@ -15,6 +15,7 @@ from app.db import Connection, Model, ModelSource
from app.services.model_resolver import ensure_v1, to_litellm
from app.services.openrouter_model_normalizer import normalize_openrouter_models
from app.services.provider_registry import Transport, provider_label, spec_for
+from app.services.requesty_model_normalizer import normalize_requesty_models
logger = logging.getLogger(__name__)
@@ -148,7 +149,7 @@ async def verify_connection(conn: Connection) -> VerifyResult:
if spec.transport == Transport.OLLAMA and base_url:
url = f"{base_url.rstrip('/')}/api/version"
- elif spec.discovery in {"openai_models", "openrouter"} and base_url:
+ elif spec.discovery in {"openai_models", "openrouter", "requesty"} and base_url:
url = f"{ensure_v1(base_url)}/models"
elif spec.discovery == "anthropic_models" and base_url:
url = f"{base_url.rstrip('/')}/models"
@@ -363,6 +364,16 @@ async def _openrouter_models(conn: Connection) -> list[dict[str, Any]]:
return normalize_openrouter_models(response.json().get("data", []))
+async def _requesty_models(conn: Connection) -> list[dict[str, Any]]:
+ base_url = _base_url_or_default(conn) or "https://router.requesty.ai/v1"
+ async with httpx.AsyncClient(timeout=DISCOVERY_TIMEOUT_SECONDS) as client:
+ response = await client.get(
+ f"{ensure_v1(base_url)}/models", headers=_auth_headers(conn)
+ )
+ response.raise_for_status()
+ return normalize_requesty_models(response.json().get("data", []))
+
+
def _litellm_static_models(conn: Connection) -> list[dict[str, Any]]:
provider = conn.provider
prefix = spec_for(provider).litellm_prefix or provider
@@ -446,6 +457,8 @@ async def discover_models(conn: Connection) -> list[dict[str, Any]]:
results = await _ollama_tags_then_show(conn)
elif spec.discovery == "openrouter":
results = await _openrouter_models(conn)
+ elif spec.discovery == "requesty":
+ results = await _requesty_models(conn)
elif spec.discovery == "anthropic_models":
results = await _discover_anthropic_models(conn)
elif spec.discovery == "openai_models":
diff --git a/surfsense_backend/app/services/provider_registry.py b/surfsense_backend/app/services/provider_registry.py
index 67d1c4db4..87c1afee3 100644
--- a/surfsense_backend/app/services/provider_registry.py
+++ b/surfsense_backend/app/services/provider_registry.py
@@ -23,6 +23,7 @@ DiscoveryKind = Literal[
"anthropic_models",
"bedrock_models",
"openrouter",
+ "requesty",
"static",
"none",
]
@@ -78,6 +79,15 @@ REGISTRY: dict[str, ProviderSpec] = {
"bearer",
"OpenRouter",
),
+ "requesty": ProviderSpec(
+ Transport.OPENAI_COMPATIBLE,
+ "openai",
+ "requesty",
+ "https://router.requesty.ai/v1",
+ False,
+ "bearer",
+ "Requesty",
+ ),
"openai_compatible": ProviderSpec(
Transport.OPENAI_COMPATIBLE,
"openai",
@@ -87,6 +97,15 @@ REGISTRY: dict[str, ProviderSpec] = {
"bearer",
"OpenAI-compatible provider",
),
+ "openai_compatible_raw": ProviderSpec(
+ Transport.NATIVE,
+ "openai",
+ "none",
+ None,
+ True,
+ "bearer",
+ "OpenAI-compatible raw endpoint",
+ ),
"lm_studio": ProviderSpec(
Transport.OPENAI_COMPATIBLE,
"openai",
diff --git a/surfsense_backend/app/services/quality_score.py b/surfsense_backend/app/services/quality_score.py
index 737dd7c2f..1aa3e7eda 100644
--- a/surfsense_backend/app/services/quality_score.py
+++ b/surfsense_backend/app/services/quality_score.py
@@ -123,6 +123,7 @@ PROVIDER_PRESTIGE_YAML: dict[str, int] = {
"perplexity": 28,
"bedrock": 28,
"openrouter": 25,
+ "requesty": 25,
"ollama_chat": 12,
"custom": 12,
}
diff --git a/surfsense_backend/app/services/requesty_model_normalizer.py b/surfsense_backend/app/services/requesty_model_normalizer.py
new file mode 100644
index 000000000..7710eb174
--- /dev/null
+++ b/surfsense_backend/app/services/requesty_model_normalizer.py
@@ -0,0 +1,123 @@
+"""Shared Requesty model normalization.
+
+Requesty (https://router.requesty.ai) is an OpenAI-compatible LLM router.
+Its ``/v1/models`` catalogue carries richer, Requesty-specific capability
+metadata than a generic OpenAI-compatible ``/models`` response, so keep all
+Requesty filtering and capability extraction here -- mirroring
+``openrouter_model_normalizer`` -- so GLOBAL catalogue generation and BYOK
+discovery agree.
+
+Unlike OpenRouter, Requesty exposes capabilities as flat booleans
+(``supports_tool_calling`` / ``supports_reasoning`` / ``supports_vision`` /
+``supports_image_generation``) rather than an ``architecture`` block plus a
+``supported_parameters`` array, and it reports context size as
+``context_window`` rather than ``context_length``. This module maps those
+fields onto the same normalized shape the rest of the backend consumes.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from app.db import ModelSource
+
+MIN_CONTEXT_LENGTH = 100_000
+
+EXCLUDED_PROVIDER_SLUGS: set[str] = {"amazon"}
+EXCLUDED_MODEL_IDS: set[str] = set()
+EXCLUDED_MODEL_SUFFIXES: tuple[str, ...] = ("-deep-research",)
+
+
+def is_image_output_model(model: dict[str, Any]) -> bool:
+ return bool(model.get("supports_image_generation"))
+
+
+def is_text_output_model(model: dict[str, Any]) -> bool:
+ # Requesty entries are chat-completion models (``api == "chat"``). Treat a
+ # model as text output whenever it is not an image-generation model.
+ return not is_image_output_model(model)
+
+
+def supports_image_input(model: dict[str, Any]) -> bool:
+ return bool(model.get("supports_vision"))
+
+
+def supports_tool_calling(model: dict[str, Any]) -> bool:
+ return bool(model.get("supports_tool_calling"))
+
+
+def has_sufficient_context(model: dict[str, Any]) -> bool:
+ return int(model.get("context_window") or 0) >= MIN_CONTEXT_LENGTH
+
+
+def is_compatible_provider(model: dict[str, Any]) -> bool:
+ model_id = str(model.get("id") or "")
+ slug = model_id.split("/", 1)[0] if "/" in model_id else ""
+ return slug not in EXCLUDED_PROVIDER_SLUGS
+
+
+def is_allowed_model(model: dict[str, Any]) -> bool:
+ model_id = str(model.get("id") or "")
+ if model_id in EXCLUDED_MODEL_IDS:
+ return False
+ base_id = model_id.split(":")[0]
+ return not base_id.endswith(EXCLUDED_MODEL_SUFFIXES)
+
+
+def is_requesty_chat_model(model: dict[str, Any]) -> bool:
+ return (
+ "/" in str(model.get("id") or "")
+ and is_text_output_model(model)
+ and supports_tool_calling(model)
+ and has_sufficient_context(model)
+ and is_compatible_provider(model)
+ and is_allowed_model(model)
+ )
+
+
+def is_requesty_image_model(model: dict[str, Any]) -> bool:
+ return (
+ "/" in str(model.get("id") or "")
+ and is_image_output_model(model)
+ and is_compatible_provider(model)
+ and is_allowed_model(model)
+ )
+
+
+def normalize_requesty_models(
+ raw_models: list[dict[str, Any]],
+) -> list[dict[str, Any]]:
+ normalized: list[dict[str, Any]] = []
+ for model in raw_models:
+ if not is_requesty_chat_model(model):
+ continue
+ model_id = str(model.get("id") or "")
+ normalized.append(
+ {
+ "model_id": model_id,
+ "display_name": model.get("name") or model_id,
+ "source": ModelSource.DISCOVERED,
+ "supports_chat": True,
+ "max_input_tokens": model.get("context_window"),
+ "supports_image_input": supports_image_input(model),
+ "supports_tools": supports_tool_calling(model),
+ "supports_image_generation": False,
+ "metadata": model,
+ }
+ )
+ return normalized
+
+
+__all__ = [
+ "MIN_CONTEXT_LENGTH",
+ "has_sufficient_context",
+ "is_allowed_model",
+ "is_compatible_provider",
+ "is_image_output_model",
+ "is_requesty_chat_model",
+ "is_requesty_image_model",
+ "is_text_output_model",
+ "normalize_requesty_models",
+ "supports_image_input",
+ "supports_tool_calling",
+]
diff --git a/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py b/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py
index fc896005c..7debd60c9 100644
--- a/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py
+++ b/surfsense_backend/app/tasks/celery_tasks/schedule_checker_task.py
@@ -48,6 +48,7 @@ async def _check_and_trigger_schedules():
# Live connectors (Linear, Slack, Jira, ClickUp, Airtable, Discord,
# Teams, Gmail, Calendar, Luma) use real-time tools instead.
from app.tasks.celery_tasks.connector_tasks import (
+ index_bookstack_pages_task,
index_confluence_pages_task,
index_elasticsearch_documents_task,
index_github_repos_task,
@@ -57,6 +58,7 @@ async def _check_and_trigger_schedules():
task_map = {
SearchSourceConnectorType.NOTION_CONNECTOR: index_notion_pages_task,
+ SearchSourceConnectorType.BOOKSTACK_CONNECTOR: index_bookstack_pages_task,
SearchSourceConnectorType.GITHUB_CONNECTOR: index_github_repos_task,
SearchSourceConnectorType.CONFLUENCE_CONNECTOR: index_confluence_pages_task,
SearchSourceConnectorType.ELASTICSEARCH_CONNECTOR: index_elasticsearch_documents_task,
diff --git a/surfsense_backend/pyproject.toml b/surfsense_backend/pyproject.toml
index 2faa2be17..30bb7ea5c 100644
--- a/surfsense_backend/pyproject.toml
+++ b/surfsense_backend/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "surf-new-backend"
-version = "0.0.31"
+version = "0.0.32"
description = "SurfSense Backend"
requires-python = ">=3.12"
dependencies = [
diff --git a/surfsense_backend/scripts/docker/entrypoint.sh b/surfsense_backend/scripts/docker/entrypoint.sh
index 2efd78f94..4e46d184a 100644
--- a/surfsense_backend/scripts/docker/entrypoint.sh
+++ b/surfsense_backend/scripts/docker/entrypoint.sh
@@ -121,7 +121,28 @@ start_beat() {
echo " Celery Beat PID=${PIDS[-1]}"
}
+# ── Headful browser display ──────────────────────────────────
+# Give the stealth browser a virtual display so it can run headful (TikTok's
+# profile feed is empty to headless Chromium). Off => every browser stays headless.
+start_xvfb() {
+ if [ "$(echo "${CRAWL_HEADED_XVFB_ENABLED:-false}" | tr '[:upper:]' '[:lower:]')" != "true" ]; then
+ return
+ fi
+ local display="${XVFB_DISPLAY:-:99}"
+ echo "Starting Xvfb on ${display} (CRAWL_HEADED_XVFB_ENABLED=true)..."
+ Xvfb "${display}" -screen 0 1920x1080x24 -nolisten tcp >/dev/null 2>&1 &
+ PIDS+=($!)
+ export DISPLAY="${display}"
+ sleep 1 # let the X server accept connections before a browser starts
+ echo " Xvfb PID=${PIDS[-1]} DISPLAY=${DISPLAY}"
+}
+
# ── Main: run based on role ──────────────────────────────────
+# migrate never launches a browser; every other role might, so start the display.
+if [ "${SERVICE_ROLE}" != "migrate" ]; then
+ start_xvfb
+fi
+
case "${SERVICE_ROLE}" in
migrate)
run_migrations
diff --git a/surfsense_backend/scripts/e2e_instagram_scraper.py b/surfsense_backend/scripts/e2e_instagram_scraper.py
new file mode 100644
index 000000000..a5e356f99
--- /dev/null
+++ b/surfsense_backend/scripts/e2e_instagram_scraper.py
@@ -0,0 +1,236 @@
+"""Manual functional e2e for the Instagram scraper (app/proprietary/platforms/instagram).
+
+Run from the backend directory:
+ cd surfsense_backend
+ uv run python scripts/e2e_instagram_scraper.py
+ # or: .venv/bin/python scripts/e2e_instagram_scraper.py
+
+This is NOT a pytest test (it needs live network + a residential/custom proxy).
+It:
+
+ Step 0 — go/no-go probe: open a proxy session, mint the anonymous
+ ``csrftoken``/``mid`` cookies, then fetch ``web_profile_info`` on the SAME
+ sticky IP and assert it returns a profile. If this fails the whole
+ approach is invalid — later steps are skipped.
+ Step 1 — scrape a profile's posts.
+ Step 2 — scrape a profile's reels.
+ Step 3 — anonymous single-post extraction for a discovered ``/p/`` URL.
+ Step 4 — fetch profile details.
+ Step 5 — run a profile discovery search (Google-backed).
+ Step 6 — dump trimmed, PII-anonymized raw fixtures into
+ tests/unit/platforms/instagram/fixtures/ for the offline parser tests.
+
+Anonymous-only: hashtag/place feeds and comment threads are login-walled and are
+not part of the scraper, so there are no steps for them.
+"""
+
+import asyncio
+import json
+import sys
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+# --- bootstrap: load .env and put the backend root on sys.path before app.* ---
+_BACKEND_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(_BACKEND_ROOT))
+for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"):
+ if _candidate.exists():
+ load_dotenv(_candidate)
+ break
+
+from app.proprietary.platforms.instagram import ( # noqa: E402
+ InstagramScrapeInput,
+ scrape_instagram,
+)
+from app.proprietary.platforms.instagram.fetch import ( # noqa: E402
+ fetch_html,
+ fetch_json,
+ proxy_session,
+ warm_session,
+)
+from app.proprietary.platforms.instagram.url_resolver import resolve_url # noqa: E402
+
+_PROFILE = "natgeo"
+_SEARCH_TERM = "national geographic"
+
+_FIXTURE_DIR = _BACKEND_ROOT / "tests" / "unit" / "platforms" / "instagram" / "fixtures"
+
+# Fields to strip from dumped fixtures so we never commit PII / volatile tokens.
+_PII_KEYS = frozenset(
+ {"profile_pic_url", "profile_pic_url_hd", "display_url", "video_url", "biography"}
+)
+
+
+def _hr(title: str) -> None:
+ print(f"\n{'=' * 70}\n{title}\n{'=' * 70}")
+
+
+def _check(label: str, ok: bool, detail: str = "") -> bool:
+ print(f" [{'PASS' if ok else 'FAIL'}] {label}{f' — {detail}' if detail else ''}")
+ return ok
+
+
+def _anonymize(obj):
+ """Recursively blank PII-ish string values so fixtures are safe to commit."""
+ if isinstance(obj, dict):
+ return {
+ k: ("" if k in _PII_KEYS and v else _anonymize(v))
+ for k, v in obj.items()
+ }
+ if isinstance(obj, list):
+ return [_anonymize(x) for x in obj]
+ return obj
+
+
+async def step0_probe() -> bool:
+ _hr("STEP 0 — go/no-go: csrftoken warm-up + sticky web_profile_info")
+ async with proxy_session() as holder:
+ if holder.session is None:
+ return _check(
+ "proxy configured", False, "no proxy -> set PROXY_PROVIDER + creds"
+ )
+ minted = await warm_session(holder.session)
+ holder.warmed = True # don't let fetch_json re-warm; we just warmed it
+ _check("csrftoken warm-up minted a session", minted)
+ data = await fetch_json(
+ "api/v1/users/web_profile_info/", {"username": _PROFILE}
+ )
+ user = (
+ (data or {}).get("data", {}).get("user") if isinstance(data, dict) else None
+ )
+ print(f" web_profile_info({_PROFILE}) -> user={'yes' if user else 'no'}")
+ return _check("sticky web_profile_info", minted and bool(user))
+
+
+async def step1_posts() -> bool:
+ _hr("STEP 1 — profile posts")
+ items = await scrape_instagram(
+ InstagramScrapeInput(
+ resultsType="posts",
+ directUrls=[f"https://www.instagram.com/{_PROFILE}/"],
+ resultsLimit=5,
+ ),
+ limit=5,
+ )
+ for it in items[:5]:
+ print(f" - {it.get('shortCode')} | likes={it.get('likesCount')}")
+ return _check("profile returned posts", len(items) > 0, f"{len(items)} posts")
+
+
+async def step2_reels() -> bool:
+ _hr("STEP 2 — profile reels")
+ items = await scrape_instagram(
+ InstagramScrapeInput(
+ resultsType="reels",
+ directUrls=[f"https://www.instagram.com/{_PROFILE}/"],
+ resultsLimit=5,
+ ),
+ limit=5,
+ )
+ print(f" {len(items)} reels for {_PROFILE}")
+ return _check("reels returned items", len(items) >= 0, f"{len(items)} reels")
+
+
+async def step3_single_post(post_url: str | None) -> bool:
+ _hr("STEP 3 — single-post extraction for a /p/ URL")
+ if not post_url:
+ return _check("had a post URL", False, "step 1 found no post")
+ items = await scrape_instagram(
+ InstagramScrapeInput(
+ resultsType="posts", directUrls=[post_url], resultsLimit=1
+ ),
+ limit=1,
+ )
+ got = items[0] if items else {}
+ print(f" {len(items)} item for {post_url} | owner={got.get('ownerUsername')}")
+ return _check("single post returned an item", len(items) > 0, post_url)
+
+
+async def step4_details() -> bool:
+ _hr("STEP 4 — profile details")
+ items = await scrape_instagram(
+ InstagramScrapeInput(
+ resultsType="details",
+ directUrls=[f"https://www.instagram.com/{_PROFILE}/"],
+ ),
+ limit=10,
+ )
+ kinds = sorted({i.get("detailKind") for i in items})
+ print(f" detail kinds={kinds}")
+ return _check("details returned", len(items) > 0, f"{len(items)} items {kinds}")
+
+
+async def step5_search() -> bool:
+ _hr("STEP 5 — profile discovery search (Google-backed)")
+ items = await scrape_instagram(
+ InstagramScrapeInput(
+ resultsType="posts",
+ search=_SEARCH_TERM,
+ searchType="profile",
+ searchLimit=3,
+ resultsLimit=3,
+ ),
+ limit=9,
+ )
+ print(f" {len(items)} items for search '{_SEARCH_TERM}'")
+ return _check("search returned results", len(items) >= 0, f"{len(items)} items")
+
+
+async def step6_dump_fixtures(post_url: str | None) -> bool:
+ _hr("STEP 6 — dump trimmed, anonymized fixtures for offline tests")
+ profile = await fetch_json("api/v1/users/web_profile_info/", {"username": _PROFILE})
+ _FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
+ wrote = []
+ if isinstance(profile, dict) and profile.get("data", {}).get("user"):
+ (_FIXTURE_DIR / "profile.json").write_text(
+ json.dumps(_anonymize(profile)), encoding="utf-8"
+ )
+ wrote.append("profile.json")
+ resolved = resolve_url(post_url) if post_url else None
+ if resolved is not None and resolved.kind in ("post", "reel"):
+ html = await fetch_html(f"p/{resolved.value}/")
+ if html:
+ (_FIXTURE_DIR / "post.json").write_text(
+ json.dumps(
+ {"url": post_url, "shortcode": resolved.value, "html": html}
+ ),
+ encoding="utf-8",
+ )
+ wrote.append("post.json")
+ return _check("dumped fixtures", bool(wrote), f"{wrote} -> {_FIXTURE_DIR}")
+
+
+async def _first_post_url() -> str | None:
+ """Discover a live post URL from the target profile's first page."""
+ items = await scrape_instagram(
+ InstagramScrapeInput(
+ resultsType="posts",
+ directUrls=[f"https://www.instagram.com/{_PROFILE}/"],
+ resultsLimit=1,
+ ),
+ limit=1,
+ )
+ return items[0].get("url") if items else None
+
+
+async def main() -> int:
+ results = [await step0_probe()]
+ if not results[-1]:
+ print("\ncookie probe failed — the approach is invalid on this IP/proxy.")
+ print("Aborting remaining steps.")
+ return 1
+ results.append(await step1_posts())
+ results.append(await step2_reels())
+ post_url = await _first_post_url()
+ results.append(await step3_single_post(post_url))
+ results.append(await step4_details())
+ results.append(await step5_search())
+ results.append(await step6_dump_fixtures(post_url))
+ _hr("SUMMARY")
+ print(f" {sum(results)}/{len(results)} steps passed")
+ return 0 if all(results) else 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(asyncio.run(main()))
diff --git a/surfsense_backend/scripts/e2e_tiktok_scrape.py b/surfsense_backend/scripts/e2e_tiktok_scrape.py
new file mode 100644
index 000000000..4a0063303
--- /dev/null
+++ b/surfsense_backend/scripts/e2e_tiktok_scrape.py
@@ -0,0 +1,301 @@
+"""Manual functional e2e for the TikTok scraper (blob + browser-listing seams).
+
+Run from the backend directory:
+ cd surfsense_backend
+ uv run python scripts/e2e_tiktok_scrape.py
+
+What it exercises (everything REAL — live network, live proxy, live browser):
+
+ Stage 1 — proxy egress proof (informational).
+ Stage 2 — profile via the full pipeline: real videos when headful
+ (CRAWL_HEADED_XVFB_ENABLED), else one ErrorItem — never silent empty.
+ Stage 3 — blob video path over HTTP (URL taken from a captured hashtag struct).
+ Stage 4 — hashtag listing via the stealth browser (captures item_list XHRs).
+ Stage 5 — full scrape_tiktok() pipeline on a hashtag.
+ Stage 6 — search via the full pipeline: same graceful-degrade contract as
+ profile (results feed doesn't load for anonymous sessions).
+ Stage 7 — comments on a real video URL (served anonymously once the panel
+ opens): real comments OR a single honest ErrorItem.
+ Stage 8 — user search: the account-discovery XHR that DOES serve anonymous
+ headless sessions — asserts real account records come back.
+ Stage 9 — trending: the Explore feed of trending videos — asserts real,
+ normalized video items come back.
+
+On success it writes raw itemStructs under tests/fixtures/tiktok/ so the parser
+suites can pin against real-shaped data without network.
+
+This is NOT a pytest test (it needs a live stack + proxy + a real browser). It is
+the manual counterpart to the unit suites and the truth check for
+``session/listing.py``, which is otherwise unverified.
+"""
+
+import asyncio
+import json
+import sys
+from pathlib import Path
+from typing import Any
+from urllib.parse import urlsplit
+
+from dotenv import load_dotenv
+
+# --- bootstrap: load .env and put the backend root on sys.path before app.* ---
+_BACKEND_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(_BACKEND_ROOT))
+for _candidate in (_BACKEND_ROOT / ".env", _BACKEND_ROOT.parent / ".env"):
+ if _candidate.exists():
+ load_dotenv(_candidate)
+ break
+
+_FIXTURES = _BACKEND_ROOT / "tests" / "fixtures" / "tiktok"
+
+# Evergreen public targets: a regular high-volume creator, a broad hashtag, and
+# a common search term.
+_PROFILE = "nasa"
+_HASHTAG = "food"
+_SEARCH = "meal prep"
+_COUNT = 5
+
+
+def _mask(url: str | None) -> str:
+ if not url:
+ return ""
+ p = urlsplit(url)
+ creds = "***@" if p.username else ""
+ port = f":{p.port}" if p.port else ""
+ return f"{p.scheme}://{creds}{p.hostname or '?'}{port}"
+
+
+def _hr(title: str) -> None:
+ print(f"\n{'=' * 70}\n{title}\n{'=' * 70}")
+
+
+def _check(label: str, ok: bool, detail: str = "") -> bool:
+ print(f" [{'PASS' if ok else 'FAIL'}] {label}{f' — {detail}' if detail else ''}")
+ return ok
+
+
+def _dump_fixture(name: str, data: Any) -> None:
+ _FIXTURES.mkdir(parents=True, exist_ok=True)
+ path = _FIXTURES / name
+ path.write_text(json.dumps(data, indent=2, ensure_ascii=False), encoding="utf-8")
+ print(f" [fixture] wrote {path.relative_to(_BACKEND_ROOT)}")
+
+
+def _url_from_struct(struct: dict[str, Any]) -> str | None:
+ """Build a canonical video URL from a captured itemStruct."""
+ author = (struct.get("author") or {}).get("uniqueId")
+ vid = struct.get("id")
+ return f"https://www.tiktok.com/@{author}/video/{vid}" if author and vid else None
+
+
+async def stage_proxy() -> bool:
+ _hr("STAGE 1 — proxy egress (informational)")
+ from app.utils.proxy import get_active_provider, get_proxy_url, is_pool_backed
+
+ provider = get_active_provider()
+ proxy_url = get_proxy_url()
+ print(f" active proxy provider : {provider.name}")
+ print(f" proxy url : {_mask(proxy_url)}")
+ print(f" pool-backed (rotates) : {is_pool_backed()}")
+ if not proxy_url:
+ print(" [INFO] no proxy configured — TikTok may block anonymous access")
+ return True
+
+
+async def stage_profile_listing() -> tuple[bool, list[dict[str, Any]]]:
+ _hr(f"STAGE 2 — profile listing graceful-degrade: @{_PROFILE}")
+ from app.proprietary.platforms.tiktok import TikTokScrapeInput, scrape_tiktok
+
+ # Accept videos OR an ErrorItem (never a silent empty): the feed degrades
+ # gracefully when the exit IP is flagged.
+ items = await scrape_tiktok(
+ TikTokScrapeInput(profiles=[_PROFILE], resultsPerPage=_COUNT), limit=_COUNT
+ )
+ has_video = any(it.get("id") and not it.get("errorCode") for it in items)
+ has_error = any(it.get("errorCode") == "no_items" for it in items)
+ ok = _check(
+ "profile yields videos or a graceful ErrorItem (never silent empty)",
+ has_video or has_error,
+ f"{len(items)} item(s); video={has_video} error={has_error}",
+ )
+ return ok, items
+
+
+async def stage_blob_video(video_url: str) -> tuple[bool, dict[str, Any] | None]:
+ _hr("STAGE 3 — blob video path (HTTP)")
+ print(f" target: {video_url}")
+ from app.proprietary.platforms.tiktok.extraction import (
+ extract_rehydration_data,
+ video_item_struct,
+ )
+ from app.proprietary.platforms.tiktok.session import fetch_html
+
+ html = await fetch_html(video_url)
+ if not _check("fetched page HTML", bool(html), f"{len(html or '')} chars"):
+ return False, None
+ data = extract_rehydration_data(html or "")
+ if not _check("extracted rehydration blob", data is not None):
+ return False, None
+ raw = video_item_struct(data or {})
+ ok = _check(
+ "blob carries the video itemStruct",
+ raw is not None and bool(raw.get("id")),
+ f"id={None if raw is None else raw.get('id')}",
+ )
+ if ok and raw is not None:
+ _dump_fixture("video_item_struct.json", raw)
+ return ok, raw
+
+
+async def stage_hashtag_listing() -> tuple[bool, list[dict[str, Any]]]:
+ _hr(f"STAGE 4 — hashtag listing (browser): #{_HASHTAG}")
+ from app.proprietary.platforms.tiktok.session import fetch_item_list
+
+ url = f"https://www.tiktok.com/tag/{_HASHTAG}"
+ raw = await fetch_item_list(url, _COUNT)
+ ok = _check(
+ "captured itemStructs for hashtag",
+ len(raw) > 0 and bool(raw[0].get("id")),
+ f"{len(raw)} struct(s)",
+ )
+ if ok:
+ _dump_fixture("listing_item.json", raw[0])
+ return ok, raw
+
+
+async def stage_pipeline() -> bool:
+ _hr("STAGE 5 — full scrape_tiktok() pipeline")
+ from app.proprietary.platforms.tiktok import TikTokScrapeInput, scrape_tiktok
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(hashtags=[_HASHTAG], resultsPerPage=3), limit=3
+ )
+ ok = _check(
+ "pipeline returns normalized video items",
+ len(items) > 0
+ and bool(items[0].get("id"))
+ and bool(items[0].get("webVideoUrl")),
+ f"{len(items)} item(s)",
+ )
+ if items:
+ print(
+ f" sample: {items[0].get('webVideoUrl')} — {items[0].get('text', '')[:60]!r}"
+ )
+ return ok
+
+
+async def stage_search_listing() -> tuple[bool, list[dict[str, Any]]]:
+ _hr(f"STAGE 6 — search listing graceful-degrade: {_SEARCH!r}")
+ from app.proprietary.platforms.tiktok import TikTokScrapeInput, scrape_tiktok
+
+ # The search results feed doesn't load for anonymous headless sessions
+ # (results XHR never fires on a cold deep-link). Same contract as profile:
+ # verify a graceful ErrorItem instead of a silent empty.
+ items = await scrape_tiktok(
+ TikTokScrapeInput(searchQueries=[_SEARCH], resultsPerPage=_COUNT), limit=_COUNT
+ )
+ has_video = any(it.get("id") and not it.get("errorCode") for it in items)
+ has_error = any(it.get("errorCode") == "no_items" for it in items)
+ ok = _check(
+ "search yields videos or a graceful ErrorItem (never silent empty)",
+ has_video or has_error,
+ f"{len(items)} item(s); video={has_video} error={has_error}",
+ )
+ return ok, items
+
+
+async def stage_comments(video_url: str) -> tuple[bool, list[dict[str, Any]]]:
+ _hr("STAGE 7 — comments graceful-degrade")
+ print(f" target: {video_url}")
+ from app.proprietary.platforms.tiktok import scrape_tiktok_comments
+
+ # Comments load over a signed /api/comment/list XHR that TikTok serves to
+ # anonymous sessions once the panel opens. Pass if real comments come back
+ # OR a graceful ErrorItem (video has none / disabled / withheld).
+ items = await scrape_tiktok_comments([video_url], per_video=_COUNT, limit=_COUNT)
+ has_comment = any(it.get("id") and not it.get("errorCode") for it in items)
+ has_error = any(it.get("errorCode") == "no_comments" for it in items)
+ ok = _check(
+ "comments yield records or a graceful ErrorItem (never silent empty)",
+ has_comment or has_error,
+ f"{len(items)} item(s); comment={has_comment} error={has_error}",
+ )
+ return ok, items
+
+
+async def stage_user_search() -> tuple[bool, list[dict[str, Any]]]:
+ _hr(f"STAGE 8 — user search (browser): {_PROFILE!r}")
+ from app.proprietary.platforms.tiktok import search_tiktok_users
+
+ # Unlike keyword *video* search, the account-search XHR serves anonymous
+ # headless sessions — so this asserts real records, not just degradation.
+ items = await search_tiktok_users([_PROFILE], per_query=_COUNT, limit=_COUNT)
+ real = [it for it in items if not it.get("errorCode")]
+ ok = _check(
+ "user search returns account records",
+ bool(real) and bool(real[0].get("uniqueId") or real[0].get("name")),
+ f"{len(items)} item(s); accounts={len(real)}",
+ )
+ if real:
+ print(f" sample: @{real[0].get('uniqueId') or real[0].get('name')}")
+ return ok, items
+
+
+async def stage_trending() -> tuple[bool, list[dict[str, Any]]]:
+ _hr("STAGE 9 — trending (browser): Explore feed")
+ from app.proprietary.platforms.tiktok import scrape_tiktok_trending
+
+ items = await scrape_tiktok_trending(count=_COUNT)
+ real = [it for it in items if not it.get("errorCode")]
+ ok = _check(
+ "trending returns normalized video items",
+ bool(real) and bool(real[0].get("id")) and bool(real[0].get("webVideoUrl")),
+ f"{len(items)} item(s); videos={len(real)}",
+ )
+ if real:
+ print(
+ f" sample: {real[0].get('webVideoUrl')} — {real[0].get('text', '')[:60]!r}"
+ )
+ return ok, items
+
+
+async def main() -> int:
+ print("TikTok scraper functional e2e — live network + proxy + browser")
+ results: dict[str, bool] = {}
+
+ results["Stage 1 proxy"] = await stage_proxy()
+
+ # Hashtag listing is the reliable browser path; use one of its captured
+ # structs to build a real video URL for the HTTP blob path.
+ ok_tag, tag_structs = await stage_hashtag_listing()
+ results["Stage 4 hashtag listing"] = ok_tag
+
+ video_url = _url_from_struct(tag_structs[0]) if tag_structs else None
+ if video_url:
+ ok_video, _ = await stage_blob_video(video_url)
+ results["Stage 3 blob video"] = ok_video
+ ok_comments, _ = await stage_comments(video_url)
+ results["Stage 7 comments"] = ok_comments
+ else:
+ print("\n [SKIP] Stage 3/7 — no captured struct to build a video URL")
+
+ ok_search, _ = await stage_search_listing()
+ results["Stage 6 search listing"] = ok_search
+
+ ok_profile, _ = await stage_profile_listing()
+ results["Stage 2 profile listing"] = ok_profile
+ results["Stage 5 pipeline"] = await stage_pipeline()
+
+ ok_users, _ = await stage_user_search()
+ results["Stage 8 user search"] = ok_users
+ ok_trending, _ = await stage_trending()
+ results["Stage 9 trending"] = ok_trending
+
+ _hr("SUMMARY")
+ for name, ok in results.items():
+ print(f" {'PASS' if ok else 'FAIL/SKIP'} — {name}")
+ return 0 if all(results.values()) else 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(asyncio.run(main()))
diff --git a/surfsense_backend/tests/fixtures/tiktok/listing_item.json b/surfsense_backend/tests/fixtures/tiktok/listing_item.json
new file mode 100644
index 000000000..374a9bf67
--- /dev/null
+++ b/surfsense_backend/tests/fixtures/tiktok/listing_item.json
@@ -0,0 +1,559 @@
+{
+ "AIGCDescription": "",
+ "CategoryType": 111,
+ "IsHDBitrate": false,
+ "anchors": [
+ {
+ "description": "CapCut · Video Editor",
+ "extraInfo": {
+ "subtype": ""
+ },
+ "icon": {
+ "urlList": [
+ "https://p16-common-sign.tiktokcdn-us.com/tiktok-obj/capcut_logo_64px_bk.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=44xwWHkyYJdE%2FIwqS3zAAjU4XQ4%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p19-common-sign.tiktokcdn-us.com/tiktok-obj/capcut_logo_64px_bk.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=lMwgk8e0wLaM3WkpnsMqYcxZN6w%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p16-common-sign.tiktokcdn-us.com/tiktok-obj/capcut_logo_64px_bk.png~tplv-tiktokx-origin.jpeg?dr=9627&x-expires=1783544400&x-signature=1EMaRlc0Snbs9Ltwr0s95%2BDvpFk%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5"
+ ]
+ },
+ "id": "0",
+ "keyword": "CapCut · Editing made easy",
+ "logExtra": "{\"anchor_id\":0,\"anchor_name\":\"CapCut · Editing made easy\",\"anchor_title_detail\":\"None\",\"anchor_title_id\":21502485763,\"anchor_type\":\"TT_CAPCUT\",\"capability_key\":\"default\",\"ccep_vip\":0,\"if_race_trigger\":0,\"is_two_line\":0,\"landing_page_style_id\":21502486275,\"maker_source\":\"\",\"publish_key\":\"default\",\"template_id\":\"none\",\"tutorial_id\":\"none\",\"viamaker_anchor_capability_key_weight\":1,\"viamaker_anchor_style_idx\":-1,\"viamaker_anchor_style_source\":3,\"video_source\":0,\"video_type_id\":1}",
+ "schema": "https://www.capcut.com/editor?scenario=tiktok&utm_source=tiktok_anchor_tool&utm_medium=tiktok_anchor&utm_campaign=70361230",
+ "thumbnail": {
+ "height": 64,
+ "urlList": [
+ "https://p19-common-sign.tiktokcdn-us.com/tiktok-obj/64x_Capcut3x.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=GOZUldk%2BrxINygKT0jwxeJz9VLQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p16-common-sign.tiktokcdn-us.com/tiktok-obj/64x_Capcut3x.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=SJ6q86bjtCoRArlVNS1DjDDFWoc%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p19-common-sign.tiktokcdn-us.com/tiktok-obj/64x_Capcut3x.png~tplv-tiktokx-origin.jpeg?dr=9627&x-expires=1783544400&x-signature=c7mUcXdyEOFc0SMcLAkkib3v9RU%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5"
+ ],
+ "width": 64
+ },
+ "type": 54
+ }
+ ],
+ "author": {
+ "avatarLarger": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:1080:1080.jpeg?dr=9640&refresh_token=c6d77c8a&x-expires=1783695600&x-signature=mUTuvDWLEGrAxXtWZlBE7gUxCl8%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "avatarMedium": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:720:720.jpeg?dr=9640&refresh_token=dfe18cb8&x-expires=1783695600&x-signature=cP14vkr0ndxwhW%2B1pWfefXfTsls%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "avatarThumb": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:100:100.jpeg?dr=9640&refresh_token=8c557171&x-expires=1783695600&x-signature=VaAysM3zFkbl%2FOT74d0h2kdkLqo%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "commentSetting": 0,
+ "downloadSetting": 3,
+ "duetSetting": 0,
+ "ftc": false,
+ "id": "7501138183701103662",
+ "isADVirtual": false,
+ "isEmbedBanned": false,
+ "nickname": "Cookwithhali",
+ "openFavorite": false,
+ "privateAccount": false,
+ "relation": 0,
+ "secUid": "MS4wLjABAAAAywZ6lCXK3u4zKz8eQxxcjVblLiugz6zMysU98G7dyKVxu6IvMOJ97mpIfpDxUL4q",
+ "secret": false,
+ "shortDramaCreator": {},
+ "signature": "From my kitchen to your screen! 🍞\nFollow for more ✨\nIG: halikit25\n\n📩 Halikitchen25@gmail.com",
+ "stitchSetting": 0,
+ "uniqueId": "cookwithhali",
+ "verified": false
+ },
+ "authorStats": {
+ "diggCount": 223,
+ "followerCount": 372600,
+ "followingCount": 0,
+ "friendCount": 0,
+ "heart": 8000000,
+ "heartCount": 8000000,
+ "videoCount": 279
+ },
+ "authorStatsV2": {
+ "diggCount": "223",
+ "followerCount": "372600",
+ "followingCount": "0",
+ "friendCount": "0",
+ "heart": "8000000",
+ "heartCount": "8000000",
+ "videoCount": "279"
+ },
+ "backendSourceEventTracking": "",
+ "challenges": [
+ {
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": "",
+ "desc": "",
+ "id": "6983",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "title": "bread"
+ },
+ {
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": "",
+ "desc": "",
+ "id": "285169",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "title": "garlicbread"
+ },
+ {
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": "",
+ "desc": "",
+ "id": "1643317566954502",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "title": "breadtok"
+ },
+ {
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": "",
+ "desc": "Whether you're baking the perfect pie or just searching for a recipe, you've come to the right place for all things #Baking.",
+ "id": "7250",
+ "profileLarger": "https://p19-common-sign.tiktokcdn-us.com/musically-maliva-obj/1aa8a54136dc06390b83508fdd683160~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=nigLPQXvvDM%2BxFQezn3ELBH%2BLwQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "profileMedium": "https://p19-common-sign.tiktokcdn-us.com/musically-maliva-obj/1aa8a54136dc06390b83508fdd683160~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=nigLPQXvvDM%2BxFQezn3ELBH%2BLwQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "profileThumb": "https://p19-common-sign.tiktokcdn-us.com/musically-maliva-obj/1aa8a54136dc06390b83508fdd683160~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=nigLPQXvvDM%2BxFQezn3ELBH%2BLwQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "title": "baking"
+ },
+ {
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": "",
+ "desc": "",
+ "id": "10488587",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "title": "homemadebread"
+ }
+ ],
+ "collected": false,
+ "contents": [
+ {
+ "desc": "Pull Apart Garlic Bread "
+ },
+ {
+ "desc": ""
+ },
+ {
+ "desc": "Ingredients"
+ },
+ {
+ "desc": ""
+ },
+ {
+ "desc": "* 1 tsp yeast"
+ },
+ {
+ "desc": "* 1 tbsp sugar"
+ },
+ {
+ "desc": "* 270g milk (about 1 cup + 2 tbsp)"
+ },
+ {
+ "desc": "* 380g flour (about 3 cups)"
+ },
+ {
+ "desc": "* 1 tsp salt"
+ },
+ {
+ "desc": "* 43g softened butter (about 3 tbsp)"
+ },
+ {
+ "desc": ""
+ },
+ {
+ "desc": "Butter Mixture"
+ },
+ {
+ "desc": ""
+ },
+ {
+ "desc": "* 100g butter, softened"
+ },
+ {
+ "desc": "* 1 tbsp chopped cilantro"
+ },
+ {
+ "desc": "* 1 garlic clove, minced"
+ },
+ {
+ "desc": "* Shredded cheese"
+ },
+ {
+ "desc": ""
+ },
+ {
+ "desc": "Instructions"
+ },
+ {
+ "desc": ""
+ },
+ {
+ "desc": "1. In a bowl, mix the sugar and yeast. Add the milk, then add the flour and mix until combined."
+ },
+ {
+ "desc": "2. Add the salt and softened butter, then coil fold until the butter is fully incorporated."
+ },
+ {
+ "desc": "3. Cover and let the dough rest for 2 hours or until doubled."
+ },
+ {
+ "desc": "4. Mix the softened butter with the cilantro and minced garlic."
+ },
+ {
+ "desc": "5. Roll the dough into a rectangle and spread the garlic butter mixture evenly over it. Sprinkle with shredded cheddar cheese."
+ },
+ {
+ "desc": "6. Cut the dough in half, stack one half on top of the other, then cut into strips. Transfer the pieces into a loaf pan."
+ },
+ {
+ "desc": "7. Cover and let rest for 20 minutes."
+ },
+ {
+ "desc": "8. Bake at 370°F for 30 minutes, or until golden brown."
+ },
+ {
+ "desc": "9. Brush with the remaining garlic butter mixture while still warm."
+ },
+ {
+ "desc": ""
+ },
+ {
+ "desc": "#bread #garlicbread #breadtok #baking #homemadebread ",
+ "textExtra": [
+ {
+ "awemeId": "",
+ "end": 6,
+ "hashtagName": "bread",
+ "isCommerce": false,
+ "start": 0,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 19,
+ "hashtagName": "garlicbread",
+ "isCommerce": false,
+ "start": 7,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 29,
+ "hashtagName": "breadtok",
+ "isCommerce": false,
+ "start": 20,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 37,
+ "hashtagName": "baking",
+ "isCommerce": false,
+ "start": 30,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 52,
+ "hashtagName": "homemadebread",
+ "isCommerce": false,
+ "start": 38,
+ "subType": 0,
+ "type": 1
+ }
+ ]
+ }
+ ],
+ "createTime": 1782509863,
+ "desc": "Pull Apart Garlic Bread Ingredients * 1 tsp yeast * 1 tbsp sugar * 270g milk (about 1 cup + 2 tbsp) * 380g flour (about 3 cups) * 1 tsp salt * 43g softened butter (about 3 tbsp) Butter Mixture * 100g butter, softened * 1 tbsp chopped cilantro * 1 garlic clove, minced * Shredded cheese Instructions 1. In a bowl, mix the sugar and yeast. Add the milk, then add the flour and mix until combined. 2. Add the salt and softened butter, then coil fold until the butter is fully incorporated. 3. Cover and let the dough rest for 2 hours or until doubled. 4. Mix the softened butter with the cilantro and minced garlic. 5. Roll the dough into a rectangle and spread the garlic butter mixture evenly over it. Sprinkle with shredded cheddar cheese. 6. Cut the dough in half, stack one half on top of the other, then cut into strips. Transfer the pieces into a loaf pan. 7. Cover and let rest for 20 minutes. 8. Bake at 370°F for 30 minutes, or until golden brown. 9. Brush with the remaining garlic butter mixture while still warm. #bread #garlicbread #breadtok #baking #homemadebread ",
+ "digged": false,
+ "diversificationId": 10040,
+ "duetDisplay": 0,
+ "duetEnabled": true,
+ "forFriend": false,
+ "id": "7655821461772406047",
+ "isAd": false,
+ "isReviewing": false,
+ "itemCommentStatus": 0,
+ "item_control": {
+ "can_repost": true
+ },
+ "music": {
+ "authorName": "Cookwithhali",
+ "coverLarge": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:1080:1080.jpeg?dr=9640&refresh_token=c6d77c8a&x-expires=1783695600&x-signature=mUTuvDWLEGrAxXtWZlBE7gUxCl8%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "coverMedium": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:720:720.jpeg?dr=9640&refresh_token=dfe18cb8&x-expires=1783695600&x-signature=cP14vkr0ndxwhW%2B1pWfefXfTsls%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "coverThumb": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:100:100.jpeg?dr=9640&refresh_token=8c557171&x-expires=1783695600&x-signature=VaAysM3zFkbl%2FOT74d0h2kdkLqo%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "duration": 62,
+ "id": "7655821492566919966",
+ "isCopyrighted": false,
+ "is_commerce_music": true,
+ "is_unlimited_music": false,
+ "original": true,
+ "playUrl": "https://v16m.tiktokcdn-us.com/ecd83422a3e7bf5450bd92ebf07e6fce/6a4ec449/video/tos/useast8/tos-useast8-v-27dcd7-tx2/ocBIhANWILSemeAAINEGoTTPe58AJaRegg6eKS/?a=1233&bti=ODszNWYuMDE6&&bt=125&ft=GSDrKInz7Th2sbSGXq8Zmo&mime_type=audio_mpeg&rc=anY5cXA5cmRpOzMzaTU8NEBpanY5cXA5cmRpOzMzaTU8NEBpYzZrMmRjL3NhLS1kMTJzYSNpYzZrMmRjL3NhLS1kMTJzcw%3D%3D&vvpl=1&l=2026070815413036F7D0D0506FB569C78C&btag=e00050000",
+ "private": false,
+ "shoot_duration": 62,
+ "title": "original sound",
+ "tt2dsp": {}
+ },
+ "officalItem": false,
+ "originalItem": false,
+ "privateItem": false,
+ "secret": false,
+ "shareEnabled": true,
+ "stats": {
+ "collectCount": 396400,
+ "commentCount": 3720,
+ "diggCount": 754100,
+ "playCount": 8300000,
+ "shareCount": 144500
+ },
+ "statsV2": {
+ "collectCount": "396397",
+ "commentCount": "3720",
+ "diggCount": "754100",
+ "playCount": "8300000",
+ "repostCount": "0",
+ "shareCount": "144500"
+ },
+ "stickersOnItem": [
+ {
+ "stickerText": [
+ "Cheesy Pull Apart Garlic Bread! 🧄🧀\n",
+ "(Bakery-Style & No-Knead)"
+ ],
+ "stickerType": 4
+ }
+ ],
+ "stitchDisplay": 0,
+ "stitchEnabled": true,
+ "textExtra": [
+ {
+ "awemeId": "",
+ "end": 1030,
+ "hashtagName": "bread",
+ "isCommerce": false,
+ "start": 1024,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 1043,
+ "hashtagName": "garlicbread",
+ "isCommerce": false,
+ "start": 1031,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 1053,
+ "hashtagName": "breadtok",
+ "isCommerce": false,
+ "start": 1044,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 1061,
+ "hashtagName": "baking",
+ "isCommerce": false,
+ "start": 1054,
+ "subType": 0,
+ "type": 1
+ },
+ {
+ "awemeId": "",
+ "end": 1076,
+ "hashtagName": "homemadebread",
+ "isCommerce": false,
+ "start": 1062,
+ "subType": 0,
+ "type": 1
+ }
+ ],
+ "textLanguage": "en",
+ "textTranslatable": true,
+ "video": {
+ "PlayAddrStruct": {
+ "DataSize": 13761060,
+ "FileCs": "c:0-51456-f8d1",
+ "FileHash": "485174c8492db027adbbf6dca7c7c201",
+ "Height": 1280,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_h264_720p_1773703",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=aa5be80671c4a73d54517ee8e4a471ad&tk=tt_chain_token&btag=e00088000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=aa5be80671c4a73d54517ee8e4a471ad&tk=tt_chain_token&btag=e00088000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=9de1abcaf21d44d081117ee2ca3d4cc2&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjcwMGExZmI3NDg4YzFiZTIwZjczM2IyMmM0OTk2YTI2&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "Width": 720
+ },
+ "VQScore": "69.69",
+ "bitrate": 1773703,
+ "bitrateInfo": [
+ {
+ "Bitrate": 1836180,
+ "BitrateFPS": 29,
+ "CodecType": "h265_hvc1",
+ "Format": "mp4",
+ "GearName": "adapt_lowest_1080_1",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 96.83, \"v960\": 97.64, \"v864\": 98.256, \"v720\": 98.838}, \"ori\": {\"v1080\": 91.834, \"v960\": 93.423, \"v864\": 94.54, \"v720\": 95.872}}}",
+ "PlayAddr": {
+ "DataSize": 14245777,
+ "FileCs": "c:0-52754-ed06",
+ "FileHash": "38996a98cfeb2714fca1b94768090c37",
+ "Height": 1920,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_bytevc1_1080p_1836180",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oIygxEfEVAP1qvGVTIBARoAvb9QYFGEKDp5v9e/?a=1988&bti=ODszNWYuMDE6&&bt=1793&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=aDlmNGg7ZTdlaGY5ZzUzZ0BpanJneW45cmVpOzMzaTczNEBjXl5hMzAvXy8xNDAuYmIyYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=db26af0a98ff05e504d7121b25048527&tk=tt_chain_token&btag=e00088000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oIygxEfEVAP1qvGVTIBARoAvb9QYFGEKDp5v9e/?a=1988&bti=ODszNWYuMDE6&&bt=1793&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=aDlmNGg7ZTdlaGY5ZzUzZ0BpanJneW45cmVpOzMzaTczNEBjXl5hMzAvXy8xNDAuYmIyYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=db26af0a98ff05e504d7121b25048527&tk=tt_chain_token&btag=e00088000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=5de0e21d367c4e2bb42b386ed1aca6fe&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjAzMmU1YzJlMTAwMDJiYjIwMmFiN2RhNTIzOWVhYWRk&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "Width": 1080
+ },
+ "QualityType": 2,
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 385342}, {\\\"time\\\": 2, \\\"offset\\\": 561014}, {\\\"time\\\": 3, \\\"offset\\\": 703814}, {\\\"time\\\": 4, \\\"offset\\\": 855867}, {\\\"time\\\": 5, \\\"offset\\\": 1019333}, {\\\"time\\\": 10, \\\"offset\\\": 2028412}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 96.83, \\\"v960\\\": 97.64, \\\"v864\\\": 98.256, \\\"v720\\\": 98.838}, \\\"ori\\\": {\\\"v1080\\\": 91.834, \\\"v960\\\": 93.423, \\\"v864\\\": 94.54, \\\"v720\\\": 95.872}}}\",\"ufq\":\"{\\\"enh\\\":89.317,\\\"playback\\\":{\\\"ori\\\":85.151,\\\"srv1\\\":90.147},\\\"src\\\":85.957,\\\"trans\\\":85.151,\\\"version\\\":\\\"v1.2\\\"}\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"11b0345f3bf185ddbe7c48bd7301980f\",\"dec_info\":\"\",\"gearvqm\":\"\",\"audio_bit_rate\":96215,\"ps_info\":\"{\\\"vps\\\": \\\"AQwC//8BYAAAAwAAAwAAAwAAAwCWAACdzkgS\\\", \\\"sps\\\": \\\"AQMBYAAAAwAAAwAAAwAAAwCWAACgAhyAHgWWdzkySUQs0RJJJJify4d/1+bPl/1+M4gQ5qAgICCAAAADAIAAAA8E\\\", \\\"pps\\\": \\\"AcElPAiQ\\\"}\"}"
+ },
+ {
+ "Bitrate": 1773703,
+ "BitrateFPS": 29,
+ "CodecType": "h264",
+ "Format": "mp4",
+ "GearName": "normal_720_0",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 84.77, \"v960\": 88.554, \"v864\": 88.982, \"v720\": 92.772}, \"ori\": {\"v1080\": 84.77, \"v960\": 88.554, \"v864\": 88.982, \"v720\": 92.772}}}",
+ "PlayAddr": {
+ "DataSize": 13761060,
+ "FileCs": "c:0-51456-f8d1",
+ "FileHash": "485174c8492db027adbbf6dca7c7c201",
+ "Height": 1280,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_h264_720p_1773703",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=aa5be80671c4a73d54517ee8e4a471ad&tk=tt_chain_token&btag=e00088000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=aa5be80671c4a73d54517ee8e4a471ad&tk=tt_chain_token&btag=e00088000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=9de1abcaf21d44d081117ee2ca3d4cc2&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjcwMGExZmI3NDg4YzFiZTIwZjczM2IyMmM0OTk2YTI2&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "Width": 720
+ },
+ "QualityType": 10,
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 400712}, {\\\"time\\\": 2, \\\"offset\\\": 659243}, {\\\"time\\\": 3, \\\"offset\\\": 886968}, {\\\"time\\\": 4, \\\"offset\\\": 1101663}, {\\\"time\\\": 5, \\\"offset\\\": 1368565}, {\\\"time\\\": 10, \\\"offset\\\": 2375704}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 84.77, \\\"v960\\\": 88.554, \\\"v864\\\": 88.982, \\\"v720\\\": 92.772}, \\\"ori\\\": {\\\"v1080\\\": 84.77, \\\"v960\\\": 88.554, \\\"v864\\\": 88.982, \\\"v720\\\": 92.772}}}\",\"ufq\":\"\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"65b88c41c0963253ad31b6f68981a242\",\"dec_info\":\"\",\"gearvqm\":\"\",\"audio_bit_rate\":64193}"
+ },
+ {
+ "Bitrate": 966380,
+ "BitrateFPS": 29,
+ "CodecType": "h265_hvc1",
+ "Format": "mp4",
+ "GearName": "adapt_lower_720_1",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 90.05, \"v960\": 91.958, \"v864\": 93.608, \"v720\": 95.869}, \"ori\": {\"v1080\": 82.259, \"v960\": 85.339, \"v864\": 87.288, \"v720\": 90.851}}}",
+ "PlayAddr": {
+ "DataSize": 7497540,
+ "FileCs": "c:0-52786-7f65",
+ "FileHash": "1376df24c917669727b27273697fb826",
+ "Height": 1280,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_bytevc1_720p_966380",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/og99rgOGqJED8xGeBpFEAADvEPARnbTIQTfoVv/?a=1988&bti=ODszNWYuMDE6&&bt=943&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=OTNkNDk2NzlmM2VkPGg7OkBpanJneW45cmVpOzMzaTczNEBeMmBiYWBfNWIxLzEyNF81YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=e3690d67b2ecae289320f5db6c338d05&tk=tt_chain_token&btag=e00088000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/og99rgOGqJED8xGeBpFEAADvEPARnbTIQTfoVv/?a=1988&bti=ODszNWYuMDE6&&bt=943&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=OTNkNDk2NzlmM2VkPGg7OkBpanJneW45cmVpOzMzaTczNEBeMmBiYWBfNWIxLzEyNF81YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=e3690d67b2ecae289320f5db6c338d05&tk=tt_chain_token&btag=e00088000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=87b87e594a2b40ec86b4c3d6e1ca50a6&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLmRjNWNmZGVmMmFkNTIyNWQ1ZTA1NzExM2NjOTVlNzMy&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "Width": 720
+ },
+ "QualityType": 14,
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 284307}, {\\\"time\\\": 2, \\\"offset\\\": 357012}, {\\\"time\\\": 3, \\\"offset\\\": 438159}, {\\\"time\\\": 4, \\\"offset\\\": 532720}, {\\\"time\\\": 5, \\\"offset\\\": 636824}, {\\\"time\\\": 10, \\\"offset\\\": 1142427}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 90.05, \\\"v960\\\": 91.958, \\\"v864\\\": 93.608, \\\"v720\\\": 95.869}, \\\"ori\\\": {\\\"v1080\\\": 82.259, \\\"v960\\\": 85.339, \\\"v864\\\": 87.288, \\\"v720\\\": 90.851}}}\",\"ufq\":\"{\\\"enh\\\":89.767,\\\"playback\\\":{\\\"ori\\\":76.711,\\\"srv1\\\":84.502},\\\"src\\\":85.957,\\\"trans\\\":76.711,\\\"version\\\":\\\"v1.2\\\"}\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"97eb57dffd9e7a0333979f6ca6c9418d\",\"dec_info\":\"\",\"gearvqm\":\"{\\\"v3.0\\\": {\\\"ori\\\": 91.625, \\\"sr20\\\": 97.355}}\",\"audio_bit_rate\":96215,\"ps_info\":\"{\\\"vps\\\": \\\"AQwC//8BYAAAAwAAAwAAAwAAAwC6AACIJElgSA==\\\", \\\"sps\\\": \\\"AQMBYAAAAwAAAwAAAwAAAwC6AACgBaIAUBZYgkSXJIkQTPCEREREREYR8xPOX+/+v82fy/+v8yh+S/3/1/mz+X/1/jOEAg5qAgICCAAAAwAIAAADAPBA\\\", \\\"pps\\\": \\\"AcElPAiQ\\\"}\"}"
+ },
+ {
+ "Bitrate": 799620,
+ "BitrateFPS": 29,
+ "CodecType": "h265_hvc1",
+ "Format": "mp4",
+ "GearName": "adapt_540_1",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 87.061, \"v960\": 89.844, \"v864\": 91.813, \"v720\": 94.499}, \"ori\": {\"v1080\": 77.717, \"v960\": 80.972, \"v864\": 83.632, \"v720\": 87.945}}}",
+ "PlayAddr": {
+ "DataSize": 6203756,
+ "FileCs": "c:0-52786-7f6d",
+ "FileHash": "41f62f740ad13045000450f28b9ff2cb",
+ "Height": 1024,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_bytevc1_540p_799620",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/oogvkDPHIAVvTxgEpE9EMoALxQRRA1FGqeB9Gf/?a=1988&bti=ODszNWYuMDE6&&bt=780&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=NThkNGRmNTo4MztpOTM3ZkBpanJneW45cmVpOzMzaTczNEAyNDMuNC0xNWExYjBiX2FgYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=05aaeb04a6cf8d012bba46bb131f00af&tk=tt_chain_token&btag=e00088000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/oogvkDPHIAVvTxgEpE9EMoALxQRRA1FGqeB9Gf/?a=1988&bti=ODszNWYuMDE6&&bt=780&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=NThkNGRmNTo4MztpOTM3ZkBpanJneW45cmVpOzMzaTczNEAyNDMuNC0xNWExYjBiX2FgYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=05aaeb04a6cf8d012bba46bb131f00af&tk=tt_chain_token&btag=e00088000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=ccd4b0e63fe840d587420c7a116a7552&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLmRlMWFkM2IxY2JkODUxODVlZDJjMDFjYTJmZWE0NTEx&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "Width": 576
+ },
+ "QualityType": 28,
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 254747}, {\\\"time\\\": 2, \\\"offset\\\": 310409}, {\\\"time\\\": 3, \\\"offset\\\": 378126}, {\\\"time\\\": 4, \\\"offset\\\": 458158}, {\\\"time\\\": 5, \\\"offset\\\": 546764}, {\\\"time\\\": 10, \\\"offset\\\": 946117}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 87.061, \\\"v960\\\": 89.844, \\\"v864\\\": 91.813, \\\"v720\\\": 94.499}, \\\"ori\\\": {\\\"v1080\\\": 77.717, \\\"v960\\\": 80.972, \\\"v864\\\": 83.632, \\\"v720\\\": 87.945}}}\",\"ufq\":\"{\\\"enh\\\":89.767,\\\"playback\\\":{\\\"ori\\\":73.304,\\\"srv1\\\":82.648},\\\"src\\\":85.957,\\\"trans\\\":73.304,\\\"version\\\":\\\"v1.2\\\"}\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"86fda13c754bde67159dea14b7d55d6a\",\"dec_info\":\"\",\"gearvqm\":\"{\\\"v3.0\\\": {\\\"ori\\\": 87.963, \\\"sr20\\\": 95.722}}\",\"audio_bit_rate\":64143,\"ps_info\":\"{\\\"vps\\\": \\\"AQwC//8BYAAAAwAAAwAAAwAAAwC6AACIJElgSA==\\\", \\\"sps\\\": \\\"AQMBYAAAAwAAAwAAAwAAAwC6AACgBIIAQBZYgkSXJIkQTPCEREREREYR8xPOX+/+v82fy/+v8yh+S/3/1/mz+X/1/jOEAg5qAgICCAAAAwAIAAADAPBA\\\", \\\"pps\\\": \\\"AcElPAiQ\\\"}\"}"
+ },
+ {
+ "Bitrate": 622546,
+ "BitrateFPS": 29,
+ "CodecType": "h264",
+ "Format": "mp4",
+ "GearName": "lowest_540_0",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 72.483, \"v960\": 75.424, \"v864\": 79.843, \"v720\": 84.897}, \"ori\": {\"v1080\": 64.527, \"v960\": 67.93, \"v864\": 72.279, \"v720\": 74.495}}}",
+ "PlayAddr": {
+ "DataSize": 4829948,
+ "FileCs": "c:0-51513-3d83",
+ "FileHash": "a74fd8fabed8b36131a8e31bf78e9d1d",
+ "Height": 1024,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_h264_540p_622546",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/o8vENAqIe9xD3BG8AvVUvfEqFEARhgPpHGoTJQ/?a=1988&bti=ODszNWYuMDE6&&bt=607&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=aGhoaDQ8ZTdlNGhmM2Q8ZkBpanJneW45cmVpOzMzaTczNEA1NjRgNTJjNi8xMl42NmIuYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=fa8ddc2bcbad8ebb4394c4a8b55b28df&tk=tt_chain_token&btag=e00088000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/o8vENAqIe9xD3BG8AvVUvfEqFEARhgPpHGoTJQ/?a=1988&bti=ODszNWYuMDE6&&bt=607&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=aGhoaDQ8ZTdlNGhmM2Q8ZkBpanJneW45cmVpOzMzaTczNEA1NjRgNTJjNi8xMl42NmIuYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=fa8ddc2bcbad8ebb4394c4a8b55b28df&tk=tt_chain_token&btag=e00088000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=1ce99a90ac5e412b94fb5981ce7e9e8e&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjU0ZWZiNDU4NWVmYzFlY2IyNmEyODAyMGY0ZWJjMGZj&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "Width": 576
+ },
+ "QualityType": 25,
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 176720}, {\\\"time\\\": 2, \\\"offset\\\": 268125}, {\\\"time\\\": 3, \\\"offset\\\": 342814}, {\\\"time\\\": 4, \\\"offset\\\": 414132}, {\\\"time\\\": 5, \\\"offset\\\": 511717}, {\\\"time\\\": 10, \\\"offset\\\": 841405}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 72.483, \\\"v960\\\": 75.424, \\\"v864\\\": 79.843, \\\"v720\\\": 84.897}, \\\"ori\\\": {\\\"v1080\\\": 64.527, \\\"v960\\\": 67.93, \\\"v864\\\": 72.279, \\\"v720\\\": 74.495}}}\",\"ufq\":\"\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"25b3faebdda74a9e438e851988a6efcc\",\"dec_info\":\"\",\"gearvqm\":\"\",\"audio_bit_rate\":32099}"
+ }
+ ],
+ "claInfo": {
+ "enableAutoCaption": true,
+ "hasOriginalAudio": true,
+ "noCaptionReason": 3
+ },
+ "codecType": "h264",
+ "cover": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-tiktokx-origin.image?dr=9636&x-expires=1783695600&x-signature=ZIp3Rdhs6GNzMatSMsNWHFZX%2F7s%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5",
+ "definition": "720p",
+ "duration": 62,
+ "dynamicCover": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oYVoPG0RxQTGE89EEqvAuADpFvfGABVMxQBeza~tplv-tiktokx-origin.image?dr=9636&x-expires=1783695600&x-signature=JWF%2F5g2txpFXhNhJzOwleSJgKik%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5",
+ "encodeUserTag": "",
+ "encodedType": "normal",
+ "format": "mp4",
+ "height": 1280,
+ "id": "7655821461772406047",
+ "originCover": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oYVoQGFsU9VvLxieEDDAQAPpRTqA8B5EzvEGfF~tplv-tiktokx-origin.image?dr=9636&x-expires=1783695600&x-signature=yHDGczHGo3joyDeerZP6Dp4S1Jc%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5",
+ "playAddr": "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12CGneI3wUcfzAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698153&l=2026070815413036F7D0D0506FB569C78C&ply_type=2&policy=2&signature=aa5be80671c4a73d54517ee8e4a471ad&tk=tt_chain_token&btag=e00088000",
+ "ratio": "720p",
+ "size": 13761060,
+ "videoID": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "videoQuality": "normal",
+ "volumeInfo": {
+ "Loudness": -48.5,
+ "Peak": 0.18408
+ },
+ "width": 720,
+ "zoomCover": {
+ "240": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:240:240.avif?dr=9616&x-expires=1783695600&x-signature=ZONw1V50ZmupszDX0NiC%2Bf8BimU%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1",
+ "480": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:480:480.avif?dr=9616&x-expires=1783695600&x-signature=LztWYGkvUC4%2BvhtQb0TO0l1ky0s%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1",
+ "720": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:720:720.avif?dr=9616&x-expires=1783695600&x-signature=X%2F3HQ%2BkiKqEAxnuk%2BdFfBfeOsko%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1",
+ "960": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:960:960.avif?dr=9616&x-expires=1783695600&x-signature=aJaYG%2BDAo%2BhO7nZ8t8XP7g0DArU%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1"
+ }
+ }
+}
\ No newline at end of file
diff --git a/surfsense_backend/tests/fixtures/tiktok/video_item_struct.json b/surfsense_backend/tests/fixtures/tiktok/video_item_struct.json
new file mode 100644
index 000000000..af3bd651b
--- /dev/null
+++ b/surfsense_backend/tests/fixtures/tiktok/video_item_struct.json
@@ -0,0 +1,660 @@
+{
+ "id": "7655821461772406047",
+ "desc": "Pull Apart Garlic Bread Ingredients * 1 tsp yeast * 1 tbsp sugar * 270g milk (about 1 cup + 2 tbsp) * 380g flour (about 3 cups) * 1 tsp salt * 43g softened butter (about 3 tbsp) Butter Mixture * 100g butter, softened * 1 tbsp chopped cilantro * 1 garlic clove, minced * Shredded cheese Instructions 1. In a bowl, mix the sugar and yeast. Add the milk, then add the flour and mix until combined. 2. Add the salt and softened butter, then coil fold until the butter is fully incorporated. 3. Cover and let the dough rest for 2 hours or until doubled. 4. Mix the softened butter with the cilantro and minced garlic. 5. Roll the dough into a rectangle and spread the garlic butter mixture evenly over it. Sprinkle with shredded cheddar cheese. 6. Cut the dough in half, stack one half on top of the other, then cut into strips. Transfer the pieces into a loaf pan. 7. Cover and let rest for 20 minutes. 8. Bake at 370°F for 30 minutes, or until golden brown. 9. Brush with the remaining garlic butter mixture while still warm. #bread #garlicbread #breadtok #baking #homemadebread ",
+ "createTime": "1782509863",
+ "scheduleTime": 0,
+ "video": {
+ "id": "7655821461772406047",
+ "height": 1280,
+ "width": 720,
+ "duration": 62,
+ "ratio": "720p",
+ "cover": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-tiktokx-origin.image?dr=9636&x-expires=1783695600&x-signature=ZIp3Rdhs6GNzMatSMsNWHFZX%2F7s%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5",
+ "originCover": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oYVoQGFsU9VvLxieEDDAQAPpRTqA8B5EzvEGfF~tplv-tiktokx-origin.image?dr=9636&x-expires=1783695600&x-signature=yHDGczHGo3joyDeerZP6Dp4S1Jc%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5",
+ "dynamicCover": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oYVoPG0RxQTGE89EEqvAuADpFvfGABVMxQBeza~tplv-tiktokx-origin.image?dr=9636&x-expires=1783695600&x-signature=Jz%2BPtpSXdbjTV4qFbG4H7JFD0f8%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5",
+ "playAddr": "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=1bb5fb748a97085b872e357d1cd5406e&tk=tt_chain_token&btag=e00090000",
+ "downloadAddr": "",
+ "shareCover": [
+ ""
+ ],
+ "reflowCover": "",
+ "bitrate": 1773703,
+ "encodedType": "normal",
+ "format": "mp4",
+ "videoQuality": "normal",
+ "encodeUserTag": "",
+ "codecType": "h264",
+ "definition": "720p",
+ "subtitleInfos": [],
+ "zoomCover": {
+ "240": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:240:240.avif?dr=9616&x-expires=1783695600&x-signature=cBXoX16WOKJyUth9fnaohYv%2B%2BBs%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1",
+ "480": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:480:480.avif?dr=9616&x-expires=1783695600&x-signature=Ff5B1aV4teSPQc6JXMoidMXqJWk%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1",
+ "720": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:720:720.avif?dr=9616&x-expires=1783695600&x-signature=X%2F3HQ%2BkiKqEAxnuk%2BdFfBfeOsko%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1",
+ "960": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-p-0068-tx2/oAiefnCIc6NNSgtgrKLRA8JGAMoVfgJSe8eiRS~tplv-photomode-zoomcover:960:960.avif?dr=9616&x-expires=1783695600&x-signature=aJaYG%2BDAo%2BhO7nZ8t8XP7g0DArU%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=43f4a2f9&idc=useast5&ftpl=1"
+ },
+ "volumeInfo": {
+ "Loudness": -48.5,
+ "Peak": 0.18408
+ },
+ "bitrateInfo": [
+ {
+ "Bitrate": 1836180,
+ "QualityType": 2,
+ "BitrateFPS": 29,
+ "GearName": "adapt_lowest_1080_1",
+ "PlayAddr": {
+ "DataSize": "14245777",
+ "Width": 1080,
+ "Height": 1920,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oIygxEfEVAP1qvGVTIBARoAvb9QYFGEKDp5v9e/?a=1988&bti=ODszNWYuMDE6&&bt=1793&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=aDlmNGg7ZTdlaGY5ZzUzZ0BpanJneW45cmVpOzMzaTczNEBjXl5hMzAvXy8xNDAuYmIyYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=0648319cea95ee08fd7b4f93a95cb14c&tk=tt_chain_token&btag=e00090000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oIygxEfEVAP1qvGVTIBARoAvb9QYFGEKDp5v9e/?a=1988&bti=ODszNWYuMDE6&&bt=1793&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=aDlmNGg7ZTdlaGY5ZzUzZ0BpanJneW45cmVpOzMzaTczNEBjXl5hMzAvXy8xNDAuYmIyYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=0648319cea95ee08fd7b4f93a95cb14c&tk=tt_chain_token&btag=e00090000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=5de0e21d367c4e2bb42b386ed1aca6fe&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjAzMmU1YzJlMTAwMDJiYjIwMmFiN2RhNTIzOWVhYWRk&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_bytevc1_1080p_1836180",
+ "FileHash": "38996a98cfeb2714fca1b94768090c37",
+ "FileCs": "c:0-52754-ed06"
+ },
+ "CodecType": "h265_hvc1",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 96.83, \"v960\": 97.64, \"v864\": 98.256, \"v720\": 98.838}, \"ori\": {\"v1080\": 91.834, \"v960\": 93.423, \"v864\": 94.54, \"v720\": 95.872}}}",
+ "Format": "mp4",
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 385342}, {\\\"time\\\": 2, \\\"offset\\\": 561014}, {\\\"time\\\": 3, \\\"offset\\\": 703814}, {\\\"time\\\": 4, \\\"offset\\\": 855867}, {\\\"time\\\": 5, \\\"offset\\\": 1019333}, {\\\"time\\\": 10, \\\"offset\\\": 2028412}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 96.83, \\\"v960\\\": 97.64, \\\"v864\\\": 98.256, \\\"v720\\\": 98.838}, \\\"ori\\\": {\\\"v1080\\\": 91.834, \\\"v960\\\": 93.423, \\\"v864\\\": 94.54, \\\"v720\\\": 95.872}}}\",\"ufq\":\"{\\\"enh\\\":89.317,\\\"playback\\\":{\\\"ori\\\":85.151,\\\"srv1\\\":90.147},\\\"src\\\":85.957,\\\"trans\\\":85.151,\\\"version\\\":\\\"v1.2\\\"}\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"11b0345f3bf185ddbe7c48bd7301980f\",\"dec_info\":\"\",\"gearvqm\":\"\",\"audio_bit_rate\":96215,\"ps_info\":\"{\\\"vps\\\": \\\"AQwC//8BYAAAAwAAAwAAAwAAAwCWAACdzkgS\\\", \\\"sps\\\": \\\"AQMBYAAAAwAAAwAAAwAAAwCWAACgAhyAHgWWdzkySUQs0RJJJJify4d/1+bPl/1+M4gQ5qAgICCAAAADAIAAAA8E\\\", \\\"pps\\\": \\\"AcElPAiQ\\\"}\"}"
+ },
+ {
+ "Bitrate": 1773703,
+ "QualityType": 10,
+ "BitrateFPS": 29,
+ "GearName": "normal_720_0",
+ "PlayAddr": {
+ "DataSize": "13761060",
+ "Width": 720,
+ "Height": 1280,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=1bb5fb748a97085b872e357d1cd5406e&tk=tt_chain_token&btag=e00090000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=1bb5fb748a97085b872e357d1cd5406e&tk=tt_chain_token&btag=e00090000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=9de1abcaf21d44d081117ee2ca3d4cc2&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjcwMGExZmI3NDg4YzFiZTIwZjczM2IyMmM0OTk2YTI2&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_h264_720p_1773703",
+ "FileHash": "485174c8492db027adbbf6dca7c7c201",
+ "FileCs": "c:0-51456-f8d1"
+ },
+ "CodecType": "h264",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 84.77, \"v960\": 88.554, \"v864\": 88.982, \"v720\": 92.772}, \"ori\": {\"v1080\": 84.77, \"v960\": 88.554, \"v864\": 88.982, \"v720\": 92.772}}}",
+ "Format": "mp4",
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 400712}, {\\\"time\\\": 2, \\\"offset\\\": 659243}, {\\\"time\\\": 3, \\\"offset\\\": 886968}, {\\\"time\\\": 4, \\\"offset\\\": 1101663}, {\\\"time\\\": 5, \\\"offset\\\": 1368565}, {\\\"time\\\": 10, \\\"offset\\\": 2375704}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 84.77, \\\"v960\\\": 88.554, \\\"v864\\\": 88.982, \\\"v720\\\": 92.772}, \\\"ori\\\": {\\\"v1080\\\": 84.77, \\\"v960\\\": 88.554, \\\"v864\\\": 88.982, \\\"v720\\\": 92.772}}}\",\"ufq\":\"\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"65b88c41c0963253ad31b6f68981a242\",\"dec_info\":\"\",\"gearvqm\":\"\",\"audio_bit_rate\":64193}"
+ },
+ {
+ "Bitrate": 966380,
+ "QualityType": 14,
+ "BitrateFPS": 29,
+ "GearName": "adapt_lower_720_1",
+ "PlayAddr": {
+ "DataSize": "7497540",
+ "Width": 720,
+ "Height": 1280,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/og99rgOGqJED8xGeBpFEAADvEPARnbTIQTfoVv/?a=1988&bti=ODszNWYuMDE6&&bt=943&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=OTNkNDk2NzlmM2VkPGg7OkBpanJneW45cmVpOzMzaTczNEBeMmBiYWBfNWIxLzEyNF81YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=3a0c7313f1a5d9908cc2f5b0dd24d048&tk=tt_chain_token&btag=e00090000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/og99rgOGqJED8xGeBpFEAADvEPARnbTIQTfoVv/?a=1988&bti=ODszNWYuMDE6&&bt=943&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=OTNkNDk2NzlmM2VkPGg7OkBpanJneW45cmVpOzMzaTczNEBeMmBiYWBfNWIxLzEyNF81YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=3a0c7313f1a5d9908cc2f5b0dd24d048&tk=tt_chain_token&btag=e00090000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=87b87e594a2b40ec86b4c3d6e1ca50a6&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLmRjNWNmZGVmMmFkNTIyNWQ1ZTA1NzExM2NjOTVlNzMy&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_bytevc1_720p_966380",
+ "FileHash": "1376df24c917669727b27273697fb826",
+ "FileCs": "c:0-52786-7f65"
+ },
+ "CodecType": "h265_hvc1",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 90.05, \"v960\": 91.958, \"v864\": 93.608, \"v720\": 95.869}, \"ori\": {\"v1080\": 82.259, \"v960\": 85.339, \"v864\": 87.288, \"v720\": 90.851}}}",
+ "Format": "mp4",
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 284307}, {\\\"time\\\": 2, \\\"offset\\\": 357012}, {\\\"time\\\": 3, \\\"offset\\\": 438159}, {\\\"time\\\": 4, \\\"offset\\\": 532720}, {\\\"time\\\": 5, \\\"offset\\\": 636824}, {\\\"time\\\": 10, \\\"offset\\\": 1142427}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 90.05, \\\"v960\\\": 91.958, \\\"v864\\\": 93.608, \\\"v720\\\": 95.869}, \\\"ori\\\": {\\\"v1080\\\": 82.259, \\\"v960\\\": 85.339, \\\"v864\\\": 87.288, \\\"v720\\\": 90.851}}}\",\"ufq\":\"{\\\"enh\\\":89.767,\\\"playback\\\":{\\\"ori\\\":76.711,\\\"srv1\\\":84.502},\\\"src\\\":85.957,\\\"trans\\\":76.711,\\\"version\\\":\\\"v1.2\\\"}\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"97eb57dffd9e7a0333979f6ca6c9418d\",\"dec_info\":\"\",\"gearvqm\":\"{\\\"v3.0\\\": {\\\"ori\\\": 91.625, \\\"sr20\\\": 97.355}}\",\"audio_bit_rate\":96215,\"ps_info\":\"{\\\"vps\\\": \\\"AQwC//8BYAAAAwAAAwAAAwAAAwC6AACIJElgSA==\\\", \\\"sps\\\": \\\"AQMBYAAAAwAAAwAAAwAAAwC6AACgBaIAUBZYgkSXJIkQTPCEREREREYR8xPOX+/+v82fy/+v8yh+S/3/1/mz+X/1/jOEAg5qAgICCAAAAwAIAAADAPBA\\\", \\\"pps\\\": \\\"AcElPAiQ\\\"}\"}"
+ },
+ {
+ "Bitrate": 799620,
+ "QualityType": 28,
+ "BitrateFPS": 29,
+ "GearName": "adapt_540_1",
+ "PlayAddr": {
+ "DataSize": "6203756",
+ "Width": 576,
+ "Height": 1024,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/oogvkDPHIAVvTxgEpE9EMoALxQRRA1FGqeB9Gf/?a=1988&bti=ODszNWYuMDE6&&bt=780&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=NThkNGRmNTo4MztpOTM3ZkBpanJneW45cmVpOzMzaTczNEAyNDMuNC0xNWExYjBiX2FgYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=a8516d4d8e79814753dc2cd0fde9f7bd&tk=tt_chain_token&btag=e00090000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/oogvkDPHIAVvTxgEpE9EMoALxQRRA1FGqeB9Gf/?a=1988&bti=ODszNWYuMDE6&&bt=780&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=NThkNGRmNTo4MztpOTM3ZkBpanJneW45cmVpOzMzaTczNEAyNDMuNC0xNWExYjBiX2FgYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=a8516d4d8e79814753dc2cd0fde9f7bd&tk=tt_chain_token&btag=e00090000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=ccd4b0e63fe840d587420c7a116a7552&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLmRlMWFkM2IxY2JkODUxODVlZDJjMDFjYTJmZWE0NTEx&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_bytevc1_540p_799620",
+ "FileHash": "41f62f740ad13045000450f28b9ff2cb",
+ "FileCs": "c:0-52786-7f6d"
+ },
+ "CodecType": "h265_hvc1",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 87.061, \"v960\": 89.844, \"v864\": 91.813, \"v720\": 94.499}, \"ori\": {\"v1080\": 77.717, \"v960\": 80.972, \"v864\": 83.632, \"v720\": 87.945}}}",
+ "Format": "mp4",
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 254747}, {\\\"time\\\": 2, \\\"offset\\\": 310409}, {\\\"time\\\": 3, \\\"offset\\\": 378126}, {\\\"time\\\": 4, \\\"offset\\\": 458158}, {\\\"time\\\": 5, \\\"offset\\\": 546764}, {\\\"time\\\": 10, \\\"offset\\\": 946117}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 87.061, \\\"v960\\\": 89.844, \\\"v864\\\": 91.813, \\\"v720\\\": 94.499}, \\\"ori\\\": {\\\"v1080\\\": 77.717, \\\"v960\\\": 80.972, \\\"v864\\\": 83.632, \\\"v720\\\": 87.945}}}\",\"ufq\":\"{\\\"enh\\\":89.767,\\\"playback\\\":{\\\"ori\\\":73.304,\\\"srv1\\\":82.648},\\\"src\\\":85.957,\\\"trans\\\":73.304,\\\"version\\\":\\\"v1.2\\\"}\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"86fda13c754bde67159dea14b7d55d6a\",\"dec_info\":\"\",\"gearvqm\":\"{\\\"v3.0\\\": {\\\"ori\\\": 87.963, \\\"sr20\\\": 95.722}}\",\"audio_bit_rate\":64143,\"ps_info\":\"{\\\"vps\\\": \\\"AQwC//8BYAAAAwAAAwAAAwAAAwC6AACIJElgSA==\\\", \\\"sps\\\": \\\"AQMBYAAAAwAAAwAAAwAAAwC6AACgBIIAQBZYgkSXJIkQTPCEREREREYR8xPOX+/+v82fy/+v8yh+S/3/1/mz+X/1/jOEAg5qAgICCAAAAwAIAAADAPBA\\\", \\\"pps\\\": \\\"AcElPAiQ\\\"}\"}"
+ },
+ {
+ "Bitrate": 622546,
+ "QualityType": 25,
+ "BitrateFPS": 29,
+ "GearName": "lowest_540_0",
+ "PlayAddr": {
+ "DataSize": "4829948",
+ "Width": 576,
+ "Height": 1024,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/o8vENAqIe9xD3BG8AvVUvfEqFEARhgPpHGoTJQ/?a=1988&bti=ODszNWYuMDE6&&bt=607&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=aGhoaDQ8ZTdlNGhmM2Q8ZkBpanJneW45cmVpOzMzaTczNEA1NjRgNTJjNi8xMl42NmIuYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=f44ac110e4573a7bc11c73155b9141d0&tk=tt_chain_token&btag=e00090000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-ve-0068c001-tx2/o8vENAqIe9xD3BG8AvVUvfEqFEARhgPpHGoTJQ/?a=1988&bti=ODszNWYuMDE6&&bt=607&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=aGhoaDQ8ZTdlNGhmM2Q8ZkBpanJneW45cmVpOzMzaTczNEA1NjRgNTJjNi8xMl42NmIuYSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=f44ac110e4573a7bc11c73155b9141d0&tk=tt_chain_token&btag=e00090000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=1ce99a90ac5e412b94fb5981ce7e9e8e&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjU0ZWZiNDU4NWVmYzFlY2IyNmEyODAyMGY0ZWJjMGZj&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_h264_540p_622546",
+ "FileHash": "a74fd8fabed8b36131a8e31bf78e9d1d",
+ "FileCs": "c:0-51513-3d83"
+ },
+ "CodecType": "h264",
+ "MVMAF": "{\"v2.0\": {\"srv1\": {\"v1080\": 72.483, \"v960\": 75.424, \"v864\": 79.843, \"v720\": 84.897}, \"ori\": {\"v1080\": 64.527, \"v960\": 67.93, \"v864\": 72.279, \"v720\": 74.495}}}",
+ "Format": "mp4",
+ "VideoExtra": "{\"PktOffsetMap\":\"[{\\\"time\\\": 1, \\\"offset\\\": 176720}, {\\\"time\\\": 2, \\\"offset\\\": 268125}, {\\\"time\\\": 3, \\\"offset\\\": 342814}, {\\\"time\\\": 4, \\\"offset\\\": 414132}, {\\\"time\\\": 5, \\\"offset\\\": 511717}, {\\\"time\\\": 10, \\\"offset\\\": 841405}]\",\"mvmaf\":\"{\\\"v2.0\\\": {\\\"srv1\\\": {\\\"v1080\\\": 72.483, \\\"v960\\\": 75.424, \\\"v864\\\": 79.843, \\\"v720\\\": 84.897}, \\\"ori\\\": {\\\"v1080\\\": 64.527, \\\"v960\\\": 67.93, \\\"v864\\\": 72.279, \\\"v720\\\": 74.495}}}\",\"ufq\":\"\",\"volume_info_json\":\"\",\"transcode_feature_id\":\"25b3faebdda74a9e438e851988a6efcc\",\"dec_info\":\"\",\"gearvqm\":\"\",\"audio_bit_rate\":32099}"
+ }
+ ],
+ "size": "13761060",
+ "VQScore": "69.69",
+ "claInfo": {
+ "hasOriginalAudio": true,
+ "enableAutoCaption": true,
+ "captionInfos": [],
+ "noCaptionReason": 3
+ },
+ "videoID": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "PlayAddrStruct": {
+ "DataSize": "13761060",
+ "Width": 720,
+ "Height": 1280,
+ "Uri": "v15044gf0000d8vf0bvog65kkvkd8ohg",
+ "UrlList": [
+ "https://v16-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=1bb5fb748a97085b872e357d1cd5406e&tk=tt_chain_token&btag=e00090000",
+ "https://v19-webapp-prime.us.tiktok.com/video/tos/useast8/tos-useast8-pve-0068-tx2/oYDTerfAgGxvqFaEctpBAvVGEPIE8Q9ARoxPNB/?a=1988&bti=ODszNWYuMDE6&&bt=1732&ft=aEeq8qT0mIoPD12qGneI3wU6LRAbMeF~O5&mime_type=video_mp4&rc=NWZnZDVoNDQzN2c1ZGU0Z0BpanJneW45cmVpOzMzaTczNEBgMTBgYjQuNWExLmNgXmI2YSNlNWhoMmRzLXNhLS1kMTJzcw%3D%3D&expire=1783698158&l=20260708154135271AF032BAC01F42EF1E&ply_type=2&policy=2&signature=1bb5fb748a97085b872e357d1cd5406e&tk=tt_chain_token&btag=e00090000",
+ "https://www.tiktok.com/aweme/v1/play/?faid=1988&file_id=9de1abcaf21d44d081117ee2ca3d4cc2&is_play_url=1&item_id=7655821461772406047&line=0&ply_type=2&signaturev3=dmlkZW9faWQ7ZmlsZV9pZDtpdGVtX2lkLjcwMGExZmI3NDg4YzFiZTIwZjczM2IyMmM0OTk2YTI2&tk=tt_chain_token&video_id=v15044gf0000d8vf0bvog65kkvkd8ohg"
+ ],
+ "UrlKey": "v15044gf0000d8vf0bvog65kkvkd8ohg_h264_720p_1773703",
+ "FileHash": "485174c8492db027adbbf6dca7c7c201",
+ "FileCs": "c:0-51456-f8d1"
+ }
+ },
+ "author": {
+ "id": "7501138183701103662",
+ "shortId": "",
+ "uniqueId": "cookwithhali",
+ "nickname": "Cookwithhali",
+ "avatarLarger": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:1080:1080.jpeg?dr=9640&refresh_token=c6d77c8a&x-expires=1783695600&x-signature=mUTuvDWLEGrAxXtWZlBE7gUxCl8%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "avatarMedium": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:720:720.jpeg?dr=9640&refresh_token=dfe18cb8&x-expires=1783695600&x-signature=cP14vkr0ndxwhW%2B1pWfefXfTsls%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "avatarThumb": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:100:100.jpeg?dr=9640&refresh_token=84bba9f5&x-expires=1783695600&x-signature=Et06On6iQeXFbtycXoVodzl0fdM%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "signature": "From my kitchen to your screen! 🍞\nFollow for more ✨\nIG: halikit25\n\n📩 Halikitchen25@gmail.com",
+ "createTime": 1746494976,
+ "verified": false,
+ "secUid": "MS4wLjABAAAAywZ6lCXK3u4zKz8eQxxcjVblLiugz6zMysU98G7dyKVxu6IvMOJ97mpIfpDxUL4q",
+ "ftc": false,
+ "relation": 0,
+ "openFavorite": false,
+ "commentSetting": 0,
+ "duetSetting": 0,
+ "stitchSetting": 0,
+ "privateAccount": false,
+ "secret": false,
+ "isADVirtual": false,
+ "roomId": "",
+ "uniqueIdModifyTime": 0,
+ "ttSeller": false,
+ "downloadSetting": 3,
+ "recommendReason": "",
+ "nowInvitationCardUrl": "",
+ "nickNameModifyTime": 0,
+ "isEmbedBanned": false,
+ "canExpPlaylist": false,
+ "suggestAccountBind": false,
+ "UserStoryStatus": 0,
+ "shortDramaCreator": {}
+ },
+ "music": {
+ "id": "7655821492566919966",
+ "title": "original sound",
+ "playUrl": "https://v16m.tiktokcdn-us.com/84ccc1629d7199ad68f976d419148401/6a4ec44e/video/tos/useast8/tos-useast8-v-27dcd7-tx2/ocBIhANWILSemeAAINEGoTTPe58AJaRegg6eKS/?a=1233&bti=ODszNWYuMDE6&&bt=125&ft=GSDrKInz7ThasbSGXq8Zmo&mime_type=audio_mpeg&rc=anY5cXA5cmRpOzMzaTU8NEBpanY5cXA5cmRpOzMzaTU8NEBpYzZrMmRjL3NhLS1kMTJzYSNpYzZrMmRjL3NhLS1kMTJzcw%3D%3D&vvpl=1&l=20260708154135271AF032BAC01F42EF1E&btag=e00050000",
+ "coverLarge": "https://p19-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:1080:1080.jpeg?dr=9640&refresh_token=c6d77c8a&x-expires=1783695600&x-signature=mUTuvDWLEGrAxXtWZlBE7gUxCl8%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "coverMedium": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:720:720.jpeg?dr=9640&refresh_token=dfe18cb8&x-expires=1783695600&x-signature=cP14vkr0ndxwhW%2B1pWfefXfTsls%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "coverThumb": "https://p16-common-sign.tiktokcdn-us.com/tos-useast8-avt-0068-tx2/b29bd2d826cffba1af22f99793a7d5dc~tplv-tiktokx-cropcenter:100:100.jpeg?dr=9640&refresh_token=84bba9f5&x-expires=1783695600&x-signature=Et06On6iQeXFbtycXoVodzl0fdM%3D&t=4d5b0474&ps=13740610&shp=a5d48078&shcp=81f88b70&idc=useast5",
+ "authorName": "Cookwithhali",
+ "original": true,
+ "private": false,
+ "duration": 62,
+ "scheduleSearchTime": 0,
+ "collected": false,
+ "preciseDuration": {
+ "preciseDuration": 62.093063,
+ "preciseShootDuration": 62.093063,
+ "preciseAuditionDuration": 62.093063,
+ "preciseVideoDuration": 62.093063
+ },
+ "isCopyrighted": false,
+ "tt2dsp": {
+ "tt_to_dsp_song_infos": []
+ },
+ "is_unlimited_music": false,
+ "is_commerce_music": true,
+ "shoot_duration": 62
+ },
+ "challenges": [
+ {
+ "id": "6983",
+ "title": "bread",
+ "desc": "",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": ""
+ },
+ {
+ "id": "285169",
+ "title": "garlicbread",
+ "desc": "",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": ""
+ },
+ {
+ "id": "1643317566954502",
+ "title": "breadtok",
+ "desc": "",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": ""
+ },
+ {
+ "id": "7250",
+ "title": "baking",
+ "desc": "Whether you're baking the perfect pie or just searching for a recipe, you've come to the right place for all things #Baking.",
+ "profileLarger": "https://p19-common-sign.tiktokcdn-us.com/musically-maliva-obj/1aa8a54136dc06390b83508fdd683160~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=nigLPQXvvDM%2BxFQezn3ELBH%2BLwQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "profileMedium": "https://p19-common-sign.tiktokcdn-us.com/musically-maliva-obj/1aa8a54136dc06390b83508fdd683160~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=nigLPQXvvDM%2BxFQezn3ELBH%2BLwQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "profileThumb": "https://p19-common-sign.tiktokcdn-us.com/musically-maliva-obj/1aa8a54136dc06390b83508fdd683160~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=nigLPQXvvDM%2BxFQezn3ELBH%2BLwQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": ""
+ },
+ {
+ "id": "10488587",
+ "title": "homemadebread",
+ "desc": "",
+ "profileLarger": "",
+ "profileMedium": "",
+ "profileThumb": "",
+ "coverLarger": "",
+ "coverMedium": "",
+ "coverThumb": ""
+ }
+ ],
+ "stats": {
+ "diggCount": 754100,
+ "shareCount": 144500,
+ "commentCount": 3720,
+ "playCount": 8300000,
+ "collectCount": "396399"
+ },
+ "statsV2": {
+ "diggCount": "754100",
+ "shareCount": "144500",
+ "commentCount": "3720",
+ "playCount": "8300000",
+ "collectCount": "396399",
+ "repostCount": "0"
+ },
+ "warnInfo": [],
+ "originalItem": false,
+ "officalItem": false,
+ "textExtra": [
+ {
+ "awemeId": "",
+ "start": 1024,
+ "end": 1030,
+ "hashtagId": "6983",
+ "hashtagName": "bread",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 1031,
+ "end": 1043,
+ "hashtagId": "285169",
+ "hashtagName": "garlicbread",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 1044,
+ "end": 1053,
+ "hashtagId": "1643317566954502",
+ "hashtagName": "breadtok",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 1054,
+ "end": 1061,
+ "hashtagId": "7250",
+ "hashtagName": "baking",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 1062,
+ "end": 1076,
+ "hashtagId": "10488587",
+ "hashtagName": "homemadebread",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ }
+ ],
+ "secret": false,
+ "forFriend": false,
+ "digged": false,
+ "itemCommentStatus": 0,
+ "takeDown": 0,
+ "effectStickers": [],
+ "authorStats": {
+ "followerCount": 372600,
+ "followingCount": 0,
+ "heart": 8000000,
+ "heartCount": 8000000,
+ "videoCount": 279,
+ "diggCount": 223,
+ "friendCount": 0
+ },
+ "privateItem": false,
+ "duetEnabled": true,
+ "stitchEnabled": true,
+ "stickersOnItem": [
+ {
+ "stickerText": [
+ "Cheesy Pull Apart Garlic Bread! 🧄🧀\n",
+ "(Bakery-Style & No-Knead)"
+ ],
+ "stickerType": 4
+ }
+ ],
+ "isAd": false,
+ "shareEnabled": true,
+ "comments": [],
+ "duetDisplay": 0,
+ "stitchDisplay": 0,
+ "indexEnabled": true,
+ "diversificationLabels": [
+ "Cooking",
+ "Food & Drink",
+ "Lifestyle"
+ ],
+ "locationCreated": "US",
+ "suggestedWords": [
+ "Garlic Bread",
+ "garlic bread recipe",
+ "pull apart bread",
+ "Sourdough Bread",
+ "How To Make Garlic Bread",
+ "Baking",
+ "food",
+ "bread",
+ "Baking Recipes",
+ "Garlic Pull Apart Bread Recipe"
+ ],
+ "contents": [
+ {
+ "desc": "Pull Apart Garlic Bread ",
+ "textExtra": []
+ },
+ {
+ "desc": "",
+ "textExtra": []
+ },
+ {
+ "desc": "Ingredients",
+ "textExtra": []
+ },
+ {
+ "desc": "",
+ "textExtra": []
+ },
+ {
+ "desc": "* 1 tsp yeast",
+ "textExtra": []
+ },
+ {
+ "desc": "* 1 tbsp sugar",
+ "textExtra": []
+ },
+ {
+ "desc": "* 270g milk (about 1 cup + 2 tbsp)",
+ "textExtra": []
+ },
+ {
+ "desc": "* 380g flour (about 3 cups)",
+ "textExtra": []
+ },
+ {
+ "desc": "* 1 tsp salt",
+ "textExtra": []
+ },
+ {
+ "desc": "* 43g softened butter (about 3 tbsp)",
+ "textExtra": []
+ },
+ {
+ "desc": "",
+ "textExtra": []
+ },
+ {
+ "desc": "Butter Mixture",
+ "textExtra": []
+ },
+ {
+ "desc": "",
+ "textExtra": []
+ },
+ {
+ "desc": "* 100g butter, softened",
+ "textExtra": []
+ },
+ {
+ "desc": "* 1 tbsp chopped cilantro",
+ "textExtra": []
+ },
+ {
+ "desc": "* 1 garlic clove, minced",
+ "textExtra": []
+ },
+ {
+ "desc": "* Shredded cheese",
+ "textExtra": []
+ },
+ {
+ "desc": "",
+ "textExtra": []
+ },
+ {
+ "desc": "Instructions",
+ "textExtra": []
+ },
+ {
+ "desc": "",
+ "textExtra": []
+ },
+ {
+ "desc": "1. In a bowl, mix the sugar and yeast. Add the milk, then add the flour and mix until combined.",
+ "textExtra": []
+ },
+ {
+ "desc": "2. Add the salt and softened butter, then coil fold until the butter is fully incorporated.",
+ "textExtra": []
+ },
+ {
+ "desc": "3. Cover and let the dough rest for 2 hours or until doubled.",
+ "textExtra": []
+ },
+ {
+ "desc": "4. Mix the softened butter with the cilantro and minced garlic.",
+ "textExtra": []
+ },
+ {
+ "desc": "5. Roll the dough into a rectangle and spread the garlic butter mixture evenly over it. Sprinkle with shredded cheddar cheese.",
+ "textExtra": []
+ },
+ {
+ "desc": "6. Cut the dough in half, stack one half on top of the other, then cut into strips. Transfer the pieces into a loaf pan.",
+ "textExtra": []
+ },
+ {
+ "desc": "7. Cover and let rest for 20 minutes.",
+ "textExtra": []
+ },
+ {
+ "desc": "8. Bake at 370°F for 30 minutes, or until golden brown.",
+ "textExtra": []
+ },
+ {
+ "desc": "9. Brush with the remaining garlic butter mixture while still warm.",
+ "textExtra": []
+ },
+ {
+ "desc": "",
+ "textExtra": []
+ },
+ {
+ "desc": "#bread #garlicbread #breadtok #baking #homemadebread ",
+ "textExtra": [
+ {
+ "awemeId": "",
+ "start": 0,
+ "end": 6,
+ "hashtagId": "6983",
+ "hashtagName": "bread",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 7,
+ "end": 19,
+ "hashtagId": "285169",
+ "hashtagName": "garlicbread",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 20,
+ "end": 29,
+ "hashtagId": "1643317566954502",
+ "hashtagName": "breadtok",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 30,
+ "end": 37,
+ "hashtagId": "7250",
+ "hashtagName": "baking",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ },
+ {
+ "awemeId": "",
+ "start": 38,
+ "end": 52,
+ "hashtagId": "10488587",
+ "hashtagName": "homemadebread",
+ "type": 1,
+ "subType": 0,
+ "isCommerce": false
+ }
+ ]
+ }
+ ],
+ "diversificationId": 10040,
+ "anchors": [
+ {
+ "id": "0",
+ "type": 54,
+ "keyword": "CapCut · Editing made easy",
+ "icon": {
+ "uri": "tiktok-obj/capcut_logo_64px_bk.png",
+ "urlList": [
+ "https://p19-common-sign.tiktokcdn-us.com/tiktok-obj/capcut_logo_64px_bk.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=lMwgk8e0wLaM3WkpnsMqYcxZN6w%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p16-common-sign.tiktokcdn-us.com/tiktok-obj/capcut_logo_64px_bk.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=44xwWHkyYJdE%2FIwqS3zAAjU4XQ4%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p19-common-sign.tiktokcdn-us.com/tiktok-obj/capcut_logo_64px_bk.png~tplv-tiktokx-origin.jpeg?dr=9627&x-expires=1783544400&x-signature=lzVxquDTAbut%2BvshGEhsFsYQc2s%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5"
+ ]
+ },
+ "schema": "https://www.capcut.com/tools/desktop-video-editor?channel=capcutpc_ttweb_anchor_edit1&download_channel=capcutpc_ttweb_anchor_edit1&enter_from=capcutpc_ttweb_anchor_edit1&force_dp=1&from_page=capcutpc_ttweb_anchor_edit1&type=tools",
+ "logExtra": "{\"anchor_id\":0,\"anchor_name\":\"CapCut · Editing made easy\",\"anchor_title_detail\":\"None\",\"anchor_title_id\":21502485763,\"anchor_type\":\"TT_CAPCUT\",\"capability_key\":\"default\",\"ccep_vip\":0,\"if_race_trigger\":0,\"is_two_line\":0,\"landing_page_style_id\":21502486275,\"maker_source\":\"\",\"publish_key\":\"default\",\"template_id\":\"none\",\"tutorial_id\":\"none\",\"viamaker_anchor_capability_key_weight\":1,\"viamaker_anchor_style_idx\":-1,\"viamaker_anchor_style_source\":3,\"video_source\":0,\"video_type_id\":1}",
+ "description": "CapCut · Video Editor",
+ "thumbnail": {
+ "uri": "tiktok-obj/64x_Capcut3x.png",
+ "urlList": [
+ "https://p16-common-sign.tiktokcdn-us.com/tiktok-obj/64x_Capcut3x.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=SJ6q86bjtCoRArlVNS1DjDDFWoc%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p19-common-sign.tiktokcdn-us.com/tiktok-obj/64x_Capcut3x.png~tplv-tiktokx-origin.image?dr=9627&x-expires=1783544400&x-signature=GOZUldk%2BrxINygKT0jwxeJz9VLQ%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5",
+ "https://p16-common-sign.tiktokcdn-us.com/tiktok-obj/64x_Capcut3x.png~tplv-tiktokx-origin.jpeg?dr=9627&x-expires=1783544400&x-signature=NBemEand1jjAoV81cQ1ANQzVqqc%3D&t=4d5b0474&ps=13740610&shp=81f88b70&shcp=9b759fb9&idc=useast5"
+ ],
+ "width": 64,
+ "height": 64
+ },
+ "extraInfo": {
+ "subtype": ""
+ }
+ }
+ ],
+ "collected": false,
+ "channelTags": [],
+ "item_control": {
+ "can_repost": true
+ },
+ "IsAigc": false,
+ "AIGCDescription": "",
+ "backendSourceEventTracking": "",
+ "CategoryType": 111,
+ "textLanguage": "en",
+ "textTranslatable": true,
+ "authorStatsV2": {
+ "followerCount": "372600",
+ "followingCount": "0",
+ "heart": "8000000",
+ "heartCount": "8000000",
+ "videoCount": "279",
+ "diggCount": "223",
+ "friendCount": "0"
+ },
+ "isReviewing": false,
+ "creatorAIComment": {
+ "hasAITopic": false,
+ "categoryList": [],
+ "eligibleVideo": false,
+ "notEligibleReason": 101
+ }
+}
\ No newline at end of file
diff --git a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py
index c3ad04250..ee889cd17 100644
--- a/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py
+++ b/surfsense_backend/tests/unit/agents/multi_agent_chat/test_subagent_composition.py
@@ -32,11 +32,13 @@ _EXPECTED_SUBAGENTS = frozenset(
"google_drive",
"google_maps",
"google_search",
+ "instagram",
"knowledge_base",
"mcp_discovery",
"memory",
"onedrive",
"reddit",
+ "tiktok",
"web_crawler",
"youtube",
}
diff --git a/surfsense_backend/tests/unit/agents/test_video_presentation_graph.py b/surfsense_backend/tests/unit/agents/test_video_presentation_graph.py
new file mode 100644
index 000000000..3283af782
--- /dev/null
+++ b/surfsense_backend/tests/unit/agents/test_video_presentation_graph.py
@@ -0,0 +1,144 @@
+from __future__ import annotations
+
+import importlib
+import operator
+import sys
+import types
+from typing import Annotated, Any
+
+import pytest
+from langgraph.graph import END, START, StateGraph
+from typing_extensions import TypedDict
+
+pytestmark = pytest.mark.unit
+
+_GRAPH_MODULE = "app.agents.video_presentation.graph"
+_NODES_MODULE = "app.agents.video_presentation.nodes"
+_TARGET_NODE = "generate_slide_scene_codes"
+_BARRIER_SOURCES = ("create_slide_audio", "assign_slide_themes")
+
+
+class _DelayedState(TypedDict, total=False):
+ calls: Annotated[list[str], operator.add]
+ audio_ready: bool
+ themes_ready: bool
+
+
+def _config(thread_id: str = "video-presentation-graph-test") -> dict[str, Any]:
+ return {
+ "configurable": {
+ "search_space_id": 1,
+ "thread_id": thread_id,
+ "video_title": "Test deck",
+ }
+ }
+
+
+def _input_state() -> dict[str, Any]:
+ return {"db_session": None, "source_content": "source material"}
+
+
+def _build_video_graph(monkeypatch: pytest.MonkeyPatch, calls: list[str]):
+ nodes = types.ModuleType(_NODES_MODULE)
+
+ async def create_presentation_slides(_state, config):
+ _ = config
+ calls.append("create_presentation_slides")
+ return {"slides": [{"slide_number": 1, "title": "Intro"}]}
+
+ async def create_slide_audio(_state, config):
+ _ = config
+ calls.append("create_slide_audio")
+ return {"slide_audio_results": [{"slide_number": 1}]}
+
+ async def assign_slide_themes(_state, config):
+ _ = config
+ calls.append("assign_slide_themes")
+ return {"slide_theme_assignments": {1: ("corporate", "light")}}
+
+ async def generate_slide_scene_codes(state, config):
+ _ = config
+ calls.append(_TARGET_NODE)
+ assert state.slide_audio_results == [{"slide_number": 1}]
+ assert state.slide_theme_assignments == {1: ("corporate", "light")}
+ return {"slide_scene_codes": [{"slide_number": 1, "code": "", "title": ""}]}
+
+ nodes.create_presentation_slides = create_presentation_slides
+ nodes.create_slide_audio = create_slide_audio
+ nodes.assign_slide_themes = assign_slide_themes
+ nodes.generate_slide_scene_codes = generate_slide_scene_codes
+
+ monkeypatch.setitem(sys.modules, _NODES_MODULE, nodes)
+ monkeypatch.delitem(sys.modules, _GRAPH_MODULE, raising=False)
+
+ graph_module = importlib.import_module(_GRAPH_MODULE)
+ return graph_module.build_graph()
+
+
+@pytest.mark.asyncio
+async def test_video_presentation_graph_generates_scene_codes_once(monkeypatch):
+ calls: list[str] = []
+ graph = _build_video_graph(monkeypatch, calls)
+
+ await graph.ainvoke(_input_state(), _config())
+
+ assert calls.count(_TARGET_NODE) == 1
+ scene_index = calls.index(_TARGET_NODE)
+ assert calls.index("create_slide_audio") < scene_index
+ assert calls.index("assign_slide_themes") < scene_index
+
+
+def test_video_presentation_graph_registers_single_barrier_trigger(monkeypatch):
+ graph = _build_video_graph(monkeypatch, [])
+
+ assert graph.builder.waiting_edges == {(_BARRIER_SOURCES, _TARGET_NODE)}
+ assert not {
+ edge
+ for edge in graph.builder.edges
+ if edge[0] in _BARRIER_SOURCES and edge[1] == _TARGET_NODE
+ }
+
+ join_channel = f"join:{'+'.join(_BARRIER_SOURCES)}:{_TARGET_NODE}"
+ assert join_channel in graph.channels
+ assert graph.nodes[_TARGET_NODE].triggers.count(join_channel) == 1
+
+
+@pytest.mark.asyncio
+async def test_barrier_fires_once_when_one_branch_finishes_a_superstep_later():
+ def create_presentation_slides(_state):
+ return {"calls": ["create_presentation_slides"]}
+
+ def create_slide_audio(_state):
+ return {"calls": ["create_slide_audio"]}
+
+ def finish_slide_audio(_state):
+ return {"calls": ["finish_slide_audio"], "audio_ready": True}
+
+ def assign_slide_themes(_state):
+ return {"calls": ["assign_slide_themes"], "themes_ready": True}
+
+ def generate_slide_scene_codes(state):
+ assert state["audio_ready"] is True
+ assert state["themes_ready"] is True
+ return {"calls": [_TARGET_NODE]}
+
+ workflow = StateGraph(_DelayedState)
+ workflow.add_node("create_presentation_slides", create_presentation_slides)
+ workflow.add_node("create_slide_audio", create_slide_audio)
+ workflow.add_node("finish_slide_audio", finish_slide_audio)
+ workflow.add_node("assign_slide_themes", assign_slide_themes)
+ workflow.add_node(_TARGET_NODE, generate_slide_scene_codes)
+
+ workflow.add_edge(START, "create_presentation_slides")
+ workflow.add_edge("create_presentation_slides", "create_slide_audio")
+ workflow.add_edge("create_presentation_slides", "assign_slide_themes")
+ workflow.add_edge("create_slide_audio", "finish_slide_audio")
+ workflow.add_edge(["finish_slide_audio", "assign_slide_themes"], _TARGET_NODE)
+ workflow.add_edge(_TARGET_NODE, END)
+
+ result = await workflow.compile().ainvoke({"calls": []})
+
+ calls = result["calls"]
+ assert calls.count(_TARGET_NODE) == 1
+ assert calls.index("assign_slide_themes") < calls.index(_TARGET_NODE)
+ assert calls.index("finish_slide_audio") < calls.index(_TARGET_NODE)
diff --git a/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py b/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py
index b2173292f..cdab68bc0 100644
--- a/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py
+++ b/surfsense_backend/tests/unit/capabilities/access/test_agent_tools.py
@@ -78,8 +78,14 @@ async def test_registry_becomes_one_tool_per_verb_plus_readers(isolate):
tools = isolate.module.build_capability_tools(workspace_id=7, capabilities=caps)
by_name = {t.name: t for t in tools}
- # One tool per verb, plus the two shared run-reader tools.
- assert set(by_name) == {"web_scrape", "web_discover", "read_run", "search_run"}
+ # One tool per verb, plus the shared run-reader tools.
+ assert set(by_name) == {
+ "web_scrape",
+ "web_discover",
+ "read_run",
+ "search_run",
+ "export_run",
+ }
assert by_name["web_scrape"].description == "web.scrape does a thing."
assert by_name["web_scrape"].args_schema is _EchoInput
diff --git a/surfsense_backend/tests/unit/capabilities/instagram/__init__.py b/surfsense_backend/tests/unit/capabilities/instagram/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/capabilities/instagram/test_executor.py b/surfsense_backend/tests/unit/capabilities/instagram/test_executor.py
new file mode 100644
index 000000000..a19e597c5
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/instagram/test_executor.py
@@ -0,0 +1,82 @@
+"""Executor tests: lean verb input → ``InstagramScrapeInput`` mapping + wrapping.
+
+A fake scraper captures the actor input the executor built (no network), so the
+snake_case→camelCase mapping and the ``InstagramAccessBlockedError`` →
+``ForbiddenError`` translation are asserted deterministically.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from app.capabilities.instagram.details.executor import build_details_executor
+from app.capabilities.instagram.details.schemas import DetailsInput, DetailsOutput
+from app.capabilities.instagram.scrape.executor import build_scrape_executor
+from app.capabilities.instagram.scrape.schemas import ScrapeInput, ScrapeOutput
+from app.exceptions import ForbiddenError
+from app.proprietary.platforms.instagram import (
+ InstagramAccessBlockedError,
+ InstagramScrapeInput,
+)
+
+pytestmark = pytest.mark.unit
+
+
+class _FakeScraper:
+ def __init__(self, items: list[dict]) -> None:
+ self.items = items
+ self.calls: list[tuple[InstagramScrapeInput, int | None]] = []
+
+ async def __call__(self, actor_input, *, limit=None):
+ self.calls.append((actor_input, limit))
+ return self.items
+
+
+async def test_scrape_maps_urls_and_wraps_items():
+ fake = _FakeScraper([{"id": "1", "shortCode": "abc", "caption": "hi"}])
+ execute = build_scrape_executor(fake)
+ out = await execute(ScrapeInput(urls=["https://www.instagram.com/natgeo/"]))
+ assert isinstance(out, ScrapeOutput)
+ assert out.items[0].shortCode == "abc"
+ actor_input, limit = fake.calls[0]
+ assert actor_input.resultsType == "posts"
+ assert actor_input.directUrls == ["https://www.instagram.com/natgeo/"]
+ assert actor_input.search == ""
+ assert limit == 10 # default max_items forwarded as the collector limit
+
+
+async def test_scrape_joins_search_queries():
+ fake = _FakeScraper([])
+ execute = build_scrape_executor(fake)
+ await execute(ScrapeInput(search_queries=["natgeo", "nasa"], search_type="profile"))
+ actor_input, _ = fake.calls[0]
+ assert actor_input.search == "natgeo,nasa"
+ assert actor_input.searchType == "profile"
+ assert actor_input.directUrls == []
+
+
+async def test_scrape_access_blocked_maps_to_forbidden():
+ async def _blocked(actor_input, *, limit=None):
+ raise InstagramAccessBlockedError("login wall")
+
+ execute = build_scrape_executor(_blocked)
+ with pytest.raises(ForbiddenError):
+ await execute(ScrapeInput(urls=["x"]))
+
+
+async def test_details_maps_and_wraps_discriminated_items():
+ fake = _FakeScraper(
+ [
+ {
+ "detailKind": "profile",
+ "username": "natgeo",
+ "url": "https://www.instagram.com/natgeo/",
+ }
+ ]
+ )
+ execute = build_details_executor(fake)
+ out = await execute(DetailsInput(urls=["https://www.instagram.com/natgeo/"]))
+ assert isinstance(out, DetailsOutput)
+ assert out.items[0].username == "natgeo"
+ actor_input, _ = fake.calls[0]
+ assert actor_input.resultsType == "details"
diff --git a/surfsense_backend/tests/unit/capabilities/instagram/test_registry.py b/surfsense_backend/tests/unit/capabilities/instagram/test_registry.py
new file mode 100644
index 000000000..503feedff
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/instagram/test_registry.py
@@ -0,0 +1,29 @@
+"""The instagram namespace registers its verbs for the doors/agent to read.
+
+Unlike the stale reddit assertion (``billing_unit is None``), these assert the
+real meters — the Capability definitions are the source of truth.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from app.capabilities import (
+ instagram, # noqa: F401 — importing the namespace registers its verbs
+)
+from app.capabilities.core.store import get_capability
+from app.capabilities.core.types import BillingUnit
+
+pytestmark = pytest.mark.unit
+
+
+def test_instagram_scrape_registered_with_item_meter():
+ cap = get_capability("instagram.scrape")
+ assert cap.name == "instagram.scrape"
+ assert cap.billing_unit is BillingUnit.INSTAGRAM_ITEM
+
+
+def test_instagram_details_registered_with_item_meter():
+ cap = get_capability("instagram.details")
+ assert cap.name == "instagram.details"
+ assert cap.billing_unit is BillingUnit.INSTAGRAM_ITEM
diff --git a/surfsense_backend/tests/unit/capabilities/instagram/test_schemas.py b/surfsense_backend/tests/unit/capabilities/instagram/test_schemas.py
new file mode 100644
index 000000000..fde5b07ab
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/instagram/test_schemas.py
@@ -0,0 +1,69 @@
+"""``instagram.*`` input guards: source exclusivity and bounded batches."""
+
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from app.capabilities.instagram.details.schemas import DetailsInput, DetailsOutput
+from app.capabilities.instagram.scrape.schemas import (
+ MAX_INSTAGRAM_ITEMS,
+ MAX_INSTAGRAM_SOURCES,
+ ScrapeInput,
+)
+
+pytestmark = pytest.mark.unit
+
+
+def test_scrape_rejects_no_source():
+ with pytest.raises(ValidationError):
+ ScrapeInput()
+
+
+def test_scrape_rejects_both_sources():
+ with pytest.raises(ValidationError):
+ ScrapeInput(urls=["https://www.instagram.com/natgeo/"], search_queries=["fit"])
+
+
+def test_scrape_accepts_urls_only():
+ payload = ScrapeInput(urls=["https://www.instagram.com/natgeo/"])
+ assert payload.search_queries == []
+ assert payload.estimated_units == payload.max_items
+
+
+def test_scrape_bounds():
+ with pytest.raises(ValidationError):
+ ScrapeInput(
+ urls=["https://www.instagram.com/x/"],
+ max_items=MAX_INSTAGRAM_ITEMS + 1,
+ )
+ with pytest.raises(ValidationError):
+ ScrapeInput(
+ urls=[
+ f"https://www.instagram.com/u{i}/"
+ for i in range(MAX_INSTAGRAM_SOURCES + 1)
+ ]
+ )
+
+
+def test_scrape_rejects_walled_search_type():
+ # Discovery is profile-only; hashtag/place are login-walled and rejected.
+ with pytest.raises(ValidationError):
+ ScrapeInput(search_queries=["travel"], search_type="hashtag")
+
+
+def test_details_wraps_profile_items():
+ out = DetailsOutput(
+ items=[
+ {"detailKind": "profile", "username": "natgeo"},
+ {"detailKind": "profile", "username": "nasa"},
+ ]
+ )
+ kinds = [type(i).__name__ for i in out.items]
+ assert kinds == ["InstagramProfile", "InstagramProfile"]
+ assert out.billable_units == 2
+
+
+def test_details_rejects_both_sources():
+ with pytest.raises(ValidationError):
+ DetailsInput(urls=["https://www.instagram.com/natgeo/"], search_queries=["x"])
diff --git a/surfsense_backend/tests/unit/capabilities/test_billing.py b/surfsense_backend/tests/unit/capabilities/test_billing.py
index 36d9a2978..85187b5f5 100644
--- a/surfsense_backend/tests/unit/capabilities/test_billing.py
+++ b/surfsense_backend/tests/unit/capabilities/test_billing.py
@@ -427,3 +427,54 @@ async def test_platform_gate_disabled_is_noop(monkeypatch):
)
session.execute.assert_not_called()
+
+
+# ===================================================================
+# Instagram per-item / per-comment billing
+# ===================================================================
+
+
+async def test_instagram_item_charges_owner_per_item(
+ monkeypatch, record_usage, _enable_platform_billing
+):
+ monkeypatch.setattr(config, "INSTAGRAM_SCRAPE_MICROS_PER_ITEM", 3500)
+ session, user = _make_session(_OWNER, balance_micros=1_000_000)
+
+ charged = await charge_capability(
+ _FakePlatformOutput(4), BillingUnit.INSTAGRAM_ITEM, _ctx(session)
+ )
+
+ assert charged == 4 * 3500
+ assert user.credit_micros_balance == 1_000_000 - 4 * 3500
+ kwargs = record_usage.await_args.kwargs
+ assert kwargs["usage_type"] == "instagram_item"
+
+
+async def test_instagram_comment_charges_owner_per_comment(
+ monkeypatch, record_usage, _enable_platform_billing
+):
+ monkeypatch.setattr(config, "INSTAGRAM_SCRAPE_MICROS_PER_COMMENT", 1500)
+ session, user = _make_session(_OWNER, balance_micros=1_000_000)
+
+ charged = await charge_capability(
+ _FakePlatformOutput(6), BillingUnit.INSTAGRAM_COMMENT, _ctx(session)
+ )
+
+ assert charged == 6 * 1500
+ assert user.credit_micros_balance == 1_000_000 - 6 * 1500
+ kwargs = record_usage.await_args.kwargs
+ assert kwargs["usage_type"] == "instagram_comment"
+
+
+async def test_instagram_gate_blocks_when_worst_case_exceeds_balance(
+ monkeypatch, _enable_platform_billing
+):
+ monkeypatch.setattr(config, "INSTAGRAM_SCRAPE_MICROS_PER_ITEM", 3500)
+ session = _gate_session(_OWNER, balance_micros=5000) # affords 1 item, not 2
+
+ with pytest.raises(InsufficientCreditsError):
+ await gate_capability(
+ _FakePlatformInput(estimated_units=2),
+ BillingUnit.INSTAGRAM_ITEM,
+ _ctx(session),
+ )
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/__init__.py b/surfsense_backend/tests/unit/capabilities/tiktok/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/comments/__init__.py b/surfsense_backend/tests/unit/capabilities/tiktok/comments/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/comments/test_executor.py b/surfsense_backend/tests/unit/capabilities/tiktok/comments/test_executor.py
new file mode 100644
index 000000000..b23149420
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/comments/test_executor.py
@@ -0,0 +1,48 @@
+"""``tiktok.comments`` executor: verb input → scraper args → typed comment items.
+
+Boundary mocked: the proprietary comments actor (injected fake). NOT mocked: the
+verb's own payload→args forwarding and the dict→CommentItem wrapping.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from app.capabilities.tiktok.comments.executor import build_comments_executor
+from app.capabilities.tiktok.comments.schemas import CommentsInput, CommentsOutput
+
+pytestmark = pytest.mark.unit
+
+_VIDEO = "https://www.tiktok.com/@bob/video/123"
+
+
+class _FakeComments:
+ """Records the urls + kwargs it was called with; returns canned items."""
+
+ def __init__(self, items: list[dict]):
+ self._items = items
+ self.calls: list[tuple[list[str], int, int | None]] = []
+
+ async def __call__(
+ self, video_urls: list[str], *, per_video: int, limit: int | None = None
+ ) -> list[dict]:
+ self.calls.append((video_urls, per_video, limit))
+ return self._items
+
+
+async def test_forwards_urls_and_limits_and_wraps_items():
+ comments = _FakeComments([{"id": "1", "text": "hi"}])
+ execute = build_comments_executor(comments_fn=comments)
+
+ out = await execute(
+ CommentsInput(video_urls=[_VIDEO], comments_per_video=15, max_items=40)
+ )
+
+ assert isinstance(out, CommentsOutput)
+ assert len(out.items) == 1
+ assert out.items[0].text == "hi"
+
+ (urls, per_video, limit) = comments.calls[0]
+ assert urls == [_VIDEO]
+ assert per_video == 15
+ assert limit == 40
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/comments/test_schemas.py b/surfsense_backend/tests/unit/capabilities/tiktok/comments/test_schemas.py
new file mode 100644
index 000000000..cdcbc201e
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/comments/test_schemas.py
@@ -0,0 +1,48 @@
+"""``tiktok.comments`` input guards and billing: a video URL is required, bounded,
+and ErrorItems are surfaced free."""
+
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from app.capabilities.tiktok.comments.schemas import CommentsInput, CommentsOutput
+from app.capabilities.tiktok.scrape.schemas import MAX_TIKTOK_ITEMS, MAX_TIKTOK_SOURCES
+
+pytestmark = pytest.mark.unit
+
+_VIDEO = "https://www.tiktok.com/@bob/video/123"
+
+
+def test_rejects_input_with_no_url():
+ with pytest.raises(ValidationError):
+ CommentsInput(video_urls=[])
+
+
+def test_defaults_and_bounds():
+ payload = CommentsInput(video_urls=[_VIDEO])
+ assert payload.max_items == 20
+ assert payload.comments_per_video == 20
+ assert payload.estimated_units == 20
+ with pytest.raises(ValidationError):
+ CommentsInput(video_urls=[_VIDEO], max_items=0)
+ with pytest.raises(ValidationError):
+ CommentsInput(video_urls=[_VIDEO], max_items=MAX_TIKTOK_ITEMS + 1)
+
+
+def test_rejects_more_urls_than_the_cap():
+ too_many = [f"{_VIDEO}{i}" for i in range(MAX_TIKTOK_SOURCES + 1)]
+ with pytest.raises(ValidationError):
+ CommentsInput(video_urls=too_many)
+
+
+def test_error_items_are_not_billed():
+ # Real comments count; ErrorItems (bad URL / empty video) are surfaced free.
+ out = CommentsOutput(
+ items=[
+ {"id": "1", "text": "hi"},
+ {"errorCode": "no_comments", "input": "123", "error": "empty"},
+ ]
+ )
+ assert len(out.items) == 2
+ assert out.billable_units == 1
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/scrape/__init__.py b/surfsense_backend/tests/unit/capabilities/tiktok/scrape/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/scrape/test_executor.py b/surfsense_backend/tests/unit/capabilities/tiktok/scrape/test_executor.py
new file mode 100644
index 000000000..367831c76
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/scrape/test_executor.py
@@ -0,0 +1,77 @@
+"""``tiktok.scrape`` executor: verb input → actor input mapping → typed items.
+
+Boundary mocked: the proprietary scraper (injected fake). NOT mocked: the verb's
+own payload→TikTokScrapeInput mapping and the dict→TikTokVideoItem wrapping.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from app.capabilities.tiktok.scrape.executor import build_scrape_executor
+from app.capabilities.tiktok.scrape.schemas import ScrapeInput, ScrapeOutput
+from app.exceptions import ForbiddenError
+from app.proprietary.platforms.tiktok import TikTokAccessBlockedError, TikTokScrapeInput
+
+pytestmark = pytest.mark.unit
+
+
+class _FakeScraper:
+ """Records the actor input + limit it was called with; returns canned items."""
+
+ def __init__(self, items: list[dict]):
+ self._items = items
+ self.calls: list[tuple[TikTokScrapeInput, int | None]] = []
+
+ async def __call__(
+ self, actor_input: TikTokScrapeInput, *, limit: int | None = None
+ ) -> list[dict]:
+ self.calls.append((actor_input, limit))
+ return self._items
+
+
+async def test_maps_urls_to_start_urls_and_wraps_items():
+ scraper = _FakeScraper([{"id": "123", "text": "hello"}])
+ execute = build_scrape_executor(scrape_fn=scraper)
+
+ out = await execute(ScrapeInput(urls=["https://www.tiktok.com/@nasa/video/123"]))
+
+ assert isinstance(out, ScrapeOutput)
+ assert len(out.items) == 1
+ assert out.items[0].id == "123"
+
+ (actor_input, _limit) = scraper.calls[0]
+ assert [u.url for u in actor_input.startUrls] == [
+ "https://www.tiktok.com/@nasa/video/123"
+ ]
+
+
+async def test_forwards_typed_sources_and_limit():
+ scraper = _FakeScraper([])
+ execute = build_scrape_executor(scrape_fn=scraper)
+
+ await execute(
+ ScrapeInput(
+ profiles=["nasa"],
+ hashtags=["food"],
+ results_per_page=7,
+ max_items=25,
+ )
+ )
+
+ (actor_input, limit) = scraper.calls[0]
+ assert actor_input.profiles == ["nasa"]
+ assert actor_input.hashtags == ["food"]
+ assert actor_input.resultsPerPage == 7
+ # The outer collection limit is the caller's total-item cap.
+ assert limit == 25
+
+
+async def test_access_blocked_maps_to_forbidden():
+ async def _blocked(actor_input: TikTokScrapeInput, *, limit: int | None = None):
+ raise TikTokAccessBlockedError("all IPs refused")
+
+ execute = build_scrape_executor(scrape_fn=_blocked)
+
+ with pytest.raises(ForbiddenError):
+ await execute(ScrapeInput(hashtags=["food"]))
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/scrape/test_schemas.py b/surfsense_backend/tests/unit/capabilities/tiktok/scrape/test_schemas.py
new file mode 100644
index 000000000..f9cec8c3d
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/scrape/test_schemas.py
@@ -0,0 +1,58 @@
+"""``tiktok.scrape`` input guards: a source is required and the batch is bounded."""
+
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from app.capabilities.tiktok.scrape.schemas import (
+ MAX_TIKTOK_ITEMS,
+ MAX_TIKTOK_SOURCES,
+ ScrapeInput,
+ ScrapeOutput,
+)
+
+pytestmark = pytest.mark.unit
+
+
+def test_rejects_input_with_no_source():
+ with pytest.raises(ValidationError):
+ ScrapeInput()
+
+
+def test_accepts_urls_only():
+ payload = ScrapeInput(urls=["https://www.tiktok.com/@nasa"])
+ assert payload.profiles == []
+
+
+def test_accepts_hashtags_only():
+ payload = ScrapeInput(hashtags=["food"])
+ assert payload.hashtags == ["food"]
+
+
+def test_defaults_and_bounds():
+ payload = ScrapeInput(hashtags=["food"])
+ assert payload.max_items == 10
+ assert payload.results_per_page == 10
+ with pytest.raises(ValidationError):
+ ScrapeInput(hashtags=["food"], max_items=0)
+ with pytest.raises(ValidationError):
+ ScrapeInput(hashtags=["food"], max_items=MAX_TIKTOK_ITEMS + 1)
+
+
+def test_rejects_more_sources_than_the_cap():
+ too_many = [f"tag{i}" for i in range(MAX_TIKTOK_SOURCES + 1)]
+ with pytest.raises(ValidationError):
+ ScrapeInput(hashtags=too_many)
+
+
+def test_error_items_are_not_billed():
+ # Real videos count; ErrorItems (blocked/empty targets) are surfaced free.
+ out = ScrapeOutput(
+ items=[
+ {"id": "1", "webVideoUrl": "https://tiktok.com/@a/video/1"},
+ {"errorCode": "no_items", "input": "nasa", "error": "blocked"},
+ ]
+ )
+ assert len(out.items) == 2
+ assert out.billable_units == 1
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/test_registry.py b/surfsense_backend/tests/unit/capabilities/tiktok/test_registry.py
new file mode 100644
index 000000000..c6314ec10
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/test_registry.py
@@ -0,0 +1,57 @@
+"""The tiktok namespace registers its verb as one Capability the doors/agent read."""
+
+from __future__ import annotations
+
+import pytest
+
+from app.capabilities import (
+ tiktok, # noqa: F401 — importing the namespace registers its verbs
+)
+from app.capabilities.core import BillingUnit
+from app.capabilities.core.store import get_capability
+from app.capabilities.tiktok.comments.schemas import CommentsInput, CommentsOutput
+from app.capabilities.tiktok.scrape.schemas import ScrapeInput, ScrapeOutput
+from app.capabilities.tiktok.trending.schemas import TrendingInput, TrendingOutput
+from app.capabilities.tiktok.user_search.schemas import (
+ UserSearchInput,
+ UserSearchOutput,
+)
+
+pytestmark = pytest.mark.unit
+
+
+def test_tiktok_scrape_is_registered_and_billed_per_video():
+ cap = get_capability("tiktok.scrape")
+
+ assert cap.name == "tiktok.scrape"
+ assert cap.input_schema is ScrapeInput
+ assert cap.output_schema is ScrapeOutput
+ assert cap.billing_unit is BillingUnit.TIKTOK_VIDEO
+
+
+def test_tiktok_user_search_is_registered_and_billed_per_profile():
+ cap = get_capability("tiktok.user_search")
+
+ assert cap.name == "tiktok.user_search"
+ assert cap.input_schema is UserSearchInput
+ assert cap.output_schema is UserSearchOutput
+ assert cap.billing_unit is BillingUnit.TIKTOK_USER
+
+
+def test_tiktok_comments_is_registered_and_billed_per_comment():
+ cap = get_capability("tiktok.comments")
+
+ assert cap.name == "tiktok.comments"
+ assert cap.input_schema is CommentsInput
+ assert cap.output_schema is CommentsOutput
+ assert cap.billing_unit is BillingUnit.TIKTOK_COMMENT
+
+
+def test_tiktok_trending_is_registered_and_billed_per_video():
+ cap = get_capability("tiktok.trending")
+
+ assert cap.name == "tiktok.trending"
+ assert cap.input_schema is TrendingInput
+ assert cap.output_schema is TrendingOutput
+ # Trending returns videos, so it shares the per-video meter.
+ assert cap.billing_unit is BillingUnit.TIKTOK_VIDEO
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/trending/__init__.py b/surfsense_backend/tests/unit/capabilities/tiktok/trending/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/trending/test_executor.py b/surfsense_backend/tests/unit/capabilities/tiktok/trending/test_executor.py
new file mode 100644
index 000000000..3539ca595
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/trending/test_executor.py
@@ -0,0 +1,38 @@
+"""``tiktok.trending`` executor: verb input → scraper count → typed video items.
+
+Boundary mocked: the proprietary trending actor (injected fake). NOT mocked: the
+verb's own count forwarding and the dict→TikTokVideoItem wrapping.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from app.capabilities.tiktok.trending.executor import build_trending_executor
+from app.capabilities.tiktok.trending.schemas import TrendingInput, TrendingOutput
+
+pytestmark = pytest.mark.unit
+
+
+class _FakeTrending:
+ """Records the count it was called with; returns canned items."""
+
+ def __init__(self, items: list[dict]):
+ self._items = items
+ self.calls: list[int] = []
+
+ async def __call__(self, *, count: int) -> list[dict]:
+ self.calls.append(count)
+ return self._items
+
+
+async def test_forwards_count_and_wraps_items():
+ trending = _FakeTrending([{"id": "1", "text": "viral"}])
+ execute = build_trending_executor(trending_fn=trending)
+
+ out = await execute(TrendingInput(max_items=30))
+
+ assert isinstance(out, TrendingOutput)
+ assert len(out.items) == 1
+ assert out.items[0].id == "1"
+ assert trending.calls == [30]
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/trending/test_schemas.py b/surfsense_backend/tests/unit/capabilities/tiktok/trending/test_schemas.py
new file mode 100644
index 000000000..f27fec71a
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/trending/test_schemas.py
@@ -0,0 +1,33 @@
+"""``tiktok.trending`` input guards and billing: bounded count, ErrorItems free."""
+
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from app.capabilities.tiktok.scrape.schemas import MAX_TIKTOK_ITEMS
+from app.capabilities.tiktok.trending.schemas import TrendingInput, TrendingOutput
+
+pytestmark = pytest.mark.unit
+
+
+def test_defaults_and_bounds():
+ payload = TrendingInput()
+ assert payload.max_items == 20
+ assert payload.estimated_units == 20
+ with pytest.raises(ValidationError):
+ TrendingInput(max_items=0)
+ with pytest.raises(ValidationError):
+ TrendingInput(max_items=MAX_TIKTOK_ITEMS + 1)
+
+
+def test_error_items_are_not_billed():
+ # Real videos count; an ErrorItem (empty/withheld feed) is surfaced free.
+ out = TrendingOutput(
+ items=[
+ {"id": "1", "webVideoUrl": "https://tiktok.com/@a/video/1"},
+ {"errorCode": "no_items", "input": "explore", "error": "empty"},
+ ]
+ )
+ assert len(out.items) == 2
+ assert out.billable_units == 1
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/user_search/__init__.py b/surfsense_backend/tests/unit/capabilities/tiktok/user_search/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/user_search/test_executor.py b/surfsense_backend/tests/unit/capabilities/tiktok/user_search/test_executor.py
new file mode 100644
index 000000000..866968808
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/user_search/test_executor.py
@@ -0,0 +1,49 @@
+"""``tiktok.user_search`` executor: verb input → search args → typed profile items.
+
+Boundary mocked: the proprietary search actor (injected fake). NOT mocked: the
+verb's own payload→args forwarding and the dict→TikTokProfileItem wrapping.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from app.capabilities.tiktok.user_search.executor import build_user_search_executor
+from app.capabilities.tiktok.user_search.schemas import (
+ UserSearchInput,
+ UserSearchOutput,
+)
+
+pytestmark = pytest.mark.unit
+
+
+class _FakeSearch:
+ """Records the queries + kwargs it was called with; returns canned items."""
+
+ def __init__(self, items: list[dict]):
+ self._items = items
+ self.calls: list[tuple[list[str], int, int | None]] = []
+
+ async def __call__(
+ self, queries: list[str], *, per_query: int, limit: int | None = None
+ ) -> list[dict]:
+ self.calls.append((queries, per_query, limit))
+ return self._items
+
+
+async def test_forwards_queries_and_limits_and_wraps_items():
+ search = _FakeSearch([{"id": "1", "name": "nasa"}])
+ execute = build_user_search_executor(search_fn=search)
+
+ out = await execute(
+ UserSearchInput(queries=["nasa"], results_per_query=7, max_items=25)
+ )
+
+ assert isinstance(out, UserSearchOutput)
+ assert len(out.items) == 1
+ assert out.items[0].name == "nasa"
+
+ (queries, per_query, limit) = search.calls[0]
+ assert queries == ["nasa"]
+ assert per_query == 7
+ assert limit == 25
diff --git a/surfsense_backend/tests/unit/capabilities/tiktok/user_search/test_schemas.py b/surfsense_backend/tests/unit/capabilities/tiktok/user_search/test_schemas.py
new file mode 100644
index 000000000..575db3fb5
--- /dev/null
+++ b/surfsense_backend/tests/unit/capabilities/tiktok/user_search/test_schemas.py
@@ -0,0 +1,49 @@
+"""``tiktok.user_search`` input guards and billing: a query is required, bounded,
+and ErrorItems are surfaced free."""
+
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from app.capabilities.tiktok.scrape.schemas import MAX_TIKTOK_ITEMS, MAX_TIKTOK_SOURCES
+from app.capabilities.tiktok.user_search.schemas import (
+ UserSearchInput,
+ UserSearchOutput,
+)
+
+pytestmark = pytest.mark.unit
+
+
+def test_rejects_input_with_no_query():
+ with pytest.raises(ValidationError):
+ UserSearchInput(queries=[])
+
+
+def test_defaults_and_bounds():
+ payload = UserSearchInput(queries=["nasa"])
+ assert payload.max_items == 10
+ assert payload.results_per_query == 10
+ assert payload.estimated_units == 10
+ with pytest.raises(ValidationError):
+ UserSearchInput(queries=["nasa"], max_items=0)
+ with pytest.raises(ValidationError):
+ UserSearchInput(queries=["nasa"], max_items=MAX_TIKTOK_ITEMS + 1)
+
+
+def test_rejects_more_queries_than_the_cap():
+ too_many = [f"q{i}" for i in range(MAX_TIKTOK_SOURCES + 1)]
+ with pytest.raises(ValidationError):
+ UserSearchInput(queries=too_many)
+
+
+def test_error_items_are_not_billed():
+ # Real accounts count; ErrorItems (empty/withheld queries) are surfaced free.
+ out = UserSearchOutput(
+ items=[
+ {"id": "1", "name": "nasa"},
+ {"errorCode": "no_users", "input": "ghost", "error": "empty"},
+ ]
+ )
+ assert len(out.items) == 2
+ assert out.billable_units == 1
diff --git a/surfsense_backend/tests/unit/config/test_embedding_settings.py b/surfsense_backend/tests/unit/config/test_embedding_settings.py
new file mode 100644
index 000000000..7483f9457
--- /dev/null
+++ b/surfsense_backend/tests/unit/config/test_embedding_settings.py
@@ -0,0 +1,76 @@
+import pytest
+
+from app.config.embedding_settings import (
+ build_embedding_kwargs,
+ resolve_embedding_base_url,
+)
+
+pytestmark = pytest.mark.unit
+
+
+def test_resolve_embedding_base_url_prefers_generic_value() -> None:
+ environ = {
+ "EMBEDDING_BASE_URL": " http://embed-host:11434 ",
+ "OLLAMA_EMBEDDING_BASE_URL": "http://ollama-embed:11434",
+ }
+
+ assert resolve_embedding_base_url(environ) == "http://embed-host:11434"
+
+
+def test_resolve_embedding_base_url_falls_back_to_ollama_specific_value() -> None:
+ environ = {
+ "EMBEDDING_BASE_URL": " ",
+ "OLLAMA_EMBEDDING_BASE_URL": "http://ollama-embed:11434",
+ }
+
+ assert resolve_embedding_base_url(environ) == "http://ollama-embed:11434"
+
+
+def test_build_embedding_kwargs_maps_base_url_to_litellm_api_base() -> None:
+ kwargs = build_embedding_kwargs(
+ {"EMBEDDING_BASE_URL": "http://host.docker.internal:11435"},
+ embedding_model="litellm://ollama/nomic-embed-text",
+ )
+
+ assert kwargs == {"api_base": "http://host.docker.internal:11435"}
+
+
+def test_build_embedding_kwargs_does_not_leak_api_base_to_other_providers() -> None:
+ kwargs = build_embedding_kwargs(
+ {"EMBEDDING_BASE_URL": "http://host.docker.internal:11435"},
+ embedding_model="cohere://embed-english-light-v3.0",
+ )
+
+ assert kwargs == {}
+
+
+def test_build_embedding_kwargs_preserves_azure_settings() -> None:
+ kwargs = build_embedding_kwargs(
+ {
+ "AZURE_OPENAI_ENDPOINT": "https://example.openai.azure.com",
+ "AZURE_OPENAI_API_KEY": "test-key",
+ },
+ embedding_model="azure_openai://text-embedding-3-small",
+ )
+
+ assert kwargs == {
+ "azure_endpoint": "https://example.openai.azure.com",
+ "azure_api_key": "test-key",
+ }
+
+
+def test_build_embedding_kwargs_combines_litellm_and_azure_env_when_set() -> None:
+ kwargs = build_embedding_kwargs(
+ {
+ "EMBEDDING_BASE_URL": "http://host.docker.internal:4000/v1",
+ "AZURE_OPENAI_ENDPOINT": "https://example.openai.azure.com",
+ "AZURE_OPENAI_API_KEY": "test-key",
+ },
+ embedding_model="litellm://openai/text-embedding-3-small",
+ )
+
+ assert kwargs == {
+ "api_base": "http://host.docker.internal:4000/v1",
+ "azure_endpoint": "https://example.openai.azure.com",
+ "azure_api_key": "test-key",
+ }
diff --git a/surfsense_backend/tests/unit/platforms/instagram/__init__.py b/surfsense_backend/tests/unit/platforms/instagram/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/platforms/instagram/test_budget.py b/surfsense_backend/tests/unit/platforms/instagram/test_budget.py
new file mode 100644
index 000000000..703002f0c
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/instagram/test_budget.py
@@ -0,0 +1,96 @@
+"""Offline budget tests: per-target caps, cross-target de-dup, and the limit guard.
+
+No network. ``fetch_json`` is stubbed with a synthetic profile payload and the
+fan-out proxy holders are replaced with no-ops, so the orchestrator's paging and
+de-dup policy is exercised deterministically.
+"""
+
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+
+import pytest
+
+from app.proprietary.platforms.instagram import scraper
+from app.proprietary.platforms.instagram.schemas import InstagramScrapeInput
+
+
+class _NoopHolder:
+ async def close(self) -> None:
+ return None
+
+
+@pytest.fixture
+def _stub_proxy(monkeypatch):
+ async def _open():
+ return _NoopHolder()
+
+ @asynccontextmanager
+ async def _bind(_holder):
+ yield _holder
+
+ monkeypatch.setattr(scraper, "open_proxy_holder", _open)
+ monkeypatch.setattr(scraper, "bind_proxy_holder", _bind)
+
+
+def _profile_payload(n: int) -> dict:
+ return {
+ "data": {
+ "user": {
+ "id": "9",
+ "username": "acct",
+ "edge_owner_to_timeline_media": {
+ "count": n,
+ "edges": [
+ {"node": {"id": str(i), "shortcode": f"S{i}"}} for i in range(n)
+ ],
+ },
+ }
+ }
+ }
+
+
+async def test_per_target_cap_limits_media(_stub_proxy, monkeypatch):
+ async def _fetch(path, params=None):
+ return _profile_payload(50)
+
+ monkeypatch.setattr(scraper, "fetch_json", _fetch)
+ model = InstagramScrapeInput(
+ resultsType="posts",
+ directUrls=["https://www.instagram.com/acct/"],
+ resultsLimit=5,
+ )
+ items = [i async for i in scraper.iter_instagram(model)]
+ assert len(items) == 5
+
+
+async def test_cross_target_dedup_by_id(_stub_proxy, monkeypatch):
+ async def _fetch(path, params=None):
+ return _profile_payload(3) # both targets return ids 0,1,2
+
+ monkeypatch.setattr(scraper, "fetch_json", _fetch)
+ model = InstagramScrapeInput(
+ resultsType="posts",
+ directUrls=[
+ "https://www.instagram.com/one/",
+ "https://www.instagram.com/two/",
+ ],
+ resultsLimit=10,
+ )
+ items = [i async for i in scraper.iter_instagram(model)]
+ ids = sorted(i["id"] for i in items)
+ assert ids == ["0", "1", "2"]
+
+
+async def test_scrape_instagram_honors_limit(_stub_proxy, monkeypatch):
+ async def _fetch(path, params=None):
+ return _profile_payload(50)
+
+ monkeypatch.setattr(scraper, "fetch_json", _fetch)
+ model = InstagramScrapeInput(
+ resultsType="posts",
+ directUrls=["https://www.instagram.com/acct/"],
+ resultsLimit=100,
+ )
+ items = await scraper.scrape_instagram(model, limit=7)
+ assert len(items) == 7
diff --git a/surfsense_backend/tests/unit/platforms/instagram/test_discovery.py b/surfsense_backend/tests/unit/platforms/instagram/test_discovery.py
new file mode 100644
index 000000000..379e3ba28
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/instagram/test_discovery.py
@@ -0,0 +1,74 @@
+"""Offline tests for Google-backed Instagram discovery.
+
+Discovery is profile-only (hashtag/place feeds are login-walled). A valid handle
+resolves directly; any other query falls back to the ``google_search`` platform
+(``site:instagram.com``), classifying organic results with ``resolve_url`` and
+keeping only profile hits. These tests inject a fake ``scrape_serps`` so there is
+no network: they pin the classification, de-dup, and ``limit`` cap.
+"""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.instagram import scraper
+
+
+def _fake_serps(*organic_urls: str):
+ async def _scrape_serps(input_model, *, limit=None):
+ assert input_model.site == "instagram.com"
+ return [{"organicResults": [{"url": u} for u in organic_urls]}]
+
+ return _scrape_serps
+
+
+async def test_google_discovery_keeps_only_profiles(monkeypatch):
+ # A non-handle query goes to Google; only profile URLs survive (hashtag /
+ # post / non-instagram results are dropped since discovery is profile-only).
+ monkeypatch.setattr(
+ scraper,
+ "scrape_serps",
+ _fake_serps(
+ "https://www.instagram.com/natgeo/",
+ "https://www.instagram.com/explore/tags/travel/",
+ "https://www.instagram.com/p/ABC123/",
+ "https://example.com/not-instagram",
+ ),
+ )
+ targets = await scraper._discover("nat geo photos", search_type="profile", limit=10)
+ assert [(t.kind, t.value) for t in targets] == [("profile", "natgeo")]
+
+
+async def test_google_discovery_dedupes(monkeypatch):
+ monkeypatch.setattr(
+ scraper,
+ "scrape_serps",
+ _fake_serps(
+ "https://www.instagram.com/natgeo/",
+ "https://www.instagram.com/natgeo/",
+ ),
+ )
+ targets = await scraper._discover("nat geo photos", search_type="profile", limit=10)
+ assert len(targets) == 1
+
+
+async def test_google_discovery_respects_limit(monkeypatch):
+ monkeypatch.setattr(
+ scraper,
+ "scrape_serps",
+ _fake_serps(
+ "https://www.instagram.com/a_a/",
+ "https://www.instagram.com/b_b/",
+ "https://www.instagram.com/c_c/",
+ ),
+ )
+ targets = await scraper._discover("some brand name", search_type="profile", limit=2)
+ assert [t.value for t in targets] == ["a_a", "b_b"]
+
+
+async def test_discover_profile_handle_fast_path_skips_google(monkeypatch):
+ # A valid handle resolves directly without touching Google.
+ async def _boom(input_model, *, limit=None):
+ raise AssertionError("Google should not be called for a valid handle")
+
+ monkeypatch.setattr(scraper, "scrape_serps", _boom)
+ targets = await scraper._discover("messi", search_type="user", limit=10)
+ assert [(t.kind, t.value) for t in targets] == [("profile", "messi")]
diff --git a/surfsense_backend/tests/unit/platforms/instagram/test_fetch_resilience.py b/surfsense_backend/tests/unit/platforms/instagram/test_fetch_resilience.py
new file mode 100644
index 000000000..44fefa430
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/instagram/test_fetch_resilience.py
@@ -0,0 +1,422 @@
+"""Offline resilience tests for the Instagram fetch seam and fan-out worker pool.
+
+No network. Fake sessions drive the ``csrftoken`` warm-up + rotate-on-block +
+backoff paths deterministically (in live runs the first IP warms and returns
+200s, so these branches rarely fire). Mirrors the reddit sibling's
+``test_fetch_resilience.py`` shape, adapted to Instagram's cookie warm-up.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+
+from app.proprietary.platforms.instagram import fetch, scraper
+from app.proprietary.platforms.instagram.fetch import (
+ InstagramAccessBlockedError,
+ _current_session,
+ fetch_json,
+)
+
+_PAYLOAD = {"data": {"user": {"username": "natgeo"}}}
+
+
+class _FakePage:
+ def __init__(
+ self, status: int, *, cookies: dict | None = None, payload=None, url=None
+ ):
+ self.status = status
+ self.cookies = cookies or {}
+ self.url = url
+ self._payload = payload if payload is not None else _PAYLOAD
+
+ def json(self):
+ return self._payload
+
+ @property
+ def body(self) -> str:
+ return json.dumps(self._payload)
+
+
+class _FakeSession:
+ """One 'IP': the warm-up GET mints csrftoken per flag; endpoint GETs return ``status``."""
+
+ def __init__(
+ self,
+ status: int = 200,
+ *,
+ csrftoken: bool = True,
+ payload=None,
+ login_wall: bool = False,
+ ) -> None:
+ self.status = status
+ self.csrftoken = csrftoken
+ self.payload = payload
+ self.login_wall = login_wall
+ self.json_calls = 0
+ self.warm_calls = 0
+
+ async def get(self, url, headers=None, cookies=None):
+ if url.rstrip("/") == "https://www.instagram.com":
+ self.warm_calls += 1
+ ck = {"csrftoken": "x", "mid": "y"} if self.csrftoken else {}
+ return _FakePage(200, cookies=ck)
+ self.json_calls += 1
+ # A soft login wall: 200, but the final URL is the login page.
+ final = "https://www.instagram.com/accounts/login/" if self.login_wall else url
+ return _FakePage(self.status, payload=self.payload, url=final)
+
+
+class _FakeHolder:
+ """Holder whose ``rotate()`` advances to the next fake session (a new IP)."""
+
+ def __init__(self, sessions: list[_FakeSession]) -> None:
+ self._sessions = sessions
+ self.session = sessions[0]
+ self.rotations = 0
+ self.warmed = False
+
+ async def rotate(self):
+ self.rotations += 1
+ self.session = self._sessions[min(self.rotations, len(self._sessions) - 1)]
+ self.warmed = False # cookies bind to the IP: re-warm on the fresh one
+ return self.session
+
+ async def pace(self) -> None:
+ return None
+
+ async def close(self) -> None:
+ return None
+
+
+def _no_sleep(monkeypatch) -> None:
+ async def _noop(_seconds):
+ return None
+
+ monkeypatch.setattr(fetch.asyncio, "sleep", _noop)
+
+
+async def test_warms_then_returns_json():
+ holder = _FakeHolder([_FakeSession(200, csrftoken=True)])
+ token = _current_session.set(holder)
+ try:
+ result = await fetch_json(
+ "api/v1/users/web_profile_info/", {"username": "natgeo"}
+ )
+ finally:
+ _current_session.reset(token)
+ assert result == _PAYLOAD
+ assert holder.rotations == 0
+ assert holder.session.warm_calls == 1 # warmed exactly once
+
+
+async def test_rotates_when_warm_fails_then_succeeds():
+ holder = _FakeHolder(
+ [_FakeSession(200, csrftoken=False), _FakeSession(200, csrftoken=True)]
+ )
+ token = _current_session.set(holder)
+ try:
+ result = await fetch_json("api/v1/users/web_profile_info/")
+ finally:
+ _current_session.reset(token)
+ assert result == _PAYLOAD
+ assert holder.rotations == 1
+
+
+async def test_raises_when_no_ip_can_warm():
+ holder = _FakeHolder(
+ [_FakeSession(200, csrftoken=False) for _ in range(fetch._MAX_ROTATIONS + 1)]
+ )
+ token = _current_session.set(holder)
+ try:
+ raised = False
+ try:
+ await fetch_json("api/v1/users/web_profile_info/")
+ except InstagramAccessBlockedError:
+ raised = True
+ finally:
+ _current_session.reset(token)
+ assert raised
+ assert holder.rotations == fetch._MAX_ROTATIONS
+
+
+async def test_rotates_and_rewarms_on_403():
+ holder = _FakeHolder([_FakeSession(403), _FakeSession(200)])
+ token = _current_session.set(holder)
+ try:
+ result = await fetch_json("api/v1/users/web_profile_info/")
+ finally:
+ _current_session.reset(token)
+ assert result == _PAYLOAD
+ assert holder.rotations == 1
+ assert holder.session.warm_calls == 1 # re-warmed on the fresh IP
+
+
+async def test_rotates_on_401_login_wall():
+ holder = _FakeHolder([_FakeSession(401), _FakeSession(200)])
+ token = _current_session.set(holder)
+ try:
+ result = await fetch_json("api/v1/users/web_profile_info/")
+ finally:
+ _current_session.reset(token)
+ assert result == _PAYLOAD
+ assert holder.rotations == 1
+
+
+async def test_login_redirect_fails_fast_without_rotating():
+ # A 302 -> /accounts/login/ (served 200) is an endpoint-level wall: rotating
+ # never clears it, so we raise immediately instead of burning IP rotations.
+ # A second healthy session is present to prove we do NOT fall through to it.
+ holder = _FakeHolder([_FakeSession(200, login_wall=True), _FakeSession(200)])
+ token = _current_session.set(holder)
+ try:
+ raised = False
+ try:
+ await fetch_json("api/v1/users/web_profile_info/", {"username": "natgeo"})
+ except InstagramAccessBlockedError:
+ raised = True
+ finally:
+ _current_session.reset(token)
+ assert raised
+ assert holder.rotations == 0 # fail fast: no rotation burned
+
+
+async def test_404_returns_none_without_rotating():
+ holder = _FakeHolder([_FakeSession(404), _FakeSession(200)])
+ token = _current_session.set(holder)
+ try:
+ result = await fetch_json("api/v1/users/web_profile_info/")
+ finally:
+ _current_session.reset(token)
+ assert result is None
+ assert holder.rotations == 0
+
+
+async def test_429_backs_off_without_rotating(monkeypatch):
+ _no_sleep(monkeypatch)
+ session = _FakeSession(429)
+
+ async def _get(url, headers=None, cookies=None):
+ if url.rstrip("/") == "https://www.instagram.com":
+ session.warm_calls += 1
+ return _FakePage(200, cookies={"csrftoken": "x"})
+ session.json_calls += 1
+ return _FakePage(429 if session.json_calls == 1 else 200)
+
+ session.get = _get # type: ignore[method-assign]
+ holder = _FakeHolder([session])
+ token = _current_session.set(holder)
+ try:
+ result = await fetch_json("api/v1/users/web_profile_info/")
+ finally:
+ _current_session.reset(token)
+ assert result == _PAYLOAD
+ assert holder.rotations == 0
+
+
+async def test_persistent_403_raises_blocked(monkeypatch):
+ _no_sleep(monkeypatch)
+ holder = _FakeHolder([_FakeSession(403) for _ in range(fetch._MAX_ROTATIONS + 1)])
+ token = _current_session.set(holder)
+ try:
+ raised = False
+ try:
+ await fetch_json("api/v1/users/web_profile_info/")
+ except InstagramAccessBlockedError:
+ raised = True
+ finally:
+ _current_session.reset(token)
+ assert raised
+ assert holder.rotations == fetch._MAX_ROTATIONS
+
+
+class _TrackingHolder:
+ """Fake fan-out session holder that records whether it was closed."""
+
+ def __init__(self) -> None:
+ self.closed = False
+
+ async def close(self) -> None:
+ self.closed = True
+
+
+async def test_fan_out_closes_all_sessions_on_early_stop(monkeypatch):
+ holders: list[_TrackingHolder] = []
+
+ async def _fake_open():
+ h = _TrackingHolder()
+ holders.append(h)
+ return h
+
+ @asynccontextmanager
+ async def _fake_bind(_holder):
+ yield _holder
+
+ monkeypatch.setattr(scraper, "open_proxy_holder", _fake_open)
+ monkeypatch.setattr(scraper, "bind_proxy_holder", _fake_bind)
+
+ async def _job(n: int) -> AsyncIterator[dict]:
+ for i in range(5):
+ yield {"job": n, "i": i}
+
+ jobs = [_job(n) for n in range(20)]
+ gen = scraper.fan_out(jobs, concurrency=4)
+ collected = []
+ async for item in gen:
+ collected.append(item)
+ if len(collected) >= 3:
+ break
+ await gen.aclose()
+
+ assert len(collected) >= 3
+ assert holders, "workers should have opened sessions"
+ assert all(h.closed for h in holders), "a worker leaked its proxy session"
+
+
+async def test_fan_out_empty_jobs_is_noop():
+ out = [x async for x in scraper.fan_out([])]
+ assert out == []
+
+
+async def _install_fake_holders(monkeypatch) -> None:
+ async def _fake_open():
+ return _TrackingHolder()
+
+ @asynccontextmanager
+ async def _fake_bind(_holder):
+ yield _holder
+
+ monkeypatch.setattr(scraper, "open_proxy_holder", _fake_open)
+ monkeypatch.setattr(scraper, "bind_proxy_holder", _fake_bind)
+
+
+async def _blocked_job() -> AsyncIterator[dict]:
+ raise InstagramAccessBlockedError("login wall")
+ yield {} # unreachable; makes this an async generator
+
+
+async def test_fan_out_all_blocked_raises_without_deadlock(monkeypatch):
+ # Regression: a blocked worker used to strand its exception on a dead task
+ # and deadlock the consumer on results.get(). When EVERY target is blocked
+ # (zero items), it must surface InstagramAccessBlockedError, not hang.
+ await _install_fake_holders(monkeypatch)
+
+ raised = False
+ try:
+ async with asyncio.timeout(5): # fail fast if the deadlock regresses
+ async for _ in scraper.fan_out([_blocked_job()], concurrency=1):
+ pass
+ except InstagramAccessBlockedError:
+ raised = True
+ assert raised, "fully-blocked run must surface the 403, not deadlock"
+
+
+async def test_fan_out_partial_results_survive_one_blocked_target(monkeypatch):
+ # Q2: one blocked target must NOT abort the batch — the good target's items
+ # come through and no exception is raised (aggregation, not a transaction).
+ await _install_fake_holders(monkeypatch)
+
+ async def _good_job() -> AsyncIterator[dict]:
+ for i in range(3):
+ yield {"id": f"good:{i}"}
+
+ async with asyncio.timeout(5):
+ items = [
+ item
+ async for item in scraper.fan_out(
+ [_blocked_job(), _good_job()], concurrency=2
+ )
+ ]
+ assert [it["id"] for it in items] == ["good:0", "good:1", "good:2"]
+
+
+def _profile_payload(username: str, n: int) -> dict:
+ # IDs namespaced per target so cross-target de-dup doesn't collapse them.
+ return {
+ "data": {
+ "user": {
+ "id": f"u_{username}",
+ "username": username,
+ "edge_owner_to_timeline_media": {
+ "count": n,
+ "edges": [{"node": {"id": f"{username}:{i}"}} for i in range(n)],
+ },
+ }
+ }
+ }
+
+
+async def test_scrape_instagram_closes_sessions_when_limit_stops_inflight_workers(
+ monkeypatch,
+):
+ """Hitting ``limit`` must tear down the whole fan-out chain deterministically.
+
+ Regression: closing the outer ``iter_instagram`` generator does NOT
+ synchronously close the inner ``fan_out`` it loops over — CPython defers that
+ to async-gen GC — so without an explicit ``aclosing`` the collector's early
+ ``break`` leaked every warm proxy session that was still mid-fetch. The
+ ``fan_out``-direct test misses this because instant jobs self-drain before
+ cancellation ever runs; here each fetch yields to the loop so workers are
+ genuinely in-flight when the limit trips.
+ """
+ holders: list[_TrackingHolder] = []
+
+ async def _fake_open():
+ h = _TrackingHolder()
+ holders.append(h)
+ return h
+
+ @asynccontextmanager
+ async def _fake_bind(_holder):
+ yield _holder
+
+ async def _fetch(path, params=None):
+ await asyncio.sleep(0) # yield control: keep sibling workers in-flight
+ username = (params or {}).get("username", "acct")
+ return _profile_payload(username, 5)
+
+ monkeypatch.setattr(scraper, "open_proxy_holder", _fake_open)
+ monkeypatch.setattr(scraper, "bind_proxy_holder", _fake_bind)
+ monkeypatch.setattr(scraper, "fetch_json", _fetch)
+
+ model = scraper.InstagramScrapeInput(
+ resultsType="posts",
+ directUrls=[f"https://www.instagram.com/acct{i}/" for i in range(50)],
+ resultsLimit=5,
+ )
+ items = await scraper.scrape_instagram(model, limit=3)
+
+ assert len(items) == 3
+ assert holders, "workers should have opened sessions"
+ assert all(h.closed for h in holders), "early stop leaked a proxy session"
+
+
+async def test_discover_profile_is_anonymous_handle_lookup():
+ # keyword search (topsearch) is login-walled, so a profile/user query resolves
+ # as a DIRECT handle lookup against the anonymous profile endpoint — no network
+ # here, just the URL resolution, so no session/monkeypatch needed.
+ targets = await scraper._discover("messi", search_type="user", limit=10)
+ assert [(t.kind, t.value) for t in targets] == [("profile", "messi")]
+
+
+async def test_discover_nonhandle_routes_through_google(monkeypatch):
+ # A non-handle profile query goes through Google (site:instagram.com) and
+ # classifies the organic results into profile targets (the only kind now).
+ async def _fake_scrape_serps(input_model, *, limit=None):
+ assert input_model.site == "instagram.com"
+ return [
+ {
+ "organicResults": [
+ {"url": "https://www.instagram.com/natgeo/"},
+ {"url": "https://www.instagram.com/p/Cabc/"}, # wrong kind
+ ]
+ }
+ ]
+
+ monkeypatch.setattr(scraper, "scrape_serps", _fake_scrape_serps)
+ targets = await scraper._discover(
+ "national geographic", search_type="profile", limit=10
+ )
+ assert [(t.kind, t.value) for t in targets] == [("profile", "natgeo")]
diff --git a/surfsense_backend/tests/unit/platforms/instagram/test_parsers.py b/surfsense_backend/tests/unit/platforms/instagram/test_parsers.py
new file mode 100644
index 000000000..1cfa386a7
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/instagram/test_parsers.py
@@ -0,0 +1,304 @@
+"""Offline parser tests: raw web JSON -> flat item dicts.
+
+Synthetic nodes cover the GraphQL ``edge_*`` flattening the anonymous web
+payloads use. A fixture-pinned test runs only when a captured fixture is present
+(the live e2e script dumps trimmed, PII-anonymized fixtures), so the suite stays
+green offline.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from app.proprietary.platforms.instagram.parsers import (
+ parse_media,
+ parse_post,
+ parse_profile,
+)
+
+_FIXTURES = Path(__file__).parent / "fixtures"
+
+
+def _edge(nodes: list[dict]) -> dict:
+ return {"edges": [{"node": n} for n in nodes]}
+
+
+def test_parse_media_flattens_edges_and_extracts_tags():
+ node = {
+ "id": "1",
+ "shortcode": "Cabc",
+ "__typename": "GraphImage",
+ "taken_at_timestamp": 1_600_000_000,
+ "edge_media_to_caption": _edge([{"text": "love #nasa shot @buzz"}]),
+ "edge_media_to_comment": {"count": 7},
+ "edge_liked_by": {"count": 42},
+ "owner": {"username": "natgeo", "id": "9"},
+ }
+ item = parse_media(node)
+ assert item["shortCode"] == "Cabc"
+ assert item["type"] == "Image"
+ assert item["hashtags"] == ["nasa"]
+ assert item["mentions"] == ["buzz"]
+ assert item["commentsCount"] == 7
+ assert item["likesCount"] == 42
+ assert item["ownerUsername"] == "natgeo"
+ assert item["url"] == "https://www.instagram.com/p/Cabc/"
+
+
+def test_parse_media_passes_through_hidden_like_count():
+ # Instagram reports -1 when the creator hid likes; never coerce it away.
+ item = parse_media({"id": "1", "edge_liked_by": {"count": -1}})
+ assert item["likesCount"] == -1
+
+
+def test_parse_media_marks_video_type():
+ item = parse_media({"id": "1", "is_video": True, "video_view_count": 99})
+ assert item["type"] == "Video"
+ assert item["videoViewCount"] == 99
+
+
+def test_parse_media_extracts_sidecar_tags_location_pinned():
+ # The anonymous profile feed node carries far more than the core fields:
+ # sidecar children, tagged users, coauthors, location, product type and pin
+ # state — all populated here from the real GraphQL key shapes.
+ node = {
+ "id": "1",
+ "shortcode": "Cabc",
+ "__typename": "GraphSidecar",
+ "taken_at_timestamp": 1_704_164_645,
+ "edge_media_to_caption": _edge([{"text": "x #tag @me"}]),
+ "pinned_for_users": [{"id": "9"}],
+ "product_type": "feed",
+ "location": {"id": "55", "name": "Paris"},
+ "coauthor_producers": [{"username": "co", "id": "7"}],
+ "edge_media_to_tagged_user": _edge(
+ [{"user": {"username": "tg", "id": "3"}, "x": 0.1, "y": 0.2}]
+ ),
+ "edge_sidecar_to_children": _edge(
+ [
+ {
+ "id": "c1",
+ "shortcode": "s1",
+ "display_url": "https://cdn/1.jpg",
+ "dimensions": {"height": 10, "width": 20},
+ },
+ {
+ "id": "c2",
+ "shortcode": "s2",
+ "is_video": True,
+ "video_url": "https://cdn/2.mp4",
+ "display_url": "https://cdn/2.jpg",
+ },
+ ]
+ ),
+ }
+ item = parse_media(node)
+ assert item["type"] == "Sidecar"
+ assert item["isPinned"] is True
+ assert item["productType"] == "feed"
+ assert item["locationName"] == "Paris"
+ assert item["locationId"] == "55"
+ assert item["taggedUsers"][0]["username"] == "tg"
+ assert item["coauthorProducers"][0]["username"] == "co"
+ assert item["images"] == ["https://cdn/1.jpg", "https://cdn/2.jpg"]
+ assert len(item["childPosts"]) == 2
+ assert item["childPosts"][1]["type"] == "Video"
+ assert item["childPosts"][1]["videoUrl"] == "https://cdn/2.mp4"
+
+
+def test_parse_media_unpinned_is_false():
+ assert parse_media({"id": "1"})["isPinned"] is False
+
+
+def test_parse_profile_flattens_counts_and_latest_posts():
+ user = {
+ "id": "9",
+ "username": "natgeo",
+ "full_name": "Nat Geo",
+ "edge_followed_by": {"count": 1000},
+ "edge_follow": {"count": 50},
+ "edge_owner_to_timeline_media": {
+ "count": 2,
+ "edges": [{"node": {"id": "p1", "shortcode": "A"}}],
+ },
+ "edge_related_profiles": _edge(
+ [{"username": "similar1", "id": "11"}, {"username": "similar2", "id": "12"}]
+ ),
+ }
+ item = parse_profile(user)
+ assert item["detailKind"] == "profile"
+ assert item["username"] == "natgeo"
+ assert item["followersCount"] == 1000
+ assert item["followsCount"] == 50
+ assert item["postsCount"] == 2
+ assert len(item["latestPosts"]) == 1
+ assert [p["username"] for p in item["relatedProfiles"]] == ["similar1", "similar2"]
+
+
+_POST_URL = "https://www.instagram.com/p/Cabc/"
+
+
+def test_parse_post_prefers_relay_json():
+ # Anonymous /p/ pages inline the mobile-v1 PolarisMedia object in an
+ # application/json script. It's the full-fidelity source (carousel children,
+ # tagged users, coauthors, location, precise timestamp), preferred over og.
+ media = {
+ "pk": "3938367641542741384",
+ "id": "POLARIS_3938367641542741384",
+ "code": "Cabc",
+ "taken_at": 1_704_164_645,
+ "media_type": 8,
+ "product_type": "carousel_container",
+ "like_count": 4200,
+ "comment_count": 37,
+ "accessibility_caption": "alt text",
+ "caption": {"text": "sunset over #bali with @friend @friend"},
+ "user": {"username": "natgeo", "full_name": "Nat Geo", "id": "9"},
+ "image_versions2": {"candidates": [{"url": "https://cdn/i.jpg"}]},
+ "carousel_media": [
+ {
+ "id": "m1",
+ "code": "c1",
+ "media_type": 1,
+ "image_versions2": {"candidates": [{"url": "https://cdn/c1.jpg"}]},
+ "original_height": 1080,
+ "original_width": 1080,
+ },
+ {
+ "id": "m2",
+ "code": "c2",
+ "media_type": 2,
+ "video_versions": [{"url": "https://cdn/c2.mp4"}],
+ "image_versions2": {"candidates": [{"url": "https://cdn/c2.jpg"}]},
+ },
+ ],
+ "usertags": {
+ "in": [
+ {"position": [0.5, 0.5], "user": {"username": "tagged1", "id": "77"}}
+ ]
+ },
+ "coauthor_producers": [
+ {"username": "coauthor1", "id": "88", "is_verified": True}
+ ],
+ "location": {"id": "123", "name": "Bali"},
+ }
+ html = (
+ '"
+ )
+ item = parse_post(html, url=_POST_URL, shortcode="Cabc")
+ assert item is not None
+ assert item["id"] == "3938367641542741384" # POLARIS_ prefix stripped
+ assert item["type"] == "Sidecar" # media_type 8
+ assert item["shortCode"] == "Cabc"
+ assert item["url"] == _POST_URL
+ assert item["caption"] == "sunset over #bali with @friend @friend"
+ assert item["hashtags"] == ["bali"]
+ assert item["mentions"] == ["friend"] # deduped
+ assert item["likesCount"] == 4200
+ assert item["commentsCount"] == 37
+ assert item["displayUrl"] == "https://cdn/i.jpg"
+ assert item["timestamp"].startswith("2024-01-02T") # real epoch -> ISO w/ time
+ assert item["ownerUsername"] == "natgeo"
+ assert item["ownerFullName"] == "Nat Geo"
+ assert item["images"] == ["https://cdn/c1.jpg", "https://cdn/c2.jpg"]
+ assert len(item["childPosts"]) == 2
+ assert item["childPosts"][1]["type"] == "Video"
+ assert item["childPosts"][1]["videoUrl"] == "https://cdn/c2.mp4"
+ assert item["taggedUsers"][0]["username"] == "tagged1"
+ assert item["coauthorProducers"][0]["username"] == "coauthor1"
+ assert item["locationName"] == "Bali"
+ assert item["locationId"] == "123"
+ assert item["productType"] == "carousel_container"
+
+
+def test_parse_post_falls_back_to_og_meta():
+ # Anonymous /p/ pages carry no ld+json; everything is lifted from the og
+ # tags. og:description gives counts + username + date; og:title gives the
+ # clean caption + full name. Entities in the caption are deduped.
+ html = """
+
+
+
+
+
+
+
+ """
+ item = parse_post(html, url=_POST_URL, shortcode="Cabc")
+ assert item is not None
+ assert item["id"] == "3938367641542741384" # numeric pk from al:ios:url meta
+ assert item["likesCount"] == 1234
+ assert item["commentsCount"] == 56
+ assert item["displayUrl"] == "https://cdn/i.jpg"
+ assert item["type"] == "Video"
+ assert item["ownerUsername"] == "natgeo"
+ assert item["ownerFullName"] == "Nat Geo"
+ assert item["timestamp"] == "2024-01-02" # og carries date only, no time
+ assert item["caption"] == "a caption #wow #wow @buzz." # @ -> @, unescaped
+ assert item["hashtags"] == ["wow"] # deduped, no @-as-#064 pollution
+ assert item["mentions"] == ["buzz"] # trailing sentence dot stripped
+
+
+def test_parse_post_og_degrades_without_crashing():
+ # A shape we don't recognise (hidden likes / a non-English locale that
+ # slipped the en-US header) must yield a partial item with None fields,
+ # never an exception or a caption polluted with the counts/date prefix.
+ html = """
+
+
+
+
+
+
+ """
+ item = parse_post(html, url=_POST_URL, shortcode="Cabc")
+ assert item is not None # og:image present -> still emits
+ assert item["displayUrl"] == "https://cdn/i.jpg"
+ assert item["likesCount"] is None
+ assert item["commentsCount"] is None
+ assert item["ownerUsername"] is None
+ assert item["timestamp"] is None
+ assert item["caption"] is None # unrecognised prefix -> no pollution
+
+
+def test_parse_post_returns_none_without_surfaces():
+ # A login interstitial / empty doc carries neither ld+json nor og -> None,
+ # never a silent empty-success item.
+ assert parse_post("login", url=_POST_URL) is None
+ assert parse_post(None, url=_POST_URL) is None
+ assert parse_post("", url=_POST_URL) is None
+
+
+@pytest.mark.skipif(
+ not (_FIXTURES / "profile.json").exists(),
+ reason="captured fixture absent (run scripts/e2e_instagram_scraper.py to dump)",
+)
+def test_fixture_profile_maps():
+ raw = json.loads((_FIXTURES / "profile.json").read_text())
+ user = raw.get("data", {}).get("user", raw)
+ item = parse_profile(user)
+ assert item["detailKind"] == "profile"
+ assert item["username"]
+
+
+@pytest.mark.skipif(
+ not (_FIXTURES / "post.json").exists(),
+ reason="captured fixture absent (run the single-post probe to dump /p/ HTML)",
+)
+def test_fixture_post_maps():
+ raw = json.loads((_FIXTURES / "post.json").read_text())
+ item = parse_post(raw["html"], url=raw["url"], shortcode=raw.get("shortcode"))
+ assert item is not None, "captured /p/ HTML produced no media item"
+ assert item["url"] == raw["url"]
+ # The relay blob (not og-meta) should drive extraction: numeric id + a
+ # precise timestamp with a time component (og-only would be date-only).
+ assert item["id"] and item["id"].isdigit()
+ assert item["ownerUsername"]
+ assert item["timestamp"] and "T" in item["timestamp"]
diff --git a/surfsense_backend/tests/unit/platforms/instagram/test_skeleton.py b/surfsense_backend/tests/unit/platforms/instagram/test_skeleton.py
new file mode 100644
index 000000000..f609883eb
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/instagram/test_skeleton.py
@@ -0,0 +1,102 @@
+"""Offline skeleton tests: input surface parity + URL classification.
+
+No network. Locks the two invariants the reference-compatible surface promises —
+no auth fields ever, and additive ``extra="allow"`` parity — plus the
+``url_resolver`` classification/normalization table (``_u/`` and profilecard
+stripping, story→profile, numeric post-ID flagging). Hashtag/place URLs are
+login-walled and deliberately resolve to ``None``.
+"""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.instagram.schemas import (
+ InstagramMediaItem,
+ InstagramScrapeInput,
+)
+from app.proprietary.platforms.instagram.url_resolver import resolve_url
+
+
+def test_input_has_no_auth_fields():
+ # Anonymous-only: the input surface must never expose a login/credential seam.
+ forbidden = {
+ "sessionid",
+ "username",
+ "password",
+ "cookies",
+ "authorization",
+ "proxyConfiguration",
+ "loginCredentials",
+ }
+ assert forbidden.isdisjoint(InstagramScrapeInput.model_fields)
+
+
+def test_input_defaults():
+ model = InstagramScrapeInput()
+ assert model.resultsType == "posts"
+ assert model.searchType == "profile"
+ assert model.directUrls == []
+ assert model.addParentData is False
+
+
+def test_input_allows_extra_inert_fields():
+ # A reference field the acquisition layer doesn't source is accepted, not rejected.
+ model = InstagramScrapeInput(enhanceUserSearchWithFacebookPage="x")
+ assert model.model_dump().get("enhanceUserSearchWithFacebookPage") == "x"
+
+
+def test_media_item_to_output_keeps_none_keys():
+ out = InstagramMediaItem(id="123", shortCode="abc").to_output()
+ assert out["id"] == "123"
+ assert out["shortCode"] == "abc"
+ # Unsourced fields stay present as None / [] for additive parity.
+ assert out["likesCount"] is None
+ assert out["requestErrorMessages"] == []
+
+
+def test_resolve_profile():
+ r = resolve_url("https://www.instagram.com/natgeo/")
+ assert r.kind == "profile"
+ assert r.value == "natgeo"
+
+
+def test_resolve_bare_profile_id():
+ r = resolve_url("natgeo")
+ assert r.kind == "profile" and r.value == "natgeo"
+
+
+def test_resolve_post_and_reel():
+ r = resolve_url("https://www.instagram.com/p/Cabc123/")
+ assert r.kind == "post" and r.value == "Cabc123" and r.numeric_post_id is False
+ r = resolve_url("https://www.instagram.com/reel/Cxyz/")
+ assert r.kind == "reel" and r.value == "Cxyz"
+
+
+def test_resolve_hashtag_and_place_unsupported():
+ # Login-walled surfaces: they must resolve to None so the orchestrator skips
+ # them rather than building a target that can only return a login wall.
+ assert resolve_url("https://www.instagram.com/explore/tags/crossfit/") is None
+ assert (
+ resolve_url("https://www.instagram.com/explore/locations/7538318/copenhagen/")
+ is None
+ )
+
+
+def test_resolve_strips_u_and_profilecard():
+ stripped_u = resolve_url("https://www.instagram.com/_u/natgeo/")
+ assert stripped_u.kind == "profile" and stripped_u.value == "natgeo"
+ card = resolve_url("https://www.instagram.com/natgeo/profilecard/")
+ assert card.kind == "profile" and card.value == "natgeo"
+
+
+def test_resolve_story_reduces_to_profile():
+ r = resolve_url("https://www.instagram.com/stories/natgeo/12345/")
+ assert r.kind == "profile" and r.value == "natgeo"
+
+
+def test_resolve_numeric_post_id_flagged():
+ r = resolve_url("https://www.instagram.com/p/12345/")
+ assert r.kind == "post" and r.numeric_post_id is True
+
+
+def test_resolve_rejects_non_instagram_host():
+ assert resolve_url("https://example.com/natgeo/") is None
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/__init__.py b/surfsense_backend/tests/unit/platforms/tiktok/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_comments.py b/surfsense_backend/tests/unit/platforms/tiktok/test_comments.py
new file mode 100644
index 000000000..e118f3c01
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_comments.py
@@ -0,0 +1,89 @@
+"""Comments orchestration over a fake fetch (no network).
+
+Drives ``scrape_tiktok_comments``: video URLs -> captured raw comments -> items.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from app.proprietary.platforms.tiktok import scrape_tiktok_comments
+
+_VIDEO = "https://www.tiktok.com/@bob/video/123"
+
+
+def _comment(cid: str, reply_id: str = "0") -> dict[str, Any]:
+ return {
+ "cid": cid,
+ "text": f"comment {cid}",
+ "digg_count": 7,
+ "reply_comment_total": 2,
+ "create_time": 1700000000,
+ "reply_id": reply_id,
+ "user": {
+ "uid": "u1",
+ "unique_id": "alice",
+ "nickname": "Alice",
+ "avatar_thumb": {"url_list": ["https://cdn/a.webp"]},
+ },
+ }
+
+
+async def test_comments_parse_dedupe_and_cap():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ return [_comment("1"), _comment("1"), _comment("2", reply_id="1")]
+
+ items = await scrape_tiktok_comments(
+ [_VIDEO], per_video=2, fetch_comments_fn=fake_fetch
+ )
+
+ assert [i["id"] for i in items] == ["1", "2"]
+ first = items[0]
+ assert first["text"] == "comment 1"
+ assert first["videoWebUrl"] == _VIDEO
+ assert first["diggCount"] == 7
+ assert first["uniqueId"] == "alice"
+ assert first["avatar"] == "https://cdn/a.webp"
+ assert first["createTimeISO"] is not None
+ assert first["repliesToId"] is None # reply_id "0" == top-level
+ assert first["scrapedAt"] is not None
+ assert items[1]["repliesToId"] == "1" # a reply carries its parent id
+
+
+async def test_empty_video_emits_error_item():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ return []
+
+ items = await scrape_tiktok_comments(
+ [_VIDEO], per_video=5, fetch_comments_fn=fake_fetch
+ )
+
+ assert len(items) == 1
+ assert items[0]["errorCode"] == "no_comments"
+ assert items[0]["input"] == "123"
+
+
+async def test_non_video_url_emits_bad_url_error():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ raise AssertionError("should not fetch for a non-video URL")
+
+ items = await scrape_tiktok_comments(
+ ["https://www.tiktok.com/@bob"], per_video=5, fetch_comments_fn=fake_fetch
+ )
+
+ assert len(items) == 1
+ assert items[0]["errorCode"] == "bad_url"
+
+
+async def test_comments_honor_limit_across_videos():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ return [_comment("1"), _comment("2")]
+
+ items = await scrape_tiktok_comments(
+ [_VIDEO, "https://www.tiktok.com/@bob/video/456"],
+ per_video=5,
+ limit=3,
+ fetch_comments_fn=fake_fetch,
+ )
+
+ assert len(items) == 3
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_fetch_resilience.py b/surfsense_backend/tests/unit/platforms/tiktok/test_fetch_resilience.py
new file mode 100644
index 000000000..a2177233e
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_fetch_resilience.py
@@ -0,0 +1,135 @@
+"""Fetch-seam resilience for the TikTok scraper (no network, fake sessions).
+
+Fake sessions drive the cookie warm-up + rotate-on-block + backoff branches
+deterministically; a live first IP normally warms and returns 200s.
+"""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.session import (
+ TikTokAccessBlockedError,
+ client,
+)
+from app.proprietary.platforms.tiktok.session.proxy import _current_session
+
+_HTML = "ok"
+
+
+class _FakePage:
+ def __init__(self, status: int, *, cookies: dict | None = None, body: str = _HTML):
+ self.status = status
+ self.cookies = cookies or {}
+ self.body = body
+
+ @property
+ def text(self) -> str:
+ return self.body
+
+
+class _FakeSession:
+ """One 'IP': homepage warm mints ``ttwid`` per flag; page GETs return ``status``."""
+
+ def __init__(self, status: int = 200, *, warms: bool = True, body: str = _HTML):
+ self.status = status
+ self.warms = warms
+ self.body = body
+ self.page_calls = 0
+ self.warm_calls = 0
+
+ async def get(self, url, headers=None, cookies=None):
+ if url.rstrip("/") == "https://www.tiktok.com":
+ self.warm_calls += 1
+ return _FakePage(200, cookies={"ttwid": "x"} if self.warms else {})
+ self.page_calls += 1
+ return _FakePage(self.status, body=self.body)
+
+
+class _FakeHolder:
+ def __init__(self, sessions: list[_FakeSession]) -> None:
+ self._sessions = sessions
+ self.session = sessions[0]
+ self.rotations = 0
+ self.warmed = False
+
+ async def rotate(self):
+ self.rotations += 1
+ self.session = self._sessions[min(self.rotations, len(self._sessions) - 1)]
+ self.warmed = False
+ return self.session
+
+ async def pace(self) -> None:
+ return None
+
+ async def close(self) -> None:
+ return None
+
+
+def _no_sleep(monkeypatch) -> None:
+ async def _noop(_seconds):
+ return None
+
+ monkeypatch.setattr(client.asyncio, "sleep", _noop)
+
+
+async def test_warms_then_returns_html():
+ holder = _FakeHolder([_FakeSession(200, warms=True)])
+ token = _current_session.set(holder)
+ try:
+ result = await client.fetch_html("https://www.tiktok.com/@scout2015")
+ finally:
+ _current_session.reset(token)
+ assert result == _HTML
+ assert holder.rotations == 0
+ assert holder.session.warm_calls == 1
+
+
+async def test_rotates_when_warm_fails_then_succeeds():
+ holder = _FakeHolder(
+ [_FakeSession(200, warms=False), _FakeSession(200, warms=True)]
+ )
+ token = _current_session.set(holder)
+ try:
+ result = await client.fetch_html("https://www.tiktok.com/@scout2015")
+ finally:
+ _current_session.reset(token)
+ assert result == _HTML
+ assert holder.rotations == 1
+
+
+async def test_404_returns_none_without_rotating():
+ holder = _FakeHolder([_FakeSession(404), _FakeSession(200)])
+ token = _current_session.set(holder)
+ try:
+ result = await client.fetch_html("https://www.tiktok.com/@missing")
+ finally:
+ _current_session.reset(token)
+ assert result is None
+ assert holder.rotations == 0
+
+
+async def test_rotates_and_rewarms_on_403():
+ holder = _FakeHolder([_FakeSession(403), _FakeSession(200, warms=True)])
+ token = _current_session.set(holder)
+ try:
+ result = await client.fetch_html("https://www.tiktok.com/@scout2015")
+ finally:
+ _current_session.reset(token)
+ assert result == _HTML
+ assert holder.rotations == 1
+ assert holder.session.warm_calls == 1
+
+
+async def test_persistent_403_raises_blocked(monkeypatch):
+ _no_sleep(monkeypatch)
+ holder = _FakeHolder([_FakeSession(403) for _ in range(client._MAX_ROTATIONS + 1)])
+ token = _current_session.set(holder)
+ try:
+ raised = False
+ try:
+ await client.fetch_html("https://www.tiktok.com/@scout2015")
+ except TikTokAccessBlockedError:
+ raised = True
+ finally:
+ _current_session.reset(token)
+ assert raised
+ assert holder.rotations == client._MAX_ROTATIONS
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_hydration.py b/surfsense_backend/tests/unit/platforms/tiktok/test_hydration.py
new file mode 100644
index 000000000..cdca568f7
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_hydration.py
@@ -0,0 +1,32 @@
+"""Rehydration-blob extraction from TikTok page HTML (pure, no network)."""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.extraction import extract_rehydration_data
+
+_BLOB = (
+ '"
+)
+
+
+def test_extracts_default_scope_from_blob():
+ data = extract_rehydration_data(_BLOB)
+ assert data is not None
+ item = data["__DEFAULT_SCOPE__"]["webapp.video-detail"]["itemInfo"]["itemStruct"]
+ assert item["id"] == "123"
+
+
+def test_returns_none_when_blob_absent():
+ assert extract_rehydration_data("no blob here") is None
+
+
+def test_returns_none_when_blob_json_malformed():
+ broken = (
+ ''
+ )
+ assert extract_rehydration_data(broken) is None
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_input.py b/surfsense_backend/tests/unit/platforms/tiktok/test_input.py
new file mode 100644
index 000000000..c080ffec3
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_input.py
@@ -0,0 +1,26 @@
+"""Input surface for the TikTok scraper (anonymous, Apify-shaped)."""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.schemas import TikTokScrapeInput
+
+
+def test_input_has_no_auth_fields():
+ forbidden = {"username", "password", "token", "login", "auth", "credentials"}
+ assert forbidden.isdisjoint(TikTokScrapeInput.model_fields)
+
+
+def test_input_defaults():
+ model = TikTokScrapeInput()
+ assert model.resultsPerPage == 1
+ assert model.profileSorting == "latest"
+ assert model.proxyCountryCode == "None"
+ assert model.hashtags == []
+ assert model.profiles == []
+ assert model.searchQueries == []
+ assert model.postURLs == []
+
+
+def test_input_allows_extra_inert_fields():
+ model = TikTokScrapeInput(shouldDownloadVideos=True, videoKvStoreIdOrName="x")
+ assert model.model_dump().get("shouldDownloadVideos") is True
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_item_list.py b/surfsense_backend/tests/unit/platforms/tiktok/test_item_list.py
new file mode 100644
index 000000000..00ab89b02
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_item_list.py
@@ -0,0 +1,25 @@
+"""Pulling item structs out of captured item_list / search API responses."""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.extraction import items_from_response
+
+
+def test_reads_item_list_shape():
+ body = {"itemList": [{"id": "1"}, {"id": "2"}], "hasMore": True}
+ assert items_from_response(body) == [{"id": "1"}, {"id": "2"}]
+
+
+def test_reads_search_data_shape():
+ body = {"data": [{"type": 1, "item": {"id": "9"}}, {"type": 4, "item": {}}]}
+ assert items_from_response(body) == [{"id": "9"}, {}]
+
+
+def test_skips_malformed_entries():
+ body = {"data": [{"type": 1}, "junk", {"item": {"id": "7"}}]}
+ assert items_from_response(body) == [{"id": "7"}]
+
+
+def test_returns_empty_for_unrelated_json():
+ assert items_from_response({"statusCode": 0}) == []
+ assert items_from_response("nope") == []
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_listing_retry.py b/surfsense_backend/tests/unit/platforms/tiktok/test_listing_retry.py
new file mode 100644
index 000000000..2676a2267
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_listing_retry.py
@@ -0,0 +1,60 @@
+"""Retry-on-empty for the browser ``item_list`` seam (no browser, fake fetch).
+
+``fetch_item_list`` re-fetches an empty capture up to ``TIKTOK_LISTING_MAX_ATTEMPTS``
+so a flagged rotating exit IP on the first draw doesn't collapse straight to an
+``ErrorItem``. These drive that loop deterministically by faking ``_fetch_sync``.
+"""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.session import listing
+
+
+class _Fake:
+ """Returns each queued result once, repeating the last; counts calls."""
+
+ def __init__(self, results: list[list[dict]]):
+ self.results = results
+ self.calls = 0
+
+ def __call__(self, *args, **kwargs):
+ out = self.results[min(self.calls, len(self.results) - 1)]
+ self.calls += 1
+ return out
+
+
+def _patch(monkeypatch, fake: _Fake, attempts: int) -> None:
+ monkeypatch.setattr(listing, "_fetch_sync", fake)
+ monkeypatch.setattr(listing.config, "TIKTOK_LISTING_MAX_ATTEMPTS", attempts)
+
+
+async def test_returns_first_nonempty_without_retrying(monkeypatch):
+ fake = _Fake([[{"id": "1"}]])
+ _patch(monkeypatch, fake, 3)
+ items = await listing.fetch_item_list("https://tt/@x", 5)
+ assert items == [{"id": "1"}]
+ assert fake.calls == 1 # a draw with items never retries
+
+
+async def test_retries_past_empty_draws_then_hits(monkeypatch):
+ fake = _Fake([[], [], [{"id": "9"}]])
+ _patch(monkeypatch, fake, 3)
+ items = await listing.fetch_item_list("https://tt/@x", 5)
+ assert items == [{"id": "9"}]
+ assert fake.calls == 3 # two empty (flagged-IP) draws retried, third lands
+
+
+async def test_stops_at_attempt_ceiling_when_always_empty(monkeypatch):
+ fake = _Fake([[]])
+ _patch(monkeypatch, fake, 3)
+ items = await listing.fetch_item_list("https://tt/@x", 5)
+ assert items == []
+ assert fake.calls == 3 # capped; caller then emits the ErrorItem
+
+
+async def test_single_attempt_config_disables_retry(monkeypatch):
+ fake = _Fake([[]])
+ _patch(monkeypatch, fake, 1)
+ items = await listing.fetch_item_list("https://tt/@x", 5)
+ assert items == []
+ assert fake.calls == 1 # static-IP setups opt out via attempts=1
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_orchestrator.py b/surfsense_backend/tests/unit/platforms/tiktok/test_orchestrator.py
new file mode 100644
index 000000000..3a4e42776
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_orchestrator.py
@@ -0,0 +1,185 @@
+"""End-to-end orchestration over a fake fetch (no network).
+
+Drives the public collector: input -> target -> blob-first flow -> items.
+"""
+
+from __future__ import annotations
+
+import json
+
+from app.proprietary.platforms.tiktok import TikTokScrapeInput, scrape_tiktok
+
+
+async def _no_html(_url: str) -> str:
+ """Fetch stub that yields no rehydration blob (skips profile metadata)."""
+ return ""
+
+
+def _profile_page(username: str, followers: int, videos: int) -> str:
+ blob = {
+ "__DEFAULT_SCOPE__": {
+ "webapp.user-detail": {
+ "userInfo": {
+ "user": {
+ "id": "u1",
+ "uniqueId": username,
+ "nickname": "Nick",
+ "verified": True,
+ },
+ "stats": {"followerCount": followers, "videoCount": videos},
+ }
+ }
+ }
+ }
+ return (
+ ''
+ )
+
+
+def _video_page(video_id: str, username: str) -> str:
+ blob = {
+ "__DEFAULT_SCOPE__": {
+ "webapp.video-detail": {
+ "itemInfo": {
+ "itemStruct": {
+ "id": video_id,
+ "desc": "hello",
+ "author": {"uniqueId": username},
+ "stats": {"diggCount": 5},
+ }
+ }
+ }
+ }
+ }
+ return (
+ ''
+ )
+
+
+async def test_scrape_video_url_returns_parsed_item():
+ url = "https://www.tiktok.com/@scout2015/video/123"
+
+ async def fake_fetch(_url: str) -> str:
+ return _video_page("123", "scout2015")
+
+ items = await scrape_tiktok(TikTokScrapeInput(postURLs=[url]), fetch=fake_fetch)
+
+ assert len(items) == 1
+ assert items[0]["id"] == "123"
+ assert items[0]["diggCount"] == 5
+ assert items[0]["webVideoUrl"] == "https://www.tiktok.com/@scout2015/video/123"
+ assert items[0]["scrapedAt"] is not None
+
+
+async def test_scrape_honors_limit_across_targets():
+ urls = [
+ "https://www.tiktok.com/@a/video/1",
+ "https://www.tiktok.com/@b/video/2",
+ ]
+
+ async def fake_fetch(url: str) -> str:
+ vid = url.rsplit("/", 1)[1]
+ user = url.split("@")[1].split("/")[0]
+ return _video_page(vid, user)
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(postURLs=urls), limit=1, fetch=fake_fetch
+ )
+ assert len(items) == 1
+
+
+async def test_scrape_skips_unrecognized_urls():
+ async def fake_fetch(_url: str) -> str:
+ return ""
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(postURLs=["https://example.com/x"]), fetch=fake_fetch
+ )
+ assert items == []
+
+
+async def test_scrape_profile_returns_listing_items():
+ async def fake_listing(_url: str, _count: int) -> list[dict]:
+ return [
+ {"id": "1", "author": {"uniqueId": "a"}},
+ {"id": "2", "author": {"uniqueId": "a"}},
+ ]
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(profiles=["a"], resultsPerPage=5),
+ fetch=_no_html,
+ fetch_listing=fake_listing,
+ )
+ assert [i["id"] for i in items] == ["1", "2"]
+ assert items[0]["webVideoUrl"] == "https://www.tiktok.com/@a/video/1"
+
+
+async def test_profile_emits_metadata_then_videos():
+ # The blob metadata item comes first and is billable; videos follow.
+ async def fake_fetch(_url: str) -> str:
+ return _profile_page("a", followers=100, videos=2)
+
+ async def fake_listing(_url: str, _count: int) -> list[dict]:
+ return [{"id": "1", "author": {"uniqueId": "a"}}]
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(profiles=["a"], resultsPerPage=5),
+ fetch=fake_fetch,
+ fetch_listing=fake_listing,
+ )
+ assert len(items) == 2
+ profile, video = items
+ assert "webVideoUrl" not in profile # metadata item, not a video
+ assert profile["name"] == "a"
+ assert profile["fans"] == 100
+ assert profile["verified"] is True
+ assert profile["scrapedAt"] is not None
+ assert video["id"] == "1"
+
+
+async def test_profile_metadata_survives_blocked_listing():
+ # Videos withheld from anonymous access: we still return the profile metadata
+ # (not just an ErrorItem), so a blocked profile isn't a total loss.
+ async def fake_fetch(_url: str) -> str:
+ return _profile_page("a", followers=100, videos=9)
+
+ async def fake_listing(_url: str, _count: int) -> list[dict]:
+ return []
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(profiles=["a"], resultsPerPage=5),
+ fetch=fake_fetch,
+ fetch_listing=fake_listing,
+ )
+ assert len(items) == 2
+ assert items[0]["name"] == "a"
+ assert items[0]["fans"] == 100
+ assert items[1]["errorCode"] == "no_items"
+
+
+async def test_listing_dedupes_then_caps_per_target():
+ async def fake_listing(_url: str, _count: int) -> list[dict]:
+ return [{"id": "1"}, {"id": "1"}, {"id": "2"}, {"id": "3"}]
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(hashtags=["x"], resultsPerPage=2), fetch_listing=fake_listing
+ )
+ assert [i["id"] for i in items] == ["1", "2"]
+
+
+async def test_empty_listing_emits_error_item():
+ # A trust-gated/empty feed (0 videos) must surface one honest ErrorItem,
+ # tagged with errorCode, rather than vanishing silently.
+ async def fake_listing(_url: str, _count: int) -> list[dict]:
+ return []
+
+ items = await scrape_tiktok(
+ TikTokScrapeInput(profiles=["nasa"], resultsPerPage=5),
+ fetch=_no_html,
+ fetch_listing=fake_listing,
+ )
+ assert len(items) == 1
+ assert items[0]["errorCode"] == "no_items"
+ assert items[0]["input"] == "nasa"
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_parsers.py b/surfsense_backend/tests/unit/platforms/tiktok/test_parsers.py
new file mode 100644
index 000000000..03fa63133
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_parsers.py
@@ -0,0 +1,102 @@
+"""Raw TikTok payload -> normalized item mapping (pure, no network)."""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.extraction import parse_author, parse_video
+
+_ITEM_STRUCT = {
+ "id": "7534061113365859586",
+ "desc": "haha #comeramabanana",
+ "createTime": 1_700_000_000,
+ "author": {
+ "id": "6733",
+ "uniqueId": "bruniela_",
+ "nickname": "Bruni",
+ "verified": False,
+ "signature": "bio here",
+ "avatarLarger": "https://cdn/avatar.jpg",
+ },
+ "authorStats": {
+ "followerCount": 51200,
+ "followingCount": 269,
+ "heartCount": 3_000_000,
+ "videoCount": 259,
+ },
+ "stats": {
+ "diggCount": 5344,
+ "shareCount": 701,
+ "playCount": 55700,
+ "commentCount": 24,
+ "collectCount": 291,
+ },
+ "music": {
+ "id": "7529",
+ "title": "som original",
+ "authorName": "fox_rus0",
+ "original": True,
+ "playUrl": "https://cdn/music.mp3",
+ },
+ "video": {
+ "height": 1024,
+ "width": 576,
+ "duration": 16,
+ "cover": "https://cdn/cover.jpg",
+ "format": "mp4",
+ "definition": "540p",
+ },
+ "challenges": [{"id": "4982299", "title": "comeramabanana"}],
+}
+
+
+def test_parse_video_maps_core_and_derived_fields():
+ item = parse_video(_ITEM_STRUCT)
+
+ assert item["id"] == "7534061113365859586"
+ assert item["text"] == "haha #comeramabanana"
+ assert item["createTimeISO"] == "2023-11-14T22:13:20.000Z"
+
+ assert item["authorMeta"]["name"] == "bruniela_"
+ assert item["authorMeta"]["nickName"] == "Bruni"
+ assert item["authorMeta"]["profileUrl"] == "https://www.tiktok.com/@bruniela_"
+ assert item["authorMeta"]["fans"] == 51200
+
+ assert item["musicMeta"]["musicName"] == "som original"
+ assert item["videoMeta"]["duration"] == 16
+
+ assert item["diggCount"] == 5344
+ assert item["playCount"] == 55700
+
+ assert item["hashtags"] == [{"id": "4982299", "name": "comeramabanana"}]
+ assert (
+ item["webVideoUrl"]
+ == "https://www.tiktok.com/@bruniela_/video/7534061113365859586"
+ )
+
+
+_USER_INFO = {
+ "user": {
+ "id": "6733",
+ "uniqueId": "bruniela_",
+ "nickname": "Bruni",
+ "verified": True,
+ "signature": "bio here",
+ "avatarLarger": "https://cdn/avatar.jpg",
+ "privateAccount": False,
+ },
+ "stats": {
+ "followerCount": 51200,
+ "followingCount": 269,
+ "heartCount": 3_000_000,
+ "videoCount": 259,
+ },
+}
+
+
+def test_parse_author_maps_user_and_stats():
+ author = parse_author(_USER_INFO)
+ assert author["name"] == "bruniela_"
+ assert author["nickName"] == "Bruni"
+ assert author["verified"] is True
+ assert author["profileUrl"] == "https://www.tiktok.com/@bruniela_"
+ assert author["fans"] == 51200
+ assert author["video"] == 259
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_scopes.py b/surfsense_backend/tests/unit/platforms/tiktok/test_scopes.py
new file mode 100644
index 000000000..7a52dd60d
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_scopes.py
@@ -0,0 +1,30 @@
+"""Navigating the rehydration blob to its useful scopes (pure, no network)."""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.extraction import user_info, video_item_struct
+
+
+def test_video_item_struct_navigates_video_detail_scope():
+ data = {
+ "__DEFAULT_SCOPE__": {
+ "webapp.video-detail": {"itemInfo": {"itemStruct": {"id": "123"}}}
+ }
+ }
+ item = video_item_struct(data)
+ assert item == {"id": "123"}
+
+
+def test_user_info_navigates_user_detail_scope():
+ data = {
+ "__DEFAULT_SCOPE__": {
+ "webapp.user-detail": {"userInfo": {"user": {"uniqueId": "scout2015"}}}
+ }
+ }
+ info = user_info(data)
+ assert info == {"user": {"uniqueId": "scout2015"}}
+
+
+def test_scopes_return_none_when_absent():
+ assert video_item_struct({}) is None
+ assert user_info({"__DEFAULT_SCOPE__": {}}) is None
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_target_resolver.py b/surfsense_backend/tests/unit/platforms/tiktok/test_target_resolver.py
new file mode 100644
index 000000000..d246085ae
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_target_resolver.py
@@ -0,0 +1,49 @@
+"""URL classification for the TikTok scraper (pure, no network)."""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok.targets import resolve_target
+
+
+def test_resolve_video_carries_username_and_id():
+ target = resolve_target(
+ "https://www.tiktok.com/@scout2015/video/6718335390845095173"
+ )
+ assert target is not None
+ assert target.kind == "video"
+ assert target.value == "6718335390845095173"
+ assert target.username == "scout2015"
+
+
+def test_resolve_profile():
+ target = resolve_target("https://www.tiktok.com/@scout2015")
+ assert target is not None
+ assert target.kind == "profile"
+ assert target.value == "scout2015"
+
+
+def test_resolve_hashtag():
+ target = resolve_target("https://www.tiktok.com/tag/funny")
+ assert target is not None
+ assert target.kind == "hashtag"
+ assert target.value == "funny"
+
+
+def test_resolve_search_top_video_and_user_sections():
+ top = resolve_target("https://www.tiktok.com/search?q=cats")
+ assert top is not None
+ assert top.kind == "search"
+ assert top.value == "cats"
+ assert top.section is None
+
+ videos = resolve_target("https://www.tiktok.com/search/video?q=cats")
+ assert videos is not None and videos.section == "video"
+
+ users = resolve_target("https://www.tiktok.com/search/user?q=cats")
+ assert users is not None and users.section == "user"
+
+
+def test_resolve_rejects_non_tiktok_and_unknown_paths():
+ assert resolve_target("https://example.com/@scout2015") is None
+ assert resolve_target("https://www.tiktok.com/") is None
+ assert resolve_target("https://www.tiktok.com/foundation") is None
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_trending.py b/surfsense_backend/tests/unit/platforms/tiktok/test_trending.py
new file mode 100644
index 000000000..d00d9c09f
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_trending.py
@@ -0,0 +1,35 @@
+"""Trending orchestration over a fake fetch (no network).
+
+Drives ``scrape_tiktok_trending``: the Explore feed -> captured itemStructs ->
+video items, reusing the listing flow (parse/dedupe/cap/empty-ErrorItem).
+"""
+
+from __future__ import annotations
+
+from app.proprietary.platforms.tiktok import scrape_tiktok_trending
+
+
+async def test_trending_parses_dedupes_and_caps():
+ async def fake_fetch(url: str, _cap: int) -> list[dict]:
+ assert url == "https://www.tiktok.com/explore"
+ return [
+ {"id": "1", "author": {"uniqueId": "a"}},
+ {"id": "1", "author": {"uniqueId": "a"}},
+ {"id": "2", "author": {"uniqueId": "b"}},
+ ]
+
+ items = await scrape_tiktok_trending(count=2, fetch_trending_fn=fake_fetch)
+
+ assert [i["id"] for i in items] == ["1", "2"]
+ assert items[0]["webVideoUrl"] == "https://www.tiktok.com/@a/video/1"
+ assert items[0]["scrapedAt"] is not None
+
+
+async def test_trending_empty_feed_emits_error_item():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ return []
+
+ items = await scrape_tiktok_trending(count=5, fetch_trending_fn=fake_fetch)
+
+ assert len(items) == 1
+ assert items[0]["errorCode"] == "no_items"
diff --git a/surfsense_backend/tests/unit/platforms/tiktok/test_user_search.py b/surfsense_backend/tests/unit/platforms/tiktok/test_user_search.py
new file mode 100644
index 000000000..c4b89e390
--- /dev/null
+++ b/surfsense_backend/tests/unit/platforms/tiktok/test_user_search.py
@@ -0,0 +1,66 @@
+"""User-search orchestration over a fake fetch (no network).
+
+Drives ``search_tiktok_users``: queries -> captured ``user_info`` -> profile items.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from app.proprietary.platforms.tiktok import search_tiktok_users
+
+
+def _user(uid: str, unique_id: str, followers: int = 10) -> dict[str, Any]:
+ return {
+ "uid": uid,
+ "unique_id": unique_id,
+ "nickname": unique_id.upper(),
+ "signature": "bio",
+ "follower_count": followers,
+ "total_favorited": 999,
+ "sec_uid": f"sec-{uid}",
+ "enterprise_verify_reason": "official" if uid == "1" else "",
+ "avatar_thumb": {"url_list": [f"https://cdn/{uid}.webp"]},
+ }
+
+
+async def test_user_search_parses_dedupes_and_caps():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ return [_user("1", "nasa"), _user("1", "nasa"), _user("2", "nasa2")]
+
+ items = await search_tiktok_users(["nasa"], per_query=2, fetch_users=fake_fetch)
+
+ assert [i["id"] for i in items] == ["1", "2"]
+ first = items[0]
+ assert first["name"] == "nasa"
+ assert first["nickName"] == "NASA"
+ assert first["profileUrl"] == "https://www.tiktok.com/@nasa"
+ assert first["verified"] is True
+ assert first["fans"] == 10
+ assert first["avatar"] == "https://cdn/1.webp"
+ assert first["secUid"] == "sec-1"
+ assert first["scrapedAt"] is not None
+ assert items[1]["verified"] is False
+
+
+async def test_user_search_empty_query_emits_error_item():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ return []
+
+ items = await search_tiktok_users(["ghost"], per_query=5, fetch_users=fake_fetch)
+
+ assert len(items) == 1
+ assert items[0]["errorCode"] == "no_users"
+ assert items[0]["input"] == "ghost"
+
+
+async def test_user_search_honors_limit_across_queries():
+ async def fake_fetch(_url: str, _cap: int) -> list[dict]:
+ return [_user("1", "a"), _user("2", "b")]
+
+ items = await search_tiktok_users(
+ ["q1", "q2"], per_query=5, limit=3, fetch_users=fake_fetch
+ )
+
+ # 2 from q1 + 1 from q2, then the cross-query limit stops it.
+ assert len(items) == 3
diff --git a/surfsense_backend/tests/unit/services/test_model_connections.py b/surfsense_backend/tests/unit/services/test_model_connections.py
index b4e7c18d7..bb5ca318e 100644
--- a/surfsense_backend/tests/unit/services/test_model_connections.py
+++ b/surfsense_backend/tests/unit/services/test_model_connections.py
@@ -64,6 +64,22 @@ def test_openai_compatible_resolver_uses_explicit_api_base() -> None:
assert ensure_v1("http://example.com/v1") == "http://example.com/v1"
+def test_openai_compatible_raw_resolver_does_not_append_v1() -> None:
+ model, kwargs = to_litellm(
+ {
+ "provider": "openai_compatible_raw",
+ "base_url": "https://ark.cn-beijing.volces.com/api/v3",
+ "api_key": "ark-key",
+ "extra": {},
+ },
+ "ep-20260101000000-test",
+ )
+
+ assert model == "openai/ep-20260101000000-test"
+ assert kwargs["api_base"] == "https://ark.cn-beijing.volces.com/api/v3"
+ assert kwargs["api_key"] == "ark-key"
+
+
def test_ollama_resolver_uses_native_api_base() -> None:
model, kwargs = to_litellm(
{
diff --git a/surfsense_backend/tests/unit/services/test_requesty_model_normalizer.py b/surfsense_backend/tests/unit/services/test_requesty_model_normalizer.py
new file mode 100644
index 000000000..2adae3ada
--- /dev/null
+++ b/surfsense_backend/tests/unit/services/test_requesty_model_normalizer.py
@@ -0,0 +1,103 @@
+"""Unit tests for Requesty model normalization.
+
+Mirrors the OpenRouter normalizer coverage but exercises Requesty's flat
+boolean capability fields (``supports_tool_calling`` / ``supports_vision``)
+and ``context_window`` sizing.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from app.services.requesty_model_normalizer import (
+ is_requesty_chat_model,
+ is_requesty_image_model,
+ normalize_requesty_models,
+ supports_image_input,
+ supports_tool_calling,
+)
+
+pytestmark = pytest.mark.unit
+
+
+def _requesty_model(
+ *,
+ model_id: str,
+ context_window: int = 128_000,
+ tools: bool = True,
+ vision: bool = False,
+ image_generation: bool = False,
+ name: str | None = None,
+) -> dict:
+ """Return a synthetic Requesty ``/v1/models`` entry.
+
+ Only the fields the normalizer inspects are populated; the live payload
+ carries many more (pricing, ``supports_caching``, ``description``, ...).
+ """
+ return {
+ "id": model_id,
+ "name": name or model_id,
+ "api": "chat",
+ "object": "model",
+ "context_window": context_window,
+ "supports_tool_calling": tools,
+ "supports_vision": vision,
+ "supports_image_generation": image_generation,
+ }
+
+
+def test_chat_model_requires_slash_tools_and_context():
+ assert is_requesty_chat_model(_requesty_model(model_id="openai/gpt-4o-mini"))
+ assert not is_requesty_chat_model(
+ _requesty_model(model_id="openai/gpt-4o-mini", tools=False)
+ )
+ assert not is_requesty_chat_model(
+ _requesty_model(model_id="openai/gpt-4o-mini", context_window=8_000)
+ )
+ assert not is_requesty_chat_model(_requesty_model(model_id="bare-model"))
+
+
+def test_excluded_provider_slug_is_filtered():
+ assert not is_requesty_chat_model(_requesty_model(model_id="amazon/nova-pro-v1"))
+
+
+def test_image_generation_models_excluded_from_chat_and_flagged():
+ image_model = _requesty_model(
+ model_id="google/gemini-2.5-flash-image", image_generation=True
+ )
+ assert not is_requesty_chat_model(image_model)
+ assert is_requesty_image_model(image_model)
+
+
+def test_capability_helpers_read_flat_booleans():
+ model = _requesty_model(
+ model_id="anthropic/claude-sonnet-4-5", vision=True, tools=True
+ )
+ assert supports_image_input(model) is True
+ assert supports_tool_calling(model) is True
+
+
+def test_normalize_maps_context_window_and_capabilities():
+ normalized = normalize_requesty_models(
+ [
+ _requesty_model(
+ model_id="openai/gpt-4o-mini",
+ context_window=128_000,
+ vision=True,
+ name="GPT-4o mini",
+ ),
+ _requesty_model(model_id="openai/gpt-4o-mini", tools=False),
+ _requesty_model(model_id="black-forest-labs/flux", image_generation=True),
+ ]
+ )
+
+ assert len(normalized) == 1
+ entry = normalized[0]
+ assert entry["model_id"] == "openai/gpt-4o-mini"
+ assert entry["display_name"] == "GPT-4o mini"
+ assert entry["supports_chat"] is True
+ assert entry["max_input_tokens"] == 128_000
+ assert entry["supports_image_input"] is True
+ assert entry["supports_tools"] is True
+ assert entry["supports_image_generation"] is False
+ assert entry["metadata"]["id"] == "openai/gpt-4o-mini"
diff --git a/surfsense_backend/tests/unit/tasks/celery_tasks/__init__.py b/surfsense_backend/tests/unit/tasks/celery_tasks/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/surfsense_backend/tests/unit/tasks/celery_tasks/test_schedule_checker_task.py b/surfsense_backend/tests/unit/tasks/celery_tasks/test_schedule_checker_task.py
new file mode 100644
index 000000000..4bca74135
--- /dev/null
+++ b/surfsense_backend/tests/unit/tasks/celery_tasks/test_schedule_checker_task.py
@@ -0,0 +1,129 @@
+"""Unit tests for the periodic schedule checker's connector-to-task dispatch."""
+
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+from datetime import UTC, datetime, timedelta
+from types import SimpleNamespace
+from unittest.mock import MagicMock
+
+import pytest
+
+from app.db import SearchSourceConnectorType
+from app.tasks.celery_tasks import schedule_checker_task
+
+pytestmark = pytest.mark.unit
+
+
+class _FakeScalars:
+ def __init__(self, connectors):
+ self._connectors = connectors
+
+ def all(self):
+ return self._connectors
+
+
+class _FakeDueConnectorsResult:
+ def __init__(self, connectors):
+ self._connectors = connectors
+
+ def scalars(self):
+ return _FakeScalars(self._connectors)
+
+
+class _FakeEmptyResult:
+ def first(self):
+ return None
+
+
+class _FakeSession:
+ """Session stub: first execute() returns due connectors, later ones no rows."""
+
+ def __init__(self, connectors):
+ self._results = [_FakeDueConnectorsResult(connectors)]
+ self.commits = 0
+
+ async def execute(self, _query):
+ if self._results:
+ return self._results.pop(0)
+ return _FakeEmptyResult()
+
+ async def commit(self):
+ self.commits += 1
+
+ async def rollback(self):
+ pass
+
+
+def _due_connector(connector_type: SearchSourceConnectorType) -> SimpleNamespace:
+ return SimpleNamespace(
+ id=42,
+ connector_type=connector_type,
+ search_space_id=7,
+ user_id="00000000-0000-0000-0000-000000000001",
+ config={},
+ periodic_indexing_enabled=True,
+ indexing_frequency_minutes=60,
+ next_scheduled_at=datetime.now(UTC) - timedelta(minutes=5),
+ )
+
+
+async def _run_checker(monkeypatch: pytest.MonkeyPatch, connector: SimpleNamespace):
+ session = _FakeSession([connector])
+
+ @asynccontextmanager
+ async def _session_ctx():
+ yield session
+
+ monkeypatch.setattr(
+ schedule_checker_task, "get_celery_session_maker", lambda: _session_ctx
+ )
+ monkeypatch.setattr(
+ schedule_checker_task, "is_connector_indexing_locked", lambda _id: False
+ )
+ await schedule_checker_task._check_and_trigger_schedules()
+ return session
+
+
+@pytest.mark.asyncio
+async def test_due_bookstack_connector_dispatches_indexing_task(monkeypatch):
+ """A due BookStack connector must dispatch index_bookstack_pages_task.
+
+ Regression test for the connector type missing from the scheduler's
+ task_map, which made periodic BookStack syncs silently no-op with only a
+ "No task found" warning.
+ """
+ from app.tasks.celery_tasks import connector_tasks
+
+ task_mock = MagicMock()
+ monkeypatch.setattr(connector_tasks, "index_bookstack_pages_task", task_mock)
+
+ connector = _due_connector(SearchSourceConnectorType.BOOKSTACK_CONNECTOR)
+ session = await _run_checker(monkeypatch, connector)
+
+ task_mock.delay.assert_called_once_with(
+ connector.id,
+ connector.search_space_id,
+ str(connector.user_id),
+ None,
+ None,
+ )
+ # The next run must be rescheduled, otherwise the connector stays "due"
+ # and is re-examined every minute.
+ assert connector.next_scheduled_at > datetime.now(UTC)
+ assert session.commits == 1
+
+
+@pytest.mark.asyncio
+async def test_unmapped_connector_type_does_not_dispatch(monkeypatch):
+ """Connector types absent from task_map are skipped without dispatching."""
+ from app.tasks.celery_tasks import connector_tasks
+
+ task_mock = MagicMock()
+ monkeypatch.setattr(connector_tasks, "index_bookstack_pages_task", task_mock)
+
+ connector = _due_connector(SearchSourceConnectorType.TAVILY_API)
+ session = await _run_checker(monkeypatch, connector)
+
+ task_mock.delay.assert_not_called()
+ assert session.commits == 0
diff --git a/surfsense_backend/uv.lock b/surfsense_backend/uv.lock
index a5b492fc8..2b1272363 100644
--- a/surfsense_backend/uv.lock
+++ b/surfsense_backend/uv.lock
@@ -36,6 +36,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'linux' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -70,6 +73,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform == 'linux' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -104,6 +110,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -138,6 +147,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -172,6 +184,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -206,6 +221,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -240,6 +258,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -274,6 +295,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -308,6 +332,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform != 'linux' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -342,6 +369,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra == 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -387,6 +417,10 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'linux' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -421,6 +455,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'linux' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -455,6 +492,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform == 'linux' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -489,6 +529,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -523,6 +566,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -557,6 +603,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -591,6 +640,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -625,6 +677,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -659,6 +714,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -693,6 +751,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform != 'linux' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -727,6 +788,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra == 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -772,6 +836,10 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'linux' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -806,6 +874,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'linux' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -840,6 +911,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform == 'linux' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -874,6 +948,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'win32' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -908,6 +985,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -942,6 +1022,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -976,6 +1059,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'win32' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1010,6 +1096,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1044,6 +1133,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1078,6 +1170,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform != 'linux' and sys_platform != 'win32' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1112,6 +1207,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform == 'win32' and extra == 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1157,6 +1255,10 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1191,6 +1293,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1258,6 +1363,12 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1292,6 +1403,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1326,6 +1440,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1393,6 +1510,12 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1460,6 +1583,12 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform != 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
"python_version < '0'",
"python_version < '0'",
@@ -1494,6 +1623,9 @@ resolution-markers = [
"python_version < '0'",
"python_version < '0'",
"python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
+ "python_version < '0'",
"python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-16-surf-new-backend-cpu' and extra != 'extra-16-surf-new-backend-cu126' and extra != 'extra-16-surf-new-backend-cu128'",
]
conflicts = [[
@@ -10353,7 +10485,7 @@ wheels = [
[[package]]
name = "surf-new-backend"
-version = "0.0.31"
+version = "0.0.32"
source = { editable = "." }
dependencies = [
{ name = "alembic" },
diff --git a/surfsense_browser_extension/package.json b/surfsense_browser_extension/package.json
index 8dda0d82f..a2c25a602 100644
--- a/surfsense_browser_extension/package.json
+++ b/surfsense_browser_extension/package.json
@@ -1,7 +1,7 @@
{
"name": "surfsense_browser_extension",
"displayName": "Surfsense Browser Extension",
- "version": "0.0.31",
+ "version": "0.0.32",
"description": "Extension to collect Browsing History for SurfSense.",
"author": "https://github.com/MODSetter",
"engines": {
diff --git a/surfsense_desktop/package.json b/surfsense_desktop/package.json
index f12b9722c..629fe801d 100644
--- a/surfsense_desktop/package.json
+++ b/surfsense_desktop/package.json
@@ -1,7 +1,7 @@
{
"name": "surfsense-desktop",
"productName": "SurfSense",
- "version": "0.0.31",
+ "version": "0.0.32",
"description": "SurfSense Desktop App",
"main": "dist/main.js",
"scripts": {
diff --git a/surfsense_evals/scripts/analyze_failure_timing.py b/surfsense_evals/scripts/analyze_failure_timing.py
index f4f8aedba..14b76852f 100644
--- a/surfsense_evals/scripts/analyze_failure_timing.py
+++ b/surfsense_evals/scripts/analyze_failure_timing.py
@@ -21,8 +21,7 @@ PDFS = REPO / "data" / "multimodal_doc" / "mmlongbench" / "pdfs"
def main() -> None:
rows = [
- json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines()
- if line.strip()
+ json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines() if line.strip()
]
# 1) SSL clustering: failures by question index per arm
@@ -35,11 +34,19 @@ def main() -> None:
arm_seen_count[arm] += 1
qid_order[f"{arm}::{row['qid']}"] = idx
err = row.get("error") or ""
- cluster = "ssl" if "SSLError" in err else (
- "empty" if not (row.get("raw_text") or "").strip() and not err else (
- "5xx" if "502" in err or "503" in err else (
- "size_limit" if "exceeds" in err.lower() and "limit" in err.lower() else (
- "other_err" if err else "ok"
+ cluster = (
+ "ssl"
+ if "SSLError" in err
+ else (
+ "empty"
+ if not (row.get("raw_text") or "").strip() and not err
+ else (
+ "5xx"
+ if "502" in err or "503" in err
+ else (
+ "size_limit"
+ if "exceeds" in err.lower() and "limit" in err.lower()
+ else ("other_err" if err else "ok")
)
)
)
@@ -100,19 +107,26 @@ def main() -> None:
err = row.get("error") or ""
empty = not (row.get("raw_text") or "").strip()
if err or empty:
- by_pdf[row["doc_id"]].append({
- "arm": row["arm"],
- "qid": row["qid"],
- "err_kind": (
- "ssl" if "SSLError" in err
- else "size_limit" if "exceeds" in err.lower() and "limit" in err.lower()
- else "5xx" if "502" in err or "503" in err
- else "json_decode" if "JSONDecodeError" in err
- else "empty" if empty and not err
- else "other"
- ),
- "pages": row.get("pages"),
- })
+ by_pdf[row["doc_id"]].append(
+ {
+ "arm": row["arm"],
+ "qid": row["qid"],
+ "err_kind": (
+ "ssl"
+ if "SSLError" in err
+ else "size_limit"
+ if "exceeds" in err.lower() and "limit" in err.lower()
+ else "5xx"
+ if "502" in err or "503" in err
+ else "json_decode"
+ if "JSONDecodeError" in err
+ else "empty"
+ if empty and not err
+ else "other"
+ ),
+ "pages": row.get("pages"),
+ }
+ )
for doc, items in sorted(by_pdf.items(), key=lambda x: (-len(x[1]), x[0])):
kinds = Counter(i["err_kind"] for i in items)
arms = sorted({i["arm"] for i in items})
diff --git a/surfsense_evals/scripts/analyze_failures.py b/surfsense_evals/scripts/analyze_failures.py
index e7ace1e1b..f60038c00 100644
--- a/surfsense_evals/scripts/analyze_failures.py
+++ b/surfsense_evals/scripts/analyze_failures.py
@@ -12,12 +12,10 @@ Outputs (printed to stdout + written to `failures_n171.json`):
from __future__ import annotations
import json
-import re
from collections import Counter, defaultdict
from pathlib import Path
from typing import Any
-
REPO = Path(__file__).resolve().parents[1]
RUN = REPO / "data" / "multimodal_doc" / "runs" / "2026-05-14T00-53-19Z" / "parser_compare"
RAW = RUN / "raw.jsonl"
@@ -53,8 +51,7 @@ def _classify(error: str | None, raw_text: str) -> str:
def main() -> None:
rows = [
- json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines()
- if line.strip()
+ json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines() if line.strip()
]
by_arm_failures: dict[str, list[dict]] = defaultdict(list)
@@ -123,7 +120,9 @@ def main() -> None:
print("=" * 90)
for entry in by_arm_failures.get("native_pdf", []):
err = (entry["error"] or "(no error string)")[:240].replace("\n", " ")
- print(f" {entry['qid']} doc={entry['doc_id']} pages={entry['pages']} cluster={entry['cluster']}")
+ print(
+ f" {entry['qid']} doc={entry['doc_id']} pages={entry['pages']} cluster={entry['cluster']}"
+ )
print(f" err: {err}")
summary: dict[str, Any] = {
@@ -132,18 +131,13 @@ def main() -> None:
"n": n_per_arm[arm],
"failures": len(by_arm_failures[arm]),
"rate": len(by_arm_failures[arm]) / n_per_arm[arm],
- "clusters": {
- cluster: len(items)
- for cluster, items in error_clusters[arm].items()
- },
+ "clusters": {cluster: len(items) for cluster, items in error_clusters[arm].items()},
"rows": by_arm_failures[arm],
}
for arm in sorted(n_per_arm)
},
"per_pdf": {
- pdf: [
- {**r, "arm": r["arm"]} for r in failures
- ]
+ pdf: [{**r, "arm": r["arm"]} for r in failures]
for pdf, failures in by_pdf_failures.items()
},
}
diff --git a/surfsense_evals/scripts/check_extraction_sizes.py b/surfsense_evals/scripts/check_extraction_sizes.py
index 712e693cb..1755e9b6c 100644
--- a/surfsense_evals/scripts/check_extraction_sizes.py
+++ b/surfsense_evals/scripts/check_extraction_sizes.py
@@ -23,9 +23,7 @@ SAFE_CHARS = (CTX_TOKENS - PROMPT_OVERHEAD_TOKENS - MAX_OUTPUT_TOKENS) * CHARS_P
def main() -> None:
rows = [
- json.loads(line)
- for line in MAP.read_text(encoding="utf-8").splitlines()
- if line.strip()
+ json.loads(line) for line in MAP.read_text(encoding="utf-8").splitlines() if line.strip()
]
total = len(rows)
diff --git a/surfsense_evals/scripts/check_uploaded_status.py b/surfsense_evals/scripts/check_uploaded_status.py
index 7021ba83d..1903502ed 100644
--- a/surfsense_evals/scripts/check_uploaded_status.py
+++ b/surfsense_evals/scripts/check_uploaded_status.py
@@ -15,7 +15,6 @@ from pathlib import Path
import httpx
from dotenv import load_dotenv
-
REPO = Path(__file__).resolve().parents[1]
PDF_DIR = REPO / "data" / "multimodal_doc" / "mmlongbench" / "pdfs"
diff --git a/surfsense_evals/scripts/compute_adjusted_accuracy.py b/surfsense_evals/scripts/compute_adjusted_accuracy.py
index 13693c055..0cd4b3073 100644
--- a/surfsense_evals/scripts/compute_adjusted_accuracy.py
+++ b/surfsense_evals/scripts/compute_adjusted_accuracy.py
@@ -67,14 +67,18 @@ def classify(error: str | None, raw_text: str) -> str:
def main() -> None:
rows = [
- json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines()
- if line.strip()
+ json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines() if line.strip()
]
- by_arm: dict[str, dict] = defaultdict(lambda: {
- "n": 0, "correct": 0,
- "transient_ssl_or_5xx": 0, "transient_empty": 0,
- "intrinsic_limit": 0, "other_error": 0,
- })
+ by_arm: dict[str, dict] = defaultdict(
+ lambda: {
+ "n": 0,
+ "correct": 0,
+ "transient_ssl_or_5xx": 0,
+ "transient_empty": 0,
+ "intrinsic_limit": 0,
+ "other_error": 0,
+ }
+ )
for row in rows:
arm = row["arm"]
m = by_arm[arm]
@@ -86,7 +90,9 @@ def main() -> None:
if kind != "ok":
m[kind] += 1
- print(f"{'arm':<25} {'raw acc%':>8} {'transient':>10} {'intrinsic':>10} {'other':>6} {'adj acc% (no transient)':>22}")
+ print(
+ f"{'arm':<25} {'raw acc%':>8} {'transient':>10} {'intrinsic':>10} {'other':>6} {'adj acc% (no transient)':>22}"
+ )
print("-" * 88)
for arm in sorted(by_arm):
m = by_arm[arm]
@@ -96,9 +102,7 @@ def main() -> None:
other = m["other_error"]
usable = m["n"] - transient
adj = m["correct"] / usable * 100 if usable else 0
- print(
- f"{arm:<25} {raw:>7.1f}% {transient:>10} {intrinsic:>10} {other:>6} {adj:>21.1f}%"
- )
+ print(f"{arm:<25} {raw:>7.1f}% {transient:>10} {intrinsic:>10} {other:>6} {adj:>21.1f}%")
print()
print("transient = SSLError / 502 / 503 / empty stream / mid-stream JSON decode (would")
diff --git a/surfsense_evals/scripts/compute_blog_extras.py b/surfsense_evals/scripts/compute_blog_extras.py
index abe88d08b..29922f54c 100644
--- a/surfsense_evals/scripts/compute_blog_extras.py
+++ b/surfsense_evals/scripts/compute_blog_extras.py
@@ -95,7 +95,7 @@ def _mcnemar_exact_pvalue(b: int, c: int) -> float:
k = min(b, c)
# Two-sided exact: 2 * P(X <= k) clipped at 1.0
cdf = sum(_binom_coef(n, i) for i in range(k + 1))
- p = 2.0 * cdf / (2 ** n)
+ p = 2.0 * cdf / (2**n)
return min(1.0, p)
@@ -116,7 +116,7 @@ def _mcnemar_table(rows: list[dict]) -> dict:
qids = sorted(by_qid)
out: dict[str, dict] = {"arms": arms, "n_qids": len(qids), "pairs": []}
for i, ai in enumerate(arms):
- for aj in arms[i + 1:]:
+ for aj in arms[i + 1 :]:
b = c = both = neither = 0
for q in qids:
row = by_qid[q]
@@ -132,12 +132,17 @@ def _mcnemar_table(rows: list[dict]) -> dict:
else:
neither += 1
p = _mcnemar_exact_pvalue(b, c)
- out["pairs"].append({
- "arm_i": ai, "arm_j": aj,
- "b_i_only": b, "c_j_only": c,
- "both_correct": both, "both_wrong": neither,
- "p_value": p,
- })
+ out["pairs"].append(
+ {
+ "arm_i": ai,
+ "arm_j": aj,
+ "b_i_only": b,
+ "c_j_only": c,
+ "both_correct": both,
+ "both_wrong": neither,
+ "p_value": p,
+ }
+ )
return out
@@ -154,9 +159,7 @@ def _per_pdf_stats(rows: list[dict]) -> dict[str, dict]:
arm = r["arm"]
pdf = r["doc_id"]
graded = r.get("graded") or {}
- bucket.setdefault(arm, {}).setdefault(pdf, []).append(
- bool(graded.get("correct"))
- )
+ bucket.setdefault(arm, {}).setdefault(pdf, []).append(bool(graded.get("correct")))
out: dict[str, dict] = {}
for arm, pdfs in bucket.items():
@@ -207,7 +210,8 @@ def _per_arm_latency(rows: list[dict]) -> dict[str, dict]:
# Coefficient of variation: std / mean (unitless tail-fatness).
"cv": (
statistics.stdev(lats) / statistics.mean(lats)
- if len(lats) > 1 and statistics.mean(lats) > 0 else 0.0
+ if len(lats) > 1 and statistics.mean(lats) > 0
+ else 0.0
),
}
return out
@@ -259,24 +263,30 @@ def _print_latency(title: str, lat: dict[str, dict]) -> None:
print()
print(title)
print("-" * len(title))
- header = (f"{'arm':<25} {'n':>4} {'mean':>7} {'std':>7} "
- f"{'p50':>7} {'p90':>7} {'p95':>7} {'p99':>7} {'max':>7} {'CV':>5}")
+ header = (
+ f"{'arm':<25} {'n':>4} {'mean':>7} {'std':>7} "
+ f"{'p50':>7} {'p90':>7} {'p95':>7} {'p99':>7} {'max':>7} {'CV':>5}"
+ )
print(header)
print("-" * len(header))
for arm in sorted(lat, key=lambda a: lat[a]["mean_s"]):
s = lat[arm]
- print(f"{arm:<25} {s['n']:>4} "
- f"{s['mean_s']:>6.1f}s {s['std_s']:>6.1f}s "
- f"{s['p50_s']:>6.1f}s {s['p90_s']:>6.1f}s {s['p95_s']:>6.1f}s "
- f"{s['p99_s']:>6.1f}s {s['max_s']:>6.1f}s {s['cv']:>5.2f}")
+ print(
+ f"{arm:<25} {s['n']:>4} "
+ f"{s['mean_s']:>6.1f}s {s['std_s']:>6.1f}s "
+ f"{s['p50_s']:>6.1f}s {s['p90_s']:>6.1f}s {s['p95_s']:>6.1f}s "
+ f"{s['p99_s']:>6.1f}s {s['max_s']:>6.1f}s {s['cv']:>5.2f}"
+ )
def _print_tokens(title: str, toks: dict[str, dict]) -> None:
print()
print(title)
print("-" * len(title))
- header = (f"{'arm':<25} {'in mean':>9} {'in p50':>9} {'in p95':>9} {'in max':>9}"
- f" {'out mean':>9} {'out p95':>9}")
+ header = (
+ f"{'arm':<25} {'in mean':>9} {'in p50':>9} {'in p95':>9} {'in max':>9}"
+ f" {'out mean':>9} {'out p95':>9}"
+ )
print(header)
print("-" * len(header))
for arm in sorted(toks):
@@ -285,25 +295,31 @@ def _print_tokens(title: str, toks: dict[str, dict]) -> None:
eout = e.get("output")
if not ein:
continue
- print(f"{arm:<25} "
- f"{ein['mean']:>9,.0f} {ein['p50']:>9,.0f} {ein['p95']:>9,.0f} {ein['max']:>9,.0f} "
- f"{(eout or {}).get('mean', 0):>9,.0f} {(eout or {}).get('p95', 0):>9,.0f}")
+ print(
+ f"{arm:<25} "
+ f"{ein['mean']:>9,.0f} {ein['p50']:>9,.0f} {ein['p95']:>9,.0f} {ein['max']:>9,.0f} "
+ f"{(eout or {}).get('mean', 0):>9,.0f} {(eout or {}).get('p95', 0):>9,.0f}"
+ )
def _print_pdf_var(title: str, var: dict[str, dict]) -> None:
print()
print(title)
print("-" * len(title))
- header = (f"{'arm':<25} {'n_pdfs':>7} {'mean':>7} {'std':>7} {'min':>7} "
- f"{'p25':>7} {'p50':>7} {'p75':>7} {'max':>7} {'#0%':>5} {'#100%':>6}")
+ header = (
+ f"{'arm':<25} {'n_pdfs':>7} {'mean':>7} {'std':>7} {'min':>7} "
+ f"{'p25':>7} {'p50':>7} {'p75':>7} {'max':>7} {'#0%':>5} {'#100%':>6}"
+ )
print(header)
print("-" * len(header))
for arm in sorted(var, key=lambda a: -var[a]["mean"]):
s = var[arm]
- print(f"{arm:<25} {s['n_pdfs']:>7} "
- f"{s['mean']*100:>6.1f}% {s['std']*100:>6.1f}% {s['min']*100:>6.1f}% "
- f"{s['p25']*100:>6.1f}% {s['p50']*100:>6.1f}% {s['p75']*100:>6.1f}% "
- f"{s['max']*100:>6.1f}% {s['n_pdfs_zero']:>5} {s['n_pdfs_perfect']:>6}")
+ print(
+ f"{arm:<25} {s['n_pdfs']:>7} "
+ f"{s['mean'] * 100:>6.1f}% {s['std'] * 100:>6.1f}% {s['min'] * 100:>6.1f}% "
+ f"{s['p25'] * 100:>6.1f}% {s['p50'] * 100:>6.1f}% {s['p75'] * 100:>6.1f}% "
+ f"{s['max'] * 100:>6.1f}% {s['n_pdfs_zero']:>5} {s['n_pdfs_perfect']:>6}"
+ )
def _print_mcnemar(title: str, table: dict) -> None:
@@ -311,8 +327,10 @@ def _print_mcnemar(title: str, table: dict) -> None:
print(title)
print("-" * len(title))
print(f"n_qids on which all arms have a graded row: {table['n_qids']}")
- header = (f"{'arm_i':<25} {'arm_j':<25} {'b':>4} {'c':>4} "
- f"{'both ok':>8} {'both wr':>8} {'p (2-sided)':>13} {'sig':>4}")
+ header = (
+ f"{'arm_i':<25} {'arm_j':<25} {'b':>4} {'c':>4} "
+ f"{'both ok':>8} {'both wr':>8} {'p (2-sided)':>13} {'sig':>4}"
+ )
print(header)
print("-" * len(header))
for pair in sorted(table["pairs"], key=lambda p: p["p_value"]):
@@ -323,10 +341,12 @@ def _print_mcnemar(title: str, table: dict) -> None:
sig = "**"
elif pair["p_value"] < 0.05:
sig = "*"
- print(f"{pair['arm_i']:<25} {pair['arm_j']:<25} "
- f"{pair['b_i_only']:>4} {pair['c_j_only']:>4} "
- f"{pair['both_correct']:>8} {pair['both_wrong']:>8} "
- f"{pair['p_value']:>13.4f} {sig:>4}")
+ print(
+ f"{pair['arm_i']:<25} {pair['arm_j']:<25} "
+ f"{pair['b_i_only']:>4} {pair['c_j_only']:>4} "
+ f"{pair['both_correct']:>8} {pair['both_wrong']:>8} "
+ f"{pair['p_value']:>13.4f} {sig:>4}"
+ )
# ---------------------------------------------------------------------------
diff --git a/surfsense_evals/scripts/compute_post_retry_accuracy.py b/surfsense_evals/scripts/compute_post_retry_accuracy.py
index 4c8c47672..29007ed41 100644
--- a/surfsense_evals/scripts/compute_post_retry_accuracy.py
+++ b/surfsense_evals/scripts/compute_post_retry_accuracy.py
@@ -47,9 +47,7 @@ def _row_key(row: dict) -> tuple[str, str]:
def _is_failure(row: dict) -> bool:
if row.get("error"):
return True
- if not (row.get("raw_text") or "").strip():
- return True
- return False
+ return bool(not (row.get("raw_text") or "").strip())
def _summarise(rows_by_arm: dict[str, list[dict]]) -> dict[str, dict]:
@@ -80,9 +78,11 @@ def _print_table(title: str, summary: dict[str, dict]) -> None:
# stable order: highest accuracy first
arms_sorted = sorted(summary.items(), key=lambda kv: -kv[1]["accuracy"])
for arm, s in arms_sorted:
- print(f"{arm:<25} {s['n']:>4} {s['n_correct']:>7} "
- f"{s['accuracy']*100:>6.1f}% {s['f1_mean']*100:>6.1f}% "
- f"{s['n_failures']:>6} {s['failure_rate']*100:>6.1f}%")
+ print(
+ f"{arm:<25} {s['n']:>4} {s['n_correct']:>7} "
+ f"{s['accuracy'] * 100:>6.1f}% {s['f1_mean'] * 100:>6.1f}% "
+ f"{s['n_failures']:>6} {s['failure_rate'] * 100:>6.1f}%"
+ )
def main() -> int:
@@ -105,9 +105,7 @@ def main() -> int:
raw_rows = _read_jsonl(raw_path)
retry_rows = _read_jsonl(retry_path)
- retry_by_key: dict[tuple[str, str], dict] = {
- _row_key(r): r for r in retry_rows
- }
+ retry_by_key: dict[tuple[str, str], dict] = {_row_key(r): r for r in retry_rows}
merged_rows: list[dict] = []
n_replaced_recovered = 0
diff --git a/surfsense_evals/scripts/inspect_first30.py b/surfsense_evals/scripts/inspect_first30.py
index e06c6c029..b3caedca6 100644
--- a/surfsense_evals/scripts/inspect_first30.py
+++ b/surfsense_evals/scripts/inspect_first30.py
@@ -44,10 +44,7 @@ def main() -> None:
f"questions covering first 30 docs: total={len(qs_in_30)} "
f"answerable={answerable} unanswerable={unanswerable}"
)
- print(
- f"avg Qs/PDF: {len(qs_in_30) / 30:.1f} "
- f"answerable/PDF: {answerable / 30:.1f}"
- )
+ print(f"avg Qs/PDF: {len(qs_in_30) / 30:.1f} answerable/PDF: {answerable / 30:.1f}")
print(f"format mix in scope: {dict(fmts)}")
print()
print("25 new PDFs to ingest:")
diff --git a/surfsense_evals/scripts/patch_manifest_for_parallel_ingest.py b/surfsense_evals/scripts/patch_manifest_for_parallel_ingest.py
index e1a2edc65..8cfb13eb7 100644
--- a/surfsense_evals/scripts/patch_manifest_for_parallel_ingest.py
+++ b/surfsense_evals/scripts/patch_manifest_for_parallel_ingest.py
@@ -27,7 +27,6 @@ from __future__ import annotations
import json
from pathlib import Path
-
REPO = Path(__file__).resolve().parents[1]
MAP_PATH = REPO / "data" / "multimodal_doc" / "maps" / "mmlongbench_doc_map.jsonl"
PDF_DIR = REPO / "data" / "multimodal_doc" / "mmlongbench" / "pdfs"
diff --git a/surfsense_evals/scripts/peek_crag_run.py b/surfsense_evals/scripts/peek_crag_run.py
index 225e5ec98..0720e24cf 100644
--- a/surfsense_evals/scripts/peek_crag_run.py
+++ b/surfsense_evals/scripts/peek_crag_run.py
@@ -10,26 +10,24 @@ from collections import defaultdict
def main() -> None:
raw_path = sorted(glob.glob("data/research/runs/*/crag/raw.jsonl"))[-1]
print(f"Reading: {raw_path}")
- rows = [json.loads(line) for line in open(raw_path, encoding="utf-8") if line.strip()]
+ with open(raw_path, encoding="utf-8") as fh:
+ rows = [json.loads(line) for line in fh if line.strip()]
by_q: dict[str, dict[str, dict]] = defaultdict(dict)
for r in rows:
by_q[r["qid"]][r["arm"]] = r
for qid, arms in list(by_q.items()):
b = arms.get("bare_llm", {})
- l = arms.get("long_context", {})
+ lc = arms.get("long_context", {})
s = arms.get("surfsense", {})
print(f"\n=== {qid} ({b.get('domain')}/{b.get('question_type')}) ===")
print(f" question: {b.get('extra', {}).get('question', '?')!r}")
print(f" gold: {b.get('gold')!r}")
- for arm_name, a in (("bare_llm", b), ("long_context", l), ("surfsense", s)):
+ for arm_name, a in (("bare_llm", b), ("long_context", lc), ("surfsense", s)):
grade = a.get("graded", {})
text = (a.get("raw_text") or "").strip()
tail = text[-200:] if text else ""
- print(
- f" [{arm_name}] grade={grade.get('grade')} "
- f"method={grade.get('method')}"
- )
+ print(f" [{arm_name}] grade={grade.get('grade')} method={grade.get('method')}")
print(f" -> {tail!r}")
diff --git a/surfsense_evals/scripts/peek_disagreements.py b/surfsense_evals/scripts/peek_disagreements.py
index c0fe0acd9..b6497570e 100644
--- a/surfsense_evals/scripts/peek_disagreements.py
+++ b/surfsense_evals/scripts/peek_disagreements.py
@@ -10,7 +10,8 @@ from collections import defaultdict
def main() -> None:
raw_path = sorted(glob.glob("data/research/runs/*/crag/raw.jsonl"))[-1]
print(f"Reading: {raw_path}")
- rows = [json.loads(line) for line in open(raw_path, encoding="utf-8") if line.strip()]
+ with open(raw_path, encoding="utf-8") as fh:
+ rows = [json.loads(line) for line in fh if line.strip()]
by_q: dict[str, dict[str, dict]] = defaultdict(dict)
for r in rows:
by_q[r["qid"]][r["arm"]] = r
diff --git a/surfsense_evals/scripts/retry_failed_questions.py b/surfsense_evals/scripts/retry_failed_questions.py
index 7cc9478e0..d65f9f0fb 100644
--- a/surfsense_evals/scripts/retry_failed_questions.py
+++ b/surfsense_evals/scripts/retry_failed_questions.py
@@ -106,9 +106,7 @@ def _is_failure_row(row: dict[str, Any]) -> bool:
if row.get("error"):
return True
- if not (row.get("raw_text") or "").strip():
- return True
- return False
+ return bool(not (row.get("raw_text") or "").strip())
@dataclass
@@ -134,17 +132,19 @@ def _load_failed_rows(raw_path: Path) -> list[FailedRow]:
row = json.loads(line)
if not _is_failure_row(row):
continue
- out.append(FailedRow(
- arm=str(row["arm"]),
- qid=str(row["qid"]),
- doc_id=str(row["doc_id"]),
- answer_format=str(row.get("answer_format") or ""),
- gold=str(row.get("gold") or ""),
- pages=int(row.get("pages") or 0),
- document_id=row.get("document_id"),
- original_error=row.get("error"),
- original_row=row,
- ))
+ out.append(
+ FailedRow(
+ arm=str(row["arm"]),
+ qid=str(row["qid"]),
+ doc_id=str(row["doc_id"]),
+ answer_format=str(row.get("answer_format") or ""),
+ gold=str(row.get("gold") or ""),
+ pages=int(row.get("pages") or 0),
+ document_id=row.get("document_id"),
+ original_error=row.get("error"),
+ original_row=row,
+ )
+ )
return out
@@ -204,8 +204,12 @@ def _qid_index(qid: str) -> int:
def _build_native_request(
- qid: str, question: str, answer_format: str, pdf_path: Path,
- *, max_output_tokens: int,
+ qid: str,
+ question: str,
+ answer_format: str,
+ pdf_path: Path,
+ *,
+ max_output_tokens: int,
) -> ArmRequest:
return ArmRequest(
question_id=qid,
@@ -216,12 +220,14 @@ def _build_native_request(
def _build_lc_request(
- qid: str, question: str, answer_format: str, doc_id: str, md_path: Path,
+ qid: str,
+ question: str,
+ answer_format: str,
+ doc_id: str,
+ md_path: Path,
) -> ArmRequest:
if not md_path.exists():
- raise FileNotFoundError(
- f"Missing parser extraction at {md_path}; cannot retry LC arm."
- )
+ raise FileNotFoundError(f"Missing parser extraction at {md_path}; cannot retry LC arm.")
markdown = md_path.read_text(encoding="utf-8")
return ArmRequest(
question_id=qid,
@@ -258,7 +264,9 @@ class RetryOutcome:
async def _retry_one(
- arm_obj: Any, request: ArmRequest, *,
+ arm_obj: Any,
+ request: ArmRequest,
+ *,
arm_name: str,
qid: str,
max_attempts: int,
@@ -276,31 +284,44 @@ async def _retry_one(
attempt_error = result.error
if not attempt_error and not raw_text:
attempt_error = "EmptyResponse: stream ended with no text"
- attempts.append(AttemptLog(
- attempt=attempt,
- started_iso=started_iso,
- latency_ms=latency_ms,
- error=attempt_error,
- raw_text_chars=len(raw_text),
- ))
+ attempts.append(
+ AttemptLog(
+ attempt=attempt,
+ started_iso=started_iso,
+ latency_ms=latency_ms,
+ error=attempt_error,
+ raw_text_chars=len(raw_text),
+ )
+ )
final = result
if not attempt_error and raw_text:
return RetryOutcome(
- arm=arm_name, qid=qid, attempts=attempts,
- final_result=result, recovered=True,
+ arm=arm_name,
+ qid=qid,
+ attempts=attempts,
+ final_result=result,
+ recovered=True,
)
if attempt < max_attempts:
delay = min(max_delay, base_delay * (2 ** (attempt - 1)))
delay = delay * (0.5 + random.random())
logger.info(
"[%s::%s] attempt %d/%d failed (%s); sleeping %.1fs",
- arm_name, qid, attempt, max_attempts, attempt_error, delay,
+ arm_name,
+ qid,
+ attempt,
+ max_attempts,
+ attempt_error,
+ delay,
)
await asyncio.sleep(delay)
assert final is not None
return RetryOutcome(
- arm=arm_name, qid=qid, attempts=attempts,
- final_result=final, recovered=False,
+ arm=arm_name,
+ qid=qid,
+ attempts=attempts,
+ final_result=final,
+ recovered=False,
)
@@ -367,7 +388,8 @@ async def _run(args: argparse.Namespace) -> int:
by_arm_count[f.arm] = by_arm_count.get(f.arm, 0) + 1
logger.info(
"Loaded %d failed rows across %d arms: %s",
- len(failed), len(by_arm_count),
+ len(failed),
+ len(by_arm_count),
", ".join(f"{a}={n}" for a, n in sorted(by_arm_count.items())),
)
@@ -385,7 +407,8 @@ async def _run(args: argparse.Namespace) -> int:
engine=PdfEngine(args.pdf_engine),
)
native_arm = NativePdfArm(
- provider=native_provider, max_output_tokens=args.max_output_tokens,
+ provider=native_provider,
+ max_output_tokens=args.max_output_tokens,
)
lc_arms: dict[str, BareLlmArm] = {}
@@ -415,7 +438,8 @@ async def _run(args: argparse.Namespace) -> int:
if qrow is None:
logger.error(
"Could not find question text for %s (idx %d) — skipping",
- f.doc_id, q_idx,
+ f.doc_id,
+ q_idx,
)
continue
question_text = str(qrow.get("question") or "").strip()
@@ -428,11 +452,14 @@ async def _run(args: argparse.Namespace) -> int:
if f.arm == "native_pdf":
pdf_path = Path(map_row["pdf_path"])
- if not pdf_path.exists():
+ if not await asyncio.to_thread(pdf_path.exists):
logger.error("PDF missing on disk: %s — skipping", pdf_path)
continue
request = _build_native_request(
- f.qid, question_text, answer_format, pdf_path,
+ f.qid,
+ question_text,
+ answer_format,
+ pdf_path,
max_output_tokens=args.max_output_tokens,
)
arm_obj = native_arm
@@ -442,11 +469,16 @@ async def _run(args: argparse.Namespace) -> int:
if not md_path_str or ext_blob.get("status") != "ok":
logger.error(
"Missing extraction for %s on %s — cannot retry; skipping",
- f.arm, f.doc_id,
+ f.arm,
+ f.doc_id,
)
continue
request = _build_lc_request(
- f.qid, question_text, answer_format, f.doc_id, Path(md_path_str),
+ f.qid,
+ question_text,
+ answer_format,
+ f.doc_id,
+ Path(md_path_str),
)
arm_obj = lc_arms[f.arm]
else:
@@ -454,13 +486,17 @@ async def _run(args: argparse.Namespace) -> int:
continue
plan.append((f, request, arm_obj))
- coros.append(_retry_one(
- arm_obj, request,
- arm_name=f.arm, qid=f.qid,
- max_attempts=args.max_attempts,
- base_delay=args.base_delay,
- max_delay=args.max_delay,
- ))
+ coros.append(
+ _retry_one(
+ arm_obj,
+ request,
+ arm_name=f.arm,
+ qid=f.qid,
+ max_attempts=args.max_attempts,
+ base_delay=args.base_delay,
+ max_delay=args.max_delay,
+ )
+ )
if not coros:
logger.warning("Nothing to retry after request building.")
@@ -469,13 +505,17 @@ async def _run(args: argparse.Namespace) -> int:
logger.info(
"Retrying %d failed rows with up to %d attempts each "
"(base_delay=%.1fs, max_delay=%.1fs, concurrency=%d).",
- len(coros), args.max_attempts, args.base_delay, args.max_delay,
+ len(coros),
+ args.max_attempts,
+ args.base_delay,
+ args.max_delay,
args.concurrency,
)
started = time.monotonic()
outcomes: list[RetryOutcome] = await _gather_with_limit(
- coros, concurrency=args.concurrency,
+ coros,
+ concurrency=args.concurrency,
)
elapsed = time.monotonic() - started
logger.info("Retry pass finished in %.1fs.", elapsed)
@@ -491,12 +531,8 @@ async def _run(args: argparse.Namespace) -> int:
for (f, _req, _arm_obj), outcome in zip(plan, outcomes, strict=True):
per_arm_total[outcome.arm] = per_arm_total.get(outcome.arm, 0) + 1
if outcome.recovered:
- per_arm_recovered[outcome.arm] = (
- per_arm_recovered.get(outcome.arm, 0) + 1
- )
- per_arm_attempts_dist.setdefault(outcome.arm, []).append(
- len(outcome.attempts)
- )
+ per_arm_recovered[outcome.arm] = per_arm_recovered.get(outcome.arm, 0) + 1
+ per_arm_attempts_dist.setdefault(outcome.arm, []).append(len(outcome.attempts))
g = grade(
pred=extract_freeform_answer(outcome.final_result.raw_text or ""),
@@ -557,12 +593,11 @@ async def _run(args: argparse.Namespace) -> int:
arm: {
"tried": per_arm_total.get(arm, 0),
"recovered": per_arm_recovered.get(arm, 0),
- "still_failed": (
- per_arm_total.get(arm, 0) - per_arm_recovered.get(arm, 0)
- ),
+ "still_failed": (per_arm_total.get(arm, 0) - per_arm_recovered.get(arm, 0)),
"recovery_rate": (
per_arm_recovered.get(arm, 0) / per_arm_total[arm]
- if per_arm_total.get(arm) else 0.0
+ if per_arm_total.get(arm)
+ else 0.0
),
"attempts_distribution": sorted(per_arm_attempts_dist.get(arm, [])),
}
@@ -595,8 +630,7 @@ async def _run(args: argparse.Namespace) -> int:
rec_total = sum(per_arm_recovered.values())
rate_total = (rec_total / total * 100) if total else 0.0
print("-" * len(header))
- print(f"{'TOTAL':<25} {total:>6} {rec_total:>10} {total - rec_total:>11} "
- f"{rate_total:>6.1f}%")
+ print(f"{'TOTAL':<25} {total:>6} {rec_total:>10} {total - rec_total:>11} {rate_total:>6.1f}%")
print()
print(f"Wrote {out_path.relative_to(REPO)}")
print(f"Wrote {summary_path.relative_to(REPO)}")
@@ -606,27 +640,37 @@ async def _run(args: argparse.Namespace) -> int:
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
- "--run-id", default="2026-05-14T00-53-19Z",
+ "--run-id",
+ default="2026-05-14T00-53-19Z",
help="Run timestamp under data/multimodal_doc/runs/. Default is the "
- "n=171 production run we wrote up in the blog.",
+ "n=171 production run we wrote up in the blog.",
)
parser.add_argument("--max-attempts", type=int, default=5)
- parser.add_argument("--base-delay", type=float, default=1.0,
- help="Base seconds for exponential backoff (default 1s).")
- parser.add_argument("--max-delay", type=float, default=30.0,
- help="Cap on per-retry sleep (default 30s).")
- parser.add_argument("--concurrency", type=int, default=2,
- help="Parallel retries in flight (default 2 — keep low "
- "to avoid the same transport stress that caused "
- "the original failures).")
+ parser.add_argument(
+ "--base-delay",
+ type=float,
+ default=1.0,
+ help="Base seconds for exponential backoff (default 1s).",
+ )
+ parser.add_argument(
+ "--max-delay", type=float, default=30.0, help="Cap on per-retry sleep (default 30s)."
+ )
+ parser.add_argument(
+ "--concurrency",
+ type=int,
+ default=2,
+ help="Parallel retries in flight (default 2 — keep low "
+ "to avoid the same transport stress that caused "
+ "the original failures).",
+ )
parser.add_argument("--llm-model", default="anthropic/claude-sonnet-4.5")
- parser.add_argument("--pdf-engine", default="native",
- choices=[e.value for e in PdfEngine])
+ parser.add_argument("--pdf-engine", default="native", choices=[e.value for e in PdfEngine])
parser.add_argument("--max-output-tokens", type=int, default=512)
parser.add_argument(
- "--include-surfsense", action="store_true",
+ "--include-surfsense",
+ action="store_true",
help="Also retry surfsense_agentic failures (requires backend + celery up). "
- "Default is to skip them since the n=171 run had 0 SurfSense failures.",
+ "Default is to skip them since the n=171 run had 0 SurfSense failures.",
)
args = parser.parse_args()
raise SystemExit(asyncio.run(_run(args)))
diff --git a/surfsense_evals/scripts/summarise_crag_run.py b/surfsense_evals/scripts/summarise_crag_run.py
index 646fb6a97..d15c4996f 100644
--- a/surfsense_evals/scripts/summarise_crag_run.py
+++ b/surfsense_evals/scripts/summarise_crag_run.py
@@ -11,7 +11,8 @@ def main() -> None:
if not runs:
print("(no CRAG runs found)")
return
- m = json.load(open(runs[-1], encoding="utf-8"))
+ with open(runs[-1], encoding="utf-8") as fh:
+ m = json.load(fh)
metrics = m["metrics"]
print(f"Reading: {runs[-1]}")
@@ -22,12 +23,12 @@ def main() -> None:
d = metrics[arm]
print(
f"{arm:14s}: "
- f"acc={d['accuracy']*100:5.1f}% (Wilson 95% CI "
- f"{d['ci_low']*100:.1f}-{d['ci_high']*100:.1f}) | "
- f"correct={d['correct_rate']*100:5.1f}% "
- f"missing={d['missing_rate']*100:5.1f}% "
- f"incorrect={d['incorrect_rate']*100:5.1f}% | "
- f"truth={d['truthfulness_score']*100:+5.1f}%"
+ f"acc={d['accuracy'] * 100:5.1f}% (Wilson 95% CI "
+ f"{d['ci_low'] * 100:.1f}-{d['ci_high'] * 100:.1f}) | "
+ f"correct={d['correct_rate'] * 100:5.1f}% "
+ f"missing={d['missing_rate'] * 100:5.1f}% "
+ f"incorrect={d['incorrect_rate'] * 100:5.1f}% | "
+ f"truth={d['truthfulness_score'] * 100:+5.1f}%"
)
print()
@@ -47,7 +48,7 @@ def main() -> None:
pieces = [f"{qt:20s} (n={n:3d}):"]
for arm in ("bare_llm", "long_context", "surfsense"):
if arm in row:
- pieces.append(f"{arm}={row[arm]['truthfulness_score']*100:+7.1f}%")
+ pieces.append(f"{arm}={row[arm]['truthfulness_score'] * 100:+7.1f}%")
print(" ".join(pieces))
print()
@@ -57,7 +58,7 @@ def main() -> None:
pieces = [f"{dom:10s} (n={n:3d}):"]
for arm in ("bare_llm", "long_context", "surfsense"):
if arm in row:
- pieces.append(f"{arm}={row[arm]['truthfulness_score']*100:+7.1f}%")
+ pieces.append(f"{arm}={row[arm]['truthfulness_score'] * 100:+7.1f}%")
print(" ".join(pieces))
diff --git a/surfsense_evals/scripts/summarise_parser_compare_run.py b/surfsense_evals/scripts/summarise_parser_compare_run.py
index c54d82784..7801a1318 100644
--- a/surfsense_evals/scripts/summarise_parser_compare_run.py
+++ b/surfsense_evals/scripts/summarise_parser_compare_run.py
@@ -16,7 +16,6 @@ import statistics
from collections import defaultdict
from pathlib import Path
-
REPO = Path(__file__).resolve().parents[1]
RUN_DIR = REPO / "data" / "multimodal_doc" / "runs" / "2026-05-14T00-53-19Z" / "parser_compare"
RAW = RUN_DIR / "raw.jsonl"
@@ -24,7 +23,9 @@ ARTIFACT = RUN_DIR / "run_artifact.json"
def main() -> None:
- rows = [json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines() if line.strip()]
+ rows = [
+ json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines() if line.strip()
+ ]
print(f"raw rows: {len(rows)}")
by_qid: dict[str, list[dict]] = defaultdict(list)
@@ -32,11 +33,19 @@ def main() -> None:
by_qid[row["qid"]].append(row)
print(f"unique questions: {len(by_qid)}")
- arm_metrics: dict[str, dict] = defaultdict(lambda: {
- "n": 0, "n_correct": 0, "n_failed": 0, "n_empty": 0,
- "costs": [], "in_tokens": [], "out_tokens": [], "latency_ms": [],
- "by_format": defaultdict(lambda: {"n": 0, "correct": 0}),
- })
+ arm_metrics: dict[str, dict] = defaultdict(
+ lambda: {
+ "n": 0,
+ "n_correct": 0,
+ "n_failed": 0,
+ "n_empty": 0,
+ "costs": [],
+ "in_tokens": [],
+ "out_tokens": [],
+ "latency_ms": [],
+ "by_format": defaultdict(lambda: {"n": 0, "correct": 0}),
+ }
+ )
for row in rows:
arm = row["arm"]
@@ -71,7 +80,9 @@ def main() -> None:
print()
print("=" * 100)
- print(f"{'arm':<25} {'n':>4} {'acc%':>6} {'F1%':>6} {'fail':>5} {'$ mean':>10} {'$ median':>10} {'in tok mean':>12} {'out tok mean':>12} {'p50 ms':>8}")
+ print(
+ f"{'arm':<25} {'n':>4} {'acc%':>6} {'F1%':>6} {'fail':>5} {'$ mean':>10} {'$ median':>10} {'in tok mean':>12} {'out tok mean':>12} {'p50 ms':>8}"
+ )
print("=" * 100)
art = json.loads(ARTIFACT.read_text(encoding="utf-8"))
per_arm_art = art["metrics"]["per_arm"]
@@ -91,7 +102,7 @@ def main() -> None:
print()
print("by answer_format (accuracy):")
- formats = sorted({f for m in arm_metrics.values() for f in m["by_format"].keys()})
+ formats = sorted({f for m in arm_metrics.values() for f in m["by_format"]})
header = f"{'arm':<25} " + " ".join(f"{f:>10}" for f in formats)
print(header)
print("-" * len(header))
@@ -111,7 +122,7 @@ def main() -> None:
print("Aggregated cost (from run_artifact.json):")
for arm, row in per_arm_art.items():
print(
- f" {arm:<25} acc={row['accuracy']*100:5.1f}% "
+ f" {arm:<25} acc={row['accuracy'] * 100:5.1f}% "
f" $/Q LLM={row['llm_cost_per_q']:.4f} "
f" preprocess total=${row['preprocess_cost_total']:.2f} "
f" $/Q total={row['total_cost_per_q']:.4f}"
diff --git a/surfsense_evals/scripts/test_context_overflow_hypothesis.py b/surfsense_evals/scripts/test_context_overflow_hypothesis.py
index 89bd6cb3d..8ccccba45 100644
--- a/surfsense_evals/scripts/test_context_overflow_hypothesis.py
+++ b/surfsense_evals/scripts/test_context_overflow_hypothesis.py
@@ -40,8 +40,7 @@ CONTEXT_HINTS = (
def main() -> None:
rows = [
- json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines()
- if line.strip()
+ json.loads(line) for line in RAW.read_text(encoding="utf-8").splitlines() if line.strip()
]
extraction_size: dict[tuple[str, str], int] = {}
@@ -73,12 +72,12 @@ def main() -> None:
print("=" * 80)
print("(b) Extraction size for OK vs FAILED rows per arm")
print("=" * 80)
- arm_buckets: dict[str, dict[str, list[int]]] = defaultdict(
- lambda: {"ok": [], "fail": []}
- )
+ arm_buckets: dict[str, dict[str, list[int]]] = defaultdict(lambda: {"ok": [], "fail": []})
parser_arms = (
- "azure_basic_lc", "azure_premium_lc",
- "llamacloud_basic_lc", "llamacloud_premium_lc",
+ "azure_basic_lc",
+ "azure_premium_lc",
+ "llamacloud_basic_lc",
+ "llamacloud_premium_lc",
)
for row in rows:
arm = row["arm"]
@@ -133,10 +132,13 @@ def main() -> None:
" 3M_2018_10K x llamacloud_premium = 908,733 chars (~227k tokens) "
"-- this is above Sonnet 4.5's 200k window."
)
- print(" If transport hypothesis is correct, this should still fail with a "
- "real overflow error.")
- print(" If transport hypothesis is correct AND the model truncates silently, "
- "it might 'succeed' but be wrong.")
+ print(
+ " If transport hypothesis is correct, this should still fail with a real overflow error."
+ )
+ print(
+ " If transport hypothesis is correct AND the model truncates silently, "
+ "it might 'succeed' but be wrong."
+ )
print()
for row in rows:
if row["doc_id"] != "3M_2018_10K.pdf":
@@ -145,10 +147,7 @@ def main() -> None:
continue
err = row.get("error") or "(none)"
graded = row.get("graded") or {}
- print(
- f" {row['qid']:<40} correct={graded.get('correct')!s:<5} "
- f"err={err[:100]}"
- )
+ print(f" {row['qid']:<40} correct={graded.get('correct')!s:<5} err={err[:100]}")
if __name__ == "__main__":
diff --git a/surfsense_evals/src/surfsense_evals/core/arms/surfsense.py b/surfsense_evals/src/surfsense_evals/core/arms/surfsense.py
index a84350dfd..f63b5cbe6 100644
--- a/surfsense_evals/src/surfsense_evals/core/arms/surfsense.py
+++ b/surfsense_evals/src/surfsense_evals/core/arms/surfsense.py
@@ -72,9 +72,7 @@ class SurfSenseArm(Arm):
try:
await self._client.delete_thread(thread_id)
except Exception as exc: # noqa: BLE001
- logger.debug(
- "Failed to delete thread %s: %s", thread_id, exc
- )
+ logger.debug("Failed to delete thread %s: %s", thread_id, exc)
letter = extract_answer_letter(answer.text)
return ArmResult(
diff --git a/surfsense_evals/src/surfsense_evals/core/auth.py b/surfsense_evals/src/surfsense_evals/core/auth.py
index a87e757c2..cf348e3ff 100644
--- a/surfsense_evals/src/surfsense_evals/core/auth.py
+++ b/surfsense_evals/src/surfsense_evals/core/auth.py
@@ -83,6 +83,7 @@ async def acquire_token(config: Config, *, http: httpx.AsyncClient | None = None
)
if config.has_local_mode():
+
async def _login(client: httpx.AsyncClient) -> TokenBundle:
response = await client.post(
f"{config.surfsense_api_base}/auth/desktop/login",
@@ -94,15 +95,12 @@ async def acquire_token(config: Config, *, http: httpx.AsyncClient | None = None
)
if response.status_code != 200:
raise CredentialError(
- f"LOCAL login failed (HTTP {response.status_code}): "
- f"{_safe_text(response)}"
+ f"LOCAL login failed (HTTP {response.status_code}): {_safe_text(response)}"
)
payload = response.json()
access = payload.get("access_token")
if not access:
- raise CredentialError(
- f"LOCAL login response missing access_token: {payload!r}"
- )
+ raise CredentialError(f"LOCAL login response missing access_token: {payload!r}")
return TokenBundle(
access_token=access,
refresh_token=payload.get("refresh_token") or None,
diff --git a/surfsense_evals/src/surfsense_evals/core/cli.py b/surfsense_evals/src/surfsense_evals/core/cli.py
index 17979fba0..21e706c49 100644
--- a/surfsense_evals/src/surfsense_evals/core/cli.py
+++ b/surfsense_evals/src/surfsense_evals/core/cli.py
@@ -32,14 +32,13 @@ from __future__ import annotations
import argparse
import asyncio
+import contextlib
import json
import logging
import sys
from dataclasses import dataclass
from typing import Any
-import sys
-
import httpx
from rich.console import Console
from rich.table import Table
@@ -51,10 +50,8 @@ from rich.table import Table
# Terminal, PowerShell, cmd) all interpret ANSI escapes natively.
if sys.platform == "win32":
for _stream in (sys.stdout, sys.stderr):
- try:
+ with contextlib.suppress(AttributeError, ValueError):
_stream.reconfigure(encoding="utf-8", errors="replace")
- except (AttributeError, ValueError):
- pass
from . import registry
from .auth import CredentialError, acquire_token, client_with_auth
@@ -207,8 +204,7 @@ async def _cmd_setup(args: argparse.Namespace) -> int:
if scenario not in SCENARIOS:
console.print(
- f"[red]Unknown scenario {scenario!r}. Pick one of: "
- f"{', '.join(SCENARIOS)}[/red]"
+ f"[red]Unknown scenario {scenario!r}. Pick one of: {', '.join(SCENARIOS)}[/red]"
)
return 2
@@ -295,9 +291,7 @@ async def _cmd_setup(args: argparse.Namespace) -> int:
if not skip_vision_setup and (vision_required or vision_llm_slug is not None):
try:
vision_candidates = await ss_client.list_global_vision_models()
- resolved = resolve_vision_llm(
- vision_candidates, explicit_slug=vision_llm_slug
- )
+ resolved = resolve_vision_llm(vision_candidates, explicit_slug=vision_llm_slug)
except VisionConfigError as exc:
console.print(f"[red]{exc}[/red]")
return 2
@@ -527,10 +521,7 @@ async def _cmd_run(args: argparse.Namespace) -> int:
)
artifact = await benchmark.run(ctx, **extra_kwargs)
- console.print(
- f"[green]run OK[/green] {args.suite}/{args.benchmark} → "
- f"{artifact.raw_path}"
- )
+ console.print(f"[green]run OK[/green] {args.suite}/{args.benchmark} → {artifact.raw_path}")
return 0
@@ -700,15 +691,21 @@ def _build_parser() -> argparse.ArgumentParser:
)
p_setup.set_defaults(_func=_cmd_setup, _async=True)
- p_teardown = sub.add_parser("teardown", help="Soft-delete the suite SearchSpace + clear state slot.")
+ p_teardown = sub.add_parser(
+ "teardown", help="Soft-delete the suite SearchSpace + clear state slot."
+ )
p_teardown.add_argument("--suite", required=True)
p_teardown.set_defaults(_func=_cmd_teardown, _async=True)
p_models = sub.add_parser("models", help="LLM-config discovery helpers.")
models_sub = p_models.add_subparsers(dest="subcommand", required=True)
p_models_list = models_sub.add_parser("list", help="List global LLM configs.")
- p_models_list.add_argument("--provider", default=None, help="Filter by provider, e.g. openrouter")
- p_models_list.add_argument("--grep", default=None, help="Substring filter on name / model_name.")
+ p_models_list.add_argument(
+ "--provider", default=None, help="Filter by provider, e.g. openrouter"
+ )
+ p_models_list.add_argument(
+ "--grep", default=None, help="Substring filter on name / model_name."
+ )
p_models_list.set_defaults(_func=_cmd_models_list, _async=True)
p_suites = sub.add_parser("suites", help="List registered suites.")
@@ -732,7 +729,9 @@ def _build_parser() -> argparse.ArgumentParser:
suite_parser = ingest_sub.add_parser(suite, help=f"Ingest a {suite} benchmark.")
suite_bench = suite_parser.add_subparsers(dest="benchmark", required=True)
for benchmark in registry.list_benchmarks(suite):
- bp = suite_bench.add_parser(benchmark.name, help=getattr(benchmark, "description", benchmark.name))
+ bp = suite_bench.add_parser(
+ benchmark.name, help=getattr(benchmark, "description", benchmark.name)
+ )
if hasattr(benchmark, "add_run_args"):
benchmark.add_run_args(bp)
bp.set_defaults(_func=_cmd_ingest, _async=True)
@@ -743,7 +742,9 @@ def _build_parser() -> argparse.ArgumentParser:
suite_parser = run_sub.add_parser(suite, help=f"Run a {suite} benchmark.")
suite_bench = suite_parser.add_subparsers(dest="benchmark", required=True)
for benchmark in registry.list_benchmarks(suite):
- bp = suite_bench.add_parser(benchmark.name, help=getattr(benchmark, "description", benchmark.name))
+ bp = suite_bench.add_parser(
+ benchmark.name, help=getattr(benchmark, "description", benchmark.name)
+ )
if hasattr(benchmark, "add_run_args"):
benchmark.add_run_args(bp)
bp.set_defaults(_func=_cmd_run, _async=True)
diff --git a/surfsense_evals/src/surfsense_evals/core/clients/documents.py b/surfsense_evals/src/surfsense_evals/core/clients/documents.py
index 362aae53b..2fd9b2766 100644
--- a/surfsense_evals/src/surfsense_evals/core/clients/documents.py
+++ b/surfsense_evals/src/surfsense_evals/core/clients/documents.py
@@ -18,6 +18,7 @@ Document processing is asynchronous:
from __future__ import annotations
import asyncio
+import contextlib
import logging
import mimetypes
from collections.abc import Iterable, Sequence
@@ -83,8 +84,7 @@ class DocumentProcessingFailed(RuntimeError):
def __init__(self, statuses: Sequence[DocumentStatus]) -> None:
details = ", ".join(
- f"id={s.document_id} ({s.title!r}): {s.reason or 'unknown'}"
- for s in statuses
+ f"id={s.document_id} ({s.title!r}): {s.reason or 'unknown'}" for s in statuses
)
super().__init__(f"Document(s) failed to process: {details}")
self.statuses = list(statuses)
@@ -157,10 +157,8 @@ class DocumentsClient:
)
finally:
for _, (_, file_obj, _) in opened:
- try:
+ with contextlib.suppress(Exception):
file_obj.close()
- except Exception: # noqa: BLE001
- pass
response.raise_for_status()
return FileUploadResult.from_payload(response.json())
@@ -241,9 +239,7 @@ class DocumentsClient:
# chunks (chunk_id -> document_id map)
# ------------------------------------------------------------------
- async def list_chunks(
- self, document_id: int, *, page_size: int = 100
- ) -> list[ChunkRow]:
+ async def list_chunks(self, document_id: int, *, page_size: int = 100) -> list[ChunkRow]:
"""Walk ``GET /documents/{id}/chunks`` until ``has_more=False``.
Used by ingestion to materialise the ``chunk_id -> document_id``
diff --git a/surfsense_evals/src/surfsense_evals/core/clients/new_chat.py b/surfsense_evals/src/surfsense_evals/core/clients/new_chat.py
index a4c23d010..397193bba 100644
--- a/surfsense_evals/src/surfsense_evals/core/clients/new_chat.py
+++ b/surfsense_evals/src/surfsense_evals/core/clients/new_chat.py
@@ -145,7 +145,7 @@ class NewChatClient:
if attempt > max_busy_retries:
raise
# Cap wait at 30s; backend retry hint is exponential anyway.
- wait = min(30.0, 0.5 * (2 ** attempt))
+ wait = min(30.0, 0.5 * (2**attempt))
logger.info(
"thread_id=%s busy (%s); retry %d/%d after %.1fs",
thread_id,
diff --git a/surfsense_evals/src/surfsense_evals/core/clients/search_space.py b/surfsense_evals/src/surfsense_evals/core/clients/search_space.py
index efd4a571d..19486aca1 100644
--- a/surfsense_evals/src/surfsense_evals/core/clients/search_space.py
+++ b/surfsense_evals/src/surfsense_evals/core/clients/search_space.py
@@ -177,16 +177,12 @@ class SearchSpaceClient:
response.raise_for_status()
payload = response.json()
if not isinstance(payload, list):
- raise RuntimeError(
- f"Unexpected /model-connections/global payload: {payload!r}"
- )
+ raise RuntimeError(f"Unexpected /model-connections/global payload: {payload!r}")
entries: list[VisionModelEntry] = []
for connection in payload:
provider = str(connection.get("provider", ""))
for model in connection.get("models") or []:
if not model.get("enabled", True) or not model.get("supports_image_input"):
continue
- entries.append(
- VisionModelEntry.from_payload({**model, "provider": provider})
- )
+ entries.append(VisionModelEntry.from_payload({**model, "provider": provider}))
return entries
diff --git a/surfsense_evals/src/surfsense_evals/core/config.py b/surfsense_evals/src/surfsense_evals/core/config.py
index 9a5a71e89..80002157e 100644
--- a/surfsense_evals/src/surfsense_evals/core/config.py
+++ b/surfsense_evals/src/surfsense_evals/core/config.py
@@ -104,7 +104,9 @@ def load_config() -> Config:
data_dir = Path(os.environ.get("EVAL_DATA_DIR") or (project_root / "data")).resolve()
reports_dir = Path(os.environ.get("EVAL_REPORTS_DIR") or (project_root / "reports")).resolve()
return Config(
- surfsense_api_base=os.environ.get("SURFSENSE_API_BASE", "http://localhost:8000").rstrip("/"),
+ surfsense_api_base=os.environ.get("SURFSENSE_API_BASE", "http://localhost:8000").rstrip(
+ "/"
+ ),
openrouter_api_key=os.environ.get("OPENROUTER_API_KEY") or None,
openrouter_base_url=os.environ.get(
"OPENROUTER_BASE_URL", "https://openrouter.ai/api/v1"
@@ -203,9 +205,7 @@ class SuiteState:
else None
),
native_arm_model=(
- str(payload["native_arm_model"])
- if payload.get("native_arm_model")
- else None
+ str(payload["native_arm_model"]) if payload.get("native_arm_model") else None
),
)
diff --git a/surfsense_evals/src/surfsense_evals/core/ingest_settings.py b/surfsense_evals/src/surfsense_evals/core/ingest_settings.py
index 8328e0d46..216ae36a4 100644
--- a/surfsense_evals/src/surfsense_evals/core/ingest_settings.py
+++ b/surfsense_evals/src/surfsense_evals/core/ingest_settings.py
@@ -95,10 +95,7 @@ class IngestSettings:
def render_label(self) -> str:
"""Human-readable single-line label for reports / log lines."""
- return (
- f"vision={'on' if self.use_vision_llm else 'off'}, "
- f"mode={self.processing_mode}"
- )
+ return f"vision={'on' if self.use_vision_llm else 'off'}, mode={self.processing_mode}"
def _coerce_bool(value: Any, default: bool) -> bool:
@@ -122,9 +119,7 @@ def _coerce_mode(value: Any, default: str) -> str:
return default
val = str(value).strip().lower()
if val not in PROCESSING_MODES:
- raise ValueError(
- f"Invalid processing_mode {val!r}; must be one of {PROCESSING_MODES}"
- )
+ raise ValueError(f"Invalid processing_mode {val!r}; must be one of {PROCESSING_MODES}")
return val
@@ -274,10 +269,7 @@ def format_ingest_settings_md(settings: Any) -> str:
return "- SurfSense ingest settings: (not recorded — re-ingest to capture)"
vision = "on" if settings.get("use_vision_llm") else "off"
mode = settings.get("processing_mode") or "basic"
- return (
- f"- SurfSense ingest settings: vision_llm=`{vision}`, "
- f"processing_mode=`{mode}`"
- )
+ return f"- SurfSense ingest settings: vision_llm=`{vision}`, processing_mode=`{mode}`"
__all__ = [
diff --git a/surfsense_evals/src/surfsense_evals/core/metrics/comparison.py b/surfsense_evals/src/surfsense_evals/core/metrics/comparison.py
index 579576f4f..332535871 100644
--- a/surfsense_evals/src/surfsense_evals/core/metrics/comparison.py
+++ b/surfsense_evals/src/surfsense_evals/core/metrics/comparison.py
@@ -67,17 +67,13 @@ def mcnemar_test(
"""
if len(arm_a_correct) != len(arm_b_correct):
- raise ValueError(
- f"Length mismatch: arm_a={len(arm_a_correct)}, arm_b={len(arm_b_correct)}"
- )
+ raise ValueError(f"Length mismatch: arm_a={len(arm_a_correct)}, arm_b={len(arm_b_correct)}")
n = len(arm_a_correct)
- b = sum(1 for a, c in zip(arm_a_correct, arm_b_correct) if a and not c)
- c = sum(1 for a, cc in zip(arm_a_correct, arm_b_correct) if (not a) and cc)
+ b = sum(1 for a, c in zip(arm_a_correct, arm_b_correct, strict=False) if a and not c)
+ c = sum(1 for a, cc in zip(arm_a_correct, arm_b_correct, strict=False) if (not a) and cc)
discordant = b + c
if discordant == 0:
- return McnemarResult(
- n_total=n, b=b, c=c, statistic=0.0, p_value=1.0, method="degenerate"
- )
+ return McnemarResult(n_total=n, b=b, c=c, statistic=0.0, p_value=1.0, method="degenerate")
if discordant < use_exact_below:
# Exact binomial: under H0 each discordant pair is a Bernoulli(0.5).
@@ -92,13 +88,11 @@ def mcnemar_test(
# Chi-square with continuity correction (McNemar-Edwards).
chi = ((abs(b - c) - 1) ** 2) / discordant
p_value = _chi2_sf(chi, df=1)
- return McnemarResult(
- n_total=n, b=b, c=c, statistic=chi, p_value=p_value, method="chi2_cc"
- )
+ return McnemarResult(n_total=n, b=b, c=c, statistic=chi, p_value=p_value, method="chi2_cc")
def _binom_pmf(n: int, k: int) -> float:
- return math.comb(n, k) * (0.5 ** n)
+ return math.comb(n, k) * (0.5**n)
def _chi2_sf(x: float, *, df: int) -> float:
diff --git a/surfsense_evals/src/surfsense_evals/core/metrics/mc_accuracy.py b/surfsense_evals/src/surfsense_evals/core/metrics/mc_accuracy.py
index 8b0188ca4..958f62600 100644
--- a/surfsense_evals/src/surfsense_evals/core/metrics/mc_accuracy.py
+++ b/surfsense_evals/src/surfsense_evals/core/metrics/mc_accuracy.py
@@ -46,9 +46,7 @@ _Z_FOR_LEVEL: dict[float, float] = {
}
-def wilson_ci(
- n_correct: int, n_total: int, *, level: float = 0.95
-) -> tuple[float, float]:
+def wilson_ci(n_correct: int, n_total: int, *, level: float = 0.95) -> tuple[float, float]:
"""Two-sided Wilson score confidence interval for a proportion.
Returns ``(low, high)``. ``n_total == 0`` returns ``(0.0, 1.0)`` —
@@ -70,9 +68,7 @@ def wilson_ci(
return low, high
-def accuracy_with_wilson_ci(
- n_correct: int, n_total: int, *, level: float = 0.95
-) -> AccuracyResult:
+def accuracy_with_wilson_ci(n_correct: int, n_total: int, *, level: float = 0.95) -> AccuracyResult:
if n_total < 0:
raise ValueError(f"n_total must be >= 0, got {n_total}")
if n_correct < 0 or n_correct > n_total:
@@ -109,10 +105,7 @@ def per_task_accuracy(
bucket[1] += 1
if row.get(correct_key):
bucket[0] += 1
- return {
- task: accuracy_with_wilson_ci(c[0], c[1], level=level)
- for task, c in counts.items()
- }
+ return {task: accuracy_with_wilson_ci(c[0], c[1], level=level) for task, c in counts.items()}
def macro_accuracy(per_task: Mapping[str, AccuracyResult]) -> float:
diff --git a/surfsense_evals/src/surfsense_evals/core/metrics/retrieval.py b/surfsense_evals/src/surfsense_evals/core/metrics/retrieval.py
index d4cfe10ae..3fd25f634 100644
--- a/surfsense_evals/src/surfsense_evals/core/metrics/retrieval.py
+++ b/surfsense_evals/src/surfsense_evals/core/metrics/retrieval.py
@@ -61,7 +61,7 @@ def _dcg_at_k(grades: Sequence[float], k: int) -> float:
s = 0.0
for i, grade in enumerate(grades[:k], start=1):
# Standard log-base-2 discount; gain = 2^grade - 1 for graded relevance.
- s += (2.0 ** grade - 1.0) / math.log2(i + 1)
+ s += (2.0**grade - 1.0) / math.log2(i + 1)
return s
@@ -106,7 +106,9 @@ def score_run(
qids = set(per_query_qrels.keys()) & set(per_query_retrieved.keys())
if not qids:
- return RetrievalScores(recall_at_k={k: 0.0 for k in ks}, mrr=0.0, ndcg_at_10=0.0, n_queries=0)
+ return RetrievalScores(
+ recall_at_k={k: 0.0 for k in ks}, mrr=0.0, ndcg_at_10=0.0, n_queries=0
+ )
recall_totals = {k: 0.0 for k in ks}
mrr_total = 0.0
diff --git a/surfsense_evals/src/surfsense_evals/core/parse/__init__.py b/surfsense_evals/src/surfsense_evals/core/parse/__init__.py
index 208c2d374..8b90de78b 100644
--- a/surfsense_evals/src/surfsense_evals/core/parse/__init__.py
+++ b/surfsense_evals/src/surfsense_evals/core/parse/__init__.py
@@ -3,7 +3,7 @@
from __future__ import annotations
from .answer_letter import AnswerLetterResult, extract_answer_letter
-from .citations import CITATION_REGEX, CitationToken, ChunkCitation, UrlCitation, parse_citations
+from .citations import CITATION_REGEX, ChunkCitation, CitationToken, UrlCitation, parse_citations
from .freeform_answer import extract_freeform_answer
from .sse import SseEvent, iter_sse_events
diff --git a/surfsense_evals/src/surfsense_evals/core/parse/citations.py b/surfsense_evals/src/surfsense_evals/core/parse/citations.py
index 1fcd35434..38bd16d31 100644
--- a/surfsense_evals/src/surfsense_evals/core/parse/citations.py
+++ b/surfsense_evals/src/surfsense_evals/core/parse/citations.py
@@ -15,7 +15,7 @@ from __future__ import annotations
import re
from dataclasses import dataclass
-from typing import Any, Union
+from typing import Any
# Pattern preserves the TS source verbatim:
# /[\[【]\u200B?citation:\s*(https?:\/\/[^\]】\u200B]+|urlcite\d+|(?:doc-)?-?\d+(?:\s*,\s*(?:doc-)?-?\d+)*)\s*\u200B?[\]】]/g
@@ -35,7 +35,7 @@ from typing import Any, Union
# the pattern source, so we splice the literal character in via an
# f-string. This keeps our pattern functionally identical to the TS
# reference and lets ``"\u200B" in CITATION_REGEX.pattern`` succeed.
-_ZWSP = "\u200B"
+_ZWSP = "\u200b"
CITATION_REGEX = re.compile(
rf"[\[【]{_ZWSP}?citation:\s*("
rf"https?://[^\]】{_ZWSP}]+|urlcite\d+|(?:doc-)?-?\d+(?:\s*,\s*(?:doc-)?-?\d+)*"
@@ -64,7 +64,7 @@ class UrlCitation:
return {"kind": "url", "url": self.url}
-CitationToken = Union[ChunkCitation, UrlCitation]
+CitationToken = ChunkCitation | UrlCitation
def parse_citations(text: str, *, url_map: dict[str, str] | None = None) -> list[CitationToken]:
diff --git a/surfsense_evals/src/surfsense_evals/core/parse/freeform_answer.py b/surfsense_evals/src/surfsense_evals/core/parse/freeform_answer.py
index 959b045a5..104176d09 100644
--- a/surfsense_evals/src/surfsense_evals/core/parse/freeform_answer.py
+++ b/surfsense_evals/src/surfsense_evals/core/parse/freeform_answer.py
@@ -56,7 +56,7 @@ def extract_freeform_answer(text: str) -> str:
marker_matches = list(_ANSWER_MARKER.finditer(text))
if marker_matches:
last = marker_matches[-1]
- tail = text[last.end():]
+ tail = text[last.end() :]
nl = tail.find("\n")
if nl >= 0:
tail = tail[:nl]
@@ -77,7 +77,7 @@ def extract_freeform_answer(text: str) -> str:
# 2. Strip wrapping quotes / parens / trailing punctuation that
# confuse the grader without changing meaning.
candidate = candidate.strip().strip("`").strip()
- if candidate.startswith(("\"", "'")) and candidate.endswith(("\"", "'")):
+ if candidate.startswith(('"', "'")) and candidate.endswith(('"', "'")):
candidate = candidate[1:-1].strip()
return candidate
diff --git a/surfsense_evals/src/surfsense_evals/core/parsers/azure_di.py b/surfsense_evals/src/surfsense_evals/core/parsers/azure_di.py
index eebad906a..16e618db4 100644
--- a/surfsense_evals/src/surfsense_evals/core/parsers/azure_di.py
+++ b/surfsense_evals/src/surfsense_evals/core/parsers/azure_di.py
@@ -25,6 +25,7 @@ import asyncio
import logging
import os
import random
+from pathlib import Path
logger = logging.getLogger(__name__)
@@ -63,8 +64,7 @@ async def parse_with_azure_di(
api_key = api_key or os.environ.get("AZURE_DI_KEY")
if not endpoint or not api_key:
raise ValueError(
- "AZURE_DI_ENDPOINT and AZURE_DI_KEY must be set "
- "(see surfsense_evals/.env)."
+ "AZURE_DI_ENDPOINT and AZURE_DI_KEY must be set (see surfsense_evals/.env)."
)
model_id = _AZURE_MODEL_BY_MODE.get(processing_mode, "prebuilt-read")
@@ -82,10 +82,13 @@ async def parse_with_azure_di(
ServiceResponseError,
)
- file_size_mb = os.path.getsize(file_path) / (1024 * 1024)
+ file_size_mb = await asyncio.to_thread(os.path.getsize, file_path) / (1024 * 1024)
logger.info(
"Azure DI parsing %s (mode=%s, model=%s, size=%.1fMB)",
- file_path, processing_mode, model_id, file_size_mb,
+ file_path,
+ processing_mode,
+ model_id,
+ file_size_mb,
)
last_exc: Exception | None = None
@@ -96,21 +99,21 @@ async def parse_with_azure_di(
credential=AzureKeyCredential(api_key),
)
async with client:
- with open(file_path, "rb") as fh:
- poller = await client.begin_analyze_document(
- model_id,
- body=fh,
- output_content_format=DocumentContentFormat.MARKDOWN,
- )
+ body = await asyncio.to_thread(Path(file_path).read_bytes)
+ poller = await client.begin_analyze_document(
+ model_id,
+ body=body,
+ output_content_format=DocumentContentFormat.MARKDOWN,
+ )
result = await poller.result()
content = (result.content or "").strip()
if not content:
- raise AzureDIError(
- f"Azure DI returned empty content for {file_path}"
- )
+ raise AzureDIError(f"Azure DI returned empty content for {file_path}")
logger.info(
"Azure DI OK: %s (%s) -> %d chars",
- file_path, model_id, len(content),
+ file_path,
+ model_id,
+ len(content),
)
return content
@@ -119,9 +122,7 @@ async def parse_with_azure_di(
except HttpResponseError as exc:
# 4xx that's not auth: don't retry, the request itself is broken.
if exc.status_code and 400 <= exc.status_code < 500:
- raise AzureDIError(
- f"Azure DI {exc.status_code} on {file_path}: {exc}"
- ) from exc
+ raise AzureDIError(f"Azure DI {exc.status_code} on {file_path}: {exc}") from exc
last_exc = exc
except (ServiceRequestError, ServiceResponseError) as exc:
last_exc = exc
@@ -132,7 +133,10 @@ async def parse_with_azure_di(
sleep_for = delay + jitter
logger.warning(
"Azure DI attempt %d/%d failed (%s); retrying in %.1fs",
- attempt, _MAX_RETRIES, type(last_exc).__name__, sleep_for,
+ attempt,
+ _MAX_RETRIES,
+ type(last_exc).__name__,
+ sleep_for,
)
await asyncio.sleep(sleep_for)
diff --git a/surfsense_evals/src/surfsense_evals/core/parsers/llamacloud.py b/surfsense_evals/src/surfsense_evals/core/parsers/llamacloud.py
index ba3d787ef..32fd97e47 100644
--- a/surfsense_evals/src/surfsense_evals/core/parsers/llamacloud.py
+++ b/surfsense_evals/src/surfsense_evals/core/parsers/llamacloud.py
@@ -61,8 +61,7 @@ def _extract_markdown(result) -> str:
if result and hasattr(result[0], "text"):
return result[0].text
return "\n\n".join(
- doc.page_content if hasattr(doc, "page_content") else str(doc)
- for doc in result
+ doc.page_content if hasattr(doc, "page_content") else str(doc) for doc in result
)
return str(result)
@@ -86,9 +85,7 @@ async def parse_with_llamacloud(
api_key = api_key or os.environ.get("LLAMA_CLOUD_API_KEY")
if not api_key:
- raise ValueError(
- "LLAMA_CLOUD_API_KEY must be set (see surfsense_evals/.env)."
- )
+ raise ValueError("LLAMA_CLOUD_API_KEY must be set (see surfsense_evals/.env).")
parse_mode = _LLAMA_PARSE_MODE_MAP.get(processing_mode, "parse_page_with_llm")
@@ -98,7 +95,7 @@ async def parse_with_llamacloud(
from llama_cloud_services.parse.base import JobFailedException
from llama_cloud_services.parse.utils import ResultType
- file_size_mb = os.path.getsize(file_path) / (1024 * 1024)
+ file_size_mb = await asyncio.to_thread(os.path.getsize, file_path) / (1024 * 1024)
# Match backend's per-page timeout heuristic so big PDFs don't drop
# mid-job: 60s baseline + 30s/page (premium agent runs longer than
# basic; both fit comfortably here).
@@ -106,13 +103,19 @@ async def parse_with_llamacloud(
upload_timeout = max(120.0, 30.0 * file_size_mb)
logger.info(
- "LlamaCloud parsing %s (mode=%s, parse_mode=%s, %.1fMB, "
- "job_timeout=%.0fs)",
- file_path, processing_mode, parse_mode, file_size_mb, job_timeout,
+ "LlamaCloud parsing %s (mode=%s, parse_mode=%s, %.1fMB, job_timeout=%.0fs)",
+ file_path,
+ processing_mode,
+ parse_mode,
+ file_size_mb,
+ job_timeout,
)
custom_timeout = httpx.Timeout(
- connect=120.0, read=upload_timeout, write=upload_timeout, pool=120.0,
+ connect=120.0,
+ read=upload_timeout,
+ write=upload_timeout,
+ pool=120.0,
)
last_exc: Exception | None = None
@@ -135,12 +138,12 @@ async def parse_with_llamacloud(
result = await parser.aparse(str(file_path))
content = _extract_markdown(result).strip()
if not content:
- raise LlamaCloudError(
- f"LlamaCloud returned empty content for {file_path}"
- )
+ raise LlamaCloudError(f"LlamaCloud returned empty content for {file_path}")
logger.info(
"LlamaCloud OK: %s (%s) -> %d chars",
- file_path, parse_mode, len(content),
+ file_path,
+ parse_mode,
+ len(content),
)
return content
@@ -156,7 +159,10 @@ async def parse_with_llamacloud(
sleep_for = delay + jitter
logger.warning(
"LlamaCloud attempt %d/%d failed (%s); retrying in %.1fs",
- attempt, _MAX_RETRIES, type(last_exc).__name__, sleep_for,
+ attempt,
+ _MAX_RETRIES,
+ type(last_exc).__name__,
+ sleep_for,
)
await asyncio.sleep(sleep_for)
diff --git a/surfsense_evals/src/surfsense_evals/core/pdf/render.py b/surfsense_evals/src/surfsense_evals/core/pdf/render.py
index 624136d7c..21866f3e5 100644
--- a/surfsense_evals/src/surfsense_evals/core/pdf/render.py
+++ b/surfsense_evals/src/surfsense_evals/core/pdf/render.py
@@ -116,11 +116,7 @@ def _normalise_paragraphs(text: str) -> list[str]:
def _escape_html(text: str) -> str:
- return (
- text.replace("&", "&")
- .replace("<", "<")
- .replace(">", ">")
- )
+ return text.replace("&", "&").replace("<", "<").replace(">", ">")
def render_pdf(
diff --git a/surfsense_evals/src/surfsense_evals/core/providers/openrouter_chat.py b/surfsense_evals/src/surfsense_evals/core/providers/openrouter_chat.py
index 2494434be..208fbb865 100644
--- a/surfsense_evals/src/surfsense_evals/core/providers/openrouter_chat.py
+++ b/surfsense_evals/src/surfsense_evals/core/providers/openrouter_chat.py
@@ -29,8 +29,8 @@ from typing import Any
import httpx
from .openrouter_pdf import (
- OpenRouterResponse,
_DEFAULT_HEADERS,
+ OpenRouterResponse,
_parse_chat_completion,
)
diff --git a/surfsense_evals/src/surfsense_evals/core/providers/openrouter_pdf.py b/surfsense_evals/src/surfsense_evals/core/providers/openrouter_pdf.py
index e98590cbf..5cd47b04e 100644
--- a/surfsense_evals/src/surfsense_evals/core/providers/openrouter_pdf.py
+++ b/surfsense_evals/src/surfsense_evals/core/providers/openrouter_pdf.py
@@ -34,7 +34,7 @@ import base64
import logging
import time
from dataclasses import dataclass
-from enum import Enum
+from enum import StrEnum
from pathlib import Path
from typing import Any
@@ -43,7 +43,7 @@ import httpx
logger = logging.getLogger(__name__)
-class PdfEngine(str, Enum):
+class PdfEngine(StrEnum):
NATIVE = "native"
MISTRAL_OCR = "mistral-ocr"
CLOUDFLARE_AI = "cloudflare-ai"
@@ -121,9 +121,7 @@ class OpenRouterPdfProvider:
body: dict[str, Any] = {
"model": self._model,
"messages": messages,
- "plugins": [
- {"id": "file-parser", "pdf": {"engine": self._engine.value}}
- ],
+ "plugins": [{"id": "file-parser", "pdf": {"engine": self._engine.value}}],
}
if max_tokens:
body["max_tokens"] = max_tokens
diff --git a/surfsense_evals/src/surfsense_evals/core/registry.py b/surfsense_evals/src/surfsense_evals/core/registry.py
index 65f64c39a..7fb64c36f 100644
--- a/surfsense_evals/src/surfsense_evals/core/registry.py
+++ b/surfsense_evals/src/surfsense_evals/core/registry.py
@@ -177,7 +177,9 @@ class Benchmark(Protocol):
def add_run_args(self, parser: argparse.ArgumentParser) -> None: # pragma: no cover - protocol
"""Add benchmark-specific flags to ``run ``."""
- def report_section(self, artifacts: list[RunArtifact]) -> ReportSection: # pragma: no cover - protocol
+ def report_section(
+ self, artifacts: list[RunArtifact]
+ ) -> ReportSection: # pragma: no cover - protocol
...
@@ -224,9 +226,7 @@ def get(suite: str, name: str) -> Benchmark:
return _REGISTRY[(suite, name)]
except KeyError as exc:
available = ", ".join(f"{s}/{n}" for s, n in sorted(_REGISTRY)) or ""
- raise KeyError(
- f"Unknown benchmark '{suite}/{name}'. Registered: {available}"
- ) from exc
+ raise KeyError(f"Unknown benchmark '{suite}/{name}'. Registered: {available}") from exc
def list_suites() -> list[str]:
diff --git a/surfsense_evals/src/surfsense_evals/core/scenarios.py b/surfsense_evals/src/surfsense_evals/core/scenarios.py
index 16874a069..fefdc6865 100644
--- a/surfsense_evals/src/surfsense_evals/core/scenarios.py
+++ b/surfsense_evals/src/surfsense_evals/core/scenarios.py
@@ -45,10 +45,7 @@ def format_scenario_md(extra: Mapping[str, Any] | None) -> str:
"(text-only model can't see images) — that's the point."
)
else:
- body = (
- f"- Scenario: head-to-head — both arms answer with `{surf_slug}` "
- "via OpenRouter."
- )
+ body = f"- Scenario: head-to-head — both arms answer with `{surf_slug}` via OpenRouter."
if vision_slug:
body += f" SurfSense ingest VLM: `{vision_slug}`."
diff --git a/surfsense_evals/src/surfsense_evals/suites/__init__.py b/surfsense_evals/src/surfsense_evals/suites/__init__.py
index 95ed958ca..f3d26f865 100644
--- a/surfsense_evals/src/surfsense_evals/suites/__init__.py
+++ b/surfsense_evals/src/surfsense_evals/suites/__init__.py
@@ -20,7 +20,7 @@ from __future__ import annotations
import importlib
import logging
import pkgutil
-from typing import Iterable
+from collections.abc import Iterable
logger = logging.getLogger(__name__)
@@ -60,7 +60,5 @@ def discover_suites() -> list[str]:
importlib.import_module(benchmark_name)
imported.append(benchmark_name)
except Exception as exc: # noqa: BLE001
- logger.warning(
- "Failed to import benchmark %s: %s", benchmark_name, exc
- )
+ logger.warning("Failed to import benchmark %s: %s", benchmark_name, exc)
return imported
diff --git a/surfsense_evals/src/surfsense_evals/suites/_demo/hello/__init__.py b/surfsense_evals/src/surfsense_evals/suites/_demo/hello/__init__.py
index 1da33926c..43dc51ac5 100644
--- a/surfsense_evals/src/surfsense_evals/suites/_demo/hello/__init__.py
+++ b/surfsense_evals/src/surfsense_evals/suites/_demo/hello/__init__.py
@@ -6,7 +6,6 @@ import argparse
from typing import Any
from ....core.registry import (
- Benchmark,
ReportSection,
RunArtifact,
RunContext,
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/cure/__init__.py b/surfsense_evals/src/surfsense_evals/suites/medical/cure/__init__.py
index e13224be7..7e9d9a07b 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/cure/__init__.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/cure/__init__.py
@@ -12,7 +12,7 @@ Recall@k / MRR / nDCG@10 against qrels.
from __future__ import annotations
-from .runner import CureBenchmark
from ....core import registry as _registry
+from .runner import CureBenchmark
_registry.register(CureBenchmark())
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/cure/ingest.py b/surfsense_evals/src/surfsense_evals/suites/medical/cure/ingest.py
index 275e28ce5..84108b4df 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/cure/ingest.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/cure/ingest.py
@@ -154,13 +154,11 @@ async def run_ingest(
if not batches:
logger.warning("Discipline %s produced 0 batches; skipping upload", discipline)
continue
- logger.info(
- "Uploading %d batches for discipline %s", len(batches), discipline
- )
+ logger.info("Uploading %d batches for discipline %s", len(batches), discipline)
upload_result = await docs_client.upload(
files=[b.path for b in batches],
search_space_id=ctx.search_space_id,
- use_vision_llm=settings.use_vision_llm,
+ use_vision_llm=settings.use_vision_llm,
processing_mode=settings.processing_mode,
)
new_doc_ids = list(upload_result.document_ids)
@@ -177,9 +175,7 @@ async def run_ingest(
)
title_to_doc = {s.title: s.document_id for s in statuses}
- per_discipline_path = (
- ctx.maps_dir() / f"cure_corpus_map_{discipline}.jsonl"
- )
+ per_discipline_path = ctx.maps_dir() / f"cure_corpus_map_{discipline}.jsonl"
with per_discipline_path.open("w", encoding="utf-8") as fh:
fh.write(settings_header_line(settings) + "\n")
for batch in batches:
@@ -202,9 +198,7 @@ async def run_ingest(
try:
chunks = await docs_client.list_chunks(int(doc_id))
except Exception as exc: # noqa: BLE001
- logger.warning(
- "Failed to list chunks for doc_id=%s: %s", doc_id, exc
- )
+ logger.warning("Failed to list chunks for doc_id=%s: %s", doc_id, exc)
continue
for chunk in chunks:
fh.write(
@@ -227,12 +221,10 @@ async def run_ingest(
def _take(it: Iterable, n: int) -> Iterable:
- yielded = 0
- for x in it:
- if yielded >= n:
+ for i, x in enumerate(it):
+ if i >= n:
return
yield x
- yielded += 1
__all__ = ["DISCIPLINES", "CorpusPassage", "PassageBatch", "run_ingest"]
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/cure/runner.py b/surfsense_evals/src/surfsense_evals/suites/medical/cure/runner.py
index 041e0e8b5..d2735c8d5 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/cure/runner.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/cure/runner.py
@@ -34,7 +34,6 @@ from ....core.ingest_settings import (
)
from ....core.metrics.retrieval import score_run
from ....core.registry import (
- Benchmark,
ReportSection,
RunArtifact,
RunContext,
@@ -192,12 +191,15 @@ class CureBenchmark:
def add_run_args(self, parser: argparse.ArgumentParser) -> None:
parser.add_argument("--lang", default="en", choices=("en", "es", "fr"))
- parser.add_argument("--discipline", default=None,
- help="Restrict to one discipline (default: all ingested).")
+ parser.add_argument(
+ "--discipline", default=None, help="Restrict to one discipline (default: all ingested)."
+ )
parser.add_argument("--n", dest="sample_n", type=int, default=None)
parser.add_argument("--concurrency", type=int, default=4)
parser.add_argument(
- "--max-passages-per-discipline", type=int, default=None,
+ "--max-passages-per-discipline",
+ type=int,
+ default=None,
help="(ingest only) cap corpus rows per discipline for smoke testing.",
)
# Per-upload knobs forwarded to /documents/fileupload at ingest;
@@ -234,11 +236,13 @@ class CureBenchmark:
# Disciplines to query are determined by the per-discipline maps
# actually present (either user-filtered or whatever was ingested).
- ingested_disciplines = sorted({
- row_disc
- for path in maps_dir.glob("cure_corpus_map_*.jsonl")
- for row_disc in [path.stem[len("cure_corpus_map_"):]]
- })
+ ingested_disciplines = sorted(
+ {
+ row_disc
+ for path in maps_dir.glob("cure_corpus_map_*.jsonl")
+ for row_disc in [path.stem[len("cure_corpus_map_") :]]
+ }
+ )
if discipline_filter:
disciplines = [discipline_filter]
else:
@@ -276,7 +280,7 @@ class CureBenchmark:
)
per_query_retrieved: dict[str, list[str]] = {}
- for q, res in zip(queries, results):
+ for q, res in zip(queries, results, strict=False):
chunk_ids: list[int] = []
seen: set[int] = set()
for citation in res.citations:
@@ -311,7 +315,7 @@ class CureBenchmark:
run_dir = ctx.runs_dir(run_timestamp=run_timestamp)
raw_path = run_dir / "raw.jsonl"
with raw_path.open("w", encoding="utf-8") as fh:
- for q, res in zip(queries, results):
+ for q, res in zip(queries, results, strict=False):
fh.write(
json.dumps(
{
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/ingest.py b/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/ingest.py
index ff43c7049..f50247acb 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/ingest.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/ingest.py
@@ -55,15 +55,15 @@ def _hf_hub_download(*args, **kwargs):
@dataclass
class MedXpertQuestion:
- qid: str # e.g. "MM-26"
- question: str # full question text (case + ask)
- options: dict[str, str] # A-E
- label: str # "A".."E"
- image_files: list[str] # filenames inside images.zip
+ qid: str # e.g. "MM-26"
+ question: str # full question text (case + ask)
+ options: dict[str, str] # A-E
+ label: str # "A".."E"
+ image_files: list[str] # filenames inside images.zip
medical_task: str
body_system: str
question_type: str
- split: str # "test" or "dev"
+ split: str # "test" or "dev"
def _load_jsonl(path: Path, *, split: str) -> list[MedXpertQuestion]:
@@ -84,17 +84,19 @@ def _load_jsonl(path: Path, *, split: str) -> list[MedXpertQuestion]:
images = row.get("images") or []
if not isinstance(images, list):
images = []
- out.append(MedXpertQuestion(
- qid=qid,
- question=question,
- options=opts,
- label=label,
- image_files=[str(x).strip() for x in images if str(x).strip()],
- medical_task=str(row.get("medical_task") or "").strip(),
- body_system=str(row.get("body_system") or "").strip(),
- question_type=str(row.get("question_type") or "").strip(),
- split=split,
- ))
+ out.append(
+ MedXpertQuestion(
+ qid=qid,
+ question=question,
+ options=opts,
+ label=label,
+ image_files=[str(x).strip() for x in images if str(x).strip()],
+ medical_task=str(row.get("medical_task") or "").strip(),
+ body_system=str(row.get("body_system") or "").strip(),
+ question_type=str(row.get("question_type") or "").strip(),
+ split=split,
+ )
+ )
return out
@@ -204,7 +206,7 @@ async def _upload_pdfs(
name_to_id: dict[str, int] = {}
pdf_list = list(pdf_paths)
for batch_start in range(0, len(pdf_list), batch_size):
- batch = pdf_list[batch_start:batch_start + batch_size]
+ batch = pdf_list[batch_start : batch_start + batch_size]
result = await docs_client.upload(
files=batch,
search_space_id=ctx.search_space_id,
@@ -226,8 +228,10 @@ async def _upload_pdfs(
name_to_id[s.title] = s.document_id
logger.info(
"Uploaded MedXpertQA batch %d-%d: %d new, %d duplicate",
- batch_start, batch_start + len(batch),
- len(result.document_ids), len(result.duplicate_document_ids),
+ batch_start,
+ batch_start + len(batch),
+ len(result.document_ids),
+ len(result.duplicate_document_ids),
)
return name_to_id
@@ -310,9 +314,11 @@ async def run_ingest(
# Materialise into bench_dir so the path is stable.
try:
from os import link as _link
+
_link(local_zip, images_zip_local)
except OSError:
from shutil import copy2
+
copy2(local_zip, images_zip_local)
_ensure_images_extracted(images_zip_local, images_dir)
@@ -354,17 +360,22 @@ async def run_ingest(
questions_jsonl = bench_dir / "questions.jsonl"
with questions_jsonl.open("w", encoding="utf-8") as fh:
for q in questions:
- fh.write(json.dumps({
- "qid": q.qid,
- "question": q.question,
- "options": q.options,
- "label": q.label,
- "image_files": q.image_files,
- "medical_task": q.medical_task,
- "body_system": q.body_system,
- "question_type": q.question_type,
- "split": q.split,
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ "qid": q.qid,
+ "question": q.question,
+ "options": q.options,
+ "label": q.label,
+ "image_files": q.image_files,
+ "medical_task": q.medical_task,
+ "body_system": q.body_system,
+ "question_type": q.question_type,
+ "split": q.split,
+ }
+ )
+ + "\n"
+ )
logger.info("Wrote %d MedXpertQA questions to %s", len(questions), questions_jsonl)
map_path = ctx.maps_dir() / "medxpertqa_doc_map.jsonl"
@@ -376,13 +387,18 @@ async def run_ingest(
local = pdf_paths.get(q.qid)
if local is None:
continue
- fh.write(json.dumps({
- "qid": q.qid,
- "document_id": name_to_id.get(local.name),
- "pdf_path": str(local),
- "n_images": len(q.image_files),
- "split": q.split,
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ "qid": q.qid,
+ "document_id": name_to_id.get(local.name),
+ "pdf_path": str(local),
+ "n_images": len(q.image_files),
+ "split": q.split,
+ }
+ )
+ + "\n"
+ )
logger.info("Wrote MedXpertQA doc map to %s", map_path)
new_state = ctx.suite_state
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/runner.py b/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/runner.py
index ac0651996..f7a3331a9 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/runner.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/medxpertqa/runner.py
@@ -129,19 +129,21 @@ def _load_questions(
n_images = int(map_row.get("n_images", 0))
if require_images and n_images <= 0:
continue
- out.append(MXQuestion(
- qid=qid,
- question=str(row.get("question") or ""),
- options={str(k).upper(): str(v) for k, v in (row.get("options") or {}).items()},
- label=str(row.get("label") or "").strip().upper(),
- medical_task=str(row.get("medical_task") or "").strip(),
- body_system=str(row.get("body_system") or "").strip(),
- question_type=str(row.get("question_type") or "").strip(),
- split=str(row.get("split") or ""),
- n_images=n_images,
- pdf_path=Path(map_row["pdf_path"]),
- document_id=map_row.get("document_id"),
- ))
+ out.append(
+ MXQuestion(
+ qid=qid,
+ question=str(row.get("question") or ""),
+ options={str(k).upper(): str(v) for k, v in (row.get("options") or {}).items()},
+ label=str(row.get("label") or "").strip().upper(),
+ medical_task=str(row.get("medical_task") or "").strip(),
+ body_system=str(row.get("body_system") or "").strip(),
+ question_type=str(row.get("question_type") or "").strip(),
+ split=str(row.get("split") or ""),
+ n_images=n_images,
+ pdf_path=Path(map_row["pdf_path"]),
+ document_id=map_row.get("document_id"),
+ )
+ )
out.sort(key=lambda q: (q.split, q.qid))
if sample_n is not None and sample_n > 0:
out = out[:sample_n]
@@ -182,51 +184,81 @@ class MedXpertQAMMBenchmark:
def add_run_args(self, parser: argparse.ArgumentParser) -> None:
parser.add_argument(
- "--split", default="test", choices=["test", "dev", "all"],
+ "--split",
+ default="test",
+ choices=["test", "dev", "all"],
help="Which MedXpertQA-MM split to run (default: test).",
)
parser.add_argument(
- "--task", default="all",
+ "--task",
+ default="all",
help="Filter by medical_task value (e.g. Diagnosis, Treatment, Basic Medicine).",
)
parser.add_argument(
- "--body-system", dest="body_filter", default="all",
+ "--body-system",
+ dest="body_filter",
+ default="all",
help="Filter by body_system value (e.g. Cardiovascular, Lymphatic).",
)
parser.add_argument(
- "--require-images", dest="require_images", action="store_true",
+ "--require-images",
+ dest="require_images",
+ action="store_true",
help="Skip rare MM rows that ended up with zero resolvable images.",
)
- parser.add_argument("--n", dest="sample_n", type=int, default=None,
- help="Run only the first N questions after filters apply.")
- parser.add_argument("--concurrency", type=int, default=4,
- help="Parallel question workers per arm.")
- parser.add_argument("--no-mentions", dest="no_mentions", action="store_true",
- help="SurfSense arm: skip mentioned_document_ids (unscoped retrieval).")
parser.add_argument(
- "--pdf-engine", default="native",
+ "--n",
+ dest="sample_n",
+ type=int,
+ default=None,
+ help="Run only the first N questions after filters apply.",
+ )
+ parser.add_argument(
+ "--concurrency", type=int, default=4, help="Parallel question workers per arm."
+ )
+ parser.add_argument(
+ "--no-mentions",
+ dest="no_mentions",
+ action="store_true",
+ help="SurfSense arm: skip mentioned_document_ids (unscoped retrieval).",
+ )
+ parser.add_argument(
+ "--pdf-engine",
+ default="native",
choices=[e.value for e in PdfEngine],
help="OpenRouter file-parser engine for the native arm.",
)
parser.add_argument(
- "--max-output-tokens", type=int, default=512,
+ "--max-output-tokens",
+ type=int,
+ default=512,
help="Cap on completion length for both arms.",
)
# Ingest-only knobs (forwarded by the CLI to ingest.run_ingest).
parser.add_argument(
- "--max-questions", dest="max_questions", type=int, default=None,
+ "--max-questions",
+ dest="max_questions",
+ type=int,
+ default=None,
help="(ingest only) cap on number of MM questions to render + upload.",
)
parser.add_argument(
- "--upload-batch-size", dest="upload_batch_size", type=int, default=8,
+ "--upload-batch-size",
+ dest="upload_batch_size",
+ type=int,
+ default=8,
help="(ingest only) PDFs per fileupload call.",
)
parser.add_argument(
- "--skip-upload", dest="skip_upload", action="store_true",
+ "--skip-upload",
+ dest="skip_upload",
+ action="store_true",
help="(ingest only) render PDFs locally but don't push to SurfSense.",
)
parser.add_argument(
- "--include-dev", dest="include_dev", action="store_true",
+ "--include-dev",
+ dest="include_dev",
+ action="store_true",
help="(ingest only) shorthand for --split all.",
)
# Per-upload knobs forwarded to /documents/fileupload at ingest;
@@ -270,7 +302,8 @@ class MedXpertQAMMBenchmark:
doc_map, ingest_settings = _load_doc_map(map_path)
questions = _load_questions(
- questions_jsonl, doc_map,
+ questions_jsonl,
+ doc_map,
split_filter=split_filter,
task_filter=task_filter if task_filter != "all" else None,
body_filter=body_filter if body_filter != "all" else None,
@@ -378,13 +411,18 @@ class MedXpertQAMMBenchmark:
manifest_path = run_dir / "run_artifact.json"
manifest_path.write_text(
- json.dumps({
- "suite": self.suite,
- "benchmark": self.name,
- "raw_path": "raw.jsonl",
- "metrics": metrics,
- "extra": artifact.extra,
- }, indent=2, sort_keys=True) + "\n",
+ json.dumps(
+ {
+ "suite": self.suite,
+ "benchmark": self.name,
+ "raw_path": "raw.jsonl",
+ "metrics": metrics,
+ "extra": artifact.extra,
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ + "\n",
encoding="utf-8",
)
return artifact
@@ -536,8 +574,12 @@ def _compute_metrics(
cost_pct = _safe_pct(surf_cost_agg.mean, native_cost_agg.mean)
lat_pct = _safe_pct(surf_lat_agg.median, native_lat_agg.median)
- per_task = _per_field(questions, native_correct, surf_correct, key=lambda q: q.medical_task or "unknown")
- per_body = _per_field(questions, native_correct, surf_correct, key=lambda q: q.body_system or "unknown")
+ per_task = _per_field(
+ questions, native_correct, surf_correct, key=lambda q: q.medical_task or "unknown"
+ )
+ per_body = _per_field(
+ questions, native_correct, surf_correct, key=lambda q: q.body_system or "unknown"
+ )
return {
"native": {
@@ -593,8 +635,7 @@ def _per_field(
"native_accuracy": (sum(n_correct) / len(pairs)) if pairs else 0.0,
"surfsense_accuracy": (sum(s_correct) / len(pairs)) if pairs else 0.0,
"delta_accuracy_pp": (
- 100.0 * (sum(s_correct) - sum(n_correct)) / len(pairs)
- if pairs else 0.0
+ 100.0 * (sum(s_correct) - sum(n_correct)) / len(pairs) if pairs else 0.0
),
}
return out
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/__init__.py b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/__init__.py
index e527b37f4..265dd62f7 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/__init__.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/__init__.py
@@ -11,7 +11,7 @@ document — the corpus is millions of biomedical snippets.
from __future__ import annotations
-from .runner import MirageBenchmark
from ....core import registry as _registry
+from .runner import MirageBenchmark
_registry.register(MirageBenchmark())
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/ingest.py b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/ingest.py
index 59006b6c0..c4aa53fe2 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/ingest.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/ingest.py
@@ -48,9 +48,7 @@ from ....core.registry import RunContext
logger = logging.getLogger(__name__)
-MIRAGE_BENCHMARK_URL = (
- "https://raw.githubusercontent.com/Teddy-XiongGZ/MIRAGE/main/benchmark.json"
-)
+MIRAGE_BENCHMARK_URL = "https://raw.githubusercontent.com/Teddy-XiongGZ/MIRAGE/main/benchmark.json"
# Upstream only ships ONE zip — top-10k retrievals across 5 retrievers,
# ~16 GB. We default to skipping it (see `--skip-snippet-filter`) and
# ingesting the chosen corpus in full; this URL is only fetched when
@@ -93,6 +91,24 @@ class SnippetRow:
# ---------------------------------------------------------------------------
+def _reuse_cached_dest(dest: Path, *, expect_zip: bool, label: str) -> Path | None:
+ """Return ``dest`` if a usable cache hit, else ``None`` (and delete corrupt zips)."""
+
+ if not dest.exists():
+ return None
+ if expect_zip and not _is_valid_zip(dest):
+ logger.warning(
+ "Cached %s at %s failed ZIP validation (size=%d B); deleting and re-downloading.",
+ label,
+ dest,
+ dest.stat().st_size,
+ )
+ dest.unlink(missing_ok=True)
+ return None
+ logger.info("Using cached %s at %s", label, dest)
+ return dest
+
+
async def _fetch_to_path(
url: str,
*,
@@ -127,19 +143,9 @@ async def _fetch_to_path(
surprise-grabbing 16 GB on a slow link.
"""
- if dest.exists():
- if expect_zip and not _is_valid_zip(dest):
- logger.warning(
- "Cached %s at %s failed ZIP validation (size=%d B); deleting "
- "and re-downloading.",
- label,
- dest,
- dest.stat().st_size,
- )
- dest.unlink(missing_ok=True)
- else:
- logger.info("Using cached %s at %s", label, dest)
- return dest
+ cached = _reuse_cached_dest(dest, expect_zip=expect_zip, label=label)
+ if cached is not None:
+ return cached
dest.parent.mkdir(parents=True, exist_ok=True)
partial = dest.with_suffix(dest.suffix + ".partial")
@@ -167,42 +173,44 @@ async def _fetch_to_path(
)
try:
- async with httpx.AsyncClient(
- timeout=httpx.Timeout(timeout_s, connect=20.0),
- follow_redirects=True,
- ) as client:
- async with client.stream("GET", url, headers=headers) as response:
- if existing_bytes and response.status_code == 200:
- logger.warning(
- "Server ignored Range header for %s; restarting from 0.",
- label,
- )
- partial.unlink(missing_ok=True)
- existing_bytes = 0
- elif response.status_code == 416:
- # Range not satisfiable — the .partial is at or
- # past the end. Treat as "already downloaded";
- # validate by closing and re-opening for atomic
- # rename below.
- logger.info(
- "Server reports %s already complete (HTTP 416).",
- label,
- )
- elif response.status_code not in (200, 206):
- response.raise_for_status()
+ async with (
+ httpx.AsyncClient(
+ timeout=httpx.Timeout(timeout_s, connect=20.0),
+ follow_redirects=True,
+ ) as client,
+ client.stream("GET", url, headers=headers) as response,
+ ):
+ if existing_bytes and response.status_code == 200:
+ logger.warning(
+ "Server ignored Range header for %s; restarting from 0.",
+ label,
+ )
+ partial.unlink(missing_ok=True)
+ existing_bytes = 0
+ elif response.status_code == 416:
+ # Range not satisfiable — the .partial is at or
+ # past the end. Treat as "already downloaded";
+ # validate by closing and re-opening for atomic
+ # rename below.
+ logger.info(
+ "Server reports %s already complete (HTTP 416).",
+ label,
+ )
+ elif response.status_code not in (200, 206):
+ response.raise_for_status()
- total_size = _planned_total_size(response, existing_bytes)
- if (
- total_size is not None
- and total_size > _LARGE_DOWNLOAD_BYTES
- and not allow_large_download
- ):
- raise _LargeDownloadAbort(label, total_size)
+ total_size = _planned_total_size(response, existing_bytes)
+ if (
+ total_size is not None
+ and total_size > _LARGE_DOWNLOAD_BYTES
+ and not allow_large_download
+ ):
+ raise _LargeDownloadAbort(label, total_size)
- mode = "ab" if existing_bytes else "wb"
- with partial.open(mode) as fh:
- async for chunk in response.aiter_bytes(chunk_size=1 << 18):
- fh.write(chunk)
+ mode = "ab" if existing_bytes else "wb"
+ with partial.open(mode) as fh:
+ async for chunk in response.aiter_bytes(chunk_size=1 << 18):
+ fh.write(chunk)
# Optional content sanity check before promoting to dest.
if expect_zip and not _is_valid_zip(partial):
raise zipfile.BadZipFile(
@@ -215,7 +223,7 @@ async def _fetch_to_path(
raise
except _RETRYABLE_NET_EXC as exc:
last_exc = exc
- wait = min(60.0, 2.0 ** attempt)
+ wait = min(60.0, 2.0**attempt)
logger.warning(
"Network error fetching %s (%s: %s); retrying in %.0fs.",
label,
@@ -228,7 +236,7 @@ async def _fetch_to_path(
last_exc = exc
# Truncated body — drop the partial and retry from scratch.
partial.unlink(missing_ok=True)
- wait = min(60.0, 2.0 ** attempt)
+ wait = min(60.0, 2.0**attempt)
logger.warning(
"Truncated ZIP for %s; restarting from byte 0 in %.0fs.",
label,
@@ -270,9 +278,9 @@ class _LargeDownloadAbort(RuntimeError):
"""Raised when a download exceeds the safety threshold without opt-in."""
def __init__(self, label: str, size_bytes: int) -> None:
- gb = size_bytes / (1024 ** 3)
+ gb = size_bytes / (1024**3)
super().__init__(
- f"{label} would download ~{gb:.1f} GB, above the {_LARGE_DOWNLOAD_BYTES / (1024 ** 3):.0f} GB safety cap. "
+ f"{label} would download ~{gb:.1f} GB, above the {_LARGE_DOWNLOAD_BYTES / (1024**3):.0f} GB safety cap. "
"Re-run with `--allow-large-download` to acknowledge, or use "
"`--skip-snippet-filter` to bypass this download entirely and "
"ingest the full corpus instead."
@@ -312,9 +320,7 @@ def _read_snippet_ids(zip_path: Path, *, tasks: list[str]) -> dict[str, set[str]
return out
-def _load_corpus(
- corpus_name: str, snippet_ids: set[str] | None
-) -> Iterable[SnippetRow]:
+def _load_corpus(corpus_name: str, snippet_ids: set[str] | None) -> Iterable[SnippetRow]:
"""Stream rows from a MedRAG HF corpus.
* ``snippet_ids=None`` → yield every row (full-corpus ingestion path).
@@ -533,10 +539,7 @@ async def run_ingest(
logger.warning("Failed to list chunks for doc_id=%s: %s", doc_id, exc)
continue
for chunk in chunks:
- fh.write(
- json.dumps({"chunk_id": chunk.id, "document_id": doc_id})
- + "\n"
- )
+ fh.write(json.dumps({"chunk_id": chunk.id, "document_id": doc_id}) + "\n")
new_state = ctx.suite_state
new_state.ingestion_maps["mirage"] = str(snippet_map_path)
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/prompt.py b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/prompt.py
index 9e5b1c618..3e5192aaa 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/prompt.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/prompt.py
@@ -8,7 +8,6 @@ from __future__ import annotations
from collections.abc import Mapping
-
_PROMPT_TEMPLATE = """\
You are a helpful medical expert. Answer the following multiple-choice
question using the relevant medical knowledge available to you (and any
diff --git a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/runner.py b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/runner.py
index b01b645a9..76e719f1f 100644
--- a/surfsense_evals/src/surfsense_evals/suites/medical/mirage/runner.py
+++ b/surfsense_evals/src/surfsense_evals/suites/medical/mirage/runner.py
@@ -29,7 +29,6 @@ from ....core.ingest_settings import (
)
from ....core.metrics.mc_accuracy import accuracy_with_wilson_ci, macro_accuracy
from ....core.registry import (
- Benchmark,
ReportSection,
RunArtifact,
RunContext,
@@ -135,15 +134,23 @@ class MirageBenchmark:
choices=("all", *_TASKS),
help="Run a single task or all (default: all).",
)
- parser.add_argument("--n", dest="sample_n", type=int, default=None,
- help="Stratified sample size across tasks.")
+ parser.add_argument(
+ "--n",
+ dest="sample_n",
+ type=int,
+ default=None,
+ help="Stratified sample size across tasks.",
+ )
parser.add_argument("--concurrency", type=int, default=4)
parser.add_argument(
- "--corpus", default="MedRAG/textbooks",
+ "--corpus",
+ default="MedRAG/textbooks",
help="HF MedRAG corpus to ingest from (default: MedRAG/textbooks).",
)
parser.add_argument(
- "--max-snippets-per-task", type=int, default=None,
+ "--max-snippets-per-task",
+ type=int,
+ default=None,
help="Cap the per-task ingestion to N snippets (smoke).",
)
# Mutually exclusive: by default we skip the upstream 16 GB
@@ -153,18 +160,24 @@ class MirageBenchmark:
# --allow-large-download).
snippet_group = parser.add_mutually_exclusive_group()
snippet_group.add_argument(
- "--use-snippet-filter", dest="use_snippet_filter", action="store_true",
+ "--use-snippet-filter",
+ dest="use_snippet_filter",
+ action="store_true",
default=False,
help="Download retrieved_snippets_10k.zip (~16 GB) and "
- "filter the corpus to those ids before ingest. "
- "Default: skip and ingest entire corpus.",
+ "filter the corpus to those ids before ingest. "
+ "Default: skip and ingest entire corpus.",
)
snippet_group.add_argument(
- "--skip-snippet-filter", dest="use_snippet_filter", action="store_false",
+ "--skip-snippet-filter",
+ dest="use_snippet_filter",
+ action="store_false",
help="(Default) Skip the 16 GB upstream zip; ingest entire corpus.",
)
parser.add_argument(
- "--allow-large-download", action="store_true", default=False,
+ "--allow-large-download",
+ action="store_true",
+ default=False,
help="Permit downloads larger than 2 GB (e.g. retrieved_snippets_10k.zip).",
)
# Per-upload knobs; ignored at run-time (runner reads the
@@ -197,16 +210,13 @@ class MirageBenchmark:
"`python -m surfsense_evals ingest medical mirage` first."
)
benchmark = json.loads(bench_path.read_text(encoding="utf-8"))
- ingest_settings = read_settings_header(
- ctx.maps_dir() / "mirage_snippet_map.jsonl"
- )
+ ingest_settings = read_settings_header(ctx.maps_dir() / "mirage_snippet_map.jsonl")
questions = _load_questions(benchmark, tasks=tasks, sample_n=sample_n)
if not questions:
raise RuntimeError(
f"No MIRAGE questions matched task={task_filter!r} sample_n={sample_n!r}."
)
- logger.info("MIRAGE: scheduled %d questions across tasks %s",
- len(questions), tasks)
+ logger.info("MIRAGE: scheduled %d questions across tasks %s", len(questions), tasks)
arm = SurfSenseArm(
client=ctx.new_chat_client(),
@@ -229,7 +239,7 @@ class MirageBenchmark:
run_dir = ctx.runs_dir(run_timestamp=run_timestamp)
raw_path = run_dir / "raw.jsonl"
with raw_path.open("w", encoding="utf-8") as fh:
- for q, res in zip(questions, results):
+ for q, res in zip(questions, results, strict=False):
fh.write(
json.dumps(
{
@@ -246,7 +256,7 @@ class MirageBenchmark:
for task in tasks:
n_correct = 0
n_total = 0
- for q, res in zip(questions, results):
+ for q, res in zip(questions, results, strict=False):
if q.task != task:
continue
n_total += 1
@@ -256,7 +266,10 @@ class MirageBenchmark:
per_task_acc[task] = acc.to_dict()
macro = macro_accuracy(
- {t: accuracy_with_wilson_ci(d["n_correct"], d["n_total"]) for t, d in per_task_acc.items()}
+ {
+ t: accuracy_with_wilson_ci(d["n_correct"], d["n_total"])
+ for t, d in per_task_acc.items()
+ }
)
metrics = {"per_task": per_task_acc, "macro_accuracy": macro}
diff --git a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/grader.py b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/grader.py
index 7edad73eb..ecb5144e8 100644
--- a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/grader.py
+++ b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/grader.py
@@ -112,8 +112,9 @@ def _grade_int(pred: str, gold: str) -> GradeResult:
if p_match is None:
return GradeResult(False, 0.0, "int_eq", str(p_match), str(g_val))
p_val = int(p_match.group(0).replace(",", ""))
- return GradeResult(p_val == g_val, 1.0 if p_val == g_val else 0.0,
- "int_eq", str(p_val), str(g_val))
+ return GradeResult(
+ p_val == g_val, 1.0 if p_val == g_val else 0.0, "int_eq", str(p_val), str(g_val)
+ )
_FLOAT_RE = re.compile(r"-?\d+(?:[.,]\d+)?")
@@ -145,15 +146,15 @@ def _grade_list(pred: str, gold: str) -> GradeResult:
return _grade_str(pred, gold)
inter = g_items & p_items
if not inter:
- return GradeResult(False, 0.0, "list_set",
- ", ".join(sorted(p_items)),
- ", ".join(sorted(g_items)))
+ return GradeResult(
+ False, 0.0, "list_set", ", ".join(sorted(p_items)), ", ".join(sorted(g_items))
+ )
precision = len(inter) / len(p_items) if p_items else 0.0
recall = len(inter) / len(g_items)
f1 = (2 * precision * recall / (precision + recall)) if (precision + recall) else 0.0
- return GradeResult(f1 >= 0.999, f1, "list_set",
- ", ".join(sorted(p_items)),
- ", ".join(sorted(g_items)))
+ return GradeResult(
+ f1 >= 0.999, f1, "list_set", ", ".join(sorted(p_items)), ", ".join(sorted(g_items))
+ )
def _grade_none(pred: str, gold: str) -> GradeResult:
@@ -188,8 +189,11 @@ def _grade_none(pred: str, gold: str) -> GradeResult:
expressed_unknown = True
break
return GradeResult(
- expressed_unknown, 1.0 if expressed_unknown else 0.0,
- "none_match", p, _normalise_text(gold),
+ expressed_unknown,
+ 1.0 if expressed_unknown else 0.0,
+ "none_match",
+ p,
+ _normalise_text(gold),
)
diff --git a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/ingest.py b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/ingest.py
index 15cdbeb77..3c736756a 100644
--- a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/ingest.py
+++ b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/ingest.py
@@ -41,6 +41,7 @@ logger = logging.getLogger(__name__)
HF_REPO_ID = "yubo2333/MMLongBench-Doc"
HF_REPO_TYPE = "dataset"
+
# Lazy import: huggingface_hub + pyarrow are heavyweight; keep the
# benchmark module importable on machines that have only the core
# install (e.g. CI lint jobs).
@@ -63,11 +64,11 @@ def _list_repo_files() -> list[str]:
@dataclass
class MMLongBenchQuestion:
- doc_id: str # filename inside the documents/ folder
+ doc_id: str # filename inside the documents/ folder
doc_type: str
question: str
answer: str
- answer_format: str # Str / Int / Float / List / None
+ answer_format: str # Str / Int / Float / List / None
evidence_pages: list[int]
evidence_sources: list[str]
@@ -161,7 +162,9 @@ def _download_questions_parquet(cache_dir: Path) -> Path:
)
parquet_paths.append(Path(local))
logger.info("Cached MMLongBench parquet shard %s -> %s", rel, local)
- return parquet_paths[0] if len(parquet_paths) == 1 else _merge_parquets(parquet_paths, cache_dir)
+ return (
+ parquet_paths[0] if len(parquet_paths) == 1 else _merge_parquets(parquet_paths, cache_dir)
+ )
def _merge_parquets(paths: list[Path], cache_dir: Path) -> Path:
@@ -221,7 +224,7 @@ async def _upload_pdfs(
name_to_id: dict[str, int] = {}
pdf_list = list(pdf_paths)
for batch_start in range(0, len(pdf_list), batch_size):
- batch = pdf_list[batch_start:batch_start + batch_size]
+ batch = pdf_list[batch_start : batch_start + batch_size]
result = await docs_client.upload(
files=batch,
search_space_id=ctx.search_space_id,
@@ -243,8 +246,10 @@ async def _upload_pdfs(
name_to_id[s.title] = s.document_id
logger.info(
"Uploaded MMLongBench batch %d-%d: %d new, %d duplicate",
- batch_start, batch_start + len(batch),
- len(result.document_ids), len(result.duplicate_document_ids),
+ batch_start,
+ batch_start + len(batch),
+ len(result.document_ids),
+ len(result.duplicate_document_ids),
)
return name_to_id
@@ -299,15 +304,20 @@ async def run_ingest(
questions_jsonl = bench_dir / "questions.jsonl"
with questions_jsonl.open("w", encoding="utf-8") as fh:
for q in questions:
- fh.write(json.dumps({
- "doc_id": q.doc_id,
- "doc_type": q.doc_type,
- "question": q.question,
- "answer": q.answer,
- "answer_format": q.answer_format,
- "evidence_pages": q.evidence_pages,
- "evidence_sources": q.evidence_sources,
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ "doc_id": q.doc_id,
+ "doc_type": q.doc_type,
+ "question": q.question,
+ "answer": q.answer,
+ "answer_format": q.answer_format,
+ "evidence_pages": q.evidence_pages,
+ "evidence_sources": q.evidence_sources,
+ }
+ )
+ + "\n"
+ )
logger.info("Wrote %d MMLongBench questions to %s", len(questions), questions_jsonl)
# Step 2: download unique PDFs
@@ -348,12 +358,17 @@ async def run_ingest(
local = pdf_paths.get(doc_id)
if local is None:
continue
- fh.write(json.dumps({
- "doc_id": doc_id,
- "document_id": name_to_id.get(local.name),
- "pdf_path": str(local),
- "n_questions": sum(1 for q in questions if q.doc_id == doc_id),
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ "doc_id": doc_id,
+ "document_id": name_to_id.get(local.name),
+ "pdf_path": str(local),
+ "n_questions": sum(1 for q in questions if q.doc_id == doc_id),
+ }
+ )
+ + "\n"
+ )
logger.info("Wrote MMLongBench doc map to %s", map_path)
new_state = ctx.suite_state
diff --git a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/prompt.py b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/prompt.py
index 27d6a0d00..70229dc15 100644
--- a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/prompt.py
+++ b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/prompt.py
@@ -18,10 +18,7 @@ _FORMAT_HINTS: dict[str, str] = {
"Respond with the answer as a short phrase, no full sentence. "
"Format your final line as `Answer: `."
),
- "int": (
- "Respond with a single integer only. "
- "Format your final line as `Answer: `."
- ),
+ "int": ("Respond with a single integer only. Format your final line as `Answer: `."),
"float": (
"Respond with a single decimal number only (no units). "
"Format your final line as `Answer: `."
diff --git a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/runner.py b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/runner.py
index b7685766e..782ba5d9a 100644
--- a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/runner.py
+++ b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/mmlongbench/runner.py
@@ -58,8 +58,8 @@ logger = logging.getLogger(__name__)
@dataclass
class MMLBQuestion:
- qid: str # synthesised from doc_id + index
- doc_id: str # filename inside the documents/ folder
+ qid: str # synthesised from doc_id + index
+ doc_id: str # filename inside the documents/ folder
doc_type: str
question: str
gold_answer: str
@@ -126,18 +126,20 @@ def _load_questions(
continue
idx = per_doc_counter.get(doc_id, 0)
per_doc_counter[doc_id] = idx + 1
- out.append(MMLBQuestion(
- qid=f"{doc_id}::Q{idx:03d}",
- doc_id=doc_id,
- doc_type=str(row.get("doc_type") or "").strip(),
- question=str(row.get("question") or "").strip(),
- gold_answer=gold,
- answer_format=answer_format,
- evidence_pages=list(row.get("evidence_pages") or []),
- evidence_sources=list(row.get("evidence_sources") or []),
- pdf_path=Path(map_row["pdf_path"]),
- document_id=map_row.get("document_id"),
- ))
+ out.append(
+ MMLBQuestion(
+ qid=f"{doc_id}::Q{idx:03d}",
+ doc_id=doc_id,
+ doc_type=str(row.get("doc_type") or "").strip(),
+ question=str(row.get("question") or "").strip(),
+ gold_answer=gold,
+ answer_format=answer_format,
+ evidence_pages=list(row.get("evidence_pages") or []),
+ evidence_sources=list(row.get("evidence_sources") or []),
+ pdf_path=Path(map_row["pdf_path"]),
+ document_id=map_row.get("document_id"),
+ )
+ )
out.sort(key=lambda q: (q.doc_id, q.qid))
if sample_n is not None and sample_n > 0:
out = out[:sample_n]
@@ -202,41 +204,61 @@ class MMLongBenchDocBenchmark:
help="Filter to one answer format. 'none' = unanswerable probes only.",
)
parser.add_argument(
- "--n", dest="sample_n", type=int, default=None,
+ "--n",
+ dest="sample_n",
+ type=int,
+ default=None,
help="Run only the first N questions after filters apply.",
)
parser.add_argument(
- "--skip-unanswerable", dest="skip_unanswerable", action="store_true",
+ "--skip-unanswerable",
+ dest="skip_unanswerable",
+ action="store_true",
help="Drop ~22%% unanswerable questions (use to compare against baselines that don't include them).",
)
parser.add_argument(
- "--concurrency", type=int, default=4,
+ "--concurrency",
+ type=int,
+ default=4,
help="Parallel question workers per arm.",
)
parser.add_argument(
- "--no-mentions", dest="no_mentions", action="store_true",
+ "--no-mentions",
+ dest="no_mentions",
+ action="store_true",
help="SurfSense arm: skip mentioned_document_ids (unscoped retrieval).",
)
parser.add_argument(
- "--pdf-engine", default="native",
+ "--pdf-engine",
+ default="native",
choices=[e.value for e in PdfEngine],
help="OpenRouter file-parser engine for the native arm.",
)
parser.add_argument(
- "--max-output-tokens", type=int, default=512,
+ "--max-output-tokens",
+ type=int,
+ default=512,
help="Cap on completion length for both arms.",
)
# Ingest-only knobs (forwarded by the CLI to ingest.run_ingest).
parser.add_argument(
- "--max-docs", dest="max_docs", type=int, default=None,
+ "--max-docs",
+ dest="max_docs",
+ type=int,
+ default=None,
help="(ingest only) cap on number of unique PDFs to download + upload.",
)
parser.add_argument(
- "--upload-batch-size", dest="upload_batch_size", type=int, default=8,
+ "--upload-batch-size",
+ dest="upload_batch_size",
+ type=int,
+ default=8,
help="(ingest only) PDFs per fileupload call.",
)
parser.add_argument(
- "--skip-upload", dest="skip_upload", action="store_true",
+ "--skip-upload",
+ dest="skip_upload",
+ action="store_true",
help="(ingest only) cache PDFs locally but don't push to SurfSense.",
)
# Per-upload knobs forwarded to /documents/fileupload at ingest;
@@ -278,7 +300,8 @@ class MMLongBenchDocBenchmark:
doc_map, ingest_settings = _load_doc_map(map_path)
questions = _load_questions(
- questions_jsonl, doc_map,
+ questions_jsonl,
+ doc_map,
doc_filter=doc_filter,
format_filter=None if format_filter == "all" else format_filter,
sample_n=sample_n,
@@ -292,9 +315,7 @@ class MMLongBenchDocBenchmark:
api_key = os.environ.get("OPENROUTER_API_KEY")
if not api_key:
- raise RuntimeError(
- "OPENROUTER_API_KEY env var is required for the native arm."
- )
+ raise RuntimeError("OPENROUTER_API_KEY env var is required for the native arm.")
# Native arm slug differs from SurfSense slug only in cost-arbitrage
# scenario; otherwise both arms answer with provider_model.
@@ -362,18 +383,30 @@ class MMLongBenchDocBenchmark:
"evidence_sources": q.evidence_sources,
"document_id": q.document_id,
}
- fh.write(json.dumps({
- **meta,
- **n_res.to_jsonl(),
- "graded": _grade_to_jsonl(n_g),
- }) + "\n")
- fh.write(json.dumps({
- **meta,
- **s_res.to_jsonl(),
- "graded": _grade_to_jsonl(s_g),
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ **meta,
+ **n_res.to_jsonl(),
+ "graded": _grade_to_jsonl(n_g),
+ }
+ )
+ + "\n"
+ )
+ fh.write(
+ json.dumps(
+ {
+ **meta,
+ **s_res.to_jsonl(),
+ "graded": _grade_to_jsonl(s_g),
+ }
+ )
+ + "\n"
+ )
- metrics = _compute_metrics(questions, native_results, surf_results, native_grades, surf_grades)
+ metrics = _compute_metrics(
+ questions, native_results, surf_results, native_grades, surf_grades
+ )
artifact = RunArtifact(
suite=self.suite,
benchmark=self.name,
@@ -398,13 +431,18 @@ class MMLongBenchDocBenchmark:
manifest_path = run_dir / "run_artifact.json"
manifest_path.write_text(
- json.dumps({
- "suite": self.suite,
- "benchmark": self.name,
- "raw_path": "raw.jsonl",
- "metrics": metrics,
- "extra": artifact.extra,
- }, indent=2, sort_keys=True) + "\n",
+ json.dumps(
+ {
+ "suite": self.suite,
+ "benchmark": self.name,
+ "raw_path": "raw.jsonl",
+ "metrics": metrics,
+ "extra": artifact.extra,
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ + "\n",
encoding="utf-8",
)
return artifact
@@ -450,9 +488,7 @@ class MMLongBenchDocBenchmark:
f"(McNemar p={_fmt(delta.get('mcnemar_p_value'), 4)}, "
f"method={delta.get('mcnemar_method')})"
)
- body_lines.append(
- f" - F1 (mean): SurfSense {_pp(delta.get('f1_pp'))} pp"
- )
+ body_lines.append(f" - F1 (mean): SurfSense {_pp(delta.get('f1_pp'))} pp")
body_lines.append(
f" - Bootstrap 95% CI on accuracy delta: "
f"[{_pp(delta.get('bootstrap_ci_low'))}pp, {_pp(delta.get('bootstrap_ci_high'))}pp]"
@@ -472,8 +508,8 @@ class MMLongBenchDocBenchmark:
for fmt, vals in sorted(per_format.items()):
body_lines.append(
f" - {fmt}: SurfSense {_pp(vals.get('delta_accuracy_pp'))} pp "
- f"(n={vals.get('n')}, native acc={vals.get('native_accuracy', 0)*100:.1f}%, "
- f"surf acc={vals.get('surfsense_accuracy', 0)*100:.1f}%)"
+ f"(n={vals.get('n')}, native acc={vals.get('native_accuracy', 0) * 100:.1f}%, "
+ f"surf acc={vals.get('surfsense_accuracy', 0) * 100:.1f}%)"
)
return ReportSection(
@@ -576,8 +612,7 @@ def _compute_metrics(
"native_accuracy": (sum(n_correct) / len(pairs)) if pairs else 0.0,
"surfsense_accuracy": (sum(s_correct) / len(pairs)) if pairs else 0.0,
"delta_accuracy_pp": (
- 100.0 * (sum(s_correct) - sum(n_correct)) / len(pairs)
- if pairs else 0.0
+ 100.0 * (sum(s_correct) - sum(n_correct)) / len(pairs) if pairs else 0.0
),
}
@@ -593,8 +628,12 @@ def _compute_metrics(
"latency_ms_mean": native_latency_agg.mean,
"latency_ms_median": native_latency_agg.median,
"latency_ms_p95": native_latency_agg.p95,
- "input_tokens_mean": (sum(native_in_tokens) / len(native_in_tokens)) if native_in_tokens else 0.0,
- "output_tokens_mean": (sum(native_out_tokens) / len(native_out_tokens)) if native_out_tokens else 0.0,
+ "input_tokens_mean": (sum(native_in_tokens) / len(native_in_tokens))
+ if native_in_tokens
+ else 0.0,
+ "output_tokens_mean": (sum(native_out_tokens) / len(native_out_tokens))
+ if native_out_tokens
+ else 0.0,
},
"surfsense": {
**surf_acc.to_dict(),
diff --git a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/ingest.py b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/ingest.py
index 93c8db4ab..a7fce60d1 100644
--- a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/ingest.py
+++ b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/ingest.py
@@ -41,8 +41,6 @@ from typing import Any
from ....core.config import set_suite_state
from ....core.parsers import (
- AzureDIError,
- LlamaCloudError,
count_pdf_pages,
parse_with_azure_di,
parse_with_llamacloud,
@@ -55,9 +53,9 @@ logger = logging.getLogger(__name__)
# Order matters for the manifest only (deterministic JSONL diffs);
# the runner doesn't rely on it.
PARSER_ARMS: tuple[tuple[str, str, str], ...] = (
- ("azure_basic_lc", "azure", "basic"),
- ("azure_premium_lc", "azure", "premium"),
- ("llamacloud_basic_lc", "llamacloud", "basic"),
+ ("azure_basic_lc", "azure", "basic"),
+ ("azure_premium_lc", "azure", "premium"),
+ ("llamacloud_basic_lc", "llamacloud", "basic"),
("llamacloud_premium_lc", "llamacloud", "premium"),
)
@@ -100,9 +98,7 @@ class PdfManifestRow:
"pdf_path": str(self.pdf_path),
"document_id": self.document_id,
"pages": self.pages,
- "extractions": {
- arm: ext.to_jsonl() for arm, ext in self.extractions.items()
- },
+ "extractions": {arm: ext.to_jsonl() for arm, ext in self.extractions.items()},
}
@@ -126,12 +122,14 @@ async def _run_one_extraction(
markdown = await parse_with_azure_di(pdf_path, processing_mode=mode)
elif parser == "llamacloud":
markdown = await parse_with_llamacloud(
- pdf_path, processing_mode=mode, estimated_pages=estimated_pages,
+ pdf_path,
+ processing_mode=mode,
+ estimated_pages=estimated_pages,
)
else:
raise ValueError(f"Unknown parser {parser!r}")
out_path.parent.mkdir(parents=True, exist_ok=True)
- out_path.write_text(markdown, encoding="utf-8")
+ await asyncio.to_thread(out_path.write_text, markdown, encoding="utf-8")
return markdown, time.monotonic() - started
@@ -170,14 +168,17 @@ async def _extract_one_pdf(
error="(cached)",
)
logger.info(
- "Cached extraction reused: %s (%d chars)", out_path.name, len(cached),
+ "Cached extraction reused: %s (%d chars)",
+ out_path.name,
+ len(cached),
)
coros.append(_noop())
else:
coros.append(
_run_one_extraction(
pdf_path,
- parser=parser, mode=mode,
+ parser=parser,
+ mode=mode,
out_path=out_path,
estimated_pages=estimated_pages,
)
@@ -192,16 +193,24 @@ async def _extract_one_pdf(
err_msg = f"{type(err).__name__}: {err}"
logger.warning(
"Extraction FAILED for %s [%s/%s]: %s",
- pdf_path.name, parser, mode, err_msg,
+ pdf_path.name,
+ parser,
+ mode,
+ err_msg,
)
out[arm_name] = ExtractionResult(
- arm=arm_name, parser=parser, mode=mode,
- status="failed", error=err_msg,
+ arm=arm_name,
+ parser=parser,
+ mode=mode,
+ status="failed",
+ error=err_msg,
)
else:
markdown, elapsed = result
out[arm_name] = ExtractionResult(
- arm=arm_name, parser=parser, mode=mode,
+ arm=arm_name,
+ parser=parser,
+ mode=mode,
markdown_path=out_path,
chars=len(markdown),
elapsed_s=elapsed,
@@ -290,9 +299,7 @@ async def run_ingest(
rows_in_scope = rows_in_scope[:max_docs]
if not rows_in_scope:
- raise RuntimeError(
- "No PDFs in scope for parser_compare. Check --docs / --max-docs."
- )
+ raise RuntimeError("No PDFs in scope for parser_compare. Check --docs / --max-docs.")
bench_dir = ctx.benchmark_data_dir()
extractions_dir = bench_dir / "extractions"
@@ -319,7 +326,8 @@ async def run_ingest(
logger.info(
"parser_compare: extracting %d PDFs x 4 parsers (concurrency=%d)",
- len(rows_in_scope), pdf_concurrency,
+ len(rows_in_scope),
+ pdf_concurrency,
)
manifest_rows = await asyncio.gather(*(_process(r) for r in rows_in_scope))
@@ -339,12 +347,13 @@ async def run_ingest(
# Quick summary log
total_extractions = sum(len(mr.extractions) for mr in manifest_rows)
failures = sum(
- 1 for mr in manifest_rows for ext in mr.extractions.values()
- if ext.status != "ok"
+ 1 for mr in manifest_rows for ext in mr.extractions.values() if ext.status != "ok"
)
logger.info(
"parser_compare ingest done: %d PDFs, %d extractions, %d failures",
- len(manifest_rows), total_extractions, failures,
+ len(manifest_rows),
+ total_extractions,
+ failures,
)
diff --git a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/prompt.py b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/prompt.py
index 7119bbd29..ccde69e71 100644
--- a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/prompt.py
+++ b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/prompt.py
@@ -34,10 +34,7 @@ _FORMAT_HINTS: dict[str, str] = {
"Respond with the answer as a short phrase, no full sentence. "
"Format your final line as `Answer: `."
),
- "int": (
- "Respond with a single integer only. "
- "Format your final line as `Answer: `."
- ),
+ "int": ("Respond with a single integer only. Format your final line as `Answer: `."),
"float": (
"Respond with a single decimal number only (no units). "
"Format your final line as `Answer: `."
@@ -69,11 +66,7 @@ _BASE_INSTRUCTION = (
def build_native_pdf_prompt(question: str, *, answer_format: str) -> str:
"""Prompt for ``NativePdfArm`` — PDF attached separately as a file part."""
- return (
- f"{_BASE_INSTRUCTION}\n\n"
- f"Question: {question.strip()}\n\n"
- f"{_format_hint(answer_format)}\n"
- )
+ return f"{_BASE_INSTRUCTION}\n\nQuestion: {question.strip()}\n\n{_format_hint(answer_format)}\n"
def build_surfsense_prompt(question: str, *, answer_format: str) -> str:
@@ -82,11 +75,7 @@ def build_surfsense_prompt(question: str, *, answer_format: str) -> str:
# SurfSense's agent already injects retrieved chunks via its tool
# loop; the prompt only carries the user-visible question + format
# hint, mirroring how a human asks the SurfSense UI.
- return (
- f"{_BASE_INSTRUCTION}\n\n"
- f"Question: {question.strip()}\n\n"
- f"{_format_hint(answer_format)}\n"
- )
+ return f"{_BASE_INSTRUCTION}\n\nQuestion: {question.strip()}\n\n{_format_hint(answer_format)}\n"
def build_long_context_prompt(
@@ -105,7 +94,7 @@ def build_long_context_prompt(
return (
f"{_BASE_INSTRUCTION}\n\n"
- f"\n"
+ f'\n'
f"{document_markdown.strip()}\n"
f" \n\n"
f"Question: {question.strip()}\n\n"
diff --git a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/runner.py b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/runner.py
index 2c4a0ffe4..6c009995f 100644
--- a/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/runner.py
+++ b/surfsense_evals/src/surfsense_evals/suites/multimodal_doc/parser_compare/runner.py
@@ -72,7 +72,7 @@ logger = logging.getLogger(__name__)
# Cost tariff (per the user's spec: $1 / 1k pages basic, $10 / 1k pages premium).
# Held as dollars-per-page so per-PDF math is a pure multiply.
PREPROCESS_USD_PER_PAGE = {
- "basic": 1.0 / 1000.0,
+ "basic": 1.0 / 1000.0,
"premium": 10.0 / 1000.0,
}
@@ -183,17 +183,19 @@ def _select_questions(
if ext_blob.get("status") == "ok" and ext_blob.get("markdown_path"):
extractions[arm_name] = Path(ext_blob["markdown_path"])
- out.append(PCQuestion(
- qid=f"{doc_id}::Q{idx:03d}",
- doc_id=doc_id,
- question=str(row.get("question") or "").strip(),
- gold_answer=str(row.get("answer") or "").strip(),
- answer_format=answer_format,
- pdf_path=Path(map_row["pdf_path"]),
- document_id=map_row.get("document_id"),
- pages=int(map_row.get("pages", 0)),
- extractions=extractions,
- ))
+ out.append(
+ PCQuestion(
+ qid=f"{doc_id}::Q{idx:03d}",
+ doc_id=doc_id,
+ question=str(row.get("question") or "").strip(),
+ gold_answer=str(row.get("answer") or "").strip(),
+ answer_format=answer_format,
+ pdf_path=Path(map_row["pdf_path"]),
+ document_id=map_row.get("document_id"),
+ pages=int(map_row.get("pages", 0)),
+ extractions=extractions,
+ )
+ )
per_doc_taken[doc_id] = per_doc_taken.get(doc_id, 0) + 1
out.sort(key=lambda q: (q.doc_id, q.qid))
@@ -242,65 +244,86 @@ class ParserCompareBenchmark:
def add_run_args(self, parser: argparse.ArgumentParser) -> None:
parser.add_argument(
- "--docs", default=None,
+ "--docs",
+ default=None,
help="Comma-separated doc_ids to include (default: all in manifest).",
)
parser.add_argument(
- "--sample-per-doc", type=int, default=1,
+ "--sample-per-doc",
+ type=int,
+ default=1,
help="Take the first N answerable questions per PDF (default 1).",
)
parser.add_argument(
- "--skip-unanswerable", dest="skip_unanswerable",
- action="store_true", default=True,
+ "--skip-unanswerable",
+ dest="skip_unanswerable",
+ action="store_true",
+ default=True,
help="Drop 'None' format probes (default true; we want signal not "
- "hallucination probes for n=5).",
+ "hallucination probes for n=5).",
)
parser.add_argument(
- "--include-unanswerable", dest="skip_unanswerable",
+ "--include-unanswerable",
+ dest="skip_unanswerable",
action="store_false",
help="Override --skip-unanswerable; include unanswerable probes too.",
)
parser.add_argument(
- "--skip-format", default=None,
+ "--skip-format",
+ default=None,
help="Comma-separated answer_format values to skip (e.g. 'none,float').",
)
parser.add_argument(
- "--concurrency", type=int, default=2,
+ "--concurrency",
+ type=int,
+ default=2,
help="Parallel question workers per arm (default 2).",
)
parser.add_argument(
- "--no-mentions", dest="no_mentions", action="store_true",
+ "--no-mentions",
+ dest="no_mentions",
+ action="store_true",
help="SurfSense arm: skip mentioned_document_ids (full-corpus retrieval).",
)
parser.add_argument(
- "--pdf-engine", default="native",
+ "--pdf-engine",
+ default="native",
choices=[e.value for e in PdfEngine],
help="OpenRouter file-parser engine for native_pdf arm.",
)
parser.add_argument(
- "--max-output-tokens", type=int, default=512,
+ "--max-output-tokens",
+ type=int,
+ default=512,
help="Cap on completion length for every arm.",
)
parser.add_argument(
- "--llm-model", default="anthropic/claude-sonnet-4.5",
+ "--llm-model",
+ default="anthropic/claude-sonnet-4.5",
help="OpenRouter slug used by the 5 OpenRouter-driven arms. "
- "SurfSense arm uses whatever provider_model is pinned on the suite.",
+ "SurfSense arm uses whatever provider_model is pinned on the suite.",
)
parser.add_argument(
- "--skip-arms", default=None,
+ "--skip-arms",
+ default=None,
help="Comma-separated arm names to skip (e.g. 'llamacloud_premium_lc').",
)
# Ingest-only flags (forwarded by the CLI to ingest.run_ingest).
parser.add_argument(
- "--max-docs", type=int, default=None,
+ "--max-docs",
+ type=int,
+ default=None,
help="(ingest only) cap number of unique PDFs to process.",
)
parser.add_argument(
- "--force-reextract", action="store_true",
+ "--force-reextract",
+ action="store_true",
help="(ingest only) re-call parsers even if cached .md exists.",
)
parser.add_argument(
- "--pdf-concurrency", type=int, default=2,
+ "--pdf-concurrency",
+ type=int,
+ default=2,
help="(ingest only) parallel PDFs (each fans out to 4 parsers).",
)
@@ -312,9 +335,7 @@ class ParserCompareBenchmark:
from .ingest import run_ingest
docs_raw: str | None = opts.get("docs")
- docs_filter = (
- [d.strip() for d in docs_raw.split(",") if d.strip()] if docs_raw else None
- )
+ docs_filter = [d.strip() for d in docs_raw.split(",") if d.strip()] if docs_raw else None
await run_ingest(
ctx,
docs_filter=docs_filter,
@@ -329,15 +350,14 @@ class ParserCompareBenchmark:
async def run(self, ctx: RunContext, **opts: Any) -> RunArtifact:
docs_raw: str | None = opts.get("docs")
- docs_filter = (
- [d.strip() for d in docs_raw.split(",") if d.strip()] if docs_raw else None
- )
+ docs_filter = [d.strip() for d in docs_raw.split(",") if d.strip()] if docs_raw else None
sample_per_doc = int(opts.get("sample_per_doc") or 1)
skip_unanswerable = bool(opts.get("skip_unanswerable", True))
skip_format_raw: str | None = opts.get("skip_format")
skip_format = (
[f.strip() for f in skip_format_raw.split(",") if f.strip()]
- if skip_format_raw else None
+ if skip_format_raw
+ else None
)
concurrency = int(opts.get("concurrency") or 2)
no_mentions = bool(opts.get("no_mentions"))
@@ -346,8 +366,7 @@ class ParserCompareBenchmark:
llm_model = str(opts.get("llm_model") or "anthropic/claude-sonnet-4.5")
skip_arms_raw: str | None = opts.get("skip_arms")
skip_arms = (
- {a.strip() for a in skip_arms_raw.split(",") if a.strip()}
- if skip_arms_raw else set()
+ {a.strip() for a in skip_arms_raw.split(",") if a.strip()} if skip_arms_raw else set()
)
active_arms = [a for a in ARM_NAMES if a not in skip_arms]
@@ -373,19 +392,20 @@ class ParserCompareBenchmark:
doc_map = _read_doc_map(map_path)
questions = _select_questions(
- questions_jsonl, doc_map,
+ questions_jsonl,
+ doc_map,
docs_filter=docs_filter,
sample_per_doc=sample_per_doc,
skip_unanswerable=skip_unanswerable,
skip_format=skip_format,
)
if not questions:
- raise RuntimeError(
- "No questions matched filters; broaden --docs / --skip-format."
- )
+ raise RuntimeError("No questions matched filters; broaden --docs / --skip-format.")
logger.info(
"parser_compare: scheduled %d questions across %d arms (%s)",
- len(questions), len(active_arms), ",".join(active_arms),
+ len(questions),
+ len(active_arms),
+ ",".join(active_arms),
)
api_key = os.environ.get("OPENROUTER_API_KEY")
@@ -396,16 +416,20 @@ class ParserCompareBenchmark:
arms: dict[str, Any] = {}
if "native_pdf" in active_arms:
native_provider = OpenRouterPdfProvider(
- api_key=api_key, base_url=ctx.config.openrouter_base_url,
- model=llm_model, engine=PdfEngine(pdf_engine_name),
+ api_key=api_key,
+ base_url=ctx.config.openrouter_base_url,
+ model=llm_model,
+ engine=PdfEngine(pdf_engine_name),
)
arms["native_pdf"] = NativePdfArm(
- provider=native_provider, max_output_tokens=max_output_tokens,
+ provider=native_provider,
+ max_output_tokens=max_output_tokens,
)
for arm_name, _, _ in PARSER_ARMS:
if arm_name in active_arms:
lc_provider = OpenRouterChatProvider(
- api_key=api_key, base_url=ctx.config.openrouter_base_url,
+ api_key=api_key,
+ base_url=ctx.config.openrouter_base_url,
model=llm_model,
)
arms[arm_name] = BareLlmArm(
@@ -441,9 +465,7 @@ class ParserCompareBenchmark:
def _lc_req(q: PCQuestion, arm_name: str) -> ArmRequest:
md_path = q.extractions.get(arm_name)
if md_path is None or not md_path.exists():
- raise FileNotFoundError(
- f"Missing extraction for {arm_name} on {q.doc_id}"
- )
+ raise FileNotFoundError(f"Missing extraction for {arm_name} on {q.doc_id}")
markdown = md_path.read_text(encoding="utf-8")
return ArmRequest(
question_id=q.qid,
@@ -483,14 +505,15 @@ class ParserCompareBenchmark:
# Run all arms in parallel (each arm bounded by `concurrency`).
per_arm_tasks: dict[str, list] = {
- arm_name: [_answer_one(arm_name, q) for q in questions]
- for arm_name in active_arms
+ arm_name: [_answer_one(arm_name, q) for q in questions] for arm_name in active_arms
}
per_arm_results: dict[str, list[ArmResult]] = {}
- gathered = await asyncio.gather(*[
- _gather_with_limit(per_arm_tasks[arm_name], concurrency=concurrency)
- for arm_name in active_arms
- ])
+ gathered = await asyncio.gather(
+ *[
+ _gather_with_limit(per_arm_tasks[arm_name], concurrency=concurrency)
+ for arm_name in active_arms
+ ]
+ )
for arm_name, results in zip(active_arms, gathered, strict=True):
per_arm_results[arm_name] = results
@@ -520,21 +543,29 @@ class ParserCompareBenchmark:
for arm_name in active_arms:
res = per_arm_results[arm_name][i]
g = per_arm_grades[arm_name][i]
- fh.write(json.dumps({
- **base,
- **res.to_jsonl(),
- "graded": {
- "correct": g.correct,
- "f1": g.f1,
- "method": g.method,
- "normalised_pred": g.normalised_pred,
- "normalised_gold": g.normalised_gold,
- },
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ **base,
+ **res.to_jsonl(),
+ "graded": {
+ "correct": g.correct,
+ "f1": g.f1,
+ "method": g.method,
+ "normalised_pred": g.normalised_pred,
+ "normalised_gold": g.normalised_gold,
+ },
+ }
+ )
+ + "\n"
+ )
# Aggregate per-arm metrics + cost
metrics = _compute_metrics(
- questions, per_arm_results, per_arm_grades, active_arms,
+ questions,
+ per_arm_results,
+ per_arm_grades,
+ active_arms,
)
artifact = RunArtifact(
@@ -564,13 +595,18 @@ class ParserCompareBenchmark:
manifest_path = run_dir / "run_artifact.json"
manifest_path.write_text(
- json.dumps({
- "suite": self.suite,
- "benchmark": self.name,
- "raw_path": "raw.jsonl",
- "metrics": metrics,
- "extra": artifact.extra,
- }, indent=2, sort_keys=True) + "\n",
+ json.dumps(
+ {
+ "suite": self.suite,
+ "benchmark": self.name,
+ "raw_path": "raw.jsonl",
+ "metrics": metrics,
+ "extra": artifact.extra,
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ + "\n",
encoding="utf-8",
)
return artifact
@@ -602,10 +638,7 @@ class ParserCompareBenchmark:
f"(LLM: `{extra.get('llm_model', '?')}`, "
f"engine: `{extra.get('pdf_engine', 'native')}`)."
)
- body.append(
- f"- Preprocess tariff: basic = $1 / 1k pages, "
- f"premium = $10 / 1k pages."
- )
+ body.append("- Preprocess tariff: basic = $1 / 1k pages, premium = $10 / 1k pages.")
body.append("")
body.append("### Per-arm summary")
body.append("")
@@ -620,13 +653,13 @@ class ParserCompareBenchmark:
continue
body.append(
f"| `{arm_name}` "
- f"| {row['accuracy']*100:.1f}% "
+ f"| {row['accuracy'] * 100:.1f}% "
f"({row['n_correct']}/{row['n']}) "
- f"| {row['f1_mean']*100:.1f}% "
+ f"| {row['f1_mean'] * 100:.1f}% "
f"| ${row['llm_cost_per_q']:.4f} "
f"| ${row['preprocess_cost_total']:.4f} "
f"| ${row['total_cost_per_q']:.4f} "
- f"| {row['latency_ms_median']/1000:.1f}s |"
+ f"| {row['latency_ms_median'] / 1000:.1f}s |"
)
body.append("")
@@ -679,8 +712,7 @@ class ParserCompareBenchmark:
else:
row_cells.append("✓" if g.get("correct") else "✗")
body.append(
- f"| `{doc_id}` | {info.get('pages', '?')} | "
- + " | ".join(row_cells) + " |"
+ f"| `{doc_id}` | {info.get('pages', '?')} | " + " | ".join(row_cells) + " |"
)
return ReportSection(
@@ -740,16 +772,16 @@ def _compute_metrics(
preprocess_per_page = 0.0
preprocess_label = "unknown"
- preprocess_cost_total = sum(
- pages * preprocess_per_page for pages in pdf_pages.values()
- )
+ preprocess_cost_total = sum(pages * preprocess_per_page for pages in pdf_pages.values())
preprocess_cost_per_q = preprocess_cost_total / n if n else 0.0
total_cost_per_q = llm_cost_per_q + preprocess_cost_per_q
latencies = sorted(int(r.latency_ms or 0) for r in results)
latency_median = latencies[len(latencies) // 2] if latencies else 0
- latency_p95 = latencies[int(len(latencies) * 0.95)] if len(latencies) >= 20 else (
- latencies[-1] if latencies else 0
+ latency_p95 = (
+ latencies[int(len(latencies) * 0.95)]
+ if len(latencies) >= 20
+ else (latencies[-1] if latencies else 0)
)
in_tokens = [int(r.input_tokens or 0) for r in results]
@@ -775,15 +807,21 @@ def _compute_metrics(
# Per-PDF breakdown (correct / not for each arm)
per_pdf: dict[str, dict[str, Any]] = {}
for i, q in enumerate(questions):
- slot = per_pdf.setdefault(q.doc_id, {
- "pages": q.pages,
- "arms": {},
- })
+ slot = per_pdf.setdefault(
+ q.doc_id,
+ {
+ "pages": q.pages,
+ "arms": {},
+ },
+ )
for arm_name in active_arms:
- slot["arms"].setdefault(arm_name, {
- "correct": per_arm_grades[arm_name][i].correct,
- "f1": per_arm_grades[arm_name][i].f1,
- })
+ slot["arms"].setdefault(
+ arm_name,
+ {
+ "correct": per_arm_grades[arm_name][i].correct,
+ "f1": per_arm_grades[arm_name][i].f1,
+ },
+ )
return {
"per_arm": per_arm,
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/crag/dataset.py b/surfsense_evals/src/surfsense_evals/suites/research/crag/dataset.py
index 224dcae5c..7154e6d14 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/crag/dataset.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/crag/dataset.py
@@ -80,7 +80,7 @@ class CragPage:
class CragQuestion:
"""One row of CRAG (Tasks 1 & 2)."""
- qid: str # synthesised "C00000".."C02705"
+ qid: str # synthesised "C00000".."C02705"
interaction_id: str
query_time: str
query: str
@@ -89,9 +89,9 @@ class CragQuestion:
domain: str
question_type: str
static_or_dynamic: str
- popularity: str # may be "" for web-sourced questions
- split: int # 0=validation, 1=public_test
- raw_index: int # row index in the source JSONL
+ popularity: str # may be "" for web-sourced questions
+ split: int # 0=validation, 1=public_test
+ raw_index: int # row index in the source JSONL
pages: list[CragPage] = field(default_factory=list)
def to_dict(self) -> dict[str, Any]:
@@ -166,16 +166,19 @@ def _parse_pages(raw_search_results: Any) -> list[CragPage]:
if not url or not html.strip():
# No URL or empty HTML => useless for retrieval.
continue
- pages.append(CragPage(
- page_name=str(entry.get("page_name") or "").strip(),
- page_url=url,
- page_snippet=str(entry.get("page_snippet") or "").strip(),
- page_html=html,
- page_last_modified=(
- str(entry.get("page_last_modified")).strip()
- if entry.get("page_last_modified") else None
- ),
- ))
+ pages.append(
+ CragPage(
+ page_name=str(entry.get("page_name") or "").strip(),
+ page_url=url,
+ page_snippet=str(entry.get("page_snippet") or "").strip(),
+ page_html=html,
+ page_last_modified=(
+ str(entry.get("page_last_modified")).strip()
+ if entry.get("page_last_modified")
+ else None
+ ),
+ )
+ )
return pages
@@ -217,21 +220,23 @@ def iter_questions(jsonl_bz2_path: Path) -> list[CragQuestion]:
continue
interaction_id = str(row.get("interaction_id") or "").strip()
pages = _parse_pages(row.get("search_results"))
- out.append(CragQuestion(
- qid=f"C{raw_idx:05d}",
- interaction_id=interaction_id,
- query_time=str(row.get("query_time") or "").strip(),
- query=query,
- gold_answer=answer,
- alt_answers=_parse_alt_answers(row.get("alt_ans")),
- domain=str(row.get("domain") or "").strip().lower(),
- question_type=str(row.get("question_type") or "").strip().lower(),
- static_or_dynamic=str(row.get("static_or_dynamic") or "").strip().lower(),
- popularity=str(row.get("popularity") or "").strip().lower(),
- split=int(row.get("split") or 0),
- raw_index=raw_idx,
- pages=pages,
- ))
+ out.append(
+ CragQuestion(
+ qid=f"C{raw_idx:05d}",
+ interaction_id=interaction_id,
+ query_time=str(row.get("query_time") or "").strip(),
+ query=query,
+ gold_answer=answer,
+ alt_answers=_parse_alt_answers(row.get("alt_ans")),
+ domain=str(row.get("domain") or "").strip().lower(),
+ question_type=str(row.get("question_type") or "").strip().lower(),
+ static_or_dynamic=str(row.get("static_or_dynamic") or "").strip().lower(),
+ popularity=str(row.get("popularity") or "").strip().lower(),
+ split=int(row.get("split") or 0),
+ raw_index=raw_idx,
+ pages=pages,
+ )
+ )
return out
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/crag/grader.py b/surfsense_evals/src/surfsense_evals/suites/research/crag/grader.py
index 63f66702b..e49660a6f 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/crag/grader.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/crag/grader.py
@@ -58,10 +58,10 @@ class CragGradeResult:
"""One graded (pred, gold) pair under CRAG's 3-class rubric."""
grade: GradeClass
- score: int # +1 / 0 / -1
- method: str # exact, numeric, substring, refusal,
- # false_premise_correct, false_premise_miss,
- # llm_judge, lexical_miss, ...
+ score: int # +1 / 0 / -1
+ method: str # exact, numeric, substring, refusal,
+ # false_premise_correct, false_premise_miss,
+ # llm_judge, lexical_miss, ...
normalised_pred: str = ""
normalised_gold: str = ""
judge_rationale: str = ""
@@ -112,10 +112,27 @@ def _normalise(s: str) -> str:
_WORD_NUMBERS = {
- "zero": 0, "one": 1, "two": 2, "three": 3, "four": 4, "five": 5,
- "six": 6, "seven": 7, "eight": 8, "nine": 9, "ten": 10, "eleven": 11,
- "twelve": 12, "thirteen": 13, "fourteen": 14, "fifteen": 15, "sixteen": 16,
- "seventeen": 17, "eighteen": 18, "nineteen": 19, "twenty": 20,
+ "zero": 0,
+ "one": 1,
+ "two": 2,
+ "three": 3,
+ "four": 4,
+ "five": 5,
+ "six": 6,
+ "seven": 7,
+ "eight": 8,
+ "nine": 9,
+ "ten": 10,
+ "eleven": 11,
+ "twelve": 12,
+ "thirteen": 13,
+ "fourteen": 14,
+ "fifteen": 15,
+ "sixteen": 16,
+ "seventeen": 17,
+ "eighteen": 18,
+ "nineteen": 19,
+ "twenty": 20,
}
_NUMERIC_RE = re.compile(r"-?\d+(?:[.,]\d+)?")
@@ -274,8 +291,11 @@ def grade_deterministic(
continue
if n_pred == cand_norm:
return CragGradeResult(
- grade="correct", score=1, method="exact",
- normalised_pred=n_pred, normalised_gold=cand_norm,
+ grade="correct",
+ score=1,
+ method="exact",
+ normalised_pred=n_pred,
+ normalised_gold=cand_norm,
)
p_num = _maybe_number(pred)
c_num = _maybe_number(candidate)
@@ -289,21 +309,30 @@ def grade_deterministic(
tol = abs(c_num) * 0.01
if abs(p_num - c_num) <= tol:
return CragGradeResult(
- grade="correct", score=1, method="numeric",
- normalised_pred=n_pred, normalised_gold=cand_norm,
+ grade="correct",
+ score=1,
+ method="numeric",
+ normalised_pred=n_pred,
+ normalised_gold=cand_norm,
)
# Numeric question with different numbers — keep looking
# at other candidates rather than declaring miss now;
# alt answers may include word forms that pass.
if _whole_word_substring(n_pred, cand_norm):
return CragGradeResult(
- grade="correct", score=1, method="substring",
- normalised_pred=n_pred, normalised_gold=cand_norm,
+ grade="correct",
+ score=1,
+ method="substring",
+ normalised_pred=n_pred,
+ normalised_gold=cand_norm,
)
if _whole_word_substring(cand_norm, n_pred) and len(n_pred) >= 3:
return CragGradeResult(
- grade="correct", score=1, method="substring_reverse",
- normalised_pred=n_pred, normalised_gold=cand_norm,
+ grade="correct",
+ score=1,
+ method="substring_reverse",
+ normalised_pred=n_pred,
+ normalised_gold=cand_norm,
)
return CragGradeResult(
@@ -326,21 +355,21 @@ _JUDGE_SYSTEM = (
"answer (and any alternative valid answers), and a model's "
"prediction, classify the prediction into exactly one of three "
"categories:\n\n"
- "* \"correct\" — the prediction expresses the same factual "
+ '* "correct" — the prediction expresses the same factual '
"content as the gold answer (paraphrasing OK; numbers as words "
"OK; partial-but-correct names OK; non-contradictory extra "
"detail OK).\n"
- "* \"missing\" — the prediction explicitly refuses, says \"I "
+ '* "missing" — the prediction explicitly refuses, says "I '
"don't know\", says there is insufficient information, or hedges "
"without committing.\n"
- "* \"incorrect\" — the prediction commits to a fact that is "
+ '* "incorrect" — the prediction commits to a fact that is '
"different from the gold answer, or fails to flag a false "
"premise when the question contains one.\n\n"
"Special case: if the question contains a false premise and the "
"gold answer says so, then a prediction that flags the false "
- "premise is \"correct\".\n\n"
+ 'premise is "correct".\n\n'
"Respond with ONLY a JSON object on a single line:\n"
- '{\"grade\": \"correct\"|\"missing\"|\"incorrect\", \"rationale\": \"\"}'
+ '{"grade": "correct"|"missing"|"incorrect", "rationale": ""}'
)
@@ -444,15 +473,17 @@ def _parse_judge_response(text: str) -> tuple[GradeClass, str]:
# Methods that should *not* trigger the LLM judge — the deterministic
# verdict is conclusive (refusal, exact match, numeric mismatch, etc.).
-_TERMINAL_METHODS = frozenset({
- "refusal",
- "exact",
- "numeric",
- "substring",
- "substring_reverse",
- "false_premise_flagged",
- "empty_gold",
-})
+_TERMINAL_METHODS = frozenset(
+ {
+ "refusal",
+ "exact",
+ "numeric",
+ "substring",
+ "substring_reverse",
+ "false_premise_flagged",
+ "empty_gold",
+ }
+)
async def grade_with_judge(
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/crag/html_extract.py b/surfsense_evals/src/surfsense_evals/suites/research/crag/html_extract.py
index 1b00aedc2..271d43d56 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/crag/html_extract.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/crag/html_extract.py
@@ -42,7 +42,7 @@ class ExtractionResult:
"""Outcome of converting one HTML blob to plain markdown."""
text: str
- method: str # "trafilatura" | "fallback_strip" | "empty"
+ method: str # "trafilatura" | "fallback_strip" | "empty"
n_chars: int
@property
@@ -94,11 +94,30 @@ class _StripHTMLParser(HTMLParser):
"""
_SKIP_TAGS = frozenset({"script", "style", "nav", "header", "footer", "aside", "svg"})
- _BLOCK_TAGS = frozenset({
- "p", "div", "section", "article", "li", "ul", "ol",
- "h1", "h2", "h3", "h4", "h5", "h6", "br", "tr",
- "td", "th", "table", "blockquote", "pre",
- })
+ _BLOCK_TAGS = frozenset(
+ {
+ "p",
+ "div",
+ "section",
+ "article",
+ "li",
+ "ul",
+ "ol",
+ "h1",
+ "h2",
+ "h3",
+ "h4",
+ "h5",
+ "h6",
+ "br",
+ "tr",
+ "td",
+ "th",
+ "table",
+ "blockquote",
+ "pre",
+ }
+ )
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
@@ -177,10 +196,7 @@ def extract_main_content(
# Prefer trafilatura output even if short, but only if it
# contained any prose at all — empty trafilatura fall-through
# to the stripped form.
- if body and stripped and len(stripped) > len(body) * 1.5:
- body = stripped
- method = "fallback_strip"
- elif not body and stripped:
+ if body and stripped and len(stripped) > len(body) * 1.5 or not body and stripped:
body = stripped
method = "fallback_strip"
elif body:
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/crag/ingest.py b/surfsense_evals/src/surfsense_evals/suites/research/crag/ingest.py
index 4e0c2bdc5..1b66f45f9 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/crag/ingest.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/crag/ingest.py
@@ -158,7 +158,10 @@ def _materialise_pages(
logger.info(
"CRAG page extraction: %s; empty=%d, total_files=%d across %d questions",
- method_counts, n_empty, len(file_to_url), len(qid_to_files),
+ method_counts,
+ n_empty,
+ len(file_to_url),
+ len(qid_to_files),
)
return qid_to_files, file_to_url
@@ -215,8 +218,10 @@ async def _upload_pages(
name_to_id[f"{s.title}.md"] = s.document_id
logger.info(
"CRAG upload batch %d-%d: %d new, %d duplicate",
- batch_start, batch_start + len(batch),
- len(result.document_ids), len(result.duplicate_document_ids),
+ batch_start,
+ batch_start + len(batch),
+ len(result.document_ids),
+ len(result.duplicate_document_ids),
)
return name_to_id
@@ -243,24 +248,26 @@ def _resolve_question_doc_ids(
doc_ids.append(doc_id)
else:
missing.append(fn)
- rows.append({
- "qid": q.qid,
- "interaction_id": q.interaction_id,
- "raw_index": q.raw_index,
- "question": q.query,
- "gold_answer": q.gold_answer,
- "alt_answers": list(q.alt_answers),
- "domain": q.domain,
- "question_type": q.question_type,
- "static_or_dynamic": q.static_or_dynamic,
- "popularity": q.popularity,
- "query_time": q.query_time,
- "split": q.split,
- "page_filenames": filenames,
- "document_ids": doc_ids,
- "missing_pages": missing,
- "n_pages": len(filenames),
- })
+ rows.append(
+ {
+ "qid": q.qid,
+ "interaction_id": q.interaction_id,
+ "raw_index": q.raw_index,
+ "question": q.query,
+ "gold_answer": q.gold_answer,
+ "alt_answers": list(q.alt_answers),
+ "domain": q.domain,
+ "question_type": q.question_type,
+ "static_or_dynamic": q.static_or_dynamic,
+ "popularity": q.popularity,
+ "query_time": q.query_time,
+ "split": q.split,
+ "page_filenames": filenames,
+ "document_ids": doc_ids,
+ "missing_pages": missing,
+ "n_pages": len(filenames),
+ }
+ )
return rows
@@ -305,7 +312,7 @@ async def run_ingest(
settings = settings or IngestSettings(
use_vision_llm=False,
processing_mode="basic",
- )
+ )
bench_dir = ctx.benchmark_data_dir()
pages_dir = bench_dir / "pages"
raw_cache = bench_dir / ".raw_cache"
@@ -336,10 +343,13 @@ async def run_ingest(
n_pages_total = sum(len(q.pages) for q in questions)
logger.info(
"CRAG: extracting up to %d pages across %d questions ...",
- n_pages_total, len(questions),
+ n_pages_total,
+ len(questions),
)
qid_to_files, file_to_url = _materialise_pages(
- questions, pages_dir=pages_dir, overwrite=overwrite_extract,
+ questions,
+ pages_dir=pages_dir,
+ overwrite=overwrite_extract,
)
n_pages_extracted = sum(len(v) for v in qid_to_files.values())
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/crag/prompt.py b/surfsense_evals/src/surfsense_evals/suites/research/crag/prompt.py
index 626834505..5b29fb90b 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/crag/prompt.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/crag/prompt.py
@@ -28,7 +28,6 @@ in the runner, so the format is mandatory.
from __future__ import annotations
-
_BASE_INSTRUCTIONS = (
"You are a careful question-answering assistant. The question is a "
"real-world factual question that may be about finance, music, "
@@ -38,7 +37,7 @@ _BASE_INSTRUCTIONS = (
"is factually wrong), say so explicitly in your final answer "
"rather than answering as if the premise were true.\n"
"2. If you are not confident in an answer, prefer saying \"I don't "
- "know\" over guessing. A wrong commit is penalised more than a "
+ 'know" over guessing. A wrong commit is penalised more than a '
"refusal.\n"
"3. Keep the final answer short — a name, a number, a date, a "
"phrase. Do not repeat the question.\n\n"
@@ -126,9 +125,7 @@ def build_long_context_prompt(
if len(body) > per_page_char_cap:
body = body[:per_page_char_cap].rstrip() + "\n[...truncated...]"
title_clean = (title or f"page_{idx}").strip().replace("\n", " ")
- blocks.append(
- f"--- PAGE {idx}: {title_clean} ---\n{body}\n"
- )
+ blocks.append(f"--- PAGE {idx}: {title_clean} ---\n{body}\n")
contexts_block = "\n".join(blocks) if blocks else "(no pages retrieved)"
return _LONG_CONTEXT_TEMPLATE.format(
instructions=_BASE_INSTRUCTIONS,
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/crag/runner.py b/surfsense_evals/src/surfsense_evals/suites/research/crag/runner.py
index 654c261a2..801e00220 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/crag/runner.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/crag/runner.py
@@ -125,21 +125,23 @@ def _filter_questions(
continue
if qtype_filter and qtype_filter not in qtype:
continue
- out.append(CragRunnerQuestion(
- qid=str(row.get("qid") or "").strip(),
- raw_index=int(row.get("raw_index") or 0),
- question=str(row.get("question") or "").strip(),
- gold_answer=str(row.get("gold_answer") or "").strip(),
- alt_answers=list(row.get("alt_answers") or []),
- domain=domain,
- question_type=qtype,
- static_or_dynamic=str(row.get("static_or_dynamic") or "").lower(),
- popularity=str(row.get("popularity") or "").lower(),
- query_time=str(row.get("query_time") or "").strip(),
- page_filenames=list(row.get("page_filenames") or []),
- document_ids=list(row.get("document_ids") or []),
- missing_pages=list(row.get("missing_pages") or []),
- ))
+ out.append(
+ CragRunnerQuestion(
+ qid=str(row.get("qid") or "").strip(),
+ raw_index=int(row.get("raw_index") or 0),
+ question=str(row.get("question") or "").strip(),
+ gold_answer=str(row.get("gold_answer") or "").strip(),
+ alt_answers=list(row.get("alt_answers") or []),
+ domain=domain,
+ question_type=qtype,
+ static_or_dynamic=str(row.get("static_or_dynamic") or "").lower(),
+ popularity=str(row.get("popularity") or "").lower(),
+ query_time=str(row.get("query_time") or "").strip(),
+ page_filenames=list(row.get("page_filenames") or []),
+ document_ids=list(row.get("document_ids") or []),
+ missing_pages=list(row.get("missing_pages") or []),
+ )
+ )
out.sort(key=lambda q: q.raw_index)
if sample_n is not None and sample_n > 0:
out = out[:sample_n]
@@ -190,15 +192,22 @@ class CragBenchmark:
def add_run_args(self, parser: argparse.ArgumentParser) -> None:
parser.add_argument(
- "--n", dest="sample_n", type=int, default=None,
+ "--n",
+ dest="sample_n",
+ type=int,
+ default=None,
help="Run only the first N questions after filters.",
)
parser.add_argument(
- "--domain", dest="domain_filter", default=None,
+ "--domain",
+ dest="domain_filter",
+ default=None,
help="Filter to a single CRAG domain (finance|music|movie|sports|open).",
)
parser.add_argument(
- "--qtype", dest="qtype_filter", default=None,
+ "--qtype",
+ dest="qtype_filter",
+ default=None,
help=(
"Filter to questions whose question_type contains this "
"substring (case-insensitive). Examples: 'multi-hop', "
@@ -206,31 +215,46 @@ class CragBenchmark:
),
)
parser.add_argument(
- "--concurrency", type=int, default=4,
+ "--concurrency",
+ type=int,
+ default=4,
help="Parallel question workers per arm.",
)
parser.add_argument(
- "--max-output-tokens", type=int, default=512,
+ "--max-output-tokens",
+ type=int,
+ default=512,
help="Cap on completion length for the chat-completion arms.",
)
parser.add_argument(
- "--per-page-char-cap", dest="per_page_char_cap", type=int, default=12_000,
+ "--per-page-char-cap",
+ dest="per_page_char_cap",
+ type=int,
+ default=12_000,
help="Long-context arm: max chars per page before truncation (default 12k).",
)
parser.add_argument(
- "--skip-bare", dest="skip_bare", action="store_true",
+ "--skip-bare",
+ dest="skip_bare",
+ action="store_true",
help="Skip the bare-LLM arm (saves cost on re-runs).",
)
parser.add_argument(
- "--skip-long-context", dest="skip_long_context", action="store_true",
+ "--skip-long-context",
+ dest="skip_long_context",
+ action="store_true",
help="Skip the long-context arm.",
)
parser.add_argument(
- "--skip-surfsense", dest="skip_surfsense", action="store_true",
+ "--skip-surfsense",
+ dest="skip_surfsense",
+ action="store_true",
help="Skip the SurfSense arm (useful when iterating on the LLM arms only).",
)
parser.add_argument(
- "--no-mention-scope", dest="no_mention_scope", action="store_true",
+ "--no-mention-scope",
+ dest="no_mention_scope",
+ action="store_true",
help=(
"SurfSense arm: don't pass mentioned_document_ids; let "
"the agent retrieve over the entire SearchSpace. Default "
@@ -239,37 +263,56 @@ class CragBenchmark:
),
)
parser.add_argument(
- "--no-judge", dest="no_judge", action="store_true",
+ "--no-judge",
+ dest="no_judge",
+ action="store_true",
help="Disable the LLM-as-judge fallback grader.",
)
parser.add_argument(
- "--judge-model", dest="judge_model",
+ "--judge-model",
+ dest="judge_model",
default="anthropic/claude-sonnet-4.5",
help="OpenRouter slug for the LLM judge.",
)
parser.add_argument(
- "--judge-concurrency", dest="judge_concurrency", type=int, default=4,
+ "--judge-concurrency",
+ dest="judge_concurrency",
+ type=int,
+ default=4,
help="Parallel judge calls.",
)
# Ingest knobs
parser.add_argument(
- "--n-questions", dest="n_questions", type=int, default=None,
+ "--n-questions",
+ dest="n_questions",
+ type=int,
+ default=None,
help="(ingest only) cap on number of questions to materialise + ingest.",
)
parser.add_argument(
- "--upload-batch-size", dest="upload_batch_size", type=int, default=16,
+ "--upload-batch-size",
+ dest="upload_batch_size",
+ type=int,
+ default=16,
help="(ingest only) markdown files per fileupload call.",
)
parser.add_argument(
- "--skip-upload", dest="skip_upload", action="store_true",
+ "--skip-upload",
+ dest="skip_upload",
+ action="store_true",
help="(ingest only) extract pages locally but don't push to SurfSense.",
)
parser.add_argument(
- "--overwrite-extract", dest="overwrite_extract", action="store_true",
+ "--overwrite-extract",
+ dest="overwrite_extract",
+ action="store_true",
help="(ingest only) re-run trafilatura even when cached markdown exists.",
)
parser.add_argument(
- "--sample-seed", dest="sample_seed", type=int, default=17,
+ "--sample-seed",
+ dest="sample_seed",
+ type=int,
+ default=17,
help="(ingest only) RNG seed for the stratified sample.",
)
add_ingest_settings_args(parser, defaults=_DEFAULT_INGEST_SETTINGS)
@@ -362,12 +405,14 @@ class CragBenchmark:
if not api_key:
logger.warning("CRAG: --no-judge implied (no OPENROUTER_API_KEY for judge)")
else:
- judge = CragLlmJudge(config=CragJudgeConfig(
- api_key=api_key,
- model=judge_model,
- base_url=ctx.config.openrouter_base_url,
- concurrency=judge_concurrency,
- ))
+ judge = CragLlmJudge(
+ config=CragJudgeConfig(
+ api_key=api_key,
+ model=judge_model,
+ base_url=ctx.config.openrouter_base_url,
+ concurrency=judge_concurrency,
+ )
+ )
run_timestamp = utc_iso_timestamp()
run_dir = ctx.runs_dir(run_timestamp=run_timestamp)
@@ -393,29 +438,53 @@ class CragBenchmark:
# internally concurrency-bounded.
tasks: list[Any] = []
if bare_arm is not None:
- tasks.append(_gather_with_limit((_bare_one(q) for q in questions), concurrency=concurrency))
+ tasks.append(
+ _gather_with_limit((_bare_one(q) for q in questions), concurrency=concurrency)
+ )
else:
tasks.append(_make_skipped_results(questions, "bare_llm"))
if long_context_arm is not None:
- tasks.append(_gather_with_limit((_long_context_one(q) for q in questions), concurrency=concurrency))
+ tasks.append(
+ _gather_with_limit(
+ (_long_context_one(q) for q in questions), concurrency=concurrency
+ )
+ )
else:
tasks.append(_make_skipped_results(questions, "long_context"))
if surf_arm is not None:
- tasks.append(_gather_with_limit((_surf_one(q) for q in questions), concurrency=concurrency))
+ tasks.append(
+ _gather_with_limit((_surf_one(q) for q in questions), concurrency=concurrency)
+ )
else:
tasks.append(_make_skipped_results(questions, "surfsense"))
bare_results, long_context_results, surf_results = await asyncio.gather(*tasks)
- bare_grades = await _grade_results(questions, bare_results, judge=judge) if bare_arm else _empty_grades(questions)
- lc_grades = await _grade_results(questions, long_context_results, judge=judge) if long_context_arm else _empty_grades(questions)
- surf_grades = await _grade_results(questions, surf_results, judge=judge) if surf_arm else _empty_grades(questions)
+ bare_grades = (
+ await _grade_results(questions, bare_results, judge=judge)
+ if bare_arm
+ else _empty_grades(questions)
+ )
+ lc_grades = (
+ await _grade_results(questions, long_context_results, judge=judge)
+ if long_context_arm
+ else _empty_grades(questions)
+ )
+ surf_grades = (
+ await _grade_results(questions, surf_results, judge=judge)
+ if surf_arm
+ else _empty_grades(questions)
+ )
with raw_path.open("w", encoding="utf-8") as fh:
for q, b_res, l_res, s_res, b_g, l_g, s_g in zip(
questions,
- bare_results, long_context_results, surf_results,
- bare_grades, lc_grades, surf_grades,
+ bare_results,
+ long_context_results,
+ surf_results,
+ bare_grades,
+ lc_grades,
+ surf_grades,
strict=False,
):
meta = {
@@ -431,18 +500,29 @@ class CragBenchmark:
"alt_answers": q.alt_answers,
}
for res, grade in (
- (b_res, b_g), (l_res, l_g), (s_res, s_g),
+ (b_res, b_g),
+ (l_res, l_g),
+ (s_res, s_g),
):
- fh.write(json.dumps({
- **meta,
- **res.to_jsonl(),
- "graded": grade.to_dict(),
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ **meta,
+ **res.to_jsonl(),
+ "graded": grade.to_dict(),
+ }
+ )
+ + "\n"
+ )
metrics = _compute_metrics(
questions=questions,
- bare_results=bare_results, long_context_results=long_context_results, surf_results=surf_results,
- bare_grades=bare_grades, lc_grades=lc_grades, surf_grades=surf_grades,
+ bare_results=bare_results,
+ long_context_results=long_context_results,
+ surf_results=surf_results,
+ bare_grades=bare_grades,
+ lc_grades=lc_grades,
+ surf_grades=surf_grades,
arms_active={
"bare_llm": bare_arm is not None,
"long_context": long_context_arm is not None,
@@ -481,13 +561,18 @@ class CragBenchmark:
manifest_path = run_dir / "run_artifact.json"
manifest_path.write_text(
- json.dumps({
- "suite": self.suite,
- "benchmark": self.name,
- "raw_path": "raw.jsonl",
- "metrics": metrics,
- "extra": artifact.extra,
- }, indent=2, sort_keys=True) + "\n",
+ json.dumps(
+ {
+ "suite": self.suite,
+ "benchmark": self.name,
+ "raw_path": "raw.jsonl",
+ "metrics": metrics,
+ "extra": artifact.extra,
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ + "\n",
encoding="utf-8",
)
return artifact
@@ -547,7 +632,9 @@ class CragBenchmark:
body_lines.append("- Headline truthfulness scores (CRAG paper rubric):")
for label, key in (
- ("Bare LLM", "bare_llm"), ("Long-Context", "long_context"), ("SurfSense", "surfsense"),
+ ("Bare LLM", "bare_llm"),
+ ("Long-Context", "long_context"),
+ ("SurfSense", "surfsense"),
):
d = m.get(key, {})
body_lines.append(
@@ -583,9 +670,7 @@ class CragBenchmark:
for arm in ("bare_llm", "long_context", "surfsense"):
if arm not in row:
continue
- pieces.append(
- f"{arm}={_signed_pct(row[arm].get('truthfulness_score'))}"
- )
+ pieces.append(f"{arm}={_signed_pct(row[arm].get('truthfulness_score'))}")
body_lines.append(" ".join(pieces))
if per_qtype:
@@ -596,9 +681,7 @@ class CragBenchmark:
for arm in ("bare_llm", "long_context", "surfsense"):
if arm not in row:
continue
- pieces.append(
- f"{arm}={_signed_pct(row[arm].get('truthfulness_score'))}"
- )
+ pieces.append(f"{arm}={_signed_pct(row[arm].get('truthfulness_score'))}")
body_lines.append(" ".join(pieces))
return ReportSection(
@@ -669,32 +752,31 @@ async def _grade_results(
rows: list[CragGradeRow] = []
for q, r in zip(questions, results, strict=False):
pred = extract_freeform_answer(r.raw_text or "")
- rows.append(CragGradeRow(
- qid=q.qid,
- question=q.question,
- gold=q.gold_answer,
- alt_answers=q.alt_answers,
- pred=pred,
- question_type=q.question_type,
- ))
+ rows.append(
+ CragGradeRow(
+ qid=q.qid,
+ question=q.question,
+ gold=q.gold_answer,
+ alt_answers=q.alt_answers,
+ pred=pred,
+ question_type=q.question_type,
+ )
+ )
return await grade_many(rows=rows, judge=judge)
def _empty_grades(questions: list[CragRunnerQuestion]) -> list[CragGradeResult]:
- return [
- CragGradeResult(grade="missing", score=0, method="skipped_arm")
- for _ in questions
- ]
+ return [CragGradeResult(grade="missing", score=0, method="skipped_arm") for _ in questions]
async def _make_skipped_results(
- questions: list[CragRunnerQuestion], arm_name: str,
+ questions: list[CragRunnerQuestion],
+ arm_name: str,
) -> list[ArmResult]:
"""Stand-in results so downstream code can assume parallel lists."""
return [
- ArmResult(arm=arm_name, question_id=q.qid, raw_text="", error="skipped")
- for q in questions
+ ArmResult(arm=arm_name, question_id=q.qid, raw_text="", error="skipped") for q in questions
]
@@ -776,20 +858,41 @@ def _compute_metrics(
deltas: dict[str, Any] = {}
for label, ref_correct, ref_t, chal_correct, chal_t, both_active in (
- ("surfsense_vs_bare", bare_correct, bare_t, surf_correct, surf_t,
- arms_active.get("bare_llm") and arms_active.get("surfsense")),
- ("surfsense_vs_long_context", lc_correct, lc_t, surf_correct, surf_t,
- arms_active.get("long_context") and arms_active.get("surfsense")),
- ("long_context_vs_bare", bare_correct, bare_t, lc_correct, lc_t,
- arms_active.get("bare_llm") and arms_active.get("long_context")),
+ (
+ "surfsense_vs_bare",
+ bare_correct,
+ bare_t,
+ surf_correct,
+ surf_t,
+ arms_active.get("bare_llm") and arms_active.get("surfsense"),
+ ),
+ (
+ "surfsense_vs_long_context",
+ lc_correct,
+ lc_t,
+ surf_correct,
+ surf_t,
+ arms_active.get("long_context") and arms_active.get("surfsense"),
+ ),
+ (
+ "long_context_vs_bare",
+ bare_correct,
+ bare_t,
+ lc_correct,
+ lc_t,
+ arms_active.get("bare_llm") and arms_active.get("long_context"),
+ ),
):
if not both_active:
continue
mc = mcnemar_test(ref_correct, chal_correct)
boot = bootstrap_delta_ci(ref_correct, chal_correct, n_resamples=2000)
deltas[label] = {
- "accuracy_pp": 100.0 * (sum(chal_correct) - sum(ref_correct)) / max(1, len(chal_correct)),
- "truthfulness_score_pp": 100.0 * (chal_t["truthfulness_score"] - ref_t["truthfulness_score"]),
+ "accuracy_pp": 100.0
+ * (sum(chal_correct) - sum(ref_correct))
+ / max(1, len(chal_correct)),
+ "truthfulness_score_pp": 100.0
+ * (chal_t["truthfulness_score"] - ref_t["truthfulness_score"]),
"mcnemar_p_value": mc.p_value,
"mcnemar_method": mc.method,
"mcnemar_b_ref_only": mc.b,
@@ -800,12 +903,18 @@ def _compute_metrics(
out["deltas"] = deltas
out["per_domain"] = _per_facet_truthfulness(
- questions, bare_grades, lc_grades, surf_grades,
+ questions,
+ bare_grades,
+ lc_grades,
+ surf_grades,
arms_active=arms_active,
key_fn=lambda q: q.domain or "(unspecified)",
)
out["per_question_type"] = _per_facet_truthfulness(
- questions, bare_grades, lc_grades, surf_grades,
+ questions,
+ bare_grades,
+ lc_grades,
+ surf_grades,
arms_active=arms_active,
key_fn=lambda q: q.question_type or "(unspecified)",
)
@@ -830,11 +939,11 @@ def _per_facet_truthfulness(
"""Bucket truthfulness scores by ``key_fn(q)``."""
buckets: dict[str, dict[str, list[CragGradeResult]]] = {}
- for q, b, l, s in zip(questions, bare_grades, lc_grades, surf_grades, strict=False):
+ for q, b, lc, s in zip(questions, bare_grades, lc_grades, surf_grades, strict=False):
key = key_fn(q)
bucket = buckets.setdefault(key, {"bare_llm": [], "long_context": [], "surfsense": []})
bucket["bare_llm"].append(b)
- bucket["long_context"].append(l)
+ bucket["long_context"].append(lc)
bucket["surfsense"].append(s)
out: dict[str, Any] = {}
for key, arms in buckets.items():
@@ -867,11 +976,11 @@ def _arm_summary_lines(d: dict[str, Any], *, indent: str) -> str:
high = d.get("ci_high", 0.0)
lines = [
f"{indent}- Accuracy: {acc * 100:.1f}% (Wilson 95% CI: {low * 100:.1f}% – {high * 100:.1f}%)",
- f"{indent}- 3-class: correct={d.get('correct_rate', 0)*100:.1f}%, "
- f"missing={d.get('missing_rate', 0)*100:.1f}%, "
- f"incorrect={d.get('incorrect_rate', 0)*100:.1f}%",
+ f"{indent}- 3-class: correct={d.get('correct_rate', 0) * 100:.1f}%, "
+ f"missing={d.get('missing_rate', 0) * 100:.1f}%, "
+ f"incorrect={d.get('incorrect_rate', 0) * 100:.1f}%",
f"{indent}- Truthfulness score (correct - incorrect)/total: "
- f"{d.get('truthfulness_score', 0)*100:+.1f}%",
+ f"{d.get('truthfulness_score', 0) * 100:+.1f}%",
f"{indent}- Cost / question: ${_dollars(d.get('cost_micros_mean'))} (mean), "
f"${_dollars(d.get('cost_micros_median'))} (median)",
f"{indent}- Latency: p50 {_ms_to_s(d.get('latency_ms_median'))}, "
@@ -916,7 +1025,7 @@ def _pct(value: Any) -> str:
if value is None:
return "?"
try:
- return f"{float(value)*100:.1f}%"
+ return f"{float(value) * 100:.1f}%"
except (TypeError, ValueError):
return "?"
@@ -925,7 +1034,7 @@ def _signed_pct(value: Any) -> str:
if value is None:
return "?"
try:
- return f"{float(value)*100:+.1f}%"
+ return f"{float(value) * 100:+.1f}%"
except (TypeError, ValueError):
return "?"
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/frames/dataset.py b/surfsense_evals/src/surfsense_evals/suites/research/frames/dataset.py
index c3b6b878e..80c7075f8 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/frames/dataset.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/frames/dataset.py
@@ -51,12 +51,12 @@ def _hf_hub_download(*args: Any, **kwargs: Any) -> str:
class FramesQuestion:
"""One row of FRAMES (post-parse)."""
- qid: str # synthesised "Q000" .. "Q823"
+ qid: str # synthesised "Q000" .. "Q823"
question: str
gold_answer: str
- wiki_urls: list[str] # deduped, in original order
- reasoning_types: list[str] # split on "|"
- raw_index: int # row index from the TSV (for debugging)
+ wiki_urls: list[str] # deduped, in original order
+ reasoning_types: list[str] # split on "|"
+ raw_index: int # row index from the TSV (for debugging)
def to_dict(self) -> dict[str, Any]:
return {
@@ -102,7 +102,7 @@ def _parse_wiki_links(raw: Any) -> list[str]:
except (SyntaxError, ValueError):
# Fall back: maybe it's a comma-separated string with no quotes.
return [tok.strip() for tok in text.strip("[]").split(",") if tok.strip()]
- if isinstance(parsed, (list, tuple)):
+ if isinstance(parsed, list | tuple):
return [str(x).strip() for x in parsed if str(x).strip()]
return [str(parsed).strip()]
@@ -146,14 +146,16 @@ def load_questions(tsv_path: Path) -> list[FramesQuestion]:
if val and val not in urls:
urls.append(val)
reasoning = _parse_reasoning_types(row.get("reasoning_types"))
- out.append(FramesQuestion(
- qid=f"Q{int(raw_idx):03d}",
- question=prompt,
- gold_answer=answer,
- wiki_urls=urls,
- reasoning_types=reasoning,
- raw_index=int(raw_idx),
- ))
+ out.append(
+ FramesQuestion(
+ qid=f"Q{int(raw_idx):03d}",
+ question=prompt,
+ gold_answer=answer,
+ wiki_urls=urls,
+ reasoning_types=reasoning,
+ raw_index=int(raw_idx),
+ )
+ )
return out
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/frames/grader.py b/surfsense_evals/src/surfsense_evals/suites/research/frames/grader.py
index d280e3eaf..32343de42 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/frames/grader.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/frames/grader.py
@@ -90,10 +90,27 @@ def _normalise(s: str) -> str:
_WORD_NUMBERS = {
- "zero": 0, "one": 1, "two": 2, "three": 3, "four": 4, "five": 5,
- "six": 6, "seven": 7, "eight": 8, "nine": 9, "ten": 10, "eleven": 11,
- "twelve": 12, "thirteen": 13, "fourteen": 14, "fifteen": 15, "sixteen": 16,
- "seventeen": 17, "eighteen": 18, "nineteen": 19, "twenty": 20,
+ "zero": 0,
+ "one": 1,
+ "two": 2,
+ "three": 3,
+ "four": 4,
+ "five": 5,
+ "six": 6,
+ "seven": 7,
+ "eight": 8,
+ "nine": 9,
+ "ten": 10,
+ "eleven": 11,
+ "twelve": 12,
+ "thirteen": 13,
+ "fourteen": 14,
+ "fifteen": 15,
+ "sixteen": 16,
+ "seventeen": 17,
+ "eighteen": 18,
+ "nineteen": 19,
+ "twenty": 20,
}
_NUMERIC_RE = re.compile(r"-?\d+(?:[.,]\d+)?")
@@ -194,7 +211,7 @@ _JUDGE_SYSTEM = (
"expresses a different fact, omits the central answer, or hedges "
"without committing.\n\n"
"Respond with ONLY a JSON object on a single line:\n"
- '{\"correct\": true|false, \"rationale\": \"\"}'
+ '{"correct": true|false, "rationale": ""}'
)
@@ -324,10 +341,7 @@ async def grade_many(
if not rows:
return []
- coros = [
- grade_with_judge(pred=p, gold=g, question=q, judge=judge)
- for _qid, q, g, p in rows
- ]
+ coros = [grade_with_judge(pred=p, gold=g, question=q, judge=judge) for _qid, q, g, p in rows]
return list(await asyncio.gather(*coros))
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/frames/ingest.py b/surfsense_evals/src/surfsense_evals/suites/research/frames/ingest.py
index 98e035f28..3ea7246a6 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/frames/ingest.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/frames/ingest.py
@@ -34,7 +34,6 @@ filename (without extension), so we round-trip via
from __future__ import annotations
-import asyncio
import json
import logging
from dataclasses import dataclass
@@ -161,8 +160,10 @@ async def _upload_markdowns(
name_to_id[s.title] = s.document_id
logger.info(
"FRAMES upload batch %d-%d: %d new, %d duplicate",
- batch_start, batch_start + len(batch),
- len(result.document_ids), len(result.duplicate_document_ids),
+ batch_start,
+ batch_start + len(batch),
+ len(result.document_ids),
+ len(result.duplicate_document_ids),
)
return name_to_id
@@ -189,14 +190,16 @@ def _resolve_question_doc_ids(
doc_id = name_to_id.get(stem) or name_to_id.get(article.markdown_path.name)
if doc_id is not None and doc_id not in doc_ids:
doc_ids.append(doc_id)
- rows.append({
- "qid": q.qid,
- "raw_index": q.raw_index,
- "n_wiki_urls": len(q.wiki_urls),
- "wiki_titles": titles,
- "document_ids": doc_ids,
- "missing_urls": missing,
- })
+ rows.append(
+ {
+ "qid": q.qid,
+ "raw_index": q.raw_index,
+ "n_wiki_urls": len(q.wiki_urls),
+ "wiki_titles": titles,
+ "document_ids": doc_ids,
+ "missing_urls": missing,
+ }
+ )
return rows
@@ -239,7 +242,7 @@ async def run_ingest(
settings = settings or IngestSettings(
use_vision_llm=False,
processing_mode="basic",
- )
+ )
bench_dir = ctx.benchmark_data_dir()
wiki_cache = bench_dir / "wiki"
wiki_cache.mkdir(parents=True, exist_ok=True)
@@ -251,8 +254,7 @@ async def run_ingest(
questions = load_questions(tsv_path)
if not questions:
raise RuntimeError(
- "FRAMES test.tsv contained no parseable rows; upstream may "
- "have changed schema."
+ "FRAMES test.tsv contained no parseable rows; upstream may have changed schema."
)
logger.info("FRAMES: parsed %d questions from %s", len(questions), tsv_path.name)
if max_questions is not None and max_questions > 0:
@@ -270,19 +272,23 @@ async def run_ingest(
unique_urls = list(seen_urls.keys())
logger.info(
"FRAMES: %d unique Wikipedia URLs across %d questions",
- len(unique_urls), len(questions),
+ len(unique_urls),
+ len(questions),
)
# 3. Fetch (with cache).
fetcher = WikiFetcher(cache_dir=wiki_cache, rate_limit_rps=fetch_rate_limit_rps)
n_cached = sum(
- 1 for url in unique_urls
+ 1
+ for url in unique_urls
if (wiki_cache / cache_filename_for_title(_safe_title(url))).exists()
)
fetched, missing_urls = await _fetch_articles(fetcher, unique_urls)
logger.info(
"FRAMES: fetched=%d, cache_hits=%d, missing=%d",
- len(fetched), n_cached, len(missing_urls),
+ len(fetched),
+ n_cached,
+ len(missing_urls),
)
# 4. Upload to SurfSense (deduped by filename).
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/frames/prompt.py b/surfsense_evals/src/surfsense_evals/suites/research/frames/prompt.py
index 16bb06da4..8f8b6e3b6 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/frames/prompt.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/frames/prompt.py
@@ -17,7 +17,6 @@ Format expectations (mirrors the FRAMES paper, section 4):
from __future__ import annotations
-
_BASE_INSTRUCTIONS = (
"You are a careful question-answering assistant. The question may "
"require combining facts from multiple sources, doing arithmetic, "
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/frames/runner.py b/surfsense_evals/src/surfsense_evals/suites/research/frames/runner.py
index 450c7ddd6..c703ab23e 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/frames/runner.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/frames/runner.py
@@ -65,7 +65,7 @@ class FramesRunnerQuestion:
question: str
gold_answer: str
reasoning_types: list[str]
- document_ids: list[int] # subset of corpus relevant to this Q (may be empty)
+ document_ids: list[int] # subset of corpus relevant to this Q (may be empty)
n_wiki_urls: int
missing_urls: list[str]
@@ -107,16 +107,18 @@ def _load_questions(
reasoning = list(row.get("reasoning_types") or [])
if reasoning_filter and reasoning_filter not in [r.lower() for r in reasoning]:
continue
- out.append(FramesRunnerQuestion(
- qid=qid,
- raw_index=int(row.get("raw_index") or 0),
- question=str(row.get("question") or "").strip(),
- gold_answer=str(row.get("gold_answer") or "").strip(),
- reasoning_types=reasoning,
- document_ids=list(map_row.get("document_ids") or []),
- n_wiki_urls=int(map_row.get("n_wiki_urls") or 0),
- missing_urls=list(map_row.get("missing_urls") or []),
- ))
+ out.append(
+ FramesRunnerQuestion(
+ qid=qid,
+ raw_index=int(row.get("raw_index") or 0),
+ question=str(row.get("question") or "").strip(),
+ gold_answer=str(row.get("gold_answer") or "").strip(),
+ reasoning_types=reasoning,
+ document_ids=list(map_row.get("document_ids") or []),
+ n_wiki_urls=int(map_row.get("n_wiki_urls") or 0),
+ missing_urls=list(map_row.get("missing_urls") or []),
+ )
+ )
out.sort(key=lambda q: q.raw_index)
if sample_n is not None and sample_n > 0:
out = out[:sample_n]
@@ -166,7 +168,10 @@ class FramesBenchmark:
def add_run_args(self, parser: argparse.ArgumentParser) -> None:
parser.add_argument(
- "--n", dest="sample_n", type=int, default=None,
+ "--n",
+ dest="sample_n",
+ type=int,
+ default=None,
help="Run only the first N questions after filters (default: all 824).",
)
parser.add_argument(
@@ -180,11 +185,15 @@ class FramesBenchmark:
),
)
parser.add_argument(
- "--concurrency", type=int, default=4,
+ "--concurrency",
+ type=int,
+ default=4,
help="Parallel question workers per arm.",
)
parser.add_argument(
- "--scope-mentions", dest="scope_mentions", action="store_true",
+ "--scope-mentions",
+ dest="scope_mentions",
+ action="store_true",
help=(
"SurfSense arm: scope retrieval to the per-question "
"document_ids (oracle-retrieval upper bound). Default "
@@ -192,11 +201,15 @@ class FramesBenchmark:
),
)
parser.add_argument(
- "--max-output-tokens", type=int, default=512,
+ "--max-output-tokens",
+ type=int,
+ default=512,
help="Cap on completion length for both arms.",
)
parser.add_argument(
- "--no-judge", dest="no_judge", action="store_true",
+ "--no-judge",
+ dest="no_judge",
+ action="store_true",
help=(
"Disable LLM-as-judge fallback grading; use only the "
"deterministic grader (faster but more pessimistic)."
@@ -217,19 +230,30 @@ class FramesBenchmark:
)
# Ingest-only knobs.
parser.add_argument(
- "--max-questions", dest="max_questions", type=int, default=None,
+ "--max-questions",
+ dest="max_questions",
+ type=int,
+ default=None,
help="(ingest only) cap on number of questions to materialise + ingest.",
)
parser.add_argument(
- "--upload-batch-size", dest="upload_batch_size", type=int, default=16,
+ "--upload-batch-size",
+ dest="upload_batch_size",
+ type=int,
+ default=16,
help="(ingest only) markdown files per fileupload call.",
)
parser.add_argument(
- "--skip-upload", dest="skip_upload", action="store_true",
+ "--skip-upload",
+ dest="skip_upload",
+ action="store_true",
help="(ingest only) cache wiki articles locally but don't push to SurfSense.",
)
parser.add_argument(
- "--fetch-rps", dest="fetch_rate_limit_rps", type=float, default=2.0,
+ "--fetch-rps",
+ dest="fetch_rate_limit_rps",
+ type=float,
+ default=2.0,
help="(ingest only) max requests/second to the Wikipedia API.",
)
add_ingest_settings_args(parser, defaults=_DEFAULT_INGEST_SETTINGS)
@@ -270,21 +294,18 @@ class FramesBenchmark:
doc_map, ingest_settings = _load_doc_map(map_path)
questions = _load_questions(
- questions_jsonl, doc_map,
+ questions_jsonl,
+ doc_map,
sample_n=sample_n,
reasoning_filter=reasoning_filter,
)
if not questions:
- raise RuntimeError(
- "No FRAMES questions matched the filters; broaden --reasoning/--n."
- )
+ raise RuntimeError("No FRAMES questions matched the filters; broaden --reasoning/--n.")
logger.info("FRAMES: scheduled %d questions", len(questions))
api_key = os.environ.get("OPENROUTER_API_KEY")
if not api_key:
- raise RuntimeError(
- "OPENROUTER_API_KEY env var is required for the bare-LLM arm."
- )
+ raise RuntimeError("OPENROUTER_API_KEY env var is required for the bare-LLM arm.")
bare_provider = OpenRouterChatProvider(
api_key=api_key,
@@ -303,12 +324,14 @@ class FramesBenchmark:
judge: LlmJudge | None = None
if not no_judge:
- judge = LlmJudge(config=JudgeConfig(
- api_key=api_key,
- model=judge_model,
- base_url=ctx.config.openrouter_base_url,
- concurrency=judge_concurrency,
- ))
+ judge = LlmJudge(
+ config=JudgeConfig(
+ api_key=api_key,
+ model=judge_model,
+ base_url=ctx.config.openrouter_base_url,
+ concurrency=judge_concurrency,
+ )
+ )
run_timestamp = utc_iso_timestamp()
run_dir = ctx.runs_dir(run_timestamp=run_timestamp)
@@ -318,9 +341,7 @@ class FramesBenchmark:
return await bare_arm.answer(_make_bare_request(q, max_output_tokens))
async def _surf_one(q: FramesRunnerQuestion) -> ArmResult:
- return await surf_arm.answer(
- _make_surfsense_request(q, scope_mentions=scope_mentions)
- )
+ return await surf_arm.answer(_make_surfsense_request(q, scope_mentions=scope_mentions))
bare_results, surf_results = await asyncio.gather(
_gather_with_limit((_bare_one(q) for q in questions), concurrency=concurrency),
@@ -343,16 +364,26 @@ class FramesBenchmark:
"n_missing_urls": len(q.missing_urls),
"gold": q.gold_answer,
}
- fh.write(json.dumps({
- **meta,
- **b_res.to_jsonl(),
- "graded": b_g.to_dict(),
- }) + "\n")
- fh.write(json.dumps({
- **meta,
- **s_res.to_jsonl(),
- "graded": s_g.to_dict(),
- }) + "\n")
+ fh.write(
+ json.dumps(
+ {
+ **meta,
+ **b_res.to_jsonl(),
+ "graded": b_g.to_dict(),
+ }
+ )
+ + "\n"
+ )
+ fh.write(
+ json.dumps(
+ {
+ **meta,
+ **s_res.to_jsonl(),
+ "graded": s_g.to_dict(),
+ }
+ )
+ + "\n"
+ )
metrics = _compute_metrics(questions, bare_results, surf_results, bare_grades, surf_grades)
artifact = RunArtifact(
@@ -380,13 +411,18 @@ class FramesBenchmark:
manifest_path = run_dir / "run_artifact.json"
manifest_path.write_text(
- json.dumps({
- "suite": self.suite,
- "benchmark": self.name,
- "raw_path": "raw.jsonl",
- "metrics": metrics,
- "extra": artifact.extra,
- }, indent=2, sort_keys=True) + "\n",
+ json.dumps(
+ {
+ "suite": self.suite,
+ "benchmark": self.name,
+ "raw_path": "raw.jsonl",
+ "metrics": metrics,
+ "extra": artifact.extra,
+ },
+ indent=2,
+ sort_keys=True,
+ )
+ + "\n",
encoding="utf-8",
)
return artifact
@@ -451,8 +487,8 @@ class FramesBenchmark:
for tag, vals in sorted(per_reasoning.items()):
body_lines.append(
f" - {tag}: SurfSense {_pp(vals.get('delta_accuracy_pp'))} pp "
- f"(n={vals.get('n')}, bare acc={vals.get('bare_accuracy', 0)*100:.1f}%, "
- f"surf acc={vals.get('surfsense_accuracy', 0)*100:.1f}%)"
+ f"(n={vals.get('n')}, bare acc={vals.get('bare_accuracy', 0) * 100:.1f}%, "
+ f"surf acc={vals.get('surfsense_accuracy', 0) * 100:.1f}%)"
)
return ReportSection(
@@ -553,8 +589,7 @@ def _compute_metrics(
"bare_accuracy": (sum(b_correct) / len(pairs)) if pairs else 0.0,
"surfsense_accuracy": (sum(s_correct) / len(pairs)) if pairs else 0.0,
"delta_accuracy_pp": (
- 100.0 * (sum(s_correct) - sum(b_correct)) / len(pairs)
- if pairs else 0.0
+ 100.0 * (sum(s_correct) - sum(b_correct)) / len(pairs) if pairs else 0.0
),
}
@@ -571,8 +606,12 @@ def _compute_metrics(
"latency_ms_mean": bare_latency_agg.mean,
"latency_ms_median": bare_latency_agg.median,
"latency_ms_p95": bare_latency_agg.p95,
- "input_tokens_mean": (sum(bare_in_tokens) / len(bare_in_tokens)) if bare_in_tokens else 0.0,
- "output_tokens_mean": (sum(bare_out_tokens) / len(bare_out_tokens)) if bare_out_tokens else 0.0,
+ "input_tokens_mean": (sum(bare_in_tokens) / len(bare_in_tokens))
+ if bare_in_tokens
+ else 0.0,
+ "output_tokens_mean": (sum(bare_out_tokens) / len(bare_out_tokens))
+ if bare_out_tokens
+ else 0.0,
},
"surfsense": {
**surf_acc.to_dict(),
diff --git a/surfsense_evals/src/surfsense_evals/suites/research/frames/wiki_fetch.py b/surfsense_evals/src/surfsense_evals/suites/research/frames/wiki_fetch.py
index 7f6b63e50..2bc96ad3a 100644
--- a/surfsense_evals/src/surfsense_evals/suites/research/frames/wiki_fetch.py
+++ b/surfsense_evals/src/surfsense_evals/suites/research/frames/wiki_fetch.py
@@ -49,10 +49,10 @@ USER_AGENT = (
class WikiArticle:
"""One fetched article + metadata."""
- title: str # canonical title returned by MW (post-redirect)
- source_url: str # the URL we were asked to fetch
- markdown_path: Path # where the cached body lives on disk
- n_chars: int # length of the body (post-prepend H1)
+ title: str # canonical title returned by MW (post-redirect)
+ source_url: str # the URL we were asked to fetch
+ markdown_path: Path # where the cached body lives on disk
+ n_chars: int # length of the body (post-prepend H1)
redirected_from: str | None = None
@@ -168,10 +168,13 @@ class WikiFetcher:
break
except (httpx.HTTPError, RuntimeError) as exc:
last_exc = exc
- wait = 1.0 * (2 ** attempt)
+ wait = 1.0 * (2**attempt)
logger.warning(
"wiki fetch %r attempt %d failed: %s; retry in %.1fs",
- title, attempt + 1, exc, wait,
+ title,
+ attempt + 1,
+ exc,
+ wait,
)
await asyncio.sleep(wait)
else:
@@ -217,10 +220,14 @@ class WikiFetcher:
}
headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
if http is not None:
- response = await http.get(WIKI_API, params=params, headers=headers, timeout=self._timeout)
+ response = await http.get(
+ WIKI_API, params=params, headers=headers, timeout=self._timeout
+ )
else:
async with httpx.AsyncClient(timeout=self._timeout) as client:
- response = await client.get(WIKI_API, params=params, headers=headers, timeout=self._timeout)
+ response = await client.get(
+ WIKI_API, params=params, headers=headers, timeout=self._timeout
+ )
response.raise_for_status()
data = response.json()
if "error" in data:
diff --git a/surfsense_evals/tests/core/test_auth.py b/surfsense_evals/tests/core/test_auth.py
index 181d8e632..ad291b295 100644
--- a/surfsense_evals/tests/core/test_auth.py
+++ b/surfsense_evals/tests/core/test_auth.py
@@ -52,9 +52,7 @@ async def test_acquire_token_local_mode_posts_desktop_login_json():
200, json={"access_token": "T", "refresh_token": "R", "token_type": "bearer"}
)
)
- config = _make_config(
- surfsense_user_email="u@example.com", surfsense_user_password="pw"
- )
+ config = _make_config(surfsense_user_email="u@example.com", surfsense_user_password="pw")
bundle = await acquire_token(config)
assert bundle.access_token == "T"
assert bundle.refresh_token == "R"
diff --git a/surfsense_evals/tests/core/test_clients.py b/surfsense_evals/tests/core/test_clients.py
index aa98f0ad4..44b78e3ba 100644
--- a/surfsense_evals/tests/core/test_clients.py
+++ b/surfsense_evals/tests/core/test_clients.py
@@ -94,10 +94,18 @@ async def test_documents_status_parses_state(respx_mock, http):
200,
json={
"items": [
- {"id": 1, "title": "a.pdf", "document_type": "FILE",
- "status": {"state": "ready", "reason": None}},
- {"id": 2, "title": "b.pdf", "document_type": "FILE",
- "status": {"state": "failed", "reason": "ETL boom"}},
+ {
+ "id": 1,
+ "title": "a.pdf",
+ "document_type": "FILE",
+ "status": {"state": "ready", "reason": None},
+ },
+ {
+ "id": 2,
+ "title": "b.pdf",
+ "document_type": "FILE",
+ "status": {"state": "failed", "reason": "ETL boom"},
+ },
]
},
)
@@ -137,14 +145,26 @@ async def test_documents_upload_returns_payload(respx_mock, http, tmp_path: Path
async def test_documents_list_chunks_paginated(respx_mock, http):
respx_mock.get("/api/v1/documents/5/chunks").mock(
side_effect=[
- httpx.Response(200, json={
- "items": [{"id": 1, "content": "a"}, {"id": 2, "content": "b"}],
- "total": 3, "page": 0, "page_size": 2, "has_more": True,
- }),
- httpx.Response(200, json={
- "items": [{"id": 3, "content": "c"}],
- "total": 3, "page": 1, "page_size": 2, "has_more": False,
- }),
+ httpx.Response(
+ 200,
+ json={
+ "items": [{"id": 1, "content": "a"}, {"id": 2, "content": "b"}],
+ "total": 3,
+ "page": 0,
+ "page_size": 2,
+ "has_more": True,
+ },
+ ),
+ httpx.Response(
+ 200,
+ json={
+ "items": [{"id": 3, "content": "c"}],
+ "total": 3,
+ "page": 1,
+ "page_size": 2,
+ "has_more": False,
+ },
+ ),
]
)
client = DocumentsClient(http, _BASE)
@@ -191,15 +211,17 @@ def _sse_body(events: list[dict]) -> bytes:
@pytest.mark.asyncio
@respx.mock(base_url=_BASE)
async def test_ask_accumulates_text_deltas(respx_mock, http):
- body = _sse_body([
- {"type": "start", "messageId": "m1"},
- {"type": "text-start", "id": "t1"},
- {"type": "text-delta", "id": "t1", "delta": "Answer "},
- {"type": "text-delta", "id": "t1", "delta": "is "},
- {"type": "text-delta", "id": "t1", "delta": "B [citation:42]."},
- {"type": "text-end", "id": "t1"},
- {"type": "finish"},
- ])
+ body = _sse_body(
+ [
+ {"type": "start", "messageId": "m1"},
+ {"type": "text-start", "id": "t1"},
+ {"type": "text-delta", "id": "t1", "delta": "Answer "},
+ {"type": "text-delta", "id": "t1", "delta": "is "},
+ {"type": "text-delta", "id": "t1", "delta": "B [citation:42]."},
+ {"type": "text-end", "id": "t1"},
+ {"type": "finish"},
+ ]
+ )
respx_mock.post("/api/v1/new_chat").mock(
return_value=httpx.Response(
200,
@@ -208,9 +230,7 @@ async def test_ask_accumulates_text_deltas(respx_mock, http):
)
)
client = NewChatClient(http, _BASE)
- answer = await client.ask(
- thread_id=1, search_space_id=2, user_query="What is the answer?"
- )
+ answer = await client.ask(thread_id=1, search_space_id=2, user_query="What is the answer?")
assert answer.text == "Answer is B [citation:42]."
assert answer.finished_normally is True
assert any(c["chunk_id"] == 42 for c in answer.citations)
@@ -219,23 +239,21 @@ async def test_ask_accumulates_text_deltas(respx_mock, http):
@pytest.mark.asyncio
@respx.mock(base_url=_BASE)
async def test_ask_409_thread_busy_retries(respx_mock, http):
- body = _sse_body([
- {"type": "text-delta", "id": "t1", "delta": "ok"},
- {"type": "finish"},
- ])
+ body = _sse_body(
+ [
+ {"type": "text-delta", "id": "t1", "delta": "ok"},
+ {"type": "finish"},
+ ]
+ )
busy = httpx.Response(
409,
json={"detail": {"errorCode": "THREAD_BUSY", "message": "busy"}},
headers={"Retry-After": "1"},
)
- success = httpx.Response(
- 200, content=body, headers={"Content-Type": "text/event-stream"}
- )
+ success = httpx.Response(200, content=body, headers={"Content-Type": "text/event-stream"})
respx_mock.post("/api/v1/new_chat").mock(side_effect=[busy, success])
client = NewChatClient(http, _BASE)
- answer = await client.ask(
- thread_id=1, search_space_id=2, user_query="hi", max_busy_retries=2
- )
+ answer = await client.ask(thread_id=1, search_space_id=2, user_query="hi", max_busy_retries=2)
assert answer.text == "ok"
@@ -250,6 +268,4 @@ async def test_ask_409_exhausts_retries(respx_mock, http):
respx_mock.post("/api/v1/new_chat").mock(return_value=busy)
client = NewChatClient(http, _BASE)
with pytest.raises(ThreadBusyError):
- await client.ask(
- thread_id=1, search_space_id=2, user_query="hi", max_busy_retries=1
- )
+ await client.ask(thread_id=1, search_space_id=2, user_query="hi", max_busy_retries=1)
diff --git a/surfsense_evals/tests/core/test_ingest_settings.py b/surfsense_evals/tests/core/test_ingest_settings.py
index fd7e7818a..cd1d1f827 100644
--- a/surfsense_evals/tests/core/test_ingest_settings.py
+++ b/surfsense_evals/tests/core/test_ingest_settings.py
@@ -46,32 +46,24 @@ class TestMerge:
def test_explicit_false_overrides_default_true(self) -> None:
defaults = IngestSettings(use_vision_llm=True)
- merged = IngestSettings.merge(
- defaults, {"use_vision_llm": False}
- )
+ merged = IngestSettings.merge(defaults, {"use_vision_llm": False})
assert merged.use_vision_llm is False
def test_explicit_true_overrides_default_false(self) -> None:
defaults = IngestSettings(use_vision_llm=False)
- merged = IngestSettings.merge(
- defaults, {"use_vision_llm": True}
- )
+ merged = IngestSettings.merge(defaults, {"use_vision_llm": True})
assert merged.use_vision_llm is True
def test_none_means_silent(self) -> None:
# Argparse with BooleanOptionalAction yields None when the
# operator passed neither --use-vision-llm nor --no-vision-llm.
defaults = IngestSettings(use_vision_llm=True)
- merged = IngestSettings.merge(
- defaults, {"use_vision_llm": None}
- )
+ merged = IngestSettings.merge(defaults, {"use_vision_llm": None})
assert merged.use_vision_llm is True
def test_processing_mode_override(self) -> None:
defaults = IngestSettings(processing_mode="basic")
- merged = IngestSettings.merge(
- defaults, {"processing_mode": "premium"}
- )
+ merged = IngestSettings.merge(defaults, {"processing_mode": "premium"})
assert merged.processing_mode == "premium"
def test_processing_mode_invalid_raises(self) -> None:
@@ -134,9 +126,7 @@ class TestAddArgs:
p = argparse.ArgumentParser()
add_ingest_settings_args(
p,
- defaults=IngestSettings(
- use_vision_llm=False, processing_mode="basic"
- ),
+ defaults=IngestSettings(use_vision_llm=False, processing_mode="basic"),
)
return p
@@ -158,31 +148,21 @@ class TestAddArgs:
args = parser.parse_args(["--processing-mode", mode])
assert args.processing_mode == mode
- def test_processing_mode_rejects_unknown(
- self, parser: argparse.ArgumentParser
- ) -> None:
+ def test_processing_mode_rejects_unknown(self, parser: argparse.ArgumentParser) -> None:
with pytest.raises(SystemExit):
parser.parse_args(["--processing-mode", "exotic"])
- def test_vision_flags_mutually_exclusive(
- self, parser: argparse.ArgumentParser
- ) -> None:
+ def test_vision_flags_mutually_exclusive(self, parser: argparse.ArgumentParser) -> None:
with pytest.raises(SystemExit):
parser.parse_args(["--use-vision-llm", "--no-vision-llm"])
def test_full_pipeline(self, parser: argparse.ArgumentParser) -> None:
# Operator passes flags + defaults are reasonable. Merge
# should yield exactly what they asked for.
- args = parser.parse_args(
- ["--use-vision-llm", "--processing-mode", "premium"]
- )
- defaults = IngestSettings(
- use_vision_llm=False, processing_mode="basic"
- )
+ args = parser.parse_args(["--use-vision-llm", "--processing-mode", "premium"])
+ defaults = IngestSettings(use_vision_llm=False, processing_mode="basic")
merged = IngestSettings.merge(defaults, vars(args))
- assert merged == IngestSettings(
- use_vision_llm=True, processing_mode="premium"
- )
+ assert merged == IngestSettings(use_vision_llm=True, processing_mode="premium")
# ---------------------------------------------------------------------------
@@ -240,16 +220,12 @@ class TestHeader:
class TestFormatMd:
def test_full_settings(self) -> None:
- out = format_ingest_settings_md(
- {"use_vision_llm": True, "processing_mode": "premium"}
- )
+ out = format_ingest_settings_md({"use_vision_llm": True, "processing_mode": "premium"})
assert "vision_llm=`on`" in out
assert "processing_mode=`premium`" in out
def test_default_off(self) -> None:
- out = format_ingest_settings_md(
- {"use_vision_llm": False, "processing_mode": "basic"}
- )
+ out = format_ingest_settings_md({"use_vision_llm": False, "processing_mode": "basic"})
assert "vision_llm=`off`" in out
assert "processing_mode=`basic`" in out
diff --git a/surfsense_evals/tests/core/test_metrics.py b/surfsense_evals/tests/core/test_metrics.py
index cde1bb957..73c85c371 100644
--- a/surfsense_evals/tests/core/test_metrics.py
+++ b/surfsense_evals/tests/core/test_metrics.py
@@ -25,7 +25,12 @@ from surfsense_evals.core.metrics import (
@pytest.mark.parametrize(
"k,n,low,high",
[
- (80, 100, 0.7111, 0.8666), # cross-checked vs statsmodels.proportion_confint(method='wilson')
+ (
+ 80,
+ 100,
+ 0.7111,
+ 0.8666,
+ ), # cross-checked vs statsmodels.proportion_confint(method='wilson')
(50, 100, 0.4038, 0.5962),
(0, 0, 0.0, 1.0),
(0, 10, 0.0, 0.2775),
@@ -74,7 +79,7 @@ def test_mcnemar_exact_branch_strong_signal():
assert res.b == 0
assert res.c == 10
assert res.method == "exact"
- expected = 2 * (0.5 ** 10)
+ expected = 2 * (0.5**10)
assert math.isclose(res.p_value, expected, rel_tol=1e-9)
diff --git a/surfsense_evals/tests/core/test_parse_answer_letter.py b/surfsense_evals/tests/core/test_parse_answer_letter.py
index 5adbf4bc3..0662adba3 100644
--- a/surfsense_evals/tests/core/test_parse_answer_letter.py
+++ b/surfsense_evals/tests/core/test_parse_answer_letter.py
@@ -11,7 +11,11 @@ from surfsense_evals.core.parse.answer_letter import AnswerLetterResult
@pytest.mark.parametrize(
"text,expected_letter,expected_strategy",
[
- ('```json\n{"step_by_step_thinking": "...", "answer_choice": "B"}\n```', "B", "json_envelope"),
+ (
+ '```json\n{"step_by_step_thinking": "...", "answer_choice": "B"}\n```',
+ "B",
+ "json_envelope",
+ ),
('Reasoning... {"step_by_step_thinking": "x", "answer_choice": "C"}', "C", "json_envelope"),
("Long reasoning.\nAnswer: D", "D", "answer_line"),
("The correct answer is (A).", "A", "answer_line"),
diff --git a/surfsense_evals/tests/core/test_parse_citations.py b/surfsense_evals/tests/core/test_parse_citations.py
index eb444dab2..488af590d 100644
--- a/surfsense_evals/tests/core/test_parse_citations.py
+++ b/surfsense_evals/tests/core/test_parse_citations.py
@@ -91,7 +91,7 @@ def test_regex_pattern_matches_ts_source():
assert "https?://" in pattern
assert "urlcite" in pattern
assert "doc-" in pattern
- assert "\u200B" in pattern
+ assert "\u200b" in pattern
assert "【" in pattern and "】" in pattern
diff --git a/surfsense_evals/tests/core/test_parse_freeform_answer.py b/surfsense_evals/tests/core/test_parse_freeform_answer.py
index bdc7d74fc..a39aad2e5 100644
--- a/surfsense_evals/tests/core/test_parse_freeform_answer.py
+++ b/surfsense_evals/tests/core/test_parse_freeform_answer.py
@@ -44,11 +44,14 @@ class TestExtractFreeformAnswer:
assert extract_freeform_answer("ANSWER: yes") == "yes"
assert extract_freeform_answer("answer: no") == "no"
- @pytest.mark.parametrize("text,expected", [
- ("Answer: 1, 2, 3", "1, 2, 3"),
- ("Answer: 3.14", "3.14"),
- ("Answer: spaced ", "spaced"),
- ])
+ @pytest.mark.parametrize(
+ "text,expected",
+ [
+ ("Answer: 1, 2, 3", "1, 2, 3"),
+ ("Answer: 3.14", "3.14"),
+ ("Answer: spaced ", "spaced"),
+ ],
+ )
def test_various_payloads(self, text: str, expected: str) -> None:
assert extract_freeform_answer(text) == expected
diff --git a/surfsense_evals/tests/core/test_parse_sse.py b/surfsense_evals/tests/core/test_parse_sse.py
index 362717288..10998a881 100644
--- a/surfsense_evals/tests/core/test_parse_sse.py
+++ b/surfsense_evals/tests/core/test_parse_sse.py
@@ -22,12 +22,16 @@ async def _astream(lines):
@pytest.mark.asyncio
async def test_basic_data_frame():
events = await _alist(
- iter_sse_events(_astream([
- 'data: {"type": "text-delta", "delta": "hi"}',
- "",
- 'data: {"type": "finish"}',
- "",
- ]))
+ iter_sse_events(
+ _astream(
+ [
+ 'data: {"type": "text-delta", "delta": "hi"}',
+ "",
+ 'data: {"type": "finish"}',
+ "",
+ ]
+ )
+ )
)
assert [e.data for e in events] == [
'{"type": "text-delta", "delta": "hi"}',
@@ -38,10 +42,14 @@ async def test_basic_data_frame():
@pytest.mark.asyncio
async def test_done_sentinel_passes_through():
events = await _alist(
- iter_sse_events(_astream([
- "data: [DONE]",
- "",
- ]))
+ iter_sse_events(
+ _astream(
+ [
+ "data: [DONE]",
+ "",
+ ]
+ )
+ )
)
assert [e.data for e in events] == ["[DONE]"]
@@ -49,11 +57,15 @@ async def test_done_sentinel_passes_through():
@pytest.mark.asyncio
async def test_multiline_data_joins_with_newline():
events = await _alist(
- iter_sse_events(_astream([
- "data: line1",
- "data: line2",
- "",
- ]))
+ iter_sse_events(
+ _astream(
+ [
+ "data: line1",
+ "data: line2",
+ "",
+ ]
+ )
+ )
)
assert events[0].data == "line1\nline2"
@@ -61,13 +73,17 @@ async def test_multiline_data_joins_with_newline():
@pytest.mark.asyncio
async def test_comments_and_other_fields_ignored():
events = await _alist(
- iter_sse_events(_astream([
- ": heartbeat",
- "event: foo",
- "id: 123",
- "data: payload",
- "",
- ]))
+ iter_sse_events(
+ _astream(
+ [
+ ": heartbeat",
+ "event: foo",
+ "id: 123",
+ "data: payload",
+ "",
+ ]
+ )
+ )
)
assert [e.data for e in events] == ["payload"]
@@ -77,8 +93,12 @@ async def test_handles_missing_trailing_blank():
"""Some servers omit the final blank line; the consumer should still emit."""
events = await _alist(
- iter_sse_events(_astream([
- "data: only-one",
- ]))
+ iter_sse_events(
+ _astream(
+ [
+ "data: only-one",
+ ]
+ )
+ )
)
assert [e.data for e in events] == ["only-one"]
diff --git a/surfsense_evals/tests/core/test_provider_openrouter.py b/surfsense_evals/tests/core/test_provider_openrouter.py
index eb78aa053..aeed6eae5 100644
--- a/surfsense_evals/tests/core/test_provider_openrouter.py
+++ b/surfsense_evals/tests/core/test_provider_openrouter.py
@@ -36,11 +36,18 @@ async def test_payload_shape_matches_openrouter_docs(respx_mock, tiny_pdf: Path)
return httpx.Response(
200,
json={
- "choices": [{
- "message": {"content": "Answer: B"},
- "finish_reason": "stop",
- }],
- "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15, "cost": 0.0001},
+ "choices": [
+ {
+ "message": {"content": "Answer: B"},
+ "finish_reason": "stop",
+ }
+ ],
+ "usage": {
+ "prompt_tokens": 10,
+ "completion_tokens": 5,
+ "total_tokens": 15,
+ "cost": 0.0001,
+ },
},
)
@@ -63,8 +70,7 @@ async def test_payload_shape_matches_openrouter_docs(respx_mock, tiny_pdf: Path)
assert file_part["file"]["filename"] == tiny_pdf.name
assert file_part["file"]["file_data"].startswith("data:application/pdf;base64,")
assert (
- base64.b64decode(file_part["file"]["file_data"].split(",", 1)[1])
- == tiny_pdf.read_bytes() # noqa: ASYNC240 — test fixture, sync read is fine
+ base64.b64decode(file_part["file"]["file_data"].split(",", 1)[1]) == tiny_pdf.read_bytes() # noqa: ASYNC240 — test fixture, sync read is fine
)
assert user["content"][1] == {"type": "text", "text": "What is the diagnosis?"}
assert captured["headers"]["authorization"] == "Bearer sk-or-test"
@@ -85,22 +91,22 @@ async def test_chat_array_content_concatenates(respx_mock, tiny_pdf: Path):
return_value=httpx.Response(
200,
json={
- "choices": [{
- "message": {
- "content": [
- {"type": "text", "text": "Hello "},
- {"type": "text", "text": "world"},
- {"type": "image_url", "image_url": "ignored"},
- ]
+ "choices": [
+ {
+ "message": {
+ "content": [
+ {"type": "text", "text": "Hello "},
+ {"type": "text", "text": "world"},
+ {"type": "image_url", "image_url": "ignored"},
+ ]
+ }
}
- }],
+ ],
"usage": {"prompt_tokens": 1, "completion_tokens": 1},
},
)
)
- provider = OpenRouterPdfProvider(
- api_key="sk-or-test", base_url=_BASE, model="x/y"
- )
+ provider = OpenRouterPdfProvider(api_key="sk-or-test", base_url=_BASE, model="x/y")
response = await provider.complete(prompt="hi", pdf_path=tiny_pdf)
assert response.text == "Hello world"
diff --git a/surfsense_evals/tests/suites/test_crag_dataset.py b/surfsense_evals/tests/suites/test_crag_dataset.py
index 36114b52e..c52e0f56b 100644
--- a/surfsense_evals/tests/suites/test_crag_dataset.py
+++ b/surfsense_evals/tests/suites/test_crag_dataset.py
@@ -11,8 +11,6 @@ import bz2
import json
from pathlib import Path
-import pytest
-
from surfsense_evals.suites.research.crag.dataset import (
CragPage,
CragQuestion,
@@ -67,13 +65,15 @@ class TestParser:
interaction_id="abc",
query="Who directed Inception?",
answer="Christopher Nolan",
- pages=[{
- "page_name": "Inception (film)",
- "page_url": "https://en.wikipedia.org/wiki/Inception",
- "page_snippet": "snippet",
- "page_result": "full html",
- "page_last_modified": "2024-01-01",
- }],
+ pages=[
+ {
+ "page_name": "Inception (film)",
+ "page_url": "https://en.wikipedia.org/wiki/Inception",
+ "page_snippet": "snippet",
+ "page_result": "full html",
+ "page_last_modified": "2024-01-01",
+ }
+ ],
),
]
path = _make_jsonl_bz2(rows, tmp_path)
@@ -122,8 +122,7 @@ class TestParser:
def test_alt_answers_parsed(self, tmp_path: Path) -> None:
rows = [
- _row(interaction_id="z", query="q?", answer="42",
- alt_ans=["forty-two", "42.0"]),
+ _row(interaction_id="z", query="q?", answer="42", alt_ans=["forty-two", "42.0"]),
]
path = _make_jsonl_bz2(rows, tmp_path)
parsed = iter_questions(path)
@@ -145,22 +144,32 @@ class TestParser:
class TestPageHash:
def test_url_hash_stable(self) -> None:
a = CragPage(
- page_name="A", page_url="https://x.test/p?q=1",
- page_snippet="", page_html=" ",
+ page_name="A",
+ page_url="https://x.test/p?q=1",
+ page_snippet="",
+ page_html=" ",
)
b = CragPage(
- page_name="B", page_url="https://x.test/p?q=1",
- page_snippet="", page_html=" ",
+ page_name="B",
+ page_url="https://x.test/p?q=1",
+ page_snippet="",
+ page_html=" ",
)
assert a.url_hash == b.url_hash
assert len(a.url_hash) == 12
def test_url_hash_unique(self) -> None:
a = CragPage(
- page_name="A", page_url="https://x.test/a", page_snippet="", page_html=" ",
+ page_name="A",
+ page_url="https://x.test/a",
+ page_snippet="",
+ page_html=" ",
)
b = CragPage(
- page_name="B", page_url="https://x.test/b", page_snippet="", page_html=" ",
+ page_name="B",
+ page_url="https://x.test/b",
+ page_snippet="",
+ page_html=" ",
)
assert a.url_hash != b.url_hash
@@ -176,21 +185,23 @@ class TestStratifiedSample:
(5, "sports", "multi-hop"),
):
for _ in range(n):
- out.append(CragQuestion(
- qid=f"C{idx:05d}",
- interaction_id=f"i{idx}",
- query_time="2024-01-01",
- query=f"q{idx}?",
- gold_answer="a",
- alt_answers=[],
- domain=domain,
- question_type=qtype,
- static_or_dynamic="static",
- popularity="head",
- split=0,
- raw_index=idx,
- pages=[],
- ))
+ out.append(
+ CragQuestion(
+ qid=f"C{idx:05d}",
+ interaction_id=f"i{idx}",
+ query_time="2024-01-01",
+ query=f"q{idx}?",
+ gold_answer="a",
+ alt_answers=[],
+ domain=domain,
+ question_type=qtype,
+ static_or_dynamic="static",
+ popularity="head",
+ split=0,
+ raw_index=idx,
+ pages=[],
+ )
+ )
idx += 1
return out
diff --git a/surfsense_evals/tests/suites/test_crag_grader.py b/surfsense_evals/tests/suites/test_crag_grader.py
index 93bf6f478..e0599c8b6 100644
--- a/surfsense_evals/tests/suites/test_crag_grader.py
+++ b/surfsense_evals/tests/suites/test_crag_grader.py
@@ -7,8 +7,6 @@ exercise the deterministic shortcut + the special-case routing for
from __future__ import annotations
-import pytest
-
from surfsense_evals.suites.research.crag.grader import (
CragGradeResult,
_flags_false_premise,
@@ -154,7 +152,9 @@ class TestGradeDeterministicHappyPath:
class TestGradeDeterministicRefusal:
def test_idk_maps_to_missing(self) -> None:
result = grade_deterministic(
- pred="I don't know.", gold="Tim Cook", question_type="simple",
+ pred="I don't know.",
+ gold="Tim Cook",
+ question_type="simple",
)
assert result.grade == "missing"
assert result.score == 0
@@ -227,8 +227,11 @@ class TestGradeDeterministicLexicalMiss:
class TestGradeResultShape:
def test_to_dict_round_trip(self) -> None:
result = CragGradeResult(
- grade="correct", score=1, method="exact",
- normalised_pred="x", normalised_gold="x",
+ grade="correct",
+ score=1,
+ method="exact",
+ normalised_pred="x",
+ normalised_gold="x",
)
d = result.to_dict()
assert d["grade"] == "correct"
diff --git a/surfsense_evals/tests/suites/test_crag_html_extract.py b/surfsense_evals/tests/suites/test_crag_html_extract.py
index a2b47aa45..368692177 100644
--- a/surfsense_evals/tests/suites/test_crag_html_extract.py
+++ b/surfsense_evals/tests/suites/test_crag_html_extract.py
@@ -11,13 +11,10 @@ We don't network-fetch trafilatura; we just verify the wrapper:
from __future__ import annotations
-import pytest
-
from surfsense_evals.suites.research.crag.html_extract import (
extract_main_content,
)
-
_RICH_HTML = """\
@@ -115,7 +112,9 @@ class TestFallbackStripper:
"""
result = extract_main_content(
- html, url="https://x.test/", page_name="Title",
+ html,
+ url="https://x.test/",
+ page_name="Title",
)
assert result.ok
assert "content one" in result.text
diff --git a/surfsense_evals/tests/suites/test_frames_dataset.py b/surfsense_evals/tests/suites/test_frames_dataset.py
index e79e7db89..f76dc0b6f 100644
--- a/surfsense_evals/tests/suites/test_frames_dataset.py
+++ b/surfsense_evals/tests/suites/test_frames_dataset.py
@@ -16,8 +16,6 @@ from __future__ import annotations
import textwrap
from pathlib import Path
-import pytest
-
from surfsense_evals.suites.research.frames.dataset import (
FramesQuestion,
_parse_reasoning_types,
@@ -25,7 +23,6 @@ from surfsense_evals.suites.research.frames.dataset import (
load_questions,
)
-
# ---------------------------------------------------------------------------
# Pure-function tests
# ---------------------------------------------------------------------------
diff --git a/surfsense_evals/tests/suites/test_frames_grader.py b/surfsense_evals/tests/suites/test_frames_grader.py
index e6e38ff8a..d4bbad79a 100644
--- a/surfsense_evals/tests/suites/test_frames_grader.py
+++ b/surfsense_evals/tests/suites/test_frames_grader.py
@@ -8,10 +8,7 @@ runner knows to consult the judge.
from __future__ import annotations
-import pytest
-
from surfsense_evals.suites.research.frames.grader import (
- GradeResult,
_maybe_number,
_normalise,
_whole_word_substring,
diff --git a/surfsense_evals/tests/suites/test_frames_wiki_fetch.py b/surfsense_evals/tests/suites/test_frames_wiki_fetch.py
index 4941756f4..483c7aa58 100644
--- a/surfsense_evals/tests/suites/test_frames_wiki_fetch.py
+++ b/surfsense_evals/tests/suites/test_frames_wiki_fetch.py
@@ -63,14 +63,22 @@ class TestCacheFilename:
@pytest.mark.asyncio
@respx.mock
async def test_fetch_success_writes_markdown(tmp_path: Path) -> None:
- respx.get(WIKI_API).mock(return_value=httpx.Response(
- 200,
- json={"query": {"pages": [{
- "pageid": 1,
- "title": "James Buchanan",
- "extract": "James Buchanan was the 15th president of the United States.",
- }]}},
- ))
+ respx.get(WIKI_API).mock(
+ return_value=httpx.Response(
+ 200,
+ json={
+ "query": {
+ "pages": [
+ {
+ "pageid": 1,
+ "title": "James Buchanan",
+ "extract": "James Buchanan was the 15th president of the United States.",
+ }
+ ]
+ }
+ },
+ )
+ )
fetcher = WikiFetcher(cache_dir=tmp_path, rate_limit_rps=100) # disable throttle
article = await fetcher.fetch("https://en.wikipedia.org/wiki/James_Buchanan")
assert article is not None
@@ -83,13 +91,21 @@ async def test_fetch_success_writes_markdown(tmp_path: Path) -> None:
@pytest.mark.asyncio
@respx.mock
async def test_fetch_missing_page_returns_none(tmp_path: Path) -> None:
- respx.get(WIKI_API).mock(return_value=httpx.Response(
- 200,
- json={"query": {"pages": [{
- "title": "DoesNotExist",
- "missing": True,
- }]}},
- ))
+ respx.get(WIKI_API).mock(
+ return_value=httpx.Response(
+ 200,
+ json={
+ "query": {
+ "pages": [
+ {
+ "title": "DoesNotExist",
+ "missing": True,
+ }
+ ]
+ }
+ },
+ )
+ )
fetcher = WikiFetcher(cache_dir=tmp_path, rate_limit_rps=100)
article = await fetcher.fetch("https://en.wikipedia.org/wiki/DoesNotExist")
assert article is None
diff --git a/surfsense_evals/tests/suites/test_mmlongbench_grader.py b/surfsense_evals/tests/suites/test_mmlongbench_grader.py
index 92cd5f0cb..89005b3cd 100644
--- a/surfsense_evals/tests/suites/test_mmlongbench_grader.py
+++ b/surfsense_evals/tests/suites/test_mmlongbench_grader.py
@@ -99,7 +99,9 @@ class TestListFormat:
assert 0.0 < r.f1 < 1.0
def test_extra_items_lower_precision(self) -> None:
- r = grade(pred="apple, banana, cherry, date", gold="apple, banana, cherry", answer_format="List")
+ r = grade(
+ pred="apple, banana, cherry, date", gold="apple, banana, cherry", answer_format="List"
+ )
assert 0.0 < r.f1 < 1.0
# Recall=1, precision=3/4 → F1 ~= 0.857
assert r.f1 == pytest.approx(2 * (3 / 4) * 1 / (3 / 4 + 1), rel=1e-3)
diff --git a/surfsense_mcp/README.md b/surfsense_mcp/README.md
index c18a857e1..95ac293d9 100644
--- a/surfsense_mcp/README.md
+++ b/surfsense_mcp/README.md
@@ -21,6 +21,9 @@ Connect it two ways:
**Scrapers (all platforms)**
- `surfsense_web_crawl`, `surfsense_google_search`, `surfsense_reddit_scrape`,
`surfsense_youtube_scrape`, `surfsense_youtube_comments`,
+ `surfsense_instagram_scrape`, `surfsense_instagram_details`,
+ `surfsense_tiktok_scrape`, `surfsense_tiktok_comments`,
+ `surfsense_tiktok_user_search`, `surfsense_tiktok_trending`,
`surfsense_google_maps_scrape`, `surfsense_google_maps_reviews`
- `surfsense_list_scraper_runs`, `surfsense_get_scraper_run` — retrieve past
results in full (useful when a large result was truncated inline)
diff --git a/surfsense_mcp/mcp_server/features/knowledge_base/__init__.py b/surfsense_mcp/mcp_server/features/knowledge_base/__init__.py
index 1a971bfe4..323fb4e48 100644
--- a/surfsense_mcp/mcp_server/features/knowledge_base/__init__.py
+++ b/surfsense_mcp/mcp_server/features/knowledge_base/__init__.py
@@ -14,9 +14,7 @@ from ...core.workspace_context import WorkspaceContext
from . import document_tools, search_tools
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register every knowledge-base tool on the server."""
search_tools.register(mcp, client, context)
document_tools.register(mcp, client, context)
diff --git a/surfsense_mcp/mcp_server/features/knowledge_base/document_tools.py b/surfsense_mcp/mcp_server/features/knowledge_base/document_tools.py
index 497a2526c..f2cc20f9e 100644
--- a/surfsense_mcp/mcp_server/features/knowledge_base/document_tools.py
+++ b/surfsense_mcp/mcp_server/features/knowledge_base/document_tools.py
@@ -20,9 +20,7 @@ from .annotations import DELETE, WRITE, DocumentId
from .note_ingestion import build_note_document
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the knowledge-base write and delete tools."""
@mcp.tool(
@@ -136,8 +134,7 @@ def register(
str,
Field(
min_length=1,
- description="New full text; replaces the existing content "
- "entirely.",
+ description="New full text; replaces the existing content entirely.",
),
],
) -> str:
diff --git a/surfsense_mcp/mcp_server/features/knowledge_base/search_tools.py b/surfsense_mcp/mcp_server/features/knowledge_base/search_tools.py
index a9e60810d..c0f7f83a9 100644
--- a/surfsense_mcp/mcp_server/features/knowledge_base/search_tools.py
+++ b/surfsense_mcp/mcp_server/features/knowledge_base/search_tools.py
@@ -17,9 +17,7 @@ from ...core.workspace_context import WorkspaceContext, WorkspaceParam
from .annotations import READ, DocumentId, DocumentTypes
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the knowledge-base read tools."""
@mcp.tool(
@@ -81,12 +79,8 @@ def register(
int | None,
Field(description="Only documents in this folder. Omit for all."),
] = None,
- page: Annotated[
- int, Field(ge=0, description="Zero-based page number.")
- ] = 0,
- page_size: Annotated[
- int, Field(ge=1, description="Documents per page.")
- ] = 20,
+ page: Annotated[int, Field(ge=0, description="Zero-based page number.")] = 0,
+ page_size: Annotated[int, Field(ge=1, description="Documents per page.")] = 20,
workspace: WorkspaceParam = None,
response_format: ResponseFormatParam = "markdown",
) -> str:
diff --git a/surfsense_mcp/mcp_server/features/scrapers/__init__.py b/surfsense_mcp/mcp_server/features/scrapers/__init__.py
index dfa2f3ab2..9aabbe0e5 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/__init__.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/__init__.py
@@ -13,14 +13,29 @@ from mcp.server.fastmcp import FastMCP
from ...core.client import SurfSenseClient
from ...core.workspace_context import WorkspaceContext
from . import run_history
-from .platforms import google_maps, google_search, reddit, web, youtube
+from .platforms import (
+ google_maps,
+ google_search,
+ instagram,
+ reddit,
+ tiktok,
+ web,
+ youtube,
+)
-_REGISTRARS = (web, google_search, reddit, youtube, google_maps, run_history)
+_REGISTRARS = (
+ web,
+ google_search,
+ reddit,
+ youtube,
+ instagram,
+ tiktok,
+ google_maps,
+ run_history,
+)
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register every scraper and run-history tool on the server."""
for module in _REGISTRARS:
module.register(mcp, client, context)
diff --git a/surfsense_mcp/mcp_server/features/scrapers/capability.py b/surfsense_mcp/mcp_server/features/scrapers/capability.py
index 7245c9f84..82f24c6e7 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/capability.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/capability.py
@@ -38,9 +38,7 @@ async def run_scraper(
return _render_markdown(platform, verb, resolved.name, result)
-def _render_markdown(
- platform: str, verb: str, workspace_name: str, result: Any
-) -> str:
+def _render_markdown(platform: str, verb: str, workspace_name: str, result: Any) -> str:
"""A readable header plus the structured payload, clipped to a safe size."""
header = f'# {platform}.{verb} — {_describe_size(result)} from "{workspace_name}"'
body = clip(to_json(result))
diff --git a/surfsense_mcp/mcp_server/features/scrapers/platforms/google_maps.py b/surfsense_mcp/mcp_server/features/scrapers/platforms/google_maps.py
index e1613ca4e..daa27cef2 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/platforms/google_maps.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/platforms/google_maps.py
@@ -16,9 +16,7 @@ from ..capability import run_scraper
ReviewSort = Literal["newest", "mostRelevant", "highestRanking", "lowestRanking"]
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the Google Maps place and review tools."""
@mcp.tool(
@@ -45,10 +43,7 @@ def register(
] = None,
location: Annotated[
str | None,
- Field(
- description="Geographic scope for a search, e.g. "
- "'Seattle, USA'."
- ),
+ Field(description="Geographic scope for a search, e.g. 'Seattle, USA'."),
] = None,
max_places: Annotated[
int, Field(ge=1, description="Maximum places to return.")
@@ -56,8 +51,7 @@ def register(
include_details: Annotated[
bool,
Field(
- description="True adds opening hours and extra contact info "
- "(slower)."
+ description="True adds opening hours and extra contact info (slower)."
),
] = False,
workspace: WorkspaceParam = None,
@@ -96,16 +90,11 @@ def register(
async def google_maps_reviews(
urls: Annotated[
list[str] | None,
- Field(
- description="Google Maps URLs of places. Provide urls OR "
- "place_ids."
- ),
+ Field(description="Google Maps URLs of places. Provide urls OR place_ids."),
] = None,
place_ids: Annotated[
list[str] | None,
- Field(
- description="Google place ids from surfsense_google_maps_scrape."
- ),
+ Field(description="Google place ids from surfsense_google_maps_scrape."),
] = None,
max_reviews: Annotated[
int, Field(ge=1, description="Maximum reviews per place.")
diff --git a/surfsense_mcp/mcp_server/features/scrapers/platforms/google_search.py b/surfsense_mcp/mcp_server/features/scrapers/platforms/google_search.py
index cc1a1f8ed..acabef1d1 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/platforms/google_search.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/platforms/google_search.py
@@ -14,9 +14,7 @@ from ..annotations import SCRAPE
from ..capability import run_scraper
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the Google Search tool."""
@mcp.tool(
@@ -46,9 +44,7 @@ def register(
] = "",
site: Annotated[
str | None,
- Field(
- description="Restrict results to one domain, e.g. 'example.com'."
- ),
+ Field(description="Restrict results to one domain, e.g. 'example.com'."),
] = None,
workspace: WorkspaceParam = None,
response_format: ResponseFormatParam = "markdown",
diff --git a/surfsense_mcp/mcp_server/features/scrapers/platforms/instagram.py b/surfsense_mcp/mcp_server/features/scrapers/platforms/instagram.py
new file mode 100644
index 000000000..62e7552d6
--- /dev/null
+++ b/surfsense_mcp/mcp_server/features/scrapers/platforms/instagram.py
@@ -0,0 +1,140 @@
+"""Instagram scraper tools: posts/reels and profile details (anonymous-only)."""
+
+from __future__ import annotations
+
+from typing import Annotated, Literal
+
+from mcp.server.fastmcp import FastMCP
+from pydantic import Field
+
+from ....core.client import SurfSenseClient
+from ....core.rendering import ResponseFormatParam
+from ....core.workspace_context import WorkspaceContext, WorkspaceParam
+from ..annotations import SCRAPE
+from ..capability import run_scraper
+
+ResultType = Literal["posts", "reels"]
+SearchType = Literal["profile", "user"]
+
+
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
+ """Register the Instagram scrape and details tools (anonymous-only)."""
+
+ @mcp.tool(
+ name="surfsense_instagram_scrape",
+ title="Scrape Instagram posts or reels",
+ annotations=SCRAPE,
+ structured_output=False,
+ )
+ async def instagram_scrape(
+ urls: Annotated[
+ list[str] | None,
+ Field(
+ description="Instagram URLs: a profile, post (/p/), or reel "
+ "(/reel/). Hashtag/place URLs are unsupported (login-walled). "
+ "Provide urls OR search_queries."
+ ),
+ ] = None,
+ search_queries: Annotated[
+ list[str] | None,
+ Field(
+ description="Terms to discover public profiles for (resolved via "
+ "Google). Google-backed discovery is slow (~30-60s per query), so "
+ "start with at most 3 queries and expand only if nothing "
+ "significant is found. Provide search_queries OR urls."
+ ),
+ ] = None,
+ search_type: Annotated[
+ SearchType,
+ Field(description="Discovery kind (profile-only)."),
+ ] = "profile",
+ result_type: Annotated[
+ ResultType,
+ Field(description="Which feed to return: 'posts' or 'reels'."),
+ ] = "posts",
+ max_items: Annotated[
+ int, Field(ge=1, description="Maximum items to return across sources.")
+ ] = 10,
+ workspace: WorkspaceParam = None,
+ response_format: ResponseFormatParam = "markdown",
+ ) -> str:
+ """Scrape public Instagram posts or reels from URLs or search queries.
+
+ Use this for Instagram content research: a creator's recent posts, a
+ single post/reel, or discovering public profiles by keyword. For a
+ profile's metadata use surfsense_instagram_details. Returns per-item
+ caption, likes, comments count, media URLs, and owner. Anonymous-only:
+ hashtag/place feeds and comment threads are login-walled and unavailable.
+ Example: urls=['https://www.instagram.com/natgeo/'], result_type='reels'.
+ """
+ return await run_scraper(
+ client,
+ context,
+ platform="instagram",
+ verb="scrape",
+ payload={
+ "urls": urls,
+ "search_queries": search_queries,
+ "search_type": search_type,
+ "result_type": result_type,
+ "max_items": max_items,
+ },
+ workspace=workspace,
+ response_format=response_format,
+ )
+
+ @mcp.tool(
+ name="surfsense_instagram_details",
+ title="Fetch Instagram profile details",
+ annotations=SCRAPE,
+ structured_output=False,
+ )
+ async def instagram_details(
+ urls: Annotated[
+ list[str] | None,
+ Field(
+ description="Profile URLs (or bare profile IDs). Provide urls OR "
+ "search_queries."
+ ),
+ ] = None,
+ search_queries: Annotated[
+ list[str] | None,
+ Field(
+ description="Terms to discover public profiles for. Google-backed "
+ "discovery is slow (~30-60s per query), so start with at most 3 "
+ "queries and expand only if nothing significant is found. Provide "
+ "search_queries OR urls."
+ ),
+ ] = None,
+ search_type: Annotated[
+ SearchType,
+ Field(description="Discovery kind (profile-only)."),
+ ] = "profile",
+ max_items: Annotated[
+ int, Field(ge=1, description="Max detail items to return.")
+ ] = 10,
+ workspace: WorkspaceParam = None,
+ response_format: ResponseFormatParam = "markdown",
+ ) -> str:
+ """Fetch Instagram profile metadata.
+
+ Use this for entity lookups: a profile's follower/post counts and bio.
+ For a feed of posts use surfsense_instagram_scrape instead. Each item
+ carries a detailKind field (always "profile"). Anonymous-only: hashtag
+ and place details are login-walled and unavailable.
+ Example: urls=['https://www.instagram.com/natgeo/'].
+ """
+ return await run_scraper(
+ client,
+ context,
+ platform="instagram",
+ verb="details",
+ payload={
+ "urls": urls,
+ "search_queries": search_queries,
+ "search_type": search_type,
+ "max_items": max_items,
+ },
+ workspace=workspace,
+ response_format=response_format,
+ )
diff --git a/surfsense_mcp/mcp_server/features/scrapers/platforms/reddit.py b/surfsense_mcp/mcp_server/features/scrapers/platforms/reddit.py
index 035193ebc..1bc2d851e 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/platforms/reddit.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/platforms/reddit.py
@@ -17,9 +17,7 @@ RedditSort = Literal["relevance", "hot", "top", "new", "rising", "comments"]
RedditTime = Literal["hour", "day", "week", "month", "year", "all"]
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the Reddit tool."""
@mcp.tool(
diff --git a/surfsense_mcp/mcp_server/features/scrapers/platforms/tiktok.py b/surfsense_mcp/mcp_server/features/scrapers/platforms/tiktok.py
new file mode 100644
index 000000000..620936063
--- /dev/null
+++ b/surfsense_mcp/mcp_server/features/scrapers/platforms/tiktok.py
@@ -0,0 +1,196 @@
+"""TikTok scraper tools: scrape (videos), comments, user search, and trending."""
+
+from __future__ import annotations
+
+from typing import Annotated
+
+from mcp.server.fastmcp import FastMCP
+from pydantic import Field
+
+from ....core.client import SurfSenseClient
+from ....core.rendering import ResponseFormatParam
+from ....core.workspace_context import WorkspaceContext, WorkspaceParam
+from ..annotations import SCRAPE
+from ..capability import run_scraper
+
+
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
+ """Register the TikTok tools."""
+
+ @mcp.tool(
+ name="surfsense_tiktok_scrape",
+ title="Scrape TikTok videos",
+ annotations=SCRAPE,
+ structured_output=False,
+ )
+ async def tiktok_scrape(
+ urls: Annotated[
+ list[str] | None,
+ Field(
+ description="TikTok URLs: a video, a profile "
+ "('https://www.tiktok.com/@nasa'), a hashtag "
+ "('https://www.tiktok.com/tag/food'), or a search URL. Provide "
+ "urls OR profiles/hashtags."
+ ),
+ ] = None,
+ profiles: Annotated[
+ list[str] | None,
+ Field(
+ description="Profile usernames to scrape, with or without a "
+ "leading '@', e.g. ['nasa']."
+ ),
+ ] = None,
+ hashtags: Annotated[
+ list[str] | None,
+ Field(
+ description="Hashtag names to scrape, without the '#', e.g. ['food']."
+ ),
+ ] = None,
+ results_per_page: Annotated[
+ int,
+ Field(ge=1, description="Max videos per profile/hashtag target."),
+ ] = 10,
+ max_items: Annotated[
+ int, Field(ge=1, description="Maximum videos to return in total.")
+ ] = 10,
+ workspace: WorkspaceParam = None,
+ response_format: ResponseFormatParam = "markdown",
+ ) -> str:
+ """Scrape public TikTok videos by hashtag, profile, or URL.
+
+ Use for TikTok video research — a creator's videos, a hashtag feed, or a
+ specific video/profile/hashtag URL — instead of a generic web search.
+ Returns videos with text, author, stats, music, and the web URL. There is
+ no keyword-video search; for accounts by keyword use
+ surfsense_tiktok_user_search. Example: hashtags=['food'], max_items=20.
+ """
+ return await run_scraper(
+ client,
+ context,
+ platform="tiktok",
+ verb="scrape",
+ payload={
+ "urls": urls,
+ "profiles": profiles,
+ "hashtags": hashtags,
+ "results_per_page": results_per_page,
+ "max_items": max_items,
+ },
+ workspace=workspace,
+ response_format=response_format,
+ )
+
+ @mcp.tool(
+ name="surfsense_tiktok_comments",
+ title="Scrape TikTok comments",
+ annotations=SCRAPE,
+ structured_output=False,
+ )
+ async def tiktok_comments(
+ video_urls: Annotated[
+ list[str],
+ Field(
+ description="TikTok video URLs "
+ "('https://www.tiktok.com/@user/video/123') to pull comments from."
+ ),
+ ],
+ comments_per_video: Annotated[
+ int, Field(ge=1, description="Max comments to return per video.")
+ ] = 20,
+ max_items: Annotated[
+ int, Field(ge=1, description="Maximum comments to return in total.")
+ ] = 20,
+ workspace: WorkspaceParam = None,
+ response_format: ResponseFormatParam = "markdown",
+ ) -> str:
+ """Scrape the public comments of TikTok videos.
+
+ Returns each comment's text, author, like count, and reply count (replies
+ carry the parent comment id). Example: video_urls=['https://www.tiktok.com/
+ @nasa/video/123'], max_items=50.
+ """
+ return await run_scraper(
+ client,
+ context,
+ platform="tiktok",
+ verb="comments",
+ payload={
+ "video_urls": video_urls,
+ "comments_per_video": comments_per_video,
+ "max_items": max_items,
+ },
+ workspace=workspace,
+ response_format=response_format,
+ )
+
+ @mcp.tool(
+ name="surfsense_tiktok_user_search",
+ title="Search TikTok accounts",
+ annotations=SCRAPE,
+ structured_output=False,
+ )
+ async def tiktok_user_search(
+ queries: Annotated[
+ list[str],
+ Field(
+ description="Keywords to find TikTok accounts by, e.g. "
+ "['nasa', 'cooking']."
+ ),
+ ],
+ results_per_query: Annotated[
+ int, Field(ge=1, description="Max accounts to return per query.")
+ ] = 10,
+ max_items: Annotated[
+ int, Field(ge=1, description="Maximum accounts to return in total.")
+ ] = 10,
+ workspace: WorkspaceParam = None,
+ response_format: ResponseFormatParam = "markdown",
+ ) -> str:
+ """Find public TikTok accounts by keyword.
+
+ Returns matching profiles with name, followers, bio, and verification —
+ the reliable account-discovery path (video search is login-walled).
+ Example: queries=['space agency'], max_items=20.
+ """
+ return await run_scraper(
+ client,
+ context,
+ platform="tiktok",
+ verb="user_search",
+ payload={
+ "queries": queries,
+ "results_per_query": results_per_query,
+ "max_items": max_items,
+ },
+ workspace=workspace,
+ response_format=response_format,
+ )
+
+ @mcp.tool(
+ name="surfsense_tiktok_trending",
+ title="Get trending TikTok videos",
+ annotations=SCRAPE,
+ structured_output=False,
+ )
+ async def tiktok_trending(
+ max_items: Annotated[
+ int,
+ Field(ge=1, description="Max trending videos to return from Explore."),
+ ] = 20,
+ workspace: WorkspaceParam = None,
+ response_format: ResponseFormatParam = "markdown",
+ ) -> str:
+ """Get the current trending TikTok videos from the Explore feed.
+
+ No input needed beyond how many to return; each video comes with caption,
+ author, stats, music, and its web URL. Example: max_items=30.
+ """
+ return await run_scraper(
+ client,
+ context,
+ platform="tiktok",
+ verb="trending",
+ payload={"max_items": max_items},
+ workspace=workspace,
+ response_format=response_format,
+ )
diff --git a/surfsense_mcp/mcp_server/features/scrapers/platforms/web.py b/surfsense_mcp/mcp_server/features/scrapers/platforms/web.py
index 9c24a4352..8c6faaaf3 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/platforms/web.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/platforms/web.py
@@ -14,9 +14,7 @@ from ..annotations import SCRAPE
from ..capability import run_scraper
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the web crawl tool."""
@mcp.tool(
diff --git a/surfsense_mcp/mcp_server/features/scrapers/platforms/youtube.py b/surfsense_mcp/mcp_server/features/scrapers/platforms/youtube.py
index 5582c82bb..1871fe711 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/platforms/youtube.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/platforms/youtube.py
@@ -16,9 +16,7 @@ from ..capability import run_scraper
CommentSort = Literal["TOP_COMMENTS", "NEWEST_FIRST"]
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the YouTube video and comment tools."""
@mcp.tool(
diff --git a/surfsense_mcp/mcp_server/features/scrapers/run_history.py b/surfsense_mcp/mcp_server/features/scrapers/run_history.py
index 9274a1a69..ed36cfca6 100644
--- a/surfsense_mcp/mcp_server/features/scrapers/run_history.py
+++ b/surfsense_mcp/mcp_server/features/scrapers/run_history.py
@@ -17,9 +17,7 @@ from ...core.workspace_context import WorkspaceContext, WorkspaceParam
from .annotations import READ_RUNS
-def register(
- mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext
-) -> None:
+def register(mcp: FastMCP, client: SurfSenseClient, context: WorkspaceContext) -> None:
"""Register the run-history tools."""
@mcp.tool(
@@ -29,9 +27,7 @@ def register(
structured_output=False,
)
async def list_scraper_runs(
- limit: Annotated[
- int, Field(ge=1, description="Maximum runs to list.")
- ] = 20,
+ limit: Annotated[int, Field(ge=1, description="Maximum runs to list.")] = 20,
capability: Annotated[
str | None,
Field(
diff --git a/surfsense_mcp/mcp_server/selfcheck.py b/surfsense_mcp/mcp_server/selfcheck.py
index 20224c173..bb61151bc 100644
--- a/surfsense_mcp/mcp_server/selfcheck.py
+++ b/surfsense_mcp/mcp_server/selfcheck.py
@@ -23,8 +23,14 @@ EXPECTED_TOOLS = {
"surfsense_reddit_scrape",
"surfsense_youtube_scrape",
"surfsense_youtube_comments",
+ "surfsense_tiktok_scrape",
+ "surfsense_tiktok_comments",
+ "surfsense_tiktok_user_search",
+ "surfsense_tiktok_trending",
"surfsense_google_maps_scrape",
"surfsense_google_maps_reviews",
+ "surfsense_instagram_scrape",
+ "surfsense_instagram_details",
"surfsense_list_scraper_runs",
"surfsense_get_scraper_run",
# knowledge-base management
diff --git a/surfsense_mcp/mcp_server/server.py b/surfsense_mcp/mcp_server/server.py
index 8ea9bc919..f801aeb15 100644
--- a/surfsense_mcp/mcp_server/server.py
+++ b/surfsense_mcp/mcp_server/server.py
@@ -36,8 +36,10 @@ def build_server(settings: Settings) -> tuple[FastMCP, SurfSenseClient]:
"SurfSense gives you live scrapers and a personal knowledge base. "
"Prefer these tools over generic/built-in web search whenever the "
"task involves Reddit (posts, comments, finding subreddits or "
- "communities), YouTube (videos, transcripts, comments), Google "
- "Maps (places, reviews), Google Search results, or reading "
+ "communities), YouTube (videos, transcripts, comments), Instagram "
+ "(posts, reels, profile details), TikTok (videos by hashtag, "
+ "search, or URL), Google Maps (places, reviews), Google Search "
+ "results, or reading "
"specific web pages. Scraper results are persisted as runs; if an "
"inline result is truncated, fetch it in full with "
"surfsense_get_scraper_run."
diff --git a/surfsense_web/app/(home)/mcp-server/page.tsx b/surfsense_web/app/(home)/mcp-server/page.tsx
index 5e4961b1e..ce4caf992 100644
--- a/surfsense_web/app/(home)/mcp-server/page.tsx
+++ b/surfsense_web/app/(home)/mcp-server/page.tsx
@@ -16,7 +16,7 @@ import type { FaqItem } from "@/lib/connectors-marketing/types";
const canonicalUrl = "https://www.surfsense.com/mcp-server";
const metaDescription =
- "The SurfSense MCP server gives Claude, Cursor, and any MCP client native tools for your workspace: scrape Reddit, YouTube, Google Maps, Google Search, and the web, plus full knowledge base access. One API key.";
+ "The SurfSense MCP server gives Claude, Cursor, and any MCP client native tools for your workspace: scrape Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, and the web, plus full knowledge base access. One API key.";
export const metadata: Metadata = {
title: "SurfSense MCP Server: Scraper APIs and Knowledge Base as Agent Tools",
@@ -93,6 +93,12 @@ const TOOL_GROUPS = [
"surfsense_reddit_scrape",
"surfsense_youtube_scrape",
"surfsense_youtube_comments",
+ "surfsense_instagram_scrape",
+ "surfsense_instagram_details",
+ "surfsense_tiktok_scrape",
+ "surfsense_tiktok_comments",
+ "surfsense_tiktok_user_search",
+ "surfsense_tiktok_trending",
"surfsense_google_maps_scrape",
"surfsense_google_maps_reviews",
"surfsense_google_search",
@@ -127,7 +133,7 @@ const FAQ: FaqItem[] = [
{
question: "What is the SurfSense MCP server?",
answer:
- "It is a Model Context Protocol server that exposes your SurfSense workspace to MCP clients like Claude Code, Cursor, and Claude Desktop. Your agents get native tools for every scraper API (Reddit, YouTube, Google Maps, Google Search, web crawl) and for searching, reading, and writing your knowledge base.",
+ "It is a Model Context Protocol server that exposes your SurfSense workspace to MCP clients like Claude Code, Cursor, and Claude Desktop. Your agents get native tools for every scraper API (Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, web crawl) and for searching, reading, and writing your knowledge base.",
},
{
question: "Which MCP clients does it work with?",
@@ -215,8 +221,9 @@ export default function McpServerPage() {
The SurfSense MCP server hands Claude, Cursor, or any MCP client the whole platform:
- scrape Reddit, YouTube, Google Maps, Google Search, and the open web, and search,
- read, and write your knowledge base. One API key, typed tools, pay as you go.
+ scrape Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, and the open
+ web, and search, read, and write your knowledge base. One API key, typed tools, pay
+ as you go.
+
-
+
diff --git a/surfsense_web/components/assistant-ui/user-message.tsx b/surfsense_web/components/assistant-ui/user-message.tsx
index 592dc4224..09a700ff1 100644
--- a/surfsense_web/components/assistant-ui/user-message.tsx
+++ b/surfsense_web/components/assistant-ui/user-message.tsx
@@ -22,6 +22,7 @@ import { currentThreadAtom } from "@/atoms/chat/current-thread.atom";
import { messageDocumentsMapAtom } from "@/atoms/chat/mentioned-documents.atom";
import { openEditorPanelAtom } from "@/atoms/editor/editor-panel.atom";
import { MentionChip } from "@/components/assistant-ui/mention-chip";
+import { MessageTimestamp } from "@/components/assistant-ui/message-timestamp";
import { TooltipIconButton } from "@/components/assistant-ui/tooltip-icon-button";
import { getConnectorIcon } from "@/contracts/enums/connectorIcons";
import { getMentionDocKey } from "@/lib/chat/mention-doc-key";
@@ -182,6 +183,7 @@ export const UserMessage: FC = () => {
)}
+ Give your AI agents competitive intelligence.
+ NotebookLM for competitive intelligence research.
@@ -717,9 +717,10 @@ export function HeroSection() {
"relative mb-8 max-w-2xl text-left text-sm text-neutral-600 antialiased sm:text-base md:text-lg dark:text-neutral-400"
)}
>
- SurfSense is an open-source competitive intelligence platform. Your AI agents monitor
- competitors, track rankings, and listen to your market with live data from platforms
- like Reddit, YouTube, Google Maps, Google Search, and the open web.
+ SurfSense is an open-source competitive intelligence platform, like NotebookLM but
+ with live scraping connectors. Your AI agents monitor competitors, track rankings, and
+ listen to your market with live data from platforms like Reddit, YouTube, Instagram,
+ TikTok, Google Maps, Google Search, and the open web.
diff --git a/surfsense_web/components/homepage/persona-paths.tsx b/surfsense_web/components/homepage/persona-paths.tsx
index 9c03c5667..50eadba0f 100644
--- a/surfsense_web/components/homepage/persona-paths.tsx
+++ b/surfsense_web/components/homepage/persona-paths.tsx
@@ -35,7 +35,7 @@ const PATHS: {
eyebrow: "For developers & agents",
title: "The whole platform is programmable",
description:
- "Everything SurfSense agents can do is a typed REST API: scrape Reddit, YouTube, Google Maps, Google Search, and the open web, search the knowledge base, run automations. One key, JSON in and out, $5 free credit, pay as you go. Already running agents in Claude, Cursor, or your own harness? The SurfSense MCP server hands them the same tools natively.",
+ "Everything SurfSense agents can do is a typed REST API: scrape Reddit, YouTube, TikTok, Google Maps, Google Search, and the open web, search the knowledge base, run automations. One key, JSON in and out, $5 free credit, pay as you go. Already running agents in Claude, Cursor, or your own harness? The SurfSense MCP server hands them the same tools natively.",
links: [
{ label: "Read the docs", href: "/docs" },
{ label: "SurfSense MCP server", href: "/mcp-server" },
diff --git a/surfsense_web/components/homepage/use-cases.tsx b/surfsense_web/components/homepage/use-cases.tsx
index 44609351d..6163f7481 100644
--- a/surfsense_web/components/homepage/use-cases.tsx
+++ b/surfsense_web/components/homepage/use-cases.tsx
@@ -28,6 +28,14 @@ const USE_CASES: {
anchor: "Reddit API",
art: "brand",
},
+ {
+ title: "Social sentiment mining",
+ description:
+ "Pull public posts, reels, and full comment threads from any creator or competitor, then score how audiences actually react to launches and campaigns.",
+ href: "/instagram",
+ anchor: "Instagram API",
+ art: "chat",
+ },
{
title: "B2B lead generation",
description:
diff --git a/surfsense_web/components/icons/providers/index.ts b/surfsense_web/components/icons/providers/index.ts
index 5c8276e62..f03fbec68 100644
--- a/surfsense_web/components/icons/providers/index.ts
+++ b/surfsense_web/components/icons/providers/index.ts
@@ -27,6 +27,7 @@ export { default as PerplexityIcon } from "./perplexity.svg";
export { default as QwenIcon } from "./qwen.svg";
export { default as RecraftIcon } from "./recraft.svg";
export { default as ReplicateIcon } from "./replicate.svg";
+export { default as RequestyIcon } from "./requesty.svg";
export { default as SambaNovaIcon } from "./sambanova.svg";
export { default as TogetherAiIcon } from "./togetherai.svg";
export { default as VertexAiIcon } from "./vertexai.svg";
diff --git a/surfsense_web/components/icons/providers/requesty.svg b/surfsense_web/components/icons/providers/requesty.svg
new file mode 100644
index 000000000..a804601b6
--- /dev/null
+++ b/surfsense_web/components/icons/providers/requesty.svg
@@ -0,0 +1 @@
+
diff --git a/surfsense_web/components/layout/providers/LayoutDataProvider.tsx b/surfsense_web/components/layout/providers/LayoutDataProvider.tsx
index f9c2c9072..81f0d974f 100644
--- a/surfsense_web/components/layout/providers/LayoutDataProvider.tsx
+++ b/surfsense_web/components/layout/providers/LayoutDataProvider.tsx
@@ -18,6 +18,7 @@ import { workspacesAtom } from "@/atoms/workspaces/workspace-query.atoms";
import { ActionLogDialog } from "@/components/agent-action-log/action-log-dialog";
import { AnnouncementSpotlight } from "@/components/announcements/AnnouncementSpotlight";
import { AnnouncementsDialog } from "@/components/announcements/AnnouncementsDialog";
+import { ActiveChatStreamRunner } from "@/components/chat/active-chat-stream-runner";
import {
AlertDialog,
AlertDialogAction,
@@ -644,6 +645,9 @@ export function LayoutDataProvider({ workspaceId, children }: LayoutDataProvider
return (
<>
+ {/* Persistent host: keeps an in-flight chat turn streaming across
+ in-app navigation and aborts it only on workspace teardown. */}
+
{tab.label}
-
+
{formatNotificationCount(tab.count)}
diff --git a/surfsense_web/components/layout/ui/sidebar/SidebarUserProfile.tsx b/surfsense_web/components/layout/ui/sidebar/SidebarUserProfile.tsx
index d5e72174e..cfb76e8dc 100644
--- a/surfsense_web/components/layout/ui/sidebar/SidebarUserProfile.tsx
+++ b/surfsense_web/components/layout/ui/sidebar/SidebarUserProfile.tsx
@@ -52,6 +52,7 @@ const LANGUAGES = [
{ code: "pt" as const, name: "Português", flag: "🇧🇷" },
{ code: "hi" as const, name: "हिन्दी", flag: "🇮🇳" },
{ code: "zh" as const, name: "简体中文", flag: "🇨🇳" },
+ { code: "ko" as const, name: "한국어", flag: "🇰🇷" },
];
// Supported themes configuration
@@ -157,7 +158,7 @@ export function SidebarUserProfile({
const showDownloadCta = !isDesktop && !isMobileOS && isDesktopViewport;
const useMobileSubmenus = !isDesktopViewport;
- const handleLanguageChange = (newLocale: "en" | "es" | "pt" | "hi" | "zh") => {
+ const handleLanguageChange = (newLocale: "en" | "es" | "pt" | "hi" | "zh" | "ko") => {
setLocale(newLocale);
};
diff --git a/surfsense_web/components/pricing/pricing-section.tsx b/surfsense_web/components/pricing/pricing-section.tsx
index 74c76ad36..1af296e57 100644
--- a/surfsense_web/components/pricing/pricing-section.tsx
+++ b/surfsense_web/components/pricing/pricing-section.tsx
@@ -35,7 +35,7 @@ const demoPlans = [
billingText: "Your first $5 of credit is free. No subscription, ever",
features: [
"$5 of free credit to start, one balance for everything",
- "Platform connectors: Reddit, YouTube, Google Maps, Google Search, and the open web",
+ "Platform connectors: Reddit, YouTube, TikTok, Google Maps, Google Search, and the open web",
"Call every connector as a REST API with your key or through the MCP server",
"Pay per item returned and per page crawled. Failed calls are never billed",
"Premium models like GPT-5.5, Claude Sonnet 5, Gemini 3.1 Pro billed at provider cost",
@@ -95,7 +95,7 @@ const faqData: FAQSection[] = [
{
question: "How does Pay As You Go work?",
answer:
- "There is no monthly subscription. Start with $5 of free credit, and when you need more, add any amount. $1 buys exactly $1 of credit, added to your balance immediately. You can enable automatic refills when your balance runs low, and turn them off any time."
+ "There is no monthly subscription. Start with $5 of free credit, and when you need more, add any amount. $1 buys exactly $1 of credit, added to your balance immediately. You can enable automatic refills when your balance runs low, and turn them off any time.",
},
{
question: "What happens if I run out of credit?",
diff --git a/surfsense_web/components/seo/json-ld.tsx b/surfsense_web/components/seo/json-ld.tsx
index c4b3ec09c..063f6a0c0 100644
--- a/surfsense_web/components/seo/json-ld.tsx
+++ b/surfsense_web/components/seo/json-ld.tsx
@@ -76,11 +76,11 @@ export function SoftwareApplicationJsonLd() {
"Free self-hosted from the open-source repo; cloud starts with $5 of free credit, then pay as you go",
},
description:
- "SurfSense is an open-source competitive intelligence platform. AI agents monitor competitors, track rankings, and listen to your market with platform-native connectors for Reddit, YouTube, Google Maps, Google Search, and the open web, through one API or MCP server.",
+ "SurfSense is an open-source competitive intelligence platform. AI agents monitor competitors, track rankings, and listen to your market with platform-native connectors for Reddit, YouTube, TikTok, Google Maps, Google Search, and the open web, through one API or MCP server.",
url: "https://www.surfsense.com",
downloadUrl: "https://github.com/MODSetter/SurfSense/releases",
featureList: [
- "Platform-native connectors: Reddit, YouTube, Google Maps, Google Search, Web Crawl",
+ "Platform-native connectors: Reddit, YouTube, TikTok, Google Maps, Google Search, Web Crawl",
"MCP server that exposes every connector as a native agent tool",
"Agent harness with retries, structured output, and credit metering",
"Competitor, brand, and rank monitoring with briefs and alerts",
diff --git a/surfsense_web/components/settings/auto-reload-settings.tsx b/surfsense_web/components/settings/auto-reload-settings.tsx
index ede826dd0..316c5975d 100644
--- a/surfsense_web/components/settings/auto-reload-settings.tsx
+++ b/surfsense_web/components/settings/auto-reload-settings.tsx
@@ -198,8 +198,8 @@ export function AutoReloadSettings() {
Last top-up failed
- Your saved card was declined and top-ups were turned off. Update your card and
- re-enable top-ups below.
+ Your saved card was declined and top-ups were turned off. Update your card and re-enable
+ top-ups below.
)}
@@ -290,7 +290,11 @@ export function AutoReloadSettings() {
-