From 36bb4a1bea72722a6cd4af5bdcbfa172b28f23ed Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 11 Jul 2026 02:20:54 +0530 Subject: [PATCH] docs(instagram): update platform scraper README --- .../proprietary/platforms/instagram/README.md | 148 +++++++++++------- 1 file changed, 92 insertions(+), 56 deletions(-) diff --git a/surfsense_backend/app/proprietary/platforms/instagram/README.md b/surfsense_backend/app/proprietary/platforms/instagram/README.md index fa42e30a6..e4d7ca0f7 100644 --- a/surfsense_backend/app/proprietary/platforms/instagram/README.md +++ b/surfsense_backend/app/proprietary/platforms/instagram/README.md @@ -1,37 +1,44 @@ -# Instagram scraper (anonymous, no browser) +# Instagram scraper (anonymous) -Platform-native Instagram scraper (anonymous, no browser). Standalone module: it -depends only on `app.utils.proxy` + `scrapling` and exposes a stable public API. -Its input/output surface is a **reference-compatible** mirror of the public -Instagram scraper actor spec (same `resultsType` / `directUrls` / camelCase field -names, additive `extra="allow"` parity), so callers written against that surface -work unchanged. It is **not** wired into ingestion or Celery — the capability -layer under `app/capabilities/instagram/` is what turns these primitives into -REST + agent + MCP surfaces. +Platform-native Instagram scraper. **Anonymous-only** and browser-free: every +flow stays on the cheap HTTP tier (`app.utils.proxy` + `scrapling`), and profile +discovery reuses the `google_search` platform (see below). It exposes a stable +public API whose input/output surface mirrors the public Instagram scraper actor +spec (same `resultsType` / `directUrls` / camelCase field names, additive +`extra="allow"` parity), so callers written against that surface work unchanged. +It is **not** wired into ingestion or Celery — the capability layer under +`app/capabilities/instagram/` turns these primitives into REST + agent + MCP +surfaces. ## Approach -Instagram's public web app exposes anonymous, logged-out JSON behind a handful of -`www.instagram.com` endpoints once a session carries an anonymous `csrftoken` + -`mid` cookie pair and the `x-ig-app-id` web header: +Instagram's public web app exposes anonymous, logged-out data once a session +carries an anonymous `csrftoken` + `mid` cookie pair and the `x-ig-app-id` web +header: > Warm an anonymous session with one plain GET to `www.instagram.com/` (mints -> `csrftoken` + `mid`), then GET the web JSON endpoints through that same +> `csrftoken` + `mid`), then GET the web endpoints through that same > Chrome-impersonated, sticky-IP session. Rotate the residential IP + re-warm on > a login wall (401/403), back off on 429. -Endpoints used (anonymous web tier only): +Surfaces used: -| Flow | Endpoint | -|---|---| -| profile / posts / reels | `api/v1/users/web_profile_info/?username=…` | -| comments | `p//?__a=1&__d=dis` | -| hashtag | `api/v1/tags/web_info/?tag_name=…` | -| place | `api/v1/locations/web_info/?location_id=…` | -| discovery search | `web/search/topsearch/?query=…` | +| Flow | Surface | Extractor | +|---|---|---| +| profile / details | `api/v1/users/web_profile_info/?username=…` (JSON) | `parse_profile` | +| profile feed (posts/reels/mentions) | the media embedded in the same profile JSON | `parse_media` | +| single post / reel | `/p//` (HTML: ld+json + og-meta) | `parse_post` | +| profile discovery | Google `site:instagram.com ` | `resolve_url` | -**No browser, no Chromium, no `solve_cloudflare`** — this stays on the cheap HTTP -tier the sibling scrapers already use. +**Why anonymous-only is a hard constraint.** Live logged-out probes show that +Instagram walls the interesting endpoints for anyone without a `sessionid` +account cookie: `api/v1/tags/web_info/`, `api/v1/locations/web_info/`, the +comment thread API (`?__a=1`), and `web/search/topsearch/` all **302 to +`/accounts/login/`**. We cannot log in (see below), so hashtag feeds, place +feeds, comment scraping, and IG's native keyword search were **removed** — they +can only ever return a login wall. What survives is what a logged-out browser can +actually read: a profile's web info + its embedded recent media, and a public +post/reel page's embedded metadata. ## Anonymous only — no authentication, ever @@ -47,59 +54,88 @@ so the capability layer can map it to a `403 INSTAGRAM_ACCESS_BLOCKED`. | File | Responsibility | |---|---| | `__init__.py` | Public exports: `InstagramScrapeInput`, item models, `iter_instagram`, `scrape_instagram`, `InstagramAccessBlockedError`. | -| `schemas.py` | `InstagramScrapeInput` (`extra="allow"`, no auth fields) + optional-field item models (`InstagramMediaItem`, `InstagramComment`, `InstagramProfile`, `InstagramHashtag`, `InstagramPlace`) each with `to_output()`. | -| `fetch.py` | The core. Rotate-on-block sticky `_RotatingSession` + `_current_session` ContextVar + `warm_session` (csrftoken/mid) + `fetch_json`. No browser imports. | -| `url_resolver.py` | Classify an Instagram URL → `profile`/`post`/`reel`/`hashtag`/`place`; non-Instagram → `None`. Strips `_u/`, `profilecard/`; story → profile. | -| `parsers.py` | Pure JSON→dict mapping (`parse_media`, `parse_comment`, `parse_profile`, `parse_hashtag`, `parse_place`, `_edges`). I/O-free. | -| `scraper.py` | Orchestrator: `_media_flow`/`_comments_flow`/`_details_flow`/`_discover`, `_targets`, `fan_out`, `iter_instagram`, `scrape_instagram`. | +| `schemas.py` | `InstagramScrapeInput` (`extra="allow"`, no auth fields) + optional-field item models (`InstagramMediaItem`, `InstagramProfile`) each with `to_output()`. | +| `fetch.py` | The core. Rotate-on-block sticky `_RotatingSession` + `_current_session` ContextVar + `warm_session` (csrftoken/mid) + `fetch_json` (JSON) / `fetch_html` (HTML) sharing one resilient `_fetch(path, params, extract)` loop. | +| `url_resolver.py` | Classify an Instagram URL → `profile`/`post`/`reel`; non-Instagram (and hashtag/place) → `None`. Strips `_u/`, `profilecard/`; story → profile. | +| `parsers.py` | Pure mapping (`parse_media`, `parse_profile`, `parse_post` [ld+json/og], `_edges`). I/O-free. | +| `scraper.py` | Orchestrator: `_media_flow`/`_details_flow`/`_discover` (+ `_discover_via_google`), `_targets`, `fan_out`, `iter_instagram`, `scrape_instagram`. | ## How it works 1. `iter_instagram` resolves `directUrls` (or runs a discovery `search` per the comma-split queries) into targets and fans them out on a pool of warm proxy - sessions (`fan_out`, 8-way; 4-way for comments). Each worker opens one - sticky-IP session and warms `csrftoken`/`mid` once, reusing it across the - sequential targets it pulls. -2. `resultsType` selects the flow: `posts`/`reels`/`mentions` → media feeds, - `comments` → per-post comment items, `details` → profile/hashtag/place - metadata. Media items de-dupe by `id` across targets. -3. `fetch_json` warms the session on first use, rotates the IP + re-warms on - 401/403, backs off on 429, returns `None` on 404. -4. Parsers map raw web JSON to flat dicts; the orchestrator stamps `scrapedAt` - and applies `resultsLimit` / `onlyPostsNewerThan` as request-time policy. + sessions (`fan_out`, 8-way). Each worker opens one sticky-IP session and warms + `csrftoken`/`mid` once, reusing it across the sequential targets it pulls. +2. `resultsType` selects the flow: `posts`/`reels`/`mentions` → media items, + `details` → profile metadata. Media items de-dupe by `id` across targets. + - A **profile** target → `web_profile_info` JSON → `parse_media` over the + embedded recent-media edges (feed) or `parse_profile` (details). + - A **post/reel** target → `fetch_html("p//")` → `parse_post`, which + reads the page's `application/ld+json` (preferred) and Open Graph meta + (fallback). Numeric-ID post URLs are skipped (the page keys on the shortCode). +3. `fetch_json` / `fetch_html` warm the session on first use, rotate the IP + + re-warm on 401/403, back off on 429, return `None` on 404, and raise + `InstagramAccessBlockedError` on a `/accounts/login/` redirect. +4. Parsers map raw web JSON/HTML to flat dicts; the orchestrator stamps + `scrapedAt` and applies `resultsLimit` / `onlyPostsNewerThan` as request-time + policy. + +## Profile discovery (Google-backed) + +Instagram's native keyword search is login-walled, so `_discover` resolves a +query that is a valid handle directly (`"messi"` → `instagram.com/messi/`) and +routes any other query (e.g. `"national geographic"`) through +`_discover_via_google`, which calls the `google_search` platform with +`site:instagram.com`, classifies each organic URL with `resolve_url`, keeps the +**profile** hits (discovery is profile-only), de-dupes, and caps at `searchLimit`. + +Caveats: + +- **Coupling**: Instagram depends on the `google_search` platform. The dependency + is one-directional and lives behind `_discover_via_google` so it stays testable. +- **Quality**: results reflect Google's index/ranking of `instagram.com`, not + IG's own relevance. This is discovery, not search parity. ## Observed limits & calibration caveats -- Anonymous web JSON is rate-limited per IP; the sticky-session pool keeps each - IP's request rate modest but a hot pool will still hit login walls — that's the - `InstagramAccessBlockedError` path, not a bug. +- Anonymous web JSON/HTML is rate-limited per IP; the sticky-session pool keeps + each IP's request rate modest but a hot pool will still hit login walls — that's + the `InstagramAccessBlockedError` path, not a bug. - `likesCount` is frequently withheld on anonymous responses (surfaces as `-1` or absent upstream); treat it as best-effort. -- Comments on the anonymous media page cap at ~50/post; deeper paging needs the - GraphQL cursor endpoint whose doc-id drifts (see the `ponytail:` note in - `scraper.py`/`fetch.py`). +- **Single-post extraction** reads whatever the public `/p/` document embeds + (ld+json + og-meta). If Instagram strips both for a given post (private, taken + down, or a login interstitial), `parse_post` returns `None` — an honest empty, + never a fabricated item. ponytail: the embedded-blob shapes can drift; a live + probe that dumps the raw HTML pins them (see Testing) and any change is contained + to `parse_post`. - The `$3.50 / 1k items` default meter assumes the proxy-bytes-per-item measured - on the reference targets; re-measure with the `references/` scale harness before - high-volume production use. + on the reference targets; re-measure with the scale harness before high-volume use. ## Testing - Offline unit tests: `tests/unit/platforms/instagram/` — `test_skeleton.py` - (schema + URL resolver), `test_parsers.py` (fixture-pinned mapping), - `test_fetch_resilience.py` (warm / rotate / backoff loop with fake sessions, no - network), `test_budget.py` (fair-share caps + de-dup). -- Live e2e (needs network + residential proxy): `scripts/e2e_instagram_scraper.py` - — step 0 is the go/no-go cookie probe; later steps exercise the flows and dump - trimmed, PII-anonymized fixtures. + (schema + URL resolver), `test_parsers.py` (mapping incl. `parse_post` + ld+json/og shapes; fixture-pinned tests skip when the fixture is absent), + `test_discovery.py` (Google-backed profile discovery with a fake `scrape_serps`), + `test_fetch_resilience.py` (warm / rotate / backoff loop + fan-out with fake + sessions, no network), `test_budget.py` (fair-share caps + de-dup). +- Stress / accuracy harness (live, needs network + residential proxy): + `scripts/stress/stress_instagram_scraper.py` — `--mode live-discovery` (profile + discovery accuracy), `--mode probe-post` (dumps a real anonymous `/p/` payload + to `fixtures/post.json` and shows what `parse_post` extracted), and + `--mode accuracy` (field coverage across the profile + single-post flows). ```bash cd surfsense_backend -.venv/bin/python -m pytest tests/unit/platforms/instagram/ -.venv/bin/python scripts/e2e_instagram_scraper.py # live; regenerates fixtures +uv run pytest tests/unit/platforms/instagram/ +# Live single-post probe: confirms /p/ is anonymously extractable + pins the shape +uv run python scripts/stress/stress_instagram_scraper.py --mode probe-post \ + --post https://www.instagram.com/p// ``` ## TODO / out of scope (v1) -- Deep feed pagination past the first web page of media (GraphQL cursor doc-id). -- Deep comment pagination past the ~50/post embedded ceiling. +- Deep feed pagination past the first web page of profile media (GraphQL cursor + doc-id). - Sticky-IP provider parity (same `__sid` caveat as the Reddit sibling).