diff --git a/surfsense_backend/app/proprietary/platforms/instagram/README.md b/surfsense_backend/app/proprietary/platforms/instagram/README.md new file mode 100644 index 000000000..fa42e30a6 --- /dev/null +++ b/surfsense_backend/app/proprietary/platforms/instagram/README.md @@ -0,0 +1,105 @@ +# Instagram scraper (anonymous, no browser) + +Platform-native Instagram scraper (anonymous, no browser). Standalone module: it +depends only on `app.utils.proxy` + `scrapling` and exposes a stable public API. +Its input/output surface is a **reference-compatible** mirror of the public +Instagram scraper actor spec (same `resultsType` / `directUrls` / camelCase field +names, additive `extra="allow"` parity), so callers written against that surface +work unchanged. It is **not** wired into ingestion or Celery — the capability +layer under `app/capabilities/instagram/` is what turns these primitives into +REST + agent + MCP surfaces. + +## Approach + +Instagram's public web app exposes anonymous, logged-out JSON behind a handful of +`www.instagram.com` endpoints once a session carries an anonymous `csrftoken` + +`mid` cookie pair and the `x-ig-app-id` web header: + +> Warm an anonymous session with one plain GET to `www.instagram.com/` (mints +> `csrftoken` + `mid`), then GET the web JSON endpoints through that same +> Chrome-impersonated, sticky-IP session. Rotate the residential IP + re-warm on +> a login wall (401/403), back off on 429. + +Endpoints used (anonymous web tier only): + +| Flow | Endpoint | +|---|---| +| profile / posts / reels | `api/v1/users/web_profile_info/?username=…` | +| comments | `p//?__a=1&__d=dis` | +| hashtag | `api/v1/tags/web_info/?tag_name=…` | +| place | `api/v1/locations/web_info/?location_id=…` | +| discovery search | `web/search/topsearch/?query=…` | + +**No browser, no Chromium, no `solve_cloudflare`** — this stays on the cheap HTTP +tier the sibling scrapers already use. + +## Anonymous only — no authentication, ever + +No login, no `sessionid` account cookie, no app password. The only cookies held +are the anonymous `csrftoken` / `mid` minted by the warm-up. There is **no** +authenticate option in the input surface or the fetch layer, by design. A +persistent block after IP rotation surfaces as `InstagramAccessBlockedError` +(mirrors Reddit's `RedditAccessBlockedError`) rather than a silent empty result, +so the capability layer can map it to a `403 INSTAGRAM_ACCESS_BLOCKED`. + +## Module map + +| File | Responsibility | +|---|---| +| `__init__.py` | Public exports: `InstagramScrapeInput`, item models, `iter_instagram`, `scrape_instagram`, `InstagramAccessBlockedError`. | +| `schemas.py` | `InstagramScrapeInput` (`extra="allow"`, no auth fields) + optional-field item models (`InstagramMediaItem`, `InstagramComment`, `InstagramProfile`, `InstagramHashtag`, `InstagramPlace`) each with `to_output()`. | +| `fetch.py` | The core. Rotate-on-block sticky `_RotatingSession` + `_current_session` ContextVar + `warm_session` (csrftoken/mid) + `fetch_json`. No browser imports. | +| `url_resolver.py` | Classify an Instagram URL → `profile`/`post`/`reel`/`hashtag`/`place`; non-Instagram → `None`. Strips `_u/`, `profilecard/`; story → profile. | +| `parsers.py` | Pure JSON→dict mapping (`parse_media`, `parse_comment`, `parse_profile`, `parse_hashtag`, `parse_place`, `_edges`). I/O-free. | +| `scraper.py` | Orchestrator: `_media_flow`/`_comments_flow`/`_details_flow`/`_discover`, `_targets`, `fan_out`, `iter_instagram`, `scrape_instagram`. | + +## How it works + +1. `iter_instagram` resolves `directUrls` (or runs a discovery `search` per the + comma-split queries) into targets and fans them out on a pool of warm proxy + sessions (`fan_out`, 8-way; 4-way for comments). Each worker opens one + sticky-IP session and warms `csrftoken`/`mid` once, reusing it across the + sequential targets it pulls. +2. `resultsType` selects the flow: `posts`/`reels`/`mentions` → media feeds, + `comments` → per-post comment items, `details` → profile/hashtag/place + metadata. Media items de-dupe by `id` across targets. +3. `fetch_json` warms the session on first use, rotates the IP + re-warms on + 401/403, backs off on 429, returns `None` on 404. +4. Parsers map raw web JSON to flat dicts; the orchestrator stamps `scrapedAt` + and applies `resultsLimit` / `onlyPostsNewerThan` as request-time policy. + +## Observed limits & calibration caveats + +- Anonymous web JSON is rate-limited per IP; the sticky-session pool keeps each + IP's request rate modest but a hot pool will still hit login walls — that's the + `InstagramAccessBlockedError` path, not a bug. +- `likesCount` is frequently withheld on anonymous responses (surfaces as `-1` or + absent upstream); treat it as best-effort. +- Comments on the anonymous media page cap at ~50/post; deeper paging needs the + GraphQL cursor endpoint whose doc-id drifts (see the `ponytail:` note in + `scraper.py`/`fetch.py`). +- The `$3.50 / 1k items` default meter assumes the proxy-bytes-per-item measured + on the reference targets; re-measure with the `references/` scale harness before + high-volume production use. + +## Testing + +- Offline unit tests: `tests/unit/platforms/instagram/` — `test_skeleton.py` + (schema + URL resolver), `test_parsers.py` (fixture-pinned mapping), + `test_fetch_resilience.py` (warm / rotate / backoff loop with fake sessions, no + network), `test_budget.py` (fair-share caps + de-dup). +- Live e2e (needs network + residential proxy): `scripts/e2e_instagram_scraper.py` + — step 0 is the go/no-go cookie probe; later steps exercise the flows and dump + trimmed, PII-anonymized fixtures. + +```bash +cd surfsense_backend +.venv/bin/python -m pytest tests/unit/platforms/instagram/ +.venv/bin/python scripts/e2e_instagram_scraper.py # live; regenerates fixtures +``` + +## TODO / out of scope (v1) + +- Deep feed pagination past the first web page of media (GraphQL cursor doc-id). +- Deep comment pagination past the ~50/post embedded ceiling. +- Sticky-IP provider parity (same `__sid` caveat as the Reddit sibling).