mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-16 23:01:06 +02:00
docs(instagram): update platform scraper README
This commit is contained in:
Anish Sarkar
parent
4813dc96e3
commit
36bb4a1bea
1 changed files with 92 additions and 56 deletions
|
|
@ -1,37 +1,44 @@
|
|||
# Instagram scraper (anonymous, no browser)
|
||||
# Instagram scraper (anonymous)
|
||||
|
||||
Platform-native Instagram scraper (anonymous, no browser). Standalone module: it
|
||||
depends only on `app.utils.proxy` + `scrapling` and exposes a stable public API.
|
||||
Its input/output surface is a **reference-compatible** mirror of the public
|
||||
Instagram scraper actor spec (same `resultsType` / `directUrls` / camelCase field
|
||||
names, additive `extra="allow"` parity), so callers written against that surface
|
||||
work unchanged. It is **not** wired into ingestion or Celery — the capability
|
||||
layer under `app/capabilities/instagram/` is what turns these primitives into
|
||||
REST + agent + MCP surfaces.
|
||||
Platform-native Instagram scraper. **Anonymous-only** and browser-free: every
|
||||
flow stays on the cheap HTTP tier (`app.utils.proxy` + `scrapling`), and profile
|
||||
discovery reuses the `google_search` platform (see below). It exposes a stable
|
||||
public API whose input/output surface mirrors the public Instagram scraper actor
|
||||
spec (same `resultsType` / `directUrls` / camelCase field names, additive
|
||||
`extra="allow"` parity), so callers written against that surface work unchanged.
|
||||
It is **not** wired into ingestion or Celery — the capability layer under
|
||||
`app/capabilities/instagram/` turns these primitives into REST + agent + MCP
|
||||
surfaces.
|
||||
|
||||
## Approach
|
||||
|
||||
Instagram's public web app exposes anonymous, logged-out JSON behind a handful of
|
||||
`www.instagram.com` endpoints once a session carries an anonymous `csrftoken` +
|
||||
`mid` cookie pair and the `x-ig-app-id` web header:
|
||||
Instagram's public web app exposes anonymous, logged-out data once a session
|
||||
carries an anonymous `csrftoken` + `mid` cookie pair and the `x-ig-app-id` web
|
||||
header:
|
||||
|
||||
> Warm an anonymous session with one plain GET to `www.instagram.com/` (mints
|
||||
> `csrftoken` + `mid`), then GET the web JSON endpoints through that same
|
||||
> `csrftoken` + `mid`), then GET the web endpoints through that same
|
||||
> Chrome-impersonated, sticky-IP session. Rotate the residential IP + re-warm on
|
||||
> a login wall (401/403), back off on 429.
|
||||
|
||||
Endpoints used (anonymous web tier only):
|
||||
Surfaces used:
|
||||
|
||||
| Flow | Endpoint |
|
||||
|---|---|
|
||||
| profile / posts / reels | `api/v1/users/web_profile_info/?username=…` |
|
||||
| comments | `p/<shortcode>/?__a=1&__d=dis` |
|
||||
| hashtag | `api/v1/tags/web_info/?tag_name=…` |
|
||||
| place | `api/v1/locations/web_info/?location_id=…` |
|
||||
| discovery search | `web/search/topsearch/?query=…` |
|
||||
| Flow | Surface | Extractor |
|
||||
|---|---|---|
|
||||
| profile / details | `api/v1/users/web_profile_info/?username=…` (JSON) | `parse_profile` |
|
||||
| profile feed (posts/reels/mentions) | the media embedded in the same profile JSON | `parse_media` |
|
||||
| single post / reel | `/p/<shortcode>/` (HTML: ld+json + og-meta) | `parse_post` |
|
||||
| profile discovery | Google `site:instagram.com <query>` | `resolve_url` |
|
||||
|
||||
**No browser, no Chromium, no `solve_cloudflare`** — this stays on the cheap HTTP
|
||||
tier the sibling scrapers already use.
|
||||
**Why anonymous-only is a hard constraint.** Live logged-out probes show that
|
||||
Instagram walls the interesting endpoints for anyone without a `sessionid`
|
||||
account cookie: `api/v1/tags/web_info/`, `api/v1/locations/web_info/`, the
|
||||
comment thread API (`?__a=1`), and `web/search/topsearch/` all **302 to
|
||||
`/accounts/login/`**. We cannot log in (see below), so hashtag feeds, place
|
||||
feeds, comment scraping, and IG's native keyword search were **removed** — they
|
||||
can only ever return a login wall. What survives is what a logged-out browser can
|
||||
actually read: a profile's web info + its embedded recent media, and a public
|
||||
post/reel page's embedded metadata.
|
||||
|
||||
## Anonymous only — no authentication, ever
|
||||
|
||||
|
|
@ -47,59 +54,88 @@ so the capability layer can map it to a `403 INSTAGRAM_ACCESS_BLOCKED`.
|
|||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `__init__.py` | Public exports: `InstagramScrapeInput`, item models, `iter_instagram`, `scrape_instagram`, `InstagramAccessBlockedError`. |
|
||||
| `schemas.py` | `InstagramScrapeInput` (`extra="allow"`, no auth fields) + optional-field item models (`InstagramMediaItem`, `InstagramComment`, `InstagramProfile`, `InstagramHashtag`, `InstagramPlace`) each with `to_output()`. |
|
||||
| `fetch.py` | The core. Rotate-on-block sticky `_RotatingSession` + `_current_session` ContextVar + `warm_session` (csrftoken/mid) + `fetch_json`. No browser imports. |
|
||||
| `url_resolver.py` | Classify an Instagram URL → `profile`/`post`/`reel`/`hashtag`/`place`; non-Instagram → `None`. Strips `_u/`, `profilecard/`; story → profile. |
|
||||
| `parsers.py` | Pure JSON→dict mapping (`parse_media`, `parse_comment`, `parse_profile`, `parse_hashtag`, `parse_place`, `_edges`). I/O-free. |
|
||||
| `scraper.py` | Orchestrator: `_media_flow`/`_comments_flow`/`_details_flow`/`_discover`, `_targets`, `fan_out`, `iter_instagram`, `scrape_instagram`. |
|
||||
| `schemas.py` | `InstagramScrapeInput` (`extra="allow"`, no auth fields) + optional-field item models (`InstagramMediaItem`, `InstagramProfile`) each with `to_output()`. |
|
||||
| `fetch.py` | The core. Rotate-on-block sticky `_RotatingSession` + `_current_session` ContextVar + `warm_session` (csrftoken/mid) + `fetch_json` (JSON) / `fetch_html` (HTML) sharing one resilient `_fetch(path, params, extract)` loop. |
|
||||
| `url_resolver.py` | Classify an Instagram URL → `profile`/`post`/`reel`; non-Instagram (and hashtag/place) → `None`. Strips `_u/`, `profilecard/`; story → profile. |
|
||||
| `parsers.py` | Pure mapping (`parse_media`, `parse_profile`, `parse_post` [ld+json/og], `_edges`). I/O-free. |
|
||||
| `scraper.py` | Orchestrator: `_media_flow`/`_details_flow`/`_discover` (+ `_discover_via_google`), `_targets`, `fan_out`, `iter_instagram`, `scrape_instagram`. |
|
||||
|
||||
## How it works
|
||||
|
||||
1. `iter_instagram` resolves `directUrls` (or runs a discovery `search` per the
|
||||
comma-split queries) into targets and fans them out on a pool of warm proxy
|
||||
sessions (`fan_out`, 8-way; 4-way for comments). Each worker opens one
|
||||
sticky-IP session and warms `csrftoken`/`mid` once, reusing it across the
|
||||
sequential targets it pulls.
|
||||
2. `resultsType` selects the flow: `posts`/`reels`/`mentions` → media feeds,
|
||||
`comments` → per-post comment items, `details` → profile/hashtag/place
|
||||
metadata. Media items de-dupe by `id` across targets.
|
||||
3. `fetch_json` warms the session on first use, rotates the IP + re-warms on
|
||||
401/403, backs off on 429, returns `None` on 404.
|
||||
4. Parsers map raw web JSON to flat dicts; the orchestrator stamps `scrapedAt`
|
||||
and applies `resultsLimit` / `onlyPostsNewerThan` as request-time policy.
|
||||
sessions (`fan_out`, 8-way). Each worker opens one sticky-IP session and warms
|
||||
`csrftoken`/`mid` once, reusing it across the sequential targets it pulls.
|
||||
2. `resultsType` selects the flow: `posts`/`reels`/`mentions` → media items,
|
||||
`details` → profile metadata. Media items de-dupe by `id` across targets.
|
||||
- A **profile** target → `web_profile_info` JSON → `parse_media` over the
|
||||
embedded recent-media edges (feed) or `parse_profile` (details).
|
||||
- A **post/reel** target → `fetch_html("p/<code>/")` → `parse_post`, which
|
||||
reads the page's `application/ld+json` (preferred) and Open Graph meta
|
||||
(fallback). Numeric-ID post URLs are skipped (the page keys on the shortCode).
|
||||
3. `fetch_json` / `fetch_html` warm the session on first use, rotate the IP +
|
||||
re-warm on 401/403, back off on 429, return `None` on 404, and raise
|
||||
`InstagramAccessBlockedError` on a `/accounts/login/` redirect.
|
||||
4. Parsers map raw web JSON/HTML to flat dicts; the orchestrator stamps
|
||||
`scrapedAt` and applies `resultsLimit` / `onlyPostsNewerThan` as request-time
|
||||
policy.
|
||||
|
||||
## Profile discovery (Google-backed)
|
||||
|
||||
Instagram's native keyword search is login-walled, so `_discover` resolves a
|
||||
query that is a valid handle directly (`"messi"` → `instagram.com/messi/`) and
|
||||
routes any other query (e.g. `"national geographic"`) through
|
||||
`_discover_via_google`, which calls the `google_search` platform with
|
||||
`site:instagram.com`, classifies each organic URL with `resolve_url`, keeps the
|
||||
**profile** hits (discovery is profile-only), de-dupes, and caps at `searchLimit`.
|
||||
|
||||
Caveats:
|
||||
|
||||
- **Coupling**: Instagram depends on the `google_search` platform. The dependency
|
||||
is one-directional and lives behind `_discover_via_google` so it stays testable.
|
||||
- **Quality**: results reflect Google's index/ranking of `instagram.com`, not
|
||||
IG's own relevance. This is discovery, not search parity.
|
||||
|
||||
## Observed limits & calibration caveats
|
||||
|
||||
- Anonymous web JSON is rate-limited per IP; the sticky-session pool keeps each
|
||||
IP's request rate modest but a hot pool will still hit login walls — that's the
|
||||
`InstagramAccessBlockedError` path, not a bug.
|
||||
- Anonymous web JSON/HTML is rate-limited per IP; the sticky-session pool keeps
|
||||
each IP's request rate modest but a hot pool will still hit login walls — that's
|
||||
the `InstagramAccessBlockedError` path, not a bug.
|
||||
- `likesCount` is frequently withheld on anonymous responses (surfaces as `-1` or
|
||||
absent upstream); treat it as best-effort.
|
||||
- Comments on the anonymous media page cap at ~50/post; deeper paging needs the
|
||||
GraphQL cursor endpoint whose doc-id drifts (see the `ponytail:` note in
|
||||
`scraper.py`/`fetch.py`).
|
||||
- **Single-post extraction** reads whatever the public `/p/` document embeds
|
||||
(ld+json + og-meta). If Instagram strips both for a given post (private, taken
|
||||
down, or a login interstitial), `parse_post` returns `None` — an honest empty,
|
||||
never a fabricated item. ponytail: the embedded-blob shapes can drift; a live
|
||||
probe that dumps the raw HTML pins them (see Testing) and any change is contained
|
||||
to `parse_post`.
|
||||
- The `$3.50 / 1k items` default meter assumes the proxy-bytes-per-item measured
|
||||
on the reference targets; re-measure with the `references/` scale harness before
|
||||
high-volume production use.
|
||||
on the reference targets; re-measure with the scale harness before high-volume use.
|
||||
|
||||
## Testing
|
||||
|
||||
- Offline unit tests: `tests/unit/platforms/instagram/` — `test_skeleton.py`
|
||||
(schema + URL resolver), `test_parsers.py` (fixture-pinned mapping),
|
||||
`test_fetch_resilience.py` (warm / rotate / backoff loop with fake sessions, no
|
||||
network), `test_budget.py` (fair-share caps + de-dup).
|
||||
- Live e2e (needs network + residential proxy): `scripts/e2e_instagram_scraper.py`
|
||||
— step 0 is the go/no-go cookie probe; later steps exercise the flows and dump
|
||||
trimmed, PII-anonymized fixtures.
|
||||
(schema + URL resolver), `test_parsers.py` (mapping incl. `parse_post`
|
||||
ld+json/og shapes; fixture-pinned tests skip when the fixture is absent),
|
||||
`test_discovery.py` (Google-backed profile discovery with a fake `scrape_serps`),
|
||||
`test_fetch_resilience.py` (warm / rotate / backoff loop + fan-out with fake
|
||||
sessions, no network), `test_budget.py` (fair-share caps + de-dup).
|
||||
- Stress / accuracy harness (live, needs network + residential proxy):
|
||||
`scripts/stress/stress_instagram_scraper.py` — `--mode live-discovery` (profile
|
||||
discovery accuracy), `--mode probe-post` (dumps a real anonymous `/p/` payload
|
||||
to `fixtures/post.json` and shows what `parse_post` extracted), and
|
||||
`--mode accuracy` (field coverage across the profile + single-post flows).
|
||||
|
||||
```bash
|
||||
cd surfsense_backend
|
||||
.venv/bin/python -m pytest tests/unit/platforms/instagram/
|
||||
.venv/bin/python scripts/e2e_instagram_scraper.py # live; regenerates fixtures
|
||||
uv run pytest tests/unit/platforms/instagram/
|
||||
# Live single-post probe: confirms /p/ is anonymously extractable + pins the shape
|
||||
uv run python scripts/stress/stress_instagram_scraper.py --mode probe-post \
|
||||
--post https://www.instagram.com/p/<shortcode>/
|
||||
```
|
||||
|
||||
## TODO / out of scope (v1)
|
||||
|
||||
- Deep feed pagination past the first web page of media (GraphQL cursor doc-id).
|
||||
- Deep comment pagination past the ~50/post embedded ceiling.
|
||||
- Deep feed pagination past the first web page of profile media (GraphQL cursor
|
||||
doc-id).
|
||||
- Sticky-IP provider parity (same `__sid` caveat as the Reddit sibling).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue