apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-07-20 23:21:06 +02:00
Code Issues Projects Releases Packages Wiki Activity
SurfSense/surfsense_backend/app/proprietary/platforms/amazon
History
Anish Sarkar 01308055dd chore: ran linting
2026-07-15 15:48:54 +05:30
..
__init__.py feat(amazon): expose public module api 2026-07-15 01:56:29 +05:30
fetch.py chore: ran linting 2026-07-15 15:48:54 +05:30
locale.py feat(amazon): add marketplace locale mapping 2026-07-15 15:31:53 +05:30
parsers.py chore: ran linting 2026-07-15 15:48:54 +05:30
README.md docs(amazon): add module status readme 2026-07-15 01:56:34 +05:30
schemas.py feat(amazon): add scrape input and output schemas 2026-07-15 01:55:09 +05:30
scraper.py feat(amazon): apply marketplace locale across scraper flows 2026-07-15 15:32:21 +05:30
url_resolver.py feat(amazon): add start-url resolver 2026-07-15 01:55:15 +05:30

README.md

Amazon Product Scraper

Current status: product details, search and category discovery, offers, sellers, best-seller rankings, on-page reviews, localized delivery sessions, and product variants are implemented. The amazon.scrape capability exposes the scraper with per-product billing and a hard per-run result ceiling.

Scope

The scraper reads public pages available to anonymous visitors. It does not log in, use account cookies, or retrieve account-gated content. Delivery localization uses an anonymous session cookie pinned to the same proxy exit.

Deep review pagination currently requires an account on Amazon and is therefore out of scope. Reviews embedded in a public product page are returned in productPageReviews and productPageReviewsFromOtherCountries.

Architecture

  • schemas.py defines the stable input, product, and error models.
  • url_resolver.py classifies product, search, best-sellers, and shortened URLs.
  • fetch.py owns proxy-aware HTTP access, block detection, retries, and anonymous location sessions.
  • parsers.py contains pure, defensive HTML parsers.
  • scraper.py coordinates discovery, enrichment, concurrency, limits, and in-stream errors.

Implementation progress

  • Done: public product detail parsing and shortened-link dispatch.
  • Done: search/category paging with card-only and detailed modes.
  • Done: public offers and seller enrichment.
  • Done: best-seller rankings and product-page reviews.
  • Done: anonymous localized delivery sessions on sticky proxy exits.
  • Done: variant expansion, variant prices, capability registration, and billing.

Verification

Offline fixtures cover every parser and flow:

cd surfsense_backend
uv run pytest tests/unit/platforms/amazon tests/unit/capabilities/amazon

The manual live check requires proxy credentials:

uv run python scripts/e2e_amazon_scraper.py