From 118411ebca84c06286fb1224da0f56fbef12cd35 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Wed, 15 Jul 2026 01:56:34 +0530 Subject: [PATCH] docs(amazon): add module status readme --- .../proprietary/platforms/amazon/README.md | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 surfsense_backend/app/proprietary/platforms/amazon/README.md diff --git a/surfsense_backend/app/proprietary/platforms/amazon/README.md b/surfsense_backend/app/proprietary/platforms/amazon/README.md new file mode 100644 index 000000000..83cb210ac --- /dev/null +++ b/surfsense_backend/app/proprietary/platforms/amazon/README.md @@ -0,0 +1,50 @@ +# Amazon Product Scraper + +Current status: product details, search and category discovery, offers, sellers, +best-seller rankings, on-page reviews, localized delivery sessions, and product +variants are implemented. The `amazon.scrape` capability exposes the scraper +with per-product billing and a hard per-run result ceiling. + +## Scope + +The scraper reads public pages available to anonymous visitors. It does not log +in, use account cookies, or retrieve account-gated content. Delivery +localization uses an anonymous session cookie pinned to the same proxy exit. + +Deep review pagination currently requires an account on Amazon and is therefore +out of scope. Reviews embedded in a public product page are returned in +`productPageReviews` and `productPageReviewsFromOtherCountries`. + +## Architecture + +- `schemas.py` defines the stable input, product, and error models. +- `url_resolver.py` classifies product, search, best-sellers, and shortened URLs. +- `fetch.py` owns proxy-aware HTTP access, block detection, retries, and + anonymous location sessions. +- `parsers.py` contains pure, defensive HTML parsers. +- `scraper.py` coordinates discovery, enrichment, concurrency, limits, and + in-stream errors. + +## Implementation progress + +- Done: public product detail parsing and shortened-link dispatch. +- Done: search/category paging with card-only and detailed modes. +- Done: public offers and seller enrichment. +- Done: best-seller rankings and product-page reviews. +- Done: anonymous localized delivery sessions on sticky proxy exits. +- Done: variant expansion, variant prices, capability registration, and billing. + +## Verification + +Offline fixtures cover every parser and flow: + +```bash +cd surfsense_backend +uv run pytest tests/unit/platforms/amazon tests/unit/capabilities/amazon +``` + +The manual live check requires proxy credentials: + +```bash +uv run python scripts/e2e_amazon_scraper.py +```