mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-24 23:41:10 +02:00
docs(walmart): add connector docs and list in indexes
This commit is contained in:
parent
2a512e0015
commit
6c98bd33a8
5 changed files with 80 additions and 3 deletions
|
|
@ -12,7 +12,7 @@ Connectors bring data into SurfSense — either as searchable knowledge or as li
|
|||
<Card
|
||||
icon={<Zap />}
|
||||
title="Native Connectors"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
href="/docs/connectors/native"
|
||||
/>
|
||||
<Card
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: Native Connectors
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, and the web
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, and the web
|
||||
---
|
||||
|
||||
import { Card, Cards } from 'fumadocs-ui/components/card';
|
||||
|
|
@ -43,6 +43,11 @@ Native connectors are SurfSense's own scraper APIs — built into the platform,
|
|||
description="Public product data: prices, ratings, offers, sellers, and best-seller ranks"
|
||||
href="/docs/connectors/native/amazon"
|
||||
/>
|
||||
<Card
|
||||
title="Walmart"
|
||||
description="Public product data and deep reviews: prices, ratings, sellers, and variants"
|
||||
href="/docs/connectors/native/walmart"
|
||||
/>
|
||||
<Card
|
||||
title="Web Crawl"
|
||||
description="Scrape any page or spider a whole site into clean markdown"
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@
|
|||
"google-maps",
|
||||
"google-search",
|
||||
"amazon",
|
||||
"walmart",
|
||||
"web-crawl"
|
||||
],
|
||||
"defaultOpen": false
|
||||
|
|
|
|||
71
surfsense_web/content/docs/connectors/native/walmart.mdx
Normal file
71
surfsense_web/content/docs/connectors/native/walmart.mdx
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
---
|
||||
title: Walmart
|
||||
description: Scrape public Walmart product data and deep reviews as structured JSON
|
||||
---
|
||||
|
||||
The Walmart scraper returns public product data as structured JSON: price and list price, rating and review count, availability, 1P and marketplace sellers, variants, specifications, and a free on-page review sample. A second verb pages the full public review history for a product. It only uses public, anonymous data — no login or seller account. Give it search terms or Walmart product, search, category, or browse URLs.
|
||||
|
||||
Walmart data is the US marketplace (`walmart.com`). The scraper reads the JSON Walmart server-renders into each page rather than the rendered DOM, which keeps extraction stable against the frequent front-end A/B testing on the site.
|
||||
|
||||
## Scrape products
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/walmart/scrape
|
||||
```
|
||||
|
||||
### Inputs
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `urls` | — | Walmart product (`/ip/`), search (`/search`), category (`/cp/`), or browse (`/browse/`) URLs. Provide `urls` or `search_terms` |
|
||||
| `search_terms` | — | Search phrases run on `walmart.com`, e.g. `air fryer`. Provide `search_terms` or `urls` |
|
||||
| `max_items` | `10` | Max products per search term or category/browse URL (1–100) |
|
||||
| `include_details` | `true` | Fetch full product detail pages; `false` returns faster card-only results |
|
||||
| `include_reviews_sample` | `true` | Attach the free on-page review sample from each detail page. For full history use the reviews verb |
|
||||
|
||||
At least one of `urls` or `search_terms` is required, with up to 20 combined sources per call.
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/walmart/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_terms": ["air fryer"],
|
||||
"max_items": 5,
|
||||
"include_reviews_sample": true
|
||||
}'
|
||||
```
|
||||
|
||||
The response is `{ "items": [...] }` — one item per product, plus structured error items for any input that could not be resolved. Billing is per returned product; error items are never billed.
|
||||
|
||||
## Fetch reviews
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/walmart/reviews
|
||||
```
|
||||
|
||||
### Inputs
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `urls` | — | Walmart product (`/ip/`) or reviews URLs to fetch reviews for. Provide `urls` or `item_ids` |
|
||||
| `item_ids` | — | Walmart numeric item ids (`usItemId`) to fetch reviews for. Provide `item_ids` or `urls` |
|
||||
| `max_reviews` | `200` | Max reviews per product (1–5000); reviews are paged 10 at a time |
|
||||
| `sort_by` | `most-recent` | Review ordering: `most-recent`, `most-helpful`, `rating-high`, or `rating-low` |
|
||||
|
||||
At least one of `urls` or `item_ids` is required, with up to 20 combined sources per call.
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/walmart/reviews" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"urls": ["https://www.walmart.com/ip/1234567890"],
|
||||
"max_reviews": 500,
|
||||
"sort_by": "most-recent"
|
||||
}'
|
||||
```
|
||||
|
||||
The response is `{ "items": [...] }` — one item per review with rating, title, text, author, date, verified-purchase flag, helpful votes, images, and any seller response. Billing is per returned review; error items are never billed.
|
||||
|
||||
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Walmart** in your workspace.
|
||||
|
|
@ -7,7 +7,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
|
|||
|
||||
# SurfSense MCP Server
|
||||
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 25 native, typed tools: every scraper (Reddit, YouTube, Instagram, TikTok, Amazon, Google Maps, Google Search, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 27 native, typed tools: every scraper (Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
|
||||
Connect it two ways: the **hosted** server at `https://mcp.surfsense.com/mcp` (nothing to install — just an API key), or run it yourself over **stdio** against any SurfSense backend, cloud or self-hosted.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue