mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-26 23:51:14 +02:00
Merge remote-tracking branch 'upstream/dev' into feature-indeed-jobs-scraper
# Conflicts: # README.es.md # README.hi.md # README.md # README.pt-BR.md # README.zh-CN.md # surfsense_backend/tests/unit/capabilities/google_maps/test_registry.py # surfsense_backend/tests/unit/capabilities/reddit/test_registry.py # surfsense_backend/tests/unit/capabilities/youtube/test_registry.py # surfsense_mcp/mcp_server/features/scrapers/__init__.py # surfsense_web/content/docs/connectors/index.mdx # surfsense_web/content/docs/connectors/native/index.mdx # surfsense_web/content/docs/connectors/native/meta.json # surfsense_web/content/docs/how-to/mcp-server.mdx # surfsense_web/lib/connectors-marketing/index.ts # surfsense_web/lib/playground/catalog.ts
This commit is contained in:
CREDO23
commit
91aa265afb
259 changed files with 13867 additions and 1883 deletions
|
|
@ -12,7 +12,7 @@ Connectors bring data into SurfSense — either as searchable knowledge or as li
|
|||
<Card
|
||||
icon={<Zap />}
|
||||
title="Native Connectors"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, Amazon, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
href="/docs/connectors/native"
|
||||
/>
|
||||
<Card
|
||||
|
|
|
|||
67
surfsense_web/content/docs/connectors/native/amazon.mdx
Normal file
67
surfsense_web/content/docs/connectors/native/amazon.mdx
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
---
|
||||
title: Amazon
|
||||
description: Scrape public Amazon product data as structured JSON
|
||||
---
|
||||
|
||||
The Amazon scraper returns public product data as structured JSON: price and list price, rating and review breakdown, availability, marketplace offers, sellers, best-seller ranks, product variants, and on-page reviews. It only uses public, anonymous data — no login or seller account. Give it search terms or Amazon product, search, category, or best-seller URLs.
|
||||
|
||||
Marketplace support is inferred from the Amazon URL or `domain` you provide. Supported public marketplaces include US (`amazon.com`), UK (`amazon.co.uk`), Germany (`amazon.de`), Italy (`amazon.it`), and Spain (`amazon.es`). France (`amazon.fr`) is supported in the scraper but remains best-effort because Amazon France is more sensitive to proxy-pool WAF challenges.
|
||||
|
||||
## Endpoint
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/amazon/scrape
|
||||
```
|
||||
|
||||
## Inputs
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `urls` | — | Amazon product, search, category, best-seller, or short (`a.co` / `amzn.to`) URLs. Marketplace is inferred from the URL, e.g. `amazon.co.uk`, `amazon.de`, `amazon.it`, or `amazon.es`. Provide `urls` or `search_terms` |
|
||||
| `search_terms` | — | Search phrases run on the selected `domain`, e.g. `wireless earbuds`. Provide `search_terms` or `urls` |
|
||||
| `max_items` | `10` | Max products per search term or category/best-seller URL (1–100) |
|
||||
| `domain` | `www.amazon.com` | Amazon marketplace domain for `search_terms`, e.g. `www.amazon.co.uk`, `www.amazon.de`, `www.amazon.it`, or `www.amazon.es`. `www.amazon.fr` is best-effort due to Amazon WAF sensitivity |
|
||||
| `include_details` | `true` | Fetch full product detail pages; `false` returns faster card-only results |
|
||||
| `max_offers` | `0` | Extra marketplace offers per product (0–100); `0` = featured offer only |
|
||||
| `include_sellers` | `false` | Enrich the product and each offer with the seller's public profile |
|
||||
| `max_variants` | `0` | Product variants to return as separate results (0–100) |
|
||||
| `include_variant_prices` | `false` | Attach per-variant prices (one extra request per variant) |
|
||||
| `country_code` | — | Two-letter delivery country for localized pricing, e.g. `us` |
|
||||
| `zip_code` | — | Delivery ZIP/postal code for localized availability, e.g. `10001` |
|
||||
| `language` | — | Content language for the domain, e.g. `en` |
|
||||
|
||||
At least one of `urls` or `search_terms` is required, with up to 20 combined sources per call.
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/amazon/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_terms": ["mechanical keyboard"],
|
||||
"max_items": 5,
|
||||
"max_offers": 3
|
||||
}'
|
||||
```
|
||||
|
||||
## Marketplace examples
|
||||
|
||||
```json
|
||||
{
|
||||
"urls": [
|
||||
"https://www.amazon.co.uk/s?k=usb+c+cable",
|
||||
"https://www.amazon.de/s?k=mechanische+tastatur",
|
||||
"https://www.amazon.it/s?k=cavo+usb+c",
|
||||
"https://www.amazon.es/s?k=cable+usb+c"
|
||||
],
|
||||
"max_items": 5,
|
||||
"include_details": false
|
||||
}
|
||||
```
|
||||
|
||||
For `search_terms`, set `domain` to the marketplace you want to search. For direct URLs, the scraper infers the marketplace from each URL.
|
||||
|
||||
The response is `{ "items": [...] }` — one item per product, plus structured error items for any input that could not be resolved. Billing is per returned product; error items are never billed.
|
||||
|
||||
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Amazon → Scrape** in your workspace.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: Native Connectors
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, and the web
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, Amazon, and the web
|
||||
---
|
||||
|
||||
import { Card, Cards } from 'fumadocs-ui/components/card';
|
||||
|
|
@ -43,6 +43,11 @@ Native connectors are SurfSense's own scraper APIs — built into the platform,
|
|||
description="Job postings by search, company, or URL — with salary and full description"
|
||||
href="/docs/connectors/native/indeed"
|
||||
/>
|
||||
<Card
|
||||
title="Amazon"
|
||||
description="Public product data: prices, ratings, offers, sellers, and best-seller ranks"
|
||||
href="/docs/connectors/native/amazon"
|
||||
/>
|
||||
<Card
|
||||
title="Web Crawl"
|
||||
description="Scrape any page or spider a whole site into clean markdown"
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@
|
|||
"google-maps",
|
||||
"google-search",
|
||||
"indeed",
|
||||
"amazon",
|
||||
"web-crawl"
|
||||
],
|
||||
"defaultOpen": false
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/web/crawl" \
|
|||
|
||||
One item per fetched page (in crawl order) with its markdown, metadata (title, description), crawl provenance (depth, referrer), links classified as internal/external/social/email/tel, and page-level contacts. The response also includes a site-wide `contacts` summary that deduplicates every email, phone, and social profile found — `siteWide: true` marks header/footer values (the company's own contacts) versus page-local finds like individual team members.
|
||||
|
||||
Contact details often live on about/contact/privacy pages, so crawl with `maxCrawlDepth >= 1` to surface them — useful for lead generation and competitive intelligence.
|
||||
Contact details often live on about/contact/privacy pages, so crawl with `maxCrawlDepth >= 1` to surface them — useful for lead generation and company research.
|
||||
|
||||
Billing is per successfully fetched page.
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
|
|||
|
||||
# SurfSense MCP Server
|
||||
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 25 native, typed tools: every scraper (Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 26 native, typed tools: every scraper (Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, Amazon, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
|
||||
Connect it two ways: the **hosted** server at `https://mcp.surfsense.com/mcp` (nothing to install — just an API key), or run it yourself over **stdio** against any SurfSense backend, cloud or self-hosted.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue