mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-26 23:51:14 +02:00
Merge remote-tracking branch 'upstream/dev' into feature-walmart-scraper
# Conflicts: # docker/.env.example # surfsense_backend/app/capabilities/core/billing.py # surfsense_backend/app/capabilities/core/types.py # surfsense_backend/app/config/__init__.py # surfsense_mcp/mcp_server/features/scrapers/__init__.py # surfsense_web/content/docs/connectors/index.mdx # surfsense_web/content/docs/connectors/native/index.mdx # surfsense_web/content/docs/how-to/mcp-server.mdx
This commit is contained in:
CREDO23
commit
ca317c4686
68 changed files with 4365 additions and 20 deletions
|
|
@ -12,7 +12,7 @@ Connectors bring data into SurfSense — either as searchable knowledge or as li
|
|||
<Card
|
||||
icon={<Zap />}
|
||||
title="Native Connectors"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
description="SurfSense's built-in scraper APIs: Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, Amazon, Walmart, and Web Crawl — usable in chat, the API Playground, or via REST"
|
||||
href="/docs/connectors/native"
|
||||
/>
|
||||
<Card
|
||||
|
|
|
|||
51
surfsense_web/content/docs/connectors/native/indeed.mdx
Normal file
51
surfsense_web/content/docs/connectors/native/indeed.mdx
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
---
|
||||
title: Indeed
|
||||
description: Scrape public Indeed job postings by search, company, or URL
|
||||
---
|
||||
|
||||
The Indeed scraper pulls structured public job postings from Indeed. Give it search terms and/or URLs (a search page, a company jobs page, or a single job), and it returns jobs with title, company, location, salary, job types, benefits, remote/hybrid flag, posting age, and description.
|
||||
|
||||
## Endpoint
|
||||
|
||||
```bash
|
||||
POST /api/v1/workspaces/{workspace_id}/scrapers/indeed/scrape
|
||||
```
|
||||
|
||||
## Inputs
|
||||
|
||||
At least one of `urls` or `search_queries` is required.
|
||||
|
||||
| Field | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
| `urls` | — | Indeed URLs: a search page (`/jobs?q=&l=`), a company jobs page (`/cmp/<slug>/jobs`), or a single job (`/viewjob?jk=`) (max 20 sources per call, combined with queries) |
|
||||
| `search_queries` | — | Job search terms, e.g. `["data analyst"]` |
|
||||
| `country` | `us` | Country code selecting the Indeed domain, e.g. `us`, `gb`, `de` |
|
||||
| `location` | — | Where to search, e.g. `Remote`, `New York, NY` |
|
||||
| `radius` | — | Search radius in miles/km around `location` |
|
||||
| `job_type` | — | `fulltime`, `parttime`, `contract`, `internship`, `temporary`, `permanent`, `seasonal`, or `freelance` |
|
||||
| `level` | — | `entry_level`, `mid_level`, or `senior_level` |
|
||||
| `remote` | — | `remote` or `hybrid` |
|
||||
| `from_days` | — | Only return jobs posted within the last N days |
|
||||
| `sort` | `relevance` | `relevance` or `date` |
|
||||
| `scrape_job_details` | `false` | Fetch each job's detail page for the full description (slower: one extra page load per job) |
|
||||
| `max_items` | `25` | Max total jobs returned across all sources (hard cap 100) |
|
||||
| `max_items_per_query` | `25` | Max jobs per search/company target |
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/indeed/scrape" \
|
||||
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"search_queries": ["data analyst"],
|
||||
"location": "Remote",
|
||||
"remote": "remote",
|
||||
"sort": "date",
|
||||
"max_items": 30
|
||||
}'
|
||||
```
|
||||
|
||||
The response is `{ "items": [...] }` — one item per job posting. Billing is per returned job.
|
||||
|
||||
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Indeed → Scrape** in your workspace.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: Native Connectors
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, and the web
|
||||
description: SurfSense's built-in scraper APIs for Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, Amazon, Walmart, and the web
|
||||
---
|
||||
|
||||
import { Card, Cards } from 'fumadocs-ui/components/card';
|
||||
|
|
@ -38,6 +38,11 @@ Native connectors are SurfSense's own scraper APIs — built into the platform,
|
|||
description="Structured SERPs: organic results, people-also-ask, AI overviews"
|
||||
href="/docs/connectors/native/google-search"
|
||||
/>
|
||||
<Card
|
||||
title="Indeed"
|
||||
description="Job postings by search, company, or URL — with salary and full description"
|
||||
href="/docs/connectors/native/indeed"
|
||||
/>
|
||||
<Card
|
||||
title="Amazon"
|
||||
description="Public product data: prices, ratings, offers, sellers, and best-seller ranks"
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@
|
|||
"tiktok",
|
||||
"google-maps",
|
||||
"google-search",
|
||||
"indeed",
|
||||
"amazon",
|
||||
"walmart",
|
||||
"web-crawl"
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
|
|||
|
||||
# SurfSense MCP Server
|
||||
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 27 native, typed tools: every scraper (Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
The SurfSense MCP server exposes your workspace to any [Model Context Protocol](https://modelcontextprotocol.io/) client. Your agent gets 28 native, typed tools: every scraper (Reddit, YouTube, Instagram, TikTok, Google Maps, Google Search, Indeed, Amazon, Walmart, web crawl), full knowledge-base access (search, read, add, upload, update, delete), and a workspace selector.
|
||||
|
||||
Connect it two ways: the **hosted** server at `https://mcp.surfsense.com/mcp` (nothing to install — just an API key), or run it yourself over **stdio** against any SurfSense backend, cloud or self-hosted.
|
||||
|
||||
|
|
@ -264,7 +264,7 @@ For self-host (stdio), all settings are environment variables passed by the clie
|
|||
| Group | Tools |
|
||||
|-------|-------|
|
||||
| Workspaces | `surfsense_list_workspaces`, `surfsense_select_workspace` |
|
||||
| Scrapers | `surfsense_reddit_scrape`, `surfsense_youtube_scrape`, `surfsense_youtube_comments`, `surfsense_instagram_scrape`, `surfsense_instagram_details`, `surfsense_tiktok_scrape`, `surfsense_tiktok_comments`, `surfsense_tiktok_user_search`, `surfsense_tiktok_trending`, `surfsense_google_maps_scrape`, `surfsense_google_maps_reviews`, `surfsense_google_search`, `surfsense_web_crawl`, `surfsense_list_scraper_runs`, `surfsense_get_scraper_run` |
|
||||
| Scrapers | `surfsense_reddit_scrape`, `surfsense_youtube_scrape`, `surfsense_youtube_comments`, `surfsense_instagram_scrape`, `surfsense_instagram_details`, `surfsense_tiktok_scrape`, `surfsense_tiktok_comments`, `surfsense_tiktok_user_search`, `surfsense_tiktok_trending`, `surfsense_google_maps_scrape`, `surfsense_google_maps_reviews`, `surfsense_google_search`, `surfsense_indeed_scrape`, `surfsense_web_crawl`, `surfsense_list_scraper_runs`, `surfsense_get_scraper_run` |
|
||||
| Knowledge base | `surfsense_search_knowledge_base`, `surfsense_list_documents`, `surfsense_get_document`, `surfsense_add_document`, `surfsense_upload_file`, `surfsense_update_document`, `surfsense_delete_document` |
|
||||
|
||||
Usage is billed exactly like the REST API — scraper tools are metered per returned item, and every call is recorded under **API Playground → Runs**.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue