mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-10 22:32:16 +02:00
59 lines
2.6 KiB
Text
59 lines
2.6 KiB
Text
|
---
|
|||
|
|
title: Web Crawl
|
|||
|
|
description: Scrape any page or crawl a whole site into clean markdown
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
The web crawler fetches one or more pages — or spiders a whole site — and returns each page as clean markdown, plus metadata, every link with its anchor text, and harvested contact signals (emails, phone numbers, social profiles). JS-rendered pages are loaded in a real browser and auto-scrolled, so lazy-loaded listings are captured too.
|
|||
|
|
|
|||
|
|
## Endpoint
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
POST /api/v1/workspaces/{workspace_id}/scrapers/web/crawl
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Inputs
|
|||
|
|
|
|||
|
|
| Field | Default | Description |
|
|||
|
|
|-------|---------|-------------|
|
|||
|
|
| `startUrls` | required | 1–20 seed URLs |
|
|||
|
|
| `maxCrawlDepth` | `0` | Link-hops to follow from each seed. `0` = fetch only the seeds; `1` = also the pages they link to; up to 5. The spider stays on the seed's site |
|
|||
|
|
| `maxCrawlPages` | `10` | Total pages fetched per call, seeds included (max 200) |
|
|||
|
|
| `maxLength` | `50000` | Max characters of markdown kept per page |
|
|||
|
|
| `includeUrlPatterns` | — | Regexes a discovered link must match to be followed (empty = follow every same-site link) |
|
|||
|
|
| `excludeUrlPatterns` | — | Regexes that exclude links from being followed (wins over include) |
|
|||
|
|
|
|||
|
|
## Example
|
|||
|
|
|
|||
|
|
Scrape a single page:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/web/crawl" \
|
|||
|
|
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
|||
|
|
-H "Content-Type: application/json" \
|
|||
|
|
-d '{ "startUrls": ["https://example.com/pricing"] }'
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Crawl a site's blog, two hops deep:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/web/crawl" \
|
|||
|
|
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
|
|||
|
|
-H "Content-Type: application/json" \
|
|||
|
|
-d '{
|
|||
|
|
"startUrls": ["https://example.com/blog"],
|
|||
|
|
"maxCrawlDepth": 2,
|
|||
|
|
"maxCrawlPages": 50,
|
|||
|
|
"includeUrlPatterns": ["/blog/"]
|
|||
|
|
}'
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Output
|
|||
|
|
|
|||
|
|
One item per fetched page (in crawl order) with its markdown, metadata (title, description), crawl provenance (depth, referrer), links classified as internal/external/social/email/tel, and page-level contacts. The response also includes a site-wide `contacts` summary that deduplicates every email, phone, and social profile found — `siteWide: true` marks header/footer values (the company's own contacts) versus page-local finds like individual team members.
|
|||
|
|
|
|||
|
|
Contact details often live on about/contact/privacy pages, so crawl with `maxCrawlDepth >= 1` to surface them — useful for lead generation and competitive intelligence.
|
|||
|
|
|
|||
|
|
Billing is per successfully fetched page.
|
|||
|
|
|
|||
|
|
For the full input and output JSON schemas and generated code snippets in your language, open **API Playground → Web → Crawl** in your workspace.
|