SurfSense/surfsense_web/content/docs/connectors/web-crawler.mdx

---
title: Web Crawler
description: Crawl and index websites with SurfSense
---

# Documentation in progress
$DESKTOP-RTLN3BA\$punk$ feat: update documentation structure and remove 'full' property from MDX files - Enhanced the DocsPage component by adding table of content and popover configurations. - Removed the 'full' property from multiple MDX files to streamline documentation structure. - Updated meta.json to reflect new documentation organization and added a 'connectors' page. 2026-01-06 03:39:25 -08:00			`---`
			`title: Web Crawler`
			`description: Crawl and index websites with SurfSense`
			`---`

$DESKTOP-RTLN3BA\$punk$ feat: added atlassian docs 2026-01-06 22:18:52 -08:00			`# Documentation in progress`