Verified revamp Phase 4-7 code citations against surfsense_backend and corrected drift:
- automations have NO delivery/notification path and automation_runs.output is never written; CI alerts wire to the separate app/notifications system (NotificationService.create_notification, Zero-synced)
- automation PENDING-gate is non-atomic (no UPDATE...WHERE status=pending); per-Tracker lock is the primary concurrency guard, not belt-and-suspenders
- folder upload uses root_folder_id (not destination_folder); KB folder scoping goes via referenced_document_ids -> SearchScope.document_ids (no folder_id search filter)
- billable predicate is 'SUCCESS and outcome.result' duplicated per caller; to be single-sourced in the 04a executor
Confirmed accurate: schedule selector (FOR UPDATE SKIP LOCKED/next_fire_at/self-heal/catchup=False/croniter), AutomationRun model, format_to_structured_document(exclude_metadata=True), MCP routing gap (constants.py), connector enum, MANUAL trigger placeholder. Added dated verification stamp to revamp/00-overview.md. Also reconciled 03b/03c/umbrella wording.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Standardized the web crawler to use Scrapling exclusively, removing Firecrawl entirely.
- Updated the crawler's location to `app/proprietary/web_crawler/connector.py` under a non-Apache-2 license boundary.
- Refactored the `WebCrawlerConnector` to eliminate the Firecrawl API key dependency, simplifying the interface for crawling URLs.
- Adjusted related components to accommodate the new structure and ensure successful crawl outcomes are properly handled.
- Updated documentation to reflect these changes and the new implementation status.
