SurfSense/docs/integration-architecture.md

SurfSense Integration Architecture

System Topography

SurfSense operates via a multi-part integration model separating state syncing, background processing, and web presentation.

1. Data Sync Layer (Zero + Next.js + PostgreSQL)

• Zero Cache (zero-cache): Acts as the real-time CVR (Client View Record) layer synchronizing mutations.
• Flow:
  • The Next.js frontend uses @rocicorp/zero to mutate or query data locally.
  • Zero Cache connects directly to the PostgreSQL upstream DB (ZERO_UPSTREAM_DB & ZERO_CVR_DB) polling for replication changes.
  • Conflicts and syncing are orchestrated between Zero Cache (port 4848) and Frontend API endpoints (/api/zero/query and /api/zero/mutate).

2. General API / Action Layer (FastAPI + Next.js)

• FastAPI Layer: When synchronous REST operations, authentication, or triggers are needed (like initiating a scrape), Next.js performs HTTP REST calls to the FastAPI backend at BACKEND_PORT: 8929/8000.
• The FastAPI layer accesses the Postgres DB directly utilizing asyncpg and SQLAlchemy.

3. AI / RAG Processing Layer (SearXNG + Celery + FastAPI)

• Search: FastAPI contacts SearXNG at http://searxng:8080 (Internal Docker network) for anonymized agent searches.
• Offloaded Processing:
  • Web scraping/Agent LLM processes are handed off from FastAPI to Celery.
  • Celery uses Redis at redis:6379/0 as its Broker and Result Backend.
  • A detached Celery Worker process natively handles LangGraph workflows, processing large web contexts and storing vector outputs back into PostgreSQL via pgvector.
  • Celery Beat provides ongoing scheduled tasks, waking up the worker for cron jobs.

Component Integrations Summary

Origin	Target	Protocol	Description
Next.js (Web)	Zero Cache	WebSocket / HTTP	Local-first State Sync API.
Next.js (Web)	FastAPI	HTTP/REST	Triggering synchronous jobs, user logic.
Zero Cache	PostgreSQL	TCP (PG Protocol)	Standard relational sync and conflict resolution via CVR.
FastAPI	PostgreSQL	TCP (PG Protocol)	AsyncPG DB Reads/Writes.
FastAPI	Redis	TCP	Celery queue pipelining / PubSub.
FastAPI	SearXNG	HTTP	Metasearch queries.
Celery Worker	Redis	TCP	Consuming asynchronous processing queues.
Celery Worker	PostgreSQL	TCP	Saving LangGraph context and PgVector embeddings.