diff --git a/surfsense_web/content/docs/connectors/meta.json b/surfsense_web/content/docs/connectors/meta.json index 9b416afdd..b41e92ab9 100644 --- a/surfsense_web/content/docs/connectors/meta.json +++ b/surfsense_web/content/docs/connectors/meta.json @@ -19,5 +19,5 @@ "elasticsearch", "bookstack" ], - "defaultOpen": true + "defaultOpen": false } diff --git a/surfsense_web/content/docs/how-to/electric-sql.mdx b/surfsense_web/content/docs/how-to/electric-sql.mdx new file mode 100644 index 000000000..8e6f4dee5 --- /dev/null +++ b/surfsense_web/content/docs/how-to/electric-sql.mdx @@ -0,0 +1,242 @@ +--- +title: Electric SQL Setup +description: Setting up Electric SQL for real-time data synchronization in SurfSense +--- + +# Electric SQL + +[Electric SQL](https://electric-sql.com/) enables real-time data synchronization in SurfSense, providing instant updates for notifications, document indexing status, and connector sync progress without manual refresh. The frontend uses [PGlite](https://pglite.dev/) (a lightweight PostgreSQL in the browser) to maintain a local database that syncs with the backend via Electric SQL. + +## What Does Electric SQL Do? + +When you index documents or receive notifications, Electric SQL pushes updates to your browser in real-time. The data flows like this: + +1. Backend writes data to PostgreSQL +2. Electric SQL detects changes and streams them to the frontend +3. PGlite (running in your browser) receives and stores the data locally in IndexedDB +4. Your UI updates instantly without refreshing + +This means: + +- **Notifications appear instantly** - No need to refresh the page +- **Document indexing progress updates live** - Watch your documents get processed +- **Connector status syncs automatically** - See when connectors finish syncing +- **Offline support** - PGlite caches data locally, so previously loaded data remains accessible + +## Docker Setup + +### All-in-One Quickstart + +The simplest way to run SurfSense with Electric SQL is using the all-in-one Docker image. This bundles everything into a single container: + +- PostgreSQL + pgvector (vector database) +- Redis (task queue) +- Electric SQL (real-time sync) +- Backend API +- Frontend + +```bash +docker run -d \ + -p 3000:3000 \ + -p 8000:8000 \ + -p 5133:5133 \ + -v surfsense-data:/data \ + --name surfsense \ + ghcr.io/modsetter/surfsense:latest +``` + +**With custom Electric SQL credentials:** + +```bash +docker run -d \ + -p 3000:3000 \ + -p 8000:8000 \ + -p 5133:5133 \ + -v surfsense-data:/data \ + -e ELECTRIC_DB_USER=your_electric_user \ + -e ELECTRIC_DB_PASSWORD=your_electric_password \ + --name surfsense \ + ghcr.io/modsetter/surfsense:latest +``` + +Access SurfSense at `http://localhost:3000`. Electric SQL is automatically configured and running on port 5133. + +### Docker Compose + +For more control over individual services, use Docker Compose. + +**Quickstart (all-in-one image):** + +```bash +docker compose -f docker-compose.quickstart.yml up -d +``` + +**Standard setup (separate services):** + +The `docker-compose.yml` includes the Electric SQL service configuration: + +```yaml +electric: + image: electricsql/electric:latest + ports: + - "${ELECTRIC_PORT:-5133}:3000" + environment: + - DATABASE_URL=${ELECTRIC_DATABASE_URL:-postgresql://${ELECTRIC_DB_USER:-electric}:${ELECTRIC_DB_PASSWORD:-electric_password}@${POSTGRES_HOST:-db}:${POSTGRES_PORT:-5432}/${POSTGRES_DB:-surfsense}?sslmode=disable} + - ELECTRIC_INSECURE=true + - ELECTRIC_WRITE_TO_PG_MODE=direct + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:3000/v1/health"] + interval: 10s + timeout: 5s + retries: 5 +``` + +No additional configuration is required - Electric SQL is pre-configured to work with the Docker PostgreSQL instance. + +## Manual Setup + +Follow the steps below based on your PostgreSQL setup. + +### Step 1: Configure Environment Variables + +Add the following variables to your root `.env` file: + +```bash +ELECTRIC_PORT=5133 +POSTGRES_DB=surfsense +POSTGRES_HOST=db +ELECTRIC_DB_USER=electric +ELECTRIC_DB_PASSWORD=electric_password +``` + +> **Note:** Set `POSTGRES_HOST=db` for Docker PostgreSQL, or `POSTGRES_HOST=host.docker.internal` for local PostgreSQL. + +--- + +### Option A: Using Docker PostgreSQL + +If you're using the Docker-managed PostgreSQL instance, follow these steps: + +**1. Start PostgreSQL and Electric SQL:** + +```bash +docker-compose up -d db electric +``` + +**2. Run database migration:** + +```bash +cd surfsense_backend +uv run alembic upgrade head +``` + +**3. Start the backend:** + +```bash +uv run main.py +``` + +Electric SQL is now configured and connected to your Docker PostgreSQL database. + +--- + +### Option B: Using Local PostgreSQL + +If you're using a local PostgreSQL installation, follow these steps: + +**1. Enable logical replication in PostgreSQL:** + +Locate your `postgresql.conf` file and add the following settings: + +```ini +# Enable logical replication (required for Electric SQL) +wal_level = logical +max_replication_slots = 10 +max_wal_senders = 10 +``` + +After saving the changes, restart your PostgreSQL server for the configuration to take effect. + +**2. Update environment variable:** + +In your root `.env` file, set: + +```bash +POSTGRES_HOST=host.docker.internal +``` + +**3. Start Electric SQL:** + +```bash +docker-compose up -d electric +``` + +**4. Run database migration:** + +```bash +cd surfsense_backend +uv run alembic upgrade head +``` + +**5. Start the backend:** + +```bash +uv run main.py +``` + +Electric SQL is now configured and connected to your local PostgreSQL database. + +## Environment Variables Reference + +| Variable | Location | Description | Default | +|----------|----------|-------------|---------| +| `ELECTRIC_PORT` | docker-compose | Port to expose Electric SQL | `5133` | +| `ELECTRIC_DB_USER` | Backend | Database user for Electric | `electric` | +| `ELECTRIC_DB_PASSWORD` | Backend | Database password for Electric | `electric_password` | +| `NEXT_PUBLIC_ELECTRIC_URL` | Frontend | Electric SQL server URL (PGlite connects to this) | `http://localhost:5133` | + +## Verify Setup + +To verify Electric SQL is running correctly: + +```bash +curl http://localhost:5133/v1/health +``` + +You should receive a healthy response. + +## Troubleshooting + +### Electric SQL Server Not Starting + +**Check PostgreSQL settings:** +- Ensure `wal_level = logical` is set +- Verify the Electric user has replication permissions +- Check database connectivity from Electric container + +### Real-time Updates Not Working + +1. Open browser DevTools → Console +2. Look for errors containing `[Electric]` +3. Check Network tab for WebSocket connections to the Electric URL + +### Connection Refused Errors + +- Verify Electric SQL server is running: `docker ps | grep electric` +- Check the `NEXT_PUBLIC_ELECTRIC_URL` matches your Electric server address +- For Docker setups, ensure the frontend can reach the Electric container + +### Data Not Syncing + +- Check Electric SQL logs: `docker logs electric` +- Verify PostgreSQL replication is working +- Ensure the Electric user has proper table permissions + +### PGlite/IndexedDB Issues + +If data appears stale or corrupted in the browser: + +1. Open browser DevTools → Application → IndexedDB +2. Delete databases starting with `surfsense-` +3. Refresh the page - PGlite will recreate the local database and resync diff --git a/surfsense_web/content/docs/how-to/meta.json b/surfsense_web/content/docs/how-to/meta.json new file mode 100644 index 000000000..2a76fd25f --- /dev/null +++ b/surfsense_web/content/docs/how-to/meta.json @@ -0,0 +1,8 @@ +{ + "title": "How to", + "pages": [ + "electric-sql" + ], + "defaultOpen": false +} + diff --git a/surfsense_web/content/docs/meta.json b/surfsense_web/content/docs/meta.json index 03790dd15..3f938f541 100644 --- a/surfsense_web/content/docs/meta.json +++ b/surfsense_web/content/docs/meta.json @@ -8,6 +8,7 @@ "installation", "docker-installation", "manual-installation", - "connectors" + "connectors", + "how-to" ] }