apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-05-01 20:03:30 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

chore: remove Electric SQL plumbing and infrastructure

Browse source 
Remove all Electric SQL client code, Docker service, env vars, CI build
args, install scripts, and documentation. Feature hooks that depend on
Electric are intentionally left in place to be rewritten with Rocicorp
Zero in subsequent commits.

Deleted:
- lib/electric/ (client.ts, context.ts, auth.ts, baseline.ts)
- ElectricProvider.tsx
- docker/scripts/init-electric-user.sh
- content/docs/how-to/electric-sql.mdx

Cleaned:
- package.json (4 @electric-sql/* deps)
- app/layout.tsx, UserDropdown.tsx, LayoutDataProvider.tsx
- docker-compose.yml, docker-compose.dev.yml
- Dockerfile, docker-entrypoint.js
- .env.example (frontend, docker, backend)
- CI workflows, install scripts, docs
This commit is contained in:
CREDO23 2026-03-23 16:40:25 +02:00
parent af5215fa44
commit 2b7465cdaa
30 changed files with 4 additions and 1511 deletions
Download patch file Download diff file Expand all files Collapse all files

1
surfsense_web/content/docs/docker-installation/dev-compose.mdx
View file

@ -25,6 +25,5 @@ The following `.env` variables are **only used by the dev compose file** (they h
| `NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE` | Frontend build arg for auth type | `LOCAL` |
| `NEXT_PUBLIC_ETL_SERVICE` | Frontend build arg for ETL service | `DOCLING` |
| `NEXT_PUBLIC_DEPLOYMENT_MODE` | Frontend build arg for deployment mode | `self-hosted` |
| `NEXT_PUBLIC_ELECTRIC_AUTH_MODE` | Frontend build arg for Electric auth | `insecure` |
In the production compose file, the `NEXT_PUBLIC_*` frontend variables are automatically derived from `AUTH_TYPE`, `ETL_SERVICE`, and the port settings. In the dev compose file, they are passed as build args since the frontend is built from source.

16
surfsense_web/content/docs/docker-installation/docker-compose.mdx
View file

@ -18,8 +18,6 @@ After starting, access SurfSense at:
- **Frontend**: [http://localhost:3929](http://localhost:3929)
- **Backend API**: [http://localhost:8929](http://localhost:8929)
- **API Docs**: [http://localhost:8929/docs](http://localhost:8929/docs)
- **Electric SQL**: [http://localhost:5929](http://localhost:5929)
---
## Configuration
@ -50,7 +48,6 @@ All configuration lives in a single `docker/.env` file (or `surfsense/.env` if y
|----------|-------------|---------|
| `FRONTEND_PORT` | Frontend service port | `3929` |
| `BACKEND_PORT` | Backend API service port | `8929` |
| `ELECTRIC_PORT` | Electric SQL service port | `5929` |
### Custom Domain / Reverse Proxy
@ -61,7 +58,6 @@ Only set these if serving SurfSense on a real domain via a reverse proxy (Caddy,
| `NEXT_FRONTEND_URL` | Public frontend URL (e.g. `https://app.yourdomain.com`) |
| `BACKEND_URL` | Public backend URL for OAuth callbacks (e.g. `https://api.yourdomain.com`) |
| `NEXT_PUBLIC_FASTAPI_BACKEND_URL` | Backend URL used by the frontend (e.g. `https://api.yourdomain.com`) |
| `NEXT_PUBLIC_ELECTRIC_URL` | Electric SQL URL used by the frontend (e.g. `https://electric.yourdomain.com`) |
### Database
@ -77,14 +73,6 @@ Defaults work out of the box. Change for security in production.
| `DB_SSLMODE` | SSL mode: `disable`, `require`, `verify-ca`, `verify-full` | `disable` |
| `DATABASE_URL` | Full connection URL override. Use for managed databases (RDS, Supabase, etc.) | *(built from above)* |
### Electric SQL
| Variable | Description | Default |
|----------|-------------|---------|
| `ELECTRIC_DB_USER` | Replication user for Electric SQL | `electric` |
| `ELECTRIC_DB_PASSWORD` | Replication password for Electric SQL | `electric_password` |
| `ELECTRIC_DATABASE_URL` | Full connection URL override for Electric. Set to `host.docker.internal` when pointing at a local Postgres instance | *(built from above)* |
### Authentication
| Variable | Description |
@ -148,7 +136,6 @@ Uncomment the connectors you want to use. Redirect URIs follow the pattern `http
| `backend` | FastAPI application server |
| `celery_worker` | Background task processing (document indexing, etc.) |
| `celery_beat` | Periodic task scheduler (connector sync) |
| `electric` | Electric SQL (real-time sync for the frontend) |
| `frontend` | Next.js web application |
All services start automatically with `docker compose up -d`.
@ -165,7 +152,6 @@ docker compose logs -f
# View logs for a specific service
docker compose logs -f backend
docker compose logs -f electric
# Stop all services
docker compose down
@ -183,6 +169,4 @@ docker compose down -v
- **Ports already in use**: Change the relevant `*_PORT` variable in `.env` and restart.
- **Permission errors on Linux**: You may need to prefix `docker` commands with `sudo`.
- **Electric SQL not connecting**: Check `docker compose logs electric`. If it shows `domain does not exist: db`, ensure `ELECTRIC_DATABASE_URL` is not set to a stale value in `.env`.
- **Real-time updates not working in browser**: Open DevTools → Console and look for `[Electric]` errors. Check that `NEXT_PUBLIC_ELECTRIC_URL` matches the running Electric SQL address.
- **Line ending issues on Windows**: Run `git config --global core.autocrlf true` before cloning.

1
surfsense_web/content/docs/docker-installation/install-script.mdx
View file

@ -38,4 +38,3 @@ After starting, access SurfSense at:
- **Frontend**: [http://localhost:3929](http://localhost:3929)
- **Backend API**: [http://localhost:8929](http://localhost:8929)
- **API Docs**: [http://localhost:8929/docs](http://localhost:8929/docs)
- **Electric SQL**: [http://localhost:5929](http://localhost:5929)

226
surfsense_web/content/docs/how-to/electric-sql.mdx
View file

@ -1,226 +0,0 @@
---
title: Electric SQL
description: Setting up Electric SQL for real-time data synchronization in SurfSense
---
[Electric SQL](https://electric-sql.com/) enables real-time data synchronization in SurfSense, providing instant updates for inbox items, document indexing status, and connector sync progress without manual refresh. The frontend uses [PGlite](https://pglite.dev/) (a lightweight PostgreSQL in the browser) to maintain a local database that syncs with the backend via Electric SQL.
## What does Electric SQL do?
When you index documents or receive inbox updates, Electric SQL pushes updates to your browser in real-time. The data flows like this:
1. Backend writes data to PostgreSQL
2. Electric SQL detects changes and streams them to the frontend
3. PGlite (running in your browser) receives and stores the data locally in IndexedDB
4. Your UI updates instantly without refreshing
This means:
- **Inbox updates appear instantly** - No need to refresh the page
- **Document indexing progress updates live** - Watch your documents get processed
- **Connector status syncs automatically** - See when connectors finish syncing
- **Offline support** - PGlite caches data locally, so previously loaded data remains accessible
## Docker Setup
- The `docker-compose.yml` includes the Electric SQL service, pre-configured to connect to the Docker-managed `db` container.
- No additional configuration is required. Electric SQL works with the Docker PostgreSQL instance out of the box.
## Manual Setup (Development Only)
This section is intended for local development environments. Follow the steps below based on your PostgreSQL setup.
### Step 1: Configure Environment Variables
Ensure your environment files are configured. If you haven't set up SurfSense yet, follow the [Manual Installation Guide](/docs/manual-installation) first.
For Electric SQL, verify these variables are set:
**Backend (`surfsense_backend/.env`):**
```bash
ELECTRIC_DB_USER=electric
ELECTRIC_DB_PASSWORD=electric_password
```
**Frontend (`surfsense_web/.env`):**
```bash
NEXT_PUBLIC_ELECTRIC_URL=http://localhost:5133
NEXT_PUBLIC_ELECTRIC_AUTH_MODE=insecure
```
Next, choose the option that matches your PostgreSQL setup:
---
### Option A: Using Docker PostgreSQL
If you're using the Docker-managed PostgreSQL instance, no extra configuration is needed. Just start the services using the development compose file (which exposes the PostgreSQL port to your host machine):
```bash
docker compose -f docker-compose.dev.yml up -d db electric
```
Then run the database migration, start the backend, and launch the frontend:
```bash
cd surfsense_backend
uv run alembic upgrade head
uv run main.py
```
In a separate terminal, start the frontend:
```bash
cd surfsense_web
pnpm run dev
```
Electric SQL is now configured and connected to your Docker PostgreSQL database.
---
### Option B: Using Local PostgreSQL
If you're using a local PostgreSQL installation (e.g. Postgres.app on macOS), follow these steps:
**1. Enable logical replication in PostgreSQL:**
Open your `postgresql.conf` file:
```bash
# Common locations:
# macOS (Postgres.app): ~/Library/Application Support/Postgres/var-17/postgresql.conf
# macOS (Homebrew): /opt/homebrew/var/postgresql@17/postgresql.conf
# Linux: /etc/postgresql/17/main/postgresql.conf
sudo vim /path/to/postgresql.conf
```
Add the following settings:
```ini
# Required for Electric SQL
wal_level = logical
max_replication_slots = 10
max_wal_senders = 10
```
After saving, restart PostgreSQL for the settings to take effect.
**2. Create the Electric replication user:**
Connect to your local database as a superuser and run:
```sql
CREATE USER electric WITH REPLICATION PASSWORD 'electric_password';
GRANT CONNECT ON DATABASE surfsense TO electric;
GRANT CREATE ON DATABASE surfsense TO electric;
GRANT USAGE ON SCHEMA public TO electric;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO electric;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO electric;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO electric;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON SEQUENCES TO electric;
CREATE PUBLICATION electric_publication_default;
```
**3. Set `ELECTRIC_DATABASE_URL` in `docker/.env`:**
Uncomment and update this line to point Electric at your local Postgres via `host.docker.internal` (the hostname Docker containers use to reach the host machine):
```bash
ELECTRIC_DATABASE_URL=postgresql://electric:electric_password@host.docker.internal:5432/surfsense?sslmode=disable
```
**4. Start Electric SQL only (skip the Docker `db` container):**
```bash
docker compose -f docker-compose.dev.yml up -d --no-deps electric
```
The `--no-deps` flag starts only the `electric` service without starting the Docker-managed `db` container.
**5. Run database migration and start the backend:**
```bash
cd surfsense_backend
uv run alembic upgrade head
uv run main.py
```
In a separate terminal, start the frontend:
```bash
cd surfsense_web
pnpm run dev
```
Electric SQL is now configured and connected to your local PostgreSQL database.
## Environment Variables Reference
**Required for manual setup:**
| Variable | Location | Description | Default |
|----------|----------|-------------|---------|
| `ELECTRIC_DB_USER` | `surfsense_backend/.env` | Database user for Electric replication | `electric` |
| `ELECTRIC_DB_PASSWORD` | `surfsense_backend/.env` | Database password for Electric replication | `electric_password` |
| `NEXT_PUBLIC_ELECTRIC_URL` | `surfsense_web/.env` | Electric SQL server URL (PGlite connects to this) | `http://localhost:5133` |
| `NEXT_PUBLIC_ELECTRIC_AUTH_MODE` | `surfsense_web/.env` | Authentication mode (`insecure` for dev, `secure` for production) | `insecure` |
**Optional / Docker-only:**
| Variable | Location | Description | Default |
|----------|----------|-------------|---------|
| `ELECTRIC_PORT` | `docker/.env` | Port to expose Electric SQL on the host | `5133` (dev), `5929` (production) |
| `ELECTRIC_DATABASE_URL` | `docker/.env` | Full connection URL override for Electric. Only needed for Option B (local Postgres via `host.docker.internal`) | *(built from above defaults)* |
## Verify Setup
To verify Electric SQL is running correctly:
```bash
curl http://localhost:5133/v1/health
```
You should receive:
```json
{"status":"active"}
```
## Troubleshooting
### Electric SQL Server Not Starting
**Check PostgreSQL settings:**
- Ensure `wal_level = logical` is set
- Verify the Electric user has replication permissions
- Check database connectivity from Electric container
### Real-time Updates Not Working
1. Open browser DevTools → Console
2. Look for errors containing `[Electric]`
3. Check Network tab for WebSocket connections to the Electric URL
### Connection Refused Errors
- Verify Electric SQL server is running: `docker ps | grep electric`
- Check the `NEXT_PUBLIC_ELECTRIC_URL` matches your Electric server address
- For Docker setups, ensure the frontend can reach the Electric container
### Data Not Syncing
- Check Electric SQL logs: `docker compose logs electric`
- Verify PostgreSQL replication is working
- Ensure the Electric user has proper table permissions
### PGlite/IndexedDB Issues
If data appears stale or corrupted in the browser:
1. Open browser DevTools → Application → IndexedDB
2. Delete databases starting with `surfsense-`
3. Refresh the page - PGlite will recreate the local database and resync

5
surfsense_web/content/docs/how-to/index.mdx
View file

@ -8,11 +8,6 @@ import { Card, Cards } from 'fumadocs-ui/components/card';
Practical guides to help you get the most out of SurfSense.
<Cards>
<Card
title="Electric SQL"
description="Setting up Electric SQL for real-time data synchronization"
href="/docs/how-to/electric-sql"
/>
<Card
title="Realtime Collaboration"
description="Invite teammates, share chats, and collaborate in realtime"

2
surfsense_web/content/docs/how-to/meta.json
View file

@ -1,6 +1,6 @@
{
"title": "How to",
"pages": ["electric-sql", "realtime-collaboration", "web-search"],
"pages": ["realtime-collaboration", "web-search"],
"icon": "Compass",
"defaultOpen": false
}

4
surfsense_web/content/docs/manual-installation.mdx
View file

@ -73,8 +73,6 @@ Edit the `.env` file and set the following variables:
| AUTH_TYPE | Authentication method: `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication |
| GOOGLE_OAUTH_CLIENT_ID | (Optional) Client ID from Google Cloud Console (required if AUTH_TYPE=GOOGLE) |
| GOOGLE_OAUTH_CLIENT_SECRET | (Optional) Client secret from Google Cloud Console (required if AUTH_TYPE=GOOGLE) |
| ELECTRIC_DB_USER | (Optional) PostgreSQL username for Electric-SQL connection (default: `electric`) |
| ELECTRIC_DB_PASSWORD | (Optional) PostgreSQL password for Electric-SQL connection (default: `electric_password`) |
| EMBEDDING_MODEL | Name of the embedding model (e.g., `sentence-transformers/all-MiniLM-L6-v2`, `openai://text-embedding-ada-002`) |
| RERANKERS_ENABLED | (Optional) Enable or disable document reranking for improved search results (e.g., `TRUE` or `FALSE`, default: `FALSE`) |
| RERANKERS_MODEL_NAME | Name of the reranker model (e.g., `ms-marco-MiniLM-L-12-v2`) (required if RERANKERS_ENABLED=TRUE) |
@ -410,8 +408,6 @@ Edit the `.env` file and set:
| NEXT_PUBLIC_FASTAPI_BACKEND_URL | Backend URL (e.g., `http://localhost:8000`) |
| NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE | Same value as set in backend AUTH_TYPE i.e `GOOGLE` for OAuth with Google, `LOCAL` for email/password authentication |
| NEXT_PUBLIC_ETL_SERVICE | Document parsing service (should match backend ETL_SERVICE): `UNSTRUCTURED`, `LLAMACLOUD`, or `DOCLING` - affects supported file formats in upload interface |
| NEXT_PUBLIC_ELECTRIC_URL | URL for Electric-SQL service (e.g., `http://localhost:5133`) |
| NEXT_PUBLIC_ELECTRIC_AUTH_MODE | Electric-SQL authentication mode (default: `insecure`) |
### 2. Install Dependencies