apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-04-26 01:06:23 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

chore: update Docker installation documentation and add new installation options for improved user guidance

Browse source
This commit is contained in:
Anish Sarkar 2026-03-10 03:01:26 +05:30
parent 4e0886e06d
commit 609086ecc8
23 changed files with 389 additions and 355 deletions
Download patch file Download diff file Expand all files Collapse all files

80
surfsense_web/content/docs/how-to/electric-sql.mdx
View file

@ -5,7 +5,7 @@ description: Setting up Electric SQL for real-time data synchronization in SurfS
[Electric SQL](https://electric-sql.com/) enables real-time data synchronization in SurfSense, providing instant updates for inbox items, document indexing status, and connector sync progress without manual refresh. The frontend uses [PGlite](https://pglite.dev/) (a lightweight PostgreSQL in the browser) to maintain a local database that syncs with the backend via Electric SQL.
## What Does Electric SQL Do?
## What does Electric SQL do?
When you index documents or receive inbox updates, Electric SQL pushes updates to your browser in real-time. The data flows like this:
@ -23,45 +23,24 @@ This means:
## Docker Setup
The `docker-compose.yml` includes the Electric SQL service. It is pre-configured to connect to the Docker-managed `db` container out of the box.
- The `docker-compose.yml` includes the Electric SQL service, pre-configured to connect to the Docker-managed `db` container.
- No additional configuration is required. Electric SQL works with the Docker PostgreSQL instance out of the box.
```bash
docker compose up -d
```
## Manual Setup (Development Only)
The Electric SQL service configuration in `docker-compose.yml`:
```yaml
electric:
image: electricsql/electric:1.4.6
ports:
- "${ELECTRIC_PORT:-5133}:3000"
environment:
DATABASE_URL: ${ELECTRIC_DATABASE_URL:-postgresql://${ELECTRIC_DB_USER:-electric}:${ELECTRIC_DB_PASSWORD:-electric_password}@${DB_HOST:-db}:${DB_PORT:-5432}/${DB_NAME:-surfsense}?sslmode=${DB_SSLMODE:-disable}}
ELECTRIC_INSECURE: "true"
ELECTRIC_WRITE_TO_PG_MODE: direct
depends_on:
db:
condition: service_healthy
```
No additional configuration is required — Electric SQL is pre-configured to work with the Docker PostgreSQL instance.
## Manual Setup
Follow the steps below based on your PostgreSQL setup.
This section is intended for local development environments. Follow the steps below based on your PostgreSQL setup.
### Step 1: Configure Environment Variables
Ensure your environment files are configured. If you haven't set up SurfSense yet, follow the [Manual Installation Guide](/docs/manual-installation) first.
For Electric SQL, verify these variables are set in `docker/.env`:
For Electric SQL, verify these variables are set:
**Backend (`surfsense_backend/.env`):**
```bash
ELECTRIC_PORT=5133
ELECTRIC_DB_USER=electric
ELECTRIC_DB_PASSWORD=electric_password
NEXT_PUBLIC_ELECTRIC_URL=http://localhost:5133
```
**Frontend (`surfsense_web/.env`):**
@ -71,17 +50,19 @@ NEXT_PUBLIC_ELECTRIC_URL=http://localhost:5133
NEXT_PUBLIC_ELECTRIC_AUTH_MODE=insecure
```
Next, choose the option that matches your PostgreSQL setup:
---
### Option A: Using Docker PostgreSQL
If you're using the Docker-managed PostgreSQL instance, no extra configuration is needed. Just start the services:
If you're using the Docker-managed PostgreSQL instance, no extra configuration is needed. Just start the services using the development compose file (which exposes the PostgreSQL port to your host machine):
```bash
docker compose up -d db electric
docker compose -f docker-compose.dev.yml up -d db electric
```
Then run the database migration and start the backend:
Then run the database migration, start the backend, and launch the frontend:
```bash
cd surfsense_backend
@ -89,6 +70,13 @@ uv run alembic upgrade head
uv run main.py
```
In a separate terminal, start the frontend:
```bash
cd surfsense_web
pnpm run dev
```
Electric SQL is now configured and connected to your Docker PostgreSQL database.
---
@ -148,7 +136,7 @@ ELECTRIC_DATABASE_URL=postgresql://electric:electric_password@host.docker.intern
**4. Start Electric SQL only (skip the Docker `db` container):**
```bash
docker compose up -d --no-deps electric
docker compose -f docker-compose.dev.yml up -d --no-deps electric
```
The `--no-deps` flag starts only the `electric` service without starting the Docker-managed `db` container.
@ -161,18 +149,32 @@ uv run alembic upgrade head
uv run main.py
```
In a separate terminal, start the frontend:
```bash
cd surfsense_web
pnpm run dev
```
Electric SQL is now configured and connected to your local PostgreSQL database.
## Environment Variables Reference
**Required for manual setup:**
| Variable | Location | Description | Default |
|----------|----------|-------------|---------|
| `ELECTRIC_PORT` | `docker/.env` | Port to expose Electric SQL | `5133` |
| `ELECTRIC_DB_USER` | `docker/.env` | Database user for Electric replication | `electric` |
| `ELECTRIC_DB_PASSWORD` | `docker/.env` | Database password for Electric replication | `electric_password` |
| `ELECTRIC_DATABASE_URL` | `docker/.env` | Full connection URL override for Electric. Set to use `host.docker.internal` when pointing at a local Postgres instance | *(built from above defaults)* |
| `NEXT_PUBLIC_ELECTRIC_URL` | Frontend `.env` | Electric SQL server URL (PGlite connects to this) | `http://localhost:5133` |
| `NEXT_PUBLIC_ELECTRIC_AUTH_MODE` | Frontend `.env` | Authentication mode (`insecure` for dev, `secure` for production) | `insecure` |
| `ELECTRIC_DB_USER` | `surfsense_backend/.env` | Database user for Electric replication | `electric` |
| `ELECTRIC_DB_PASSWORD` | `surfsense_backend/.env` | Database password for Electric replication | `electric_password` |
| `NEXT_PUBLIC_ELECTRIC_URL` | `surfsense_web/.env` | Electric SQL server URL (PGlite connects to this) | `http://localhost:5133` |
| `NEXT_PUBLIC_ELECTRIC_AUTH_MODE` | `surfsense_web/.env` | Authentication mode (`insecure` for dev, `secure` for production) | `insecure` |
**Optional / Docker-only:**
| Variable | Location | Description | Default |
|----------|----------|-------------|---------|
| `ELECTRIC_PORT` | `docker/.env` | Port to expose Electric SQL on the host | `5133` (dev), `5929` (production) |
| `ELECTRIC_DATABASE_URL` | `docker/.env` | Full connection URL override for Electric. Only needed for Option B (local Postgres via `host.docker.internal`) | *(built from above defaults)* |
## Verify Setup

2
surfsense_web/content/docs/how-to/meta.json
View file

@ -1,6 +1,6 @@
{
"title": "How to",
"pages": ["electric-sql", "realtime-collaboration", "migrate-from-allinone"],
"pages": ["electric-sql", "realtime-collaboration"],
"icon": "BookOpen",
"defaultOpen": false
}

195
surfsense_web/content/docs/how-to/migrate-from-allinone.mdx
View file

@ -1,195 +0,0 @@
---
title: Migrate from the All-in-One Container
description: How to migrate your data from the legacy surfsense all-in-one Docker image to the current multi-container setup
---
The original SurfSense all-in-one image (`ghcr.io/modsetter/surfsense:latest`, run via `docker-compose.quickstart.yml`) stored all data — PostgreSQL, Redis, and configuration — in a single Docker volume named `surfsense-data`. The current setup uses separate named volumes and has upgraded PostgreSQL from **version 14 to 17**.
Because PostgreSQL data files are not compatible between major versions, a **logical dump and restore** is required. This is a one-time migration.
<Callout type="warn">
This guide only applies to users who ran the legacy `docker-compose.quickstart.yml` (the all-in-one `surfsense` container). If you were already using `docker/docker-compose.yml`, you do not need to migrate.
</Callout>
---
## Option A — One command (recommended)
`install.sh` detects the legacy `surfsense-data` volume and handles the full migration automatically — no separate migration script needed. Just run the same install command you would use for a fresh install:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
**What it does automatically:**
1. Downloads all SurfSense files (including `migrate-database.sh`) into `./surfsense/`
2. Detects the `surfsense-data` volume and enters migration mode
3. Stops the old all-in-one container if it is still running
4. Starts a temporary PostgreSQL 14 container and dumps your database
5. Recovers your `SECRET_KEY` from the old volume
6. Starts PostgreSQL 17, restores the dump, runs a smoke test
7. Starts all services
Your original `surfsense-data` volume is **never deleted** — you remove it manually after verifying.
### After it completes
1. Open [http://localhost:3000](http://localhost:3000) and confirm your data is intact.
2. Once satisfied, remove the old volume (irreversible):
```bash
docker volume rm surfsense-data
```
3. Delete the dump file once you no longer need it as a backup:
```bash
rm ./surfsense_migration_backup.sql
```
### If the migration fails mid-way
The dump file is saved to `./surfsense_migration_backup.sql` as a checkpoint. Simply re-run `install.sh` — it will detect the existing dump and skip straight to the restore step without re-extracting.
---
## Option B — Manual migration script (custom credentials)
If you launched the old all-in-one container with custom database credentials (`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` environment variables), the automatic path will use wrong credentials. Run `migrate-database.sh` manually first:
```bash
# 1. Extract data with your custom credentials
bash ./surfsense/scripts/migrate-database.sh --db-user myuser --db-password mypass --db-name mydb
# 2. Install and restore (detects the dump automatically)
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
Or download and run if you haven't run `install.sh` yet:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/migrate-database.sh -o migrate-database.sh
bash migrate-database.sh --db-user myuser --db-password mypass --db-name mydb
```
### Migration script options
| Flag | Description | Default |
|------|-------------|---------|
| `--db-user USER` | Old PostgreSQL username | `surfsense` |
| `--db-password PASS` | Old PostgreSQL password | `surfsense` |
| `--db-name NAME` | Old PostgreSQL database | `surfsense` |
| `--yes` / `-y` | Skip confirmation prompts (used automatically by `install.sh`) | — |
---
## Option C — Manual steps
For users who prefer full control or whose platform doesn't support bash scripts (e.g. Windows without WSL2).
### Step 1 — Stop the old all-in-one container
Before mounting the `surfsense-data` volume into a new container, stop the existing one to prevent two PostgreSQL processes from writing to the same data directory:
```bash
docker stop surfsense 2>/dev/null || true
```
### Step 2 — Start a temporary PostgreSQL 14 container
```bash
docker run -d --name surfsense-pg14-temp \
-v surfsense-data:/data \
-e PGDATA=/data/postgres \
-e POSTGRES_USER=surfsense \
-e POSTGRES_PASSWORD=surfsense \
-e POSTGRES_DB=surfsense \
pgvector/pgvector:pg14
```
Wait ~10 seconds, then confirm it is healthy:
```bash
docker exec surfsense-pg14-temp pg_isready -U surfsense
```
### Step 3 — Dump the database
```bash
docker exec -e PGPASSWORD=surfsense surfsense-pg14-temp \
pg_dump -U surfsense surfsense > surfsense_backup.sql
```
### Step 4 — Recover your SECRET\_KEY
```bash
docker run --rm -v surfsense-data:/data alpine cat /data/.secret_key
```
### Step 5 — Set up the new stack
```bash
mkdir -p surfsense/scripts
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/docker-compose.yml -o surfsense/docker-compose.yml
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/.env.example -o surfsense/.env.example
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/postgresql.conf -o surfsense/postgresql.conf
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/init-electric-user.sh -o surfsense/scripts/init-electric-user.sh
chmod +x surfsense/scripts/init-electric-user.sh
cp surfsense/.env.example surfsense/.env
```
Set `SECRET_KEY` in `surfsense/.env` to the value from Step 4.
### Step 6 — Start PostgreSQL 17 and restore
```bash
cd surfsense
docker compose up -d db
docker compose exec db pg_isready -U surfsense # wait until ready
docker compose exec -T db psql -U surfsense -d surfsense < ../surfsense_backup.sql
```
### Step 7 — Start all services
```bash
docker compose up -d
```
### Step 8 — Clean up
```bash
docker stop surfsense-pg14-temp && docker rm surfsense-pg14-temp
docker volume rm surfsense-data # only after verifying migration succeeded
```
---
## Troubleshooting
### `install.sh` runs normally with a blank database (no migration happened)
The legacy volume was not detected. Confirm it exists:
```bash
docker volume ls | grep surfsense-data
```
If it doesn't appear, the old container may have used a different volume name. Check with:
```bash
docker volume ls | grep -i surfsense
```
### Extraction fails with permission errors
The script detects the UID of the data files and runs the temporary PG14 container as that user. If you see permission errors in `./surfsense-migration.log`, run `migrate-database.sh` manually and check the log for details.
### Cannot find `/data/.secret_key`
The all-in-one entrypoint always writes the key to `/data/.secret_key` unless you explicitly set `SECRET_KEY=` as an environment variable. If the key is missing, the migration script auto-generates a new one (with a warning). You can update it manually in `./surfsense/.env` afterwards. Note that a new key invalidates all existing browser sessions — users will need to log in again.
### Restore errors after re-running `install.sh`
If `surfsense-postgres` volume already exists from a previous partial run, remove it before retrying:
```bash
docker volume rm surfsense-postgres
```