apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-04-25 08:46:22 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

chore: update Docker installation documentation and add new installation options for improved user guidance

Browse source
This commit is contained in:
Anish Sarkar 2026-03-10 03:01:26 +05:30
parent 4e0886e06d
commit 609086ecc8
23 changed files with 389 additions and 355 deletions
Download patch file Download diff file Expand all files Collapse all files

30
surfsense_web/content/docs/docker-installation/dev-compose.mdx Normal file
View file

@ -0,0 +1,30 @@
---
title: Development Compose
description: Building SurfSense from source using docker-compose.dev.yml
---
If you're contributing to SurfSense and want to build from source, use `docker-compose.dev.yml` instead:
```bash
cd SurfSense/docker
docker compose -f docker-compose.dev.yml up --build
```
This file builds the backend and frontend from your local source code (instead of pulling prebuilt images) and includes pgAdmin for database inspection at [http://localhost:5050](http://localhost:5050). Use the production `docker-compose.yml` for all other cases.
## Dev-Only Environment Variables
The following `.env` variables are **only used by the dev compose file** (they have no effect on the production `docker-compose.yml`):
| Variable | Description | Default |
|----------|-------------|---------|
| `PGADMIN_PORT` | pgAdmin web UI port | `5050` |
| `PGADMIN_DEFAULT_EMAIL` | pgAdmin login email | `admin@surfsense.com` |
| `PGADMIN_DEFAULT_PASSWORD` | pgAdmin login password | `surfsense` |
| `REDIS_PORT` | Exposed Redis port (internal-only in prod) | `6379` |
| `NEXT_PUBLIC_FASTAPI_BACKEND_AUTH_TYPE` | Frontend build arg for auth type | `LOCAL` |
| `NEXT_PUBLIC_ETL_SERVICE` | Frontend build arg for ETL service | `DOCLING` |
| `NEXT_PUBLIC_DEPLOYMENT_MODE` | Frontend build arg for deployment mode | `self-hosted` |
| `NEXT_PUBLIC_ELECTRIC_AUTH_MODE` | Frontend build arg for Electric auth | `insecure` |
In the production compose file, the `NEXT_PUBLIC_*` frontend variables are automatically derived from `AUTH_TYPE`, `ETL_SERVICE`, and the port settings. In the dev compose file, they are passed as build args since the frontend is built from source.

188
surfsense_web/content/docs/docker-installation/docker-compose.mdx Normal file
View file

@ -0,0 +1,188 @@
---
title: Docker Compose
description: Manual Docker Compose setup for SurfSense
---
## Setup
```bash
git clone https://github.com/MODSetter/SurfSense.git
cd SurfSense/docker
cp .env.example .env
# Edit .env, at minimum set SECRET_KEY
docker compose up -d
```
After starting, access SurfSense at:
- **Frontend**: [http://localhost:3929](http://localhost:3929)
- **Backend API**: [http://localhost:8929](http://localhost:8929)
- **API Docs**: [http://localhost:8929/docs](http://localhost:8929/docs)
- **Electric SQL**: [http://localhost:5929](http://localhost:5929)
---
## Configuration
All configuration lives in a single `docker/.env` file (or `surfsense/.env` if you used the install script). Copy `.env.example` to `.env` and edit the values you need.
### Required
| Variable | Description |
|----------|-------------|
| `SECRET_KEY` | JWT secret key. Generate with: `openssl rand -base64 32`. Auto-generated by the install script. |
### Core Settings
| Variable | Description | Default |
|----------|-------------|---------|
| `SURFSENSE_VERSION` | Image tag to deploy. Use `latest`, a clean version (e.g. `0.0.14`), or a specific build (e.g. `0.0.14.1`) | `latest` |
| `AUTH_TYPE` | Authentication method: `LOCAL` (email/password) or `GOOGLE` (OAuth) | `LOCAL` |
| `ETL_SERVICE` | Document parsing: `DOCLING` (local), `UNSTRUCTURED`, or `LLAMACLOUD` | `DOCLING` |
| `EMBEDDING_MODEL` | Embedding model for vector search | `sentence-transformers/all-MiniLM-L6-v2` |
| `TTS_SERVICE` | Text-to-speech provider for podcasts | `local/kokoro` |
| `STT_SERVICE` | Speech-to-text provider for audio files | `local/base` |
| `REGISTRATION_ENABLED` | Allow new user registrations | `TRUE` |
### Ports
| Variable | Description | Default |
|----------|-------------|---------|
| `FRONTEND_PORT` | Frontend service port | `3929` |
| `BACKEND_PORT` | Backend API service port | `8929` |
| `ELECTRIC_PORT` | Electric SQL service port | `5929` |
### Custom Domain / Reverse Proxy
Only set these if serving SurfSense on a real domain via a reverse proxy (Caddy, Nginx, Cloudflare Tunnel, etc.). Leave commented out for standard localhost deployments.
| Variable | Description |
|----------|-------------|
| `NEXT_FRONTEND_URL` | Public frontend URL (e.g. `https://app.yourdomain.com`) |
| `BACKEND_URL` | Public backend URL for OAuth callbacks (e.g. `https://api.yourdomain.com`) |
| `NEXT_PUBLIC_FASTAPI_BACKEND_URL` | Backend URL used by the frontend (e.g. `https://api.yourdomain.com`) |
| `NEXT_PUBLIC_ELECTRIC_URL` | Electric SQL URL used by the frontend (e.g. `https://electric.yourdomain.com`) |
### Database
Defaults work out of the box. Change for security in production.
| Variable | Description | Default |
|----------|-------------|---------|
| `DB_USER` | PostgreSQL username | `surfsense` |
| `DB_PASSWORD` | PostgreSQL password | `surfsense` |
| `DB_NAME` | PostgreSQL database name | `surfsense` |
| `DB_HOST` | PostgreSQL host | `db` |
| `DB_PORT` | PostgreSQL port | `5432` |
| `DB_SSLMODE` | SSL mode: `disable`, `require`, `verify-ca`, `verify-full` | `disable` |
| `DATABASE_URL` | Full connection URL override. Use for managed databases (RDS, Supabase, etc.) | *(built from above)* |
### Electric SQL
| Variable | Description | Default |
|----------|-------------|---------|
| `ELECTRIC_DB_USER` | Replication user for Electric SQL | `electric` |
| `ELECTRIC_DB_PASSWORD` | Replication password for Electric SQL | `electric_password` |
| `ELECTRIC_DATABASE_URL` | Full connection URL override for Electric. Set to `host.docker.internal` when pointing at a local Postgres instance | *(built from above)* |
### Authentication
| Variable | Description |
|----------|-------------|
| `GOOGLE_OAUTH_CLIENT_ID` | Google OAuth client ID (required if `AUTH_TYPE=GOOGLE`) |
| `GOOGLE_OAUTH_CLIENT_SECRET` | Google OAuth client secret (required if `AUTH_TYPE=GOOGLE`) |
Create credentials at the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
### External API Keys
| Variable | Description |
|----------|-------------|
| `FIRECRAWL_API_KEY` | [Firecrawl](https://www.firecrawl.dev/) API key for web crawling |
| `UNSTRUCTURED_API_KEY` | [Unstructured.io](https://unstructured.io/) API key (required if `ETL_SERVICE=UNSTRUCTURED`) |
| `LLAMA_CLOUD_API_KEY` | [LlamaCloud](https://cloud.llamaindex.ai/) API key (required if `ETL_SERVICE=LLAMACLOUD`) |
### Connector OAuth Keys
Uncomment the connectors you want to use. Redirect URIs follow the pattern `http://localhost:8000/api/v1/auth/<connector>/connector/callback`.
| Connector | Variables |
|-----------|-----------|
| Google Drive / Gmail / Calendar | `GOOGLE_DRIVE_REDIRECT_URI`, `GOOGLE_GMAIL_REDIRECT_URI`, `GOOGLE_CALENDAR_REDIRECT_URI` |
| Notion | `NOTION_CLIENT_ID`, `NOTION_CLIENT_SECRET`, `NOTION_REDIRECT_URI` |
| Slack | `SLACK_CLIENT_ID`, `SLACK_CLIENT_SECRET`, `SLACK_REDIRECT_URI` |
| Discord | `DISCORD_CLIENT_ID`, `DISCORD_CLIENT_SECRET`, `DISCORD_BOT_TOKEN`, `DISCORD_REDIRECT_URI` |
| Jira & Confluence | `ATLASSIAN_CLIENT_ID`, `ATLASSIAN_CLIENT_SECRET`, `JIRA_REDIRECT_URI`, `CONFLUENCE_REDIRECT_URI` |
| Linear | `LINEAR_CLIENT_ID`, `LINEAR_CLIENT_SECRET`, `LINEAR_REDIRECT_URI` |
| ClickUp | `CLICKUP_CLIENT_ID`, `CLICKUP_CLIENT_SECRET`, `CLICKUP_REDIRECT_URI` |
| Airtable | `AIRTABLE_CLIENT_ID`, `AIRTABLE_CLIENT_SECRET`, `AIRTABLE_REDIRECT_URI` |
| Microsoft Teams | `TEAMS_CLIENT_ID`, `TEAMS_CLIENT_SECRET`, `TEAMS_REDIRECT_URI` |
### Observability (optional)
| Variable | Description |
|----------|-------------|
| `LANGSMITH_TRACING` | Enable LangSmith tracing (`true` / `false`) |
| `LANGSMITH_ENDPOINT` | LangSmith API endpoint |
| `LANGSMITH_API_KEY` | LangSmith API key |
| `LANGSMITH_PROJECT` | LangSmith project name |
### Advanced (optional)
| Variable | Description | Default |
|----------|-------------|---------|
| `SCHEDULE_CHECKER_INTERVAL` | How often to check for scheduled connector tasks (e.g. `5m`, `1h`) | `5m` |
| `RERANKERS_ENABLED` | Enable document reranking for improved search | `FALSE` |
| `RERANKERS_MODEL_NAME` | Reranker model name (e.g. `ms-marco-MiniLM-L-12-v2`) | |
| `RERANKERS_MODEL_TYPE` | Reranker model type (e.g. `flashrank`) | |
| `PAGES_LIMIT` | Max pages per user for ETL services | unlimited |
---
## Docker Services
| Service | Description |
|---------|-------------|
| `db` | PostgreSQL with pgvector extension |
| `redis` | Message broker for Celery |
| `backend` | FastAPI application server |
| `celery_worker` | Background task processing (document indexing, etc.) |
| `celery_beat` | Periodic task scheduler (connector sync) |
| `electric` | Electric SQL (real-time sync for the frontend) |
| `frontend` | Next.js web application |
All services start automatically with `docker compose up -d`.
The backend includes a health check. Dependent services (workers, frontend) wait until the API is fully ready before starting. You can monitor startup progress with `docker compose ps` (look for `(health: starting)` → `(healthy)`).
---
## Useful Commands
```bash
# View logs (all services)
docker compose logs -f
# View logs for a specific service
docker compose logs -f backend
docker compose logs -f electric
# Stop all services
docker compose down
# Restart a specific service
docker compose restart backend
# Stop and remove all containers + volumes (destructive!)
docker compose down -v
```
---
## Troubleshooting
- **Ports already in use**: Change the relevant `*_PORT` variable in `.env` and restart.
- **Permission errors on Linux**: You may need to prefix `docker` commands with `sudo`.
- **Electric SQL not connecting**: Check `docker compose logs electric`. If it shows `domain does not exist: db`, ensure `ELECTRIC_DATABASE_URL` is not set to a stale value in `.env`.
- **Real-time updates not working in browser**: Open DevTools → Console and look for `[Electric]` errors. Check that `NEXT_PUBLIC_ELECTRIC_URL` matches the running Electric SQL address.
- **Line ending issues on Windows**: Run `git config --global core.autocrlf true` before cloning.

41
surfsense_web/content/docs/docker-installation/install-script.mdx Normal file
View file

@ -0,0 +1,41 @@
---
title: Install Script
description: One-command installation of SurfSense using Docker
---
Downloads the compose files, generates a `SECRET_KEY`, starts all services, and sets up [Watchtower](https://github.com/nicholas-fedor/watchtower) for automatic daily updates.
**Prerequisites:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) must be installed and running.
### For Linux/macOS users:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
### For Windows users (PowerShell):
```powershell
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex
```
This creates a `./surfsense/` directory with `docker-compose.yml` and `.env`, then runs `docker compose up -d`.
To skip Watchtower (e.g. in production where you manage updates yourself):
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash -s -- --no-watchtower
```
To customise the check interval (default 24h), use `--watchtower-interval=SECONDS`.
---
## Access SurfSense
After starting, access SurfSense at:
- **Frontend**: [http://localhost:3929](http://localhost:3929)
- **Backend API**: [http://localhost:8929](http://localhost:8929)
- **API Docs**: [http://localhost:8929/docs](http://localhost:8929/docs)
- **Electric SQL**: [http://localhost:5929](http://localhost:5929)

6
surfsense_web/content/docs/docker-installation/meta.json Normal file
View file

@ -0,0 +1,6 @@
{
"title": "Docker Installation",
"pages": ["install-script", "docker-compose", "updating", "dev-compose", "migrate-from-allinone"],
"icon": "Container",
"defaultOpen": false
}

195
surfsense_web/content/docs/docker-installation/migrate-from-allinone.mdx Normal file
View file

@ -0,0 +1,195 @@
---
title: Migrate from the All-in-One Container
description: How to migrate your data from the legacy surfsense all-in-one Docker image to the current multi-container setup
---
The original SurfSense all-in-one image (`ghcr.io/modsetter/surfsense:latest`, run via `docker-compose.quickstart.yml`) stored all data — PostgreSQL, Redis, and configuration — in a single Docker volume named `surfsense-data`. The current setup uses separate named volumes and has upgraded PostgreSQL from **version 14 to 17**.
Because PostgreSQL data files are not compatible between major versions, a **logical dump and restore** is required. This is a one-time migration.
<Callout type="warn">
This guide only applies to users who ran the legacy `docker-compose.quickstart.yml` (the all-in-one `surfsense` container). If you were already using `docker/docker-compose.yml`, you do not need to migrate.
</Callout>
---
## Option A — One command (recommended)
`install.sh` detects the legacy `surfsense-data` volume and handles the full migration automatically — no separate migration script needed. Just run the same install command you would use for a fresh install:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
**What it does automatically:**
1. Downloads all SurfSense files (including `migrate-database.sh`) into `./surfsense/`
2. Detects the `surfsense-data` volume and enters migration mode
3. Stops the old all-in-one container if it is still running
4. Starts a temporary PostgreSQL 14 container and dumps your database
5. Recovers your `SECRET_KEY` from the old volume
6. Starts PostgreSQL 17, restores the dump, runs a smoke test
7. Starts all services
Your original `surfsense-data` volume is **never deleted** — you remove it manually after verifying.
### After it completes
1. Open [http://localhost:3000](http://localhost:3000) and confirm your data is intact.
2. Once satisfied, remove the old volume (irreversible):
```bash
docker volume rm surfsense-data
```
3. Delete the dump file once you no longer need it as a backup:
```bash
rm ./surfsense_migration_backup.sql
```
### If the migration fails mid-way
The dump file is saved to `./surfsense_migration_backup.sql` as a checkpoint. Simply re-run `install.sh` — it will detect the existing dump and skip straight to the restore step without re-extracting.
---
## Option B — Manual migration script (custom credentials)
If you launched the old all-in-one container with custom database credentials (`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` environment variables), the automatic path will use wrong credentials. Run `migrate-database.sh` manually first:
```bash
# 1. Extract data with your custom credentials
bash ./surfsense/scripts/migrate-database.sh --db-user myuser --db-password mypass --db-name mydb
# 2. Install and restore (detects the dump automatically)
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
Or download and run if you haven't run `install.sh` yet:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/migrate-database.sh -o migrate-database.sh
bash migrate-database.sh --db-user myuser --db-password mypass --db-name mydb
```
### Migration script options
| Flag | Description | Default |
|------|-------------|---------|
| `--db-user USER` | Old PostgreSQL username | `surfsense` |
| `--db-password PASS` | Old PostgreSQL password | `surfsense` |
| `--db-name NAME` | Old PostgreSQL database | `surfsense` |
| `--yes` / `-y` | Skip confirmation prompts (used automatically by `install.sh`) | — |
---
## Option C — Manual steps
For users who prefer full control or whose platform doesn't support bash scripts (e.g. Windows without WSL2).
### Step 1 — Stop the old all-in-one container
Before mounting the `surfsense-data` volume into a new container, stop the existing one to prevent two PostgreSQL processes from writing to the same data directory:
```bash
docker stop surfsense 2>/dev/null || true
```
### Step 2 — Start a temporary PostgreSQL 14 container
```bash
docker run -d --name surfsense-pg14-temp \
-v surfsense-data:/data \
-e PGDATA=/data/postgres \
-e POSTGRES_USER=surfsense \
-e POSTGRES_PASSWORD=surfsense \
-e POSTGRES_DB=surfsense \
pgvector/pgvector:pg14
```
Wait ~10 seconds, then confirm it is healthy:
```bash
docker exec surfsense-pg14-temp pg_isready -U surfsense
```
### Step 3 — Dump the database
```bash
docker exec -e PGPASSWORD=surfsense surfsense-pg14-temp \
pg_dump -U surfsense surfsense > surfsense_backup.sql
```
### Step 4 — Recover your SECRET\_KEY
```bash
docker run --rm -v surfsense-data:/data alpine cat /data/.secret_key
```
### Step 5 — Set up the new stack
```bash
mkdir -p surfsense/scripts
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/docker-compose.yml -o surfsense/docker-compose.yml
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/.env.example -o surfsense/.env.example
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/postgresql.conf -o surfsense/postgresql.conf
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/init-electric-user.sh -o surfsense/scripts/init-electric-user.sh
chmod +x surfsense/scripts/init-electric-user.sh
cp surfsense/.env.example surfsense/.env
```
Set `SECRET_KEY` in `surfsense/.env` to the value from Step 4.
### Step 6 — Start PostgreSQL 17 and restore
```bash
cd surfsense
docker compose up -d db
docker compose exec db pg_isready -U surfsense # wait until ready
docker compose exec -T db psql -U surfsense -d surfsense < ../surfsense_backup.sql
```
### Step 7 — Start all services
```bash
docker compose up -d
```
### Step 8 — Clean up
```bash
docker stop surfsense-pg14-temp && docker rm surfsense-pg14-temp
docker volume rm surfsense-data # only after verifying migration succeeded
```
---
## Troubleshooting
### `install.sh` runs normally with a blank database (no migration happened)
The legacy volume was not detected. Confirm it exists:
```bash
docker volume ls | grep surfsense-data
```
If it doesn't appear, the old container may have used a different volume name. Check with:
```bash
docker volume ls | grep -i surfsense
```
### Extraction fails with permission errors
The script detects the UID of the data files and runs the temporary PG14 container as that user. If you see permission errors in `./surfsense-migration.log`, run `migrate-database.sh` manually and check the log for details.
### Cannot find `/data/.secret_key`
The all-in-one entrypoint always writes the key to `/data/.secret_key` unless you explicitly set `SECRET_KEY=` as an environment variable. If the key is missing, the migration script auto-generates a new one (with a warning). You can update it manually in `./surfsense/.env` afterwards. Note that a new key invalidates all existing browser sessions — users will need to log in again.
### Restore errors after re-running `install.sh`
If `surfsense-postgres` volume already exists from a previous partial run, remove it before retrying:
```bash
docker volume rm surfsense-postgres
```

50
surfsense_web/content/docs/docker-installation/updating.mdx Normal file
View file

@ -0,0 +1,50 @@
---
title: Updating
description: How to update your SurfSense Docker deployment
---
## Watchtower Daemon (recommended)
Auto-updates every 24 hours. If you used the [install script](/docs/docker-installation/install-script), Watchtower is already running. No extra setup needed.
For [manual Docker Compose](/docs/docker-installation/docker-compose) installs, start Watchtower separately:
```bash
docker run -d --name watchtower \
--restart unless-stopped \
-v /var/run/docker.sock:/var/run/docker.sock \
nickfedor/watchtower \
--label-enable \
--interval 86400
```
## Watchtower One-Time Update
```bash
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
nickfedor/watchtower --run-once \
--label-filter "com.docker.compose.project=surfsense"
```
<Callout type="warn">
Use `nickfedor/watchtower`. The original `containrrr/watchtower` is no longer maintained and may fail with newer Docker versions.
</Callout>
## Manual Update
```bash
cd surfsense # or SurfSense/docker if you cloned manually
docker compose pull && docker compose up -d
```
Database migrations are applied automatically on every startup.
---
## Migrating from the All-in-One Container
<Callout type="warn">
If you were previously using `docker-compose.quickstart.yml` (the legacy all-in-one `surfsense` container), your data lives in a `surfsense-data` volume and requires a **one-time migration** before switching to the current setup. PostgreSQL has been upgraded from version 14 to 17, so a simple volume swap will not work.
See the full step-by-step guide: [Migrate from the All-in-One Container](/docs/docker-installation/migrate-from-allinone).
</Callout>