apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-04-26 17:26:23 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: enhance installation and migration scripts to automate legacy data migration process

Browse source 
- Updated install.sh to handle fresh installations and migrations from the legacy all-in-one container.
- Added checks for Docker and Docker Compose prerequisites.
- Implemented a wait-for-postgres function to ensure database readiness.
- Enhanced migration script to extract data and recover SECRET_KEY automatically.
- Updated documentation to reflect the new automated migration process.
This commit is contained in:
Anish Sarkar 2026-02-26 17:24:02 +05:30
parent 5030dec96b
commit 025643ffa2
3 changed files with 357 additions and 329 deletions
Download patch file Download diff file Expand all files Collapse all files

197
surfsense_web/content/docs/how-to/migrate-from-allinone.mdx
View file

@ -11,82 +11,79 @@ Because PostgreSQL data files are not compatible between major versions, a **log
This guide only applies to users who ran the legacy `docker-compose.quickstart.yml` (the all-in-one `surfsense` container). If you were already using `docker/docker-compose.yml`, you do not need to migrate.
</Callout>
<Callout type="info">
If you try to run `install.sh` while the old `surfsense-data` volume exists, the script will detect it and stop with instructions to migrate first.
</Callout>
---
## Option A — One command (recommended)
`install.sh` detects the legacy `surfsense-data` volume and handles the full migration automatically — no separate migration script needed. Just run the same install command you would use for a fresh install:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
**What it does automatically:**
1. Downloads all SurfSense files (including `migrate-database.sh`) into `./surfsense/`
2. Detects the `surfsense-data` volume and enters migration mode
3. Stops the old all-in-one container if it is still running
4. Starts a temporary PostgreSQL 14 container and dumps your database
5. Recovers your `SECRET_KEY` from the old volume
6. Starts PostgreSQL 17, restores the dump, runs a smoke test
7. Starts all services
Your original `surfsense-data` volume is **never deleted** — you remove it manually after verifying.
### After it completes
1. Open [http://localhost:3000](http://localhost:3000) and confirm your data is intact.
2. Once satisfied, remove the old volume (irreversible):
```bash
docker volume rm surfsense-data
```
3. Delete the dump file once you no longer need it as a backup:
```bash
rm ./surfsense_migration_backup.sql
```
### If the migration fails mid-way
The dump file is saved to `./surfsense_migration_backup.sql` as a checkpoint. Simply re-run `install.sh` — it will detect the existing dump and skip straight to the restore step without re-extracting.
---
## Option A — Migration Script (recommended)
## Option B — Manual migration script (custom credentials)
A single script handles the entire process automatically: it dumps your PostgreSQL 14 data, recovers your `SECRET_KEY`, sets up the new stack, and restores into PostgreSQL 17.
**Prerequisites:** Docker running, ~500 MB free disk space, internet access.
If you launched the old all-in-one container with custom database credentials (`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` environment variables), the automatic path will use wrong credentials. Run `migrate-database.sh` manually first:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/migrate-database.sh | bash
# 1. Extract data with your custom credentials
bash ./surfsense/scripts/migrate-database.sh --db-user myuser --db-password mypass --db-name mydb
# 2. Install and restore (detects the dump automatically)
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash
```
Or download and inspect it first (recommended):
Or download and run if you haven't run `install.sh` yet:
```bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/migrate-database.sh -o migrate-database.sh
# Review the script, then run:
bash migrate-database.sh
bash migrate-database.sh --db-user myuser --db-password mypass --db-name mydb
```
### Options
### Migration script options
| Flag | Description | Default |
|------|-------------|---------|
| `--db-user USER` | Old PostgreSQL username | `surfsense` |
| `--db-password PASS` | Old PostgreSQL password | `surfsense` |
| `--db-name NAME` | Old PostgreSQL database | `surfsense` |
| `--install-dir DIR` | New installation directory | `./surfsense` |
| `--yes` / `-y` | Skip confirmation prompts | — |
If you customised the database credentials in your old all-in-one container, pass them explicitly:
```bash
bash migrate-database.sh --db-user myuser --db-password mypass --db-name mydb
```
### What the script does
1. Checks prerequisites and confirms the `surfsense-data` volume exists
2. Starts a temporary `postgres:14` container against the old data
3. Runs `pg_dump` and validates the dump file (size + header check)
4. Recovers your `SECRET_KEY` from the old volume (or prompts if not found)
5. Downloads the new compose files into `./surfsense/` (skips if already present)
6. Writes the recovered `SECRET_KEY` into `./surfsense/.env`
7. Starts the new `db` service (PostgreSQL 17), waits for readiness
8. Restores the dump with `psql` and runs a smoke test
9. Starts all remaining services
The original `surfsense-data` volume is **never deleted** — you remove it manually after verifying the migration.
### After the script completes
1. Open [http://localhost:3000](http://localhost:3000) and confirm your data is intact.
2. Once satisfied, remove the old volume:
```bash
docker volume rm surfsense-data
```
3. Delete the backup dump once you no longer need it:
```bash
rm ./surfsense_migration_backup.sql
```
| `--yes` / `-y` | Skip confirmation prompts (used automatically by `install.sh`) | — |
---
## Option B — Manual Steps
## Option C — Manual steps
Use these steps if the migration script doesn't work on your platform (e.g. Windows without WSL2), or if you want full control over each step.
### Before you start
- Confirm the old volume exists: `docker volume ls | grep surfsense-data`
- Have ~500 MB free disk space for the SQL dump.
For users who prefer full control or whose platform doesn't support bash scripts (e.g. Windows without WSL2).
### Step 1 — Start a temporary PostgreSQL 14 container
@ -97,7 +94,7 @@ docker run -d --name surfsense-pg14-temp \
-e POSTGRES_USER=surfsense \
-e POSTGRES_PASSWORD=surfsense \
-e POSTGRES_DB=surfsense \
postgres:14
pgvector/pgvector:pg14
```
Wait ~10 seconds, then confirm it is healthy:
@ -113,114 +110,78 @@ docker exec -e PGPASSWORD=surfsense surfsense-pg14-temp \
pg_dump -U surfsense surfsense > surfsense_backup.sql
```
Verify the dump is valid:
```bash
wc -l surfsense_backup.sql
grep "PostgreSQL database dump" surfsense_backup.sql
```
### Step 3 — Recover your SECRET\_KEY
```bash
docker run --rm -v surfsense-data:/data alpine cat /data/.secret_key
```
Copy the printed value for the next step.
### Step 4 — Set up the new stack
```bash
git clone https://github.com/MODSetter/SurfSense.git
cd SurfSense/docker
cp .env.example .env
mkdir -p surfsense/scripts
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/docker-compose.yml -o surfsense/docker-compose.yml
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/.env.example -o surfsense/.env.example
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/postgresql.conf -o surfsense/postgresql.conf
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/init-electric-user.sh -o surfsense/scripts/init-electric-user.sh
chmod +x surfsense/scripts/init-electric-user.sh
cp surfsense/.env.example surfsense/.env
```
Set `SECRET_KEY` in `.env` to the value recovered above.
Set `SECRET_KEY` in `surfsense/.env` to the value from Step 3.
### Step 5 — Start PostgreSQL 17
### Step 5 — Start PostgreSQL 17 and restore
```bash
cd surfsense
docker compose up -d db
docker compose exec db pg_isready -U surfsense # wait until ready
docker compose exec -T db psql -U surfsense -d surfsense < ../surfsense_backup.sql
```
Wait until ready:
```bash
docker compose exec db pg_isready -U surfsense
```
### Step 6 — Restore the database
```bash
docker compose exec -T db \
psql -U surfsense -d surfsense < surfsense_backup.sql
```
Harmless notices like `ERROR: role "surfsense" already exists` are expected.
### Step 7 — Start all services
### Step 6 — Start all services
```bash
docker compose up -d
```
### Step 8 — Clean up
After verifying everything works:
### Step 7 — Clean up
```bash
# Remove temporary PG14 container
docker stop surfsense-pg14-temp && docker rm surfsense-pg14-temp
# Remove old volume (irreversible — only after confirming migration success)
docker volume rm surfsense-data
docker volume rm surfsense-data # only after verifying migration succeeded
```
---
## Troubleshooting
### Script exits with "surfsense-postgres already exists"
### `install.sh` runs normally with a blank database (no migration happened)
A previous migration attempt partially completed. Remove the incomplete volume and retry:
The legacy volume was not detected. Confirm it exists:
```bash
docker volume rm surfsense-postgres
bash migrate-database.sh
docker volume ls | grep surfsense-data
```
### PostgreSQL 14 container fails to start
Check the container logs:
If it doesn't appear, the old container may have used a different volume name. Check with:
```bash
docker logs surfsense-pg14-temp
docker volume ls | grep -i surfsense
```
If you see permission errors, the data directory may need ownership correction. Run:
### Extraction fails with permission errors
```bash
docker exec surfsense-pg14-temp chown -R postgres:postgres /data/postgres
```
Then restart the container.
### Empty or corrupt dump file
If `surfsense_backup.sql` is smaller than expected, run the dump command again with verbose output:
```bash
docker exec -e PGPASSWORD=surfsense surfsense-pg14-temp \
pg_dump -U surfsense surfsense -v 2>&1 | head -40
```
The script detects the UID of the data files and runs the temporary PG14 container as that user. If you see permission errors in `./surfsense-migration.log`, run `migrate-database.sh` manually and check the log for details.
### Cannot find `/data/.secret_key`
If the all-in-one was launched with `SECRET_KEY` set explicitly as an environment variable, the key was never written to the volume. Set the same value manually in `docker/.env`. If it is lost, generate a new one:
The all-in-one entrypoint always writes the key to `/data/.secret_key` unless you explicitly set `SECRET_KEY=` as an environment variable. If the key is missing, the migration script auto-generates a new one (with a warning). You can update it manually in `./surfsense/.env` afterwards. Note that a new key invalidates all existing browser sessions — users will need to log in again.
### Restore errors after re-running `install.sh`
If `surfsense-postgres` volume already exists from a previous partial run, remove it before retrying:
```bash
openssl rand -base64 32
docker volume rm surfsense-postgres
```
Note: a new key invalidates all existing browser sessions — users will need to log in again.