apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-04-26 17:26:23 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: added celery beat to docker

Browse source
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2025-10-23 00:56:28 -07:00
parent 8706df6716
commit aed8163aee
6 changed files with 134 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

18
surfsense_web/content/docs/docker-installation.mdx
View file

@ -98,6 +98,8 @@ Before you begin, ensure you have:
| LLAMA_CLOUD_API_KEY | API key for LlamaCloud service for document parsing (required if ETL_SERVICE=LLAMACLOUD) |
| CELERY_BROKER_URL | Redis connection URL for Celery broker (e.g., `redis://localhost:6379/0`) |
| CELERY_RESULT_BACKEND | Redis connection URL for Celery result backend (e.g., `redis://localhost:6379/0`) |
| SCHEDULE_CHECKER_INTERVAL | (Optional) How often to check for scheduled connector tasks. Format: `<number><unit>` where unit is `m` (minutes) or `h` (hours). Examples: `1m`, `5m`, `1h`, `2h` (default: `1m`) |
| REGISTRATION_ENABLED | (Optional) Enable or disable new user registration (e.g., `TRUE` or `FALSE`, default: `TRUE`) |
**Optional Backend LangSmith Observability:**
@ -181,6 +183,22 @@ For more details, see the [Uvicorn documentation](https://www.uvicorn.org/#comma
- API Documentation: [http://localhost:8000/docs](http://localhost:8000/docs)
- pgAdmin: [http://localhost:5050](http://localhost:5050)
## Docker Services Overview
The Docker setup includes several services that work together:
- **Backend**: FastAPI application server
- **Frontend**: Next.js web application
- **PostgreSQL (db)**: Database with pgvector extension
- **Redis**: Message broker for Celery
- **Celery Worker**: Handles background tasks (document processing, indexing, etc.)
- **Celery Beat**: Scheduler for periodic tasks (enables scheduled connector indexing)
- The schedule interval can be configured using the `SCHEDULE_CHECKER_INTERVAL` environment variable in your backend `.env` file
- Default: checks every minute for connectors that need indexing
- **pgAdmin**: Database management interface
All services start automatically with `docker compose up`. The Celery Beat service ensures that periodic indexing functionality works out of the box.
## Using pgAdmin
pgAdmin is included in the Docker setup to help manage your PostgreSQL database. To connect:

20
surfsense_web/content/docs/manual-installation.mdx
View file

@ -86,6 +86,8 @@ Edit the `.env` file and set the following variables:
| LLAMA_CLOUD_API_KEY | API key for LlamaCloud service for document parsing (required if ETL_SERVICE=LLAMACLOUD) |
| CELERY_BROKER_URL | Redis connection URL for Celery broker (e.g., `redis://localhost:6379/0`) |
| CELERY_RESULT_BACKEND | Redis connection URL for Celery result backend (e.g., `redis://localhost:6379/0`) |
| SCHEDULE_CHECKER_INTERVAL | (Optional) How often to check for scheduled connector tasks. Format: `<number><unit>` where unit is `m` (minutes) or `h` (hours). Examples: `1m`, `5m`, `1h`, `2h` (default: `1m`) |
| REGISTRATION_ENABLED | (Optional) Enable or disable new user registration (e.g., `TRUE` or `FALSE`, default: `TRUE`) |
**(Optional) Backend LangSmith Observability:**
@ -249,7 +251,23 @@ uv run celery -A celery_worker.celery_app flower --port=5555
Access Flower at [http://localhost:5555](http://localhost:5555) to monitor your Celery tasks.
### 5. Run the Backend
### 5. Start Celery Beat (Scheduler)
In another new terminal window, start Celery Beat to enable periodic tasks (like scheduled connector indexing):
**Linux/macOS/Windows:**
```bash
# Make sure you're in the surfsense_backend directory
cd surfsense_backend
# Start Celery Beat
uv run celery -A celery_worker.celery_app beat --loglevel=info
```
**Important**: Celery Beat is required for the periodic indexing functionality to work. Without it, scheduled connector tasks won't run automatically. The schedule interval can be configured using the `SCHEDULE_CHECKER_INTERVAL` environment variable.
### 6. Run the Backend
Start the backend server: