feat(gateway): introduce GATEWAY_TELEGRAM_INTAKE_MODE for Telegram integration

This commit is contained in:
Anish Sarkar 2026-05-28 05:02:07 +05:30
parent 7ff0120fc9
commit c958fe5bc6
5 changed files with 53 additions and 14 deletions

View file

@ -39,6 +39,39 @@ Complete all the [setup steps](/docs), including:
The backend is the core of SurfSense. Follow these steps to set it up:
### Optional: Telegram External Chat Surface
SurfSense can expose the same canonical chat agent through Telegram. The `external_chat_*` tables store adapter identity, delivery configuration, and durable inbox rows. The actual chat thread and messages remain in `new_chat_threads` and `new_chat_messages`, and all chat-message sources are eligible for Zero replication so a future SurfSense UI layer can render external chat surfaces. The web app initially shows pairing, health, revoke, and resume controls under **User Settings > Messaging Channels**.
Add these variables to `surfsense_backend/.env` when enabling the Telegram surface:
```bash
TELEGRAM_SHARED_BOT_TOKEN=123456:bot-token-from-botfather
TELEGRAM_SHARED_BOT_USERNAME=your_bot_username
TELEGRAM_WEBHOOK_SECRET=generate-a-long-random-secret
GATEWAY_BASE_URL=https://api.example.com
GATEWAY_TELEGRAM_INTAKE_MODE=webhook
```
`GATEWAY_TELEGRAM_INTAKE_MODE` must be `webhook`, `longpoll`, or `disabled`. Use `webhook` for production/SaaS deployments, `longpoll` only for single-replica self-host installs that cannot expose a public HTTPS webhook, and `disabled` to skip Telegram intake. `TELEGRAM_WEBHOOK_SECRET` must use only `A-Z`, `a-z`, `0-9`, `_`, and `-` characters. `REDIS_APP_URL` is reused for external chat rate limits and per-thread locks. The webhook URL shape is:
```text
${GATEWAY_BASE_URL}/api/v1/gateway/webhooks/telegram/{account_id}
```
After deploying the backend, register the webhook:
```bash
cd surfsense_backend
uv run python scripts/register_webhook.py
```
Keep the FastAPI backend, Celery worker, and Celery beat running. Telegram webhooks write inbound updates into `external_chat_inbound_events`. The FastAPI process owns external chat inbox processing and runs the same SurfSense agent used by web UI chats, then replies back to Telegram. Celery remains maintenance-only for external chat reconciliation, health checks, and retention sweeps. There is no separate gateway service, `SERVICE_ROLE=gateway` process, or Celery agent-processing path.
For self-hosted BYO Telegram bots without a public HTTPS URL, set `GATEWAY_TELEGRAM_INTAKE_MODE=longpoll`. The FastAPI process starts one lifespan long-poll supervisor per non-system Telegram account and writes updates into the same durable inbox. The FastAPI inbox worker then processes those rows in-process through the canonical `new_chat_*` surface. This fallback is intended for single-replica self-hosted installs. For SaaS-style multi-replica deployments, prefer public webhooks and keep `GATEWAY_TELEGRAM_INTAKE_MODE=webhook` so API replicas skip BYO polling entirely. Telegram does not allow `get_updates()` while a webhook is active, so delete any existing webhook for a BYO bot before relying on long polling.
When upgrading from an older gateway-runner deployment, apply the rewritten migration 144 external chat schema, deploy the new backend, worker, and beat images, then stop the old `gateway` service. Wait about 30 seconds for any old Telegram `getUpdates` long-poll request to release its advisory lock before starting the new API process. Register each webhook again with the account-id URL above and the per-account `webhook_secret`. If you roll back before using the migration in production, restore the old image and downgrade the schema first.
### 1. Environment Configuration
First, create and configure your environment variables by copying the example file:
@ -350,7 +383,7 @@ redis-cli ping
### 6. Start Celery Worker
In a new terminal window, start the Celery worker to handle background tasks:
In a new terminal window, start the Celery worker to handle background tasks. For external chat surfaces, Celery only runs maintenance tasks; agent turns run inside the FastAPI process.
**If using uv:**
@ -358,9 +391,9 @@ In a new terminal window, start the Celery worker to handle background tasks:
# Make sure you're in the surfsense_backend directory
cd surfsense_backend
# Start Celery worker (consume both default and connectors queues)
# Start Celery worker (consume default, connectors, and external chat maintenance queues)
DEFAULT_Q="${CELERY_TASK_DEFAULT_QUEUE:-surfsense}"
uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="${DEFAULT_Q},${DEFAULT_Q}.connectors"
uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="${DEFAULT_Q},${DEFAULT_Q}.connectors,${DEFAULT_Q}.gateway"
```
**If using pip/venv:**
@ -374,9 +407,9 @@ source .venv/bin/activate # Linux/macOS
# OR
.venv\Scripts\activate # Windows
# Start Celery worker (consume both default and connectors queues)
# Start Celery worker (consume default, connectors, and external chat maintenance queues)
DEFAULT_Q="${CELERY_TASK_DEFAULT_QUEUE:-surfsense}"
celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="${DEFAULT_Q},${DEFAULT_Q}.connectors"
celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="${DEFAULT_Q},${DEFAULT_Q}.connectors,${DEFAULT_Q}.gateway"
```
**Optional: Start Flower for monitoring Celery tasks:**