apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-04-25 00:36:31 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

refactor: added batch commits and Increased task time limits in celery_app.py

Browse source 
- Increased task time limits in celery_app.py for longer processing times.
- Enhanced pagination logic in NotionHistoryConnector to handle large result sets.
- Implemented batch commits every 10 documents across various indexers (Airtable, ClickUp, Confluence, Discord, GitHub, Google Calendar, Gmail, JIRA, Linear, Luma, Notion, Slack) to improve performance and reduce database load.
- Updated final commit logging for clarity on total documents processed.
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2025-11-03 15:57:19 -08:00
parent 49b7cb9ad5
commit e65d74f2e2
15 changed files with 305 additions and 102 deletions
Download patch file Download diff file Expand all files Collapse all files

4
surfsense_backend/app/celery_app.py
View file

@ -76,8 +76,8 @@ celery_app.conf.update(
enable_utc=True,
# Task execution settings
task_track_started=True,
task_time_limit=3600, # 1 hour hard limit
task_soft_time_limit=3000, # 50 minutes soft limit
task_time_limit=28800, # 8 hour hard limit
task_soft_time_limit=28200, # 7 hours 50 minutes soft limit
# Result backend settings
result_expires=86400, # Results expire after 24 hours
result_extended=True,