SurfSense/surfsense_backend
Claude 5fac3db849
Parallelize connector searches for 40-60% latency improvement
- Added asyncio.gather() to run all connector searches concurrently
- Replaced sequential for-loop (~600 lines) with parallel execution
- Added search_single_connector() helper function for cleaner code
- Added CONNECTOR_EMOJIS and CONNECTOR_RESULT_NAMES mappings
- Streams "Searching N sources in parallel" before concurrent execution
- Processes results as they complete and streams found counts

Before: 20 connectors × ~1s each = ~20s total
After: All connectors run simultaneously = ~2-3s total

This is the highest impact performance optimization identified in the audit.
2025-11-18 23:00:02 +00:00
..
alembic Add disable_registration toggle to site configuration 2025-11-18 14:28:16 +02:00
app Parallelize connector searches for 40-60% latency improvement 2025-11-18 23:00:02 +00:00
scripts SECURITY: Remove hardcoded password from admin script 2025-11-18 20:16:53 +00:00
.dockerignore feat: Added Docker Support and missing dependencies. 2025-03-20 18:52:06 -07:00
.env.example Fix CORS configuration to prevent startup error 2025-11-18 22:33:02 +00:00
.gitignore feat: added global llm configurations 2025-11-14 21:53:46 -08:00
.python-version feat: SurfSense v0.0.6 init 2025-03-14 18:53:14 -07:00
alembic.ini add github connector, add alembic for db migrations, fix bug updating connectors 2025-04-13 13:56:22 -07:00
celery_worker.py fix: celery_app path and gmail indexing 2025-10-21 21:11:41 -07:00
Dockerfile fix: docker backend entrpoint issues 2025-10-28 23:35:53 -07:00
main.py Fixed all ruff lint and formatting errors 2025-07-24 14:43:48 -07:00
OLLAMA_MEMORY_OPTIMIZATION.md Fix translation keys, session persistence, and add admin tools 2025-11-18 21:01:28 +02:00
pyproject.toml revert: remove setuptools configuration from pyproject.toml 2025-11-04 09:26:52 +05:30
uv.lock feat: added file limit tracking for a user 2025-10-30 14:58:08 -07:00