SurfSense/surfsense_backend/app
Claude 5fac3db849
Parallelize connector searches for 40-60% latency improvement
- Added asyncio.gather() to run all connector searches concurrently
- Replaced sequential for-loop (~600 lines) with parallel execution
- Added search_single_connector() helper function for cleaner code
- Added CONNECTOR_EMOJIS and CONNECTOR_RESULT_NAMES mappings
- Streams "Searching N sources in parallel" before concurrent execution
- Processes results as they complete and streams found counts

Before: 20 connectors × ~1s each = ~20s total
After: All connectors run simultaneously = ~2-3s total

This is the highest impact performance optimization identified in the audit.
2025-11-18 23:00:02 +00:00
..
agents Parallelize connector searches for 40-60% latency improvement 2025-11-18 23:00:02 +00:00
config Improve code quality per PR #11 and #12 review feedback 2025-11-18 22:46:17 +00:00
connectors chore: linting 2025-11-03 16:00:58 -08:00
prompts Fixed all ruff lint and formatting errors 2025-07-24 14:43:48 -07:00
retriver recurse fix 2025-08-20 10:21:59 -07:00
routes Improve code quality per PR #11 and #12 review feedback 2025-11-18 22:46:17 +00:00
schemas Add disable_registration toggle to site configuration 2025-11-18 14:28:16 +02:00
services Update streaming service and database connection handling 2025-11-17 20:34:56 +02:00
tasks Update streaming service and database connection handling 2025-11-17 20:34:56 +02:00
utils feat: Implement local-first European AI architecture with Mistral NeMo and TildeOpen 2025-11-17 19:58:20 +02:00
__init__.py feat: SurfSense v0.0.6 init 2025-03-14 18:53:14 -07:00
app.py Refactor auth token usage to use AUTH_TOKEN_KEY constant 2025-11-18 22:03:56 +00:00
celery_app.py refactor: added batch commits and Increased task time limits in celery_app.py 2025-11-03 15:57:19 -08:00
db.py Add disable_registration toggle to site configuration 2025-11-18 14:28:16 +02:00
users.py Update streaming service and database connection handling 2025-11-17 20:34:56 +02:00