Commit graph

426 commits

Author SHA1 Message Date
Anish Sarkar
8e7cda31c5 feat: update Google indexing functions to track skipped messages
- Modified the indexing functions for Google Calendar and Gmail to return the count of skipped messages alongside indexed messages, enhancing performance tracking.
- Updated related tests to accommodate the new return values, ensuring comprehensive coverage of the indexing process.
- Improved error handling to maintain consistency in returned values across different indexing functions.
2026-03-19 20:56:40 +05:30
Anish Sarkar
e9485ab2df feat: update Google Drive indexing to include skipped file tracking 2026-03-19 20:27:50 +05:30
Anish Sarkar
eac4cb6075 feat: enhance Google Drive indexing to track skipped files
- Updated the indexing function to return the count of skipped files alongside indexed files, improving tracking of indexing performance.
- Added logic to accumulate skipped file counts during the indexing process, providing better insights into potential issues.
- Enhanced notification updates to include skipped file counts, ensuring comprehensive progress reporting for users.
2026-03-19 20:27:36 +05:30
Anish Sarkar
2390bd7d26 feat: enhance Google Drive authentication error handling
- Improved error handling for Google Drive indexing and listing operations to manage authentication failures more effectively.
- Added logic to mark connectors as 'auth_expired' when a 401 error or invalid credentials are detected, prompting users to re-authenticate.
- Updated error messages to provide clearer guidance on authentication issues, ensuring a better user experience.
2026-03-19 18:24:41 +05:30
Anish Sarkar
83152e8e7e refactor: unify all 3 google Composio and non-Composio connector types and pipelines keeping same credential adapters 2026-03-19 05:08:21 +05:30
Anish Sarkar
ac0f2fa2eb chore: ran linting 2026-03-17 04:40:46 +05:30
DESKTOP-RTLN3BA\$punk
2b33dfe728 refactor: update safe_set_chunks function to be asynchronous and modify all connector and document processor files to use the new async implementation 2026-03-15 00:44:27 -07:00
DESKTOP-RTLN3BA\$punk
d8a05ae4d5 feat: refactor agent tools management and add UI integration
- Added endpoint to list agent tools with metadata, excluding hidden tools.
- Updated NewChatRequest and RegenerateRequest schemas to include disabled tools.
- Integrated disabled tools management in the NewChatPage and Composer components.
- Improved tool instructions and visibility in the system prompt.
- Refactored tool registration to support hidden tools and default enabled states.
- Enhanced document chunk creation to handle strict zip behavior.
- Cleaned up imports and formatting across various files for consistency.
2026-03-10 17:36:26 -07:00
DESKTOP-RTLN3BA\$punk
403097646d feat: implement batch unread counts for notifications to reduce API calls and improve performance 2026-03-10 01:26:37 -07:00
Anish Sarkar
e8cf677b25 refactor: update title generation logic to improve user experience by generating titles in parallel with assistant responses 2026-03-09 01:28:53 +05:30
Anish Sarkar
6d00b0debf Merge remote-tracking branch 'upstream/dev' into refactor/upload-document-adapter-class 2026-03-01 22:35:17 +05:30
DESKTOP-RTLN3BA\$punk
ecb0a25cc8 feat: enhance memory management and session handling in database operations
- Introduced a shielded async session context manager to ensure safe session closure during cancellations.
- Updated various database operations to utilize the new shielded session, preventing orphaned connections.
- Added environment variables to optimize glibc memory management, improving overall application performance.
- Implemented a function to trim the native heap, allowing for better memory reclamation on Linux systems.
2026-02-28 23:59:28 -08:00
DESKTOP-RTLN3BA\$punk
dd3da2bc36 refactor: improve session management and cleanup in chat streaming
- Added proper session closure to prevent connection leaks during streaming.
- Implemented a fresh session for cleanup tasks to ensure data integrity.
- Enhanced error handling during session operations to improve robustness.
- Removed unnecessary session parameters from function signatures for clarity.
2026-02-28 23:17:11 -08:00
DESKTOP-RTLN3BA\$punk
d959a6a6c8 feat: optimize document upload process and enhance memory management
- Increased maximum file upload limit from 10 to 50 to improve user experience.
- Implemented batch processing for document uploads to avoid proxy timeouts, splitting files into manageable chunks.
- Enhanced garbage collection in chat streaming functions to prevent memory leaks and improve performance.
- Added memory delta tracking in system snapshots for better monitoring of resource usage.
- Updated LLM router and service configurations to prevent unbounded internal accumulation and improve efficiency.
2026-02-28 17:22:34 -08:00
DESKTOP-RTLN3BA\$punk
f4b2ab0899 feat: enhance caching mechanisms to prevent memory leaks
- Improved in-memory rate limiting by evicting timestamps outside the current window and cleaning up empty keys.
- Updated LLM router service to cache context profiles and avoid redundant computations.
- Introduced cache eviction logic for MCP tools and sandbox instances to manage memory usage effectively.
- Added garbage collection triggers in chat streaming functions to reclaim resources promptly.
2026-02-27 17:56:00 -08:00
DESKTOP-RTLN3BA\$punk
664c43ca13 feat: add performance logging middleware and enhance performance tracking across services
- Introduced RequestPerfMiddleware to log request performance metrics, including slow request thresholds.
- Updated various services and retrievers to utilize the new performance logging utility for better tracking of execution times.
- Enhanced existing methods with detailed performance logs for operations such as embedding, searching, and indexing.
- Removed deprecated logging setup in stream_new_chat and replaced it with the new performance logger.
2026-02-27 16:32:30 -08:00
Anish Sarkar
23a98d802c refactor: implement UploadDocumentAdapter for file indexing and reindexing 2026-02-28 01:38:32 +05:30
DESKTOP-RTLN3BA\$punk
e9892c8fe9 feat: added configable summary calculation and various improvements
- Replaced direct embedding calls with a utility function across various components to streamline embedding logic.
- Added enable_summary flag to several models and routes to control summary generation behavior.
2026-02-26 18:24:57 -08:00
DESKTOP-RTLN3BA\$punk
23f553ef84 Merge branch 'dev' of https://github.com/MODSetter/SurfSense into dev 2026-02-26 13:01:24 -08:00
DESKTOP-RTLN3BA\$punk
aabc24f82c feat: enhance performance logging and caching in various components
- Introduced slow callback logging in FastAPI to identify blocking calls.
- Added performance logging for agent creation and tool loading processes.
- Implemented caching for MCP tools to reduce redundant server calls.
- Enhanced sandbox management with in-process caching for improved efficiency.
- Refactored several functions for better readability and performance tracking.
- Updated tests to ensure proper functionality of new features and optimizations.
2026-02-26 13:00:31 -08:00
Anish Sarkar
9ccee054a5 chore: ran linting 2026-02-26 03:05:20 +05:30
Anish Sarkar
f59a70f7a5 Merge remote-tracking branch 'upstream/dev' into feat/document-test 2026-02-26 02:22:10 +05:30
Rohan Verma
2e99f1e853
Merge pull request #837 from CREDO23/test-document-creation
[Refactor] Core document creation / indexing pipeline with unit and e2e tests
2026-02-25 12:27:41 -08:00
DESKTOP-RTLN3BA\$punk
54dd6f904c feat: implement local file persistence for Daytona sandboxes
- Added functionality to persist sandbox files locally before deletion.
- Introduced methods for retrieving and deleting locally stored sandbox files.
- Updated routes to handle local file downloads and background deletion of sandboxes.
- Enhanced streaming tasks to support sandbox file management.
2026-02-25 12:25:59 -08:00
Anish Sarkar
380c1c3877 fix: Refactor document ID usage in file processing to improve clarity 2026-02-26 01:28:09 +05:30
CREDO23
ab31bba3ad merge upstream/dev 2026-02-25 20:52:05 +02:00
CREDO23
4293910e8e plug file upload into indexing pipeline adapter and add integration tests 2026-02-25 20:20:52 +02:00
DESKTOP-RTLN3BA\$punk
d198c8b89d chore; linting 2026-02-25 01:50:28 -08:00
DESKTOP-RTLN3BA\$punk
d570cae3c6 feat: added file handling for daytona sandboxes
- Added _TimeoutAwareSandbox class to handle per-command timeouts in DaytonaSandbox.
- Updated _find_or_create function to manage sandbox states and restart stopped/archived sandboxes.
- Enhanced get_or_create_sandbox to return the new sandbox class.
- Introduced file download functionality in the frontend, allowing users to download generated files from the sandbox.
- Updated system prompt to include guidelines for sharing generated files.
2026-02-25 01:36:30 -08:00
DESKTOP-RTLN3BA\$punk
2869d3a9c8 Merge commit 'd0ee8b12b6' into codesandbox_integration 2026-02-24 16:38:09 -08:00
DESKTOP-RTLN3BA\$punk
c007f0e056 feat: unut codesandbox integration 2026-02-24 16:36:11 -08:00
Rohan Verma
d0ee8b12b6
Merge pull request #834 from CREDO23/google-drive-hitl
[Feature] Add human in the loop for google drive sensitive actions (create:docs,sheets & delete)
2026-02-24 16:34:25 -08:00
CREDO23
e89b68520a rename trash_google_drive_file to delete_google_drive_file 2026-02-24 13:18:39 +02:00
CREDO23
33bcf19b62 stream full output for google drive HITL tools 2026-02-23 16:13:18 +02:00
DESKTOP-RTLN3BA\$punk
0fb6caa01f fix: improve error handling for AI responding state clearance in chat streaming 2026-02-21 20:05:17 -08:00
DESKTOP-RTLN3BA\$punk
08c75127f1 feat: update migration to skip blocknode conversion and update transaction handling in chat routes 2026-02-20 22:43:25 -08:00
Rohan Verma
f15833fec6
Merge pull request #826 from AnishSarkar22/fix/report-artifact
fix: improve report artifact & enhance revision handling
2026-02-19 19:10:21 -08:00
Rohan Verma
9aef655566
Merge pull request #825 from CREDO23/sur-169-feat-implement-human-in-the-loop-for-linear-sensitive
[Feat] Add human in the loop for linear sensitive actions
2026-02-19 19:09:50 -08:00
Anish Sarkar
e059ee4193 chore: ran linting 2026-02-19 19:34:25 +05:30
Anish Sarkar
4afdfb580d feat: enhance chat streaming by incorporating recent report context for improved LLM decision-making 2026-02-19 19:28:03 +05:30
Anish Sarkar
69dfe943b7 refactor: enhance report generation tool with improved source strategies and revision handling 2026-02-19 15:33:32 +05:30
Anish Sarkar
f3ec48fb00 feat: enhance chat event streaming by tracking active tool depth to prevent inner-tool LLM token leakage 2026-02-19 13:11:18 +05:30
CREDO23
1d6c1a2287 feat(linear): disable linear tools when no connector, handle in streaming 2026-02-18 18:49:03 +02:00
CREDO23
7d1bd1fab4 Implement KB sync after Notion page updates with block ID verification
- Add NotionKBSyncService for immediate KB updates after page changes
- Implement block ID verification to ensure content freshness
- Refactor duplicate block processing logic to shared utils
- Add user-friendly status messages
- Include debug logging for troubleshooting
2026-02-17 20:30:12 +02:00
Anish Sarkar
8b497da130 feat: add source_markdown column to documents and implement migration logic for existing records using a pure-Python BlockNote JSON to Markdown converter 2026-02-17 11:34:11 +05:30
DESKTOP-RTLN3BA\$punk
1849b451a5 feat: update Celery worker configuration and improve connector deletion process
- Added support for multiple queues in Celery worker configuration.
- Modified connector deletion to handle documents inline instead of using a background task.
- Updated response messages for document creation and connector deletion to reflect new processing status.
- Removed the obsolete connector deletion Celery task file.
2026-02-16 00:07:23 -08:00
Anish Sarkar
793c19556d chore: ran linting 2026-02-16 03:54:19 +05:30
Anish Sarkar
bff52c71ef feat: improve report generation functionality to chat streaming service 2026-02-16 03:53:33 +05:30
CREDO23
79d1f7d603 pass full notion tool output in stream 2026-02-11 17:38:42 +02:00
CREDO23
8f81c9859d deduplicate backend streaming loop into shared _stream_agent_events helper 2026-02-10 19:47:10 +02:00