Commit graph

1075 commits

Author SHA1 Message Date
DESKTOP-RTLN3BA\$punk
1e4b8d3e89 feat: enhance document formatting and context management for LLM tools
- Introduced dynamic character budget calculation for document formatting based on model's context window.
- Updated `format_documents_for_context` to respect character limits and improve output quality.
- Added `max_input_tokens` parameter to various functions to facilitate context-aware processing.
- Enhanced error handling for context overflow in LLM router service.
2026-02-26 20:47:19 -08:00
DESKTOP-RTLN3BA\$punk
a4dc84d1ab feat: add should_summarize parameter to task dispatchers
- Introduced should_summarize parameter in TaskDispatcher and CeleryTaskDispatcher to control summary generation.
- Updated InlineTaskDispatcher to support the new parameter for document processing.
2026-02-26 19:12:37 -08:00
DESKTOP-RTLN3BA\$punk
6f4bf11a32 Merge branch 'dev' of https://github.com/MODSetter/SurfSense into dev 2026-02-26 18:25:05 -08:00
DESKTOP-RTLN3BA\$punk
e9892c8fe9 feat: added configable summary calculation and various improvements
- Replaced direct embedding calls with a utility function across various components to streamline embedding logic.
- Added enable_summary flag to several models and routes to control summary generation behavior.
2026-02-26 18:24:57 -08:00
Rohan Verma
2f08dc9cf4
Merge pull request #839 from AnishSarkar22/feat/document-test
fix: enhanced document upload, page limit, upload limit tests
2026-02-26 13:48:45 -08:00
DESKTOP-RTLN3BA\$punk
23f553ef84 Merge branch 'dev' of https://github.com/MODSetter/SurfSense into dev 2026-02-26 13:01:24 -08:00
DESKTOP-RTLN3BA\$punk
aabc24f82c feat: enhance performance logging and caching in various components
- Introduced slow callback logging in FastAPI to identify blocking calls.
- Added performance logging for agent creation and tool loading processes.
- Implemented caching for MCP tools to reduce redundant server calls.
- Enhanced sandbox management with in-process caching for improved efficiency.
- Refactored several functions for better readability and performance tracking.
- Updated tests to ensure proper functionality of new features and optimizations.
2026-02-26 13:00:31 -08:00
Anish Sarkar
3393e435f9 feat: implement task dispatcher for document processing
- Introduced a TaskDispatcher abstraction to decouple the upload endpoint from Celery, allowing for easier testing with synchronous implementations.
- Updated the create_documents_file_upload function to utilize the new dispatcher for task management.
- Removed direct Celery task imports from the upload function, enhancing modularity.
- Added integration tests for document upload, including page limit enforcement and file size restrictions.
2026-02-26 23:55:47 +05:30
Anish Sarkar
9ccee054a5 chore: ran linting 2026-02-26 03:05:20 +05:30
Anish Sarkar
f59a70f7a5 Merge remote-tracking branch 'upstream/dev' into feat/document-test 2026-02-26 02:22:10 +05:30
Rohan Verma
2e99f1e853
Merge pull request #837 from CREDO23/test-document-creation
[Refactor] Core document creation / indexing pipeline with unit and e2e tests
2026-02-25 12:27:41 -08:00
DESKTOP-RTLN3BA\$punk
54dd6f904c feat: implement local file persistence for Daytona sandboxes
- Added functionality to persist sandbox files locally before deletion.
- Introduced methods for retrieving and deleting locally stored sandbox files.
- Updated routes to handle local file downloads and background deletion of sandboxes.
- Enhanced streaming tasks to support sandbox file management.
2026-02-25 12:25:59 -08:00
Anish Sarkar
25df3dff64 chore: ran linting 2026-02-26 01:29:25 +05:30
Anish Sarkar
380c1c3877 fix: Refactor document ID usage in file processing to improve clarity 2026-02-26 01:28:09 +05:30
Anish Sarkar
a57ab02900 feat: Implement file upload limits and page limit enforcement in backend
- Added constants for maximum files per upload, per-file size, and total upload size.
- Enhanced document upload route to validate file counts and sizes, returning appropriate HTTP errors.
- Introduced end-to-end tests for upload limits and page limit enforcement, ensuring correct behavior under various scenarios.
- Updated test helpers to support notification retrieval for page limit exceeded scenarios.
2026-02-26 01:25:34 +05:30
CREDO23
c50d661d7d fix wrong status key in adapter error reporting 2026-02-25 21:00:55 +02:00
CREDO23
ab31bba3ad merge upstream/dev 2026-02-25 20:52:05 +02:00
CREDO23
d0fdd3224a fix metadata keys casing and set content_needs_reindexing in adapter 2026-02-25 20:39:18 +02:00
CREDO23
4293910e8e plug file upload into indexing pipeline adapter and add integration tests 2026-02-25 20:20:52 +02:00
CREDO23
cad400be1b add file upload adapter and make index() return refreshed document 2026-02-25 19:56:59 +02:00
CREDO23
86ecb82c6e fix: tighten indexing pipeline exception handling and logging 2026-02-25 17:44:35 +02:00
CREDO23
5be58b78ad simplify indexing pipeline DB error handling 2026-02-25 16:59:09 +02:00
CREDO23
66d7d3da8a fix bugs in indexing pipeline exception handling 2026-02-25 16:27:12 +02:00
CREDO23
b6c25628c8 add structured logging to indexing pipeline 2026-02-25 16:04:35 +02:00
CREDO23
610080bfef extract persistence helpers into document_persistence.py 2026-02-25 15:30:25 +02:00
CREDO23
0aeb888be0 add structured error handling to indexing pipeline 2026-02-25 15:26:04 +02:00
CREDO23
ca870cf660 add fallback document sumary 2026-02-25 13:47:36 +02:00
CREDO23
36d1fba75f fix: isolate per-document errors in prepare_for_indexing 2026-02-25 13:00:34 +02:00
CREDO23
e6b7ce7345 fix: handle IntegrityError in prepare_for_indexing and add within-batch content dedup test 2026-02-25 12:03:00 +02:00
DESKTOP-RTLN3BA\$punk
96595d3ff7 refactor: remove security notes from system prompt
- Eliminated the security notes section from the code execution system prompt to streamline the instructions and focus on user guidance.
2026-02-25 01:56:16 -08:00
DESKTOP-RTLN3BA\$punk
d198c8b89d chore; linting 2026-02-25 01:50:28 -08:00
DESKTOP-RTLN3BA\$punk
70686a1eb2 feat: enhance code execution system prompt
- Added detailed instructions for prioritizing code execution over text responses in specific scenarios, such as data visualization and file generation.
- Included examples to illustrate correct and incorrect approaches to handling user requests involving code execution.
- Emphasized the importance of printing file paths for user downloads and clarified the handling of sandbox-generated files.
2026-02-25 01:48:54 -08:00
DESKTOP-RTLN3BA\$punk
d570cae3c6 feat: added file handling for daytona sandboxes
- Added _TimeoutAwareSandbox class to handle per-command timeouts in DaytonaSandbox.
- Updated _find_or_create function to manage sandbox states and restart stopped/archived sandboxes.
- Enhanced get_or_create_sandbox to return the new sandbox class.
- Introduced file download functionality in the frontend, allowing users to download generated files from the sandbox.
- Updated system prompt to include guidelines for sharing generated files.
2026-02-25 01:36:30 -08:00
CREDO23
c5ae62140d fix: rescue stuck documents with unchanged content on next indexing run 2026-02-25 11:13:25 +02:00
DESKTOP-RTLN3BA\$punk
a6563f396a chore: moved to daytona due to lack of windows support in microsandbox 2026-02-25 00:38:27 -08:00
DESKTOP-RTLN3BA\$punk
421bb29466 chore: update microsandbox command to bind to all interfaces and add compatibility shims for missing types 2026-02-24 23:53:03 -08:00
CREDO23
0363cb9c17 fix: updated_at on title change, LLM fallback, stale chunks deleted on re-index 2026-02-25 08:40:13 +02:00
CREDO23
af22fa7c88 refactor: remove redundant and low-value tests, enforce connector_id and created_by_id constraints 2026-02-25 08:29:53 +02:00
DESKTOP-RTLN3BA\$punk
2869d3a9c8 Merge commit 'd0ee8b12b6' into codesandbox_integration 2026-02-24 16:38:09 -08:00
DESKTOP-RTLN3BA\$punk
c007f0e056 feat: unut codesandbox integration 2026-02-24 16:36:11 -08:00
Rohan Verma
d0ee8b12b6
Merge pull request #834 from CREDO23/google-drive-hitl
[Feature] Add human in the loop for google drive sensitive actions (create:docs,sheets & delete)
2026-02-24 16:34:25 -08:00
CREDO23
5b616eac5a fix: plug all gaps found in deep review of indexing pipeline 2026-02-25 02:20:44 +02:00
CREDO23
61e50834e6 feat: implement and test index method 2026-02-25 01:40:30 +02:00
CREDO23
497ed681d5 feat: implement and test index happy path 2026-02-25 00:30:11 +02:00
CREDO23
579a9e2cb5 feat: implement and test prepare_for_indexing 2026-02-25 00:06:34 +02:00
CREDO23
a0134a5830 test: add document hashing unit tests and clean up conftest mocks 2026-02-24 22:48:40 +02:00
CREDO23
d5e10bd8f9 test: add ConnectorDocument unit tests and factory fixture 2026-02-24 22:20:08 +02:00
CREDO23
30ab464ba7 fix reauth error handling and stale log messages 2026-02-24 13:33:47 +02:00
CREDO23
e89b68520a rename trash_google_drive_file to delete_google_drive_file 2026-02-24 13:18:39 +02:00
CREDO23
6265e9a437 add delete_from_kb support to trash google drive file tool 2026-02-24 13:01:55 +02:00