Anish Sarkar
9b1b9a90c0
Merge remote-tracking branch 'upstream/dev' into feat/obsidian-plugin
2026-04-24 21:34:55 +05:30
Rohan Verma
a0f2851784
Merge pull request #1299 from AnishSarkar22/feat/swappable-filesystem
...
Build and Push Docker Images / tag_release (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Has been cancelled
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Has been cancelled
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Has been cancelled
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Has been cancelled
feat: introduce swappable filesystem on desktop & monaco editor to edit local files
2026-04-23 19:38:33 -07:00
Anish Sarkar
7063d6d1e4
feat(filesystem): improve path normalization in SurfSenseFilesystemMiddleware to handle Windows-style paths and mixed separators
2026-04-24 06:09:22 +05:30
Anish Sarkar
30b55a9baa
feat(filesystem): refactor local filesystem handling to use mounts instead of root paths, enhancing mount management and path normalization
2026-04-24 05:59:21 +05:30
Anish Sarkar
6721919398
feat(filesystem): add multi-root local folder support in backend
2026-04-24 01:44:23 +05:30
Anish Sarkar
1eadecee23
feat(new-chat): integrate filesystem flow into agent pipeline
2026-04-23 15:45:33 +05:30
Anish Sarkar
42d2d2222e
feat(filesystem): add local folder backend and verification coverage
2026-04-23 15:44:12 +05:30
Anish Sarkar
15a9e8b085
feat(middleware): detect file intent in chat messages
2026-04-23 15:03:32 +05:30
Anish Sarkar
6ac5256431
feat: implement background processing for binary attachments in Obsidian plugin
...
- Added a new Celery task for indexing non-markdown attachments.
- Enhanced the Obsidian plugin schema to support binary attachments.
- Updated routes to enqueue binary attachments for background processing.
- Improved metadata handling for binary attachments during indexing.
- Added tests for binary attachment processing and validation.
2026-04-22 23:00:34 +05:30
Anish Sarkar
6eeaa2db4d
feat: enhance Obsidian plugin schema with HeadingRef class
2026-04-22 20:26:58 +05:30
Anish Sarkar
144e513068
feat: implement resume page limit functionality
2026-04-22 18:46:17 +05:30
DESKTOP-RTLN3BA\$punk
80d3f624d4
chore: linting
2026-04-22 01:05:31 -07:00
Anish Sarkar
3eb4d55ef5
chore: ran linting
2026-04-22 06:40:39 +05:30
Anish Sarkar
4a75603d4f
feat: implement sync notifications for Obsidian plugin
...
- Added functionality to create and update notifications during the Obsidian sync process.
- Improved handling of sync completion and failure notifications.
- Updated connector naming convention in various locations for consistency.
2026-04-22 06:38:51 +05:30
Anish Sarkar
16ea8e2401
Merge remote-tracking branch 'upstream/dev' into feat/obsidian-plugin
2026-04-21 23:35:22 +05:30
DESKTOP-RTLN3BA\$punk
dbcea0d50b
feat: enhance HTTP exception handling with 5xx sanitization policy
...
- Updated the HTTP exception handler to sanitize 500 responses, replacing them with a generic message to prevent sensitive information leakage.
- Preserved detailed messages for other 5xx statuses (e.g., 503, 502) to provide user-friendly feedback while logging the errors server-side.
- Added unit tests to ensure that specific 5xx responses retain their detail for frontend rendering.
2026-04-21 01:19:41 -07:00
Anish Sarkar
54ce2666f5
feat: implement cross-device deduplication for Obsidian connectors using vault fingerprinting and enhance connector management
2026-04-21 04:21:33 +05:30
Anish Sarkar
2d90ed0fec
feat: deactivate legacy Obsidian connectors and implement partial unique index for improved upsert handling
2026-04-21 03:18:44 +05:30
DESKTOP-RTLN3BA\$punk
635bf8ba96
fix: update LlamaCloud test assertions to reflect new parse modes
...
- Changed assertions in `test_llamacloud_basic_uses_cost_effective_tier` to check for `parse_page_with_llm` instead of `cost_effective` tier.
- Updated `test_llamacloud_premium_uses_agentic_plus_tier` to verify `parse_page_with_agent` instead of `agentic_plus` tier.
- Ensured that `tier` is no longer included in the call arguments.
2026-04-16 01:32:42 -07:00
DESKTOP-RTLN3BA\$punk
656e061f84
feat: add processing mode support for document uploads and ETL pipeline, improded error handling ux
...
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
- Introduced a `ProcessingMode` enum to differentiate between basic and premium processing modes.
- Updated `EtlRequest` to include a `processing_mode` field, defaulting to basic.
- Enhanced ETL pipeline services to utilize the selected processing mode for Azure Document Intelligence and LlamaCloud parsing.
- Modified various routes and services to handle processing mode, affecting document upload and indexing tasks.
- Improved error handling and logging to include processing mode details.
- Added tests to validate processing mode functionality and its impact on ETL operations.
2026-04-14 21:26:00 -07:00
DESKTOP-RTLN3BA\$punk
4bee367d4a
feat: added ai file sorting
2026-04-14 01:43:30 -07:00
CREDO23
c30cc08771
Merge upstream/dev into feat/kb-export-and-folder-upload
2026-04-11 10:28:40 +02:00
CREDO23
f556446d07
Fix test mocks for vision_llm kwarg
2026-04-10 18:20:49 +02:00
CREDO23
a95bf58c8f
Make Vision LLM opt-in for uploads and connectors
2026-04-10 16:45:51 +02:00
Anish Sarkar
8c9440998a
chore: ran linting
2026-04-10 14:20:11 +05:30
Anish Sarkar
b8e1c9801b
feat: update memory extraction and management protocols to enforce structured bullet formats, utilize user first names, and enhance validation for team and user memory entries
2026-04-10 04:36:51 +05:30
Anish Sarkar
a0883d2ab6
feat: implement team memory extraction and validation mechanisms, enhancing memory management by enforcing scope restrictions and improving memory persistence for shared threads
2026-04-10 01:54:00 +05:30
CREDO23
7e90a8ed3c
Route uploaded images to vision LLM with document-parser fallback
2026-04-09 14:33:33 +02:00
Anish Sarkar
56c5809170
chore: ran linting
2026-04-08 18:23:03 +05:30
Anish Sarkar
37c52ce7ea
feat: implement indexing progress management in local folder indexing process and enhance related test coverage
2026-04-08 18:01:55 +05:30
Anish Sarkar
8455451ce1
chore: ran linting
2026-04-08 05:20:03 +05:30
Anish Sarkar
d072ca60bb
test: enhance file classification tests for Azure DI configuration
2026-04-08 05:13:17 +05:30
Anish Sarkar
20fa93f0ba
refactor: make Azure Document Intelligence an internal LLAMACLOUD accelerator instead of a standalone ETL service
2026-04-08 03:26:24 +05:30
Anish Sarkar
1fa8d1220b
feat: add support for Azure Document Intelligence in ETL pipeline
2026-04-08 00:59:12 +05:30
Anish Sarkar
0a26a6c5bb
chore: ran linting
2026-04-07 05:55:39 +05:30
Anish Sarkar
aba5f6a124
refactor: improve file handling logic in Dropbox and OneDrive connectors to include unsupported file extension information
2026-04-07 05:19:23 +05:30
Anish Sarkar
a624c86b04
refactor: update file skipping logic in Dropbox, Google Drive, and OneDrive connectors to return unsupported extension information
2026-04-07 05:11:15 +05:30
Anish Sarkar
122be76133
refactor: update _index_selected_files method signatures in Dropbox, Google Drive, and OneDrive indexers to include unsupported file count, enhancing error reporting and consistency across connectors
2026-04-07 03:16:46 +05:30
Anish Sarkar
3a1d700817
refactor: enhance file skipping logic across Dropbox, Google Drive, and OneDrive connectors to return unsupported extensions, improving error reporting and maintainability
2026-04-07 03:16:34 +05:30
Anish Sarkar
e7beeb2a36
refactor: unify file skipping logic across Dropbox, Google Drive, and OneDrive connectors by replacing classification checks with a centralized service-based approach, enhancing maintainability and consistency in file handling
2026-04-07 02:19:31 +05:30
Anish Sarkar
f03bf05aaa
refactor: enhance Google Drive indexer to support file extension filtering, improving file handling and error reporting
2026-04-06 22:34:49 +05:30
Anish Sarkar
63a75052ca
Merge remote-tracking branch 'upstream/dev' into feat/unified-etl-pipeline
2026-04-06 22:04:51 +05:30
Anish Sarkar
dc7047f64d
refactor: implement file type classification for supported extensions across Dropbox, Google Drive, and OneDrive connectors, enhancing file handling and error management
2026-04-06 22:03:47 +05:30
Anish Sarkar
47f4be08d9
refactor: remove allowed_formats from DocumentConverter initialization in DoclingService to allow acceptance of all supported formats
2026-04-06 19:31:42 +05:30
Anish Sarkar
caca491774
test: add unit tests for Dropbox integration, covering delta sync methods, file type filtering, and re-authentication behavior
2026-04-06 18:36:48 +05:30
Anish Sarkar
f8913adaa3
test: add unit tests for content extraction from cloud connectors and ETL pipeline functionality
2026-04-05 17:46:04 +05:30
Anish Sarkar
a2b3541046
chore: ran linting
2026-04-04 03:11:56 +05:30
Anish Sarkar
0d2acc665d
Merge remote-tracking branch 'upstream/dev' into feat/page-limit-connectors
2026-04-04 03:08:27 +05:30
Anish Sarkar
ce40da80ea
feat: implement page limit estimation and enforcement in file based connector indexers
...
- Added a static method `estimate_pages_from_metadata` to `PageLimitService` for estimating page counts based on file metadata.
- Integrated page limit checks in Google Drive, Dropbox, and OneDrive indexers to prevent exceeding user quotas during file indexing.
- Updated relevant indexing methods to utilize the new page estimation logic and enforce limits accordingly.
- Enhanced tests for page limit functionality, ensuring accurate estimation and enforcement across different file types.
2026-04-04 02:51:28 +05:30
Anish Sarkar
9c0af6569d
feat: implement page limit checks in local folder indexing to manage user page usage
2026-04-03 19:13:25 +05:30
