DESKTOP-RTLN3BA\$punk
635bf8ba96
fix: update LlamaCloud test assertions to reflect new parse modes
...
- Changed assertions in `test_llamacloud_basic_uses_cost_effective_tier` to check for `parse_page_with_llm` instead of `cost_effective` tier.
- Updated `test_llamacloud_premium_uses_agentic_plus_tier` to verify `parse_page_with_agent` instead of `agentic_plus` tier.
- Ensured that `tier` is no longer included in the call arguments.
2026-04-16 01:32:42 -07:00
DESKTOP-RTLN3BA\$punk
656e061f84
feat: add processing mode support for document uploads and ETL pipeline, improded error handling ux
...
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
- Introduced a `ProcessingMode` enum to differentiate between basic and premium processing modes.
- Updated `EtlRequest` to include a `processing_mode` field, defaulting to basic.
- Enhanced ETL pipeline services to utilize the selected processing mode for Azure Document Intelligence and LlamaCloud parsing.
- Modified various routes and services to handle processing mode, affecting document upload and indexing tasks.
- Improved error handling and logging to include processing mode details.
- Added tests to validate processing mode functionality and its impact on ETL operations.
2026-04-14 21:26:00 -07:00
DESKTOP-RTLN3BA\$punk
4bee367d4a
feat: added ai file sorting
2026-04-14 01:43:30 -07:00
CREDO23
c30cc08771
Merge upstream/dev into feat/kb-export-and-folder-upload
2026-04-11 10:28:40 +02:00
CREDO23
f556446d07
Fix test mocks for vision_llm kwarg
2026-04-10 18:20:49 +02:00
CREDO23
a95bf58c8f
Make Vision LLM opt-in for uploads and connectors
2026-04-10 16:45:51 +02:00
Anish Sarkar
8c9440998a
chore: ran linting
2026-04-10 14:20:11 +05:30
Anish Sarkar
b8e1c9801b
feat: update memory extraction and management protocols to enforce structured bullet formats, utilize user first names, and enhance validation for team and user memory entries
2026-04-10 04:36:51 +05:30
Anish Sarkar
a0883d2ab6
feat: implement team memory extraction and validation mechanisms, enhancing memory management by enforcing scope restrictions and improving memory persistence for shared threads
2026-04-10 01:54:00 +05:30
CREDO23
7e90a8ed3c
Route uploaded images to vision LLM with document-parser fallback
2026-04-09 14:33:33 +02:00
Anish Sarkar
56c5809170
chore: ran linting
2026-04-08 18:23:03 +05:30
Anish Sarkar
37c52ce7ea
feat: implement indexing progress management in local folder indexing process and enhance related test coverage
2026-04-08 18:01:55 +05:30
Anish Sarkar
8455451ce1
chore: ran linting
2026-04-08 05:20:03 +05:30
Anish Sarkar
d072ca60bb
test: enhance file classification tests for Azure DI configuration
2026-04-08 05:13:17 +05:30
Anish Sarkar
20fa93f0ba
refactor: make Azure Document Intelligence an internal LLAMACLOUD accelerator instead of a standalone ETL service
2026-04-08 03:26:24 +05:30
Anish Sarkar
1fa8d1220b
feat: add support for Azure Document Intelligence in ETL pipeline
2026-04-08 00:59:12 +05:30
Anish Sarkar
0a26a6c5bb
chore: ran linting
2026-04-07 05:55:39 +05:30
Anish Sarkar
aba5f6a124
refactor: improve file handling logic in Dropbox and OneDrive connectors to include unsupported file extension information
2026-04-07 05:19:23 +05:30
Anish Sarkar
a624c86b04
refactor: update file skipping logic in Dropbox, Google Drive, and OneDrive connectors to return unsupported extension information
2026-04-07 05:11:15 +05:30
Anish Sarkar
122be76133
refactor: update _index_selected_files method signatures in Dropbox, Google Drive, and OneDrive indexers to include unsupported file count, enhancing error reporting and consistency across connectors
2026-04-07 03:16:46 +05:30
Anish Sarkar
3a1d700817
refactor: enhance file skipping logic across Dropbox, Google Drive, and OneDrive connectors to return unsupported extensions, improving error reporting and maintainability
2026-04-07 03:16:34 +05:30
Anish Sarkar
e7beeb2a36
refactor: unify file skipping logic across Dropbox, Google Drive, and OneDrive connectors by replacing classification checks with a centralized service-based approach, enhancing maintainability and consistency in file handling
2026-04-07 02:19:31 +05:30
Anish Sarkar
f03bf05aaa
refactor: enhance Google Drive indexer to support file extension filtering, improving file handling and error reporting
2026-04-06 22:34:49 +05:30
Anish Sarkar
63a75052ca
Merge remote-tracking branch 'upstream/dev' into feat/unified-etl-pipeline
2026-04-06 22:04:51 +05:30
Anish Sarkar
dc7047f64d
refactor: implement file type classification for supported extensions across Dropbox, Google Drive, and OneDrive connectors, enhancing file handling and error management
2026-04-06 22:03:47 +05:30
Anish Sarkar
47f4be08d9
refactor: remove allowed_formats from DocumentConverter initialization in DoclingService to allow acceptance of all supported formats
2026-04-06 19:31:42 +05:30
Anish Sarkar
caca491774
test: add unit tests for Dropbox integration, covering delta sync methods, file type filtering, and re-authentication behavior
2026-04-06 18:36:48 +05:30
Anish Sarkar
f8913adaa3
test: add unit tests for content extraction from cloud connectors and ETL pipeline functionality
2026-04-05 17:46:04 +05:30
Anish Sarkar
a2b3541046
chore: ran linting
2026-04-04 03:11:56 +05:30
Anish Sarkar
0d2acc665d
Merge remote-tracking branch 'upstream/dev' into feat/page-limit-connectors
2026-04-04 03:08:27 +05:30
Anish Sarkar
ce40da80ea
feat: implement page limit estimation and enforcement in file based connector indexers
...
- Added a static method `estimate_pages_from_metadata` to `PageLimitService` for estimating page counts based on file metadata.
- Integrated page limit checks in Google Drive, Dropbox, and OneDrive indexers to prevent exceeding user quotas during file indexing.
- Updated relevant indexing methods to utilize the new page estimation logic and enforce limits accordingly.
- Enhanced tests for page limit functionality, ensuring accurate estimation and enforcement across different file types.
2026-04-04 02:51:28 +05:30
Anish Sarkar
9c0af6569d
feat: implement page limit checks in local folder indexing to manage user page usage
2026-04-03 19:13:25 +05:30
Anish Sarkar
edda5b98cb
chore: ran linting
2026-04-03 17:38:29 +05:30
Anish Sarkar
b759bb36a9
feat: add direct conversion support for CSV, TSV, and HTML files in local folder indexing
2026-04-03 17:36:48 +05:30
Anish Sarkar
746c730b2e
chore: ran linting
2026-04-03 13:14:40 +05:30
Anish Sarkar
62b44889d1
Merge remote-tracking branch 'upstream/dev' into feat/local-folder-sync
2026-04-03 11:42:43 +05:30
Anish Sarkar
2b9d79d44c
feat: add integration tests for batch processing of local folder indexing, covering multiple file scenarios and error handling
2026-04-03 10:04:14 +05:30
Anish Sarkar
1fa8e1cc83
feat: refactor folder indexing to support batch processing of multiple files, enhancing performance and error handling
2026-04-03 10:02:36 +05:30
DESKTOP-RTLN3BA\$punk
62e698d8aa
refactor: streamline document upload limits and enhance handling of mentioned documents
...
- Updated maximum file size limit to 500 MB per file.
- Removed restrictions on the number of files per upload and total upload size.
- Enhanced handling of user-mentioning documents in the knowledge base search middleware.
- Improved document reading and processing logic to accommodate new features and optimizations.
2026-04-02 19:39:10 -07:00
Anish Sarkar
53df393cf7
refactor: streamline local folder indexing logic by removing unused imports, enhancing content hashing, and improving document creation process
2026-04-02 23:28:23 +05:30
Anish Sarkar
c27d24a117
feat: enhance folder indexing by adding root folder ID support and implement folder creation and cleanup logic
2026-04-02 22:41:45 +05:30
Anish Sarkar
caf2525ab5
fix: update folder ID collection logic to include deleted directories and adjust test cases for document titles
2026-04-02 22:29:07 +05:30
Anish Sarkar
22ee5c99cc
refactor: remove Local Folder connector and related tasks, implement new folder indexing endpoints
2026-04-02 22:21:31 +05:30
Anish Sarkar
775dea7894
feat: add integration and unit tests for local folder indexing and document versioning
2026-04-02 11:12:16 +05:30
DESKTOP-RTLN3BA\$punk
ad0e77c3d6
feat: enhance knowledge base search with date filtering
2026-03-31 20:13:46 -07:00
DESKTOP-RTLN3BA\$punk
a9fd45844d
feat: integrate Stripe for page purchases and reconciliation tasks
2026-03-31 18:39:45 -07:00
Anish Sarkar
272de1bb40
feat: add integration and unit tests for Dropbox indexing pipeline and parallel downloads
2026-03-30 22:19:15 +05:30
Anish Sarkar
04691d572b
chore: ran linting
2026-03-30 01:50:41 +05:30
Anish Sarkar
5a3eece397
Merge remote-tracking branch 'upstream/dev' into feat/onedrive-connector
2026-03-29 11:55:06 +05:30
DESKTOP-RTLN3BA\$punk
2cc2d339e6
feat: made agent file sytem optimized
2026-03-28 16:39:46 -07:00
