Commit graph

64 commits

Author SHA1 Message Date
Anish Sarkar
bd21c2842d feat: enhance document upload and folder synchronization UI with improved processing state indicators and responsive design adjustments 2026-04-03 04:14:09 +05:30
Anish Sarkar
f0a7c7134a feat: move endpoint above to retrieve watched folders based on search space ID 2026-04-03 00:09:17 +05:30
Anish Sarkar
25358fddcf feat: implement local folder synchronization and versioning with new metadata handling and document_versions table 2026-04-02 23:46:21 +05:30
Anish Sarkar
c27d24a117 feat: enhance folder indexing by adding root folder ID support and implement folder creation and cleanup logic 2026-04-02 22:41:45 +05:30
Anish Sarkar
22ee5c99cc refactor: remove Local Folder connector and related tasks, implement new folder indexing endpoints 2026-04-02 22:21:31 +05:30
Anish Sarkar
8e58094a86 refactor: update permission checks in document and connector routes to use specific permission values 2026-04-02 12:13:14 +05:30
Anish Sarkar
3b92e99d28 feat: add local folder connector and document versioning functionality 2026-04-02 10:35:32 +05:30
DESKTOP-RTLN3BA\$punk
685ad0c02d feat: add folder management features including creation, deletion, and organization of documents within folders 2026-03-27 01:39:15 -07:00
CREDO23
cf21eaacfc fix: critical timestamp parsing and audit fixes
- Fix timestamp conversion: String(epochMs) → new Date(epochMs).toISOString()
  in use-messages-sync, use-comments-sync, use-documents, use-inbox.
  Without this, date comparisons (isEdited, cutoff filters) would fail.
- Fix updated_at: undefined → null in use-inbox to match InboxItem type
- Fix ZeroProvider: skip Zero connection for unauthenticated users
- Clean 30+ stale "Electric SQL" comments in backend Python code
2026-03-23 19:49:28 +02:00
DESKTOP-RTLN3BA\$punk
403097646d feat: implement batch unread counts for notifications to reduce API calls and improve performance 2026-03-10 01:26:37 -07:00
Anish Sarkar
dc6c18b3f6 refactor: add sorting functionality to document retrieval and enhance DocumentsSidebar for improved search and pagination handling 2026-03-06 11:22:33 +05:30
DESKTOP-RTLN3BA\$punk
d959a6a6c8 feat: optimize document upload process and enhance memory management
- Increased maximum file upload limit from 10 to 50 to improve user experience.
- Implemented batch processing for document uploads to avoid proxy timeouts, splitting files into manageable chunks.
- Enhanced garbage collection in chat streaming functions to prevent memory leaks and improve performance.
- Added memory delta tracking in system snapshots for better monitoring of resource usage.
- Updated LLM router and service configurations to prevent unbounded internal accumulation and improve efficiency.
2026-02-28 17:22:34 -08:00
DESKTOP-RTLN3BA\$punk
6f4bf11a32 Merge branch 'dev' of https://github.com/MODSetter/SurfSense into dev 2026-02-26 18:25:05 -08:00
DESKTOP-RTLN3BA\$punk
e9892c8fe9 feat: added configable summary calculation and various improvements
- Replaced direct embedding calls with a utility function across various components to streamline embedding logic.
- Added enable_summary flag to several models and routes to control summary generation behavior.
2026-02-26 18:24:57 -08:00
Anish Sarkar
3393e435f9 feat: implement task dispatcher for document processing
- Introduced a TaskDispatcher abstraction to decouple the upload endpoint from Celery, allowing for easier testing with synchronous implementations.
- Updated the create_documents_file_upload function to utilize the new dispatcher for task management.
- Removed direct Celery task imports from the upload function, enhancing modularity.
- Added integration tests for document upload, including page limit enforcement and file size restrictions.
2026-02-26 23:55:47 +05:30
Anish Sarkar
a57ab02900 feat: Implement file upload limits and page limit enforcement in backend
- Added constants for maximum files per upload, per-file size, and total upload size.
- Enhanced document upload route to validate file counts and sizes, returning appropriate HTTP errors.
- Introduced end-to-end tests for upload limits and page limit enforcement, ensuring correct behavior under various scenarios.
- Updated test helpers to support notification retrieval for page limit exceeded scenarios.
2026-02-26 01:25:34 +05:30
Anish Sarkar
f3652ad7cf feat: add created_by_email field to document schema and update related components for improved user information display 2026-02-21 23:41:00 +05:30
DESKTOP-RTLN3BA\$punk
1849b451a5 feat: update Celery worker configuration and improve connector deletion process
- Added support for multiple queues in Celery worker configuration.
- Modified connector deletion to handle documents inline instead of using a background task.
- Updated response messages for document creation and connector deletion to reflect new processing status.
- Removed the obsolete connector deletion Celery task file.
2026-02-16 00:07:23 -08:00
DESKTOP-RTLN3BA\$punk
db652116d6 chore: linting 2026-02-09 16:49:11 -08:00
DESKTOP-RTLN3BA\$punk
c979609041 feat: simplified document upload handling
- Introduced a new endpoint for batch document status retrieval, allowing users to check the status of multiple documents in a search space.
- Enhanced the document upload process to return duplicate document IDs and improved response structure.
- Updated schemas to include new response models for document status.
- Removed unused attachment processing code from chat routes and UI components to streamline functionality.
2026-02-09 16:46:54 -08:00
Anish Sarkar
e3faf4cc5e feat: enhance document upload handling by managing duplicates and updating statuses for existing documents 2026-02-06 18:12:46 +05:30
Anish Sarkar
aa66928154 chore: ran linting 2026-02-06 05:35:15 +05:30
Anish Sarkar
ed2fc5c636 feat: enhance document upload process with two-phase indexing and real-time status updates 2026-02-06 05:15:47 +05:30
Anish Sarkar
aef59d04eb feat: add document status management with JSONB column for processing states in documents 2026-02-05 21:59:31 +05:30
Anish Sarkar
90f9fad95c feat: enhance document management with user information and connector dialog 2026-02-04 12:55:38 +05:30
Anish Sarkar
293de6876a feat: implement fuzzy search in mention document 2026-01-17 20:46:47 +05:30
Anish Sarkar
b001b65067 feat: add pg_trgm indexes and lightweight document title search
- Introduced pg_trgm extension and GIN trigram indexes for efficient document title searches, enhancing performance for mention picker functionality.
- Implemented a new API endpoint for lightweight document title searches, returning only essential fields.
- Updated frontend components to utilize the new title search feature with throttling for improved user experience.
- Added necessary schemas and types for the new search functionality.
2026-01-17 20:45:10 +05:30
DESKTOP-RTLN3BA\$punk
c768730b8c feat: fixed issues of note management
Issues Fixed

- Missing pagination fields in API response schemas (page, page_size, has_more)
- NOTE enum missing from frontend Zod schema
- Missing fields in DocumentRead response construction (content_hash, updated_at)
- BlockNote slash menu clipped by overflow-hidden CSS
- Sidebar click conflicts - hidden action buttons intercepting clicks
- Rewrote All Notes sidebar - replaced fragile custom portal with shadcn Sheet
- Missing translation keys for new UI strings
- Missing NOTE retrieval logic in researcher agent
- Added search to All Notes sidebar
- Removed frontend logging - was causing toasters on every page refresh
- Added backend logging to document reindex Celery task
2025-12-17 00:09:43 -08:00
WayChan
3c423436cc fix: retrieve wrong field for content in saving extension document. 2025-12-04 00:31:50 +00:00
WayChan
081080233a fix: saving document from browser extension fails due to missing and mismatch fields of backend data models 2025-12-03 15:32:32 +00:00
DESKTOP-RTLN3BA\$punk
e9d32c3516 feat: Implement Role-Based Access Control (RBAC) for search space resources.
-Introduce granular permissions for documents, chats, podcasts, and logs.
- Update routes to enforce permission checks for creating, reading, updating, and deleting resources. - Refactor user and search space interactions to align with RBAC model, removing ownership checks in favor of permission validation.
2025-11-27 22:45:04 -08:00
samkul-swe
6d19e0fad8 Fixing search logic 2025-11-22 13:33:16 -08:00
DESKTOP-RTLN3BA\$punk
ecd07d6155 refactor: update API endpoint paths to remove trailing slashes
- Modified various FastAPI route definitions to remove trailing slashes for consistency across the application.
- Updated corresponding fetch calls in the frontend to align with the new endpoint structure.
- Ensured that all affected routes maintain their functionality without trailing slashes.
2025-10-31 01:33:01 -07:00
DESKTOP-RTLN3BA\$punk
18adf79649 feat(fix): document type filtering 2025-10-21 21:53:55 -07:00
DESKTOP-RTLN3BA\$punk
c80bbfa867 feat: added celery and removed background_tasks for MQ's
- removed pre commit hooks
- updated docker setup
- updated github docker actions
- updated docs
2025-10-20 00:30:00 -07:00
DESKTOP-RTLN3BA\$punk
70b547c9c9 chore: updated docs & refactored sst_service.py 2025-10-15 14:31:38 -07:00
Rohan Verma
5ed9aa2b0b
Merge pull request #387 from nabthebest135/local-stt
local STT implementation with Faster-Whisper
2025-10-15 14:08:09 -07:00
Nabhan
9b72ec65b5 fix: address code review feedback for STT implementation
- Add header to local STT transcription for consistency
- Add empty text validation for external STT path
- Refactor external STT to eliminate duplication in atranscription calls
- Ensure both local and external paths have consistent error handling
2025-10-13 14:26:36 +05:00
Differ
917cf4f398 feat: add Chinese LLM providers support with auto-fill API Base URL
- Add support for DeepSeek, Qwen (Alibaba), Kimi (Moonshot), and GLM (Zhipu)
- Implement auto-fill API Base URL when selecting Chinese LLM providers
- Add smart validation and warnings for missing API endpoints
- Fix session state management in task logging service
- Add comprehensive Chinese setup documentation
- Add database migration for new LLM provider enums

Closes #383
2025-10-12 19:10:46 +08:00
Nabhan
15ba2b86f6 fix: add defensive dictionary access and error handling for local STT
- Use .get() for safe dictionary access instead of direct key access
- Add explicit try-catch for local STT transcription failures
- Validate transcription result is not empty
- Provide clear error messages for corrupted audio files
- Match error handling pattern with external STT service
2025-10-12 11:14:12 +05:00
Nabhan
504399ad01 refactor: eliminate duplicated STT service condition check
- Compute stt_service_type once and reuse
- Follow DRY principles
- Improve code maintainability
2025-10-12 11:13:30 +05:00
Nabhan
cf0e265107 refactor: integrate local STT with existing upload flow
- Simplify STT_SERVICE config to local/MODEL_SIZE format
- Remove separate STT routes, integrate with document upload
- Add local STT support to audio file processing pipeline
- Remove React component, use existing upload interface
- Support both local Faster-Whisper and external STT services
- Tested with real speech: 99% accuracy, 2.87s processing
2025-10-12 10:50:55 +05:00
Natsume Ryuhane
797fe26f53
Implemented serverside pagination;
Enabled searchspace file mgmt panel to use serverside pagination;
2025-10-01 13:05:22 -07:00
DESKTOP-RTLN3BA\$punk
76732c36ba feat: added jump to source referencing of citations 2025-08-23 18:48:18 -07:00
DESKTOP-RTLN3BA\$punk
1c4c61eb04 feat: Fixed Document Summary Content across connectors and processors 2025-08-18 20:51:48 -07:00
DESKTOP-RTLN3BA\$punk
5aa52375c3 refactor: refactored background_tasks & indexing_tasks 2025-08-12 15:28:13 -07:00
Utkarsh-Patel-13
d359a59f6d Fixed all ruff lint and formatting errors 2025-07-24 14:43:48 -07:00
MSI\ModSetter
9e8929ed2d refactor: Update import path for TaskLoggingService in documents_routes.py 2025-07-21 06:20:44 -07:00
MSI\ModSetter
931fafa403 refactor: Remove deprecated document processing services and update imports
- Deleted the document_processing module and its associated docling_service.
- Updated imports in documents_routes.py and background_tasks.py to reflect the new service structure.
- Ensured compatibility with the task logging system by adjusting type hints for log entries.
2025-07-21 06:19:37 -07:00
Abdullah 3li
f117d94ef7 fix: Resolve merge conflict in documents_routes.py
- Integrated Docling ETL service with new task logging system
- Maintained consistent logging pattern across all ETL services
- Added progress and success/failure logging for Docling processing
2025-07-21 10:43:15 +03:00