- Replaced direct embedding calls with a utility function across various components to streamline embedding logic.
- Added enable_summary flag to several models and routes to control summary generation behavior.
- Introduced a TaskDispatcher abstraction to decouple the upload endpoint from Celery, allowing for easier testing with synchronous implementations.
- Updated the create_documents_file_upload function to utilize the new dispatcher for task management.
- Removed direct Celery task imports from the upload function, enhancing modularity.
- Added integration tests for document upload, including page limit enforcement and file size restrictions.
- Added constants for maximum files per upload, per-file size, and total upload size.
- Enhanced document upload route to validate file counts and sizes, returning appropriate HTTP errors.
- Introduced end-to-end tests for upload limits and page limit enforcement, ensuring correct behavior under various scenarios.
- Updated test helpers to support notification retrieval for page limit exceeded scenarios.
- Added support for multiple queues in Celery worker configuration.
- Modified connector deletion to handle documents inline instead of using a background task.
- Updated response messages for document creation and connector deletion to reflect new processing status.
- Removed the obsolete connector deletion Celery task file.
- Introduced a new endpoint for batch document status retrieval, allowing users to check the status of multiple documents in a search space.
- Enhanced the document upload process to return duplicate document IDs and improved response structure.
- Updated schemas to include new response models for document status.
- Removed unused attachment processing code from chat routes and UI components to streamline functionality.
- Introduced pg_trgm extension and GIN trigram indexes for efficient document title searches, enhancing performance for mention picker functionality.
- Implemented a new API endpoint for lightweight document title searches, returning only essential fields.
- Updated frontend components to utilize the new title search feature with throttling for improved user experience.
- Added necessary schemas and types for the new search functionality.
-Introduce granular permissions for documents, chats, podcasts, and logs.
- Update routes to enforce permission checks for creating, reading, updating, and deleting resources. - Refactor user and search space interactions to align with RBAC model, removing ownership checks in favor of permission validation.
- Modified various FastAPI route definitions to remove trailing slashes for consistency across the application.
- Updated corresponding fetch calls in the frontend to align with the new endpoint structure.
- Ensured that all affected routes maintain their functionality without trailing slashes.
- Add header to local STT transcription for consistency
- Add empty text validation for external STT path
- Refactor external STT to eliminate duplication in atranscription calls
- Ensure both local and external paths have consistent error handling
- Add support for DeepSeek, Qwen (Alibaba), Kimi (Moonshot), and GLM (Zhipu)
- Implement auto-fill API Base URL when selecting Chinese LLM providers
- Add smart validation and warnings for missing API endpoints
- Fix session state management in task logging service
- Add comprehensive Chinese setup documentation
- Add database migration for new LLM provider enums
Closes#383
- Use .get() for safe dictionary access instead of direct key access
- Add explicit try-catch for local STT transcription failures
- Validate transcription result is not empty
- Provide clear error messages for corrupted audio files
- Match error handling pattern with external STT service
- Simplify STT_SERVICE config to local/MODEL_SIZE format
- Remove separate STT routes, integrate with document upload
- Add local STT support to audio file processing pipeline
- Remove React component, use existing upload interface
- Support both local Faster-Whisper and external STT services
- Tested with real speech: 99% accuracy, 2.87s processing
- Deleted the document_processing module and its associated docling_service.
- Updated imports in documents_routes.py and background_tasks.py to reflect the new service structure.
- Ensured compatibility with the task logging system by adjusting type hints for log entries.
- Integrated Docling ETL service with new task logging system
- Maintained consistent logging pattern across all ETL services
- Added progress and success/failure logging for Docling processing
- Added DOCLING as third ETL_SERVICE option (alongside UNSTRUCTURED/LLAMACLOUD)
- Implemented add_received_file_document_using_docling function
- Added Docling processing logic in documents_routes.py
- Enhanced chunking with configurable overlap support
- Added comprehensive document processing service
- Supports both CPU and GPU processing with user selection
Addresses #161 - Add Docling Support as an ETL_SERVICE
Follows same pattern as LlamaCloud integration (PR #123)
- Added support for processing YouTube videos, including transcript extraction and document creation.
- Implemented a new background task for adding YouTube video documents.
- Enhanced the connector service to search for YouTube videos and return relevant results.
- Updated frontend components to include YouTube video options in the dashboard and connector sources.
- Added necessary dependencies for YouTube transcript API.
