- Refactored GitHubConnector to utilize gitingest CLI via subprocess, improving performance and avoiding async issues with Celery.
- Updated ingestion method to handle repository digests more efficiently, including error handling for subprocess execution.
- Adjusted GitHub indexer to call the new synchronous ingestion method.
- Clarified documentation regarding the optional nature of the Personal Access Token for public repositories.
- Added gitingest as a dependency to streamline the ingestion of GitHub repositories.
- Refactored GitHubConnector to utilize gitingest for efficient repository digest generation, reducing API calls.
- Updated GitHub indexer to process entire repository digests, enhancing performance and simplifying the indexing process.
- Modified GitHub connect form to indicate that the Personal Access Token is optional for public repositories.
- Updated Google Drive API calls to include md5Checksum in file metadata retrieval for improved content tracking.
- Added logic to check for rename-only updates based on md5Checksum, optimizing document processing by preventing unnecessary ETL operations for unchanged content.
- Enhanced existing document update logic to handle renaming and metadata updates more effectively, particularly for Google Drive files.
- Added support for content hash fallback in document migration to prevent duplicate entries from different sources.
- Improved existing document update logic to handle renaming and metadata updates more effectively, particularly for Google Drive files.
- Updated functions to check for existing documents with enhanced logging for better traceability of duplicate content detection.
- Implemented support for both new file_id-based and legacy filename-based hash schemes in document processing.
- Added functions to generate unique identifier hashes and find existing documents with migration support.
- Improved existing document update logic to handle content changes and metadata updates, particularly for Google Drive files.
- Enhanced UI components to display appropriate file icons based on file types in the Google Drive connector.
- Updated document processing functions to accommodate the new connector structure and ensure seamless integration.
- Added logic to refresh connector and notification attributes after indexing to ensure up-to-date information.
- Enhanced periodic sync configuration to disable the option when no folders or files are selected for Google Drive, providing user feedback through a message.
- Updated the connector edit view to reflect the new disabled state for periodic sync based on selected items.
- Implemented validation in the connector dialog to prevent enabling periodic sync without selected items, improving user experience.
- Updated the Google Drive indexing functionality to include indexing options such as max files per folder, incremental sync, and inclusion of subfolders.
- Modified the API to accept a new 'indexing_options' parameter in the request body.
- Enhanced the UI to allow users to configure these options when selecting folders and files for indexing.
- Updated related components and tasks to support the new indexing options, ensuring a more flexible and efficient indexing process.
- Simplified chat document processing display by removing the book emoji for a cleaner look.
- Enhanced the greeting function to prioritize user display names over email for a more personalized experience.
- Adjusted the ChatShareButton component by removing unused imports and unnecessary elements for better clarity and performance.
- Updated the title in the Electric SQL documentation for conciseness.
- Enhanced error handling in the indexing process to differentiate between actual failures and cases where no new documents are processed.
- Updated notification messages to reflect the status accurately, including a message for when no new items are synced.
- Standardized return values across various indexer tasks to return `None` on success, simplifying logging and error management.
- Added session refresh for notifications to prevent stale data after rollbacks in multiple document processing tasks.
- Wrapped notification update logic in try-except blocks to handle potential failures gracefully and log errors without crashing the process.
- Improved error handling for notification updates in various document processing functions, enhancing overall robustness.
- Introduced a new DocumentProcessingNotificationHandler to manage notifications for document processing stages.
- Updated existing notification methods to include detailed progress updates for various stages (queued, parsing, chunking, embedding, storing, completed, failed).
- Refactored NotificationService to support the new document processing notification type and metadata schema.
- Updated multiple document processing tasks to create and manage notifications throughout the processing lifecycle.
- Adjusted UI components to reflect changes in notification types and improve user experience during document uploads and processing.
- Added setup_electric_replication function to handle Electric SQL replication for the notifications table during app startup.
- Updated alembic migration script to remove direct SQL commands for replication, now managed in app/db.py.
- Refactored indexing functions in search_source_connectors_routes to support new start_date and end_date parameters for improved flexibility.
- Enhanced Google Gmail indexing task to utilize new date parameters, ensuring better control over indexing periods.
