I've implemented several improvements and new features for the Slack connector,
addressing your requirements for more granular control over indexing
and data synchronization.
Key changes include:
Backend (`surfsense_backend`):
- I've updated the `SearchSourceConnector` schema for Slack to include new
configuration options:
- `slack_periodic_indexing_enabled` (boolean)
- `slack_periodic_indexing_frequency` (string: "daily", "weekly", "monthly")
- `slack_max_messages_per_channel_periodic` (integer)
- I've modified the `index_slack_messages` task:
- It now supports on-demand re-indexing of specific `target_channel_ids`.
- It allows `force_reindex_all_messages` to override `last_indexed_at`
for specified channels, using initial indexing settings or custom
date ranges (`reindex_start_date_str`, `reindex_latest_date_str`).
- It uses `slack_max_messages_per_channel_periodic` for regular
periodic updates.
- I've updated the Slack Connector Routes:
- The `/slack/{connector_id}/reindex-channels` endpoint now accepts
`channel_ids`, `force_reindex_all_messages`, `reindex_start_date`,
and `reindex_latest_date` to trigger targeted re-indexing.
- The main `/connector/{id}/index` endpoint for Slack can now accept
`force_full_reindex` to re-index all configured channels from scratch.
Frontend (`surfsense_web`):
- I've created `EditSlackConnectorConfigForm.tsx` to provide a dedicated UI
for Slack connector settings, including the new periodic indexing fields.
- I've integrated this form into the main connector editing page
(`.../connectors/[connector_id]/edit/page.tsx`).
- I've enhanced the Slack connector edit page with a "Channel Management" tab:
- UI for discovering Slack channels via `/api/v1/slack/{id}/discover-channels`.
- Allows selection of channels to be saved into
`config.slack_selected_channel_ids` when membership filter is "selected".
- UI for triggering on-demand re-indexing of selected channels via
`/api/v1/slack/{id}/reindex-channels`, with options for forcing
full re-index and specifying date ranges.
- I've updated the `useSearchSourceConnectors.ts` hook:
- I've added the `discoverSlackChannels` function.
- I've added the `reindexSlackChannels` function with parameters for
channel IDs, force flag, and date ranges.
These changes fulfill your requirements for:
1. Configurable Membership/Join Behavior (via existing `slack_membership_filter_type` and new channel selection UI).
2. Configurable Periodic Indexing (new backend schema and UI fields).
3. Granular Channel Selection (new UI for discovering and selecting channels).
4. On-Demand Re-index (new backend and UI capabilities for specific channels).
5. Initial Indexing Timestamp Range (I've verified existing backend logic and UI).
- Added support for processing YouTube videos, including transcript extraction and document creation.
- Implemented a new background task for adding YouTube video documents.
- Enhanced the connector service to search for YouTube videos and return relevant results.
- Updated frontend components to include YouTube video options in the dashboard and connector sources.
- Added necessary dependencies for YouTube transcript API.
