I've implemented several improvements and new features for the Slack connector,
addressing your requirements for more granular control over indexing
and data synchronization.
Key changes include:
Backend (`surfsense_backend`):
- I've updated the `SearchSourceConnector` schema for Slack to include new
configuration options:
- `slack_periodic_indexing_enabled` (boolean)
- `slack_periodic_indexing_frequency` (string: "daily", "weekly", "monthly")
- `slack_max_messages_per_channel_periodic` (integer)
- I've modified the `index_slack_messages` task:
- It now supports on-demand re-indexing of specific `target_channel_ids`.
- It allows `force_reindex_all_messages` to override `last_indexed_at`
for specified channels, using initial indexing settings or custom
date ranges (`reindex_start_date_str`, `reindex_latest_date_str`).
- It uses `slack_max_messages_per_channel_periodic` for regular
periodic updates.
- I've updated the Slack Connector Routes:
- The `/slack/{connector_id}/reindex-channels` endpoint now accepts
`channel_ids`, `force_reindex_all_messages`, `reindex_start_date`,
and `reindex_latest_date` to trigger targeted re-indexing.
- The main `/connector/{id}/index` endpoint for Slack can now accept
`force_full_reindex` to re-index all configured channels from scratch.
Frontend (`surfsense_web`):
- I've created `EditSlackConnectorConfigForm.tsx` to provide a dedicated UI
for Slack connector settings, including the new periodic indexing fields.
- I've integrated this form into the main connector editing page
(`.../connectors/[connector_id]/edit/page.tsx`).
- I've enhanced the Slack connector edit page with a "Channel Management" tab:
- UI for discovering Slack channels via `/api/v1/slack/{id}/discover-channels`.
- Allows selection of channels to be saved into
`config.slack_selected_channel_ids` when membership filter is "selected".
- UI for triggering on-demand re-indexing of selected channels via
`/api/v1/slack/{id}/reindex-channels`, with options for forcing
full re-index and specifying date ranges.
- I've updated the `useSearchSourceConnectors.ts` hook:
- I've added the `discoverSlackChannels` function.
- I've added the `reindexSlackChannels` function with parameters for
channel IDs, force flag, and date ranges.
These changes fulfill your requirements for:
1. Configurable Membership/Join Behavior (via existing `slack_membership_filter_type` and new channel selection UI).
2. Configurable Periodic Indexing (new backend schema and UI fields).
3. Granular Channel Selection (new UI for discovering and selecting channels).
4. On-Demand Re-index (new backend and UI capabilities for specific channels).
5. Initial Indexing Timestamp Range (I've verified existing backend logic and UI).
This commit introduces Phase 1 of Slack connector enhancements, providing more granular control over the indexing process.
Key changes include:
1. **Updated Connector Configuration Schema (`search_source_connector.py`):**
* Added new fields to `SearchSourceConnector.config` for Slack:
* `slack_membership_filter_type`: ("all_member_channels", "selected_member_channels")
* `slack_selected_channel_ids`: List of channel IDs for selective indexing.
* `slack_indexing_frequency`: For periodic re-indexing schedule.
* `slack_initial_indexing_days`: Depth for initial message import.
* `slack_initial_max_messages_per_channel`: Max messages for initial import.
* Updated Pydantic validation logic for these new fields.
2. **Modified Indexing Task (`connectors_indexing_tasks.py`):**
* `index_slack_messages` now reads and utilizes the new configuration fields.
* Channel filtering is applied based on `slack_membership_filter_type` and `slack_selected_channel_ids`.
* Initial indexing runs use `slack_initial_indexing_days` and `slack_initial_max_messages_per_channel` to determine the scope of the first message fetch.
* The Slack API call was changed from `get_history_by_date_range` to a direct call to `get_conversation_history` using more precise timestamp-based parameters (`oldest`, `latest`) and message limits.
3. **New API Endpoint for Channel Discovery (`search_source_connectors_routes.py`):**
* Added `GET /connectors/slack/{connector_id}/discover-channels`.
* This endpoint allows the frontend to fetch a list of channels where the bot is currently a member, facilitating your selection for indexing.
4. **New API Endpoint for Specific Channel Re-index Trigger (`search_source_connectors_routes.py`):**
* Added `POST /slack/{connector_id}/reindex-channels`.
* This endpoint is structured to accept a list of channel IDs for targeted re-indexing.
* Note: The underlying modification to `index_slack_messages` to fully handle the specialized logic for these targeted re-indexes (ignoring last_indexed_at, etc.) was not completed. This endpoint sets up the route and request model.
These changes lay the backend foundation for a more configurable and user-friendly Slack integration. Frontend implementation for these configurations and the completion of the specific channel re-indexing logic are pending.
