Backend Tests (`surfsense_backend`):
- `test_slack_history.py`:
- Tests for `SlackHistory` class methods including `get_all_channels`,
`get_conversation_history`, timestamp conversions, and message
formatting. Covers various API response scenarios, pagination,
and error handling.
- `test_connectors_indexing_tasks.py`:
- Unit tests for `index_slack_messages` task.
- Scenarios include initial indexing, periodic indexing, on-demand
re-indexing (targeted channels, force re-index, custom dates),
channel filtering, and error handling.
- Mocks `SlackHistory` and database interactions.
- `test_search_source_connectors_routes.py`:
- Tests for `/slack/{connector_id}/discover-channels` endpoint.
- Tests for `/slack/{connector_id}/reindex-channels` endpoint,
verifying parameter passing to the background task.
- Tests for the main `/search-source-connectors/{connector_id}/index`
endpoint for Slack connectors, ensuring the `force_full_reindex`
flag is handled.
Frontend Tests (`surfsense_web`):
- `components/editConnector/EditSlackConnectorConfigForm.test.tsx`:
- Tests rendering of all Slack configuration fields.
- Verifies `onConfigChange` callback functionality.
- Tests conditional rendering of periodic indexing settings.
- Checks disabled state of the form.
- `app/dashboard/[search_space_id]/connectors/[connector_id]/edit/page.test.tsx`:
- Tests for the "Channel Management" tab specific to Slack connectors.
- Verifies UI for discovering channels (mocking API calls).
- Tests selection of channels and updating the connector configuration.
- Tests UI for triggering on-demand re-indexing with various options.
- `hooks/useSearchSourceConnectors.test.ts`:
- Unit tests for the `discoverSlackChannels` function, mocking `fetch`.
- Unit tests for the `reindexSlackChannels` function, mocking `fetch`
and verifying correct request body construction.
- Includes testing of the `fetchWithAuth` helper function.
I've implemented several improvements and new features for the Slack connector,
addressing your requirements for more granular control over indexing
and data synchronization.
Key changes include:
Backend (`surfsense_backend`):
- I've updated the `SearchSourceConnector` schema for Slack to include new
configuration options:
- `slack_periodic_indexing_enabled` (boolean)
- `slack_periodic_indexing_frequency` (string: "daily", "weekly", "monthly")
- `slack_max_messages_per_channel_periodic` (integer)
- I've modified the `index_slack_messages` task:
- It now supports on-demand re-indexing of specific `target_channel_ids`.
- It allows `force_reindex_all_messages` to override `last_indexed_at`
for specified channels, using initial indexing settings or custom
date ranges (`reindex_start_date_str`, `reindex_latest_date_str`).
- It uses `slack_max_messages_per_channel_periodic` for regular
periodic updates.
- I've updated the Slack Connector Routes:
- The `/slack/{connector_id}/reindex-channels` endpoint now accepts
`channel_ids`, `force_reindex_all_messages`, `reindex_start_date`,
and `reindex_latest_date` to trigger targeted re-indexing.
- The main `/connector/{id}/index` endpoint for Slack can now accept
`force_full_reindex` to re-index all configured channels from scratch.
Frontend (`surfsense_web`):
- I've created `EditSlackConnectorConfigForm.tsx` to provide a dedicated UI
for Slack connector settings, including the new periodic indexing fields.
- I've integrated this form into the main connector editing page
(`.../connectors/[connector_id]/edit/page.tsx`).
- I've enhanced the Slack connector edit page with a "Channel Management" tab:
- UI for discovering Slack channels via `/api/v1/slack/{id}/discover-channels`.
- Allows selection of channels to be saved into
`config.slack_selected_channel_ids` when membership filter is "selected".
- UI for triggering on-demand re-indexing of selected channels via
`/api/v1/slack/{id}/reindex-channels`, with options for forcing
full re-index and specifying date ranges.
- I've updated the `useSearchSourceConnectors.ts` hook:
- I've added the `discoverSlackChannels` function.
- I've added the `reindexSlackChannels` function with parameters for
channel IDs, force flag, and date ranges.
These changes fulfill your requirements for:
1. Configurable Membership/Join Behavior (via existing `slack_membership_filter_type` and new channel selection UI).
2. Configurable Periodic Indexing (new backend schema and UI fields).
3. Granular Channel Selection (new UI for discovering and selecting channels).
4. On-Demand Re-index (new backend and UI capabilities for specific channels).
5. Initial Indexing Timestamp Range (I've verified existing backend logic and UI).
This commit introduces Phase 1 of Slack connector enhancements, providing more granular control over the indexing process.
Key changes include:
1. **Updated Connector Configuration Schema (`search_source_connector.py`):**
* Added new fields to `SearchSourceConnector.config` for Slack:
* `slack_membership_filter_type`: ("all_member_channels", "selected_member_channels")
* `slack_selected_channel_ids`: List of channel IDs for selective indexing.
* `slack_indexing_frequency`: For periodic re-indexing schedule.
* `slack_initial_indexing_days`: Depth for initial message import.
* `slack_initial_max_messages_per_channel`: Max messages for initial import.
* Updated Pydantic validation logic for these new fields.
2. **Modified Indexing Task (`connectors_indexing_tasks.py`):**
* `index_slack_messages` now reads and utilizes the new configuration fields.
* Channel filtering is applied based on `slack_membership_filter_type` and `slack_selected_channel_ids`.
* Initial indexing runs use `slack_initial_indexing_days` and `slack_initial_max_messages_per_channel` to determine the scope of the first message fetch.
* The Slack API call was changed from `get_history_by_date_range` to a direct call to `get_conversation_history` using more precise timestamp-based parameters (`oldest`, `latest`) and message limits.
3. **New API Endpoint for Channel Discovery (`search_source_connectors_routes.py`):**
* Added `GET /connectors/slack/{connector_id}/discover-channels`.
* This endpoint allows the frontend to fetch a list of channels where the bot is currently a member, facilitating your selection for indexing.
4. **New API Endpoint for Specific Channel Re-index Trigger (`search_source_connectors_routes.py`):**
* Added `POST /slack/{connector_id}/reindex-channels`.
* This endpoint is structured to accept a list of channel IDs for targeted re-indexing.
* Note: The underlying modification to `index_slack_messages` to fully handle the specialized logic for these targeted re-indexes (ignoring last_indexed_at, etc.) was not completed. This endpoint sets up the route and request model.
These changes lay the backend foundation for a more configurable and user-friendly Slack integration. Frontend implementation for these configurations and the completion of the specific channel re-indexing logic are pending.
I modified the Slack indexing process in `index_slack_messages` to skip any channel (public or private) where the bot's `is_member` status is false.
Previously, the indexing process would only explicitly skip private channels where the bot was not a member. For public channels, it would attempt to fetch history, relying on a `not_in_channel` error during history fetching to implicitly skip.
This change makes the skipping behavior explicit for all channel types based on the `is_member` flag retrieved from `conversations.list` API call. This aims to reduce unnecessary API calls to `conversations.history` for channels the bot hasn't been invited to, potentially speeding up the indexing process and avoiding hitting rate limits unnecessarily.
The `is_member` flag is reliably provided by the `SlackHistory.get_all_channels` method. Logging messages have been updated to reflect this change.
Here's a rundown of what I did:
Fix: Robust Slack rate limiting, error handling & GitHub org repos
This update delivers comprehensive improvements to Slack connector stability and enhances the GitHub connector.
**Slack Connector (`slack_history.py`, `connectors_indexing_tasks.py`):**
- I've implemented proactive delays (1.2s for `conversations.history`, 3s for `conversations.list` pagination) and `Retry-After` header handling for 429 rate limit errors across `conversations.list`, `conversations.history`, and `users.info` API calls.
- I'll now gracefully handle `not_in_channel` errors when fetching conversation history by logging a warning and skipping the channel.
- I've refactored channel info fetching: `get_all_channels` now returns richer channel data (including `is_member`, `is_private`).
- I've removed direct calls to `conversations.info` from `connectors_indexing_tasks.py`, using the richer data from `get_all_channels` instead, to prevent associated rate limits.
- I corrected a `SyntaxError` (non-printable character) in `slack_history.py`.
- I've enhanced logging for rate limit actions, delays, and errors.
- I've updated unit tests in `test_slack_history.py` to cover all new logic.
**GitHub Connector (`github_connector.py`):**
- I've modified `get_user_repositories` to fetch all repositories accessible by you (owned, collaborated, organization) by changing the API call parameter from `type='owner'` to `type='all'`.
- I've included unit tests in `test_github_connector.py` for this change.
- Added support for processing YouTube videos, including transcript extraction and document creation.
- Implemented a new background task for adding YouTube video documents.
- Enhanced the connector service to search for YouTube videos and return relevant results.
- Updated frontend components to include YouTube video options in the dashboard and connector sources.
- Added necessary dependencies for YouTube transcript API.
