Updated the frontend API call to use the correct path `/api/v1/slack/:connectorId/discover-channels` instead of the previous incorrect path.
This change ensures the frontend aligns with the backend routing structure for fetching Slack channels where the bot is a member.
Addresses a 404 error you reported when attempting to discover Slack channels.
Replaced mock data in the Slack connector's channel management UI with a real API call to the backend.
- Implemented `discoverSlackChannelsAPI` in the `useConnectorEditPage` hook to fetch channels from `/api/search-source-connectors/slack/:connectorId/discover-channels`.
- Updated the `handleDiscoverChannels` function in `EditConnectorPage` to use this new API function.
- Ensured consistent `SlackChannelInfo` type usage.
This fixes a bug where incorrect Slack channel names were displayed due to the UI using stale mock data instead of actual data reflecting the bot's true channel memberships.
Backend Tests (`surfsense_backend`):
- `test_slack_history.py`:
- Tests for `SlackHistory` class methods including `get_all_channels`,
`get_conversation_history`, timestamp conversions, and message
formatting. Covers various API response scenarios, pagination,
and error handling.
- `test_connectors_indexing_tasks.py`:
- Unit tests for `index_slack_messages` task.
- Scenarios include initial indexing, periodic indexing, on-demand
re-indexing (targeted channels, force re-index, custom dates),
channel filtering, and error handling.
- Mocks `SlackHistory` and database interactions.
- `test_search_source_connectors_routes.py`:
- Tests for `/slack/{connector_id}/discover-channels` endpoint.
- Tests for `/slack/{connector_id}/reindex-channels` endpoint,
verifying parameter passing to the background task.
- Tests for the main `/search-source-connectors/{connector_id}/index`
endpoint for Slack connectors, ensuring the `force_full_reindex`
flag is handled.
Frontend Tests (`surfsense_web`):
- `components/editConnector/EditSlackConnectorConfigForm.test.tsx`:
- Tests rendering of all Slack configuration fields.
- Verifies `onConfigChange` callback functionality.
- Tests conditional rendering of periodic indexing settings.
- Checks disabled state of the form.
- `app/dashboard/[search_space_id]/connectors/[connector_id]/edit/page.test.tsx`:
- Tests for the "Channel Management" tab specific to Slack connectors.
- Verifies UI for discovering channels (mocking API calls).
- Tests selection of channels and updating the connector configuration.
- Tests UI for triggering on-demand re-indexing with various options.
- `hooks/useSearchSourceConnectors.test.ts`:
- Unit tests for the `discoverSlackChannels` function, mocking `fetch`.
- Unit tests for the `reindexSlackChannels` function, mocking `fetch`
and verifying correct request body construction.
- Includes testing of the `fetchWithAuth` helper function.
I've implemented several improvements and new features for the Slack connector,
addressing your requirements for more granular control over indexing
and data synchronization.
Key changes include:
Backend (`surfsense_backend`):
- I've updated the `SearchSourceConnector` schema for Slack to include new
configuration options:
- `slack_periodic_indexing_enabled` (boolean)
- `slack_periodic_indexing_frequency` (string: "daily", "weekly", "monthly")
- `slack_max_messages_per_channel_periodic` (integer)
- I've modified the `index_slack_messages` task:
- It now supports on-demand re-indexing of specific `target_channel_ids`.
- It allows `force_reindex_all_messages` to override `last_indexed_at`
for specified channels, using initial indexing settings or custom
date ranges (`reindex_start_date_str`, `reindex_latest_date_str`).
- It uses `slack_max_messages_per_channel_periodic` for regular
periodic updates.
- I've updated the Slack Connector Routes:
- The `/slack/{connector_id}/reindex-channels` endpoint now accepts
`channel_ids`, `force_reindex_all_messages`, `reindex_start_date`,
and `reindex_latest_date` to trigger targeted re-indexing.
- The main `/connector/{id}/index` endpoint for Slack can now accept
`force_full_reindex` to re-index all configured channels from scratch.
Frontend (`surfsense_web`):
- I've created `EditSlackConnectorConfigForm.tsx` to provide a dedicated UI
for Slack connector settings, including the new periodic indexing fields.
- I've integrated this form into the main connector editing page
(`.../connectors/[connector_id]/edit/page.tsx`).
- I've enhanced the Slack connector edit page with a "Channel Management" tab:
- UI for discovering Slack channels via `/api/v1/slack/{id}/discover-channels`.
- Allows selection of channels to be saved into
`config.slack_selected_channel_ids` when membership filter is "selected".
- UI for triggering on-demand re-indexing of selected channels via
`/api/v1/slack/{id}/reindex-channels`, with options for forcing
full re-index and specifying date ranges.
- I've updated the `useSearchSourceConnectors.ts` hook:
- I've added the `discoverSlackChannels` function.
- I've added the `reindexSlackChannels` function with parameters for
channel IDs, force flag, and date ranges.
These changes fulfill your requirements for:
1. Configurable Membership/Join Behavior (via existing `slack_membership_filter_type` and new channel selection UI).
2. Configurable Periodic Indexing (new backend schema and UI fields).
3. Granular Channel Selection (new UI for discovering and selecting channels).
4. On-Demand Re-index (new backend and UI capabilities for specific channels).
5. Initial Indexing Timestamp Range (I've verified existing backend logic and UI).
- Added support for processing YouTube videos, including transcript extraction and document creation.
- Implemented a new background task for adding YouTube video documents.
- Enhanced the connector service to search for YouTube videos and return relevant results.
- Updated frontend components to include YouTube video options in the dashboard and connector sources.
- Added necessary dependencies for YouTube transcript API.
