I've modified `useConnectorEditPage.ts` to correctly save all Slack connector settings.
- I updated `editConnectorSchema` in `types.ts` to include a `config` field in `EditConnectorFormValues`.
- I replaced the `handleSaveChanges` function in `useConnectorEditPage.ts` with a version that reads the complete Slack configuration from `formData.config` (populated by `EditSlackConnectorConfigForm` and channel selection).
- This ensures that settings like `slack_selected_channel_ids`, `slack_membership_filter_type`, and other Slack-specific options are included in the payload to the backend and persisted correctly.
- I updated post-save logic to correctly reset the form's `config` field for Slack connectors.
This addresses the issue where changes to Slack connector configurations, other than just the bot token, were not being saved.
Updated the frontend API call to use the correct path `/api/v1/slack/:connectorId/discover-channels` instead of the previous incorrect path.
This change ensures the frontend aligns with the backend routing structure for fetching Slack channels where the bot is a member.
Addresses a 404 error you reported when attempting to discover Slack channels.
Replaced mock data in the Slack connector's channel management UI with a real API call to the backend.
- Implemented `discoverSlackChannelsAPI` in the `useConnectorEditPage` hook to fetch channels from `/api/search-source-connectors/slack/:connectorId/discover-channels`.
- Updated the `handleDiscoverChannels` function in `EditConnectorPage` to use this new API function.
- Ensured consistent `SlackChannelInfo` type usage.
This fixes a bug where incorrect Slack channel names were displayed due to the UI using stale mock data instead of actual data reflecting the bot's true channel memberships.
Backend Tests (`surfsense_backend`):
- `test_slack_history.py`:
- Tests for `SlackHistory` class methods including `get_all_channels`,
`get_conversation_history`, timestamp conversions, and message
formatting. Covers various API response scenarios, pagination,
and error handling.
- `test_connectors_indexing_tasks.py`:
- Unit tests for `index_slack_messages` task.
- Scenarios include initial indexing, periodic indexing, on-demand
re-indexing (targeted channels, force re-index, custom dates),
channel filtering, and error handling.
- Mocks `SlackHistory` and database interactions.
- `test_search_source_connectors_routes.py`:
- Tests for `/slack/{connector_id}/discover-channels` endpoint.
- Tests for `/slack/{connector_id}/reindex-channels` endpoint,
verifying parameter passing to the background task.
- Tests for the main `/search-source-connectors/{connector_id}/index`
endpoint for Slack connectors, ensuring the `force_full_reindex`
flag is handled.
Frontend Tests (`surfsense_web`):
- `components/editConnector/EditSlackConnectorConfigForm.test.tsx`:
- Tests rendering of all Slack configuration fields.
- Verifies `onConfigChange` callback functionality.
- Tests conditional rendering of periodic indexing settings.
- Checks disabled state of the form.
- `app/dashboard/[search_space_id]/connectors/[connector_id]/edit/page.test.tsx`:
- Tests for the "Channel Management" tab specific to Slack connectors.
- Verifies UI for discovering channels (mocking API calls).
- Tests selection of channels and updating the connector configuration.
- Tests UI for triggering on-demand re-indexing with various options.
- `hooks/useSearchSourceConnectors.test.ts`:
- Unit tests for the `discoverSlackChannels` function, mocking `fetch`.
- Unit tests for the `reindexSlackChannels` function, mocking `fetch`
and verifying correct request body construction.
- Includes testing of the `fetchWithAuth` helper function.
I've implemented several improvements and new features for the Slack connector,
addressing your requirements for more granular control over indexing
and data synchronization.
Key changes include:
Backend (`surfsense_backend`):
- I've updated the `SearchSourceConnector` schema for Slack to include new
configuration options:
- `slack_periodic_indexing_enabled` (boolean)
- `slack_periodic_indexing_frequency` (string: "daily", "weekly", "monthly")
- `slack_max_messages_per_channel_periodic` (integer)
- I've modified the `index_slack_messages` task:
- It now supports on-demand re-indexing of specific `target_channel_ids`.
- It allows `force_reindex_all_messages` to override `last_indexed_at`
for specified channels, using initial indexing settings or custom
date ranges (`reindex_start_date_str`, `reindex_latest_date_str`).
- It uses `slack_max_messages_per_channel_periodic` for regular
periodic updates.
- I've updated the Slack Connector Routes:
- The `/slack/{connector_id}/reindex-channels` endpoint now accepts
`channel_ids`, `force_reindex_all_messages`, `reindex_start_date`,
and `reindex_latest_date` to trigger targeted re-indexing.
- The main `/connector/{id}/index` endpoint for Slack can now accept
`force_full_reindex` to re-index all configured channels from scratch.
Frontend (`surfsense_web`):
- I've created `EditSlackConnectorConfigForm.tsx` to provide a dedicated UI
for Slack connector settings, including the new periodic indexing fields.
- I've integrated this form into the main connector editing page
(`.../connectors/[connector_id]/edit/page.tsx`).
- I've enhanced the Slack connector edit page with a "Channel Management" tab:
- UI for discovering Slack channels via `/api/v1/slack/{id}/discover-channels`.
- Allows selection of channels to be saved into
`config.slack_selected_channel_ids` when membership filter is "selected".
- UI for triggering on-demand re-indexing of selected channels via
`/api/v1/slack/{id}/reindex-channels`, with options for forcing
full re-index and specifying date ranges.
- I've updated the `useSearchSourceConnectors.ts` hook:
- I've added the `discoverSlackChannels` function.
- I've added the `reindexSlackChannels` function with parameters for
channel IDs, force flag, and date ranges.
These changes fulfill your requirements for:
1. Configurable Membership/Join Behavior (via existing `slack_membership_filter_type` and new channel selection UI).
2. Configurable Periodic Indexing (new backend schema and UI fields).
3. Granular Channel Selection (new UI for discovering and selecting channels).
4. On-Demand Re-index (new backend and UI capabilities for specific channels).
5. Initial Indexing Timestamp Range (I've verified existing backend logic and UI).
- Added support for processing YouTube videos, including transcript extraction and document creation.
- Implemented a new background task for adding YouTube video documents.
- Enhanced the connector service to search for YouTube videos and return relevant results.
- Updated frontend components to include YouTube video options in the dashboard and connector sources.
- Added necessary dependencies for YouTube transcript API.
