Backend Tests (`surfsense_backend`):
- `test_slack_history.py`:
- Tests for `SlackHistory` class methods including `get_all_channels`,
`get_conversation_history`, timestamp conversions, and message
formatting. Covers various API response scenarios, pagination,
and error handling.
- `test_connectors_indexing_tasks.py`:
- Unit tests for `index_slack_messages` task.
- Scenarios include initial indexing, periodic indexing, on-demand
re-indexing (targeted channels, force re-index, custom dates),
channel filtering, and error handling.
- Mocks `SlackHistory` and database interactions.
- `test_search_source_connectors_routes.py`:
- Tests for `/slack/{connector_id}/discover-channels` endpoint.
- Tests for `/slack/{connector_id}/reindex-channels` endpoint,
verifying parameter passing to the background task.
- Tests for the main `/search-source-connectors/{connector_id}/index`
endpoint for Slack connectors, ensuring the `force_full_reindex`
flag is handled.
Frontend Tests (`surfsense_web`):
- `components/editConnector/EditSlackConnectorConfigForm.test.tsx`:
- Tests rendering of all Slack configuration fields.
- Verifies `onConfigChange` callback functionality.
- Tests conditional rendering of periodic indexing settings.
- Checks disabled state of the form.
- `app/dashboard/[search_space_id]/connectors/[connector_id]/edit/page.test.tsx`:
- Tests for the "Channel Management" tab specific to Slack connectors.
- Verifies UI for discovering channels (mocking API calls).
- Tests selection of channels and updating the connector configuration.
- Tests UI for triggering on-demand re-indexing with various options.
- `hooks/useSearchSourceConnectors.test.ts`:
- Unit tests for the `discoverSlackChannels` function, mocking `fetch`.
- Unit tests for the `reindexSlackChannels` function, mocking `fetch`
and verifying correct request body construction.
- Includes testing of the `fetchWithAuth` helper function.
This commit addresses recurring `SyntaxError: invalid non-printable character U+001B`
errors in `surfsense_backend/app/connectors/slack_history.py`.
The file was cleaned to remove all occurrences of the
U+001B (ESCAPE) character. This ensures that previously introduced
problematic control characters are fully removed, allowing the application
to parse and load the module correctly.
Here's a rundown of what I did:
Fix: Robust Slack rate limiting, error handling & GitHub org repos
This update delivers comprehensive improvements to Slack connector stability and enhances the GitHub connector.
**Slack Connector (`slack_history.py`, `connectors_indexing_tasks.py`):**
- I've implemented proactive delays (1.2s for `conversations.history`, 3s for `conversations.list` pagination) and `Retry-After` header handling for 429 rate limit errors across `conversations.list`, `conversations.history`, and `users.info` API calls.
- I'll now gracefully handle `not_in_channel` errors when fetching conversation history by logging a warning and skipping the channel.
- I've refactored channel info fetching: `get_all_channels` now returns richer channel data (including `is_member`, `is_private`).
- I've removed direct calls to `conversations.info` from `connectors_indexing_tasks.py`, using the richer data from `get_all_channels` instead, to prevent associated rate limits.
- I corrected a `SyntaxError` (non-printable character) in `slack_history.py`.
- I've enhanced logging for rate limit actions, delays, and errors.
- I've updated unit tests in `test_slack_history.py` to cover all new logic.
**GitHub Connector (`github_connector.py`):**
- I've modified `get_user_repositories` to fetch all repositories accessible by you (owned, collaborated, organization) by changing the API call parameter from `type='owner'` to `type='all'`.
- I've included unit tests in `test_github_connector.py` for this change.
This commit includes two main improvements:
1. Slack Connector (`slack_history.py`):
- Addresses API rate limiting for `conversations.list` by introducing a 3-second delay between paginated calls.
- Implements handling for the `Retry-After` header when HTTP 429 errors occur.
- Fixes a `SyntaxError` caused by a non-printable character accidentally introduced in a previous modification.
- Adds comprehensive unit tests for the rate limiting and retry logic in `test_slack_history.py`.
2. GitHub Connector (`github_connector.py`):
- Modifies `get_user_repositories` to fetch all repositories accessible by you (including organization repositories) by changing the API call parameter from `type='owner'` to `type='all'`.
- Adds unit tests in `test_github_connector.py` to verify this change and other connector functionalities.
The `get_all_channels` method in `slack_history.py` was making paginated
requests to `conversations.list` without any delay, leading to HTTP 429
errors when fetching channels from large Slack workspaces.
This commit introduces the following changes:
- Adds a 3-second delay between paginated calls to `conversations.list`
to comply with Slack's Tier 2 rate limits (approx. 20 requests/minute).
- Implements handling for the `Retry-After` header when a 429 error is
received. The system will wait for the specified duration before
retrying. If the header is missing or invalid, a default of 60 seconds
is used.
- Adds comprehensive unit tests to verify the new delay and retry logic,
covering scenarios with and without the `Retry-After` header, as well
as other API errors.
