- Added DEXSCREENER_CONNECTOR to _CONNECTOR_TYPE_TO_SEARCHABLE in chat_deepagent.py
- This fixes LLM's inability to search DexScreener data despite it being indexed
- Root cause: connector was enabled in DB but missing from mapping, causing it to be filtered out
- Verified: LLM now successfully retrieves WETH price (~$2,442) with DexScreener citations
Related files:
- chat_deepagent.py: Added connector mapping
- knowledge_base.py: Added debug logging for DexScreener search
- connector_service.py: Fixed metadata field names (base_symbol, quote_symbol, dex)
- 85_add_dexscreener_connector.py: Migration for connector type
- Create app/utils/airtable_token_utils.py for token refresh logic
- Update all connectors to use centralized token refresh function
- Affected connectors: airtable, clickup, confluence, discord, jira, notion, slack, teams
- Improves code reusability and maintainability
- No functional changes, pure refactoring
Core Implementation:
- Add DexScreenerConnector class with public API integration
- Implement token pair data fetching and indexing
- Add API routes: add, delete, test endpoints
- Register connector in task indexers and Celery tasks
- Add DEXSCREENER_CONNECTOR enum to database models
Features:
- Support up to 50 tokens per connector
- Track prices, volume, liquidity across multiple DEXs
- EVM and Solana address validation
- Periodic sync support
- No API key required (public DexScreener API)
API Endpoints:
- POST /api/v1/connectors/dexscreener/add
- DELETE /api/v1/connectors/dexscreener
- GET /api/v1/connectors/dexscreener/test
All endpoints require JWT authentication ✅
Integration tests passing ✅
Ready for production deployment ✅
- Enhanced Google Calendar and Composio connector indexing to track and log duplicate content, preventing re-indexing of already processed events.
- Implemented robust error handling during final commits to manage integrity errors gracefully, ensuring successful indexing despite potential duplicates.
- Updated notification service to differentiate between actual errors and warnings for duplicate content, improving user feedback.
- Refactored date handling to ensure valid date ranges and adjusted end dates when necessary for better indexing accuracy.
- Added normalization for "undefined" strings to None in date parameters to prevent parsing errors.
- Improved date range validation to ensure start_date is strictly before end_date, adjusting end_date if necessary.
- Updated Google Calendar and Composio connector indexing logic to handle duplicate content more effectively, logging warnings for skipped events.
- Enhanced error handling during final commits to manage integrity errors gracefully.
- Refactored date handling in various connector indexers for consistency and reliability.
- Added user-friendly re-authentication messages for expired or revoked tokens in both Google Calendar and Gmail connectors.
- Updated error handling in indexing tasks to log specific authentication errors and provide clearer feedback to users.
- Enhanced the connector UI to handle indexing failures more effectively, improving overall user experience.
- Added methods to retrieve the starting page token and list changes in Google Drive, enabling delta sync capabilities.
- Updated Composio service to handle file download directory configuration.
- Modified indexing tasks to support delta sync, improving efficiency by processing only changed files.
- Adjusted date handling in connector tasks to allow optional start and end dates.
- Improved error handling and logging throughout the Composio indexing process.
- Updated the list_gmail_messages method to support pagination with page tokens, allowing for more efficient message retrieval.
- Modified the return structure to include next_page_token and result_size_estimate for better client-side handling.
- Improved error handling and logging throughout the Gmail indexing process, ensuring better visibility into failures.
- Implemented batch processing for Gmail messages, committing changes incrementally to prevent data loss.
- Ensured consistent timestamp updates for connectors, even when no documents are indexed, to maintain accurate UI states.
- Refactored the indexing logic to streamline message processing and enhance overall performance.
- Refactored GitHubConnector to utilize gitingest CLI via subprocess, improving performance and avoiding async issues with Celery.
- Updated ingestion method to handle repository digests more efficiently, including error handling for subprocess execution.
- Adjusted GitHub indexer to call the new synchronous ingestion method.
- Clarified documentation regarding the optional nature of the Personal Access Token for public repositories.
- Added gitingest as a dependency to streamline the ingestion of GitHub repositories.
- Refactored GitHubConnector to utilize gitingest for efficient repository digest generation, reducing API calls.
- Updated GitHub indexer to process entire repository digests, enhancing performance and simplifying the indexing process.
- Modified GitHub connect form to indicate that the Personal Access Token is optional for public repositories.
- Updated Google Drive API calls to include md5Checksum in file metadata retrieval for improved content tracking.
- Added logic to check for rename-only updates based on md5Checksum, optimizing document processing by preventing unnecessary ETL operations for unchanged content.
- Enhanced existing document update logic to handle renaming and metadata updates more effectively, particularly for Google Drive files.
- Added ClickUp OAuth authentication flow with new environment variables for client ID, client secret, and redirect URI.
- Introduced ClickUpHistoryConnector to manage OAuth-based authentication and token refresh for ClickUp API access.
- Created ClickUp connector routes for OAuth flow, including authorization and callback handling.
- Updated indexing logic to utilize the new ClickUpHistoryConnector, supporting both OAuth and legacy API token methods.
- Enhanced frontend components to reflect the new ClickUp integration and removed legacy API token forms.
- Introduced AirtableHistoryConnector to manage OAuth-based authentication and token refresh for Airtable API access.
- Added date string validation in AirtableConnector to ensure valid date inputs before processing.
- Updated indexing logic to utilize the new AirtableHistoryConnector, improving credential management and token handling.
- Added support for legacy API token authentication alongside OAuth 2.0 in ConfluenceHistoryConnector.
- Implemented logic to handle both authentication methods, ensuring backward compatibility.
- Refactored token management to accommodate legacy credentials and updated API request handling accordingly.
- Enhanced error handling for credential validation and improved logging for better traceability.
- Introduced JiraHistoryConnector to handle OAuth-based authentication and automatic token refresh for Jira API access.
- Refactored Jira indexing logic to utilize the new connector, simplifying credential management and enhancing token refresh capabilities.
- Removed legacy token handling code from the Jira indexer, streamlining the integration process.
- Ensured compatibility with both OAuth 2.0 and legacy API token methods for improved flexibility.
- Added support for Confluence OAuth with new environment variables for client ID, client secret, and redirect URI.
- Implemented Confluence connector routes for OAuth flow, including authorization and callback handling.
- Enhanced Confluence connector to support both OAuth 2.0 and legacy API token authentication methods.
- Updated Confluence indexing logic to utilize OAuth credentials with auto-refresh capabilities.
- Removed outdated Confluence UI components and adjusted frontend logic to reflect the new integration.
- Added support for Jira OAuth with new environment variables for client ID, client secret, and redirect URI.
- Implemented Jira connector routes for OAuth flow, including authorization and callback handling.
- Enhanced Jira connector to support both OAuth 2.0 and legacy API token authentication methods.
- Updated Jira indexing logic to utilize OAuth credentials with auto-refresh capabilities.
- Removed outdated Jira UI components and adjusted frontend logic to reflect the new integration.
- Introduced Discord OAuth support with new environment variables for client ID, client secret, and redirect URI.
- Implemented Discord connector routes for OAuth flow, including authorization and callback handling.
- Enhanced Discord connector to support both OAuth-based authentication and legacy bot token usage.
- Updated Discord indexing logic to utilize OAuth credentials with auto-refresh capabilities.
- Removed outdated Discord UI components and adjusted frontend logic to reflect the new integration.
- Updated SlackHistory class to enforce the use of session and connector_id for token refresh, raising a ValueError for legacy token usage.
- Simplified conditional checks for client initialization in SlackHistory.
- Cleaned up unnecessary comments and whitespace in the codebase.
- Enhanced LinearConnector and NotionHistoryConnector classes to support automatic token refresh, improving reliability in accessing APIs.
- Updated initialization to require session and connector ID, allowing for dynamic credential management.
- Introduced new credential schemas for Linear and Notion, encapsulating access and refresh tokens with expiration handling.
- Refactored indexers to utilize the new connector structure, ensuring seamless integration with the updated authentication flow.
- Improved error handling and logging during token refresh processes for better debugging and user feedback.
