- Added DEXSCREENER_CONNECTOR to _CONNECTOR_TYPE_TO_SEARCHABLE in chat_deepagent.py
- This fixes LLM's inability to search DexScreener data despite it being indexed
- Root cause: connector was enabled in DB but missing from mapping, causing it to be filtered out
- Verified: LLM now successfully retrieves WETH price (~$2,442) with DexScreener citations
Related files:
- chat_deepagent.py: Added connector mapping
- knowledge_base.py: Added debug logging for DexScreener search
- connector_service.py: Fixed metadata field names (base_symbol, quote_symbol, dex)
- 85_add_dexscreener_connector.py: Migration for connector type
- Create app/utils/airtable_token_utils.py for token refresh logic
- Update all connectors to use centralized token refresh function
- Affected connectors: airtable, clickup, confluence, discord, jira, notion, slack, teams
- Improves code reusability and maintainability
- No functional changes, pure refactoring
Core Implementation:
- Add DexScreenerConnector class with public API integration
- Implement token pair data fetching and indexing
- Add API routes: add, delete, test endpoints
- Register connector in task indexers and Celery tasks
- Add DEXSCREENER_CONNECTOR enum to database models
Features:
- Support up to 50 tokens per connector
- Track prices, volume, liquidity across multiple DEXs
- EVM and Solana address validation
- Periodic sync support
- No API key required (public DexScreener API)
API Endpoints:
- POST /api/v1/connectors/dexscreener/add
- DELETE /api/v1/connectors/dexscreener
- GET /api/v1/connectors/dexscreener/test
All endpoints require JWT authentication ✅
Integration tests passing ✅
Ready for production deployment ✅
- Enhanced the logic for determining success cases during indexing by distinguishing between duplicate warnings and empty results.
- Updated notification messages to provide clearer feedback for empty results, improving user understanding of indexing outcomes.
- Ensured that notifications reflect accurate statuses, maintaining consistency in user feedback during the indexing process.
- Enhanced Google Calendar and Composio connector indexing to track and log duplicate content, preventing re-indexing of already processed events.
- Implemented robust error handling during final commits to manage integrity errors gracefully, ensuring successful indexing despite potential duplicates.
- Updated notification service to differentiate between actual errors and warnings for duplicate content, improving user feedback.
- Refactored date handling to ensure valid date ranges and adjusted end dates when necessary for better indexing accuracy.
- Added normalization for "undefined" strings to None in date parameters to prevent parsing errors.
- Improved date range validation to ensure start_date is strictly before end_date, adjusting end_date if necessary.
- Updated Google Calendar and Composio connector indexing logic to handle duplicate content more effectively, logging warnings for skipped events.
- Enhanced error handling during final commits to manage integrity errors gracefully.
- Refactored date handling in various connector indexers for consistency and reliability.
- Updated the Composio connector naming logic to dynamically generate user-friendly names based on existing connectors.
- Introduced new utility functions for counting connectors and retrieving base names for specific connector types.
- Enhanced the UI components to improve accessibility and focus management, ensuring a better user experience when interacting with connector dialogs.
- Refactored Composio connector configuration components to improve modularity and maintainability.
- Simplified the ComposioCalendarConfig, ComposioGmailConfig, and ComposioDriveConfig components by removing unnecessary state management and UI elements.
- Added functionality to remove selected folders and files in the Google Drive and Composio Drive configurations, enhancing user experience.
- Updated connector display names for better clarity in the UI.
- Improved the overall structure of the connector edit view for better readability and usability.
- Added methods to retrieve the starting page token and list changes in Google Drive, enabling delta sync capabilities.
- Updated Composio service to handle file download directory configuration.
- Modified indexing tasks to support delta sync, improving efficiency by processing only changed files.
- Adjusted date handling in connector tasks to allow optional start and end dates.
- Improved error handling and logging throughout the Composio indexing process.
- Added support for extracting connected account IDs from query parameters in the Composio callback, accommodating both camelCase and snake_case formats.
- Improved logging for received query parameters in the Composio callback to enhance debugging.
- Updated Google Drive folder listing logs to remove unnecessary emoji for consistency.
- Expanded the connector dialog to include Composio Google Drive as a recognized connector type, improving user interface clarity.
- Enhanced the handling of file content from Composio, supporting both binary and text files with appropriate processing methods.
- Introduced robust error logging and handling for file content extraction, ensuring better visibility into issues during processing.
- Updated the indexing logic to accommodate new content processing methods, improving overall reliability and user feedback on errors.
- Added temporary file handling for binary files to facilitate text extraction using the ETL service.
- Added a new endpoint to list folders and files in a user's Composio Google Drive, supporting hierarchical structure.
- Implemented UI components for selecting specific folders and files to index, improving user control over indexing options.
- Introduced indexing options for maximum files per folder and inclusion of subfolders, allowing for customizable indexing behavior.
- Enhanced error handling and logging for Composio Drive operations, ensuring better visibility into issues during file retrieval and indexing.
- Updated the Composio configuration component to reflect new selection capabilities and indexing options.
- Updated the list_gmail_messages method to support pagination with page tokens, allowing for more efficient message retrieval.
- Modified the return structure to include next_page_token and result_size_estimate for better client-side handling.
- Improved error handling and logging throughout the Gmail indexing process, ensuring better visibility into failures.
- Implemented batch processing for Gmail messages, committing changes incrementally to prevent data loss.
- Ensured consistent timestamp updates for connectors, even when no documents are indexed, to maintain accurate UI states.
- Refactored the indexing logic to streamline message processing and enhance overall performance.
