SurfSense/_bmad-output/dexscreener-connector-implementation-plan.md

DexScreener Connector - Implementation Plan

📋 Tổng Quan

Sau khi research kỹ DexScreener API Documentation và phân tích source code SurfSense, đây là phương án implementation chính xác nhất cho DexScreener Connector.

🔍 DexScreener API Research Findings

Base Information

• Base URL: https://api.dexscreener.com
• Authentication: KHÔNG cần API key (public API)
• Rate Limits:
  • Profile/Ads endpoints: 60 requests/minute
  • Pair/Token endpoints: 300 requests/minute
• Max Results: Search endpoint trả về tối đa 30 pairs

Core Endpoints

1. Search Pairs

GET /latest/dex/search?q={query}
Rate Limit: 300 req/min
Max Results: 30 pairs

Use Case: Tìm kiếm trading pairs theo token name, symbol, hoặc address

Response Structure:

{
  "schemaVersion": "1.0.0",
  "pairs": [
    {
      "chainId": "solana",
      "dexId": "raydium",
      "url": "https://dexscreener.com/solana/...",
      "pairAddress": "...",
      "baseToken": {
        "address": "...",
        "name": "Token Name",
        "symbol": "TKN"
      },
      "quoteToken": {
        "address": "...",
        "name": "USD Coin",
        "symbol": "USDC"
      },
      "priceNative": "0.00123",
      "priceUsd": "1.23",
      "txns": {
        "m5": { "buys": 10, "sells": 5 },
        "h1": { "buys": 100, "sells": 50 },
        "h6": { "buys": 500, "sells": 250 },
        "h24": { "buys": 2000, "sells": 1000 }
      },
      "volume": {
        "h24": 1000000,
        "h6": 250000,
        "h1": 50000,
        "m5": 5000
      },
      "priceChange": {
        "m5": 1.5,
        "h1": 5.2,
        "h6": 10.5,
        "h24": 25.3
      },
      "liquidity": {
        "usd": 500000,
        "base": 1000000,
        "quote": 500000
      },
      "fdv": 10000000,
      "marketCap": 5000000,
      "pairCreatedAt": 1640000000000
    }
  ]
}

2. Get Token Pairs

GET /latest/dex/tokens/{chainId}/{tokenAddress}
Rate Limit: 300 req/min

Use Case: Lấy tất cả pools/pairs của một token cụ thể

3. Get Specific Pair

GET /latest/dex/pairs/{chainId}/{pairAddress}
Rate Limit: 300 req/min

Use Case: Lấy thông tin chi tiết của một pair cụ thể

4. Get Multiple Tokens

GET /tokens/v1/{chainId}/{tokenAddresses}
Rate Limit: 300 req/min
Max: 30 addresses (comma-separated)

Use Case: Batch query nhiều tokens cùng lúc

🏗️ SurfSense Architecture Analysis

Pattern Đã Xác Định

1. Connector Class Pattern

File: app/connectors/{name}_connector.py

Responsibilities:

• Initialize với API credentials (nếu cần)
• Methods để fetch data từ external API
• Methods để format data sang markdown
• Error handling cho API calls

Example từ LumaConnector:

class LumaConnector:
    def __init__(self, api_key: str | None = None):
        self.api_key = api_key
        self.base_url = "https://api.lu.ma"
    
    def make_request(self, endpoint: str, params: dict | None = None):
        # Handle API calls with error handling
        
    def get_events_by_date_range(self, start_date: str, end_date: str):
        # Fetch data from API
        
    def format_event_to_markdown(self, event: dict) -> str:
        # Convert to markdown for indexing

2. Indexer Pattern

File: app/tasks/connector_indexers/{name}_indexer.py

Responsibilities:

• Async function index_{name}()
• Get connector từ database
• Extract config (API keys, etc.)
• Initialize connector class
• Fetch data từ API
• Loop qua items:
  • Generate unique_identifier_hash (để track duplicates)
  • Generate content_hash (để detect content changes)
  • Check existing documents
  • Create/Update Document objects với:
    • chunks (text chunks cho vector search)
    • embedding (vector embedding)
    • metadata (structured data)
• Batch commit to database
• Update last_indexed_at timestamp

Key Functions Used:

from app.utils.document_converters import (
    create_document_chunks,
    generate_content_hash,
    generate_document_summary,
    generate_unique_identifier_hash,
)

3. Routes Pattern

File: app/routes/{name}_add_connector_route.py

Endpoints:

• POST /connectors/{name}/add - Add/Update connector
• DELETE /connectors/{name} - Delete connector
• GET /connectors/{name}/test - Test connection

Example từ luma_add_connector_route.py:

@router.post("/connectors/luma/add")
async def add_luma_connector(
    request: AddLumaConnectorRequest,
    user: User = Depends(current_active_user),
    session: AsyncSession = Depends(get_async_session),
):
    # Check existing connector
    # Create or update SearchSourceConnector
    # Store config in connector.config JSON field

4. Database Schema

File: app/db.py

SearchSourceConnectorType Enum:

class SearchSourceConnectorType(str, Enum):
    LUMA_CONNECTOR = "LUMA_CONNECTOR"
    SLACK_CONNECTOR = "SLACK_CONNECTOR"
    # ... thêm DEXSCREENER_CONNECTOR

SearchSourceConnector Model:

class SearchSourceConnector(Base):
    id: int
    name: str
    connector_type: SearchSourceConnectorType
    config: dict  # JSON field để store API keys, settings
    search_space_id: int
    user_id: UUID
    is_indexable: bool
    last_indexed_at: datetime

5. Celery Tasks

File: app/tasks/celery_tasks/connector_tasks.py

Pattern:

@celery_app.task(name="index_luma_events", bind=True)
def index_luma_events_task(
    self,
    connector_id: int,
    search_space_id: int,
    user_id: str,
    start_date: str | None = None,
    end_date: str | None = None,
):
    # Wrapper cho async indexer function
    return asyncio.run(_index_luma_events(...))

6. Periodic Scheduler

File: app/utils/periodic_scheduler.py

Mapping:

CONNECTOR_TYPE_TO_TASK_NAME = {
    SearchSourceConnectorType.LUMA_CONNECTOR: "index_luma_events",
    # ... thêm mapping cho DexScreener
}

CONNECTOR_TYPE_TO_TASK = {
    SearchSourceConnectorType.LUMA_CONNECTOR: index_luma_events_task,
    # ... thêm task cho DexScreener
}

📝 Implementation Plan

Phase 1: Core Components

1.1. Database Schema Update

File: app/db.py

Changes:

class SearchSourceConnectorType(str, Enum):
    # ... existing types
    DEXSCREENER_CONNECTOR = "DEXSCREENER_CONNECTOR"

class DocumentType(str, Enum):
    # ... existing types
    DEXSCREENER_CONNECTOR = "DEXSCREENER_CONNECTOR"

1.2. Connector Class

File: app/connectors/dexscreener_connector.py

Xem full implementation trong artifacts.

1.3. Indexer

File: app/tasks/connector_indexers/dexscreener_indexer.py

Xem full implementation trong artifacts.

Phase 2: API Routes & Integration

2.1. Routes

File: app/routes/dexscreener_add_connector_route.py

Xem full implementation trong artifacts.

2.2. Celery Task

File: app/tasks/celery_tasks/connector_tasks.py

Add to existing file:

# Add import
from app.tasks.connector_indexers import index_dexscreener_pairs

# Add task
@celery_app.task(name="index_dexscreener_pairs", bind=True)
def index_dexscreener_pairs_task(
    self,
    connector_id: int,
    search_space_id: int,
    user_id: str,
):
    """Celery task for indexing DexScreener pairs."""
    try:
        return asyncio.run(
            _index_dexscreener_pairs(
                connector_id=connector_id,
                search_space_id=search_space_id,
                user_id=user_id,
            )
        )
    except Exception as e:
        logger.error(f"DexScreener indexing task failed: {str(e)}", exc_info=True)
        raise


async def _index_dexscreener_pairs(
    connector_id: int,
    search_space_id: int,
    user_id: str,
):
    """Async wrapper for DexScreener indexing."""
    async with get_async_session_context() as session:
        return await index_dexscreener_pairs(
            session=session,
            connector_id=connector_id,
            search_space_id=search_space_id,
            user_id=user_id,
        )

2.3. Periodic Scheduler

File: app/utils/periodic_scheduler.py

Add to existing mappings:

# Add to CONNECTOR_TYPE_TO_TASK_NAME
CONNECTOR_TYPE_TO_TASK_NAME = {
    # ... existing mappings
    SearchSourceConnectorType.DEXSCREENER_CONNECTOR: "index_dexscreener_pairs",
}

# Add import
from app.tasks.celery_tasks.connector_tasks import index_dexscreener_pairs_task

# Add to CONNECTOR_TYPE_TO_TASK
CONNECTOR_TYPE_TO_TASK = {
    # ... existing mappings
    SearchSourceConnectorType.DEXSCREENER_CONNECTOR: index_dexscreener_pairs_task,
}

2.4. Routes Registration

File: app/routes/__init__.py

Add:

# Add import
from app.routes.dexscreener_add_connector_route import router as dexscreener_add_connector_router

# Add to router includes (after other connector routes)
router.include_router(dexscreener_add_connector_router)

2.5. Indexer Export

File: app/tasks/connector_indexers/__init__.py

Add:

# Add import
from .dexscreener_indexer import index_dexscreener_pairs

# Add to __all__
__all__ = [
    # ... existing exports
    "index_dexscreener_pairs",
]

🔄 Usage Flow

1. Add Connector via API

curl -X POST "http://localhost:8000/api/connectors/dexscreener/add" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "space_id": 1,
    "tokens": [
      {
        "chain": "solana",
        "address": "So11111111111111111111111111111111111111112",
        "name": "Wrapped SOL"
      },
      {
        "chain": "ethereum",
        "address": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
        "name": "Wrapped ETH"
      }
    ]
  }'

2. Test Connection

curl -X GET "http://localhost:8000/api/connectors/dexscreener/test?chain=solana&token_address=So11111111111111111111111111111111111111112" \
  -H "Authorization: Bearer YOUR_TOKEN"

3. Trigger Manual Indexing

Indexing sẽ được trigger tự động qua:

• Periodic scheduler: Mỗi 60 phút (configurable)
• Manual trigger: Qua search_source_connectors_routes.py endpoint

4. Search Indexed Data

Data được index sẽ tự động available trong:

• AI Chat với context từ DexScreener
• Search results
• Document retrieval

⚠️ Important Considerations

Rate Limiting

• DexScreener API: 300 requests/minute
• Với 50 tokens tracked, mỗi lần index = 50 requests
• Recommended indexing interval: 60 minutes
• Implement exponential backoff nếu hit rate limit

Data Freshness

• Crypto market data thay đổi nhanh
• Consider shorter intervals (15-30 min) cho high-priority tokens
• Implement priority queue cho important tokens

Storage Optimization

• Mỗi pair = 1 document với chunks
• 50 tokens × 5 pairs average = 250 documents
• Monitor storage usage và implement cleanup cho old data

Error Handling

• Network failures: Retry với exponential backoff
• API errors: Log và skip, không block toàn bộ indexing
• Invalid data: Validate trước khi index

🎯 Next Steps

1. Implement Phase 1: Core components (connector, indexer, DB schema)
2. Test locally: Verify API calls và data formatting
3. Implement Phase 2: Routes và integration
4. Test end-to-end: Add connector → Index → Search
5. Deploy: Monitor performance và adjust intervals
6. Optimize: Based on usage patterns và feedback

📊 Success Metrics

• ✅ Connector successfully fetches data from DexScreener API
• ✅ Data được format chính xác sang markdown
• ✅ Documents được index với proper chunks và embeddings
• ✅ Search results include DexScreener data
• ✅ AI Chat có context từ crypto market data
• ✅ Periodic indexing runs without errors
• ✅ Rate limits được respect

---

Note: Implementation này dựa trên:

• Official DexScreener API Documentation
• Existing SurfSense connector patterns (Luma, Slack, etc.)
• Best practices từ production connectors