mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-05-08 23:32:40 +02:00
docs: Add BMAD stories and implementation plan for DexScreener
- Add Story 1.1: DexScreener Connector Backend (complete) - Add Story 1.2: DexScreener Connector Frontend UI (ready for dev) - Add implementation plan with architecture decisions - Include assets and documentation structure - Stories follow BMAD methodology for AI-driven development
This commit is contained in:
API Test Bot
parent
8dd41adae4
commit
44dc7904f3
8 changed files with 1674 additions and 0 deletions
473
_bmad-output/dexscreener-connector-implementation-plan.md
Normal file
473
_bmad-output/dexscreener-connector-implementation-plan.md
Normal file
|
|
@ -0,0 +1,473 @@
|
|||
# DexScreener Connector - Implementation Plan
|
||||
|
||||
## 📋 Tổng Quan
|
||||
|
||||
Sau khi research kỹ [DexScreener API Documentation](https://docs.dexscreener.com/api/reference) và phân tích source code SurfSense, đây là phương án implementation chính xác nhất cho DexScreener Connector.
|
||||
|
||||
## 🔍 DexScreener API Research Findings
|
||||
|
||||
### Base Information
|
||||
- **Base URL**: `https://api.dexscreener.com`
|
||||
- **Authentication**: KHÔNG cần API key (public API)
|
||||
- **Rate Limits**:
|
||||
- Profile/Ads endpoints: 60 requests/minute
|
||||
- Pair/Token endpoints: **300 requests/minute**
|
||||
- **Max Results**: Search endpoint trả về tối đa 30 pairs
|
||||
|
||||
### Core Endpoints
|
||||
|
||||
#### 1. Search Pairs
|
||||
```
|
||||
GET /latest/dex/search?q={query}
|
||||
Rate Limit: 300 req/min
|
||||
Max Results: 30 pairs
|
||||
```
|
||||
|
||||
**Use Case**: Tìm kiếm trading pairs theo token name, symbol, hoặc address
|
||||
|
||||
**Response Structure**:
|
||||
```json
|
||||
{
|
||||
"schemaVersion": "1.0.0",
|
||||
"pairs": [
|
||||
{
|
||||
"chainId": "solana",
|
||||
"dexId": "raydium",
|
||||
"url": "https://dexscreener.com/solana/...",
|
||||
"pairAddress": "...",
|
||||
"baseToken": {
|
||||
"address": "...",
|
||||
"name": "Token Name",
|
||||
"symbol": "TKN"
|
||||
},
|
||||
"quoteToken": {
|
||||
"address": "...",
|
||||
"name": "USD Coin",
|
||||
"symbol": "USDC"
|
||||
},
|
||||
"priceNative": "0.00123",
|
||||
"priceUsd": "1.23",
|
||||
"txns": {
|
||||
"m5": { "buys": 10, "sells": 5 },
|
||||
"h1": { "buys": 100, "sells": 50 },
|
||||
"h6": { "buys": 500, "sells": 250 },
|
||||
"h24": { "buys": 2000, "sells": 1000 }
|
||||
},
|
||||
"volume": {
|
||||
"h24": 1000000,
|
||||
"h6": 250000,
|
||||
"h1": 50000,
|
||||
"m5": 5000
|
||||
},
|
||||
"priceChange": {
|
||||
"m5": 1.5,
|
||||
"h1": 5.2,
|
||||
"h6": 10.5,
|
||||
"h24": 25.3
|
||||
},
|
||||
"liquidity": {
|
||||
"usd": 500000,
|
||||
"base": 1000000,
|
||||
"quote": 500000
|
||||
},
|
||||
"fdv": 10000000,
|
||||
"marketCap": 5000000,
|
||||
"pairCreatedAt": 1640000000000
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Get Token Pairs
|
||||
```
|
||||
GET /latest/dex/tokens/{chainId}/{tokenAddress}
|
||||
Rate Limit: 300 req/min
|
||||
```
|
||||
|
||||
**Use Case**: Lấy tất cả pools/pairs của một token cụ thể
|
||||
|
||||
#### 3. Get Specific Pair
|
||||
```
|
||||
GET /latest/dex/pairs/{chainId}/{pairAddress}
|
||||
Rate Limit: 300 req/min
|
||||
```
|
||||
|
||||
**Use Case**: Lấy thông tin chi tiết của một pair cụ thể
|
||||
|
||||
#### 4. Get Multiple Tokens
|
||||
```
|
||||
GET /tokens/v1/{chainId}/{tokenAddresses}
|
||||
Rate Limit: 300 req/min
|
||||
Max: 30 addresses (comma-separated)
|
||||
```
|
||||
|
||||
**Use Case**: Batch query nhiều tokens cùng lúc
|
||||
|
||||
## 🏗️ SurfSense Architecture Analysis
|
||||
|
||||
### Pattern Đã Xác Định
|
||||
|
||||
#### 1. Connector Class Pattern
|
||||
**File**: `app/connectors/{name}_connector.py`
|
||||
|
||||
**Responsibilities**:
|
||||
- Initialize với API credentials (nếu cần)
|
||||
- Methods để fetch data từ external API
|
||||
- Methods để format data sang markdown
|
||||
- Error handling cho API calls
|
||||
|
||||
**Example từ LumaConnector**:
|
||||
```python
|
||||
class LumaConnector:
|
||||
def __init__(self, api_key: str | None = None):
|
||||
self.api_key = api_key
|
||||
self.base_url = "https://api.lu.ma"
|
||||
|
||||
def make_request(self, endpoint: str, params: dict | None = None):
|
||||
# Handle API calls with error handling
|
||||
|
||||
def get_events_by_date_range(self, start_date: str, end_date: str):
|
||||
# Fetch data from API
|
||||
|
||||
def format_event_to_markdown(self, event: dict) -> str:
|
||||
# Convert to markdown for indexing
|
||||
```
|
||||
|
||||
#### 2. Indexer Pattern
|
||||
**File**: `app/tasks/connector_indexers/{name}_indexer.py`
|
||||
|
||||
**Responsibilities**:
|
||||
- Async function `index_{name}()`
|
||||
- Get connector từ database
|
||||
- Extract config (API keys, etc.)
|
||||
- Initialize connector class
|
||||
- Fetch data từ API
|
||||
- Loop qua items:
|
||||
- Generate `unique_identifier_hash` (để track duplicates)
|
||||
- Generate `content_hash` (để detect content changes)
|
||||
- Check existing documents
|
||||
- Create/Update `Document` objects với:
|
||||
- `chunks` (text chunks cho vector search)
|
||||
- `embedding` (vector embedding)
|
||||
- `metadata` (structured data)
|
||||
- Batch commit to database
|
||||
- Update `last_indexed_at` timestamp
|
||||
|
||||
**Key Functions Used**:
|
||||
```python
|
||||
from app.utils.document_converters import (
|
||||
create_document_chunks,
|
||||
generate_content_hash,
|
||||
generate_document_summary,
|
||||
generate_unique_identifier_hash,
|
||||
)
|
||||
```
|
||||
|
||||
#### 3. Routes Pattern
|
||||
**File**: `app/routes/{name}_add_connector_route.py`
|
||||
|
||||
**Endpoints**:
|
||||
- `POST /connectors/{name}/add` - Add/Update connector
|
||||
- `DELETE /connectors/{name}` - Delete connector
|
||||
- `GET /connectors/{name}/test` - Test connection
|
||||
|
||||
**Example từ luma_add_connector_route.py**:
|
||||
```python
|
||||
@router.post("/connectors/luma/add")
|
||||
async def add_luma_connector(
|
||||
request: AddLumaConnectorRequest,
|
||||
user: User = Depends(current_active_user),
|
||||
session: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
# Check existing connector
|
||||
# Create or update SearchSourceConnector
|
||||
# Store config in connector.config JSON field
|
||||
```
|
||||
|
||||
#### 4. Database Schema
|
||||
**File**: `app/db.py`
|
||||
|
||||
**SearchSourceConnectorType Enum**:
|
||||
```python
|
||||
class SearchSourceConnectorType(str, Enum):
|
||||
LUMA_CONNECTOR = "LUMA_CONNECTOR"
|
||||
SLACK_CONNECTOR = "SLACK_CONNECTOR"
|
||||
# ... thêm DEXSCREENER_CONNECTOR
|
||||
```
|
||||
|
||||
**SearchSourceConnector Model**:
|
||||
```python
|
||||
class SearchSourceConnector(Base):
|
||||
id: int
|
||||
name: str
|
||||
connector_type: SearchSourceConnectorType
|
||||
config: dict # JSON field để store API keys, settings
|
||||
search_space_id: int
|
||||
user_id: UUID
|
||||
is_indexable: bool
|
||||
last_indexed_at: datetime
|
||||
```
|
||||
|
||||
#### 5. Celery Tasks
|
||||
**File**: `app/tasks/celery_tasks/connector_tasks.py`
|
||||
|
||||
**Pattern**:
|
||||
```python
|
||||
@celery_app.task(name="index_luma_events", bind=True)
|
||||
def index_luma_events_task(
|
||||
self,
|
||||
connector_id: int,
|
||||
search_space_id: int,
|
||||
user_id: str,
|
||||
start_date: str | None = None,
|
||||
end_date: str | None = None,
|
||||
):
|
||||
# Wrapper cho async indexer function
|
||||
return asyncio.run(_index_luma_events(...))
|
||||
```
|
||||
|
||||
#### 6. Periodic Scheduler
|
||||
**File**: `app/utils/periodic_scheduler.py`
|
||||
|
||||
**Mapping**:
|
||||
```python
|
||||
CONNECTOR_TYPE_TO_TASK_NAME = {
|
||||
SearchSourceConnectorType.LUMA_CONNECTOR: "index_luma_events",
|
||||
# ... thêm mapping cho DexScreener
|
||||
}
|
||||
|
||||
CONNECTOR_TYPE_TO_TASK = {
|
||||
SearchSourceConnectorType.LUMA_CONNECTOR: index_luma_events_task,
|
||||
# ... thêm task cho DexScreener
|
||||
}
|
||||
```
|
||||
|
||||
## 📝 Implementation Plan
|
||||
|
||||
### Phase 1: Core Components
|
||||
|
||||
#### 1.1. Database Schema Update
|
||||
|
||||
**File**: `app/db.py`
|
||||
|
||||
**Changes**:
|
||||
```python
|
||||
class SearchSourceConnectorType(str, Enum):
|
||||
# ... existing types
|
||||
DEXSCREENER_CONNECTOR = "DEXSCREENER_CONNECTOR"
|
||||
|
||||
class DocumentType(str, Enum):
|
||||
# ... existing types
|
||||
DEXSCREENER_CONNECTOR = "DEXSCREENER_CONNECTOR"
|
||||
```
|
||||
|
||||
#### 1.2. Connector Class
|
||||
|
||||
**File**: `app/connectors/dexscreener_connector.py`
|
||||
|
||||
Xem full implementation trong artifacts.
|
||||
|
||||
#### 1.3. Indexer
|
||||
|
||||
**File**: `app/tasks/connector_indexers/dexscreener_indexer.py`
|
||||
|
||||
Xem full implementation trong artifacts.
|
||||
|
||||
### Phase 2: API Routes & Integration
|
||||
|
||||
#### 2.1. Routes
|
||||
|
||||
**File**: `app/routes/dexscreener_add_connector_route.py`
|
||||
|
||||
Xem full implementation trong artifacts.
|
||||
|
||||
#### 2.2. Celery Task
|
||||
|
||||
**File**: `app/tasks/celery_tasks/connector_tasks.py`
|
||||
|
||||
**Add to existing file**:
|
||||
```python
|
||||
# Add import
|
||||
from app.tasks.connector_indexers import index_dexscreener_pairs
|
||||
|
||||
# Add task
|
||||
@celery_app.task(name="index_dexscreener_pairs", bind=True)
|
||||
def index_dexscreener_pairs_task(
|
||||
self,
|
||||
connector_id: int,
|
||||
search_space_id: int,
|
||||
user_id: str,
|
||||
):
|
||||
"""Celery task for indexing DexScreener pairs."""
|
||||
try:
|
||||
return asyncio.run(
|
||||
_index_dexscreener_pairs(
|
||||
connector_id=connector_id,
|
||||
search_space_id=search_space_id,
|
||||
user_id=user_id,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"DexScreener indexing task failed: {str(e)}", exc_info=True)
|
||||
raise
|
||||
|
||||
|
||||
async def _index_dexscreener_pairs(
|
||||
connector_id: int,
|
||||
search_space_id: int,
|
||||
user_id: str,
|
||||
):
|
||||
"""Async wrapper for DexScreener indexing."""
|
||||
async with get_async_session_context() as session:
|
||||
return await index_dexscreener_pairs(
|
||||
session=session,
|
||||
connector_id=connector_id,
|
||||
search_space_id=search_space_id,
|
||||
user_id=user_id,
|
||||
)
|
||||
```
|
||||
|
||||
#### 2.3. Periodic Scheduler
|
||||
|
||||
**File**: `app/utils/periodic_scheduler.py`
|
||||
|
||||
**Add to existing mappings**:
|
||||
```python
|
||||
# Add to CONNECTOR_TYPE_TO_TASK_NAME
|
||||
CONNECTOR_TYPE_TO_TASK_NAME = {
|
||||
# ... existing mappings
|
||||
SearchSourceConnectorType.DEXSCREENER_CONNECTOR: "index_dexscreener_pairs",
|
||||
}
|
||||
|
||||
# Add import
|
||||
from app.tasks.celery_tasks.connector_tasks import index_dexscreener_pairs_task
|
||||
|
||||
# Add to CONNECTOR_TYPE_TO_TASK
|
||||
CONNECTOR_TYPE_TO_TASK = {
|
||||
# ... existing mappings
|
||||
SearchSourceConnectorType.DEXSCREENER_CONNECTOR: index_dexscreener_pairs_task,
|
||||
}
|
||||
```
|
||||
|
||||
#### 2.4. Routes Registration
|
||||
|
||||
**File**: `app/routes/__init__.py`
|
||||
|
||||
**Add**:
|
||||
```python
|
||||
# Add import
|
||||
from app.routes.dexscreener_add_connector_route import router as dexscreener_add_connector_router
|
||||
|
||||
# Add to router includes (after other connector routes)
|
||||
router.include_router(dexscreener_add_connector_router)
|
||||
```
|
||||
|
||||
#### 2.5. Indexer Export
|
||||
|
||||
**File**: `app/tasks/connector_indexers/__init__.py`
|
||||
|
||||
**Add**:
|
||||
```python
|
||||
# Add import
|
||||
from .dexscreener_indexer import index_dexscreener_pairs
|
||||
|
||||
# Add to __all__
|
||||
__all__ = [
|
||||
# ... existing exports
|
||||
"index_dexscreener_pairs",
|
||||
]
|
||||
```
|
||||
|
||||
## 🔄 Usage Flow
|
||||
|
||||
### 1. Add Connector via API
|
||||
|
||||
```bash
|
||||
curl -X POST "http://localhost:8000/api/connectors/dexscreener/add" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"space_id": 1,
|
||||
"tokens": [
|
||||
{
|
||||
"chain": "solana",
|
||||
"address": "So11111111111111111111111111111111111111112",
|
||||
"name": "Wrapped SOL"
|
||||
},
|
||||
{
|
||||
"chain": "ethereum",
|
||||
"address": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
|
||||
"name": "Wrapped ETH"
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### 2. Test Connection
|
||||
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/api/connectors/dexscreener/test?chain=solana&token_address=So11111111111111111111111111111111111111112" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
```
|
||||
|
||||
### 3. Trigger Manual Indexing
|
||||
|
||||
Indexing sẽ được trigger tự động qua:
|
||||
- **Periodic scheduler**: Mỗi 60 phút (configurable)
|
||||
- **Manual trigger**: Qua search_source_connectors_routes.py endpoint
|
||||
|
||||
### 4. Search Indexed Data
|
||||
|
||||
Data được index sẽ tự động available trong:
|
||||
- AI Chat với context từ DexScreener
|
||||
- Search results
|
||||
- Document retrieval
|
||||
|
||||
## ⚠️ Important Considerations
|
||||
|
||||
### Rate Limiting
|
||||
- DexScreener API: 300 requests/minute
|
||||
- Với 50 tokens tracked, mỗi lần index = 50 requests
|
||||
- Recommended indexing interval: **60 minutes**
|
||||
- Implement exponential backoff nếu hit rate limit
|
||||
|
||||
### Data Freshness
|
||||
- Crypto market data thay đổi nhanh
|
||||
- Consider shorter intervals (15-30 min) cho high-priority tokens
|
||||
- Implement priority queue cho important tokens
|
||||
|
||||
### Storage Optimization
|
||||
- Mỗi pair = 1 document với chunks
|
||||
- 50 tokens × 5 pairs average = 250 documents
|
||||
- Monitor storage usage và implement cleanup cho old data
|
||||
|
||||
### Error Handling
|
||||
- Network failures: Retry với exponential backoff
|
||||
- API errors: Log và skip, không block toàn bộ indexing
|
||||
- Invalid data: Validate trước khi index
|
||||
|
||||
## 🎯 Next Steps
|
||||
|
||||
1. **Implement Phase 1**: Core components (connector, indexer, DB schema)
|
||||
2. **Test locally**: Verify API calls và data formatting
|
||||
3. **Implement Phase 2**: Routes và integration
|
||||
4. **Test end-to-end**: Add connector → Index → Search
|
||||
5. **Deploy**: Monitor performance và adjust intervals
|
||||
6. **Optimize**: Based on usage patterns và feedback
|
||||
|
||||
## 📊 Success Metrics
|
||||
|
||||
- ✅ Connector successfully fetches data from DexScreener API
|
||||
- ✅ Data được format chính xác sang markdown
|
||||
- ✅ Documents được index với proper chunks và embeddings
|
||||
- ✅ Search results include DexScreener data
|
||||
- ✅ AI Chat có context từ crypto market data
|
||||
- ✅ Periodic indexing runs without errors
|
||||
- ✅ Rate limits được respect
|
||||
|
||||
---
|
||||
|
||||
**Note**: Implementation này dựa trên:
|
||||
- Official DexScreener API Documentation
|
||||
- Existing SurfSense connector patterns (Luma, Slack, etc.)
|
||||
- Best practices từ production connectors
|
||||
58
_bmad-stories/README.md
Normal file
58
_bmad-stories/README.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# SurfSense Stories
|
||||
|
||||
Danh sách các user stories cho dự án SurfSense.
|
||||
|
||||
## 📋 Active Stories
|
||||
|
||||
### Story 1.1: DexScreener Connector Integration
|
||||
- **Status**: Ready for Development
|
||||
- **Priority**: High
|
||||
- **Epic**: SurfSense Connectors Enhancement
|
||||
- **File**: [story-1.1-dexscreener-connector.md](./story-1.1-dexscreener-connector.md)
|
||||
- **Implementation Plan**: [dexscreener-connector-implementation-plan.md](../_bmad-output/dexscreener-connector-implementation-plan.md)
|
||||
|
||||
**Summary**: Integrate DexScreener API connector để users có thể track và search crypto trading pairs data.
|
||||
|
||||
**Key Features**:
|
||||
- Configure tracked tokens across multiple chains
|
||||
- Auto-index trading pair data (prices, volume, liquidity)
|
||||
- Search indexed crypto market data
|
||||
- AI chat with DexScreener context
|
||||
|
||||
**Acceptance Criteria**: 6 categories, 20+ checkboxes
|
||||
- ✅ Connector Configuration
|
||||
- ✅ API Integration
|
||||
- ✅ Data Indexing
|
||||
- ✅ Periodic Indexing
|
||||
- ✅ Search Integration
|
||||
- ✅ AI Chat Integration
|
||||
|
||||
---
|
||||
|
||||
## 📁 Story Organization
|
||||
|
||||
```
|
||||
_bmad-stories/
|
||||
├── README.md # This file
|
||||
├── story-1.1-dexscreener-connector.md # Story 1.1
|
||||
└── [future stories...]
|
||||
```
|
||||
|
||||
## 🔗 Related Documentation
|
||||
|
||||
- [Connectors Architecture](../_bmad-output/connectors-explained.md)
|
||||
- [Custom Connector Guide](../_bmad-output/custom-connector-guide.md)
|
||||
- [User Guide](../_bmad-output/user-guide.md)
|
||||
|
||||
## 📊 Story Status Legend
|
||||
|
||||
- **Ready for Development**: Story đã được review và sẵn sàng implement
|
||||
- **In Progress**: Đang được develop
|
||||
- **In Review**: Code đã hoàn thành, đang review
|
||||
- **Testing**: Đang test
|
||||
- **Done**: Hoàn thành và deployed
|
||||
- **Blocked**: Bị block bởi dependencies hoặc issues
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-31
|
||||
BIN
_bmad-stories/assets/story-1.2/dexscreener-config-edit.png
Normal file
BIN
_bmad-stories/assets/story-1.2/dexscreener-config-edit.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 381 KiB |
BIN
_bmad-stories/assets/story-1.2/dexscreener-connect-form.png
Normal file
BIN
_bmad-stories/assets/story-1.2/dexscreener-connect-form.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 477 KiB |
BIN
_bmad-stories/assets/story-1.2/dexscreener-token-card.png
Normal file
BIN
_bmad-stories/assets/story-1.2/dexscreener-token-card.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 697 KiB |
286
_bmad-stories/assets/story-1.2/ui-mockups.md
Normal file
286
_bmad-stories/assets/story-1.2/ui-mockups.md
Normal file
|
|
@ -0,0 +1,286 @@
|
|||
# UI Mockups - Story 1.2: DexScreener Connector Frontend
|
||||
|
||||
> [!NOTE]
|
||||
> **Purpose**: Visual design specifications for DexScreener connector UI components
|
||||
>
|
||||
> **Design System**: ShadCN UI with dark theme
|
||||
>
|
||||
> **Reference**: Based on existing Luma connector patterns
|
||||
|
||||
---
|
||||
|
||||
## 1. Connect Form - Main Interface
|
||||
|
||||
|
||||
|
||||
### 📋 Component Breakdown
|
||||
|
||||
#### Header Section
|
||||
- **Info Alert**: Light blue background với icon
|
||||
- Message: "No API Key Required - DexScreener API is free and public"
|
||||
- Purpose: Inform users về public API nature
|
||||
|
||||
#### Connector Name Section
|
||||
- **Label**: "Connector Name"
|
||||
- **Input Field**: Text input với placeholder "My DexScreener Connector"
|
||||
- **Help Text**: "A friendly name to identify this connector"
|
||||
- **Validation**: Required field
|
||||
|
||||
#### Token Management Section
|
||||
- **Section Title**: "Tracked Tokens (2/50)"
|
||||
- Shows current count / maximum limit
|
||||
- **Token Cards**: Stacked vertically
|
||||
- Each card contains:
|
||||
- Chain dropdown (Ethereum, BSC, Polygon, etc.)
|
||||
- Token address input (0x... format)
|
||||
- Optional name field
|
||||
- Delete button (red X icon)
|
||||
- **Add Button**: Blue "+ Add Token" button
|
||||
- Disabled when limit reached (50 tokens)
|
||||
|
||||
#### Indexing Configuration
|
||||
- **Date Range**: Two date pickers (Start Date, End Date)
|
||||
- **Periodic Sync Toggle**: Switch component (enabled/disabled)
|
||||
- **Sync Frequency**: Dropdown (Daily, Weekly, Monthly)
|
||||
|
||||
#### Benefits Section
|
||||
- **Title**: "What you get with DexScreener integration:"
|
||||
- **List Items**:
|
||||
- Access real-time crypto trading data across multiple chains
|
||||
- Track live token prices and market capitalization
|
||||
- Analyze liquidity pairs and trading volume trends
|
||||
- Monitor transaction history and large trades
|
||||
- Stay updated with new token listings and pair information
|
||||
|
||||
#### Action Button
|
||||
- **Connect Button**: Blue, bottom-right alignment
|
||||
- **States**: Default, Hover, Loading, Disabled
|
||||
|
||||
---
|
||||
|
||||
## 2. Token Card Component - Detailed View
|
||||
|
||||
|
||||
|
||||
### 🎨 Design Specifications
|
||||
|
||||
#### Layout Structure
|
||||
- **Container**: Horizontal layout với border và rounded corners
|
||||
- **Left Section** (70% width):
|
||||
- Chain selector với logo icon
|
||||
- Token address input (monospace font)
|
||||
- Optional name input
|
||||
- **Right Section** (30% width):
|
||||
- Delete button (circular, red X)
|
||||
|
||||
#### Validation States
|
||||
|
||||
**Valid Address**:
|
||||
- Green checkmark icon next to address field
|
||||
- Success message: "✓ Valid ERC-20 address"
|
||||
- Green border on input field
|
||||
|
||||
**Invalid Address**:
|
||||
- Red X icon next to address field
|
||||
- Error message: "✗ Invalid address format"
|
||||
- Red border on input field
|
||||
|
||||
**Empty State**:
|
||||
- Neutral gray border
|
||||
- Placeholder text: "0x..."
|
||||
|
||||
#### Chain Options
|
||||
Support các chains:
|
||||
- Ethereum (ETH)
|
||||
- Binance Smart Chain (BSC)
|
||||
- Polygon (MATIC)
|
||||
- Arbitrum
|
||||
- Optimism
|
||||
- Avalanche
|
||||
- Fantom
|
||||
- Base
|
||||
|
||||
#### Interaction States
|
||||
- **Hover**: Subtle background color change
|
||||
- **Focus**: Blue border on active input
|
||||
- **Delete Hover**: Red background on delete button
|
||||
|
||||
---
|
||||
|
||||
## 3. Config Edit Interface
|
||||
|
||||
|
||||
|
||||
### ⚙️ Configuration Management
|
||||
|
||||
#### Header
|
||||
- **Title**: "Edit Connector"
|
||||
- **Status Badge**: Green "Active" badge
|
||||
- Shows connector status (Active, Paused, Error)
|
||||
|
||||
#### Configuration Section
|
||||
- **Connector Name**: Editable text input
|
||||
- **Edit Icon**: Pencil icon for inline editing
|
||||
|
||||
#### Current Tokens Display
|
||||
- **Section Title**: "Tracked Tokens (3)"
|
||||
- **Token List**: Compact card view
|
||||
- Chain icon + name
|
||||
- Shortened address (0x1f98...F984)
|
||||
- Token name
|
||||
- Action icons (Edit, Delete)
|
||||
|
||||
#### Add New Token
|
||||
- **Expandable Section**: "+ Add New Token" button
|
||||
- **Expanded State**: Shows full token form
|
||||
- Same fields as connect form
|
||||
- Inline validation
|
||||
|
||||
#### Action Buttons
|
||||
- **Cancel**: Gray button, left-aligned
|
||||
- Discards changes
|
||||
- Returns to connector list
|
||||
- **Save Changes**: Blue button, right-aligned
|
||||
- Validates all fields
|
||||
- Updates connector config
|
||||
- Shows success toast
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Implementation Notes
|
||||
|
||||
### Responsive Design
|
||||
- **Desktop**: Full width form với side-by-side layouts
|
||||
- **Tablet**: Stacked sections, maintained spacing
|
||||
- **Mobile**: Single column, full-width inputs
|
||||
|
||||
### Accessibility
|
||||
- **ARIA Labels**: All form fields có proper labels
|
||||
- **Keyboard Navigation**: Tab order logical
|
||||
- **Screen Reader**: Descriptive text cho all actions
|
||||
- **Focus Indicators**: Visible focus states
|
||||
|
||||
### Validation Rules
|
||||
|
||||
**Connector Name**:
|
||||
- Required field
|
||||
- Min length: 3 characters
|
||||
- Max length: 50 characters
|
||||
|
||||
**Token Address**:
|
||||
- Required field
|
||||
- Must start with "0x"
|
||||
- Must be 42 characters (0x + 40 hex)
|
||||
- Hex characters only (0-9, a-f, A-F)
|
||||
|
||||
**Token Limit**:
|
||||
- Maximum 50 tokens per connector
|
||||
- Minimum 1 token required
|
||||
|
||||
### Error Handling
|
||||
|
||||
**Network Errors**:
|
||||
- Toast notification với retry option
|
||||
- Form remains editable
|
||||
|
||||
**Validation Errors**:
|
||||
- Inline error messages
|
||||
- Red border on invalid fields
|
||||
- Prevent form submission
|
||||
|
||||
**Success States**:
|
||||
- Green toast notification
|
||||
- Redirect to connector list
|
||||
- Update connector status
|
||||
|
||||
---
|
||||
|
||||
## 📱 Mobile Considerations
|
||||
|
||||
### Touch Targets
|
||||
- Minimum 44x44px for all interactive elements
|
||||
- Increased spacing between tokens
|
||||
- Larger delete buttons
|
||||
|
||||
### Form Layout
|
||||
- Single column layout
|
||||
- Full-width inputs
|
||||
- Stacked date pickers
|
||||
- Bottom sheet for chain selector
|
||||
|
||||
### Performance
|
||||
- Lazy load token list (virtual scrolling)
|
||||
- Debounced address validation
|
||||
- Optimistic UI updates
|
||||
|
||||
---
|
||||
|
||||
## 🔄 State Management
|
||||
|
||||
### Form States
|
||||
1. **Initial**: Empty form với default values
|
||||
2. **Editing**: User input in progress
|
||||
3. **Validating**: Checking address formats
|
||||
4. **Submitting**: API call in progress
|
||||
5. **Success**: Connector created/updated
|
||||
6. **Error**: Validation or API error
|
||||
|
||||
### Token List States
|
||||
1. **Empty**: No tokens added
|
||||
2. **Adding**: New token form visible
|
||||
3. **Editing**: Existing token being modified
|
||||
4. **Deleting**: Confirmation dialog shown
|
||||
5. **Maximum**: 50 tokens reached
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Design Tokens
|
||||
|
||||
### Colors
|
||||
- **Primary**: Blue (#3B82F6)
|
||||
- **Success**: Green (#10B981)
|
||||
- **Error**: Red (#EF4444)
|
||||
- **Background**: Slate (#1E293B)
|
||||
- **Border**: Slate (#334155)
|
||||
|
||||
### Typography
|
||||
- **Headings**: Inter, 600 weight
|
||||
- **Body**: Inter, 400 weight
|
||||
- **Monospace**: JetBrains Mono (addresses)
|
||||
|
||||
### Spacing
|
||||
- **Section Gap**: 24px
|
||||
- **Input Gap**: 16px
|
||||
- **Card Padding**: 16px
|
||||
- **Button Padding**: 12px 24px
|
||||
|
||||
---
|
||||
|
||||
## ✅ Implementation Checklist
|
||||
|
||||
- [ ] Create `dexscreener-connect-form.tsx` component
|
||||
- [ ] Implement token card component với validation
|
||||
- [ ] Create `dexscreener-config.tsx` edit interface
|
||||
- [ ] Add chain selector dropdown với icons
|
||||
- [ ] Implement address validation logic
|
||||
- [ ] Add form state management (React Hook Form)
|
||||
- [ ] Create Zod validation schema
|
||||
- [ ] Implement error handling và toast notifications
|
||||
- [ ] Add responsive breakpoints
|
||||
- [ ] Test accessibility compliance
|
||||
- [ ] Add loading states
|
||||
- [ ] Implement optimistic UI updates
|
||||
|
||||
---
|
||||
|
||||
## 📚 Reference Components
|
||||
|
||||
### Existing Patterns
|
||||
- [luma-connect-form.tsx](file:///Users/mac_1/Documents/GitHub/SurfSense/surfsense_web/components/assistant-ui/connector-popup/connect-forms/components/luma-connect-form.tsx) - Form structure
|
||||
- [luma-config.tsx](file:///Users/mac_1/Documents/GitHub/SurfSense/surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/luma-config.tsx) - Config pattern
|
||||
- [connector-benefits.ts](file:///Users/mac_1/Documents/GitHub/SurfSense/surfsense_web/components/assistant-ui/connector-popup/connect-forms/connector-benefits.ts) - Benefits system
|
||||
|
||||
### Design System
|
||||
- ShadCN UI components
|
||||
- Tailwind CSS utilities
|
||||
- Radix UI primitives
|
||||
403
_bmad-stories/story-1.1-dexscreener-connector.md
Normal file
403
_bmad-stories/story-1.1-dexscreener-connector.md
Normal file
|
|
@ -0,0 +1,403 @@
|
|||
# Story 1.1: DexScreener Connector Integration
|
||||
|
||||
## 📋 Story Overview
|
||||
|
||||
**Story ID**: 1.1
|
||||
**Story Title**: DexScreener Connector Integration
|
||||
**Epic**: SurfSense Connectors Enhancement
|
||||
**Priority**: High
|
||||
**Status**: Ready for Development
|
||||
**Created**: 2026-01-31
|
||||
|
||||
## 🎯 User Story
|
||||
|
||||
**As a** SurfSense user tracking cryptocurrency markets
|
||||
**I want** to connect my DexScreener data to SurfSense
|
||||
**So that** I can search and chat with AI about my tracked trading pairs and token data
|
||||
|
||||
## 📝 Description
|
||||
|
||||
Implement a custom connector for DexScreener API that allows users to:
|
||||
1. Configure tracked tokens across multiple blockchain networks
|
||||
2. Automatically index trading pair data (prices, volume, liquidity, etc.)
|
||||
3. Search and retrieve indexed crypto market data
|
||||
4. Use AI chat with context from DexScreener trading pairs
|
||||
|
||||
This connector will integrate with SurfSense's existing connector architecture, following the established patterns from Luma, Slack, and other connectors.
|
||||
|
||||
## ✅ Acceptance Criteria
|
||||
|
||||
### AC1: Connector Configuration
|
||||
- [ ] User can add DexScreener connector via API endpoint
|
||||
- [ ] User can configure multiple tokens to track (up to 50)
|
||||
- [ ] Each token config includes: chain ID, token address, optional name
|
||||
- [ ] User can update connector configuration
|
||||
- [ ] User can delete connector
|
||||
- [ ] Configuration is persisted in database
|
||||
|
||||
### AC2: API Integration
|
||||
- [ ] Connector successfully calls DexScreener API endpoints
|
||||
- [ ] Handles rate limits (300 req/min) appropriately
|
||||
- [ ] Implements retry logic with exponential backoff
|
||||
- [ ] Validates API responses
|
||||
- [ ] Handles API errors gracefully (network failures, invalid data, etc.)
|
||||
|
||||
### AC3: Data Indexing
|
||||
- [ ] Fetches trading pairs for all configured tokens
|
||||
- [ ] Converts pair data to markdown format with all key metrics:
|
||||
- Token information (names, symbols, addresses)
|
||||
- Price data (USD, native, 24h changes)
|
||||
- Volume metrics (24h, 6h, 1h)
|
||||
- Liquidity information
|
||||
- Market cap and FDV
|
||||
- Transaction counts
|
||||
- [ ] Generates unique identifier hash for each pair
|
||||
- [ ] Generates content hash to detect changes
|
||||
- [ ] Creates document chunks for vector search
|
||||
- [ ] Generates embeddings using configured LLM
|
||||
- [ ] Stores documents in database with proper metadata
|
||||
- [ ] Updates existing documents when content changes
|
||||
- [ ] Skips unchanged documents
|
||||
|
||||
### AC4: Periodic Indexing
|
||||
- [ ] Indexing task is registered with Celery
|
||||
- [ ] Periodic scheduler triggers indexing (default: 60 min interval)
|
||||
- [ ] Manual indexing can be triggered via API
|
||||
- [ ] Last indexed timestamp is updated after successful indexing
|
||||
- [ ] Indexing errors are logged properly
|
||||
- [ ] Failed indexing doesn't block future attempts
|
||||
|
||||
### AC5: Search Integration
|
||||
- [ ] Indexed DexScreener data appears in search results
|
||||
- [ ] Documents are searchable by:
|
||||
- Token names and symbols
|
||||
- Pair addresses
|
||||
- Chain IDs
|
||||
- DEX names
|
||||
- Price ranges
|
||||
- Volume metrics
|
||||
- [ ] Search results include relevant metadata
|
||||
- [ ] Vector search returns semantically similar pairs
|
||||
|
||||
### AC6: AI Chat Integration
|
||||
- [ ] AI chat can access DexScreener data as context
|
||||
- [ ] Chat responses include relevant trading pair information
|
||||
- [ ] Citations link to DexScreener URLs
|
||||
- [ ] Metadata is properly formatted in chat responses
|
||||
|
||||
## 🏗️ Technical Implementation
|
||||
|
||||
### Database Schema Changes
|
||||
|
||||
**File**: `app/db.py`
|
||||
|
||||
```python
|
||||
class SearchSourceConnectorType(str, Enum):
|
||||
# ... existing types
|
||||
DEXSCREENER_CONNECTOR = "DEXSCREENER_CONNECTOR"
|
||||
|
||||
class DocumentType(str, Enum):
|
||||
# ... existing types
|
||||
DEXSCREENER_CONNECTOR = "DEXSCREENER_CONNECTOR"
|
||||
```
|
||||
|
||||
### Components to Implement
|
||||
|
||||
#### 1. Connector Class
|
||||
**File**: `app/connectors/dexscreener_connector.py`
|
||||
|
||||
**Methods**:
|
||||
- `__init__()` - Initialize connector (no API key needed)
|
||||
- `make_request(endpoint, params)` - Generic API request handler
|
||||
- `search_pairs(query)` - Search for trading pairs
|
||||
- `get_token_pairs(chain_id, token_address)` - Get all pairs for a token
|
||||
- `get_pair_by_address(chain_id, pair_address)` - Get specific pair details
|
||||
- `get_multiple_tokens(chain_id, token_addresses)` - Batch query tokens
|
||||
- `format_pair_to_markdown(pair)` - Convert pair data to markdown
|
||||
|
||||
#### 2. Indexer
|
||||
**File**: `app/tasks/connector_indexers/dexscreener_indexer.py`
|
||||
|
||||
**Function**: `async def index_dexscreener_pairs(session, connector_id, search_space_id, user_id, start_date=None, end_date=None, update_last_indexed=True)`
|
||||
|
||||
**Required Imports**:
|
||||
```python
|
||||
from .base import (
|
||||
check_document_by_unique_identifier,
|
||||
check_duplicate_document_by_hash,
|
||||
get_connector_by_id,
|
||||
get_current_timestamp,
|
||||
logger,
|
||||
update_connector_last_indexed,
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
**Logic**:
|
||||
1. Get connector from database
|
||||
2. Extract token configuration from `connector.config.get("tokens")`
|
||||
3. Initialize DexScreener connector
|
||||
4. For each tracked token:
|
||||
- Fetch all trading pairs
|
||||
- For each pair:
|
||||
- Format to markdown
|
||||
- Generate hashes
|
||||
- Check if document exists
|
||||
- Create or update document
|
||||
- Batch commit every 10 documents
|
||||
5. Update last_indexed_at timestamp
|
||||
6. Log success/failure
|
||||
|
||||
|
||||
#### 3. API Routes
|
||||
**File**: `app/routes/dexscreener_add_connector_route.py`
|
||||
|
||||
**Endpoints**:
|
||||
- `POST /connectors/dexscreener/add` - Add/update connector
|
||||
- `DELETE /connectors/dexscreener` - Delete connector
|
||||
- `GET /connectors/dexscreener/test` - Test API connection
|
||||
|
||||
#### 4. Celery Task
|
||||
**File**: `app/tasks/celery_tasks/connector_tasks.py`
|
||||
|
||||
**Task**: `index_dexscreener_pairs_task(connector_id, search_space_id, user_id, start_date, end_date)`
|
||||
|
||||
**Note**: Requires both the Celery task wrapper and async helper function:
|
||||
- `@celery_app.task(bind=True)` decorator on `index_dexscreener_pairs_task`
|
||||
- `async def _index_dexscreener_pairs(...)` helper that creates session and calls indexer
|
||||
|
||||
#### 5. Scheduler Integration
|
||||
**File**: `app/utils/periodic_scheduler.py`
|
||||
|
||||
**Mappings**:
|
||||
Add to `CONNECTOR_TASK_MAP` dictionary:
|
||||
```python
|
||||
SearchSourceConnectorType.DEXSCREENER_CONNECTOR: "index_dexscreener_pairs"
|
||||
```
|
||||
|
||||
#### 6. Routes Registration
|
||||
**File**: `app/routes/__init__.py`
|
||||
|
||||
Import and include `dexscreener_add_connector_router`
|
||||
|
||||
#### 7. Indexer Export
|
||||
**File**: `app/tasks/connector_indexers/__init__.py`
|
||||
|
||||
Export `index_dexscreener_pairs`
|
||||
|
||||
## 🔗 Dependencies
|
||||
|
||||
### External APIs
|
||||
- **DexScreener API**: `https://api.dexscreener.com`
|
||||
- No authentication required
|
||||
- Rate limit: 300 requests/minute
|
||||
- Free tier available
|
||||
|
||||
### Internal Dependencies
|
||||
- `httpx` - HTTP client for API requests
|
||||
- `app.utils.document_converters` - Document processing utilities
|
||||
- `app.services.llm_service` - LLM for embeddings and summaries
|
||||
- `app.services.task_logging_service` - Task logging
|
||||
- SQLAlchemy models and sessions
|
||||
- Celery for background tasks
|
||||
|
||||
## 📊 Data Model
|
||||
|
||||
### Connector Config Schema
|
||||
```json
|
||||
{
|
||||
"tokens": [
|
||||
{
|
||||
"chain": "solana",
|
||||
"address": "So11111111111111111111111111111111111111112",
|
||||
"name": "Wrapped SOL"
|
||||
},
|
||||
{
|
||||
"chain": "ethereum",
|
||||
"address": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
|
||||
"name": "Wrapped ETH"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Document Metadata Schema
|
||||
```json
|
||||
{
|
||||
"pair_address": "string",
|
||||
"chain_id": "string",
|
||||
"dex_id": "string",
|
||||
"base_token_symbol": "string",
|
||||
"base_token_address": "string",
|
||||
"quote_token_symbol": "string",
|
||||
"quote_token_address": "string",
|
||||
"price_usd": "string",
|
||||
"price_native": "string",
|
||||
"volume_h24": "number",
|
||||
"liquidity_usd": "number",
|
||||
"market_cap": "number",
|
||||
"fdv": "number",
|
||||
"url": "string",
|
||||
"indexed_at": "string (ISO 8601)"
|
||||
}
|
||||
```
|
||||
|
||||
## 🧪 Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- [ ] Test `DexScreenerConnector` API methods
|
||||
- [ ] Test markdown formatting logic
|
||||
- [ ] Test error handling for API failures
|
||||
- [ ] Test rate limit handling
|
||||
- [ ] Test data validation
|
||||
|
||||
### Integration Tests
|
||||
- [ ] Test full indexing flow
|
||||
- [ ] Test connector CRUD operations
|
||||
- [ ] Test periodic scheduling
|
||||
- [ ] Test search integration
|
||||
- [ ] Test AI chat context retrieval
|
||||
|
||||
### Manual Testing
|
||||
1. Add connector with test tokens
|
||||
2. Verify API test endpoint works
|
||||
3. Trigger manual indexing
|
||||
4. Check database for created documents
|
||||
5. Search for indexed pairs
|
||||
6. Use AI chat to query trading data
|
||||
7. Verify periodic indexing runs
|
||||
8. Test connector update and deletion
|
||||
|
||||
## ⚠️ Edge Cases & Error Handling
|
||||
|
||||
### API Errors
|
||||
- **Rate Limit Exceeded (429)**: Implement exponential backoff, log warning
|
||||
- **Network Timeout**: Retry with timeout, skip token if persistent
|
||||
- **Invalid Response**: Log error, skip pair, continue indexing
|
||||
- **Token Not Found**: Log warning, skip token
|
||||
|
||||
### Data Validation
|
||||
- **Missing pair_address**: Skip pair, log warning
|
||||
- **Empty markdown content**: Skip pair, log warning
|
||||
- **Invalid chain_id**: Validate against known chains
|
||||
- **Malformed token config**: Reject at API level with clear error
|
||||
|
||||
### Database Errors
|
||||
- **Duplicate documents**: Update existing based on content hash
|
||||
- **Transaction failures**: Rollback, log error, retry
|
||||
- **Connection issues**: Retry with backoff
|
||||
|
||||
### Performance Considerations
|
||||
- **Large token lists**: Batch commits every 10 documents
|
||||
- **Slow API responses**: Set reasonable timeout (30s)
|
||||
- **Memory usage**: Process pairs iteratively, not all at once
|
||||
|
||||
## 📈 Success Metrics
|
||||
|
||||
### Functional Metrics
|
||||
- Connector successfully adds/updates/deletes
|
||||
- API test endpoint returns valid data
|
||||
- Indexing completes without errors
|
||||
- Documents searchable within 5 minutes of indexing
|
||||
- AI chat provides accurate trading pair information
|
||||
|
||||
### Performance Metrics
|
||||
- API response time < 5 seconds for test endpoint
|
||||
- Indexing time < 2 minutes for 50 tokens (avg 5 pairs each)
|
||||
- Search latency < 500ms
|
||||
- Rate limit compliance: 0 violations
|
||||
|
||||
### Quality Metrics
|
||||
- 0 critical bugs in production
|
||||
- < 1% indexing failure rate
|
||||
- 100% test coverage for core logic
|
||||
- All acceptance criteria met
|
||||
|
||||
## 🚀 Deployment Plan
|
||||
|
||||
### Pre-deployment
|
||||
1. Review and merge PR
|
||||
2. Run all tests in CI/CD
|
||||
3. Database migration (add enum values)
|
||||
4. Deploy to staging environment
|
||||
5. Smoke test on staging
|
||||
|
||||
### Deployment Steps
|
||||
1. Deploy backend changes
|
||||
2. Verify Celery workers pick up new task
|
||||
3. Verify periodic scheduler includes new connector
|
||||
4. Monitor logs for errors
|
||||
5. Test connector addition via API
|
||||
|
||||
### Post-deployment
|
||||
1. Monitor error logs for 24 hours
|
||||
2. Check indexing task success rate
|
||||
3. Verify search results quality
|
||||
4. Gather user feedback
|
||||
|
||||
### Rollback Plan
|
||||
If critical issues occur:
|
||||
1. Remove connector type from periodic scheduler
|
||||
2. Disable connector routes
|
||||
3. Revert database migration if needed
|
||||
4. Investigate and fix issues
|
||||
5. Redeploy with fixes
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### User Documentation
|
||||
- [ ] Add DexScreener to connector list in user guide
|
||||
- [ ] Document how to add DexScreener connector
|
||||
- [ ] Explain token configuration format
|
||||
- [ ] Provide example API requests
|
||||
- [ ] Document supported chains
|
||||
|
||||
### Developer Documentation
|
||||
- [ ] Add inline code comments
|
||||
- [ ] Document API endpoints in OpenAPI spec
|
||||
- [ ] Update connector architecture docs
|
||||
- [ ] Add troubleshooting guide
|
||||
- [ ] Document rate limit handling
|
||||
|
||||
## 🔐 Security Considerations
|
||||
|
||||
- **No API Keys**: DexScreener API is public, no sensitive data to store
|
||||
- **Input Validation**: Validate chain IDs and token addresses
|
||||
- **Rate Limiting**: Respect API rate limits to avoid IP bans
|
||||
- **Data Privacy**: No PII collected or stored
|
||||
- **Error Messages**: Don't expose internal system details in API responses
|
||||
|
||||
## 🎯 Definition of Done
|
||||
|
||||
- [ ] All acceptance criteria met
|
||||
- [ ] Code reviewed and approved
|
||||
- [ ] Unit tests written and passing
|
||||
- [ ] Integration tests written and passing
|
||||
- [ ] Manual testing completed
|
||||
- [ ] Documentation updated
|
||||
- [ ] Deployed to staging and tested
|
||||
- [ ] Deployed to production
|
||||
- [ ] Monitoring in place
|
||||
- [ ] User guide updated
|
||||
- [ ] Story marked as complete
|
||||
|
||||
## 📎 Related Documents
|
||||
|
||||
- [DexScreener API Documentation](https://docs.dexscreener.com/api/reference)
|
||||
- [Implementation Plan](./dexscreener-connector-implementation-plan.md)
|
||||
- [SurfSense Connector Architecture](../../../Documents/GitHub/SurfSense/_bmad-output/connectors-explained.md)
|
||||
- [Custom Connector Guide](../../../Documents/GitHub/SurfSense/_bmad-output/custom-connector-guide.md)
|
||||
|
||||
## 💬 Notes
|
||||
|
||||
- DexScreener API is free and doesn't require authentication
|
||||
- Rate limit of 300 req/min is generous for typical use cases
|
||||
- Consider implementing priority queue for high-value tokens in future
|
||||
- May want to add support for custom indexing intervals per token
|
||||
- Consider adding alerts for significant price changes in future enhancement
|
||||
|
||||
---
|
||||
|
||||
**Story Created By**: Antigravity AI
|
||||
**Date**: 2026-01-31
|
||||
**Last Updated**: 2026-01-31
|
||||
454
_bmad-stories/story-1.2-dexscreener-frontend.md
Normal file
454
_bmad-stories/story-1.2-dexscreener-frontend.md
Normal file
|
|
@ -0,0 +1,454 @@
|
|||
# Story 1.2: DexScreener Connector Frontend UI
|
||||
|
||||
## 📋 Story Overview
|
||||
|
||||
**Story ID**: 1.2
|
||||
**Story Title**: DexScreener Connector Frontend UI
|
||||
**Epic**: SurfSense Connectors Enhancement
|
||||
**Priority**: High
|
||||
**Status**: Ready for Development
|
||||
**Created**: 2026-01-31
|
||||
**Depends On**: Story 1.1 (Backend API)
|
||||
|
||||
## 🎯 User Story
|
||||
|
||||
**As a** SurfSense user tracking cryptocurrency markets
|
||||
**I want** an intuitive UI to configure my DexScreener connector
|
||||
**So that** I can easily add/manage tracked tokens and view connector status
|
||||
|
||||
## 📝 Description
|
||||
|
||||
Implement frontend UI components for the DexScreener connector that allows users to:
|
||||
1. Add new DexScreener connector with token configuration
|
||||
2. Manage tracked tokens (add, edit, remove)
|
||||
3. View connector status and indexed data
|
||||
4. Configure periodic sync settings
|
||||
5. Access connector documentation
|
||||
|
||||
This story implements the user-facing components following SurfSense's established connector UI patterns.
|
||||
|
||||
## ✅ Acceptance Criteria
|
||||
|
||||
### AC1: Connect Form Component
|
||||
- [ ] User can access DexScreener connector from connector popup
|
||||
- [ ] Form includes connector name field (min 3 characters)
|
||||
- [ ] User can add multiple tokens (up to 50)
|
||||
- [ ] Each token has: chain selector, token address input, optional name
|
||||
- [ ] Form validates token addresses (40-character hex)
|
||||
- [ ] User can remove tokens from list
|
||||
- [ ] Date range selector for initial indexing
|
||||
- [ ] Periodic sync toggle with frequency selector
|
||||
- [ ] "What you get" benefits section displayed
|
||||
- [ ] Form submits to backend API endpoint
|
||||
|
||||
### AC2: Token Management UI
|
||||
- [ ] Dynamic token list with add/remove buttons
|
||||
- [ ] Chain selector dropdown with popular chains:
|
||||
- Ethereum
|
||||
- BSC (Binance Smart Chain)
|
||||
- Polygon
|
||||
- Arbitrum
|
||||
- Optimism
|
||||
- Base
|
||||
- Avalanche
|
||||
- Solana
|
||||
- [ ] Token address input with validation
|
||||
- [ ] Optional token name/label field
|
||||
- [ ] Visual feedback for validation errors
|
||||
- [ ] Responsive design for mobile/desktop
|
||||
|
||||
### AC3: Connector Config Component
|
||||
- [ ] Edit mode for existing connector
|
||||
- [ ] Update connector name
|
||||
- [ ] Add/remove tokens from tracked list
|
||||
- [ ] View current token configuration
|
||||
- [ ] Save changes button
|
||||
- [ ] Cancel/discard changes option
|
||||
|
||||
### AC4: Connector Benefits
|
||||
- [ ] Display benefits list in connect form
|
||||
- [ ] Benefits include:
|
||||
- "Real-time cryptocurrency trading pair data"
|
||||
- "Track prices, volume, and liquidity across multiple DEXs"
|
||||
- "Search and analyze token market data with AI"
|
||||
- "Monitor your crypto portfolio with automated updates"
|
||||
- "Access historical price and volume trends"
|
||||
|
||||
### AC5: Documentation
|
||||
- [ ] MDX documentation file created
|
||||
- [ ] Setup guide with screenshots
|
||||
- [ ] Token configuration instructions
|
||||
- [ ] Chain selection guide
|
||||
- [ ] Troubleshooting section
|
||||
- [ ] Link to DexScreener API docs
|
||||
|
||||
### AC6: Integration
|
||||
- [ ] Connector registered in connector registry
|
||||
- [ ] Icon/logo added to public assets
|
||||
- [ ] Connector appears in connector list
|
||||
- [ ] Form properly integrated with connector popup
|
||||
- [ ] Config component properly integrated
|
||||
|
||||
## 🏗️ Technical Implementation
|
||||
|
||||
### 1. Connect Form Component
|
||||
**File**: `surfsense_web/components/assistant-ui/connector-popup/connect-forms/components/dexscreener-connect-form.tsx`
|
||||
|
||||
**Component Structure**:
|
||||
```typescript
|
||||
export const DexScreenerConnectForm: FC<ConnectFormProps> = ({ onSubmit, isSubmitting }) => {
|
||||
// State management
|
||||
const [tokens, setTokens] = useState<TokenConfig[]>([]);
|
||||
const [startDate, setStartDate] = useState<Date | undefined>();
|
||||
const [endDate, setEndDate] = useState<Date | undefined>();
|
||||
const [periodicEnabled, setPeriodicEnabled] = useState(false);
|
||||
const [frequencyMinutes, setFrequencyMinutes] = useState("1440");
|
||||
|
||||
// Form validation schema
|
||||
const formSchema = z.object({
|
||||
name: z.string().min(3),
|
||||
tokens: z.array(tokenSchema).min(1).max(50),
|
||||
});
|
||||
|
||||
// Token management functions
|
||||
const addToken = () => { /* ... */ };
|
||||
const removeToken = (index: number) => { /* ... */ };
|
||||
const updateToken = (index: number, field: string, value: string) => { /* ... */ };
|
||||
|
||||
// Submit handler
|
||||
const handleSubmit = async (values) => {
|
||||
await onSubmit({
|
||||
name: values.name,
|
||||
connector_type: EnumConnectorName.DEXSCREENER_CONNECTOR,
|
||||
config: { tokens: values.tokens },
|
||||
// ... other fields
|
||||
});
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
**Token Config Schema**:
|
||||
```typescript
|
||||
interface TokenConfig {
|
||||
chain: string;
|
||||
address: string;
|
||||
name?: string;
|
||||
}
|
||||
|
||||
const tokenSchema = z.object({
|
||||
chain: z.string().min(1, "Chain is required"),
|
||||
address: z.string().regex(/^0x[a-fA-F0-9]{40}$/, "Invalid token address"),
|
||||
name: z.string().optional(),
|
||||
});
|
||||
```
|
||||
|
||||
**UI Sections**:
|
||||
1. **Alert**: Info about DexScreener API (no auth required)
|
||||
2. **Connector Name**: Text input with validation
|
||||
3. **Token List**: Dynamic list with add/remove
|
||||
- Chain selector (dropdown)
|
||||
- Token address input
|
||||
- Optional name field
|
||||
- Remove button
|
||||
4. **Add Token Button**: Add new token to list
|
||||
5. **Indexing Configuration**:
|
||||
- Date range selector
|
||||
- Periodic sync toggle
|
||||
- Frequency selector
|
||||
6. **Benefits Section**: Display connector benefits
|
||||
|
||||
### 2. Connector Config Component
|
||||
**File**: `surfsense_web/components/assistant-ui/connector-popup/connector-configs/components/dexscreener-config.tsx`
|
||||
|
||||
**Component Structure**:
|
||||
```typescript
|
||||
export const DexScreenerConfig: FC<ConnectorConfigProps> = ({
|
||||
connector,
|
||||
onConfigChange,
|
||||
onNameChange
|
||||
}) => {
|
||||
const [tokens, setTokens] = useState<TokenConfig[]>(
|
||||
connector.config?.tokens || []
|
||||
);
|
||||
const [name, setName] = useState(connector.name || "");
|
||||
|
||||
// Update handlers
|
||||
const handleTokensChange = (newTokens: TokenConfig[]) => {
|
||||
setTokens(newTokens);
|
||||
onConfigChange({ ...connector.config, tokens: newTokens });
|
||||
};
|
||||
|
||||
const handleNameChange = (newName: string) => {
|
||||
setName(newName);
|
||||
onNameChange?.(newName);
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
**UI Sections**:
|
||||
1. **Connector Name**: Editable text input
|
||||
2. **Token Configuration**:
|
||||
- List of current tokens
|
||||
- Add/remove token functionality
|
||||
- Edit token details
|
||||
|
||||
### 3. Connector Benefits
|
||||
**File**: `surfsense_web/components/assistant-ui/connector-popup/connect-forms/connector-benefits.ts`
|
||||
|
||||
**Add to benefits object**:
|
||||
```typescript
|
||||
DEXSCREENER_CONNECTOR: [
|
||||
"Real-time cryptocurrency trading pair data from multiple DEXs",
|
||||
"Track token prices, volume, and liquidity across chains",
|
||||
"Search and analyze market data with AI-powered insights",
|
||||
"Monitor your crypto portfolio with automated updates",
|
||||
"Access historical price trends and trading volumes",
|
||||
],
|
||||
```
|
||||
|
||||
### 4. Connector Registry
|
||||
**Files to Update**:
|
||||
- `surfsense_web/components/assistant-ui/connector-popup/connect-forms/index.tsx`
|
||||
- `surfsense_web/components/assistant-ui/connector-popup/connector-configs/index.tsx`
|
||||
|
||||
**Register Components**:
|
||||
```typescript
|
||||
// In connect-forms/index.tsx
|
||||
import { DexScreenerConnectForm } from "./components/dexscreener-connect-form";
|
||||
|
||||
// Add to component map
|
||||
case EnumConnectorName.DEXSCREENER_CONNECTOR:
|
||||
return <DexScreenerConnectForm {...props} />;
|
||||
|
||||
// In connector-configs/index.tsx
|
||||
import { DexScreenerConfig } from "./components/dexscreener-config";
|
||||
|
||||
// Add to component map
|
||||
case EnumConnectorName.DEXSCREENER_CONNECTOR:
|
||||
return <DexScreenerConfig {...props} />;
|
||||
```
|
||||
|
||||
### 5. Documentation
|
||||
**File**: `surfsense_web/content/docs/connectors/dexscreener.mdx`
|
||||
|
||||
**Structure**:
|
||||
```markdown
|
||||
---
|
||||
title: DexScreener
|
||||
description: Connect DexScreener trading pair data to SurfSense
|
||||
---
|
||||
|
||||
# DexScreener Integration Setup Guide
|
||||
|
||||
## How it works
|
||||
- Fetches real-time trading pair data
|
||||
- Tracks prices, volume, liquidity
|
||||
- No API key required
|
||||
|
||||
## Authorization
|
||||
No authentication needed - DexScreener API is public.
|
||||
|
||||
## Connecting to SurfSense
|
||||
1. Navigate to Connector Dashboard
|
||||
2. Select DexScreener Connector
|
||||
3. Configure tracked tokens:
|
||||
- Select blockchain network
|
||||
- Enter token contract address
|
||||
- Add optional label
|
||||
4. Configure sync settings
|
||||
5. Click Connect
|
||||
|
||||
## What Gets Indexed
|
||||
- Token pair information
|
||||
- Price data (USD, native)
|
||||
- Volume metrics (24h, 6h, 1h)
|
||||
- Liquidity information
|
||||
- DEX information
|
||||
```
|
||||
|
||||
### 6. Assets
|
||||
**Files to Add**:
|
||||
- `surfsense_web/public/connectors/dexscreener.svg` - Connector icon
|
||||
|
||||
**Icon Requirements**:
|
||||
- SVG format
|
||||
- 24x24px viewBox
|
||||
- Monochrome design
|
||||
- Matches SurfSense design system
|
||||
|
||||
### 7. Enum Registration
|
||||
**File**: `surfsense_web/contracts/enums/connector.ts`
|
||||
|
||||
**Add to EnumConnectorName**:
|
||||
```typescript
|
||||
export enum EnumConnectorName {
|
||||
// ... existing connectors
|
||||
DEXSCREENER_CONNECTOR = "DEXSCREENER_CONNECTOR",
|
||||
}
|
||||
```
|
||||
|
||||
## 🔗 Dependencies
|
||||
|
||||
### Internal Dependencies
|
||||
- Story 1.1 (Backend API) - **MUST BE COMPLETED FIRST**
|
||||
- `@/components/ui/*` - ShadCN UI components
|
||||
- `@/contracts/enums/connector` - Connector enums
|
||||
- React Hook Form - Form management
|
||||
- Zod - Validation schema
|
||||
|
||||
### External Dependencies
|
||||
- None (DexScreener API is public)
|
||||
|
||||
## 📊 Data Models
|
||||
|
||||
### Token Configuration
|
||||
```typescript
|
||||
interface TokenConfig {
|
||||
chain: string; // Blockchain network
|
||||
address: string; // Token contract address (0x...)
|
||||
name?: string; // Optional display name
|
||||
}
|
||||
```
|
||||
|
||||
### Form Submission
|
||||
```typescript
|
||||
interface DexScreenerConnectorSubmission {
|
||||
name: string;
|
||||
connector_type: "DEXSCREENER_CONNECTOR";
|
||||
config: {
|
||||
tokens: TokenConfig[];
|
||||
};
|
||||
is_indexable: true;
|
||||
periodic_indexing_enabled: boolean;
|
||||
indexing_frequency_minutes: number | null;
|
||||
startDate?: Date;
|
||||
endDate?: Date;
|
||||
}
|
||||
```
|
||||
|
||||
## 🧪 Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- [ ] Token validation logic
|
||||
- [ ] Add/remove token functionality
|
||||
- [ ] Form submission with valid data
|
||||
- [ ] Form validation with invalid data
|
||||
- [ ] Chain selector options
|
||||
- [ ] Token address format validation
|
||||
|
||||
### Integration Tests
|
||||
- [ ] Form submission to backend API
|
||||
- [ ] Config component updates connector
|
||||
- [ ] Benefits display correctly
|
||||
- [ ] Documentation renders properly
|
||||
|
||||
### Manual Testing
|
||||
- [ ] Add connector with single token
|
||||
- [ ] Add connector with multiple tokens (10+)
|
||||
- [ ] Edit existing connector
|
||||
- [ ] Remove tokens from config
|
||||
- [ ] Test all chain options
|
||||
- [ ] Verify responsive design
|
||||
- [ ] Test form validation errors
|
||||
|
||||
## 🎨 UI/UX Considerations
|
||||
|
||||
### Design Patterns
|
||||
- Follow Luma connector UI patterns
|
||||
- Use consistent spacing and typography
|
||||
- Responsive design for mobile/tablet/desktop
|
||||
- Clear validation error messages
|
||||
- Loading states during submission
|
||||
|
||||
### User Flow
|
||||
1. User opens connector popup
|
||||
2. Selects "DexScreener" from list
|
||||
3. Enters connector name
|
||||
4. Adds first token (chain + address)
|
||||
5. Optionally adds more tokens
|
||||
6. Configures sync settings
|
||||
7. Reviews benefits
|
||||
8. Clicks "Connect"
|
||||
9. Sees success message
|
||||
|
||||
### Error Handling
|
||||
- Invalid token address format
|
||||
- Duplicate tokens
|
||||
- Maximum tokens exceeded (50)
|
||||
- Network errors during submission
|
||||
- Backend validation errors
|
||||
|
||||
## 📈 Success Metrics
|
||||
|
||||
- [ ] User can add DexScreener connector in < 2 minutes
|
||||
- [ ] Form validation prevents invalid submissions
|
||||
- [ ] Token management is intuitive
|
||||
- [ ] Documentation is clear and helpful
|
||||
- [ ] UI is responsive on all devices
|
||||
|
||||
## 🚀 Deployment Plan
|
||||
|
||||
### Phase 1: Component Development
|
||||
1. Create connect form component
|
||||
2. Create config component
|
||||
3. Add connector benefits
|
||||
4. Register in connector registry
|
||||
|
||||
### Phase 2: Documentation & Assets
|
||||
1. Write MDX documentation
|
||||
2. Add connector icon
|
||||
3. Update connector list
|
||||
|
||||
### Phase 3: Testing & QA
|
||||
1. Unit tests
|
||||
2. Integration tests
|
||||
3. Manual testing
|
||||
4. Bug fixes
|
||||
|
||||
### Phase 4: Release
|
||||
1. Merge to main branch
|
||||
2. Deploy to staging
|
||||
3. Verify functionality
|
||||
4. Deploy to production
|
||||
|
||||
## 📚 Documentation Requirements
|
||||
|
||||
- [ ] Component documentation (JSDoc)
|
||||
- [ ] User guide (MDX)
|
||||
- [ ] Developer notes for future maintenance
|
||||
- [ ] Screenshot examples in docs
|
||||
|
||||
## 🔒 Security Considerations
|
||||
|
||||
- Token addresses validated on frontend
|
||||
- No sensitive data stored in config
|
||||
- HTTPS for all API calls
|
||||
- Input sanitization for token names
|
||||
|
||||
## ✅ Definition of Done
|
||||
|
||||
- [ ] All acceptance criteria met
|
||||
- [ ] Components implemented and tested
|
||||
- [ ] Documentation complete
|
||||
- [ ] Code reviewed and approved
|
||||
- [ ] Merged to main branch
|
||||
- [ ] Deployed to production
|
||||
- [ ] User can successfully add DexScreener connector
|
||||
- [ ] Connector appears in connector list
|
||||
- [ ] Token management works correctly
|
||||
|
||||
---
|
||||
|
||||
## 📎 Related Files
|
||||
|
||||
- Backend Story: [Story 1.1](./story-1.1-dexscreener-connector.md)
|
||||
- Luma Reference: `surfsense_web/components/assistant-ui/connector-popup/connect-forms/components/luma-connect-form.tsx`
|
||||
- Connector Docs: `surfsense_web/content/docs/connectors/`
|
||||
|
||||
## 💡 Implementation Notes
|
||||
|
||||
- DexScreener API is public (no auth required)
|
||||
- Token addresses must be valid EVM addresses (0x + 40 hex chars)
|
||||
- Support up to 50 tokens per connector
|
||||
- Chain list may need updates as new chains are added to DexScreener
|
||||
- Consider adding token search/autocomplete in future iteration
|
||||
Loading…
Add table
Add a link
Reference in a new issue