mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-18 23:11:12 +02:00
test: Add Vitest configuration and initial tests for the DexScreener connect form.
This commit is contained in:
API Test Bot
parent
828d7d695a
commit
fd9eddf7fa
11 changed files with 1480 additions and 116 deletions
|
|
@ -38,3 +38,75 @@ Backend của SurfSense là một ứng dụng **Python FastAPI** mạnh mẽ,
|
|||
- Nếu là tác vụ AI: Đẩy vào LangGraph runner để streaming phản hồi.
|
||||
- Nếu là tác vụ dài (Ingestion): Đẩy job vào Redis queue cho Celery.
|
||||
5. **Response**: Trả về JSON hoặc Streaming Response (SSE).
|
||||
|
||||
## Critical RAG Pipeline Fix (Feb 2026)
|
||||
|
||||
### DexScreener Connector Integration
|
||||
|
||||
**Issue Discovered**: DexScreener connector was successfully implemented and indexed data into `search_space_id = 7`, but the LLM could not retrieve this data when users asked about crypto prices.
|
||||
|
||||
**Root Cause**: Missing connector mapping in `_CONNECTOR_TYPE_TO_SEARCHABLE` dictionary.
|
||||
|
||||
**File**: `surfsense_backend/app/agents/new_chat/chat_deepagent.py`
|
||||
|
||||
**The Problem**:
|
||||
```python
|
||||
# BEFORE (Missing mapping)
|
||||
_CONNECTOR_TYPE_TO_SEARCHABLE = {
|
||||
"GMAIL": "GMAIL",
|
||||
"GOOGLE_DRIVE_CONNECTOR": "GOOGLE_DRIVE",
|
||||
"SLACK_CONNECTOR": "SLACK",
|
||||
# ... other connectors ...
|
||||
# ❌ DEXSCREENER_CONNECTOR was MISSING
|
||||
}
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
1. `connector_service.get_available_connectors()` returned DexScreener connector type
|
||||
2. `_map_connectors_to_searchable_types()` could not find mapping → ignored DexScreener
|
||||
3. LLM's tool description didn't mention DexScreener as available
|
||||
4. LLM never searched DexScreener data, responded "can't see price data"
|
||||
|
||||
**The Fix**:
|
||||
```python
|
||||
# AFTER (Fixed)
|
||||
_CONNECTOR_TYPE_TO_SEARCHABLE = {
|
||||
"GMAIL": "GMAIL",
|
||||
"GOOGLE_DRIVE_CONNECTOR": "GOOGLE_DRIVE",
|
||||
"SLACK_CONNECTOR": "SLACK",
|
||||
# ... other connectors ...
|
||||
"DEXSCREENER_CONNECTOR": "DEXSCREENER_CONNECTOR", # ✅ Added
|
||||
}
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
- User query: *"What's the current price of WETH?"*
|
||||
- LLM successfully retrieved: ~$2,442 USD with DexScreener citations
|
||||
- Citations linked to indexed trading pairs with metadata (chain, DEX, liquidity, volume)
|
||||
|
||||
**Lesson Learned**: When adding new connectors, **ALWAYS** update the `_CONNECTOR_TYPE_TO_SEARCHABLE` mapping to enable RAG retrieval. This is a critical step that's easy to miss during implementation.
|
||||
|
||||
---
|
||||
|
||||
## Connector Architecture Pattern
|
||||
|
||||
### Adding New Connectors (Best Practices)
|
||||
|
||||
Khi thêm connector mới, cần update **4 locations**:
|
||||
|
||||
1. **Connector Class** (`app/connectors/`)
|
||||
- Implement data fetching logic
|
||||
- Format data to markdown for indexing
|
||||
|
||||
2. **Database Enum** (`app/db.py`)
|
||||
- Add to `SearchSourceConnectorType` enum
|
||||
|
||||
3. **API Routes** (`app/routes/`)
|
||||
- Create add/delete/test endpoints
|
||||
|
||||
4. **RAG Mapping** (`app/agents/new_chat/chat_deepagent.py`) ⚠️ **CRITICAL**
|
||||
- Add to `_CONNECTOR_TYPE_TO_SEARCHABLE` dictionary
|
||||
- **Failure to do this = LLM cannot access connector data**
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -18,8 +18,9 @@
|
|||
- 🎫 **Jira** - Tìm kiếm tickets
|
||||
- 📚 **Confluence** - Tìm kiếm wiki pages
|
||||
- 🗂️ **Microsoft Teams** - Tìm kiếm chats và files
|
||||
- 💰 **DexScreener** - Theo dõi giá token crypto và trading pairs
|
||||
|
||||
**Tổng cộng:** SurfSense hỗ trợ **26+ connectors** khác nhau!
|
||||
**Tổng cộng:** SurfSense hỗ trợ **27+ connectors** khác nhau!
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -238,6 +239,30 @@ User → SurfSense → Notion OAuth → Access Token → SurfSense
|
|||
- Chỉ hoạt động khi SurfSense chạy self-hosted
|
||||
- Truy cập trực tiếp vào local file system
|
||||
|
||||
### 5. API-Based (No Authentication)
|
||||
|
||||
**Ví dụ:** DexScreener Connector
|
||||
|
||||
- Không cần OAuth hay API key (public API)
|
||||
- User chỉ cần cấu hình tokens muốn theo dõi
|
||||
- Ưu điểm:
|
||||
- Setup cực kỳ đơn giản (không cần đăng ký API key)
|
||||
- Miễn phí hoàn toàn
|
||||
- Real-time data từ public blockchain
|
||||
- Nhược điểm:
|
||||
- Bị giới hạn rate limit của public API
|
||||
- Không có personalized data
|
||||
|
||||
**Flow:**
|
||||
```
|
||||
User → Nhập token addresses → SurfSense → DexScreener Public API → Token Price Data
|
||||
```
|
||||
|
||||
**Use Case:**
|
||||
- Theo dõi giá crypto tokens (WETH, USDC, etc.)
|
||||
- Phân tích trading pairs trên các DEX
|
||||
- AI có thể trả lời: *"What's the current price of WETH?"*
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Cấu Hình Connector
|
||||
|
|
@ -278,6 +303,24 @@ Mỗi connector có các settings:
|
|||
}
|
||||
```
|
||||
|
||||
**DexScreener:**
|
||||
```json
|
||||
{
|
||||
"tokens": [
|
||||
{
|
||||
"chain": "ethereum",
|
||||
"address": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
|
||||
"name": "WETH"
|
||||
},
|
||||
{
|
||||
"chain": "bsc",
|
||||
"address": "0xbb4CdB9CBd36B01bD1cBaEBF2De08d9173bc095c",
|
||||
"name": "WBNB"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💡 Use Cases
|
||||
|
|
@ -320,6 +363,23 @@ Mỗi connector có các settings:
|
|||
3. Slack discussion về issue
|
||||
4. Confluence: Payment API Architecture
|
||||
|
||||
### 3. Crypto Trader
|
||||
|
||||
**Scenario:** Theo dõi giá token và phân tích market trends.
|
||||
|
||||
**Connectors kết nối:**
|
||||
- DexScreener (token prices và trading pairs)
|
||||
- Twitter/X (crypto news - nếu có connector)
|
||||
- Notion (trading notes)
|
||||
|
||||
**Search query trong AI Chat:** *"What's the current price of WETH and how has it changed in the last 24 hours?"*
|
||||
|
||||
**Kết quả:**
|
||||
- AI trả lời với real-time price data từ DexScreener
|
||||
- Hiển thị price changes (5m, 1h, 24h)
|
||||
- Liquidity và volume information
|
||||
- Citations link đến DexScreener pairs
|
||||
|
||||
---
|
||||
|
||||
## 🚨 Lưu Ý Quan Trọng
|
||||
|
|
|
|||
|
|
@ -796,6 +796,76 @@ async def index_connector_data(connector: SearchSourceConnector):
|
|||
# ... other connector types
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ CRITICAL: Enable RAG Retrieval
|
||||
|
||||
**This is the most commonly missed step when adding new connectors!**
|
||||
|
||||
### The Problem
|
||||
|
||||
Even if your connector successfully:
|
||||
1. ✅ Stores data in the database
|
||||
2. ✅ Indexes data into vector store
|
||||
3. ✅ Shows up in the UI
|
||||
|
||||
The LLM **WILL NOT** be able to retrieve this data unless you add the connector to the RAG mapping.
|
||||
|
||||
### The Fix
|
||||
|
||||
**File:** `surfsense_backend/app/agents/new_chat/chat_deepagent.py`
|
||||
|
||||
**Add your connector to `_CONNECTOR_TYPE_TO_SEARCHABLE`:**
|
||||
|
||||
```python
|
||||
_CONNECTOR_TYPE_TO_SEARCHABLE = {
|
||||
"GMAIL": "GMAIL",
|
||||
"GOOGLE_DRIVE_CONNECTOR": "GOOGLE_DRIVE",
|
||||
"SLACK_CONNECTOR": "SLACK",
|
||||
"NOTION_CONNECTOR": "NOTION",
|
||||
# ... other connectors ...
|
||||
|
||||
# ✅ ADD YOUR NEW CONNECTOR HERE
|
||||
"DEXSCREENER_CONNECTOR": "DEXSCREENER_CONNECTOR",
|
||||
"YOUR_CONNECTOR": "YOUR_CONNECTOR", # Example
|
||||
}
|
||||
```
|
||||
|
||||
### Why This Matters
|
||||
|
||||
This mapping is used by `_map_connectors_to_searchable_types()` to:
|
||||
1. Build the list of available search spaces for the LLM
|
||||
2. Include connector types in the tool description
|
||||
3. Enable the LLM to search your connector's data
|
||||
|
||||
**Without this mapping:**
|
||||
- LLM won't know your connector exists
|
||||
- LLM can't search your indexed data
|
||||
- Users will get "I don't have access to that data" responses
|
||||
|
||||
### Verification
|
||||
|
||||
After adding the mapping, test with a user query:
|
||||
|
||||
```bash
|
||||
# Example for DexScreener
|
||||
curl -X POST http://localhost:8000/api/chat \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d '{
|
||||
"message": "What is the current price of WETH?",
|
||||
"space_id": 7
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected:** LLM retrieves data and provides answer with citations.
|
||||
|
||||
**If failed:** Check that:
|
||||
1. Connector is in `_CONNECTOR_TYPE_TO_SEARCHABLE`
|
||||
2. Connector type matches exactly (case-sensitive)
|
||||
3. Data is indexed in the correct `search_space_id`
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. **Error Handling**
|
||||
|
|
|
|||
|
|
@ -195,6 +195,11 @@ SurfSense cho phép bạn kết nối với **26+ ứng dụng bên ngoài** nh
|
|||
- **Kết nối Google Drive:** Tìm files và documents ngay trong SurfSense
|
||||
- **Kết nối Slack:** Tìm conversations và shared files từ workspace
|
||||
- **Kết nối Notion:** Tìm kiếm trong pages và databases
|
||||
- **Kết nối DexScreener:** Theo dõi giá crypto tokens real-time
|
||||
- Không cần API key
|
||||
- Chỉ cần nhập token addresses muốn theo dõi
|
||||
- AI có thể trả lời: *"What's the current price of WETH?"*
|
||||
- Xem trading pairs, liquidity, volume, price changes
|
||||
|
||||
**Quản lý Connectors:**
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue