mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-12 22:42:13 +02:00
4bf09b8bde
Backend Tests (`surfsense_backend`):
- `test_slack_history.py`:
- Tests for `SlackHistory` class methods including `get_all_channels`,
`get_conversation_history`, timestamp conversions, and message
formatting. Covers various API response scenarios, pagination,
and error handling.
- `test_connectors_indexing_tasks.py`:
- Unit tests for `index_slack_messages` task.
- Scenarios include initial indexing, periodic indexing, on-demand
re-indexing (targeted channels, force re-index, custom dates),
channel filtering, and error handling.
- Mocks `SlackHistory` and database interactions.
- `test_search_source_connectors_routes.py`:
- Tests for `/slack/{connector_id}/discover-channels` endpoint.
- Tests for `/slack/{connector_id}/reindex-channels` endpoint,
verifying parameter passing to the background task.
- Tests for the main `/search-source-connectors/{connector_id}/index`
endpoint for Slack connectors, ensuring the `force_full_reindex`
flag is handled.
Frontend Tests (`surfsense_web`):
- `components/editConnector/EditSlackConnectorConfigForm.test.tsx`:
- Tests rendering of all Slack configuration fields.
- Verifies `onConfigChange` callback functionality.
- Tests conditional rendering of periodic indexing settings.
- Checks disabled state of the form.
- `app/dashboard/[search_space_id]/connectors/[connector_id]/edit/page.test.tsx`:
- Tests for the "Channel Management" tab specific to Slack connectors.
- Verifies UI for discovering channels (mocking API calls).
- Tests selection of channels and updating the connector configuration.
- Tests UI for triggering on-demand re-indexing with various options.
- `hooks/useSearchSourceConnectors.test.ts`:
- Unit tests for the `discoverSlackChannels` function, mocking `fetch`.
- Unit tests for the `reindexSlackChannels` function, mocking `fetch`
and verifying correct request body construction.
- Includes testing of the `fetchWithAuth` helper function.
|
||
|---|---|---|
| .. | ||
| .vscode | ||
| alembic | ||
| app | ||
| .dockerignore | ||
| .env.example | ||
| .gitignore | ||
| .python-version | ||
| alembic.ini | ||
| Dockerfile | ||
| draw.py | ||
| main.py | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
Surf Backend
Technology Stack Overview
This application is a modern AI-powered search and knowledge management platform built with the following technology stack:
Core Framework and Environment
- Python 3.12+: The application requires Python 3.12 or newer
- FastAPI: Modern, fast web framework for building APIs with Python
- Uvicorn: ASGI server implementation, running the FastAPI application
- PostgreSQL with pgvector: Database with vector search capabilities for similarity searches
- SQLAlchemy: SQL toolkit and ORM (Object-Relational Mapping) for database interactions
- FastAPI Users: Authentication and user management with JWT and OAuth support
Key Features and Components
Authentication and User Management
- JWT-based authentication
- OAuth integration (Google)
- User registration, login, and password reset flows
Search and Retrieval System
- Hybrid Search: Combines vector similarity and full-text search for optimal results using Reciprocal Rank Fusion (RRF)
- Vector Embeddings: Document and text embeddings for semantic search
- pgvector: PostgreSQL extension for efficient vector similarity operations
- Chonkie: Advanced document chunking and embedding library
- Uses
AutoEmbeddingsfor flexible embedding model selection LateChunkerfor optimized document chunking based on embedding model's max sequence length
- Uses
AI and NLP Capabilities
- LangChain: Framework for developing AI-powered applications
- Used for document processing, research, and response generation
- Integration with various LLM models through LiteLLM
- Document conversion utilities for standardized processing
- GPT Integration: Integration with LLM models through LiteLLM
- Multiple LLM configurations for different use cases:
- Fast LLM: Quick responses (default: gpt-4o-mini)
- Smart LLM: More comprehensive analysis (default: gpt-4o-mini)
- Strategic LLM: Complex reasoning (default: gpt-4o-mini)
- Long Context LLM: For processing large documents (default: gemini-2.0-flash-thinking)
- Multiple LLM configurations for different use cases:
- Rerankers with FlashRank: Advanced result ranking for improved search relevance
- Configurable reranking models (default: ms-marco-MiniLM-L-12-v2)
- Supports multiple reranking backends (FlashRank, Cohere, etc.)
- Improves search result quality by reordering based on semantic relevance
- GPT-Researcher: Advanced research capabilities
- Multiple research modes (GENERAL, DEEP, DEEPER)
- Customizable report formats with proper citations
- Streaming research results for real-time updates
External Integrations
- Slack Connector: Integration with Slack for data retrieval and notifications
- Notion Connector: Integration with Notion for document retrieval
- Search APIs: Integration with Tavily and Serper API for web search
- Firecrawl: Web crawling and data extraction capabilities
Data Processing
- Unstructured: Tools for processing unstructured data
- Markdownify: Converting HTML to Markdown
- Playwright: Web automation and scraping capabilities
Main Modules
- Search Spaces: Isolated search environments for different contexts or projects
- Documents: Storage and retrieval of various document types
- Chunks: Document fragments for more precise retrieval
- Chats: Conversation management with different depth levels (GENERAL, DEEP)
- Podcasts: Audio content management with generation capabilities
- Search Source Connectors: Integration with various data sources
Development Tools
- Poetry: Python dependency management (indicated by pyproject.toml)
- CORS support: Cross-Origin Resource Sharing enabled for API access
- Environment Variables: Configuration through .env files
Database Schema
The application uses a relational database with the following main entities:
- Users: Authentication and user management
- SearchSpaces: Isolated search environments owned by users
- Documents: Various document types with content and embeddings
- Chunks: Smaller pieces of documents for granular retrieval
- Chats: Conversation tracking with different depth levels
- Podcasts: Audio content with generation capabilities
- SearchSourceConnectors: External data source integrations
API Endpoints
The API is structured with the following main route groups:
/auth/*: Authentication endpoints (JWT, OAuth)/users/*: User management/api/v1/search-spaces/*: Search space management/api/v1/documents/*: Document management/api/v1/podcasts/*: Podcast functionality/api/v1/chats/*: Chat and conversation endpoints/api/v1/search-source-connectors/*: External data source management
Deployment
The application is configured to run with Uvicorn and can be deployed with:
python main.py
This will start the server on all interfaces (0.0.0.0) with info-level logging.
Requirements
See pyproject.toml for detailed dependency information. Key dependencies include:
- asyncpg: Asynchronous PostgreSQL client
- chonkie: Document chunking and embedding library
- fastapi and related packages
- fastapi-users: Authentication and user management
- firecrawl-py: Web crawling capabilities
- langchain components for AI workflows
- litellm: LLM model integration
- pgvector: Vector similarity search in PostgreSQL
- rerankers with FlashRank: Advanced result ranking
- Various AI and NLP libraries
- Integration clients for Slack, Notion, etc.