mirror of
https://github.com/MODSetter/SurfSense.git
synced 2026-07-12 22:42:13 +02:00
d1f11d4fbb
I've implemented several improvements and new features for the Slack connector,
addressing your requirements for more granular control over indexing
and data synchronization.
Key changes include:
Backend (`surfsense_backend`):
- I've updated the `SearchSourceConnector` schema for Slack to include new
configuration options:
- `slack_periodic_indexing_enabled` (boolean)
- `slack_periodic_indexing_frequency` (string: "daily", "weekly", "monthly")
- `slack_max_messages_per_channel_periodic` (integer)
- I've modified the `index_slack_messages` task:
- It now supports on-demand re-indexing of specific `target_channel_ids`.
- It allows `force_reindex_all_messages` to override `last_indexed_at`
for specified channels, using initial indexing settings or custom
date ranges (`reindex_start_date_str`, `reindex_latest_date_str`).
- It uses `slack_max_messages_per_channel_periodic` for regular
periodic updates.
- I've updated the Slack Connector Routes:
- The `/slack/{connector_id}/reindex-channels` endpoint now accepts
`channel_ids`, `force_reindex_all_messages`, `reindex_start_date`,
and `reindex_latest_date` to trigger targeted re-indexing.
- The main `/connector/{id}/index` endpoint for Slack can now accept
`force_full_reindex` to re-index all configured channels from scratch.
Frontend (`surfsense_web`):
- I've created `EditSlackConnectorConfigForm.tsx` to provide a dedicated UI
for Slack connector settings, including the new periodic indexing fields.
- I've integrated this form into the main connector editing page
(`.../connectors/[connector_id]/edit/page.tsx`).
- I've enhanced the Slack connector edit page with a "Channel Management" tab:
- UI for discovering Slack channels via `/api/v1/slack/{id}/discover-channels`.
- Allows selection of channels to be saved into
`config.slack_selected_channel_ids` when membership filter is "selected".
- UI for triggering on-demand re-indexing of selected channels via
`/api/v1/slack/{id}/reindex-channels`, with options for forcing
full re-index and specifying date ranges.
- I've updated the `useSearchSourceConnectors.ts` hook:
- I've added the `discoverSlackChannels` function.
- I've added the `reindexSlackChannels` function with parameters for
channel IDs, force flag, and date ranges.
These changes fulfill your requirements for:
1. Configurable Membership/Join Behavior (via existing `slack_membership_filter_type` and new channel selection UI).
2. Configurable Periodic Indexing (new backend schema and UI fields).
3. Granular Channel Selection (new UI for discovering and selecting channels).
4. On-Demand Re-index (new backend and UI capabilities for specific channels).
5. Initial Indexing Timestamp Range (I've verified existing backend logic and UI).
|
||
|---|---|---|
| .. | ||
| .vscode | ||
| alembic | ||
| app | ||
| .dockerignore | ||
| .env.example | ||
| .gitignore | ||
| .python-version | ||
| alembic.ini | ||
| Dockerfile | ||
| draw.py | ||
| main.py | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
Surf Backend
Technology Stack Overview
This application is a modern AI-powered search and knowledge management platform built with the following technology stack:
Core Framework and Environment
- Python 3.12+: The application requires Python 3.12 or newer
- FastAPI: Modern, fast web framework for building APIs with Python
- Uvicorn: ASGI server implementation, running the FastAPI application
- PostgreSQL with pgvector: Database with vector search capabilities for similarity searches
- SQLAlchemy: SQL toolkit and ORM (Object-Relational Mapping) for database interactions
- FastAPI Users: Authentication and user management with JWT and OAuth support
Key Features and Components
Authentication and User Management
- JWT-based authentication
- OAuth integration (Google)
- User registration, login, and password reset flows
Search and Retrieval System
- Hybrid Search: Combines vector similarity and full-text search for optimal results using Reciprocal Rank Fusion (RRF)
- Vector Embeddings: Document and text embeddings for semantic search
- pgvector: PostgreSQL extension for efficient vector similarity operations
- Chonkie: Advanced document chunking and embedding library
- Uses
AutoEmbeddingsfor flexible embedding model selection LateChunkerfor optimized document chunking based on embedding model's max sequence length
- Uses
AI and NLP Capabilities
- LangChain: Framework for developing AI-powered applications
- Used for document processing, research, and response generation
- Integration with various LLM models through LiteLLM
- Document conversion utilities for standardized processing
- GPT Integration: Integration with LLM models through LiteLLM
- Multiple LLM configurations for different use cases:
- Fast LLM: Quick responses (default: gpt-4o-mini)
- Smart LLM: More comprehensive analysis (default: gpt-4o-mini)
- Strategic LLM: Complex reasoning (default: gpt-4o-mini)
- Long Context LLM: For processing large documents (default: gemini-2.0-flash-thinking)
- Multiple LLM configurations for different use cases:
- Rerankers with FlashRank: Advanced result ranking for improved search relevance
- Configurable reranking models (default: ms-marco-MiniLM-L-12-v2)
- Supports multiple reranking backends (FlashRank, Cohere, etc.)
- Improves search result quality by reordering based on semantic relevance
- GPT-Researcher: Advanced research capabilities
- Multiple research modes (GENERAL, DEEP, DEEPER)
- Customizable report formats with proper citations
- Streaming research results for real-time updates
External Integrations
- Slack Connector: Integration with Slack for data retrieval and notifications
- Notion Connector: Integration with Notion for document retrieval
- Search APIs: Integration with Tavily and Serper API for web search
- Firecrawl: Web crawling and data extraction capabilities
Data Processing
- Unstructured: Tools for processing unstructured data
- Markdownify: Converting HTML to Markdown
- Playwright: Web automation and scraping capabilities
Main Modules
- Search Spaces: Isolated search environments for different contexts or projects
- Documents: Storage and retrieval of various document types
- Chunks: Document fragments for more precise retrieval
- Chats: Conversation management with different depth levels (GENERAL, DEEP)
- Podcasts: Audio content management with generation capabilities
- Search Source Connectors: Integration with various data sources
Development Tools
- Poetry: Python dependency management (indicated by pyproject.toml)
- CORS support: Cross-Origin Resource Sharing enabled for API access
- Environment Variables: Configuration through .env files
Database Schema
The application uses a relational database with the following main entities:
- Users: Authentication and user management
- SearchSpaces: Isolated search environments owned by users
- Documents: Various document types with content and embeddings
- Chunks: Smaller pieces of documents for granular retrieval
- Chats: Conversation tracking with different depth levels
- Podcasts: Audio content with generation capabilities
- SearchSourceConnectors: External data source integrations
API Endpoints
The API is structured with the following main route groups:
/auth/*: Authentication endpoints (JWT, OAuth)/users/*: User management/api/v1/search-spaces/*: Search space management/api/v1/documents/*: Document management/api/v1/podcasts/*: Podcast functionality/api/v1/chats/*: Chat and conversation endpoints/api/v1/search-source-connectors/*: External data source management
Deployment
The application is configured to run with Uvicorn and can be deployed with:
python main.py
This will start the server on all interfaces (0.0.0.0) with info-level logging.
Requirements
See pyproject.toml for detailed dependency information. Key dependencies include:
- asyncpg: Asynchronous PostgreSQL client
- chonkie: Document chunking and embedding library
- fastapi and related packages
- fastapi-users: Authentication and user management
- firecrawl-py: Web crawling capabilities
- langchain components for AI workflows
- litellm: LLM model integration
- pgvector: Vector similarity search in PostgreSQL
- rerankers with FlashRank: Advanced result ranking
- Various AI and NLP libraries
- Integration clients for Slack, Notion, etc.