Commit graph

246 commits

Author SHA1 Message Date
Anish Sarkar
ac0f2fa2eb chore: ran linting 2026-03-17 04:40:46 +05:30
Anish Sarkar
60d12b0a70 Merge remote-tracking branch 'upstream/dev' into feat/web-search 2026-03-16 19:34:38 +05:30
Rohan Verma
ee07ed8168
Merge pull request #883 from octo-patch/feat/complete-minimax-provider-support
feat: add MiniMax LLM provider support
2026-03-15 00:45:16 -07:00
DESKTOP-RTLN3BA\$punk
2b33dfe728 refactor: update safe_set_chunks function to be asynchronous and modify all connector and document processor files to use the new async implementation 2026-03-15 00:44:27 -07:00
Anish Sarkar
c1fe95939f refactor: remove legacy SearXNG API references and web search settings
- Deleted the migration script for adding web search columns to the database.
- Removed web search related fields from the SearchSpace model and schemas.
- Eliminated web search health check endpoint and associated service logic.
- Updated frontend components to remove web search settings management.
- Cleaned up references to SearXNG API in various modules, transitioning to a platform service approach.
2026-03-15 03:49:37 +05:30
Anish Sarkar
d9e312ddaa refactor: simplify SearXNG service configuration in Docker setup
- Consolidated volume mappings for SearXNG to use a single directory.
- Removed unnecessary port mappings and legacy data volume definitions.
- Updated web search service documentation to clarify Redis usage and circuit breaker implementation, eliminating Redis dependency for circuit breaker logic.
2026-03-14 20:34:34 +05:30
Anish Sarkar
d40c6bf743 feat: integrate SearXNG web search service with platform capabilities
- Added SearXNG service configuration to Docker setup, including environment variables and health checks.
- Introduced new settings management for web search in the frontend, allowing users to enable/disable and configure search engines and language preferences.
- Updated backend to support web search functionality, including database schema changes and service integration.
- Implemented health check endpoint for the web search service and integrated it into the application.
- Removed legacy SearXNG API connector references in favor of the new platform service approach.
2026-03-14 20:25:25 +05:30
PR Bot
760aa38225 feat: complete MiniMax LLM provider integration
Add full MiniMax provider support across the entire stack:

Backend:
- Add MINIMAX to LiteLLMProvider enum in db.py
- Add MINIMAX mapping to all provider_map dicts in llm_service.py,
  llm_router_service.py, and llm_config.py
- Add Alembic migration (rev 106) for PostgreSQL enum
- Add MiniMax M2.5 example in global_llm_config.example.yaml

Frontend:
- Add MiniMax to LLM_PROVIDERS enum with apiBase
- Add MiniMax-M2.5 and MiniMax-M2.5-highspeed to LLM_MODELS
- Add MINIMAX to Zod validation schema
- Add MiniMax SVG icon and wire up in provider-icons

Docs:
- Add MiniMax setup guide in chinese-llm-setup.md

MiniMax uses an OpenAI-compatible API (https://api.minimax.io/v1)
with models supporting up to 204K context window.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 07:27:47 +08:00
DESKTOP-RTLN3BA\$punk
eec4db4a3b feat: Enhance LLM configuration and routing with model profile attachment
- Added `_attach_model_profile` function to attach model context metadata to `ChatLiteLLM`.
- Updated `create_chat_litellm_from_config` and `create_chat_litellm_from_agent_config` to utilize the new profile attachment.
- Improved context profile caching in `llm_router_service.py` to include both minimum and maximum input tokens, along with token model names for better context management.
- Introduced new methods for token counting and context trimming based on model profiles.
2026-03-10 18:18:59 -07:00
CREDO23
89ed845dcd merge upstream/dev into improve-ux-connectors
Resolve 3 conflicts:
- connector-popup.tsx: keep Picker modal coordination (pickerOpen state)
- google-drive-config.tsx: keep Google Picker, discard upstream folder tree
- composio-drive-config.tsx: accept upstream collapsible folder tree UX
2026-03-10 23:40:04 +02:00
Rohan Verma
547077e5b9
Merge pull request #865 from CREDO23/sur-182-fix-ux-experience-for-composio-google-drive-connector
[Perf] Batch embedding, non-blocking search, chunks index & Google Drive UX fix
2026-03-10 12:52:16 -07:00
CREDO23
1e2c54eea6 add unified drive-picker-token endpoint and GOOGLE_PICKER_API_KEY config 2026-03-10 20:06:45 +02:00
DESKTOP-RTLN3BA\$punk
403097646d feat: implement batch unread counts for notifications to reduce API calls and improve performance 2026-03-10 01:26:37 -07:00
CREDO23
a49a4db6d6 perf: use asyncio.to_thread for embed_text in FastAPI paths 2026-03-09 16:33:24 +02:00
DESKTOP-RTLN3BA\$punk
40a091f8cc feat: enhance knowledge base search and document retrieval
- Introduced a mechanism to identify degenerate queries that lack meaningful search signals, improving search accuracy.
- Implemented a fallback method for browsing recent documents when queries are degenerate, ensuring relevant results are returned.
- Added limits on the number of chunks fetched per document to optimize performance and prevent excessive data loading.
- Updated the ConnectorService to allow for reusable query embeddings, enhancing efficiency in search operations.
- Enhanced LLM router service to support context window fallbacks, improving robustness during context window limitations.
2026-02-28 19:40:24 -08:00
DESKTOP-RTLN3BA\$punk
d959a6a6c8 feat: optimize document upload process and enhance memory management
- Increased maximum file upload limit from 10 to 50 to improve user experience.
- Implemented batch processing for document uploads to avoid proxy timeouts, splitting files into manageable chunks.
- Enhanced garbage collection in chat streaming functions to prevent memory leaks and improve performance.
- Added memory delta tracking in system snapshots for better monitoring of resource usage.
- Updated LLM router and service configurations to prevent unbounded internal accumulation and improve efficiency.
2026-02-28 17:22:34 -08:00
DESKTOP-RTLN3BA\$punk
f4b2ab0899 feat: enhance caching mechanisms to prevent memory leaks
- Improved in-memory rate limiting by evicting timestamps outside the current window and cleaning up empty keys.
- Updated LLM router service to cache context profiles and avoid redundant computations.
- Introduced cache eviction logic for MCP tools and sandbox instances to manage memory usage effectively.
- Added garbage collection triggers in chat streaming functions to reclaim resources promptly.
2026-02-27 17:56:00 -08:00
DESKTOP-RTLN3BA\$punk
0e723a5b8b feat: perf optimizations
- improved search_knowledgebase_tool
- Added new endpoint to batch-fetch comments for multiple messages, reducing the number of API calls.
- Introduced CommentBatchRequest and CommentBatchResponse schemas for handling batch requests and responses.
- Updated chat_comments_service to validate message existence and permissions before fetching comments.
- Enhanced frontend with useBatchCommentsPreload hook to optimize comment loading for assistant messages.
2026-02-27 17:19:25 -08:00
DESKTOP-RTLN3BA\$punk
664c43ca13 feat: add performance logging middleware and enhance performance tracking across services
- Introduced RequestPerfMiddleware to log request performance metrics, including slow request thresholds.
- Updated various services and retrievers to utilize the new performance logging utility for better tracking of execution times.
- Enhanced existing methods with detailed performance logs for operations such as embedding, searching, and indexing.
- Removed deprecated logging setup in stream_new_chat and replaced it with the new performance logger.
2026-02-27 16:32:30 -08:00
DESKTOP-RTLN3BA\$punk
1e4b8d3e89 feat: enhance document formatting and context management for LLM tools
- Introduced dynamic character budget calculation for document formatting based on model's context window.
- Updated `format_documents_for_context` to respect character limits and improve output quality.
- Added `max_input_tokens` parameter to various functions to facilitate context-aware processing.
- Enhanced error handling for context overflow in LLM router service.
2026-02-26 20:47:19 -08:00
DESKTOP-RTLN3BA\$punk
a4dc84d1ab feat: add should_summarize parameter to task dispatchers
- Introduced should_summarize parameter in TaskDispatcher and CeleryTaskDispatcher to control summary generation.
- Updated InlineTaskDispatcher to support the new parameter for document processing.
2026-02-26 19:12:37 -08:00
DESKTOP-RTLN3BA\$punk
6f4bf11a32 Merge branch 'dev' of https://github.com/MODSetter/SurfSense into dev 2026-02-26 18:25:05 -08:00
DESKTOP-RTLN3BA\$punk
e9892c8fe9 feat: added configable summary calculation and various improvements
- Replaced direct embedding calls with a utility function across various components to streamline embedding logic.
- Added enable_summary flag to several models and routes to control summary generation behavior.
2026-02-26 18:24:57 -08:00
Anish Sarkar
3393e435f9 feat: implement task dispatcher for document processing
- Introduced a TaskDispatcher abstraction to decouple the upload endpoint from Celery, allowing for easier testing with synchronous implementations.
- Updated the create_documents_file_upload function to utilize the new dispatcher for task management.
- Removed direct Celery task imports from the upload function, enhancing modularity.
- Added integration tests for document upload, including page limit enforcement and file size restrictions.
2026-02-26 23:55:47 +05:30
Anish Sarkar
25df3dff64 chore: ran linting 2026-02-26 01:29:25 +05:30
Rohan Verma
d0ee8b12b6
Merge pull request #834 from CREDO23/google-drive-hitl
[Feature] Add human in the loop for google drive sensitive actions (create:docs,sheets & delete)
2026-02-24 16:34:25 -08:00
DESKTOP-RTLN3BA\$punk
a3cd598e01 feat: added improved llm model selector 2026-02-20 14:28:01 -08:00
CREDO23
856d258318 fix bugs and pattern inconsistencies in google drive tools 2026-02-20 19:41:29 +02:00
CREDO23
2220314041 add google drive tool metadata service 2026-02-20 16:05:54 +02:00
CREDO23
c8413ee2bf fix: resolve graphql complexity, json extraction, kb sync, and ui prefill for linear HIL 2026-02-19 18:30:20 +02:00
CREDO23
9b838ec015 fix(linear): use .astext for JSONB text extraction in _resolve_issue
cast(document_metadata["key"], String) generates CAST(col->'key' AS TEXT)
which preserves JSON string quotes (e.g. '"Fix login bug"'), causing
case-insensitive comparisons to never match.

Replace with .astext which generates the ->> operator (unquoted text
extraction), making issue lookups by title and identifier work correctly.
2026-02-19 15:47:40 +02:00
CREDO23
e79ec9234a fix label clear bug, invalid delete URL, and remove dead connector method 2026-02-19 14:35:30 +02:00
CREDO23
c5bd930a71 add create-linear-issue tool UI and multi-account creation context 2026-02-18 20:01:05 +02:00
CREDO23
58955cdc9f feat(linear): add LinearKBSyncService and services __init__ 2026-02-18 17:51:53 +02:00
CREDO23
3fd80a61f9 feat(linear): add LinearToolMetadataService 2026-02-18 17:34:52 +02:00
CREDO23
594379ea69 Disable streaming for KB sync summary LLM to prevent token leaking into chat; remove workspace_icon from update-notion-page 2026-02-18 15:10:51 +02:00
CREDO23
d28aed14b3 Guard KB sync on document_id; fix comment style 2026-02-18 12:31:28 +02:00
CREDO23
64a170efce Fix critical bugs in KB sync implementation
- Add rollback on exception to prevent dirty database state
- Fix block verification threshold from >=1 to >=80%
- Remove technical 'failed' language from user messages
- Add empty content validation with warning
2026-02-17 21:27:58 +02:00
CREDO23
7d1bd1fab4 Implement KB sync after Notion page updates with block ID verification
- Add NotionKBSyncService for immediate KB updates after page changes
- Implement block ID verification to ensure content freshness
- Refactor duplicate block processing logic to shared utils
- Add user-friendly status messages
- Include debug logging for troubleshooting
2026-02-17 20:30:12 +02:00
CREDO23
2f65536c6d use document_id instead of page_id for kb sync lookup 2026-02-17 18:25:56 +02:00
CREDO23
bd3bd701a7 fix jsonb query syntax for page_id lookup 2026-02-17 18:02:00 +02:00
CREDO23
f5195b4c2f fix import path for generate_content_hash 2026-02-17 17:50:47 +02:00
CREDO23
e92aa68523 fix circular import in kb sync service 2026-02-17 17:48:24 +02:00
CREDO23
b240bc3e1a add kb sync for notion updates 2026-02-17 17:37:47 +02:00
DESKTOP-RTLN3BA\$punk
81c70befcf chore: linting 2026-02-15 23:38:15 -08:00
CREDO23
66a6fb685e Merge upstream/dev into feature/human-in-the-loop 2026-02-13 20:17:59 +02:00
CREDO23
9411edf057 Refactor update_notion_page to append-only with simplified parameters 2026-02-13 13:07:19 +02:00
CREDO23
0d1b61d7e6 Implement update notion page tool 2026-02-13 10:45:51 +02:00
Anish Sarkar
a2dd5fb671 chore: ran linting 2026-02-13 02:43:26 +05:30
CREDO23
d33c0dd32f fix: use cast() instead of astext for JSONB query 2026-02-12 19:32:36 +02:00