Commit graph

327 commits

Author SHA1 Message Date
DESKTOP-RTLN3BA\$punk
351b38801b Merge commit '97ed791ca0' into dev 2025-11-03 22:35:18 -08:00
DESKTOP-RTLN3BA\$punk
4dd7e8fc1f chore: Updated UserRead schema to include pages_limit and pages_used fields
- Expanded installation options in README to include SurfSense Cloud as a new method.
- Updated UserRead schema to include pages_limit and pages_used fields.
- Added AnnouncementBanner component to the dashboard layout for improved user notifications.
- Refactored DashboardPage to utilize useUser hook for user state management.
- Integrated page usage display in AppSidebar to show user-specific page limits and usage.
- Removed deprecated apiClient code and replaced it with hooks for better API interaction.
2025-11-03 22:34:37 -08:00
Rohan Verma
97ed791ca0
Merge pull request #463 from codeBunny2022/fix/pdfminer-warning-fix
fix: suppress pdfminer warnings to prevent upload halting
2025-11-03 20:22:38 -08:00
Chirag
2efce16f5f revert: remove setuptools configuration from pyproject.toml 2025-11-04 09:26:52 +05:30
DESKTOP-RTLN3BA\$punk
a3a5b13f48 chore: linting 2025-11-03 16:00:58 -08:00
DESKTOP-RTLN3BA\$punk
e65d74f2e2 refactor: added batch commits and Increased task time limits in celery_app.py
- Increased task time limits in celery_app.py for longer processing times.
- Enhanced pagination logic in NotionHistoryConnector to handle large result sets.
- Implemented batch commits every 10 documents across various indexers (Airtable, ClickUp, Confluence, Discord, GitHub, Google Calendar, Gmail, JIRA, Linear, Luma, Notion, Slack) to improve performance and reduce database load.
- Updated final commit logging for clarity on total documents processed.
2025-11-03 15:57:19 -08:00
Chirag
b3026e4412 fix: resolve ruff F823 error by importing getLogger and ERROR directly 2025-11-02 12:03:03 +05:30
Chirag
094bdfad45 fix: suppress pdfminer warnings to prevent upload halting
- Added warning suppression for pdfminer warnings during Docling PDF processing
- Suppresses 'Cannot set gray non-stroke color' warnings that cause uploads to halt
- Temporarily sets pdfminer logger to ERROR level during document processing
- Fixes issue where files ~34MB would fail due to pdfminer warning spam

Resolves issue where PDF uploads would halt with repeated pdfminer warnings
2025-11-01 17:42:23 +05:30
DESKTOP-RTLN3BA\$punk
ecd07d6155 refactor: update API endpoint paths to remove trailing slashes
- Modified various FastAPI route definitions to remove trailing slashes for consistency across the application.
- Updated corresponding fetch calls in the frontend to align with the new endpoint structure.
- Ensured that all affected routes maintain their functionality without trailing slashes.
2025-10-31 01:33:01 -07:00
DESKTOP-RTLN3BA\$punk
6062bf63d7 feat: add ProxyHeadersMiddleware to support HTTPS redirects behind proxies
- Integrated ProxyHeadersMiddleware to ensure FastAPI correctly handles HTTPS in redirects when deployed behind a proxy, such as Cloudflare.
- This addition enhances security and proper URL handling in the application.
2025-10-31 00:24:53 -07:00
DESKTOP-RTLN3BA\$punk
a0f9c3aad8 feat: add BACKEND_URL configuration for OAuth redirect
- Introduced BACKEND_URL in the configuration to allow overriding the HTTP to HTTPS in the OAuth redirect URI.
- Updated the Google OAuth router to conditionally use the BACKEND_URL for the redirect URI when specified.
2025-10-30 23:52:14 -07:00
DESKTOP-RTLN3BA\$punk
a2fb9faad6 feat: enhance Azure OpenAI embeddings and add voice support for Azure provider
- Introduced a fixed parameter order for AzureOpenAIEmbeddings to resolve compatibility issues.
- Updated the voice selection logic to include Azure voices in the podcaster utility.
- Modified the page limit service to use a more efficient method for retrieving users.
2025-10-30 22:33:47 -07:00
DESKTOP-RTLN3BA\$punk
4be9d099bf feat: added file limit tracking for a user 2025-10-30 14:58:08 -07:00
DESKTOP-RTLN3BA\$punk
d76da8a59d fix: podcast tasks for Windows compatibility
- Added WindowsProactorEventLoopPolicy for better async subprocess support on Windows.
- Ensured proper cleanup of the event loop after task execution.
- Removed redundant asyncio import from the generate_chat_podcast_task function.
2025-10-30 12:05:06 -07:00
DESKTOP-RTLN3BA\$punk
0987440893 chore: update configuration for rerankers
- Added RERANKERS_ENABLED option to control reranking functionality.
- Updated rerank_documents function to handle cases when reranking is disabled.
- Enhanced documentation for environment variables related to rerankers in installation guides.
2025-10-29 23:23:08 -07:00
DESKTOP-RTLN3BA\$punk
71e4860495 chore: updated chonkie and temp fix for azure embeddings registry
- TODO: Raise PR in upstream for fix in next version
2025-10-29 14:42:05 -07:00
DESKTOP-RTLN3BA\$punk
133e2639ec fix: docker backend entrpoint issues 2025-10-28 23:35:53 -07:00
DESKTOP-RTLN3BA\$punk
81ddc81026 feat(removed): sub_section_writer
- Its bad and not needed.
2025-10-27 20:30:10 -07:00
DESKTOP-RTLN3BA\$punk
d5ccced6b9 feat: frontend docker to use nextjs production build
- Damn this made things fast af.
2025-10-23 20:37:01 -07:00
DESKTOP-RTLN3BA\$punk
590baff093 fix: reduced docker size by 67% 2025-10-23 15:49:16 -07:00
DESKTOP-RTLN3BA\$punk
aed8163aee feat: added celery beat to docker 2025-10-23 00:56:28 -07:00
MSI\ModSetter
8706df6716 feat: added meta-scheduler for running periodic tasks 2025-10-22 19:49:44 -07:00
MSI\ModSetter
a90767a478 feat: added meta-scheduler for running periodic tasks 2025-10-22 19:49:30 -07:00
DESKTOP-RTLN3BA\$punk
182f815bb7 feat: added periodic tasks in backend db and frontend hooks
- TODO: Add celery redbeat and create tasks dinamically in our redis
2025-10-22 16:14:25 -07:00
DESKTOP-RTLN3BA\$punk
18adf79649 feat(fix): document type filtering 2025-10-21 21:53:55 -07:00
DESKTOP-RTLN3BA\$punk
0e6669ac4e fix: celery_app path and gmail indexing 2025-10-21 21:11:41 -07:00
Anish Sarkar
fd94193fd5 fix: ran linting for both backend and frontend 2025-10-20 16:25:14 +05:30
Anish Sarkar
19ed0becce feat: implement registration toggle in backend and handle disabled state in frontend 2025-10-20 15:54:52 +05:30
DESKTOP-RTLN3BA\$punk
c80bbfa867 feat: added celery and removed background_tasks for MQ's
- removed pre commit hooks
- updated docker setup
- updated github docker actions
- updated docs
2025-10-20 00:30:00 -07:00
DESKTOP-RTLN3BA\$punk
5b957ec21c feat: bumped version to v0.0.8 2025-10-16 22:44:12 -07:00
Anish Sarkar
bbb2abfc02 fix: ran formatter as per coderrabbitai 2025-10-17 02:44:44 +05:30
Anish Sarkar
0ff1b586a2 feat: update Elasticsearch integration and logging
- revised Elasticsearch connector enum revision IDs
- added `TaskLoggingService` to elasticsearch_indexer
- integrated Elasticsearch into prompts.py as requested
2025-10-17 02:21:56 +05:30
Anish Sarkar
8e1e81ebae Merge remote-tracking branch 'upstream/main' into feature/elasticsearch-connector 2025-10-17 01:02:29 +05:30
Differ
a479fea588 fix: Baidu search UI rendering and clean debug code
- Fix: Add BAIDU_SEARCH_API to direct render sources
- Fix: Remove API key exposure in debug logs
- Resolves 404 error when clicking Baidu search results
2025-10-16 22:45:35 +08:00
Anish Sarkar
82438c7396 refactor: streamline Elasticsearch indexing by removing unused services and integrating document chunking, also added documentation 2025-10-16 17:48:28 +05:30
Differ
beaf8f89a6 feat: add Baidu AI Search integration
- Add BAIDU_SEARCH_API connector type to support Chinese web search
- Implement search_baidu() method in connector_service.py
- Add frontend configuration page for Baidu Search API
- Create Alembic migration for new enum values
- Add validation rules and agent integration
- Support configurable model, search source, and deep search options
- Update .gitignore to exclude .env.local and other env files

Addresses integration with Chinese search ecosystem for better local market support.
Baidu AI Search provides intelligent search with automatic summarization.
2025-10-16 19:30:42 +08:00
Anish Sarkar
3c0f5e42f3 updated enum revision id and uv.lock 2025-10-16 16:52:27 +05:30
Anish Sarkar
929035f802 Merge remote-tracking branch 'upstream/main' into feature/elasticsearch-connector 2025-10-16 16:24:37 +05:30
DESKTOP-RTLN3BA\$punk
70b547c9c9 chore: updated docs & refactored sst_service.py 2025-10-15 14:31:38 -07:00
Rohan Verma
5ed9aa2b0b
Merge pull request #387 from nabthebest135/local-stt
local STT implementation with Faster-Whisper
2025-10-15 14:08:09 -07:00
DESKTOP-RTLN3BA\$punk
97f06899e4 Merge branch 'dev' of https://github.com/MODSetter/SurfSense into dev 2025-10-14 21:12:17 -07:00
DESKTOP-RTLN3BA\$punk
c99cd710ea feat: add unique identifier hash for documents to prevent duplicates across various connectors 2025-10-14 21:11:19 -07:00
MSI\ModSetter
031dc714eb chore: fixed validators 2025-10-14 17:03:38 -07:00
DESKTOP-RTLN3BA\$punk
31982cea9a chore: removed content trunking for better UI 2025-10-14 14:19:48 -07:00
DESKTOP-RTLN3BA\$punk
ba5bb91a7b chore: removed chinese comments to pass ruff checks and updated migration nos 2025-10-13 20:07:32 -07:00
Rohan Verma
c99469bfdf
Merge pull request #391 from unitagain/feature/chinese-llm-support
feat: add Chinese LLM providers support with auto-fill API Base URL
2025-10-13 19:52:21 -07:00
DESKTOP-RTLN3BA\$punk
acd900c41d ruff fix 2025-10-13 14:01:55 -07:00
DESKTOP-RTLN3BA\$punk
576dc19260 refactor: updated SearxNG connector migration and fixed validation for SearxNG host URL 2025-10-13 13:57:58 -07:00
Rohan Verma
08661e686c
Merge pull request #392 from Aki-07/feature/add-searxng-connector
Add Searxng connector
2025-10-13 13:41:37 -07:00
Nabhan
9b72ec65b5 fix: address code review feedback for STT implementation
- Add header to local STT transcription for consistency
- Add empty text validation for external STT path
- Refactor external STT to eliminate duplication in atranscription calls
- Ensure both local and external paths have consistent error handling
2025-10-13 14:26:36 +05:00