Commit graph

1075 commits

Author SHA1 Message Date
CREDO23
64cd65bc1f use trafilatura to extrack page content from the chromium result 2025-12-19 10:05:51 +02:00
DESKTOP-RTLN3BA\$punk
c6cc7c2a6a feat: test script of new agent 2025-12-18 23:57:57 -08:00
CREDO23
1f60d1c22f add user agent to AsyncChromiumLoader 2025-12-17 19:43:54 +02:00
CREDO23
c6cb754aac refactor: update the webcrawler index to compare hashes without metadata 2025-12-17 18:44:58 +02:00
CREDO23
4cfeffb38a refactor: update the webcrawler connector formater 2025-12-17 18:42:37 +02:00
DESKTOP-RTLN3BA\$punk
c768730b8c feat: fixed issues of note management
Issues Fixed

- Missing pagination fields in API response schemas (page, page_size, has_more)
- NOTE enum missing from frontend Zod schema
- Missing fields in DocumentRead response construction (content_hash, updated_at)
- BlockNote slash menu clipped by overflow-hidden CSS
- Sidebar click conflicts - hidden action buttons intercepting clicks
- Rewrote All Notes sidebar - replaced fragile custom portal with shadcn Sheet
- Missing translation keys for new UI strings
- Missing NOTE retrieval logic in researcher agent
- Added search to All Notes sidebar
- Removed frontend logging - was causing toasters on every page refresh
- Added backend logging to document reindex Celery task
2025-12-17 00:09:43 -08:00
Anish Sarkar
4cef8cc93f feat: enhance validation and formatting in editor and sidebar components 2025-12-16 21:27:31 +05:30
Anish Sarkar
7bd638668a Merge remote-tracking branch 'upstream/main' into feature/note-management
updating
2025-12-16 20:28:35 +05:30
Anish Sarkar
e8e393a0d5 fix: fixed linting for some files 2025-12-16 20:25:50 +05:30
Anish Sarkar
e1e813702a feat: enhance note creation and editing experience
- Added dynamic title extraction from the first block of BlockNote documents for NOTE type.
- Updated editor routes to support new note creation with a BlockNote editor.
- Implemented unsaved changes dialog for better user experience when navigating away from the editor.
- Refactored BlockNoteEditor to ensure the first block is a heading when creating new notes.
- Removed the old note creation page in favor of the new streamlined editor experience.
2025-12-16 13:57:06 +05:30
Anish Sarkar
8eceb7a5cb feat: add note management functionality with BlockNote support
- Introduced a new ENUM value 'NOTE' for document types in the database.
- Implemented backend routes for creating, listing, and deleting notes.
- Added a new frontend page for creating notes with a BlockNote editor.
- Updated sidebar to include recent notes and an option to add new notes.
- Enhanced API service for notes with validation and request/response schemas.
- Updated translations to support new note-related terms.
2025-12-16 12:28:30 +05:30
DESKTOP-RTLN3BA\$punk
fd1f3eaf60 chore: update langchain
- Updated langchain-unstructured to version 1.0.0 and langchain to 1.2.0.
- Updated litellm to version 1.80.10 and grpcio versions based on Python version.
- Updated fastuuid to version 0.14.0 and langgraph to 1.0.5.
- Updated openai to version 2.12.0 and langgraph-sdk to 0.3.0.
- Added uuid-utils package version 0.12.0.
- Refactored imports in utils.py, chats_routes.py, and query_service.py to use langchain_core.messages.
2025-12-15 21:25:44 -08:00
DESKTOP-RTLN3BA\$punk
fea1837186 roadmap(1.3): Update citation prompt to use new whole document structure
- Modified the document extraction and citation formatting to accommodate a new structure that includes a `chunks` list for each document.
- Enhanced the citation format to reference `chunk_id` instead of `source_id`, ensuring accurate citations in the UI.
- Updated various components, including the connector service and reranker service, to handle the new document format and maintain compatibility with existing functionalities.
- Improved documentation and comments to reflect changes in the data structure and citation requirements.
2025-12-14 22:07:31 -08:00
DESKTOP-RTLN3BA\$punk
08fb488995 feat(roadmap-1.2): removed SearchMode enum and combined hybrid search 2025-12-12 04:17:30 -08:00
DESKTOP-RTLN3BA\$punk
919c323ef3 feat: implement time-based filtering for document retrieval using 'updated_at' timestamp 2025-12-12 02:42:20 -08:00
DESKTOP-RTLN3BA\$punk
8c9aa68faa feat: update document tracking to use 'updated_at' timestamp instead of 'last_edited_at' 2025-12-12 01:32:14 -08:00
DESKTOP-RTLN3BA\$punk
1cbe479af4 chore: ruff format 2025-12-11 00:31:58 -08:00
DESKTOP-RTLN3BA\$punk
3207fbf9ea feat(fix): add PAGES_LIMIT configuration for ETL services 2025-12-11 00:29:56 -08:00
Rohan Verma
e1e14c7ab6
Merge pull request #534 from ankitpasayat/fix/retriever-typo
fix(backend): rename retriver directory to retriever
2025-12-06 17:54:18 -08:00
Ankit Pasayat
b30f53bf0b fix(backend): rename retriver directory to retriever
Fixes typo in directory name and updates all import paths:
- Renamed surfsense_backend/app/retriver/ to surfsense_backend/app/retriever/
- Updated imports in db.py
- Updated imports in connector_service.py
2025-12-06 22:21:30 +05:30
Differ
e238fab638 Merge remote-tracking branch 'upstream/main' into feat/bookstack-connector 2025-12-06 09:15:02 +08:00
Rohan Verma
601489bec2
Merge pull request #525 from CREDO23/refactor/jira-connector-migrate-tot-rest-v3
[Refactor] Jira Connector | Migrate to atlassian rest api v3
2025-12-05 12:14:46 -08:00
DESKTOP-RTLN3BA\$punk
c97887a63d fix: implement real-time streaming for responses
- Added streaming service support to the Q&A agent for real-time token streaming.
- Updated `answer_question` method to stream responses token-by-token to the frontend.
- Modified `handle_qna_workflow` to handle both custom and values streaming modes.
- Enhanced state management to include streaming service for improved user experience.
2025-12-05 00:14:36 -08:00
Differ
500bc60d02 fix: add input validation, retry limit, code formatting, and exclude i18n from secret detection 2025-12-05 09:58:49 +08:00
CREDO23
803f792a9d clean up 2025-12-04 12:55:19 +02:00
CREDO23
521cea3ef0 update query parmas for get issues by date range method 2025-12-04 12:53:18 +02:00
Differ
6b1b8d0f2e feat: add BookStack connector for wiki documentation indexing 2025-12-04 14:08:44 +08:00
WayChan
3c423436cc fix: retrieve wrong field for content in saving extension document. 2025-12-04 00:31:50 +00:00
CREDO23
107f013ff9 jira-connector: update get_issues_by_date_range method 2025-12-04 01:21:46 +02:00
CREDO23
abf017eabb jira-connector: update get_issues_by_date_range method 2025-12-04 00:48:54 +02:00
CREDO23
4df6b09db9 jira-connector: update get all issues method 2025-12-04 00:42:10 +02:00
CREDO23
875924e5fd jira-connector: update make_api_request to accespt POST with payload 2025-12-04 00:38:13 +02:00
WayChan
081080233a fix: saving document from browser extension fails due to missing and mismatch fields of backend data models 2025-12-03 15:32:32 +00:00
DESKTOP-RTLN3BA\$punk
ea94c778c9 refactor: streamline document actions
- Simplified the RowActions component by removing unused imports and replacing the dropdown menu with tooltip buttons for editing, viewing metadata, and deleting documents.
- Updated the JsonMetadataViewer component to support controlled mode for better integration with the RowActions component.
- Adjusted CSS variables for destructive actions in globals.css for improved visual consistency.
2025-11-30 15:06:48 -08:00
DESKTOP-RTLN3BA\$punk
d0c7be7eca feat: update document editor routes to include search space context
- Modified API endpoints to include search space ID in the URL for fetching and saving documents.
- Added permission checks for reading and updating documents based on search space.
2025-11-30 14:27:27 -08:00
Anish Sarkar
f92112a79a fix: code cleanup 2025-11-30 11:37:50 +05:30
Anish Sarkar
5e53207edc refactor: update alembic migration revision ID and added some defaults for blocknote in file_processors.py file 2025-11-30 04:57:07 +05:30
Anish Sarkar
4e6b4cc57d fixed: cleaned up code and added import in init 2025-11-30 04:25:27 +05:30
Anish Sarkar
e419702ebd fix: run ruff formatter to fix code quality 2025-11-30 04:15:38 +05:30
Anish Sarkar
b98c312fb1 Merge remote-tracking branch 'upstream/main' into feature/blocknote-editor 2025-11-30 04:10:49 +05:30
Anish Sarkar
f8e4926969 feat: Implement document saving with reindexing
- Updated the document saving endpoint to trigger reindexing after saving.
- Introduced a new Celery task for reindexing documents.
- Refactored the editor page to reflect the changes in the API endpoint and method.
2025-11-30 04:08:12 +05:30
Anish Sarkar
91bc344b56 feat: Added celery tasks to populate blocknote_document for existing documents 2025-11-30 03:49:43 +05:30
DESKTOP-RTLN3BA\$punk
07234d7849 ruff format 2025-11-29 00:16:41 -08:00
DESKTOP-RTLN3BA\$punk
72236f65f7 refactor: remove UserSearchSpacePreference model and related relationships
- Deleted UserSearchSpacePreference class and its relationships from SearchSpace and User models.
- Cleaned up unused code to streamline database interactions.
2025-11-29 00:12:30 -08:00
DESKTOP-RTLN3BA\$punk
e9d32c3516 feat: Implement Role-Based Access Control (RBAC) for search space resources.
-Introduce granular permissions for documents, chats, podcasts, and logs.
- Update routes to enforce permission checks for creating, reading, updating, and deleting resources. - Refactor user and search space interactions to align with RBAC model, removing ownership checks in favor of permission validation.
2025-11-27 22:45:04 -08:00
DESKTOP-RTLN3BA\$punk
d1c9c533b5 fix: crawled url search 2025-11-26 14:53:44 -08:00
DESKTOP-RTLN3BA\$punk
0b1ca97acf refactor(webcrawler): update scraping logic to use v2 API and improve error handling 2025-11-26 14:30:08 -08:00
DESKTOP-RTLN3BA\$punk
ab6ea7e0ab feat(UI): reorganized connectors 2025-11-26 13:44:38 -08:00
DESKTOP-RTLN3BA\$punk
8f30cfd69a chore(lint): ruff checks 2025-11-26 13:22:31 -08:00
Anish Sarkar
289b4de52d refactor: comment out unused blocknote converter imports in editor_routes.py 2025-11-23 16:47:09 +05:30