Commit graph

158 commits

Author SHA1 Message Date
Anish Sarkar
2bc6a0c3bc chore: ran linting 2026-03-22 00:43:53 +05:30
Anish Sarkar
de8841fb86 chore: ran linting 2026-03-21 13:20:13 +05:30
Anish Sarkar
aaf34800e6 feat: enhance legacy document migration for Google connectors
- Implemented fallback logic in Google Calendar, Drive, and Gmail indexers to handle legacy Composio document types, ensuring smooth migration to native types.
- Updated document indexing functions to check for existing documents using both primary and legacy hashes, improving data integrity during indexing.
2026-03-20 03:39:05 +05:30
Anish Sarkar
8e7cda31c5 feat: update Google indexing functions to track skipped messages
- Modified the indexing functions for Google Calendar and Gmail to return the count of skipped messages alongside indexed messages, enhancing performance tracking.
- Updated related tests to accommodate the new return values, ensuring comprehensive coverage of the indexing process.
- Improved error handling to maintain consistency in returned values across different indexing functions.
2026-03-19 20:56:40 +05:30
Anish Sarkar
e9485ab2df feat: update Google Drive indexing to include skipped file tracking 2026-03-19 20:27:50 +05:30
Anish Sarkar
eac4cb6075 feat: enhance Google Drive indexing to track skipped files
- Updated the indexing function to return the count of skipped files alongside indexed files, improving tracking of indexing performance.
- Added logic to accumulate skipped file counts during the indexing process, providing better insights into potential issues.
- Enhanced notification updates to include skipped file counts, ensuring comprehensive progress reporting for users.
2026-03-19 20:27:36 +05:30
Anish Sarkar
2390bd7d26 feat: enhance Google Drive authentication error handling
- Improved error handling for Google Drive indexing and listing operations to manage authentication failures more effectively.
- Added logic to mark connectors as 'auth_expired' when a 401 error or invalid credentials are detected, prompting users to re-authenticate.
- Updated error messages to provide clearer guidance on authentication issues, ensuring a better user experience.
2026-03-19 18:24:41 +05:30
Anish Sarkar
83152e8e7e refactor: unify all 3 google Composio and non-Composio connector types and pipelines keeping same credential adapters 2026-03-19 05:08:21 +05:30
Anish Sarkar
ac0f2fa2eb chore: ran linting 2026-03-17 04:40:46 +05:30
DESKTOP-RTLN3BA\$punk
2b33dfe728 refactor: update safe_set_chunks function to be asynchronous and modify all connector and document processor files to use the new async implementation 2026-03-15 00:44:27 -07:00
DESKTOP-RTLN3BA\$punk
e9892c8fe9 feat: added configable summary calculation and various improvements
- Replaced direct embedding calls with a utility function across various components to streamline embedding logic.
- Added enable_summary flag to several models and routes to control summary generation behavior.
2026-02-26 18:24:57 -08:00
CREDO23
7d1bd1fab4 Implement KB sync after Notion page updates with block ID verification
- Add NotionKBSyncService for immediate KB updates after page changes
- Implement block ID verification to ensure content freshness
- Refactor duplicate block processing logic to shared utils
- Add user-friendly status messages
- Include debug logging for troubleshooting
2026-02-17 20:30:12 +02:00
Rohan Verma
26fd61fcbb
Merge pull request #796 from AnishSarkar22/feat/sur-149-batch-index
impr: batch index for messaging connectors & some fixes
2026-02-09 15:00:16 -08:00
DESKTOP-RTLN3BA\$punk
17b7348f61 feat: fixed and improved search and background task management. 2026-02-09 14:03:56 -08:00
Anish Sarkar
20ab128b05 feat: implement batch indexing for Microsoft Teams messages to improve efficiency and conversational context 2026-02-09 14:31:22 +05:30
Anish Sarkar
e2dd80c604 chore: ran linting 2026-02-08 12:43:31 +05:30
Anish Sarkar
7cede99d29 feat: implement batch indexing for Slack messages to enhance efficiency and conversational context 2026-02-07 18:30:06 +05:30
Anish Sarkar
98870a9f9a feat: implement batch indexing for Discord messages to improve efficiency and context 2026-02-07 18:26:29 +05:30
Anish Sarkar
0fdd194d92 Merge remote-tracking branch 'upstream/dev' into fix/documents 2026-02-06 12:13:26 +05:30
DESKTOP-RTLN3BA\$punk
1511c26ef5 feat: add residential proxy configuration for web crawling and YouTube transcript fetching 2026-02-05 20:44:13 -08:00
Anish Sarkar
aa66928154 chore: ran linting 2026-02-06 05:35:15 +05:30
Anish Sarkar
cc1e796c12 feat: implement two-phase document indexing for webcrawler and YouTube video processors with real-time status updates 2026-02-06 04:54:50 +05:30
Anish Sarkar
629f6f9cf5 feat: implement two-phase document indexing for Obsidian and Circleback connectors with real-time status updates 2026-02-06 04:35:13 +05:30
Anish Sarkar
0f61a249c0 feat: implement two-phase document indexing for BookStack, Elasticsearch, and Luma connectors with real-time status updates 2026-02-06 04:31:55 +05:30
Anish Sarkar
bfa3be655e feat: implement two-phase document indexing for ClickUp and GitHub connectors with real-time status updates 2026-02-06 04:06:14 +05:30
Anish Sarkar
1d870e45a4 feat: implement two-phase document indexing for Confluence and Jira connectors with real-time status updates 2026-02-06 03:54:24 +05:30
Anish Sarkar
0249ea20a5 feat: implement two-phase document indexing for Discord and Teams connectors with real-time status updates 2026-02-06 03:42:03 +05:30
Anish Sarkar
2077344934 feat: implement two-phase document indexing for Linear and Slack connectors with real-time status updates 2026-02-06 02:59:21 +05:30
Anish Sarkar
c12401c1e8 feat: implement two-phase document indexing across Google connectors with real-time status updates 2026-02-06 02:24:35 +05:30
Manoj Aggarwal
e6c0fabd0a Merge branch 'dev' into bugs_prod 2026-02-05 10:53:16 -08:00
Anish Sarkar
3bbac0d4ea feat: implement two-phase document indexing for Airtable and Notion connectors with real-time status updates 2026-02-06 00:12:48 +05:30
Anish Sarkar
aef59d04eb feat: add document status management with JSONB column for processing states in documents 2026-02-05 21:59:31 +05:30
Manoj Aggarwal
33165830e5 add parse date flexible 2026-02-04 13:18:33 -08:00
Anish Sarkar
04884caeef refactor: simplify document title assignment across various connectors by removing prefix formatting 2026-02-05 02:30:20 +05:30
Manoj Aggarwal
48e646607b Fix google calendar and notion erros 2026-02-02 12:07:53 -08:00
Anish Sarkar
bf08982029 feat: add connector_id to documents for source tracking and implement connector deletion task 2026-02-02 16:23:26 +05:30
Anish Sarkar
e0ade20e68 feat: add created_by_id column to documents for ownership tracking and update related connectors 2026-02-02 12:32:24 +05:30
Anish Sarkar
cf339ff350 chore: ran backend linting 2026-02-02 00:19:19 +05:30
Anish Sarkar
085653d3e3 chore: ran frontend and backend linting 2026-02-01 22:54:25 +05:30
Anish Sarkar
024a683b4f feat: add heartbeat callback support for long-running indexing tasks and implement stale notification cleanup task 2026-02-01 02:17:06 +05:30
Anish Sarkar
9771a88380 fix: refine date handling in Google Calendar connector to ensure accurate same-day queries 2026-01-30 20:51:03 +05:30
Anish Sarkar
4526b656a4 fix: update default date range for Google Calendar events and improve query parameter handling 2026-01-30 19:55:48 +05:30
DESKTOP-RTLN3BA\$punk
d39bf3510f chore: linting 2026-01-28 22:20:23 -08:00
Anish Sarkar
1658724fb2 Merge remote-tracking branch 'upstream/dev' into fix/notion-connector 2026-01-29 10:45:31 +05:30
Anish Sarkar
c6d25ed7d8 feat(backend): Add legacy token handling in NotionHistoryConnector and log warnings for legacy usage in indexing 2026-01-28 22:53:34 +05:30
CREDO23
b20fbaca4b fix: skip webcrawler indexing gracefully when no URLs configured 2026-01-28 17:54:46 +02:00
Anish Sarkar
b3f553802c fix(backend): Update Notion page indexing log message to clarify sharing requirements and adjust return value for no pages found 2026-01-28 18:58:57 +05:30
CREDO23
4f7ed8439f fix(backend): Use calculate_date_range for Gmail indexer
Gmail indexer was using a hardcoded 30-day default instead of respecting
last_indexed_at like other connectors. Now uses calculate_date_range()
for consistent behavior (last_indexed_at → now, or 365 days for first run).
2026-01-28 15:20:07 +02:00
Anish Sarkar
33316fa6db feat(backend): Add retry logic for Notion API calls with user notifications on rate limits and errors 2026-01-28 18:36:42 +05:30
CREDO23
a9d393327d fix(backend): Add duplicate content_hash check to connector indexers
Prevent UniqueViolationError on ix_documents_content_hash constraint by
adding check_duplicate_document_by_hash() before inserting new documents
in 15 connector indexers that were missing this check.

Affected: clickup, luma, linear, jira, google_gmail, confluence,
bookstack, github, webcrawler, teams, slack, notion, discord,
airtable, obsidian indexers.
2026-01-28 14:51:54 +02:00