Matt Van Horn
456dd7417c
fix(connectors): refresh Redis heartbeat during long Phase 1 indexing
...
Closes #1295
The connector indexing route's `_run_indexing_with_notifications` set the
Redis heartbeat key once at the start of indexing and relied on
`on_heartbeat_callback` (only fired in Phase 2 per-document loops) to
refresh it. The GitHub connector's Phase 1 runs `gitingest` as a blocking
subprocess via `asyncio.to_thread`, so for any repo larger than the
2-minute TTL, the key expires before Phase 2 starts. The
`cleanup_stale_indexing_notifications_task` then marks the document as
failed with the misleading "Sync was interrupted unexpectedly. Please
retry." message — even though the indexing thread is still running and
gitingest's own subprocess timeout is 900 seconds.
Add a background asyncio coroutine that refreshes the Redis key every
60 seconds for the duration of the indexing call. Same pattern already
in use at app/tasks/celery_tasks/document_tasks.py:_run_heartbeat_loop,
just adapted to use the route's get_heartbeat_redis_client() and
_get_heartbeat_key() helpers.
Cancellation runs in the `finally` block BEFORE the heartbeat-key
delete so the loop cannot race and re-create the key after we have
deleted it. The new `HEARTBEAT_REFRESH_INTERVAL = 60` constant mirrors
the celery task module's value.
2026-04-26 02:46:43 -07:00
CREDO23
9396ee9c85
fix: delete broken sandbox before creating replacement to prevent orphan accumulation
2026-04-13 21:57:06 +02:00
CREDO23
facfb1f7e7
disable file sync to sandbox, LLM handles data extraction itself
2026-04-13 20:59:31 +02:00
CREDO23
5ea2732bf2
make execute_code a Python tool: auto-wrap, print() nudge on empty output
2026-04-13 20:53:17 +02:00
CREDO23
5719a794a0
refine execute_code prompts: tool description + XML workflow guidance
2026-04-13 19:51:34 +02:00
CREDO23
bb41c09eef
clean execute_code tool description, remove sandbox internals
2026-04-13 18:03:22 +02:00
CREDO23
c9e3b1a08a
fix KB search skipping when system message follows user message
2026-04-13 18:03:16 +02:00
CREDO23
823e3ebb1d
pass thread_id to filesystem middleware
2026-04-13 16:23:53 +02:00
CREDO23
a7e70020b1
add execute_code tool with sandbox integration
2026-04-13 16:23:47 +02:00
CREDO23
9e8ea1fd1c
harden sandbox: security params, file sync, path traversal fix
2026-04-13 16:23:40 +02:00
CREDO23
c30cc08771
Merge upstream/dev into feat/kb-export-and-folder-upload
2026-04-11 10:28:40 +02:00
Rohan Verma
7ec2973223
Merge pull request #1214 from AnishSarkar22/fix/azure-document-intelligence
...
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
fix: update Azure Document Intelligence to use `prebuilt-layout`
2026-04-10 14:37:55 -07:00
Anish Sarkar
bb5b90e5bd
fix: update Azure Document Intelligence parser to use prebuilt-layout model
2026-04-10 20:40:54 +05:30
CREDO23
a95bf58c8f
Make Vision LLM opt-in for uploads and connectors
2026-04-10 16:45:51 +02:00
Anish Sarkar
c6730c5551
feat: implement download functionality for different OS in SidebarUserProfile and DownloadButton components, enhance user experience with localized download messages
2026-04-10 19:53:13 +05:30
Anish Sarkar
0925f89b7c
fix: guard display_name.split() against whitespace-only IndexError (recurseml feedback)
2026-04-10 15:29:56 +05:30
Anish Sarkar
8c9440998a
chore: ran linting
2026-04-10 14:20:11 +05:30
Anish Sarkar
8d7b5bfe36
feat: enhance memory management protocols to enforce structured headings, utilize user first names in entries, and improve clarity in memory update rules
2026-04-10 04:51:56 +05:30
Anish Sarkar
b8e1c9801b
feat: update memory extraction and management protocols to enforce structured bullet formats, utilize user first names, and enhance validation for team and user memory entries
2026-04-10 04:36:51 +05:30
Anish Sarkar
c06b82647d
feat: refine team memory extraction decision policy to prioritize durable context and clarify criteria for updates, enhancing memory relevance and management
2026-04-10 02:02:14 +05:30
Anish Sarkar
a0883d2ab6
feat: implement team memory extraction and validation mechanisms, enhancing memory management by enforcing scope restrictions and improving memory persistence for shared threads
2026-04-10 01:54:00 +05:30
CREDO23
0aefcbd504
Remove vision LLM from desktop folder watcher
2026-04-09 22:06:06 +02:00
Anish Sarkar
33626d4f91
chore: ran linting
2026-04-10 00:31:43 +05:30
Anish Sarkar
84fc72e596
feat: implement background memory extraction and editing capabilities for user and team memory management, enhancing long-term memory persistence and user interaction
2026-04-10 00:21:55 +05:30
Anish Sarkar
cd72fa9a48
refactor: remove memory extraction functionality and update memory management protocols to ensure immediate updates for user and team interactions
2026-04-09 23:30:17 +05:30
CREDO23
4ccdd80e26
Harden vision LLM fallback, folder upload validation, and export memory
2026-04-09 16:14:53 +02:00
CREDO23
e164fe0612
Fix misleading log when vision LLM fails vs not provided
2026-04-09 15:29:39 +02:00
CREDO23
ff2a9c77f9
Pass vision_llm in legacy process_file_in_background path
2026-04-09 15:28:21 +02:00
CREDO23
55661bcde6
Replace mimetypes fallback with explicit extension-to-MIME mapping
2026-04-09 15:21:32 +02:00
CREDO23
71db53fc55
Add 5MB file size guard before base64 encoding for vision LLM
2026-04-09 15:17:08 +02:00
CREDO23
d6c4fb8938
Add try/except fallback in _extract_image for vision LLM failures
2026-04-09 15:11:24 +02:00
CREDO23
caaec2e0a7
Simplify vision LLM image description prompt
2026-04-09 14:56:18 +02:00
CREDO23
afd3c2cde2
Pass vision LLM through local folder indexer call chain
2026-04-09 14:50:24 +02:00
Anish Sarkar
f38ea77940
chore: ran linting
2026-04-09 18:10:34 +05:30
Anish Sarkar
b8091114b5
feat: drop legacy memory tables and update memory management tools to streamline user and shared memory handling
2026-04-09 18:05:30 +05:30
CREDO23
7e90a8ed3c
Route uploaded images to vision LLM with document-parser fallback
2026-04-09 14:33:33 +02:00
Anish Sarkar
7caf3b25e2
feat: implement background memory extraction for user and team interactions, enhancing long-term memory management in the SurfSense agent
2026-04-09 17:43:52 +05:30
Anish Sarkar
a335f7621a
refactor: update memory management tools to enforce character limits, enhance pinned section handling, and improve user feedback in MemoryContent and TeamMemoryManager components
2026-04-09 17:24:31 +05:30
CREDO23
1af5725bd1
fix: track dedup'd filename in used_paths to prevent collisions
2026-04-09 13:51:32 +02:00
CREDO23
b5f6e44fc3
security: sanitize folder names in ZIP export paths
2026-04-09 13:39:36 +02:00
CREDO23
e7107b751d
fix: strip folder prefix from filename in folder upload
2026-04-09 13:01:23 +02:00
CREDO23
47f1d7e373
feat: implement KB export as ZIP with service layer
2026-04-09 12:17:43 +02:00
CREDO23
473eece896
feat: add export route skeleton
2026-04-09 12:10:37 +02:00
Anish Sarkar
ab3cb0e1c5
Merge remote-tracking branch 'upstream/dev' into refactor/persistent-memory
2026-04-09 15:33:23 +05:30
Anish Sarkar
d1c6caddba
refactor: enhance MemoryInjectionMiddleware to auto-persist user memory seed and improve button styles in MemoryContent and TeamMemoryManager components
2026-04-09 13:57:54 +05:30
Anish Sarkar
1f59382ab0
refactor: update knowledge base policies and enhance memory loading logic in MemoryInjectionMiddleware
2026-04-09 03:36:45 +05:30
Anish Sarkar
71e88156f4
feat: enhance MemoryInjectionMiddleware to include user display name and seed memory with first name
2026-04-09 00:22:30 +05:30
Anish Sarkar
dfa6005af5
feat: implement update_memory tool and routes for user memory management
2026-04-08 23:54:29 +05:30
Anish Sarkar
dec381d87e
feat: add shared_memory_md field and enforce character limit in search space updates
2026-04-08 23:37:23 +05:30
Anish Sarkar
6fc941e4c5
refactor: rename and consolidate memory tools to update_memory
2026-04-08 23:29:26 +05:30
