Move the lower-level runtime/infra modules out of multi_agent_chat/shared/
(they were never used by subagents, so they failed the shared-by-all-siblings
rule) and unify them with the already-relocated checkpointer:
agents/runtime/ -> agents/chat/runtime/
mac/shared/errors.py -> chat/runtime/errors.py
mac/shared/llm_config.py -> chat/runtime/llm_config.py
mac/shared/prompt_caching.py -> chat/runtime/prompt_caching.py
mac/shared/mention_resolver.py -> chat/runtime/mention_resolver.py
mac/shared/path_resolver.py -> chat/runtime/path_resolver.py
These sit below the agent packages: the boundary + agent factory + shared
middleware depend on them, and they import no agent code (acyclic).
Recursive shared-folder rule: a shared/ must be shared by ALL siblings at its
level. The kernel (context, compaction, retry_after, web_search) was shared by
only 2 of the agents -- anonymous_chat + multi_agent_chat -- never by podcaster
or video_presentation. Those 2 are the "chat" category, so their shared code
belongs in that category's shared/, not the top-level one.
app/agents/anonymous_chat/ -> app/agents/chat/anonymous_chat/
app/agents/multi_agent_chat/ -> app/agents/chat/multi_agent_chat/
app/agents/shared/ -> app/agents/chat/shared/ (anon<->mac kernel)
Top-level app/agents/shared/ is gone: nothing was shared across all three
categories (chat / podcaster / video_presentation).
~289 import sites rewritten (app.agents.{anonymous_chat,multi_agent_chat,shared}
-> app.agents.chat.*); all moves are git renames (history preserved).
app/agents/ now: chat/, podcaster/, video_presentation/, runtime/.
app/agents/shared/ is a sibling of anonymous_chat/podcaster/multi_agent_chat/
video_presentation, so it should only hold code shared across 2+ of those
agents. In practice podcaster and video_presentation import nothing from it,
and anonymous_chat needs only context + compaction + retry_after + web_search.
Everything else was multi_agent_chat-only (the boundary just passes through).
Move the multi_agent_chat-only cluster into multi_agent_chat/shared/ (files
moved verbatim via git rename; ~116 import sites rewritten):
errors, feature_flags, filesystem_selection, path_resolver, prompt_caching,
sandbox, llm_config, mention_resolver
middleware/busy_mutex, middleware/kb_persistence
busy_mutex/llm_config/mention_resolver are boundary-only but import the moved
modules, so they were folded in to avoid a backwards shared -> multi_agent_chat
dependency. main_agent builders now import the impls directly; the shared
middleware barrel keeps only the genuinely-shared compaction + retry_after.
Also delete the dead leftover shared/plugins and shared/skills dirs (live
copies already live under main_agent/).
Remaining in app/agents/shared/: context, system_prompt(+prompts), checkpointer,
middleware/{compaction,retry_after,dedup_tool_calls}, tools/. checkpointer and
system_prompt are boundary-only infra pending a dedicated home decision.
Relocate three leaf filesystem-cluster modules to the shared kernel and flip
all 38 importers. No re-export shims needed (no frozen single-agent importer).
This also resolves the pre-existing shared->new_chat back-edge from
shared/receipt_command.py onto filesystem_state.
filesystem_backends is intentionally deferred to slice 5: it depends on
new_chat middleware (kb_postgres_backend, multi_root_local_folder_backend)
that have not yet moved, so relocating it now would create a shared->new_chat edge.
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
- Introduced a `ProcessingMode` enum to differentiate between basic and premium processing modes.
- Updated `EtlRequest` to include a `processing_mode` field, defaulting to basic.
- Enhanced ETL pipeline services to utilize the selected processing mode for Azure Document Intelligence and LlamaCloud parsing.
- Modified various routes and services to handle processing mode, affecting document upload and indexing tasks.
- Improved error handling and logging to include processing mode details.
- Added tests to validate processing mode functionality and its impact on ETL operations.
- Updated maximum file size limit to 500 MB per file.
- Removed restrictions on the number of files per upload and total upload size.
- Enhanced handling of user-mentioning documents in the knowledge base search middleware.
- Improved document reading and processing logic to accommodate new features and optimizations.
- Fix timestamp conversion: String(epochMs) → new Date(epochMs).toISOString()
in use-messages-sync, use-comments-sync, use-documents, use-inbox.
Without this, date comparisons (isEdited, cutoff filters) would fail.
- Fix updated_at: undefined → null in use-inbox to match InboxItem type
- Fix ZeroProvider: skip Zero connection for unauthenticated users
- Clean 30+ stale "Electric SQL" comments in backend Python code
- Increased maximum file upload limit from 10 to 50 to improve user experience.
- Implemented batch processing for document uploads to avoid proxy timeouts, splitting files into manageable chunks.
- Enhanced garbage collection in chat streaming functions to prevent memory leaks and improve performance.
- Added memory delta tracking in system snapshots for better monitoring of resource usage.
- Updated LLM router and service configurations to prevent unbounded internal accumulation and improve efficiency.
- Replaced direct embedding calls with a utility function across various components to streamline embedding logic.
- Added enable_summary flag to several models and routes to control summary generation behavior.
- Introduced a TaskDispatcher abstraction to decouple the upload endpoint from Celery, allowing for easier testing with synchronous implementations.
- Updated the create_documents_file_upload function to utilize the new dispatcher for task management.
- Removed direct Celery task imports from the upload function, enhancing modularity.
- Added integration tests for document upload, including page limit enforcement and file size restrictions.
- Added constants for maximum files per upload, per-file size, and total upload size.
- Enhanced document upload route to validate file counts and sizes, returning appropriate HTTP errors.
- Introduced end-to-end tests for upload limits and page limit enforcement, ensuring correct behavior under various scenarios.
- Updated test helpers to support notification retrieval for page limit exceeded scenarios.
- Added support for multiple queues in Celery worker configuration.
- Modified connector deletion to handle documents inline instead of using a background task.
- Updated response messages for document creation and connector deletion to reflect new processing status.
- Removed the obsolete connector deletion Celery task file.
- Introduced a new endpoint for batch document status retrieval, allowing users to check the status of multiple documents in a search space.
- Enhanced the document upload process to return duplicate document IDs and improved response structure.
- Updated schemas to include new response models for document status.
- Removed unused attachment processing code from chat routes and UI components to streamline functionality.
- Introduced pg_trgm extension and GIN trigram indexes for efficient document title searches, enhancing performance for mention picker functionality.
- Implemented a new API endpoint for lightweight document title searches, returning only essential fields.
- Updated frontend components to utilize the new title search feature with throttling for improved user experience.
- Added necessary schemas and types for the new search functionality.
