Promote the filesystem mode contracts (FilesystemMode, FilesystemSelection,
ClientPlatform, LocalFilesystemMount) out of `new_chat` into the cross-agent
`app/agents/shared` kernel.
Pure leaf consumed across the whole multi-agent filesystem middleware/tool tree,
the chat flows/monolith, routes and tests. git mv (content unchanged) + flipped
all ~48 importers. A re-export shim remains at new_chat/filesystem_selection.py
only for the not-yet-retired single-agent (chat_deepagent).
Also updated the stream parity test's annotation normalizer to strip the new
app.agents.shared.filesystem_selection. prefix (the dataclasses' __module__
changed with the move), keeping monolith<->flows signature parity intact.
Behavior-preserving: only import paths change. 1326 tests green.
Promote the agent feature-flag resolver (AgentFeatureFlags / get_flags) out of
`new_chat` into the cross-agent `app/agents/shared` kernel.
feature_flags is a pure leaf consumed across the multi-agent middleware stack,
the chat routes, and tests. Moved it via git mv (content unchanged) and flipped
all 37 importers to app.agents.shared.feature_flags. A thin re-export shim
remains at new_chat/feature_flags.py only for the not-yet-retired single-agent
(chat_deepagent); it goes away with the single-agent deletion.
Behavior-preserving: only import paths change. 1243 tests green.
Continue promoting the shared agent toolkit out of `new_chat` into the
cross-agent `app/agents/shared` kernel.
- state_reducers.py: clean move (no single-agent importer); all 7 importers
flipped to app.agents.shared.state_reducers.
- context.py: moved to app.agents.shared.context; flipped the multi-agent,
app, automations, chat-flows and monolith importers. A thin re-export shim
remains at new_chat/context.py because the not-yet-retired single-agent
(chat_deepagent) and the new_chat package __init__ still import it; the shim
goes away with the single-agent deletion.
- Updated the stream parity test's annotation normalizer to strip the new
app.agents.shared.context. prefix (SurfSenseContextSchema.__module__ changed
with the move), keeping monolith<->flows signature parity intact.
Behavior-preserving: definitions unchanged; only import paths move. 1219 tests green.
First slice of promoting the shared agent toolkit out of the misnamed
`new_chat` package into the cross-agent `app/agents/shared` kernel.
`errors.py` is a leaf module (no intra-package deps) consumed by the
multi-agent chat, the chat streaming flows/monolith, and tests — i.e. it is
shared infrastructure, not single-agent code. Moved it verbatim to
`app.agents.shared.errors` and flipped all 12 importers. No re-export shim
remains since zero importers needed it.
Behavior-preserving: identical class/enum definitions; only the import path
changes. 1208 agent + chat-task tests green.
The multi-agent factory reached into the single-agent factory module
(chat_deepagent) for `_map_connectors_to_searchable_types`. Move this
agent-agnostic helper (and its two lookup tables) into a dedicated
`connector_searchable_types` module and point both factories at it.
Behavior-preserving: the function body is unchanged; only its home and
visibility (now public `map_connectors_to_searchable_types`) change. This
removes the cross-dependency on the dying single-agent module so it can be
retired later without breaking the multi-agent path.
- Removed the eligibility gate for model selection in the automation creation process, allowing users to choose models directly in the builder.
- Updated the `AutomationBuilderForm` to incorporate model selection logic, ensuring that selected models are validated and preserved during automation creation and editing.
- Simplified the `AutomationsContent` and `AutomationNewContent` components by eliminating unnecessary eligibility checks and alerts.
- Enhanced the user experience by integrating model selection directly into the automation approval process, ensuring that only billable models are used.
- Refactored related tests to cover new model selection behavior and ensure proper validation of user-selected models.
- Added model eligibility checks to ensure automations can only use billable models (premium or BYOK).
- Introduced new API endpoint to report model eligibility status for search spaces.
- Updated frontend components to display eligibility alerts and disable creation options when models are not billable.
- Enhanced automation creation forms to reflect model eligibility, preventing users from submitting invalid configurations.
- Implemented server-side logic to capture and preserve model preferences across automation edits, ensuring consistent behavior during execution.
- Deleted the `search_surfsense_docs` tool and its associated files, streamlining the agent's toolset.
- Updated various components and prompts to remove references to the now-removed tool, ensuring consistency across the codebase.
- Adjusted documentation to direct users to the SurfSense documentation link for product-related queries instead.
Closes the create loop in chat: the agent describes user intent → the
drafter sub-LLM produces an AutomationCreate JSON → this card surfaces
a structured preview → approve persists; reject cancels. Edits flow
through chat refinement (re-call with a refined intent), not in-card,
so the card stays simple and the multi-turn checkpointer carries the
context.
Tool UI (components/tool-ui/automation/):
- create-automation.tsx — entry dispatcher + ApprovalCard chrome
(pending/processing/complete/rejected via useHitlPhase) + SavedCard
(links to the detail page) + InvalidCard (lists drafter validation
issues) + ErrorCard (verbatim message). Rejection result is hidden
because the approval card itself shows the rejected phase inline.
- automation-draft-preview.tsx — structured preview body: name +
description + goal, triggers (humanised cron + tz + static-input
keys), plan steps (step_id → action), and a collapsible raw JSON
for power users.
Wiring:
- components/tool-ui/index.ts — re-export.
- features/chat-messages/timeline/tool-registry/registry.ts —
register create_automation → CreateAutomationToolUI (dynamic import,
same pattern as other connector tools).
- contracts/enums/toolIcons.tsx — Workflow icon + "Create automation"
display name so fallback chrome (and timeline headers) are honest.
Shared util:
- lib/automations/describe-cron.ts — lifted from the route slice's
lib/ folder since both the dashboard slice and the new approval card
now render schedule descriptions. Slice imports updated; the now-
empty slice lib/ folder is gone.
Backend prompt fragments:
- main_agent/system_prompt/.../create_automation/description.md and
the tool's docstring no longer promise in-card edits. They make the
refinement path explicit: if the user wants changes after seeing the
draft, they reply in chat and the agent calls the tool again with a
refined intent.
v1 deliberately excludes:
- In-card edit form / right-side edit panel — defer until we see real
demand. The chat refinement loop covers the common case.
- approve_always / persistent allow rules — automations are a single
artifact, not a repeated mutation, so the "trust this kind of call"
affordance doesn't apply.
Single tool exposed to the main agent. The main agent passes a natural-language
`intent`; a focused drafter sub-LLM turns it into a full AutomationCreate JSON;
that JSON is surfaced via request_approval (action_type "automation_create") so
the user can edit/approve it on a frontend card; on approval the tool persists
via AutomationService. Three phases, one tool call.
Scope split:
- main agent sees only `intent: str` (no schema knowledge leaks into the calling
graph) — prompt fragments scoped accordingly.
- drafter sub-LLM owns the schema + few-shot intent→JSON examples — lives in
the generating graph's prompt (tools/automation/prompt.py).
Files:
- main_agent/tools/automation/{create.py, prompt.py, __init__.py}: new tool
+ drafter system prompt with two few-shot intent→JSON examples.
- system_prompt/prompts/tools/create_automation/{description.md, example.md}:
intent-only guidance for the main agent.
- main_agent/tools/index.py: add create_automation to the main-agent allowlist.
- new_chat/tools/registry.py: deferred-import factory to break the
multi_agent_chat ↔ registry cycle; one ToolDefinition entry.
- Added new environment variables for controlling task execution limits, including `SURFSENSE_SUBAGENT_INVOKE_TIMEOUT_SECONDS`, `SURFSENSE_TASK_BATCH_CONCURRENCY`, and `SURFSENSE_TASK_BATCH_MAX_SIZE`.
- Updated documentation to reflect new batch processing capabilities for `task` calls, allowing for concurrent execution of multiple subagent tasks.
- Improved error handling and receipt generation for deliverables, ensuring consistent feedback on task status.
- Refactored middleware to incorporate search space ID for better task management.
The citations fix (cacb27e0) added a "Chunk citations in your prose"
section to system_prompt_desktop.md telling the KB subagent to always
leave `evidence.chunk_ids` null and emit no `[citation:...]` markers in
desktop mode, but left the pre-existing line declaring that
`chunk_ids` apply to `<priority_documents>` hits. The two rules
contradicted each other; the model picked one per turn.
Strike the stale conditional clause and point at the dedicated section
as the single source of truth. Matches the parallel line in
system_prompt_cloud.md and the already-consistent
system_prompt_readonly_desktop.md.
Resolves: surfsense_backend/app/agents/new_chat/middleware/memory_injection.py
- Took both imports: upstream moved MEMORY_HARD_LIMIT/SOFT_LIMIT to
app.services.memory; kept our perf-logger import for timing.
Pulls in upstream changes:
- Memory document feature (services/memory refactor, removal of
app.agents.new_chat.memory_extraction and background extraction in
stream_new_chat — agent now drives memory via update_memory tool).
- BACKEND_URL env refactor across web tool-ui/editor/chat/dashboard/lib.
- GitHub Actions backend test workflow + pre-commit biome bump.
- Token-display polish in MessageInfoDropdown; save_memory no-update
sentinel.
Verified: 1723 unit tests pass, ruff clean. No semantic regression in
stream_new_chat (their memory-extraction deletion and our preflight
removal touch different functions).
Adds an optional planner LLM role wired through KnowledgePriorityMiddleware
so KB query rewriting, date extraction, and recency classification run on a
cheap model (e.g. gpt-4o-mini, Haiku, Azure nano) instead of the user's
chat LLM. Operators opt in by setting is_planner: true on exactly one
global config; without it, behavior is unchanged.
Only MCP tools have a persistence target for 'approve_always' (the
connector's trusted-tools list); for native tools the decision lives
only in the in-memory runtime ruleset. Reflect that in the wire palette
so the FE can stay a pure renderer of allowed_decisions instead of
peeking at context.mcp_connector_id to decide whether to show the
'Always Allow' button.
The backend still accepts an 'approve_always' reply for any tool kind
(in-memory promotion is harmless), it just doesn't advertise it when
there's nowhere to persist.
Renames the SurfSense HITL extension decision-type from "always" to
"approve_always" so it sits in the same verb-first family as "approve",
"reject", and "edit". The Python constant is now SURFSENSE_DECISION_APPROVE_ALWAYS;
the wire value, the permission-domain decision_type, and the FE union members
all match (no wire/internal mismatch).
Both the multi_agent_chat permission middleware and the legacy new_chat one
accept the new wire value; the FE types.ts union is updated accordingly.
The "context.always" payload key is intentionally left untouched - it's the
patterns-to-promote field, semantically distinct from the decision type.
Until now an "Always Allow" reply only updated the in-memory runtime
ruleset, evaporating after the session ended. Persist it to the
existing connector.config['trusted_tools'] list so the next session's
fetch_user_allowlist_rulesets picks it up and the user is never asked
again for the same (connector, tool) pair.
- TrustedToolSaver + make_trusted_tool_saver(user_id) in
user_tool_allowlist: opens its own session via async_session_maker
per call, logs and swallows failures (in-memory promotion is the
canonical "always" path, durable persistence is opportunistic).
- PermissionMiddleware._process is now pure: returns
(state_update, list[_AlwaysPromotion]). aafter_model awaits the
saver for each promotion; after_model discards them. Promotions are
only emitted for tools whose metadata exposes mcp_connector_id, so
native tools and KB FS ops are correctly skipped.
- main_agent factory builds the saver once per turn and stashes it in
dependencies["trusted_tool_saver"]; pack_subagent and the KB
middleware stack forward it through build_permission_mw.
- Renamed pm._process(state, None) call sites in two existing tests to
pm.after_model(state, None) so they exercise the public hook
contract instead of the now-tuple-returning private method.
The FE permission card needs mcp_connector_id, mcp_server, and
tool_description in the interrupt context to render "Always Allow"
against the right connected account. Thread the tool through the
ask pipeline:
- pack_subagent → build_permission_mw(tools=...) → PermissionMiddleware
(tools_by_name) → request_permission_decision(tool=...) →
build_permission_ask_payload(tool=...) projects card fields out of
BaseTool.
- mcp_tool.py: stdio path now stashes mcp_connector_id in metadata for
parity with the HTTP path.
