Commit graph

2128 commits

Author SHA1 Message Date
CREDO23
7d866a2279 refactor(agents): sink sandbox.py into filesystem subsystem
shared/sandbox.py was used only by the filesystem middleware/tools (and the
boundary) -- never by main_agent or subagents as shared code. Move it next to
its only agent-side consumer:

  multi_agent_chat/shared/sandbox.py
  -> multi_agent_chat/shared/middleware/filesystem/sandbox.py
2026-06-05 13:15:57 +02:00
CREDO23
24b62a63b4 refactor(agents): introduce chat/ category; dissolve top-level agents/shared
Recursive shared-folder rule: a shared/ must be shared by ALL siblings at its
level. The kernel (context, compaction, retry_after, web_search) was shared by
only 2 of the agents -- anonymous_chat + multi_agent_chat -- never by podcaster
or video_presentation. Those 2 are the "chat" category, so their shared code
belongs in that category's shared/, not the top-level one.

  app/agents/anonymous_chat/   -> app/agents/chat/anonymous_chat/
  app/agents/multi_agent_chat/ -> app/agents/chat/multi_agent_chat/
  app/agents/shared/           -> app/agents/chat/shared/   (anon<->mac kernel)

Top-level app/agents/shared/ is gone: nothing was shared across all three
categories (chat / podcaster / video_presentation).

~289 import sites rewritten (app.agents.{anonymous_chat,multi_agent_chat,shared}
-> app.agents.chat.*); all moves are git renames (history preserved).
app/agents/ now: chat/, podcaster/, video_presentation/, runtime/.
2026-06-05 12:54:02 +02:00
CREDO23
d59bb2b5aa refactor(agents): evict mac-only tools/middleware from shared kernel
These were never shared with anonymous_chat (nor podcaster/video_presentation)
-- only multi_agent_chat (subagents/main agent) and the boundary use them:

  shared/tools/mcp/             -> multi_agent_chat/shared/tools/mcp/
  shared/tools/hitl.py          -> multi_agent_chat/shared/tools/hitl.py
  shared/tools/catalog.py       -> multi_agent_chat/shared/tools/catalog.py
  shared/middleware/dedup_tool_calls.py
                                -> multi_agent_chat/shared/middleware/dedup_tool_calls.py

app/agents/shared/ now holds only the genuine anon<->mac kernel:
context, middleware/{compaction,retry_after}, tools/web_search.
2026-06-05 12:50:46 +02:00
CREDO23
b7ea829371 refactor(agents): relocate boundary-only infra out of shared/
Neither module is imported by any sibling agent package, so neither belongs in
the cross-agent shared kernel:

- checkpointer.py -> app/agents/runtime/checkpointer.py
  LangGraph Postgres checkpoint saver. It's cross-agent *runtime infra* wired by
  the boundary (app lifespan + anonymous_chat & multi_agent_chat flows), not
  agent code. New app/agents/runtime/ layer holds boundary-wired agent infra.

- shared/system_prompt.py + shared/prompts/ -> app/prompts/
  The legacy single-agent prompt composer. The live agents don't use it
  (main_agent has its own system_prompt/ builder; anonymous_chat builds inline);
  its only consumer is new_llm_config_routes for displaying default instructions.
  Moved to the existing non-agent prompt domain:
    system_prompt.py        -> app/prompts/default_system_instructions.py
    prompts/                 -> app/prompts/system_prompt_composer/

app/agents/shared/ now contains only genuinely cross-agent code: context,
middleware/{compaction,retry_after,dedup_tool_calls}, tools/.

NOTE: get_default_system_instructions() (LLM-config UI) composes from the legacy
library, which differs from what the live agents actually run -- pre-existing
latent staleness, not changed here.
2026-06-05 12:36:44 +02:00
CREDO23
82c5dc5b02 refactor(agents): move mac-only modules out of the cross-agent shared kernel
app/agents/shared/ is a sibling of anonymous_chat/podcaster/multi_agent_chat/
video_presentation, so it should only hold code shared across 2+ of those
agents. In practice podcaster and video_presentation import nothing from it,
and anonymous_chat needs only context + compaction + retry_after + web_search.
Everything else was multi_agent_chat-only (the boundary just passes through).

Move the multi_agent_chat-only cluster into multi_agent_chat/shared/ (files
moved verbatim via git rename; ~116 import sites rewritten):

  errors, feature_flags, filesystem_selection, path_resolver, prompt_caching,
  sandbox, llm_config, mention_resolver
  middleware/busy_mutex, middleware/kb_persistence

busy_mutex/llm_config/mention_resolver are boundary-only but import the moved
modules, so they were folded in to avoid a backwards shared -> multi_agent_chat
dependency. main_agent builders now import the impls directly; the shared
middleware barrel keeps only the genuinely-shared compaction + retry_after.

Also delete the dead leftover shared/plugins and shared/skills dirs (live
copies already live under main_agent/).

Remaining in app/agents/shared/: context, system_prompt(+prompts), checkpointer,
middleware/{compaction,retry_after,dedup_tool_calls}, tools/. checkpointer and
system_prompt are boundary-only infra pending a dedicated home decision.
2026-06-05 12:30:15 +02:00
CREDO23
c0c4f57f5d refactor(agents): delete dead PermissionMiddleware twin in shared kernel
app/agents/shared/middleware/permission.py was an older, monolithic
PermissionMiddleware superseded by the modular permissions/ package under
multi_agent_chat/shared/middleware/ (core + evaluation + ask/ + factory).
Production wires only the package (main_agent stack + every subagent
builder); the kernel file was reachable only through the shared barrel
re-export (itself unused) and two tests pinned to its dead internals
(_raise_interrupt, _normalize_permission_decision, old after_model shape).

- delete app/agents/shared/middleware/permission.py
- drop PermissionMiddleware from the shared middleware barrel
- delete test_permission_middleware.py (covered the dead impl only; live
  behavior is covered by tests/.../middleware/shared/permissions/*)
- test_desktop_safety_rules.py: keep the ruleset-level regression tests,
  drop the dead import + TestPermissionMiddlewareIntegration class
2026-06-05 12:10:08 +02:00
CREDO23
8ae190a11d refactor(agents): move MAC middleware impls out of shared kernel
knowledge_search, memory_injection and scoped_model_fallback no longer
belong in the cross-agent kernel (app/agents/shared/middleware): they are
consumed only inside multi_agent_chat. Relocate each impl next to the
builder that uses it:

- knowledge_search.py -> multi_agent_chat/shared/middleware/ (genuinely
  shared: its _render_priority_message feeds kb_context_projection, used by
  both the main agent and the KB subagent)
- memory_injection.py -> multi_agent_chat/shared/middleware/ (beside its
  memory.py builder)
- scoped_model_fallback.py -> multi_agent_chat/shared/middleware/resilience/
  (beside fallback.py/bundle.py)

Impls moved verbatim (git rename). Builders/consumers now import the local
sibling; main_agent knowledge_priority imports the new shared path; shared
middleware barrel trimmed.

Tests: repoint imports; convert the knowledge_search monkeypatch targets
from brittle dotted-string form to object-based patching (monkeypatch.setattr
on the imported module), which is robust to import ordering. No behavior
change.
2026-06-05 12:04:31 +02:00
CREDO23
9493519c61 refactor(agents): colocate 8 main-agent-only middleware as per-concept folders
Each main-agent-only middleware now lives in its own folder under
main_agent/middleware/<concept>/ with builder.py (flag-gated construction)
+ middleware.py (the impl), re-exported via __init__.py. This kills the
cross-folder hop into agents/shared/middleware and keeps each middleware's
two responsibilities (build vs behavior) as colocated siblings.

Moved (impl from shared/middleware, builder from main_agent/middleware):
action_log, anonymous_document, context_editing, doom_loop, knowledge_tree,
noop_injection, otel_span, tool_call_repair.

Impls moved verbatim (git rename, no body edits) so behavior is unchanged.
Builders now import from the local .middleware sibling. stack.py import
paths updated for the 3 renamed folders; shared middleware barrel trimmed;
tests repointed (imports + patch targets).
2026-06-05 11:42:58 +02:00
CREDO23
fbd5ccc35a refactor(agents): split dedup_tool_calls; move HITL middleware to main_agent
DedupHITLToolCallsMiddleware is only wired by the main_agent stack, but
its module also exports dedup-key resolvers consumed by the shared MCP
tool layer. Splitting keeps the resolvers (dedup_key_full_args,
wrap_dedup_key_by_arg_name, DedupResolver) in shared and moves the
middleware class verbatim into main_agent/middleware/dedup_hitl.py
(merged with its builder), eliminating the shared->main_agent dependency
that a flat move would create. No behavior change.
2026-06-05 11:17:44 +02:00
CREDO23
afa51e97cf refactor(agents): delete dead single-agent-only middleware
file_intent (FileIntentMiddleware) and flatten_system
(FlattenSystemMessageMiddleware) were only ever instantiated in the
single-agent chat_deepagent stack, which was removed in 14bbea085. They
have no production consumer in multi_agent_chat. Delete both modules and
their unit tests.

Also drop the vestigial KnowledgeBaseSearchMiddleware alias (= the live
KnowledgePriorityMiddleware); its tests now target the real class so the
behavior coverage is preserved. Trim the three barrel/__all__ entries and
strip the now-dead class names from comments.
2026-06-05 11:15:13 +02:00
CREDO23
21509e7eca refactor(agents): group filesystem backends under filesystem/backends/
The concrete filesystem backends are consumed only by the MAC filesystem
layer (tools, path-resolution middleware, the resolver, skills backend) and
tests -- no external app code. Group them next to the filesystem middleware
they serve:

- filesystem_backends.py            -> filesystem/backends/resolver.py
- middleware/kb_postgres_backend.py -> filesystem/backends/kb_postgres.py
- middleware/local_folder_backend.py -> filesystem/backends/local_folder.py
- middleware/multi_root_local_folder_backend.py -> .../multi_root_local_folder.py
- document_xml.py                   -> filesystem/backends/document_xml.py

Repoint all 21 importers. No behavior change; import-all + filesystem
backend/path-resolution/knowledge-search unit tests stay green (478).
2026-06-05 11:02:26 +02:00
CREDO23
f615d6b530 refactor(agents): relocate remaining MAC-only kernel (permissions, deliverable_wait)
permissions.py (authorization Rule/Ruleset model) is consumed across all
MAC subagents + the permissions middleware, with a single external
consumer (user_tool_allowlist service) -> move to
multi_agent_chat/shared/permissions.py and repoint all 42 sites.

deliverable_wait.py (wait_for_deliverable) is used only by the podcast and
video_presentation deliverable tools -> colocate into
subagents/builtins/deliverables/.

No behavior change; import-all + permission/allowlist/deliverable unit
tests stay green.
2026-06-05 10:58:49 +02:00
CREDO23
714c5ffea9 refactor(agents): group tool-outcome receipts into multi_agent_chat/shared/receipts/
receipt.py (Receipt model + make_receipt) and receipt_command.py
(with_receipt Command helper) are a tight pair used only by MAC subagent
tools, the graph state, and the kb_persistence middleware -- no external
code imports them (the streaming tool_end handler only references them in a
docstring). Move both into a dedicated receipts/ package
(receipts/receipt.py + receipts/command.py) and repoint importers.

No behavior change; import-all + receipt/deliverable unit tests stay green.
2026-06-05 10:56:37 +02:00
CREDO23
1d2519730e refactor(agents): move MAC graph-state schema into multi_agent_chat/shared/state/
filesystem_state.py (the multi-agent graph state) and state_reducers.py
(its merge reducers) are consumed only by multi_agent_chat (filesystem
tools/middleware, kb projection, and the MAC-only shared middleware) plus
two unit tests -- no external app code. Relocate them into a dedicated
multi_agent_chat/shared/state/ package (filesystem_state.py + reducers.py)
and repoint every importer.

No behavior change; import-all + the full unit/middleware + unit/agents
suites (1066 tests) stay green.
2026-06-05 10:54:15 +02:00
CREDO23
2db4ad479e refactor(agents): colocate KB-search tool with its sole consumer; fix report ImportError
shared/tools/knowledge_base.py had exactly one production consumer: the
report deliverable, which imported it via `from .knowledge_base import ...`
-- a sibling path that did not exist, so the report KB-search path would
raise ImportError at runtime.

Move the module next to report.py (subagents/builtins/deliverables/tools/)
which makes that relative import valid, and move its only dependency
(shared/utils.py date helpers) to multi_agent_chat/shared/date_filters.py,
shared between the KB tool and the knowledge_search middleware.

Drop the now-unused knowledge-base re-exports from the shared/tools barrel
and repoint the integration tests. import-all + error-contract stay green.
2026-06-05 10:28:56 +02:00
CREDO23
a7d7155039 refactor(agents): colocate main_agent-only kernel into main_agent/
Move modules out of agents/shared/ that are consumed by a single package
(main_agent), placing each next to its only consumer instead of in a
"shared" grab-bag:

- agent_cache.py        -> main_agent/runtime/agent_cache_store.py
- connector_searchable_types.py -> main_agent/runtime/
- plugin_loader.py + plugins/    -> main_agent/plugins/
- skills/ + skills_backends.py   -> main_agent/skills/
- tools/invalid_tool.py          -> main_agent/tools/

Drop the skills_backends re-export from the shared middleware barrel and
repoint all consumers + tests. No behavior change; import-all,
error-contract, and the moved tests stay green.
2026-06-04 21:25:39 +02:00
CREDO23
c51aca6ccc refactor(agents): group MCP tools into shared/tools/mcp/ subpackage
The three MCP siblings (mcp_client/mcp_tool/mcp_tools_cache) served one
objective but sat loose at the top of shared/tools. Grouped them into an
mcp/ package and dropped the redundant prefix: client.py, tool.py, cache.py.
Updated all importers (routes, mcp_tools subagent, e2e fake patch targets,
unit test) to the new paths.
2026-06-04 20:35:38 +02:00
CREDO23
8d0090c6a1 refactor(agents): delete deliverable dead twins in shared/tools; fix live image api_base bug
The deliverables subagent runs its own generate_image/podcast/report/resume/
video_presentation (via tools/index.py); the shared/tools copies had zero
production importers — classic dead twins. Removed them so deliverable tools
live only in their vertical slice.

While repointing the 2 stranded unit tests at the LIVE deliverables modules,
found the OpenRouter empty-api_base defense (resolve_api_base) existed ONLY in
the dead shared generate_image, never propagated to the live multi-agent copy.
Ported the fix into deliverables/tools/generate_image.py (both the global-config
and user-DB-config branches) so an empty api_base no longer falls through to
LiteLLM's global api_base (Azure) and 404s.

Tests now exercise the live Command/receipt-returning tools (invoke the raw
coroutine with a hand-built ToolRuntime; resume progress events neutralized).
2026-06-04 20:30:30 +02:00
CREDO23
64512c604d refactor(agents): colocate gmail + calendar connector tools into subagent slices
Gmail and Calendar are handled together because both Google connectors share
the _build_credentials helper that lived in shared/tools/gmail.

- relocate the gmail helpers (_get_token_encryption, _build_credentials,
  _gmail_headers, _format_gmail_summary) into the gmail subagent slice
  (tools/_helpers.py); repoint gmail search_emails/read_email to it.
- calendar search_events now imports _build_credentials from the gmail slice
  (preserving the existing cross-connector Google-auth dependency).
- repoint both dead tools/__init__ shims at the live local impls.
- fix tests/e2e native_google fake: it patched the dead shared
  google_calendar.*.build paths; point it at the live subagent calendar
  modules (which actually import googleapiclient build).
- delete dead shared/tools/{gmail,google_calendar} twins.

shared/tools now has zero connector dirs. agents unit suite green (942).
2026-06-04 20:09:37 +02:00
CREDO23
70fb19890b refactor(agents): colocate google_drive connector tools into subagent slice
Repoint the dead tools/__init__ shim at the live local impls and delete the
dead shared/tools/google_drive twin (subagent already ran its local copies via
tools/index.py). No runtime behavior change.
2026-06-04 20:03:58 +02:00
CREDO23
97ec27c786 refactor(agents): colocate onedrive connector tools into subagent slice
Repoint the dead tools/__init__ shim at the live local impls and delete the
dead shared/tools/onedrive twin (subagent already ran its local copies via
tools/index.py). No runtime behavior change.
2026-06-04 20:03:58 +02:00
CREDO23
1a778883b3 refactor(agents): colocate dropbox connector tools into subagent slice
Repoint the dead tools/__init__ shim at the live local impls and delete the
dead shared/tools/dropbox twin (subagent already ran its local copies via
tools/index.py). No runtime behavior change.
2026-06-04 20:03:58 +02:00
CREDO23
1b9c2820e8 refactor(agents): colocate teams connector tools into subagent slice
Repoint the dead tools/__init__ shim at the live local impls and delete the
dead shared/tools/teams twin (subagent already ran its local copies via
tools/index.py). No runtime behavior change.
2026-06-04 20:03:58 +02:00
CREDO23
c6525c4f52 refactor(agents): colocate discord connector tools into subagent slice
Repoint the dead tools/__init__ shim at the live local impls and delete the
dead shared/tools/discord twin (subagent already ran its local copies via
tools/index.py). No runtime behavior change.
2026-06-04 20:03:58 +02:00
CREDO23
425e6e50a3 refactor(agents): colocate luma connector tools into subagent slice
The luma subagent already ran its own local tool impls (tools/index.py
imports the local create_event/list_events/read_event). The shared/tools/luma
copies were a dead twin, only referenced by the subagent's unused
tools/__init__ shim. Repoint that shim at the local modules and delete the
dead shared copies. No runtime behavior change.
2026-06-04 20:02:19 +02:00
CREDO23
4d02af2a53 refactor(agents): delete orphaned shared/tools connectors
With the registry gone, these shared/tools dirs had no importers:

- notion/, confluence/: the live subagents (subagents/connectors/{notion,
  confluence}) own self-contained tool implementations; the shared copies
  were dead duplicates.
- linear/: stale empty dir (no tracked files).
- connected_accounts.py: get_connected_accounts tool was only ever built by
  the deleted registry; no builder instantiates it. Removed its impl and its
  catalog entry (was advertising an unbuildable tool).

agents unit suite green (978 passed; -9 import-smoke cases for the deleted
modules, no coverage lost).
2026-06-04 19:55:31 +02:00
CREDO23
003924062d refactor(agents): split tool registry into pure-data catalog, decouple connectors
Replace the connector-coupled BUILTIN_TOOLS registry with a pure-data
catalog so shared/tools no longer imports any connector module, making the
connector packages independently deletable.

- add shared/tools/catalog.py (ToolMetadata + TOOL_CATALOG, 41 tools, no imports)
- point GET /agent/tools (the only live consumer) at the catalog
- relocate ToolDefinition into action_log middleware (its sole consumer);
  drop the inert tool_definitions wiring (no tool defines reverse)
- delete shared/tools/registry.py: connector imports, dead factories,
  dead get_connector_gated_tools, and BUILTIN_TOOLS
- drop stale dedup-propagation test (path removed in C1) + refresh docstrings

import-all guardrail + agents unit suite green (987 passed).
2026-06-04 19:43:50 +02:00
CREDO23
c3238d8840 refactor(agents): remove dead tool-building machinery from registry
After the main agent moved to its own build_main_agent_tools, nothing calls
the shared registry's builders. Delete the dead functions (build_tools,
build_tools_async, get_tool_by_name, get_all_tool_names,
get_default_enabled_tools) plus the now-orphaned load_mcp_tools import and the
stale __init__ re-exports.

BUILTIN_TOOLS, ToolDefinition, and get_connector_gated_tools are retained:
the catalog is still consumed for tool *metadata* (action_log revert/dedup
resolvers and the /agent/tools listing). Also drop stale references to the
deleted chat_deepagent.py within the agents module.

Verified: full unit suite green (2431 passed, 1 skipped); lints clean.
2026-06-04 19:24:17 +02:00
CREDO23
66103c68f6 refactor(agents): colocate main-agent-only tools (scrape_webpage, update_memory)
These two tools were "shared-by-folder, not shared-by-use": the only live
consumer of shared/tools/{scrape_webpage,update_memory} was the main agent
(the research/memory subagents carry their own local copies; web_search,
by contrast, is genuinely shared with anonymous_chat and stays put).

Move both into main_agent/tools/ (their sole owner). The shared BUILTIN_TOOLS
catalog still lists them for action_log/revert + /agent/tools, now via
deferred-import factories (_build_scrape_webpage_tool, _build_update_memory_tool)
mirroring the create_automation precedent to avoid a multi_agent_chat import
cycle. Removed the now-dead re-exports from shared/tools/__init__.py.

Verified: full unit suite green (2431 passed, 1 skipped).
2026-06-04 19:10:48 +02:00
CREDO23
482aefc32a refactor(agents): SRP main-agent tool registry, decoupled from BUILTIN_TOOLS
The main agent only exposes 4 SurfSense tools (web_search, scrape_webpage,
update_memory, create_automation) and delegates connectors/MCP/deliverables
to subagents. Yet it built those 4 by importing and iterating the 900-line,
connector-laden shared BUILTIN_TOOLS via build_tools_async.

Introduce app/agents/multi_agent_chat/main_agent/tools/registry.py owning
just those 4 factories, and switch runtime/factory.py to build_main_agent_tools.
Binding order is preserved (scrape_webpage, web_search, create_automation,
update_memory) to match prior behavior exactly.

shared/tools/registry.py BUILTIN_TOOLS is intentionally unchanged: it remains
the app-wide tool *metadata* catalog used by action_log (revert/dedup
resolvers for subagent-executed connector tools) and the /agent/tools
listing endpoint.

Verified: full unit suite green (2431 passed, 1 skipped); import-all guard ok.
2026-06-04 19:01:44 +02:00
CREDO23
add9e14694 refactor(agents): colocate middleware into vertical slices
Eliminate the top-level multi_agent_chat/middleware/ package so each slice
owns its middleware (vertical-slice colocation):

- middleware/shared/   -> shared/middleware/        (cross-slice middleware)
- middleware/subagent/ -> subagents/shared/middleware/ (subagent stack)
- main_agent/middleware/ already colocated in Slice A

The moved shared/ subtree is internally consistent (all relative imports
stay within it), so only external absolute refs were rewritten. The
subagent stack's ..shared.* relatives were promoted to absolute paths to
the new shared/middleware/ location.

multi_agent_chat/ root is now: main_agent/, shared/, subagents/.
Verified: 2430 unit tests pass, 1 skipped (baseline unchanged).
2026-06-04 18:13:47 +02:00
CREDO23
9c845d562e refactor(agents): colocate main-agent middleware under main_agent/ slice
Vertical-slice colocation: all main-agent code should live under
main_agent/ instead of being split across a parallel middleware/main_agent
tree. Move multi_agent_chat/middleware/main_agent/ -> main_agent/middleware/
and its assembler middleware/stack.py -> main_agent/middleware/stack.py, so
the main-agent slice is self-contained (graph, runtime, system_prompt, tools,
middleware).

Genuinely cross-slice middleware (middleware/shared/, middleware/subagent/)
stays under multi_agent_chat/middleware/ for a later slice; the moved builders
now reference it via absolute imports.

Pure move + import rewrite (git-tracked renames). Verified: full unit suite
green (2430 passed, 1 skipped), including test_import_all and the
checkpointed-subagent middleware suite.
2026-06-04 18:03:49 +02:00
CREDO23
1acde6a470 test(agents): cover live filesystem middleware, retire dead twin
The single-agent-era filesystem middleware (app/agents/shared/middleware/
filesystem.py, ~2000 lines) was never instantiated in production, yet three
unit suites validated it — an illusory guardrail while the live decomposed
middleware (multi_agent_chat/middleware/shared/filesystem) was unguarded.

Close the gap before reorganizing the agents module:
- Add 14 integration tests driving live B's tools in desktop mode (real
  on-disk effects) and cloud mode (in-state staging, namespace policy).
- Port all high-value dead-twin assertions onto the live path: cloud rm/rmdir
  staging + guard rails, KBPostgresBackend delete-view filter, mode-scoped
  system prompt, cwd/relative/namespace resolution, multi-root mount
  normalization.
- Delete dead twin filesystem.py, drop its __init__ re-export, and retire its
  3 dead-twin tests.

Verified: test_import_all + middleware unit + FS integration all green.
2026-06-04 17:46:49 +02:00
CREDO23
5b45f78a16 refactor(chat): delete legacy stream_new_chat monolith (cutover complete)
The flows orchestrators (new_chat/resume_chat) are now the sole live path
after the byte-for-byte differential proof, so the monolith and its
monolith-vs-flows parity scaffolding are removed.

- Repoint the last live importer (anonymous_chat_routes) to
  streaming.agent.event_loop.stream_agent_events + shared.stream_result.StreamResult
  (drop-in; the keyword-only fallback-commit params default to inert for anon).
- Repoint e2e launcher patch targets to flows.shared.llm_bundle.
- Repoint helper unit tests (chunk_parts, thinking-step ids, tool-input
  streaming) to their flows homes to preserve coverage.
- Delete the monolith, the contract test, and the parity tests
  (parallel_refactor, stage_1, stage_2, orchestrator_frame) whose sole
  purpose was comparing against the now-removed monolith.

Full suite green (2622 passed, 1 skipped); the two excluded live-app dirs
(document_upload, composio) have a pre-existing, env-gated registration 404
unrelated to this change.
2026-06-04 14:35:45 +02:00
CREDO23
b9937cf4b1 refactor(chat): cut live chat streaming over to flows orchestrators (cutover)
Flip the live callers (new_chat_routes + gateway/agent_invoke) from the
legacy monolithic app.tasks.chat.stream_new_chat to the side-by-side
app.tasks.chat.streaming.flows orchestrators.

Adds a byte-for-byte differential parity test driving BOTH implementations
on identical, fully-deterministic glue inputs (frozen time/uuid, stubbed
LLM/persistence/agent seams). All glue paths are byte-identical:
  * new: auto-pin failure, LLM-load failure, persist-user fail,
    persist-assistant fail (full initial-frame ordering + handshake),
    pre-stream exception (top-level except path)
  * resume: persist-assistant fail

The differential test also surfaced one INTENTIONAL divergence: on a resume
turn whose auto-pin / LLM-load fails, the monolith crashes with
UnboundLocalError (_resume_premium_request_id read in finally before its
post-early-return definition); the flows version emits a clean terminal
error. The flows path is therefore byte-identical or strictly more correct.

The agent-content stream itself is shared, unforkable code
(stream_output -> EventRelay) so it cannot diverge.

Monolith + old parity test deletion follows in a separate commit.
2026-06-04 14:24:18 +02:00
CREDO23
8faa03889d docs(agents): refresh comments that referenced the deleted single-agent stack (bucket B6)
After deleting app/agents/new_chat/, several shared-kernel comments still cited
new_chat paths/cycles. Update the two lazy-import comments in middleware to state
the real reason (tools.registry <-> shared.middleware cycle), and repoint dangling
``new_chat/tools/hitl.py`` / ``chat_deepagent`` doc references to their shared
locations. Comment-only; suite unaffected.
2026-06-04 13:47:10 +02:00
CREDO23
305a8fe7e6 refactor(config): remove dead MULTI_AGENT_CHAT_ENABLED flag (bucket B5)
Multi-agent is now the only chat path (B1-B4), so the flag is dead config.
Drop it from app/config and .env.example. Suite green (2710 passed).
2026-06-04 13:44:02 +02:00
CREDO23
14bbea0854 refactor(agents): delete single-agent stack + new_chat shim package (bucket B3/B4)
With multi-agent the only live factory (B1), the single-agent stack is dead.
Remove app/agents/new_chat/ entirely: chat_deepagent.py, subagents/, and all
re-export shims (errors/context/llm_config/permissions/tools/middleware/...) that
existed only to serve frozen single-agent code. Live code already imports the
shared kernel (app.agents.shared.*) directly.

Tests: delete single-agent-only suites (test_resolve_prompt_model_name,
test_specialized_subagents) and the chat_deepagent source-shape contract assertion;
repoint test_scoped_model_fallback to the shared middleware path. Suite green
(2710 passed).
2026-06-04 13:40:44 +02:00
CREDO23
724bbd6deb refactor(agents): hardwire multi-agent as the only chat factory (bucket B1)
Make create_multi_agent_chat_deep_agent the unconditional agent factory in
all three streaming entry points (stream_new_chat monolith + new_chat/resume
flow orchestrators). Drop the MULTI_AGENT_CHAT_ENABLED branch and the now-unused
create_surfsense_deep_agent / _app_config imports. The single-agent
implementation (chat_deepagent.py, subagents/) is left in place; it is deleted
in a later phase. Suite green (2758 passed).
2026-06-04 13:35:38 +02:00
CREDO23
64d2ad6451 refactor(agents): promote anonymous_agent to its own anonymous_chat/ package (slice 8)
The anonymous / free-chat agent is a distinct live agent (not part of the
single-agent stack and not shared infrastructure), so it gets its own top-level
package app/agents/anonymous_chat/ (sibling to multi_agent_chat). Moved
new_chat/anonymous_agent.py -> anonymous_chat/agent.py with a package __init__
re-exporting create_anonymous_chat_agent + build_anonymous_system_prompt.

Repointed its only new_chat import (the context shim) to app.agents.shared.context
and updated the single importer (anonymous_chat_routes). The test_import_all
guard auto-discovers the new package via pkgutil.walk_packages, so it is covered.
2026-06-04 13:25:23 +02:00
CREDO23
a019f18d1c refactor(agents): move connector_searchable_types, agent_cache, system_prompt + prompts to app/agents/shared (slice 7b)
Three live shared leaves discovered while taking stock after slice 7 (all are
consumed by the multi-agent stack and/or live routes, not single-agent-only):

- connector_searchable_types -> shared + shim (multi-agent factory uses it)
- agent_cache -> shared + shim (multi-agent runtime/agent_cache uses it)
- system_prompt + prompts/ (42 .md fragments) -> shared together + shim.
  Repointed composer's _PROMPTS_PACKAGE to app.agents.shared.prompts so
  importlib.resources fragment loading keeps working; system_prompt's relative
  ".prompts.composer" import is preserved by moving both as a unit.

Each keeps a re-export shim for the frozen chat_deepagent. After this slice,
new_chat/ holds only the frozen single-agent stack (chat_deepagent, subagents/,
__init__) plus shims.
2026-06-04 13:21:45 +02:00
CREDO23
13a96851ef refactor(agents): move skills/, plugins/, plugin_loader to app/agents/shared (slice 7)
- skills/ (builtin SKILL.md assets) has zero Python importers; it is read by
  filesystem path only. Moved the dir and restored
  skills_backends._default_builtin_root() to the clean
  parent.parent / "skills" / "builtin" form (undoing the transient path from 5c).
- plugin_loader.py -> shared (frozen chat_deepagent uses it -> re-export shim).
- plugins/ package -> shared (year_substituter rewired to shared.plugin_loader;
  docstring entry-point example updated to the shared dotted path). No shim
  needed (only a test imported it). Plugin discovery is via importlib entry
  points (group "surfsense.plugins"), not dotted-path import, and nothing is
  registered in pyproject, so the move does not affect runtime discovery.
2026-06-04 13:16:22 +02:00
CREDO23
aab95b9130 refactor(agents): move tools package to app/agents/shared (slice 6)
Relocate the entire new_chat/tools/ package (62 files incl. registry, hitl, MCP
cluster, and all connector subpackages: gmail/slack/discord/teams/drive/etc.)
to the shared kernel. The package turned out to be a clean cohesive cluster:
its only references to non-tools new_chat modules were comments, and its
middleware deps were already flipped to shared in slice 5c.

Flip 33 live importers (multi-agent, flows, routes, services, anonymous_agent,
tests). Re-export shims remain for the frozen single-agent stack: a package
__init__ mirroring the public surface (new_chat.__init__ imports it) plus
invalid_tool + registry submodule shims (chat_deepagent imports those).

Resolves slice 5c's two transient back-edges: shared/middleware/action_log
(TYPE_CHECKING ToolDefinition) and tool_call_repair (local INVALID_TOOL_NAME)
now point at app.agents.shared.tools.
2026-06-04 13:11:56 +02:00
CREDO23
a7fde2a48e refactor(agents): move filesystem_backends to app/agents/shared (slice 5d)
Completes slice 5. filesystem_backends was deferred from 3b because it depends
on middleware.{kb_postgres_backend,multi_root_local_folder_backend}; those moved
to shared in 5c, so it now relocates cleanly. Flip the 2 non-frozen importers
(multi-agent factory + test); a re-export shim remains for the frozen
chat_deepagent (build_backend_resolver).
2026-06-04 13:03:15 +02:00
CREDO23
227983a104 refactor(agents): move middleware package to app/agents/shared (slice 5c)
Relocate the entire new_chat/middleware/ package to the shared kernel as one
cohesive unit (it is live shared infrastructure: the multi-agent stack wraps
nearly every middleware via multi_agent_chat/middleware/main_agent/*, and
anonymous_agent consumes it too). Flip 69 live importers across both the
package-path and submodule-path forms.

Shims left for the frozen single-agent stack: a package __init__ re-export plus
submodule shims for permission, skills_backends, and scoped_model_fallback
(the three imported via submodule path by chat_deepagent/subagents).

Cycle break: importing shared.middleware previously reached back into
new_chat.tools at module load, which dragged in new_chat.__init__ ->
chat_deepagent -> the middleware shim -> half-initialized shared.middleware.
Made action_log's ToolDefinition import TYPE_CHECKING-only and
tool_call_repair's INVALID_TOOL_NAME import function-local. These tools-package
back-edges fully resolve in slice 6.

Asset note: skills_backends._default_builtin_root now walks to
app/agents/new_chat/skills/builtin (the skills/ tree migrates in slice 7).
2026-06-04 13:00:41 +02:00
CREDO23
6f488d9564 refactor(agents): move checkpointer + mention_resolver to app/agents/shared (slice 5b)
Two independent leaf modules (no intra-new_chat deps, no frozen importer),
consumed only by flows/routes/tests. Flipped 8 importers across both the
dotted-path and module-style (from app.agents.new_chat import mention_resolver)
forms. No shims needed.
2026-06-04 12:52:54 +02:00
CREDO23
dcdf8f776b refactor(agents): move utils + document_xml to app/agents/shared (slice 5a)
Two pure leaf modules with no intra-new_chat deps and no frozen importer.
Moving them now (before the middleware package) pre-empts two shared->new_chat
back-edges that the middleware move would otherwise create
(knowledge_search->utils, kb_postgres_backend->document_xml).
2026-06-04 12:50:38 +02:00
CREDO23
946f8a8c5d refactor(agents): move llm_config + prompt_caching to app/agents/shared (slice 4b)
Relocate the mutually-dependent LLM config layer and the LiteLLM prompt-caching
helper to the shared kernel as one unit, rewiring their internal cross-reference
to the shared paths. Flip 21 non-frozen importers. Re-export shims remain at
new_chat/{llm_config,prompt_caching}.py for the frozen single-agent stack
(chat_deepagent); they will be removed when that stack is retired.
2026-06-04 12:41:52 +02:00
CREDO23
8fca2753aa refactor(agents): move permissions to app/agents/shared (slice 4a)
Relocate the permission evaluator (wildcard matcher + rule evaluation) to the
shared kernel and flip 43 non-frozen importers. A re-export shim remains at
new_chat/permissions.py for the frozen single-agent stack (chat_deepagent and
subagents/{config,providers/linear,providers/slack}); it will be removed when
that stack is retired.
2026-06-04 12:38:30 +02:00
CREDO23
3efe51e6ec refactor(agents): move filesystem_state, path_resolver, sandbox to app/agents/shared (slice 3b)
Relocate three leaf filesystem-cluster modules to the shared kernel and flip
all 38 importers. No re-export shims needed (no frozen single-agent importer).
This also resolves the pre-existing shared->new_chat back-edge from
shared/receipt_command.py onto filesystem_state.

filesystem_backends is intentionally deferred to slice 5: it depends on
new_chat middleware (kb_postgres_backend, multi_root_local_folder_backend)
that have not yet moved, so relocating it now would create a shared->new_chat edge.
2026-06-04 12:34:28 +02:00