diff --git a/surfsense_backend/AFTER_WORK_FIXES.md b/surfsense_backend/AFTER_WORK_FIXES.md deleted file mode 100644 index d74b7c7da..000000000 --- a/surfsense_backend/AFTER_WORK_FIXES.md +++ /dev/null @@ -1,36 +0,0 @@ -# After Work Fixes - -## Middleware Risk Flags (new_chat) - -These are known "policy/routing via middleware" risks to review later. - -1. `FileIntentMiddleware` -- Risk: `file_write` classification can force `write_file`/`edit_file` and override deliverable or connector tool selection. -- Example failure: user asks for video/report artifact, agent writes into `/documents/*` instead. - -2. `KnowledgePriorityMiddleware` -- Risk: KB planner and injected priority hints can over-anchor turns to KB reads when connector action is the better path. - -3. `KnowledgeTreeMiddleware` -- Risk: injected workspace tree can bias behavior toward file navigation/writes by default. - -4. `SurfSenseFilesystemMiddleware` + `KnowledgeBasePersistenceMiddleware` -- Risk: mistaken `write_file` actions become persisted NOTE documents in KB, making wrong-path behavior durable. - -5. `PermissionMiddleware` -- Risk: deny/ask rules can hide or block the correct tool, appearing as "model chose wrong tool" when it never had access. - -6. Subagent middleware parity (`chat_deepagent.py`) -- Risk: parent vs subagent stack differences can produce inconsistent behavior across similar tasks. - -7. `SpillingContextEditingMiddleware` + compaction -- Risk: context trimming can remove critical tool evidence and cause wrong retries/tool choices. - -8. `ToolCallNameRepairMiddleware` -- Risk: malformed calls may be auto-repaired to unintended tools in edge cases. - -9. `DedupHITLToolCallsMiddleware` / `DoomLoopMiddleware` -- Risk: legitimate repeated calls can be suppressed or stopped early. - -10. `MemoryInjectionMiddleware` -- Risk: injected memory may bias tool choice away from fresh connector/KB evidence. diff --git a/surfsense_backend/app/agents/multi_agent_chat/CAPABILITY_PARITY_CHECKLIST.md b/surfsense_backend/app/agents/multi_agent_chat/CAPABILITY_PARITY_CHECKLIST.md deleted file mode 100644 index b759e3221..000000000 --- a/surfsense_backend/app/agents/multi_agent_chat/CAPABILITY_PARITY_CHECKLIST.md +++ /dev/null @@ -1,57 +0,0 @@ -# Multi-Agent Capability Parity Checklist - -This checklist tracks whether `multi_agent_chat` has the required capability coverage -to be manually tested against `new_chat` in LangSmith. - -Legend: -- `[x]` implemented -- `[~]` implemented with intentional difference -- `[ ]` pending - -## 1) Prompting - -- [x] Supervisor prompt has explicit delegation policy. -- [x] Supervisor prompt consumes structured expert outputs (`status`, `evidence`, `next_step`, `missing_fields`, `assumptions`). -- [x] Supervisor available specialist list is dynamically rendered from currently registered tools. -- [x] All expert prompts are normalized to a shared JSON output contract shape with invariant rules. -- [x] Memory wording adapts to thread visibility (user vs team). -- [~] `generic_mcp` specialist prompt exists but route is intentionally disabled. - -## 2) Tooling and Routing - -- [x] Built-in specialist routes are wired (`research`, `memory`, `deliverables` when eligible). -- [x] Connector specialist routes are gated by available connector inventory. -- [x] MCP tools are partitioned and merged into matching specialists. -- [x] MCP-only named specialists are routed when present (`linear`, `slack`, `jira`, `clickup`, `airtable`). -- [~] `generic_mcp` route is intentionally disabled by product decision. -- [x] Delegated child tasks include explicit structured context envelope tags. -- [x] Domain-agent outputs are parsed and validated as JSON with safe fallback envelope. - -## 3) Middleware / Runtime - -- [x] Supervisor middleware stack mirrors SurfSense shell used by `new_chat` for core protections. -- [~] `SubAgentMiddleware` intentionally omitted (multi-agent architecture uses explicit specialists). -- [~] `PermissionMiddleware` intentionally omitted by decision (route gating used instead). -- [x] Action-log / compaction / retry / fallback / filesystem / KB middleware are wired for supervisor path. -- [x] Agent graph compile path uses `asyncio.to_thread` for heavy build operations. - -## 4) Entry-Point Wiring - -- [x] Authenticated streaming path can route to `create_multi_agent_chat` via feature flag (`MULTI_AGENT_CHAT_ENABLED`). -- [x] Resume streaming path can route to `create_multi_agent_chat` via feature flag. -- [~] Authenticated stream falls back to `new_chat` when `disabled_tools` is provided (multi-agent does not yet implement disabled-tool filtering parity). -- [ ] Anonymous stream path wired to multi-agent (left unchanged for now due anonymous tool allow-list differences). - -## 5) Observability and Validation Readiness - -- [x] Ready for manual LangSmith trace inspection once `MULTI_AGENT_CHAT_ENABLED=true`. -- [ ] Formal routing eval harness and benchmark dataset. -- [ ] Automated regression checks in CI for routing quality. - -## 6) Manual Benchmark Readiness Decision - -Status: **Ready for manual benchmarking in authenticated flows**. - -Before declaring "better than `new_chat`", still required: -- Build and run formal eval/benchmark harness. -- Close anonymous-path and disabled-tools parity gaps if they are in benchmark scope. diff --git a/surfsense_backend/app/agents/multi_agent_chat/IMPLEMENTATION_PLAN.md b/surfsense_backend/app/agents/multi_agent_chat/IMPLEMENTATION_PLAN.md deleted file mode 100644 index 186a55975..000000000 --- a/surfsense_backend/app/agents/multi_agent_chat/IMPLEMENTATION_PLAN.md +++ /dev/null @@ -1,122 +0,0 @@ -# `multi_agent_chat` — layout & alignment with `new_chat` - -## Mission - -**Preserve** everything that makes SurfSense chat agents production-grade in `new_chat` (KB, middleware, tools, prompts, safety, observability). **Rework** how those pieces are composed: a clearer **multi-agent** layout (supervisor + domain slices + routing), less accidental coupling, and one explicit assembly path—so the agent stays **excellent** (correct tools, grounded KB, safe permissions, debuggable traces), not just “different folders.” - -Implementation strategy: **reuse `new_chat` modules** (middleware classes, tool factories, KB helpers, prompts composer pieces) wherever possible; **`multi_agent_chat` owns structure and wiring**, not reimplemented business logic. - ---- - -## What we must not lose from `new_chat` (capability inventory) - -Use this as a checklist when porting middleware/KB into `multi_agent_chat`. Items map to `surfsense_backend/app/agents/new_chat/`. - -| Area | Capabilities to preserve | Typical locations | -|------|-------------------------|-------------------| -| **KB & documents** | Hybrid search → priority docs → lazy XML load; workspace tree; anon-document path; KB persistence / commit staging | `middleware/knowledge_search.py`, `knowledge_tree.py`, `kb_persistence.py`, `anonymous_document.py`, `tools/knowledge_base.py`, `search_surfsense_docs.py` | -| **Filesystem** | Virtual FS, backends, path resolver, file intent | `middleware/filesystem.py`, `filesystem_backends.py`, `path_resolver.py`, `file_intent.py` | -| **Memory & context** | Memory injection, team/private protocols, context schema | `middleware/memory_injection.py`, `prompts/base/memory_protocol_*.md`, `context.py` | -| **Safety & quality** | Permissions, doom-loop detection, dedup HITL tool calls, tool-call repair, action logging | `middleware/permission.py`, `doom_loop.py`, `dedup_tool_calls.py`, `tool_call_repair.py`, `action_log.py` | -| **Model / context limits** | Compaction, context editing / spill, summarization, model & tool call limits, retries / fallback | `middleware/compaction.py`, `context_editing.py`, `chat_deepagent.py` stack | -| **Concurrency & ops** | Busy mutex (single-flight turns), OTel spans | `middleware/busy_mutex.py`, `otel_span.py` | -| **Skills & subagents** | Skills backends, subagent specs and wrapping patterns | `middleware/skills_backends.py`, `subagents/` | -| **Tools** | Async registry, connector gating, MCP loading, feature-flagged tools | `tools/registry.py`, `feature_flags.py`, `tools/mcp_tool.py` | -| **Prompts** | Composer, provider fragments, tool routing (KB vs live connectors), citations | `prompts/composer.py`, `prompts/base/tool_routing_*.md`, `system_prompt.py` | -| **Runtime** | Checkpointer, LLM config, `create_agent` + middleware ordering discipline | `checkpointer.py`, `llm_config.py`, `chat_deepagent.py` | - -Not every row applies to the **first** multi-agent graph (e.g. you may start with a subset of middleware). The rule is: **if `new_chat` does it for correctness or safety, we either reuse it or consciously document why this graph differs.** - ---- - -## Rework principles (better arrangement, same substance) - -1. **Expert agents**: **`expert_agent/builtins/`** — broad registry **categories** (e.g. research, deliverables), not a single vendor. **`expert_agent/connectors/`** — **external integrations** (one package per product route: Discord, Notion, Gmail, …), each using the same pattern: ``slice_tools.py`` (registry subset or factories) + ``domain_prompt.md`` + ``agent.py``. Cross-cutting helpers live in `core/` or are imported from `new_chat`. -2. **Explicit graphs**: supervisor vs domain agents vs routing tools are **named** and testable; avoid one opaque megagraph where behavior is hard to reason about. -3. **Single composer**: integration eventually mirrors `create_surfsense_deep_agent` in spirit—**one factory** that attaches middleware, KB, and tools in documented order (see `chat_deepagent.py` comments on ordering). -4. **No duplicate KB pipelines**: align with `KnowledgePriorityMiddleware` / tree semantics; don’t invent a second hybrid-search path for the same turn. -5. **Parity tests**: when wiring completes, compare behavior against `new_chat` for the same user message + search space where scopes overlap (KB snippet quality, tool allow/deny). - ---- - -## Supervisor vs domain agents — tools and context - -**Supervisor (orchestrator)** - -- Keeps a **small tool surface**: one **routing** tool per builtin category (`research`, `memory`, …) and per connector route (`notion`, `gmail`, …) — **not** the full flat `registry.py` tool list on the supervisor. -- **KB** should primarily benefit the model via **`new_chat`-style middleware** (e.g. hybrid priority docs → state / system adjunct), not by stacking redundant search tools, unless product explicitly requires them. -- **Single hybrid search per user turn** at this layer when possible: full retrieval is expensive; avoid running it again inside every sub-agent for the same message. -- Does **not** own **on-demand connector discovery** (e.g. `get_connected_accounts`): orchestration is route-by-intent, not ID resolution. - -**Domain agents (every connector slice — same shape)** - -- Carry tools built from **`new_chat`** (`registry` subsets via ``build_registry_tools_for_category`` per ``TOOL_NAMES_BY_CATEGORY``, plus MCP merge where applicable). -- **Curated context belongs in the task message**: when the supervisor calls **any** routing tool, the handler composes the child’s task string so it includes **only** what that domain needs (KB snippets, constraints, distilled facts) — folded into how the task is written — not the full parent transcript. The sub-agent `invoke` stays a tight payload (`messages` + task content); domain middleware can still add connector-local hints. Still **no second full hybrid search** for the same turn unless the subdomain explicitly needs a new query. -- **Middleware here** still fits **domain-only** grounding (connector availability, search-space hints, metadata) shared across tools in that subgraph. Reuse or thin-wrap `new_chat.middleware` where it applies to a subgraph. -- **Reactive discovery** (resolve a service id mid-task) stays a **tool** on that domain (or shared factory), e.g. `get_connected_accounts` when the model needs it — not something the supervisor must call. - -**Tool grouping by category** - -- Group “horizontal” registry tools by **job** (research, deliverables, creative, …) into **separate compiled subgraphs**; supervisor gets **one routing tool per category** (subagents-as-tools), matching LangChain multi-agent guidance. See prior discussion: not all 10 non–connector-gated tools on the supervisor. - -### KB + virtual filesystem — where it belongs - -In `new_chat`, KB + **virtual FS** (`KnowledgePriorityMiddleware`, tree, **`SurfSenseFilesystemMiddleware`** / **`KBPostgresBackend`**) serves the **orchestrator** that may **read and traverse** the workspace. - -**Connector domain agents** are **not** mini-parents: the **supervisor** should already decide *what* to do and pass a **clear task** (plus any curated KB snippet folded into **`compose_child_task`**). The specialist runs **connector APIs**, not a second document crawl — duplicating full KB+VFS on every domain subgraph **shifts the parent’s exploration work onto the wrong agent** and adds noise. - -So **no child-side filesystem stack by default** for narrow connector subgraphs unless product demands it. Reserve **KB + VFS on a subgraph** for roles that **actually** need heavy document work (research, coding/explore-style agents, deliverables that grep the KB), matching how `new_chat` uses specialists. - ---- - -## Inspiration map (`new_chat` → `multi_agent_chat`) - -| Concern in `new_chat` | Primary references | Role in `multi_agent_chat` | -|----------------------|-------------------|---------------------------| -| **Main factory** | `chat_deepagent.py` (`create_surfsense_deep_agent`) | `integration/create_multi_agent_chat.py` — eventual single composer after KB + middleware land | -| **Tool lists** | `tools/registry.py`, `build_tools_async` | **`expert_agent/builtins/`** — category bundles (research, deliverables). **`expert_agent/connectors/`** — per-integration graphs (may use hand-written factories or registry subsets). | -| **Middleware stack** | `chat_deepagent.py` → `_build_compiled_agent_blocking`, `middleware/*.py` | **Planned:** `middleware/` — compose `create_agent(..., middleware=[...])` on supervisor and/or domain graphs; reuse or thin-wrap `new_chat.middleware` (ordering matters: see `new_chat` comments, e.g. BusyMutex → OTel → KB priority → filesystem → …) | -| **KB / hybrid search** | `middleware/knowledge_search.py` (`KnowledgePriorityMiddleware`), `middleware/knowledge_tree.py`, `tools/knowledge_base.py` | **Planned:** hybrid priority **once per user turn** at orchestrator; **curated KB/context folded into the routing task message** to children (no second full search for the same message unless explicitly scoped otherwise). | -| **Prompts** | `prompts/composer.py`, `prompts/base/*`, provider fragments | Vertical **`domain_prompt.md`** per slice + **`supervisor/supervisor_prompt.md`**; optional later: thin composer that injects KB/tool-routing fragments like `tool_routing_*.md` | -| **Context / checkpointer** | `context.py`, `checkpointer.py` | Pass **`Checkpointer`** into `create_multi_agent_chat` / `build_supervisor_agent`; align thread IDs with route layer when wired | -| **Subagent middleware** | `subagents/config.py` (`_wrap_with_subagent_essentials`) | Domain agents may eventually take **`middleware=`** on `create_agent` mirroring “inherit parent essentials + local rules” | - ---- - -## Current package tree - -``` -multi_agent_chat/ - __init__.py - - core/ # one concern per subfolder (SRP) - prompts/ # read_prompt_md — markdown next to packages - agents/ # build_domain_agent — compile subgraph + prompt - delegation/ # compose_child_task — supervisor → child message - invocation/ # extract_last_assistant_text — invoke result parsing - bindings/ # ``connector_binding`` — DB/search-space kwargs (not ``expert_agent.connectors`` vendors) - registry/ # TOOL_NAMES_BY_CATEGORY, build_registry_tools_for_category, build_registry_dependencies - - expert_agent/ - builtins/ # broad categories: research, deliverables - connectors/ # one subgraph per vendor route (see TOOL_NAMES_BY_CATEGORY keys) - - routing/ - domain_routing_spec.py - from_domain_agents.py - supervisor_routing.py - - supervisor/ - supervisor_prompt.md - graph.py - - integration/ - create_multi_agent_chat.py -``` - ---- - -## References - -- LangChain: [Multi-agent](https://docs.langchain.com/oss/python/langchain/multi-agent), [Subagents](https://docs.langchain.com/oss/python/langchain/multi-agent/subagents). -- Internal: `surfsense_backend/app/agents/new_chat/chat_deepagent.py`, `middleware/`, `tools/registry.py`. diff --git a/surfsense_backend/app/agents/multi_agent_chat/PROMPT_TUNING_PLAYBOOK.md b/surfsense_backend/app/agents/multi_agent_chat/PROMPT_TUNING_PLAYBOOK.md deleted file mode 100644 index db8e0b495..000000000 --- a/surfsense_backend/app/agents/multi_agent_chat/PROMPT_TUNING_PLAYBOOK.md +++ /dev/null @@ -1,159 +0,0 @@ -# Multi-Agent Prompt Tuning Playbook - -This playbook defines how to tune `multi_agent_chat` prompts for better outcomes than `new_chat` on delegation quality, lower confusion, and stable tool behavior. - -It is intentionally architecture-aware: this system is a **supervisor + expert tools** pattern, not a single flat tool agent. - -## Why this matters in our architecture - -- The supervisor only sees **routing tools** (e.g. `research`, `gmail`, `calendar`), not low-level connector APIs. -- Experts are invoked through `routing/from_domain_agents.py` and receive a single natural-language task via `compose_child_task(...)`. -- Because expert context is compact and delegated, prompt quality is the primary control lever for routing accuracy and downstream tool behavior. - -## Authoritative guidance we should follow - -- Anthropic prompt engineering best practices (role clarity, XML structure, explicit tool-use policy, few-shot examples): [Anthropic docs](https://docs.anthropic.com/en/docs/use-xml-tags) -- OpenAI function-calling reliability guidance (clear tool descriptions, when/when-not tool usage, small callable surface): [OpenAI function calling guide](https://developers.openai.com/docs/guides/function-calling) -- OpenAI prompt engineering (instruction hierarchy and explicit output contracts): [OpenAI prompt engineering guide](https://developers.openai.com/api/docs/guides/prompt-engineering) -- LangChain supervisor/subagents guidance (clear subagent names/descriptions, context engineering, routing intent): [LangChain supervisor docs](https://docs.langchain.com/oss/python/langchain/supervisor), [LangChain subagents docs](https://docs.langchain.com/oss/python/langchain/multi-agent/subagents) - -## Current weakness audit (as of now) - -- `supervisor/supervisor_prompt.md` is short and does not define decision policy for ambiguous/multi-domain tasks. -- Most expert `domain_prompt.md` files are one-line role statements with no: - - scope boundaries and refusal policy, - - parameter-resolution behavior, - - completion criteria (what must be returned), - - failure handling rules, - - concrete examples. -- Tool descriptions in routing are generic ("Pass a clear natural-language task"), which weakens handoff quality. - -## Prompt design standards (required) - -Apply these standards to supervisor and every expert prompt. - -1. **Role + objective first** - - One sentence for identity. - - One sentence for success criterion. - -2. **Explicit routing/usage rules** - - Tell the model when to use this agent/tool. - - Tell it when not to use it. - - Include ambiguity fallback ("ask one clarifying question" or "do X conservative default"). - -3. **Structured task contract** - - Require concise but complete execution reports. - - Require IDs/links/timestamps when tool outputs produce them. - - For no-op paths, explain why no action was taken. - -4. **Safety + reliability contract** - - Never fabricate tool results. - - Never claim action if no successful tool call happened. - - Surface irreversible/risky actions clearly. - -5. **Few-shot examples** - - Include 2-4 minimal examples per domain: - - direct success, - - ambiguous input, - - out-of-scope reroute. - -6. **Concise formatting rules** - - Avoid verbosity. - - Stable response structure improves orchestration and observability. - -## Supervisor prompt blueprint - -The supervisor prompt should contain these sections in order: - -1. `Role` -2. `Available experts` (name + scope + non-scope) -3. `Delegation policy` - - single-domain -> one expert - - multi-domain -> sequence or parallel where independent - - no expert needed -> answer directly -4. `Task-writing policy` for delegated calls - - include user goal, constraints, success criteria - - include only needed context -5. `Result synthesis policy` - - merge expert outputs into one user-facing response - - preserve concrete identifiers from expert outputs -6. `Failure policy` - - retry on recoverable mismatch - - ask clarifying question when required field is missing - -## Expert prompt blueprint (per domain) - -Each `domain_prompt.md` should include: - -1. `Role and scope` -2. `In-scope actions` (mapped to the exact provided tools) -3. `Out-of-scope behavior` (what to return for reroute) -4. `Execution rules` - - choose the minimum tool sequence that satisfies request - - do not guess IDs or parameters - - ask concise clarification only when necessary -5. `Output contract` - - action summary - - concrete artifacts/IDs/links generated - - unresolved items and next step -6. `Examples` (2-4 realistic, short) - -## Domain-specific tuning checklist - -- `research`: enforce source-grounded summaries and explicit uncertainty. -- `memory`: strict save criteria (durable preference/fact only) and secret-handling policy. -- `deliverables`: require output artifact references and constraints echo. -- `gmail` / `calendar`: require recipient/date-time disambiguation policy and timezone handling. -- `docs connectors` (`notion`, `confluence`, `drive`, `dropbox`, `onedrive`): require exact page/file target resolution before mutate actions. -- chat connectors (`discord`, `teams`, `slack`): require channel/thread context clarity before send actions. -- MCP experts: require strict tool-description adherence and no assumption about unavailable endpoints. - -## Tool description tuning rules (routing layer) - -Routing tool descriptions should include: - -- best-fit task types, -- disallowed task types, -- required task payload hints (e.g. "include recipient + intent + constraints"), -- expected result shape. - -This is especially important because supervisor tool choice is heavily influenced by `name + description`. - -## Evaluation plan (before wiring to production) - -Create a prompt eval set with at least 20 tasks: - -- 8 single-domain tasks, -- 6 ambiguous tasks (should clarify or route conservatively), -- 6 multi-domain tasks (should sequence experts correctly). - -Track: - -- routing accuracy, -- unnecessary delegation rate, -- tool-call success rate, -- clarification precision (ask only when needed), -- final answer completeness. - -Use same test set against: - -- current prompts, -- tuned prompts v1, -- tuned prompts v2. - -Promote only when v2 improves routing accuracy and reduces unnecessary delegation with no regression in tool-call success. - -## Immediate implementation plan - -1. Rewrite `supervisor/supervisor_prompt.md` using the supervisor blueprint. -2. Rewrite all expert `domain_prompt.md` files with the expert blueprint. -3. Upgrade routing tool descriptions in `routing/supervisor_routing.py` to add "when to use / when not to use". -4. Add a lightweight prompt eval script or fixture set for reproducible tuning. - -## Definition of done - -- Every supervisor/expert prompt has explicit scope, failure policy, and output contract. -- Every route description encodes clear decision boundaries. -- Prompt eval shows measurable gains on routing accuracy and lower unnecessary delegation. -- Team can iterate prompt versions without changing core orchestration code. - diff --git a/surfsense_backend/app/tasks/chat/stream_new_chat.py b/surfsense_backend/app/tasks/chat/stream_new_chat.py index c9ae38f24..3339ff10a 100644 --- a/surfsense_backend/app/tasks/chat/stream_new_chat.py +++ b/surfsense_backend/app/tasks/chat/stream_new_chat.py @@ -28,6 +28,7 @@ from sqlalchemy import func from sqlalchemy.future import select from sqlalchemy.orm import selectinload +from app.agents.multi_agent_chat.integration import create_multi_agent_chat from app.agents.new_chat.chat_deepagent import create_surfsense_deep_agent from app.agents.new_chat.checkpointer import get_checkpointer from app.agents.new_chat.filesystem_selection import FilesystemMode, FilesystemSelection @@ -38,7 +39,6 @@ from app.agents.new_chat.llm_config import ( load_agent_config, load_global_llm_config_by_id, ) -from app.agents.multi_agent_chat.integration import create_multi_agent_chat from app.agents.new_chat.memory_extraction import ( extract_and_save_memory, extract_and_save_team_memory, @@ -69,6 +69,7 @@ from app.utils.user_message_multimodal import build_human_message_content _background_tasks: set[asyncio.Task] = set() _perf_log = get_perf_logger() +logger = logging.getLogger(__name__) def format_mentioned_surfsense_docs_as_context(