agent: retire eager KB priority/planner path and its dead flags

The pull-based KB design (on-demand search_knowledge_base tool + pre-injected
workspace tree) fully replaced the old eager retrieval path. Remove its last
remnants:

- Delete KnowledgePriorityMiddleware (knowledge_search.py) and its tests.
- Drop the kb_priority state field + reducer default; trim
  KbContextProjectionMiddleware to project only workspace_tree_text.
- Remove the now-dead feature flags enable_kb_priority_preinjection and
  enable_kb_planner_runnable across backend (flags, route schema, tests,
  env examples) and frontend (settings toggle, zod schema).
- Scrub <priority_documents> and stale KnowledgePriorityMiddleware references
  from prompts, docstrings, and the ADR.

No functional change: nothing wrote kb_priority and neither flag gated live
behavior after the cutover. Full backend suite green (pre-existing unrelated
failures aside).
This commit is contained in:
CREDO23 2026-06-25 18:37:14 +02:00
parent 0148647b98
commit 2beafbdec8
34 changed files with 62 additions and 1890 deletions

View file

@ -6,8 +6,6 @@ read-only). This middleware loads it once on the first turn into
* :class:`KnowledgeTreeMiddleware` can render the synthetic ``/documents``
view without touching the DB.
* :class:`KnowledgePriorityMiddleware` skips hybrid search and emits a
degenerate priority list.
* :class:`KBPostgresBackend` (``als_info`` / ``aread`` / ``_load_file_data``)
recognises the synthetic path.

View file

@ -8,11 +8,6 @@ standing instructions. It also reports current character usage versus the
hard limit so you can manage the budget. Treat it as background colour for
your answer, not as the task itself.
`<priority_documents>` lists the workspace documents most relevant to the
latest user message, ranked by relevance score, with `[USER-MENTIONED]`
flagged on anything the user explicitly referenced. When the task is about
workspace content, read these first.
`<workspace_tree>` shows the full `/documents/` folder and file layout. Use
it to resolve paths the user describes in natural language ("my Q2 roadmap",
"last week's meeting notes") into concrete document references before

View file

@ -7,11 +7,6 @@ decisions, conventions, architecture notes, processes, key facts. It also
reports current character usage versus the hard limit so you can manage the
budget. Treat it as background colour for your answer, not as the task itself.
`<priority_documents>` lists the workspace documents most relevant to the
latest user message, ranked by relevance score, with `[USER-MENTIONED]`
flagged on anything someone in the thread explicitly referenced. When the
task is about workspace content, read these first.
`<workspace_tree>` shows the full `/documents/` folder and file layout. Use
it to resolve paths described in natural language ("the Q2 roadmap", "last
week's planning notes") into concrete document references before delegating

View file

@ -14,5 +14,5 @@ Workflow (Understand → Plan → Act → Verify):
Discipline:
- Do not imply access to connectors, MCP tools, or deliverable generators except via **task**.
- Pass paths to **task(knowledge_base, …)** only when you saw them in `<workspace_tree>` or `<priority_documents>`. Otherwise describe the document in natural language and let the subagent resolve it.
- Pass paths to **task(knowledge_base, …)** only when you saw them in `<workspace_tree>`. Otherwise describe the document in natural language and let the subagent resolve it.
</provider_hints>

View file

@ -53,14 +53,6 @@ class AgentFeatureFlags:
# Skills + subagents
enable_skills: bool = True
enable_specialized_subagents: bool = True
enable_kb_planner_runnable: bool = True
# KB retrieval mode — when False (default), the main agent retrieves KB
# content lazily via the on-demand ``search_knowledge_base`` tool and the
# expensive per-turn pre-injection (planner LLM + embed + hybrid search,
# ~2.3s) is skipped; explicit @-mentions are still surfaced cheaply. Set
# True to restore the original eager ``<priority_documents>`` pre-injection.
enable_kb_priority_preinjection: bool = False
# Snapshot / revert
enable_action_log: bool = True
@ -118,9 +110,6 @@ class AgentFeatureFlags:
enable_llm_tool_selector=False,
enable_skills=False,
enable_specialized_subagents=False,
enable_kb_planner_runnable=False,
# Full rollback restores the original eager KB pre-injection.
enable_kb_priority_preinjection=True,
enable_action_log=False,
enable_revert_route=False,
enable_plugin_loader=False,
@ -156,12 +145,6 @@ class AgentFeatureFlags:
enable_specialized_subagents=_env_bool(
"SURFSENSE_ENABLE_SPECIALIZED_SUBAGENTS", True
),
enable_kb_planner_runnable=_env_bool(
"SURFSENSE_ENABLE_KB_PLANNER_RUNNABLE", True
),
enable_kb_priority_preinjection=_env_bool(
"SURFSENSE_ENABLE_KB_PRIORITY_PREINJECTION", False
),
# Snapshot / revert
enable_action_log=_env_bool("SURFSENSE_ENABLE_ACTION_LOG", True),
enable_revert_route=_env_bool("SURFSENSE_ENABLE_REVERT_ROUTE", True),
@ -198,7 +181,6 @@ class AgentFeatureFlags:
self.enable_llm_tool_selector,
self.enable_skills,
self.enable_specialized_subagents,
self.enable_kb_planner_runnable,
self.enable_action_log,
self.enable_revert_route,
self.enable_plugin_loader,

View file

@ -44,12 +44,6 @@ to page through a large document. Cite a passage by writing its `[n]` after the
statement it supports the same `[n]` that passage had in
`search_knowledge_base` results.
## Priority List
You receive a `<priority_documents>` system message each turn listing the
top-K paths most relevant to the user's query (by hybrid search). Read those
first.
## Workspace Tree
You receive a `<workspace_tree>` system message each turn with the current

View file

@ -37,13 +37,4 @@ directory (`cwd`).
- Cross-mount moves are not supported.
- Desktop deletes hit disk immediately and cannot be undone via the
agent's revert flow — confirm before calling `rm`/`rmdir`.
## Priority List
You may receive a `<priority_documents>` system message listing the top-K
documents from the user's SurfSense knowledge base — these are cloud-ingested
via connectors (Notion, Slack, etc.), not local files. Treat it as a hint:
consult it when the task spans both local and cloud sources (e.g. drafting a
local note from a Notion summary); skip when the task is purely about local
files.
"""

View file

@ -1,4 +1,4 @@
"""Project ``workspace_tree_text`` + ``kb_priority`` from state into SystemMessages."""
"""Project ``workspace_tree_text`` from state into a SystemMessage."""
from __future__ import annotations
@ -14,18 +14,15 @@ from app.agents.chat.multi_agent_chat.shared.state.filesystem_state import (
)
from app.utils.perf import get_perf_logger
from .knowledge_search import _render_priority_message
_perf_log = get_perf_logger()
class KbContextProjectionMiddleware(AgentMiddleware): # type: ignore[type-arg]
"""Emit ``<workspace_tree>`` + ``<priority_documents>`` from shared state.
"""Emit the ``<workspace_tree>`` from shared state.
Read-only consumer: no DB, no LLM, no state writes. The orchestrator's
renderer middlewares populate the source fields; this projection lets any
agent (orchestrator or subagent) put the same content in front of its
own LLM call.
``KnowledgeTreeMiddleware`` populates ``workspace_tree_text``; this
projection lets a subagent put the same tree in front of its own LLM call.
"""
tools = ()
@ -39,28 +36,19 @@ class KbContextProjectionMiddleware(AgentMiddleware): # type: ignore[type-arg]
del runtime
start = time.perf_counter()
tree_text = state.get("workspace_tree_text")
priority = state.get("kb_priority")
if not tree_text and not priority:
if not tree_text:
_perf_log.info(
"[kb_context_projection] tree=0 priority=0 elapsed=%.3fs",
"[kb_context_projection] tree=0 elapsed=%.3fs",
time.perf_counter() - start,
)
return None
messages = list(state.get("messages") or [])
insert_at = max(len(messages) - 1, 0)
tree_chars = 0
if tree_text:
tree_chars = len(tree_text)
messages.insert(insert_at, SystemMessage(content=tree_text))
priority_count = 0
if priority:
priority_count = len(priority) if hasattr(priority, "__len__") else 1
messages.insert(insert_at, _render_priority_message(priority))
messages.insert(insert_at, SystemMessage(content=tree_text))
_perf_log.info(
"[kb_context_projection] tree_chars=%d priority_items=%d elapsed=%.3fs",
tree_chars,
priority_count,
"[kb_context_projection] tree_chars=%d elapsed=%.3fs",
len(tree_text),
time.perf_counter() - start,
)
return {"messages": messages}

View file

@ -13,7 +13,6 @@ extra fields needed to implement Postgres-backed virtual filesystem semantics:
* ``dirty_paths`` paths whose state file content differs from DB.
* ``dirty_path_tool_calls`` sidecar map ``path -> latest tool_call_id`` for
dirty paths; used to bind the per-path snapshot to an action_id.
* ``kb_priority`` top-K priority hints rendered into a system message.
* ``kb_anon_doc`` Redis-loaded anonymous document (if any).
* ``citation_registry`` per-conversation ``[n]`` -> source map for citations.
* ``tree_version`` bumped by persistence; invalidates the tree render cache.
@ -69,14 +68,6 @@ class PendingDelete(TypedDict, total=False):
tool_call_id: str
class KbPriorityEntry(TypedDict, total=False):
path: str
score: float
document_id: int | None
title: str
mentioned: bool
class KbAnonDoc(TypedDict, total=False):
"""In-memory anonymous-session document loaded from Redis."""
@ -161,9 +152,6 @@ class SurfSenseFilesystemState(FilesystemState):
to the latest action_id (the one the user is most likely to revert).
"""
kb_priority: NotRequired[Annotated[list[KbPriorityEntry], _replace_reducer]]
"""Top-K priority hints rendered as a system message before the user turn."""
kb_anon_doc: NotRequired[Annotated[KbAnonDoc | None, _replace_reducer]]
"""Anonymous-session document loaded from Redis (read-only, no DB row)."""
@ -212,7 +200,6 @@ class SurfSenseFilesystemState(FilesystemState):
__all__ = [
"KbAnonDoc",
"KbPriorityEntry",
"PendingDelete",
"PendingMove",
"SurfSenseFilesystemState",

View file

@ -2,7 +2,7 @@
These reducers back the extra state fields used by the cloud-mode filesystem
agent (`cwd`, `staged_dirs`, `pending_moves`, `dirty_paths`, `doc_id_by_path`,
`kb_priority`, `kb_anon_doc`, `tree_version`).
`kb_anon_doc`, `tree_version`).
Tools mutate these fields ONLY via `Command(update={...})` returns; the
reducers are responsible for merging successive updates atomically and for
@ -258,7 +258,6 @@ def _initial_filesystem_state() -> dict[str, Any]:
"doc_id_by_path": {},
"dirty_paths": [],
"dirty_path_tool_calls": {},
"kb_priority": [],
"kb_anon_doc": None,
"tree_version": 0,
}

View file

@ -6,10 +6,9 @@ You are the SurfSense knowledge base specialist for the user's `/documents/` wor
- If the supervisor already provided a precise path (e.g. `/documents/notes/2026-05-11.md`), use it directly — skip the lookup steps below.
- Otherwise, most requests reference documents by description (`"my meeting notes from last week"`, `"the design doc"`). Resolve them yourself:
1. Consult `<priority_documents>` — it's a hint about top-K likely matches, not a directive. Skip when the ranked entries don't fit the task.
2. Walk `<workspace_tree>` for descriptive folder/filename matches.
3. Use the `glob` tool for filename patterns the tree didn't surface, and the `grep` tool when the description points at *content* rather than a name.
4. Only return `status=blocked` with `missing_fields=["path"]` when the description is genuinely ambiguous after a thorough lookup.
1. Walk `<workspace_tree>` for descriptive folder/filename matches.
2. Use the `glob` tool for filename patterns the tree didn't surface, and the `grep` tool when the description points at *content* rather than a name.
3. Only return `status=blocked` with `missing_fields=["path"]` when the description is genuinely ambiguous after a thorough lookup.
For writes (where you choose the path yourself):
@ -89,7 +88,7 @@ A KB document reads back like this — only the bracketed `[n]` is a citation la
**Example 2 — edit by inference:**
- *Supervisor task:* `"Add a bullet about the new feature flag to my Q2 roadmap"`
- *You:* search for the roadmap doc — check `<priority_documents>` and `<workspace_tree>` first; if neither surfaces it, widen with the `glob` tool (try filename patterns the user's language suggests) or the `grep` tool (search by content). Suppose `<priority_documents>` hits `/documents/planning/q2-roadmap.md``read_file("/documents/planning/q2-roadmap.md")``edit_file("/documents/planning/q2-roadmap.md", old, new)` → success.
- *You:* search for the roadmap doc — check `<workspace_tree>` first; if it doesn't surface the doc, widen with the `glob` tool (try filename patterns the user's language suggests) or the `grep` tool (search by content). Suppose the tree hits `/documents/planning/q2-roadmap.md``read_file("/documents/planning/q2-roadmap.md")``edit_file("/documents/planning/q2-roadmap.md", old, new)` → success.
- *Output:* `status=success`, evidence includes path and the inserted snippet.
**Example 3 — blocked, multiple candidates:**

View file

@ -9,8 +9,7 @@ You are the SurfSense workspace specialist for the user's local folders.
1. If you do not know which mounts exist, call `ls('/')` first.
2. Walk likely folders with the `ls` and `list_tree` tools.
3. Use the `glob` tool for filename patterns; use the `grep` tool when the description points at *content* rather than a name.
4. `<priority_documents>` lists top-K cloud-ingested docs, not local files — consult it only when the task spans both worlds (e.g. drafting a local note from a Notion source). Skip otherwise.
5. Only return `status=blocked` with `missing_fields=["path"]` when the description is genuinely ambiguous after a thorough lookup.
4. Only return `status=blocked` with `missing_fields=["path"]` when the description is genuinely ambiguous after a thorough lookup.
For writes (where you choose the path yourself):

View file

@ -6,9 +6,8 @@ You answer workspace questions for another agent. The end user does **not** see
The caller's question often references documents by description (`"my meeting notes from last week"`, `"the design doc"`). Resolve them yourself:
1. Consult `<priority_documents>` — a hint about top-K likely matches, not a directive. Skip when the ranked entries don't fit.
2. Walk `<workspace_tree>` for descriptive folder/filename matches.
3. Use `glob` for filename patterns the tree didn't surface, and `grep` when the description points at *content* rather than a name.
1. Walk `<workspace_tree>` for descriptive folder/filename matches.
2. Use `glob` for filename patterns the tree didn't surface, and `grep` when the description points at *content* rather than a name.
If a precise path was already given, use it directly — skip the lookup.

View file

@ -9,7 +9,6 @@ The caller's question often references files by description (`"my meeting notes
1. If you do not know which mounts exist, call `ls('/')` first.
2. Walk likely folders with the `ls` and `list_tree` tools.
3. Use `glob` for filename patterns; use `grep` when the description points at *content* rather than a name.
4. `<priority_documents>` lists top-K cloud-ingested docs, not local files — consult it only when the task spans both worlds (e.g. drafting a local note from a Notion source). Skip otherwise.
If a precise path was already given, use it directly — skip the lookup.

View file

@ -74,8 +74,9 @@ class ResolvedMentionSet:
``@Project``).
``mentioned_document_ids`` is an ordered, deduped list consumed by
the priority middleware downstream see
``KnowledgePriorityMiddleware._compute_priority_paths``.
the on-demand ``search_knowledge_base`` tool downstream (via
``referenced_document_ids``) to pin @-mentioned docs into the
retrieval scope.
"""
mentions: list[ResolvedMention] = field(default_factory=list)
@ -113,8 +114,8 @@ async def resolve_mentions(
* Legacy clients that haven't migrated to the unified chip list
still send the id arrays we treat the union as authoritative.
* The id arrays are the canonical input to
``KnowledgePriorityMiddleware`` (via ``SurfSenseContextSchema``);
* The id arrays are the canonical input to the retrieval scope
(via ``SurfSenseContextSchema`` ``referenced_document_ids``);
returning the deduped, validated lists lets the route forward
them unchanged.

View file

@ -4,7 +4,6 @@ This module is the single source of truth for mapping ``Document`` rows to
virtual paths under ``/documents/`` and back. It is used by:
* :class:`KnowledgeTreeMiddleware` (rendering the workspace tree)
* :class:`KnowledgePriorityMiddleware` (computing priority paths)
* :class:`KBPostgresBackend` (``als_info`` / ``aread`` / move operations)
* :class:`KnowledgeBasePersistenceMiddleware` (resolving moves and creates)

View file

@ -11,9 +11,9 @@ MUST live on this context object instead of being captured into a
middleware ``__init__`` closure. Middlewares read fields back via
``runtime.context.<field>``; tools read them via ``runtime.context``.
This object is read inside both ``KnowledgePriorityMiddleware`` (for
``mentioned_document_ids``) and any future middleware that needs
per-request state without invalidating the compiled-agent cache.
This object is read by the ``search_knowledge_base`` tool (for
``mentioned_document_ids``) and any middleware that needs per-request
state without invalidating the compiled-agent cache.
"""
from __future__ import annotations
@ -43,13 +43,12 @@ class SurfSenseContextSchema:
Phase 1.5 fields:
search_space_id: Search space the request is scoped to.
mentioned_document_ids: KB documents the user @-mentioned this turn.
Read by ``KnowledgePriorityMiddleware`` to seed its priority
list. Stays out of the compiled-agent cache key that's the
whole point of putting it here.
Read by the ``search_knowledge_base`` tool to pin these docs
into the retrieval scope. Stays out of the compiled-agent cache
key that's the whole point of putting it here.
mentioned_folder_ids: KB folders the user @-mentioned this turn
(cloud filesystem mode). Surfaced as ``[USER-MENTIONED]``
entries in ``<priority_documents>`` so the agent prioritises
walking those folders with ``ls`` / ``find_documents``.
(cloud filesystem mode). Pinned into the ``search_knowledge_base``
retrieval scope so matches from those folders are prioritised.
file_operation_contract: One-shot file operation contract for the
upcoming turn (reserved; not currently populated).
turn_id / request_id: Correlation IDs surfaced by the streaming

View file

@ -4,7 +4,7 @@ Extends ``SummarizationMiddleware`` with three SurfSense behaviors:
1. A structured summary template (:data:`SURFSENSE_SUMMARY_PROMPT`) instead of
the base freeform prompt.
2. Protected SystemMessages (injected hints like ``<priority_documents>``) are
2. Protected SystemMessages (injected hints like ``<workspace_tree>``) are
kept verbatim instead of being summarized away.
3. ``content=None`` is sanitized before ``get_buffer_string`` (some providers
stream tool-only AIMessages with ``None`` content, which would crash it).
@ -77,7 +77,6 @@ Respond ONLY with the structured summary. Do not include any text before or afte
# compaction step happens *before* re-injection in some paths, so we
# must preserve them verbatim across the cutoff.
PROTECTED_SYSTEM_PREFIXES: tuple[str, ...] = (
"<priority_documents>", # KnowledgePriorityMiddleware
"<workspace_tree>", # KnowledgeTreeMiddleware
"<file_operation_contract>", # reserved file-operation contract prefix
"<user_memory>", # MemoryInjectionMiddleware

View file

@ -78,7 +78,7 @@ async def _resolve_mention_context(
Automation always runs in cloud filesystem mode, so we mirror the chat
``new_chat`` flow: substitute ``@title`` tokens with canonical
``/documents/...`` paths, prepend a ``<mentioned_connectors>`` block, and
build a ``SurfSenseContextSchema`` that ``KnowledgePriorityMiddleware``
build a ``SurfSenseContextSchema`` that the ``search_knowledge_base`` tool
reads via ``runtime.context``. Returns ``(query, None)`` unchanged when
there are no mentions.
"""
@ -210,7 +210,7 @@ async def run_agent_task(
runtime_context.turn_id = turn_id
# The compiled graph declares ``context_schema=SurfSenseContextSchema``;
# mentions only reach ``KnowledgePriorityMiddleware`` via ``context=``.
# mentions only reach the ``search_knowledge_base`` tool via ``context=``.
invoke_kwargs: dict[str, Any] = {"config": config}
if runtime_context is not None:
invoke_kwargs["context"] = runtime_context

View file

@ -53,7 +53,6 @@ class AgentFeatureFlagsRead(BaseModel):
enable_skills: bool
enable_specialized_subagents: bool
enable_kb_planner_runnable: bool
enable_action_log: bool
enable_revert_route: bool

View file

@ -246,10 +246,10 @@ class NewChatRequest(BaseModel):
description=(
"Optional knowledge-base folder IDs the user mentioned with "
"@. Resolved to virtual paths (``/documents/.../``) by "
"``mention_resolver`` and surfaced to the agent via "
"(a) backtick-wrapped substitution in ``user_query`` and "
"(b) a ``[USER-MENTIONED]`` entry in ``<priority_documents>``. "
"The agent's ``ls`` tool can then walk the folder itself."
"``mention_resolver``, surfaced to the agent via backtick-wrapped "
"substitution in ``user_query`` and pinned into the "
"``search_knowledge_base`` retrieval scope. The agent's ``ls`` "
"tool can then walk the folder itself."
),
)
mentioned_documents: list[MentionedDocumentInfo] | None = Field(

View file

@ -22,7 +22,8 @@ def build_new_chat_runtime_context(
request_id: str | None,
turn_id: str,
) -> SurfSenseContextSchema:
"""``mentioned_document_ids`` is consumed by ``KnowledgePriorityMiddleware``.
"""``mentioned_document_ids`` is consumed by the ``search_knowledge_base``
tool (via ``referenced_document_ids``) to pin mentioned docs into scope.
``accepted_folder_ids`` (post-resolve) wins over the raw
``mentioned_folder_ids`` from the request: the resolver drops chips that