feat: refine private and team memory protocols

surfsense_backend/app/agents/multi_agent_chat/main_agent/system_prompt/prompts/memory_protocol/private.md

@@ -6,4 +6,10 @@ standing instructions?
 If yes, call `update_memory` **alongside** your normal response — don't
 defer it to a later turn. Skip ephemeral chat noise (one-off Q/A, greetings,
 session logistics). Stay within the budget shown in `<user_memory>`.
+
+Memory is heading-based markdown. New entries should be under `##` headings
+such as `## Facts`, `## Preferences`, or `## Instructions`, with bullets like
+`- YYYY-MM-DD: text`. If existing memory contains legacy
+`(YYYY-MM-DD) [fact|pref|instr]` markers, preserve the information but write
+new saves in the heading-based format.
 </memory_protocol>

surfsense_backend/app/agents/multi_agent_chat/main_agent/system_prompt/prompts/memory_protocol/team.md

@@ -6,4 +6,12 @@ key facts?
 If yes, call `update_memory` **alongside** your normal response — don't
 defer it to a later turn. Skip ephemeral chat noise (one-off Q/A, greetings,
 session logistics). Stay within the budget shown in `<team_memory>`.
+
+Team memory is heading-based markdown. New entries should be under `##`
+headings such as `## Product Decisions`, `## Engineering Conventions`,
+`## Project Facts`, or `## Open Questions`, with bullets like
+`- YYYY-MM-DD: text`. If existing memory contains legacy `(YYYY-MM-DD) [fact]`
+markers, preserve the information but write new saves in the heading-based
+format. Do not create personal headings such as `## Preferences` or
+`## Instructions`.
 </memory_protocol>

surfsense_backend/app/agents/multi_agent_chat/main_agent/system_prompt/prompts/tools/update_memory/private/description.md

@@ -9,7 +9,9 @@
   - Skip ephemeral chat noise (one-off Q/A, greetings, session logistics).
   - Args: `updated_memory` — FULL replacement markdown (merge and curate,
     don't only append).
-  - Formatting: bullets `- (YYYY-MM-DD) [marker] text` with markers `[fact]`,
-    `[pref]`, `[instr]` (priority when trimming: `instr > pref > fact`).
-    Group bullets under short `##` headings; stay under the limit shown in
-    `<user_memory>`.
+  - Formatting: heading-based markdown with entries under `##` headings.
+    Recommended headings are `## Facts`, `## Preferences`, `## Instructions`,
+    though clearer natural headings are allowed. New bullets should look like
+    `- YYYY-MM-DD: text`; stay under the limit shown in `<user_memory>`.
+  - If existing memory uses legacy `(YYYY-MM-DD) [fact|pref|instr]` markers,
+    preserve the information but write the updated document in the new format.

surfsense_backend/app/agents/multi_agent_chat/main_agent/system_prompt/prompts/tools/update_memory/private/example.md

@@ -1,28 +1,28 @@
 <example>
 <user_name>Alex</user_name>, <user_memory> is empty.
 user: "I'm a space enthusiast, explain astrophage to me"
-→ update_memory(updated_memory="## Interests & background\n- (2025-03-15) [fact] Alex is a space enthusiast\n")
+→ update_memory(updated_memory="## Facts\n- 2025-03-15: Alex is a space enthusiast\n")
 (Casual durable fact; use first name, neutral heading.)
 </example>
 
 <example>
 user: "Remember that I prefer concise answers over detailed explanations"
-→ update_memory(updated_memory="## Interests & background\n- (2025-03-15) [fact] Alex is a space enthusiast\n\n## Response style\n- (2025-03-15) [pref] Alex prefers concise answers over detailed explanations\n")
+→ update_memory(updated_memory="## Facts\n- 2025-03-15: Alex is a space enthusiast\n\n## Preferences\n- 2025-03-15: Alex prefers concise answers over detailed explanations\n")
 (Durable preference; merge with existing memory.)
 </example>
 
 <example>
 user: "I actually moved to Tokyo last month"
-→ update_memory(updated_memory="...\n\n## Personal context\n- (2025-03-15) [fact] Alex lives in Tokyo (previously London)\n...")
+→ update_memory(updated_memory="...\n\n## Facts\n- 2025-03-15: Alex lives in Tokyo (previously London)\n...")
 (Updated fact; date reflects when recorded.)
 </example>
 
 <example>
 user: "I'm a freelance photographer working on a nature documentary"
-→ update_memory(updated_memory="...\n\n## Current focus\n- (2025-03-15) [fact] Alex is a freelance photographer\n- (2025-03-15) [fact] Alex is working on a nature documentary\n")
+→ update_memory(updated_memory="...\n\n## Current Focus\n- 2025-03-15: Alex is a freelance photographer\n- 2025-03-15: Alex is working on a nature documentary\n")
 </example>
 
 <example>
 user: "Always respond in bullet points"
-→ update_memory(updated_memory="...\n\n## Response style\n- (2025-03-15) [instr] Always respond to Alex in bullet points\n")
+→ update_memory(updated_memory="...\n\n## Instructions\n- 2025-03-15: Always respond to Alex in bullet points\n")
 </example>

surfsense_backend/app/agents/multi_agent_chat/main_agent/system_prompt/prompts/tools/update_memory/team/description.md

@@ -9,8 +9,14 @@
   - Skip ephemeral chat noise (one-off Q/A, greetings, session logistics).
   - Args: `updated_memory` — FULL replacement markdown (merge and curate,
     don't only append).
-  - Formatting: bullets `- (YYYY-MM-DD) [fact] text`. Team memory uses ONLY
-    the `[fact]` marker (never `[pref]` or `[instr]`). Group bullets under
-    short `##` headings (2-3 words each); stay under the limit shown in
-    `<team_memory>`. When trimming, prioritise: decisions/conventions > key
-    facts > current priorities.
+  - Formatting: heading-based markdown with entries under `##` headings.
+    Recommended headings are `## Product Decisions`,
+    `## Engineering Conventions`, `## Project Facts`, and `## Open Questions`.
+    New bullets should look like `- YYYY-MM-DD: text`; stay under the limit
+    shown in `<team_memory>`.
+  - If existing memory uses legacy `(YYYY-MM-DD) [fact]` markers, preserve the
+    information but write the updated document in the new format.
+  - Do not create personal headings such as `## Preferences`,
+    `## Instructions`, `## Personal Notes`, or `## Personal Instructions`.
+    When trimming, prioritise: decisions/conventions > key facts > current
+    priorities.

surfsense_backend/app/agents/multi_agent_chat/main_agent/system_prompt/prompts/tools/update_memory/team/example.md

@@ -1,9 +1,9 @@
 <example>
 user: "Let's remember that we decided to do weekly standup meetings on Mondays"
-→ update_memory(updated_memory="...\n\n## Team rituals\n- (2025-03-15) [fact] Weekly standup meetings on Mondays\n...")
+→ update_memory(updated_memory="...\n\n## Product Decisions\n- 2025-03-15: Weekly standup meetings happen on Mondays\n...")
 </example>
 
 <example>
 user: "Our office is in downtown Seattle, 5th floor"
-→ update_memory(updated_memory="...\n\n## Workspace\n- (2025-03-15) [fact] Office location: downtown Seattle, 5th floor\n...")
+→ update_memory(updated_memory="...\n\n## Project Facts\n- 2025-03-15: Office location is downtown Seattle, 5th floor\n...")
 </example>

surfsense_backend/app/agents/multi_agent_chat/subagents/builtins/memory/system_prompt.md

@@ -18,6 +18,10 @@ Persist durable preferences/facts/instructions with `update_memory` while avoidi
 - Do not store transient chatter.
 - Do not store secrets unless explicitly instructed.
 - If memory intent is unclear, return `status=blocked` with the missing intent signal.
+- Persisted memory is heading-based markdown. New saved bullets should look like
+  `- YYYY-MM-DD: text` under `##` headings. If existing memory has legacy
+  `(YYYY-MM-DD) [fact|pref|instr]` markers, preserve the information but write
+  the updated document in the heading-based format.
 </tool_policy>
 
 <out_of_scope>
@@ -53,4 +57,7 @@ Rules:
 - `status=success` -> `next_step=null`, `missing_fields=null`.
 - `status=partial|blocked|error` -> `next_step` must be non-null.
 - `status=blocked` due to missing required inputs -> `missing_fields` must be non-null.
+- `evidence.memory_category` is a semantic classification for supervisor logs
+  only. It is not the persisted storage format and must not force inline
+  `[fact|preference|instruction]` markers into saved memory.
 </output_contract>

surfsense_backend/app/agents/new_chat/middleware/memory_injection.py

@@ -17,8 +17,8 @@ from langgraph.runtime import Runtime
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.agents.new_chat.tools.update_memory import MEMORY_HARD_LIMIT, MEMORY_SOFT_LIMIT
 from app.db import ChatVisibility, SearchSpace, User, shielded_async_session
+from app.services.memory import MEMORY_HARD_LIMIT, MEMORY_SOFT_LIMIT
 
 logger = logging.getLogger(__name__)
 

surfsense_backend/app/agents/new_chat/prompts/base/memory_protocol_private.md

@@ -3,4 +3,10 @@ IMPORTANT — After understanding each user message, ALWAYS check: does this mes
 reveal durable facts about the user (role, interests, preferences, projects,
 background, or standing instructions)? If yes, you MUST call update_memory
 alongside your normal response — do not defer this to a later turn.
+
+Memory is stored as a heading-based markdown document. New entries should be
+under `##` headings such as `## Facts`, `## Preferences`, or `## Instructions`
+with bullets like `- YYYY-MM-DD: text`. If existing memory contains legacy
+`(YYYY-MM-DD) [fact|pref|instr]` markers, preserve the information but write
+new saves in the heading-based format.
 </memory_protocol>

surfsense_backend/app/agents/new_chat/prompts/base/memory_protocol_team.md

@@ -3,4 +3,12 @@ IMPORTANT — After understanding each user message, ALWAYS check: does this mes
 reveal durable facts about the team (decisions, conventions, architecture, processes,
 or key facts)? If yes, you MUST call update_memory alongside your normal response —
 do not defer this to a later turn.
+
+Team memory is stored as a heading-based markdown document. New entries should
+be under `##` headings such as `## Product Decisions`,
+`## Engineering Conventions`, `## Project Facts`, or `## Open Questions` with
+bullets like `- YYYY-MM-DD: text`. If existing memory contains legacy
+`(YYYY-MM-DD) [fact]` markers, preserve the information but write new saves in
+the heading-based format. Do not create personal headings such as
+`## Preferences` or `## Instructions`.
 </memory_protocol>

surfsense_backend/app/agents/new_chat/prompts/examples/update_memory_private.md

@@ -1,16 +1,16 @@
 
 - <user_name>Alex</user_name>, <user_memory> is empty. User: "I'm a space enthusiast, explain astrophage to me"
-  - The user casually shared a durable fact. Use their first name in the entry, short neutral heading:
-    update_memory(updated_memory="## Interests & background\n- (2025-03-15) [fact] Alex is a space enthusiast\n")
+  - The user casually shared a durable fact:
+    update_memory(updated_memory="## Facts\n- 2025-03-15: Alex is a space enthusiast\n")
 - User: "Remember that I prefer concise answers over detailed explanations"
-  - Durable preference. Merge with existing memory, add a new heading:
-    update_memory(updated_memory="## Interests & background\n- (2025-03-15) [fact] Alex is a space enthusiast\n\n## Response style\n- (2025-03-15) [pref] Alex prefers concise answers over detailed explanations\n")
+  - Durable preference. Merge with existing memory:
+    update_memory(updated_memory="## Facts\n- 2025-03-15: Alex is a space enthusiast\n\n## Preferences\n- 2025-03-15: Alex prefers concise answers over detailed explanations\n")
 - User: "I actually moved to Tokyo last month"
   - Updated fact, date prefix reflects when recorded:
-    update_memory(updated_memory="## Interests & background\n...\n\n## Personal context\n- (2025-03-15) [fact] Alex lives in Tokyo (previously London)\n...")
+    update_memory(updated_memory="## Facts\n- 2025-03-15: Alex lives in Tokyo (previously London)\n...")
 - User: "I'm a freelance photographer working on a nature documentary"
   - Durable background info under a fitting heading:
-    update_memory(updated_memory="...\n\n## Current focus\n- (2025-03-15) [fact] Alex is a freelance photographer\n- (2025-03-15) [fact] Alex is working on a nature documentary\n")
+    update_memory(updated_memory="...\n\n## Current Focus\n- 2025-03-15: Alex is a freelance photographer\n- 2025-03-15: Alex is working on a nature documentary\n")
 - User: "Always respond in bullet points"
   - Standing instruction:
-    update_memory(updated_memory="...\n\n## Response style\n- (2025-03-15) [instr] Always respond to Alex in bullet points\n")
+    update_memory(updated_memory="...\n\n## Instructions\n- 2025-03-15: Always respond to Alex in bullet points\n")

surfsense_backend/app/agents/new_chat/prompts/examples/update_memory_team.md

@@ -1,7 +1,7 @@
 
 - User: "Let's remember that we decided to do weekly standup meetings on Mondays"
   - Durable team decision:
-    update_memory(updated_memory="- (2025-03-15) [fact] Weekly standup meetings on Mondays\n...")
+    update_memory(updated_memory="## Product Decisions\n- 2025-03-15: Weekly standup meetings happen on Mondays\n...")
 - User: "Our office is in downtown Seattle, 5th floor"
   - Durable team fact:
-    update_memory(updated_memory="- (2025-03-15) [fact] Office location: downtown Seattle, 5th floor\n...")
+    update_memory(updated_memory="## Project Facts\n- 2025-03-15: Office location is downtown Seattle, 5th floor\n...")

surfsense_backend/app/agents/new_chat/prompts/tools/update_memory_private.md

@@ -1,31 +1,26 @@
 
 - update_memory: Update your personal memory document about the user.
-  - Your current memory is already in <user_memory> in your context.  The `chars` and
-    `limit` attributes show your current usage and the maximum allowed size.
-  - This is your curated long-term memory — the distilled essence of what you know about
-    the user, not raw conversation logs.
-  - Call update_memory when:
-    * The user explicitly asks to remember or forget something
-    * The user shares durable facts or preferences that will matter in future conversations
-  - The user's first name is provided in <user_name>. Use it in memory entries
-    instead of "the user" (e.g. "{name} works at..." not "The user works at...").
-    Do not store the name itself as a separate memory entry.
-  - Do not store short-lived or ephemeral info: one-off questions, greetings,
-    session logistics, or things that only matter for the current task.
+  - Your current memory is already in <user_memory> in your context. The `chars`
+    and `limit` attributes show current usage and the maximum allowed size.
+  - This is curated long-term memory, not raw conversation logs.
+  - Call update_memory when the user explicitly asks to remember/forget
+    something or shares durable facts, preferences, or standing instructions.
+  - The user's first name is provided in <user_name>. Use it in entries instead
+    of "the user" when helpful. Do not store the name alone as a memory entry.
+  - Do not store short-lived info: one-off questions, greetings, session
+    logistics, or things that only matter for the current task.
   - Args:
-    - updated_memory: The FULL updated markdown document (not a diff).
-      Merge new facts with existing ones, update contradictions, remove outdated entries.
-      Treat every update as a curation pass — consolidate, don't just append.
-  - Every bullet MUST use this format: - (YYYY-MM-DD) [marker] text
-    Markers:
-      [fact]  — durable facts (role, background, projects, tools, expertise)
-      [pref]  — preferences (response style, languages, formats, tools)
-      [instr] — standing instructions (always/never do, response rules)
-  - Keep it concise and well under the character limit shown in <user_memory>.
-  - Every entry MUST be under a `##` heading. Keep heading names short (2-3 words) and
-    natural. Do NOT include the user's name in headings. Organize by context — e.g.
-    who they are, what they're focused on, how they prefer things. Create, split, or
-    merge headings freely as the memory grows.
-  - Each entry MUST be a single bullet point. Be descriptive but concise — include relevant
-    details and context rather than just a few words.
-  - During consolidation, prioritize keeping: [instr] > [pref] > [fact].
+    - updated_memory: The FULL updated markdown document, not a diff. Merge new
+      facts with existing ones, update contradictions, remove outdated entries,
+      and consolidate instead of only appending.
+  - Use heading-based Markdown:
+    * Every entry must be under a `##` heading.
+    * Recommended headings: `## Facts`, `## Preferences`, `## Instructions`.
+      Specific natural headings are allowed when clearer.
+    * New bullets should use `- YYYY-MM-DD: text`.
+    * Each entry should be one concise but descriptive bullet.
+  - If existing memory uses legacy `(YYYY-MM-DD) [fact|pref|instr]` markers,
+    preserve the information but write the updated document in the new
+    heading-based format.
+  - During consolidation, prioritize durable instructions and preferences before
+    generic facts.

surfsense_backend/app/agents/new_chat/prompts/tools/update_memory_team.md

@@ -1,26 +1,28 @@
 
 - update_memory: Update the team's shared memory document for this search space.
-  - Your current team memory is already in <team_memory> in your context.  The `chars`
-    and `limit` attributes show current usage and the maximum allowed size.
-  - This is the team's curated long-term memory — decisions, conventions, key facts.
-  - NEVER store personal memory in team memory (e.g. personal bio, individual
-    preferences, or user-only standing instructions).
-  - Call update_memory when:
-    * A team member explicitly asks to remember or forget something
-    * The conversation surfaces durable team decisions, conventions, or facts
-      that will matter in future conversations
-  - Do not store short-lived or ephemeral info: one-off questions, greetings,
-    session logistics, or things that only matter for the current task.
+  - Your current team memory is already in <team_memory> in your context. The
+    `chars` and `limit` attributes show current usage and the maximum allowed size.
+  - This is curated long-term team memory: decisions, conventions, architecture,
+    processes, and key shared facts.
+  - NEVER store personal memory in team memory: individual bios, personal
+    preferences, or user-only standing instructions.
+  - Call update_memory when a team member asks to remember/forget something, or
+    when the conversation surfaces durable team context that matters later.
+  - Do not store short-lived info: one-off questions, greetings, session
+    logistics, or things that only matter for the current task.
   - Args:
-    - updated_memory: The FULL updated markdown document (not a diff).
-      Merge new facts with existing ones, update contradictions, remove outdated entries.
-      Treat every update as a curation pass — consolidate, don't just append.
-  - Every bullet MUST use this format: - (YYYY-MM-DD) [fact] text
-    Team memory uses ONLY the [fact] marker. Never use [pref] or [instr] in team memory.
-  - Keep it concise and well under the character limit shown in <team_memory>.
-  - Every entry MUST be under a `##` heading. Keep heading names short (2-3 words) and
-    natural. Organize by context — e.g. what the team decided, current architecture,
-    active processes. Create, split, or merge headings freely as the memory grows.
-  - Each entry MUST be a single bullet point. Be descriptive but concise — include relevant
-    details and context rather than just a few words.
-  - During consolidation, prioritize keeping: decisions/conventions > key facts > current priorities.
+    - updated_memory: The FULL updated markdown document, not a diff. Merge new
+      facts with existing ones, update contradictions, remove outdated entries,
+      and consolidate instead of only appending.
+  - Use heading-based Markdown:
+    * Every entry must be under a `##` heading.
+    * Recommended headings: `## Product Decisions`, `## Engineering Conventions`,
+      `## Project Facts`, `## Open Questions`.
+    * New bullets should use `- YYYY-MM-DD: text`.
+    * Each entry should be one concise but descriptive bullet.
+  - If existing memory uses legacy `(YYYY-MM-DD) [fact]` markers, preserve the
+    information but write the updated document in the new heading-based format.
+  - Do not create personal headings such as `## Preferences`, `## Instructions`,
+    `## Personal Notes`, or `## Personal Instructions`.
+  - During consolidation, prioritize decisions/conventions, then key facts, then
+    current priorities.