feat: refine private and team memory protocols

surfsense_backend/app/agents/new_chat/middleware/memory_injection.py

@@ -17,8 +17,8 @@ from langgraph.runtime import Runtime
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.agents.new_chat.tools.update_memory import MEMORY_HARD_LIMIT, MEMORY_SOFT_LIMIT
 from app.db import ChatVisibility, SearchSpace, User, shielded_async_session
+from app.services.memory import MEMORY_HARD_LIMIT, MEMORY_SOFT_LIMIT
 
 logger = logging.getLogger(__name__)
 

surfsense_backend/app/agents/new_chat/prompts/base/memory_protocol_private.md

@@ -3,4 +3,10 @@ IMPORTANT — After understanding each user message, ALWAYS check: does this mes
 reveal durable facts about the user (role, interests, preferences, projects,
 background, or standing instructions)? If yes, you MUST call update_memory
 alongside your normal response — do not defer this to a later turn.
+
+Memory is stored as a heading-based markdown document. New entries should be
+under `##` headings such as `## Facts`, `## Preferences`, or `## Instructions`
+with bullets like `- YYYY-MM-DD: text`. If existing memory contains legacy
+`(YYYY-MM-DD) [fact|pref|instr]` markers, preserve the information but write
+new saves in the heading-based format.
 </memory_protocol>

surfsense_backend/app/agents/new_chat/prompts/base/memory_protocol_team.md

@@ -3,4 +3,12 @@ IMPORTANT — After understanding each user message, ALWAYS check: does this mes
 reveal durable facts about the team (decisions, conventions, architecture, processes,
 or key facts)? If yes, you MUST call update_memory alongside your normal response —
 do not defer this to a later turn.
+
+Team memory is stored as a heading-based markdown document. New entries should
+be under `##` headings such as `## Product Decisions`,
+`## Engineering Conventions`, `## Project Facts`, or `## Open Questions` with
+bullets like `- YYYY-MM-DD: text`. If existing memory contains legacy
+`(YYYY-MM-DD) [fact]` markers, preserve the information but write new saves in
+the heading-based format. Do not create personal headings such as
+`## Preferences` or `## Instructions`.
 </memory_protocol>

surfsense_backend/app/agents/new_chat/prompts/examples/update_memory_private.md

@@ -1,16 +1,16 @@
 
 - <user_name>Alex</user_name>, <user_memory> is empty. User: "I'm a space enthusiast, explain astrophage to me"
-  - The user casually shared a durable fact. Use their first name in the entry, short neutral heading:
-    update_memory(updated_memory="## Interests & background\n- (2025-03-15) [fact] Alex is a space enthusiast\n")
+  - The user casually shared a durable fact:
+    update_memory(updated_memory="## Facts\n- 2025-03-15: Alex is a space enthusiast\n")
 - User: "Remember that I prefer concise answers over detailed explanations"
-  - Durable preference. Merge with existing memory, add a new heading:
-    update_memory(updated_memory="## Interests & background\n- (2025-03-15) [fact] Alex is a space enthusiast\n\n## Response style\n- (2025-03-15) [pref] Alex prefers concise answers over detailed explanations\n")
+  - Durable preference. Merge with existing memory:
+    update_memory(updated_memory="## Facts\n- 2025-03-15: Alex is a space enthusiast\n\n## Preferences\n- 2025-03-15: Alex prefers concise answers over detailed explanations\n")
 - User: "I actually moved to Tokyo last month"
   - Updated fact, date prefix reflects when recorded:
-    update_memory(updated_memory="## Interests & background\n...\n\n## Personal context\n- (2025-03-15) [fact] Alex lives in Tokyo (previously London)\n...")
+    update_memory(updated_memory="## Facts\n- 2025-03-15: Alex lives in Tokyo (previously London)\n...")
 - User: "I'm a freelance photographer working on a nature documentary"
   - Durable background info under a fitting heading:
-    update_memory(updated_memory="...\n\n## Current focus\n- (2025-03-15) [fact] Alex is a freelance photographer\n- (2025-03-15) [fact] Alex is working on a nature documentary\n")
+    update_memory(updated_memory="...\n\n## Current Focus\n- 2025-03-15: Alex is a freelance photographer\n- 2025-03-15: Alex is working on a nature documentary\n")
 - User: "Always respond in bullet points"
   - Standing instruction:
-    update_memory(updated_memory="...\n\n## Response style\n- (2025-03-15) [instr] Always respond to Alex in bullet points\n")
+    update_memory(updated_memory="...\n\n## Instructions\n- 2025-03-15: Always respond to Alex in bullet points\n")

surfsense_backend/app/agents/new_chat/prompts/examples/update_memory_team.md

@@ -1,7 +1,7 @@
 
 - User: "Let's remember that we decided to do weekly standup meetings on Mondays"
   - Durable team decision:
-    update_memory(updated_memory="- (2025-03-15) [fact] Weekly standup meetings on Mondays\n...")
+    update_memory(updated_memory="## Product Decisions\n- 2025-03-15: Weekly standup meetings happen on Mondays\n...")
 - User: "Our office is in downtown Seattle, 5th floor"
   - Durable team fact:
-    update_memory(updated_memory="- (2025-03-15) [fact] Office location: downtown Seattle, 5th floor\n...")
+    update_memory(updated_memory="## Project Facts\n- 2025-03-15: Office location is downtown Seattle, 5th floor\n...")

surfsense_backend/app/agents/new_chat/prompts/tools/update_memory_private.md

@@ -1,31 +1,26 @@
 
 - update_memory: Update your personal memory document about the user.
-  - Your current memory is already in <user_memory> in your context.  The `chars` and
-    `limit` attributes show your current usage and the maximum allowed size.
-  - This is your curated long-term memory — the distilled essence of what you know about
-    the user, not raw conversation logs.
-  - Call update_memory when:
-    * The user explicitly asks to remember or forget something
-    * The user shares durable facts or preferences that will matter in future conversations
-  - The user's first name is provided in <user_name>. Use it in memory entries
-    instead of "the user" (e.g. "{name} works at..." not "The user works at...").
-    Do not store the name itself as a separate memory entry.
-  - Do not store short-lived or ephemeral info: one-off questions, greetings,
-    session logistics, or things that only matter for the current task.
+  - Your current memory is already in <user_memory> in your context. The `chars`
+    and `limit` attributes show current usage and the maximum allowed size.
+  - This is curated long-term memory, not raw conversation logs.
+  - Call update_memory when the user explicitly asks to remember/forget
+    something or shares durable facts, preferences, or standing instructions.
+  - The user's first name is provided in <user_name>. Use it in entries instead
+    of "the user" when helpful. Do not store the name alone as a memory entry.
+  - Do not store short-lived info: one-off questions, greetings, session
+    logistics, or things that only matter for the current task.
   - Args:
-    - updated_memory: The FULL updated markdown document (not a diff).
-      Merge new facts with existing ones, update contradictions, remove outdated entries.
-      Treat every update as a curation pass — consolidate, don't just append.
-  - Every bullet MUST use this format: - (YYYY-MM-DD) [marker] text
-    Markers:
-      [fact]  — durable facts (role, background, projects, tools, expertise)
-      [pref]  — preferences (response style, languages, formats, tools)
-      [instr] — standing instructions (always/never do, response rules)
-  - Keep it concise and well under the character limit shown in <user_memory>.
-  - Every entry MUST be under a `##` heading. Keep heading names short (2-3 words) and
-    natural. Do NOT include the user's name in headings. Organize by context — e.g.
-    who they are, what they're focused on, how they prefer things. Create, split, or
-    merge headings freely as the memory grows.
-  - Each entry MUST be a single bullet point. Be descriptive but concise — include relevant
-    details and context rather than just a few words.
-  - During consolidation, prioritize keeping: [instr] > [pref] > [fact].
+    - updated_memory: The FULL updated markdown document, not a diff. Merge new
+      facts with existing ones, update contradictions, remove outdated entries,
+      and consolidate instead of only appending.
+  - Use heading-based Markdown:
+    * Every entry must be under a `##` heading.
+    * Recommended headings: `## Facts`, `## Preferences`, `## Instructions`.
+      Specific natural headings are allowed when clearer.
+    * New bullets should use `- YYYY-MM-DD: text`.
+    * Each entry should be one concise but descriptive bullet.
+  - If existing memory uses legacy `(YYYY-MM-DD) [fact|pref|instr]` markers,
+    preserve the information but write the updated document in the new
+    heading-based format.
+  - During consolidation, prioritize durable instructions and preferences before
+    generic facts.

surfsense_backend/app/agents/new_chat/prompts/tools/update_memory_team.md

@@ -1,26 +1,28 @@
 
 - update_memory: Update the team's shared memory document for this search space.
-  - Your current team memory is already in <team_memory> in your context.  The `chars`
-    and `limit` attributes show current usage and the maximum allowed size.
-  - This is the team's curated long-term memory — decisions, conventions, key facts.
-  - NEVER store personal memory in team memory (e.g. personal bio, individual
-    preferences, or user-only standing instructions).
-  - Call update_memory when:
-    * A team member explicitly asks to remember or forget something
-    * The conversation surfaces durable team decisions, conventions, or facts
-      that will matter in future conversations
-  - Do not store short-lived or ephemeral info: one-off questions, greetings,
-    session logistics, or things that only matter for the current task.
+  - Your current team memory is already in <team_memory> in your context. The
+    `chars` and `limit` attributes show current usage and the maximum allowed size.
+  - This is curated long-term team memory: decisions, conventions, architecture,
+    processes, and key shared facts.
+  - NEVER store personal memory in team memory: individual bios, personal
+    preferences, or user-only standing instructions.
+  - Call update_memory when a team member asks to remember/forget something, or
+    when the conversation surfaces durable team context that matters later.
+  - Do not store short-lived info: one-off questions, greetings, session
+    logistics, or things that only matter for the current task.
   - Args:
-    - updated_memory: The FULL updated markdown document (not a diff).
-      Merge new facts with existing ones, update contradictions, remove outdated entries.
-      Treat every update as a curation pass — consolidate, don't just append.
-  - Every bullet MUST use this format: - (YYYY-MM-DD) [fact] text
-    Team memory uses ONLY the [fact] marker. Never use [pref] or [instr] in team memory.
-  - Keep it concise and well under the character limit shown in <team_memory>.
-  - Every entry MUST be under a `##` heading. Keep heading names short (2-3 words) and
-    natural. Organize by context — e.g. what the team decided, current architecture,
-    active processes. Create, split, or merge headings freely as the memory grows.
-  - Each entry MUST be a single bullet point. Be descriptive but concise — include relevant
-    details and context rather than just a few words.
-  - During consolidation, prioritize keeping: decisions/conventions > key facts > current priorities.
+    - updated_memory: The FULL updated markdown document, not a diff. Merge new
+      facts with existing ones, update contradictions, remove outdated entries,
+      and consolidate instead of only appending.
+  - Use heading-based Markdown:
+    * Every entry must be under a `##` heading.
+    * Recommended headings: `## Product Decisions`, `## Engineering Conventions`,
+      `## Project Facts`, `## Open Questions`.
+    * New bullets should use `- YYYY-MM-DD: text`.
+    * Each entry should be one concise but descriptive bullet.
+  - If existing memory uses legacy `(YYYY-MM-DD) [fact]` markers, preserve the
+    information but write the updated document in the new heading-based format.
+  - Do not create personal headings such as `## Preferences`, `## Instructions`,
+    `## Personal Notes`, or `## Personal Instructions`.
+  - During consolidation, prioritize decisions/conventions, then key facts, then
+    current priorities.