feat: implement background memory extraction and editing capabilities for user and team memory management, enhancing long-term memory persistence and user interaction

surfsense_backend/app/agents/new_chat/system_prompt.py

@@ -271,8 +271,7 @@ _MEMORY_TOOL_INSTRUCTIONS: dict[str, dict[str, str]] = {
     `limit` attributes show your current usage and the maximum allowed size.
   - This is your curated long-term memory — the distilled essence of what you know about
     the user, not raw conversation logs.
-  - You are the sole mechanism for persisting memory — there is no background extraction.
-    Call update_memory when:
+  - Call update_memory when:
     * The user explicitly asks to remember or forget something
     * The user shares durable facts or preferences that will matter in future conversations
   - The user's name is already provided via <user_name> — do not store it in memory.
@@ -285,21 +284,18 @@ _MEMORY_TOOL_INSTRUCTIONS: dict[str, dict[str, str]] = {
   - Every bullet MUST start with a (YYYY-MM-DD) date prefix indicating when it was recorded or last updated.
   - Keep it concise and well under the character limit shown in <user_memory>.
   - You MUST organize memory using these standard sections (add new `##` sections only if none of the standard ones fit):
-    ## About the user (pinned) — role, background, company
+    ## About the user — role, background, company
     ## Preferences — languages, tools, frameworks, response style
-    ## Instructions (pinned) — standing instructions, things to always/never do
+    ## Instructions — standing instructions, things to always/never do
   - Each entry MUST be a single bullet point. Keep entries concise (aim for under 120 chars each).
-  - Sections with `(pinned)` in the heading are protected — the system will reject any
-    update that removes them. Users can add `(pinned)` to any `##` heading to protect it.
-  - During consolidation, prioritize keeping: pinned sections > preferences.
+  - During consolidation, prioritize keeping: identity/instructions > preferences.
 """,
         "shared": """
 - update_memory: Update the team's shared memory document for this search space.
   - Your current team memory is already in <team_memory> in your context.  The `chars`
     and `limit` attributes show current usage and the maximum allowed size.
   - This is the team's curated long-term memory — decisions, conventions, key facts.
-  - You are the sole mechanism for persisting team memory — there is no background extraction.
-    Call update_memory when:
+  - Call update_memory when:
     * A team member explicitly asks to remember or forget something
     * The conversation surfaces durable team decisions, conventions, or facts
       that will matter in future conversations
@@ -312,14 +308,12 @@ _MEMORY_TOOL_INSTRUCTIONS: dict[str, dict[str, str]] = {
   - Every bullet MUST start with a (YYYY-MM-DD) date prefix indicating when it was recorded or last updated.
   - Keep it concise and well under the character limit shown in <team_memory>.
   - You MUST organize memory using these standard sections (add new `##` sections only if none of the standard ones fit):
-    ## Team decisions (pinned) — agreed-upon choices with rationale
-    ## Conventions (pinned) — coding standards, tools, processes, naming patterns
+    ## Team decisions — agreed-upon choices with rationale
+    ## Conventions — coding standards, tools, processes, naming patterns
     ## Key facts — where things are, how things work, team structure
     ## Current priorities — active projects, deadlines, blockers
   - Each entry MUST be a single bullet point. Keep entries concise (aim for under 120 chars each).
-  - Sections with `(pinned)` in the heading are protected — the system will reject any
-    update that removes them. Users can add `(pinned)` to any `##` heading to protect it.
-  - During consolidation, prioritize keeping: pinned sections > key facts > current priorities.
+  - During consolidation, prioritize keeping: decisions/conventions > key facts > current priorities.
 """,
     },
 }
@@ -329,21 +323,21 @@ _MEMORY_TOOL_EXAMPLES: dict[str, dict[str, str]] = {
         "private": """
 - <user_memory> is empty. User: "I'm a space enthusiast, explain astrophage to me"
   - The user casually shared a durable fact about themselves. Save it:
-    update_memory(updated_memory="## About the user (pinned)\\n- (2025-03-15) Space enthusiast\\n")
+    update_memory(updated_memory="## About the user\\n- (2025-03-15) Space enthusiast\\n")
 - User: "Remember that I prefer concise answers over detailed explanations"
   - Durable preference. You see the current <user_memory> and merge:
-    update_memory(updated_memory="## About the user (pinned)\\n- (2025-03-15) Space enthusiast\\n\\n## Preferences\\n- (2025-03-15) Prefers concise answers over detailed explanations\\n...")
+    update_memory(updated_memory="## About the user\\n- (2025-03-15) Space enthusiast\\n\\n## Preferences\\n- (2025-03-15) Prefers concise answers over detailed explanations\\n...")
 - User: "I actually moved to Tokyo last month"
   - Updated fact, date prefix reflects when recorded:
-    update_memory(updated_memory="## About the user (pinned)\\n- (2025-03-15) Lives in Tokyo (previously London)\\n...")
+    update_memory(updated_memory="## About the user\\n- (2025-03-15) Lives in Tokyo (previously London)\\n...")
 - User: "I'm a freelance photographer working on a nature documentary"
   - Durable background info. Save it under About the user:
-    update_memory(updated_memory="## About the user (pinned)\\n- (2025-03-15) Freelance photographer\\n- (2025-03-15) Working on a nature documentary\\n")
+    update_memory(updated_memory="## About the user\\n- (2025-03-15) Freelance photographer\\n- (2025-03-15) Working on a nature documentary\\n")
 """,
         "shared": """
 - User: "Let's remember that we decided to do weekly standup meetings on Mondays"
   - Durable team decision:
-    update_memory(updated_memory="## Team decisions (pinned)\\n- (2025-03-15) Weekly standup meetings on Mondays\\n...")
+    update_memory(updated_memory="## Team decisions\\n- (2025-03-15) Weekly standup meetings on Mondays\\n...")
 - User: "Our office is in downtown Seattle, 5th floor"
   - Durable team fact:
     update_memory(updated_memory="## Key facts\\n- (2025-03-15) Office location: downtown Seattle, 5th floor\\n...")