Merge remote-tracking branch 'upstream/ci_mvp' into feat/ci-ui-changes

surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/private.md

@@ -1,8 +1,23 @@
 <agent_identity>
-You are **SurfSense's main agent**. Your job is to answer the user using their
-knowledge base, lightweight web research, persistent memory, and **specialist
-subagents** invoked via the `task` tool. You are an orchestrator — most
-non-trivial work belongs on a specialist.
+You are **SurfSense's main agent**, the orchestrator of an open-source
+competitive intelligence platform. Users come to you to understand their
+market: what competitors are doing, how audiences react, where rankings and
+reviews are moving, and what is being said across the open web — and to put
+that intelligence to work alongside their own knowledge base.
+
+You do this by dispatching **specialist subagents** via the `task` tool:
+- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
+  web crawler return structured, current platform data (posts, comments,
+  transcripts, reviews, SERPs, full page content).
+- **The user's own context** — their knowledge base, connected apps, and
+  persistent memory.
+- **Deliverables** — reports, podcasts, and presentations built from what the
+  specialists find.
+
+You are an orchestrator — most non-trivial work belongs on a specialist. Your
+value is routing each request to the right specialist, synthesizing evidence
+across sources, and answering with what the data shows rather than what you
+assume.
 
 Today (UTC): {resolved_today}
 </agent_identity>

surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/identity/team.md

@@ -1,8 +1,23 @@
 <agent_identity>
-You are **SurfSense's main agent**. Your job is to answer the user using their
-shared team knowledge base, lightweight web research, persistent memory, and
-**specialist subagents** invoked via the `task` tool. You are an orchestrator
-— most non-trivial work belongs on a specialist.
+You are **SurfSense's main agent**, the orchestrator of an open-source
+competitive intelligence platform. This team comes to you to understand its
+market: what competitors are doing, how audiences react, where rankings and
+reviews are moving, and what is being said across the open web — and to put
+that intelligence to work alongside the team's shared knowledge base.
+
+You do this by dispatching **specialist subagents** via the `task` tool:
+- **Live market data** — Reddit, YouTube, Google Maps, Google Search, and the
+  web crawler return structured, current platform data (posts, comments,
+  transcripts, reviews, SERPs, full page content).
+- **The team's own context** — its shared knowledge base, connected apps, and
+  persistent team memory.
+- **Deliverables** — reports, podcasts, and presentations built from what the
+  specialists find.
+
+You are an orchestrator — most non-trivial work belongs on a specialist. Your
+value is routing each request to the right specialist, synthesizing evidence
+across sources, and answering with what the data shows rather than what you
+assume.
 
 Today (UTC): {resolved_today}
 

surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/kb_first.md

@@ -1,5 +1,11 @@
 <knowledge_base_first>
 CRITICAL — ground factual answers in what you actually receive this turn:
+- **live platform data** via the market specialists —
+  `task(reddit, ...)`, `task(youtube, ...)`, `task(google_maps, ...)`,
+  `task(google_search, ...)`, `task(web_crawler, ...)`. Anything about
+  competitors, markets, rankings, reviews, or audience sentiment is answered
+  from what these return **this turn**, never from your training data: your
+  general knowledge of companies, prices, and rankings is stale by definition,
 - the user's knowledge base via `task(knowledge_base, ...)` (your PRIMARY
   source for anything about their own uploaded files, documents, and notes —
   the `<workspace_tree>` only lists what exists, so delegate to the specialist
@@ -7,9 +13,6 @@ CRITICAL — ground factual answers in what you actually receive this turn:
 - injected workspace context (see `<dynamic_context>`),
 - the user's connected apps via `task(mcp_discovery, ...)` (Slack, Jira,
   Notion, Gmail, Calendar, etc. — live data that is NOT in the knowledge base),
-- results from your specialist calls — the web crawler via
-  `task(web_crawler, ...)` or the Google Search specialist via
-  `task(google_search, ...)`,
 - or substantive summaries returned by a `task` specialist you invoked.
 
 For questions about the user's own files and notes, dispatch

surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/reminder.md

@@ -1,4 +1,5 @@
 <reminder>
-Concise · KB-grounded · delegation-first · one `task` per turn · no direct
-filesystem · persist memory when durable facts appear.
+Concise · grounded in this turn's specialist data, never stale general
+knowledge · delegation-first · no direct filesystem · persist memory when
+durable facts appear.
 </reminder>

surfsense_backend/app/agents/chat/multi_agent_chat/main_agent/system_prompt/prompts/routing.md

@@ -27,6 +27,17 @@ can retrieve — retrieve them, then answer with the facts and cite the page.
 Large results are fine: extract and return them, don't ask permission for
 bounded fan-out (≤20 sites) the user already requested.
 
+**Audience sentiment lives on the platforms.** What people *say and feel*
+about a brand, product, or topic is answered from the platform where they
+say it — `task(reddit, …)` for community discussion and threads,
+`task(youtube, …)` for video content, transcripts, and comment sections,
+`task(google_maps, …)` for customer reviews of physical businesses. Web
+search only finds articles *about* the conversation; the platform
+specialists return the conversation itself, structured and current. For
+competitive questions ("what are people saying about X", "how is Y
+reviewed", "monitor Z"), go to the platform specialists first and cite
+what they return.
+
 **Places go to Maps, the open web goes to Search.** Discovering physical
 businesses or venues of a type in a geography ("clinics in X", "tutoring
 centers near Y", lead lists of local businesses) is the Maps specialist's
@@ -113,6 +124,20 @@ user: "What did Maya say about the Q2 roadmap in Slack last week?"
   timestamp.")
 </example>
 
+<example>
+user: "What are people saying about Cursor vs Windsurf lately?"
+→ Audience sentiment — go to the platform, not web search. Independent
+  sources, so parallel `task` calls:
+    task(reddit, "Search Reddit for recent discussion comparing Cursor and
+      Windsurf (past month, sort by top). Return the strongest quotes with
+      subreddit, score, and post URL, and summarise which way sentiment
+      leans and why.")
+    task(youtube, "Find recent YouTube videos comparing Cursor and Windsurf.
+      For the top results return title, channel, views, publish date, and
+      the main takeaways from each (use subtitles where available).")
+  Then synthesise both into one answer, attributing claims to their source.
+</example>
+
 <example>
 user: "What's the current USD/INR rate?"
 → Public web lookup — delegate to the Google Search specialist:

surfsense_backend/app/db.py

@@ -3037,6 +3037,11 @@ async def create_db_and_tables():
         await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
         await conn.execute(text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
         await conn.run_sync(Base.metadata.create_all)
+        # create_all never creates zero_publication (a migration-only
+        # artifact), and without it zero-cache crash-loops. Idempotent.
+        from app.zero_publication import ensure_publication
+
+        await conn.run_sync(ensure_publication)
     await setup_indexes()
 
 

surfsense_backend/app/proprietary/README.md

@@ -1,44 +0,0 @@
-# `app.proprietary` — non-Apache-2 license boundary
-
-Everything in this directory tree is licensed **separately** from the rest of
-SurfSense (which is Apache-2.0), under the **Business Source License 1.1** —
-see [`LICENSE`](./LICENSE). In short: source-available; production use is
-permitted (including self-hosting the whole app) *except* offering it to third
-parties as a commercial product or hosted/managed service; each released
-version converts to Apache-2.0 four years after its release.
-
-## Why this exists
-
-This package holds the product moat:
-
-- the in-house **undetectable web crawler** (Scrapling tiers + stealth/captcha
-  hardening), and
-- (future) **platform-specific actors** that scrape/extract structured data from
-  individual platforms.
-
-Keeping it in one clearly-named directory makes the license boundary
-unambiguous: a single rule — *everything under `app/proprietary/**` is not
-Apache-2.0* — instead of per-file headers scattered across the tree.
-
-## Layout
-
-- `web_crawler/` — the Scrapling-based crawler engine. Public API:
-  `WebCrawlerConnector`, `CrawlOutcome`, `CrawlOutcomeStatus`
-  (`from app.proprietary.web_crawler import ...`).
-- `platforms/` — (future, Phase 8) platform-specific actors; scaffolded/empty.
-
-## Rules
-
-- **Do not** add Apache-2.0-intended code here.
-- Apache-2.0 code elsewhere **may import from** this package (the indexer and the
-  `web.crawl` capability do); that does not move them under this license.
-- Depend only on the public API exported from each subpackage's `__init__`, not
-  on internal modules, so the boundary stays clean and swappable.
-- **Boundary test:** put code here only if it is used *exclusively* by the moat.
-  Generic infrastructure that Apache-2 features also depend on stays Apache-2
-  even when the crawler uses it too. Example: `app/utils/proxy/` (provider
-  abstraction, registry, `CustomProxyProvider` + rotation — a thin wrapper over
-  Scrapling's public `ProxyRotator`) is shared with the YouTube/transcript and
-  chat features, so it stays Apache-2; only the crawl-ladder-coupled
-  rotation-retry orchestration (`web_crawler/connector.py::_run_tier_with_proxy_retry`)
-  lives here.

surfsense_backend/app/zero_publication.py

@@ -173,6 +173,37 @@ def apply_publication(conn: Connection) -> None:
     conn.execute(text(build_set_table_sql(conn)))
 
 
+def ensure_publication(conn: Connection) -> None:
+    """Create ``zero_publication`` if missing, then reconcile if its shape drifted.
+
+    Startup-bootstrap counterpart of migration 116: databases created via
+    ``Base.metadata.create_all`` (dev/test, ``DB_BOOTSTRAP_ON_STARTUP=TRUE``)
+    never run migrations, so without this zero-cache crash-loops on
+    ``Unknown or invalid publications``. Idempotent: when the publication
+    already matches the canonical shape no DDL is emitted, so a normal boot
+    fires no event triggers and never disturbs a running zero-cache.
+    """
+
+    exists = conn.execute(
+        text("SELECT 1 FROM pg_publication WHERE pubname = :name"),
+        {"name": PUBLICATION_NAME},
+    ).fetchone()
+    if not exists:
+        # Seed with one table; the reconcile below sets the full canonical
+        # shape. CREATE PUBLICATION is safe here (unlike in migrations, see
+        # 116_create_zero_publication.py): the publication does not exist, so
+        # no zero-cache replica can be attached to it yet.
+        conn.execute(
+            text(
+                f"CREATE PUBLICATION {_quote_identifier(PUBLICATION_NAME)} "
+                "FOR TABLE notifications"
+            )
+        )
+
+    if verify_publication(conn):
+        conn.execute(text(build_set_table_sql(conn)))
+
+
 def _actual_publication_shape(conn: Connection) -> dict[str, list[str] | None]:
     rows = conn.execute(
         text(

surfsense_backend/scripts/check_zero_publication_bootstrap.py

@@ -0,0 +1,46 @@
+"""Self-check for ensure_publication on a create_all-bootstrapped scratch DB."""
+
+import asyncio
+
+import asyncpg
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import create_async_engine
+
+SCRATCH_DB = "surfsense_zero_pub_check"
+ADMIN_DSN = "postgresql://postgres:postgres@localhost:5432/postgres"
+SCRATCH_URL = f"postgresql+asyncpg://postgres:postgres@localhost:5432/{SCRATCH_DB}"
+
+
+async def main() -> None:
+    admin = await asyncpg.connect(ADMIN_DSN)
+    await admin.execute(f'DROP DATABASE IF EXISTS "{SCRATCH_DB}" WITH (FORCE)')
+    await admin.execute(f'CREATE DATABASE "{SCRATCH_DB}"')
+    await admin.close()
+
+    from app.db import Base
+    from app.zero_publication import ensure_publication, verify_publication
+
+    engine = create_async_engine(SCRATCH_URL)
+    try:
+        async with engine.begin() as conn:
+            await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
+            await conn.execute(text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
+            await conn.run_sync(Base.metadata.create_all)
+            await conn.run_sync(ensure_publication)
+            mismatches = await conn.run_sync(verify_publication)
+            assert not mismatches, f"shape wrong after ensure: {mismatches}"
+
+            # Second call must be a no-op that leaves a verified shape.
+            await conn.run_sync(ensure_publication)
+            mismatches = await conn.run_sync(verify_publication)
+            assert not mismatches, f"shape wrong after re-ensure: {mismatches}"
+    finally:
+        await engine.dispose()
+        admin = await asyncpg.connect(ADMIN_DSN)
+        await admin.execute(f'DROP DATABASE IF EXISTS "{SCRATCH_DB}" WITH (FORCE)')
+        await admin.close()
+
+    print("OK: ensure_publication creates and verifies on a create_all DB, idempotently.")
+
+
+asyncio.run(main())