Adds 34 tests under tests/unit/tasks/chat/streaming/ that cover the
new flows tree against the legacy stream_new_chat.py module to gate
the upcoming cutover. Coverage:
* Public entry points: stream_new_chat and stream_resume_chat are
async generator functions whose parameter signatures (name, kind,
annotation, default) match the legacy versions one-for-one. Uses a
normalized-annotation comparison so PEP-563 vs eager-annotation
representation differences are tolerated.
* Extracted helpers: image-capability gate, runtime-context builders
for new-chat and resume-chat, LLM-bundle dispatcher, premium-quota
needs check + reservation dataclass, rate-limit recovery truth
table, persistence-spawn registration/self-unregistration, await
helpers.
* SSE frame iterators: iter_initial_frames + iter_final_frames emit
the canonical sequence; iter_token_usage_frame skips on None.
* Initial thinking step: 4 parametrized branches (text, image-only,
empty, mentioned-docs), long-query truncation, many-docs collapse.
These tests are scaffolding for the cutover and will be removed once
the legacy module is deleted.
Slim composition root for the resume-chat streaming flow. Mirrors the
new_chat orchestrator but specialized for resumed turns:
* no fresh user turn, no title generation, no image-capability gate
* persists a fresh assistant shell for the resumed turn
* applies build_resume_routing to dispatch user decisions to the
correct paused subagent before invoking the agent
* shares the same stream_loop + flow-local _recover closure for in-
stream provider rate-limit recovery
Also lands flows/__init__.py, which becomes the public chat-flow API:
from app.tasks.chat.streaming.flows import stream_new_chat, stream_resume_chat
Existing wiring (routes, contract test) still imports from the legacy
app.tasks.chat.stream_new_chat module. Cutover is the next phase.
Three focused modules used by the upcoming resume-chat orchestrator:
* runtime_context: build_resume_chat_runtime_context assembles the
SurfSenseContextSchema for a resume turn (handles empty mention
lists, since resume requests do not carry fresh @-mentions).
* assistant_shell: persist_resume_assistant_shell writes a fresh
assistant row for the resumed turn so the post-stream finalize
has a target.
* resume_routing: build_resume_routing collects the pending
interrupts across paused subagents and slices the flat list of
ResumeDecision[] into the correct (thread, subagent) buckets so
LangGraph routes each decision back to the right paused tool call.
Add-only; no orchestrator yet (next commit).
Slim composition root for the new-chat streaming flow. Sequences:
1. validate inputs and load the LLM bundle (negative id => YAML)
2. open the OTEL chat_request span; set agent_mode tag
3. spawn the four pre-stream DB writes (set-ai-responding, persist
user turn, persist assistant shell, first-assistant probe)
4. reserve premium quota (with free-fallback retry on denial)
5. build connector + checkpointer + agent + input_state
6. emit first frames (message-start, step-start, initial thinking step)
7. spawn the background title generator
8. run the shared stream_loop with a flow-local _recover closure that
reroutes to the next auto-pin config on provider 429s
9. finalize: emit terminal title/token frames, shielded assistant
finalize, release-or-finalize premium quota, close session, GC,
record OTEL outcome
Public entry-point flows/new_chat/__init__ re-exports stream_new_chat.
Existing wiring (routes, tests) still imports the legacy function from
app.tasks.chat.stream_new_chat. Cutover is a later commit.
Seven focused modules that the upcoming new_chat orchestrator
composes:
* auto_pin: resolve_initial_auto_pin selects the initial config (with
vision-capable filtering and error classification).
* llm_capability: check_image_input_capability blocks routing an
image-bearing turn to a known text-only model.
* runtime_context: build_new_chat_runtime_context assembles the
SurfSenseContextSchema for a new-chat turn.
* persistence_spawn: spawn_set_ai_responding_bg, spawn_persist_user_task,
spawn_persist_assistant_shell_task, and await_persist_task background
the four pre-stream DB writes so they overlap with agent build.
* initial_thinking_step: build_initial_thinking_step +
iter_initial_thinking_step_frame produce the very first thinking-1 SSE
step ("Understanding your request" / "Analyzing referenced content").
* title_gen: spawn_title_task + maybe_emit_title_update +
await_pending_title_update background the thread-title generator and
interleave its update into the stream when ready.
* input_state: build_new_chat_input_state assembles the LangGraph
input_state (history bootstrap, mentions resolution, context blocks,
human-message construction). The heavy one.
Add-only; no orchestrator yet (next commit).
Extracts finalize_assistant_message: the post-stream server-side write
of the final assistant message (with content parts + token usage)
guarded by asyncio.shield + shielded_async_session so a client
disconnect cannot abort the persist.
Add-only; legacy stream_new_chat.py keeps its inline finalize block
until cutover.
Two cooperating modules that wrap stream_agent_events with in-stream
recovery from provider 429s:
* rate_limit_recovery: can_recover_provider_rate_limit truth-table
guard, reroute_to_next_auto_pin (selects the next eligible auto-pin
config and reloads the LLM bundle), log_rate_limit_recovered.
* stream_loop: run_stream_loop drives stream_agent_events in a
while-True loop, delegating recovery to a flow-supplied RecoverFn
callback so new_chat and resume_chat can share the same loop while
keeping their own nonlocal state.
Add-only; not yet wired into any orchestrator.
Extracts handle_terminal_exception: the shared except-branch behavior for
the chat orchestrators. Classifies the raised exception, logs the
structured chat_stream error event, and emits the terminal-error SSE
frame + done sentinel via the streaming service.
Add-only; nothing imports it yet.
Centralizes the premium-credits lifecycle for chat turns:
* needs_premium_quota: gate check (premium user + non-fallback config).
* PremiumReservation: dataclass capturing reservation state + token totals.
* reserve_premium / finalize_premium / release_premium: idempotent
reservation, commit, and rollback used by the orchestrators.
Add-only; legacy stream_new_chat.py keeps its inline quota handling
until cutover.
Six small, single-purpose modules shared by the upcoming new_chat and
resume_chat orchestrators:
* llm_bundle: dispatches negative config_id to the YAML loader and
non-negative config_id to the DB loader, returning (llm, AgentConfig).
* pre_stream_setup: builds the connector service, resolves the
Firecrawl API key, and returns the chat checkpointer.
* first_frames: iter_initial_frames + iter_final_frames emit the canonical
message-start / step-start / idle / finish / done SSE envelope.
* finalize_emit: iter_token_usage_frame emits the per-turn usage frame
from a TokenAccumulator summary.
* finally_cleanup: close_session_and_clear_ai_responding and run_gc_pass
centralize the finally-block bookkeeping.
* span: open_chat_request_span / set_agent_mode / close_chat_request_span /
record_outcome_attrs wrap the OpenTelemetry chat_request span.
Add-only; these are not yet wired into stream_new_chat.py.
Extracts the inner agent-streaming driver previously inlined as
_stream_agent_events in stream_new_chat.py.
stream_agent_events drives graph_stream.event_stream.stream_output and,
after the agent finishes, performs the post-stream safety-net work:
* commit any pending content the agent never explicitly finished
* evaluate file-operation contract outcomes and emit the appropriate
contract verdict for desktop_local_folder turns
This unit is what flows/shared/stream_loop.py wraps in the rate-limit
recovery while-loop. Add-only; no existing wiring uses it yet.
Extracts the agent-construction wrapper that the chat streamers call to
materialize the LangGraph agent for a given thread. Centralizes how we
pass the agent factory plus checkpointer, runtime context, and the
in-memory content builder.
Add-only; pre-existing inline equivalent in stream_new_chat.py stays
until cutover.
Extracts the desktop_local_folder file-operation contract helpers:
* contract_enforcement_active: gates the contract on filesystem mode.
* evaluate_file_contract_outcome: scores tool outputs as success/no-op.
* log_file_contract: structured logging of contract verdicts.
This is the unit responsible for catching agents that claim to have
written/edited a file without actually invoking the filesystem tool.
Add-only; stream_new_chat.py keeps its inline duplicates until cutover.
Extracts two pure context helpers used during input-state assembly:
* mentioned_docs.format_mentioned_surfsense_docs_as_context: renders the
user's @-mentioned SurfSense docs into the LLM context block.
* deepagents_todos.extract_todos_from_deepagents: pulls the in-progress
todo list from a deep-agents state snapshot for the title generator.
Add-only; existing call sites in stream_new_chat.py remain untouched
until cutover.
Foundation layer for the parallel refactor of stream_new_chat.py.
Extracts the StreamResult dataclass (tracks per-turn streaming state)
and a small set of shared utilities (resume_step_prefix, safe_float).
Add-only; no existing code imports from this package yet. Existing
stream_new_chat.py keeps its inline equivalents until cutover.
The `documentsPanel` prop on `LayoutShell` declared `isDocked` and
`onDockedChange` alongside `open` / `onOpenChange`, but the shell never
forwarded those two extra fields to its consumers. `RightPanel` already
defines its own interface accepting only `open` / `onOpenChange`, and
`DocumentsSidebar` keeps both as optional props with safe fallbacks for
the rare cases that pass them directly.
Trim the interface to the two fields the shell actually plumbs through
and drop the matching `isDocumentsDocked` state and prop entries from
`LayoutDataProvider`, which was the only caller still populating the
dead fields (`FreeLayoutDataProvider` already passed the minimal pair).
`DocumentsSidebar` itself is untouched — its props remain optional so
direct consumers that want docking behaviour can still wire it.
Replace the boolean "skip first render" ref with a ref that stores the
previously-seen tick value. The effect now compares against the stored
value and only fires when it differs, which makes the dependency
naturally used (removes the `void slideoutOpenedTick;` acknowledgement)
and self-documents the intent of the guard.
Behavior is unchanged — both forms preserve the one-shot-per-event
semantics of the prior window-event implementation. The JSDoc on
`slideoutOpenedTickAtom` is updated to describe the new pattern.
PR #1428 (issue #1366) extracted the inline `hasPermission` callback into
a shared `canPerform` helper but left the original arrow-function body,
its dependency array, and trailing `)` behind after the new
`useCallback` block. The result was a syntactically invalid statement
that broke `pnpm build` on the `dev` branch and is now blocking every
E2E job in the PR queue.
Delete the orphaned lines so the file parses again. No behavior change —
the working `useCallback(canPerform(access, permission))` already
supplies the same predicate the duplicated body did.
Replace the `SLIDEOUT_PANEL_OPENED_EVENT` window event with a
`slideoutOpenedTickAtom` jotai atom. The dispatcher in
`SidebarSlideOutPanel` now bumps the tick via `useSetAtom`, and the
listener in `Thread` reads it via `useAtomValue` and reacts on change
behind a ref guard that skips the initial render — preserving the
one-shot-per-open semantics of the previous event.
This removes the implicit cross-module string contract, makes the
signal traceable through React DevTools / jotai inspector, and lets
TypeScript catch typos that the string-based event API silently
swallowed.
Replace the duplicated `OAUTH_RESULT_COOKIE` constant and inline payload
type across the callback route and connector dialog hook with a shared
`contracts/types/oauth.types.ts` module that exports:
- OAUTH_RESULT_COOKIE constant
- oauthCallbackResultSchema Zod schema
- OAuthCallbackResult type (inferred from the schema)
- parseOAuthCallbackResult() helper that returns null on invalid JSON
or shape mismatch
The route handler now uses the shared type to constrain the cookie
payload at compile time. The consumer hook validates the cookie value
through the helper instead of an unchecked JSON.parse, removing the
silent runtime risk when the cookie is tampered with or its shape
drifts.
electron-builder was given a single 2048x2048 icon.png and dumped it into
hicolor/2048x2048/apps/, a bucket no Linux desktop environment indexes —
launchers and taskbar fell back to a generic placeholder. Pre-render the
standard sizes (16, 32, 48, 64, 128, 256, 512, 1024) and point linux.icon
at the directory so each PNG lands in the matching hicolor/NxN/apps/ slot.
Tray icon was unaffected (loaded at runtime via Electron's Tray API from
bundled resources, no theme lookup).
Setting HOSTNAME=0.0.0.0 made Next.js standalone canonicalize request.url to
http://0.0.0.0:PORT. The connector OAuth callback's NextResponse.redirect built
its Location from that URL, so navigating it flipped window.location.origin from
http://localhost:PORT to http://0.0.0.0:PORT. The backend CORS allowlist matches
localhost/127.0.0.1 only, blocking every subsequent API call until app restart —
producing the "no internet" / app-down state after connecting any connector.
Adds two diagnostic events to surface OAuth-redirect failures we can't
reproduce on Linux:
- desktop_oauth_redirect_intercepted fires from inside onBeforeRequest
with the original host, path, and target port — confirms the rewrite
actually ran.
- desktop_oauth_redirect_missed fires from a read-only onCompleted
listener when a /dashboard/*/connectors/callback URL lands off-localhost,
meaning the rewrite filter didn't catch it. This is the smoking-gun
event for "connector OAuth dies on mac/win" reports.
Read-only; no behavior change.
The interceptor previously matched a strict `${HOSTED_FRONTEND_URL}/*`
prefix and did a naive String.replace, which broke whenever the backend
NEXT_FRONTEND_URL differed at all (apex vs www, http vs https, or a
self-hosted domain). Now:
- Match by host: apex + www. sibling, both http and https.
- Rewrite via URL parsing so only protocol/host change; query strings
containing the host as a value are left intact.
- Read HOSTED_FRONTEND_URL through getHostedFrontendUrl() which honors
a SURFSENSE_HOSTED_FRONTEND_URL_OVERRIDE env var, letting self-hosters
point their builds at their own frontend without rebuilding.
Default behavior is identical when override is unset and backend host
matches the baked-in value.
Linux registered the scheme via desktop-file MIME, but mac.extendInfo
never declared CFBundleURLTypes, leaving install-time LaunchServices
unaware of the protocol. The runtime app.setAsDefaultProtocolClient
call still runs as a fallback.
setupDeepLinks() only listened for second-instance and open-url events.
On Windows/Linux a fresh launch via `surfsense://` delivers the URL in
argv of the first instance, where it was silently dropped. Scan argv on
setup so the existing handlePendingDeepLink() pass picks it up.
The citations fix (cacb27e0) added a "Chunk citations in your prose"
section to system_prompt_desktop.md telling the KB subagent to always
leave `evidence.chunk_ids` null and emit no `[citation:...]` markers in
desktop mode, but left the pre-existing line declaring that
`chunk_ids` apply to `<priority_documents>` hits. The two rules
contradicted each other; the model picked one per turn.
Strike the stale conditional clause and point at the dedicated section
as the single source of truth. Matches the parallel line in
system_prompt_cloud.md and the already-consistent
system_prompt_readonly_desktop.md.
