merge dev into feature/apps-v1

Union resolutions — dev's changes kept, apps-v1 additions appended:
- use_case.ts: dev's meeting_prep + our app_llm_generate/app_copilot_run
- ipc.ts: dev's HttpAuthRequestSchema import + our AppSummarySchema
- shared/index.ts: dev's channels export + our rowboatApp export
- builtin-tools app-navigation: dev's read-view/open-item actions + our open-app
- chat-conversation: dev's read-view/open-item labels + our open-app labels
- sidebar: dev's nav-workspaces tour id + our Apps entry
- App.tsx: dev's open-item handler + our open-app handler
This commit is contained in:
Gagan 2026-07-05 01:08:35 +05:30
commit dc28b0b868
186 changed files with 29957 additions and 1357 deletions

40
.github/workflows/x-tests.yml vendored Normal file
View file

@ -0,0 +1,40 @@
name: apps/x tests
on:
pull_request:
paths:
- "apps/x/**"
- ".github/workflows/x-tests.yml"
push:
branches: [main]
paths:
- "apps/x/**"
- ".github/workflows/x-tests.yml"
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
defaults:
run:
working-directory: apps/x
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: 10
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
cache-dependency-path: apps/x/pnpm-lock.yaml
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Builds @x/shared (core/renderer tests import its dist), then runs
# the shared, core, and renderer vitest suites in order.
- name: Run tests
run: npm test

3
.gitignore vendored
View file

@ -4,3 +4,6 @@
data/
.venv/
.claude/
# Local-only meeting-prep planning doc
/PLAN.md

101
AGENTS.md Normal file
View file

@ -0,0 +1,101 @@
# AGENTS.md — Working on the Rowboat runtime
Context for AI coding agents (and humans) working on this repo. General
codebase orientation lives in `CLAUDE.md`; this file covers the **new
turn/session runtime** in `apps/x` — its storage, its debugging tools, and
the invariants you must not break.
## The runtime in one paragraph
Chats are **sessions**; each user message starts a **turn**. Both are
append-only JSONL event logs under `~/.rowboat/storage/{turns,sessions}/YYYY/MM/DD/`.
All state is derived by pure reducers (`reduceTurn`, `reduceSession` in
`@x/shared/src/turns.ts` / `sessions.ts`) shared byte-for-byte between the
main process and the renderer. Design specs:
`apps/x/packages/core/docs/turn-runtime-design.md` and `session-design.md`
read the relevant spec before changing runtime behavior.
## Storage is reference-based — files store each fact once
Three applications of the same mechanism:
1. **Context**: a session turn's `context` is `{ previousTurnId }`; the
conversation prefix is materialized by walking the chain
(`TurnRepoContextResolver`).
2. **Model requests**: `model_call_requested.request.messages` is a list of
string refs into the turn's own events — `"context"`, `"input"`,
`"assistant:<index>"`, `"toolResult:<toolCallId>"` — recording only what
is NEW since the previous call.
3. **Agent snapshots**: when a turn's system prompt + tools are
byte-identical to its predecessor's, `turn_created.agent.resolved` is
`{ agentId, model, inheritedFrom }` instead of re-persisting ~70KB. The
model stays concrete (mid-session model switches still inherit).
The exact provider payload is rebuilt by `composeModelRequest`
(`packages/core/src/turns/compose-model-request.ts`) — **the same code path
the loop transmits through**, so the file plus the composer reproduce the
wire bytes exactly (there is a property test asserting composed == sent).
## Inspecting turns and sessions
```bash
cd apps/x/packages/core
# A whole session: title, per-turn status/size/input preview
npm run inspect -- <sessionId | path/to/session.jsonl>
# One turn: per model call, the EXACT provider payload — resolved system
# prompt, tool list, wire-form messages (user-message context woven in,
# tool-result envelopes), and the response/failure
npm run inspect -- <turnId | path/to/turn.jsonl> [modelCallIndex] [--full]
# Cascade full turn inspection across a session
npm run inspect -- <sessionId> --turns
```
`--full` prints untruncated system prompts and message contents. Turn vs
session ids are auto-detected. This is the intended way to see "what did the
model actually receive" — the raw JSONL deliberately stores structural facts
and references, never the derived wire form.
## The legacy runs runtime is code-mode-only
The old runs runtime (`packages/core/src/runs/`, `AgentRuntime`/`streamAgent`
in `packages/core/src/agents/runtime.ts`, the `runs:*` IPC channels, and the
renderer's legacy chat-tab state machine in `App.tsx`) remains in the tree
**solely for code-mode sessions** — a deliberate, documented carve-out:
- A code session IS a run (`sessionId === runId`). Direct-mode prompts drive
an external ACP agent and hand-assemble `code-run-event`s into the run log
(`code-mode/sessions/service.ts`); rowboat-mode code tabs go through
`runs:createMessage` → the old `AgentRuntime` loop.
- Everything else — main chat, background tasks, live notes, knowledge
pipelines, scheduled agents — runs on turns/sessions. Do NOT add new
callers of `createRun`/`createMessage`/`streamAgent`; headless work uses
`packages/core/src/agents/headless.ts` (`startHeadlessAgent`/
`runHeadlessAgent`).
- Temporary bridges that die when code-mode is unified: the renderer
transcript loader's `runs:fetch` fallback (`lib/agent-transcript.ts`), the
`runs:downloadLog` fallback in the chat sidebar, and the `notify-user`
gate's `fetchRun` fallback (`application/lib/builtin-tools.ts`).
- The remaining migration (designed but deferred): port code sessions onto
sessions/turns — rowboat-mode prompts become normal turns with
`codeMode`/`codeCwd` composition (already supported by the agent
resolver's composition overrides), direct-mode prompts become a delegated
turn kind carrying opaque code events. That project deletes `runs/`
entirely.
## Invariants to respect
- Turn/session files are **append-only**; reducers reject impossible
histories loudly (`TurnCorruptionError`). Never hand-edit files.
- Durable events are persisted **before** side effects (model calls, tool
invocations). Deltas (`text_delta`, …) are stream-only, never persisted.
- The reducers in `@x/shared` must stay pure (no I/O, no node imports) —
the renderer imports them directly.
- Every behavior change needs tests: reducers in `packages/shared`,
runtime/sessions in `packages/core`, renderer stores/views in
`apps/renderer` (all vitest; run `npm test` per package).
- Schema changes: the schema is pre-release (`schemaVersion: 1` throughout);
breaking changes are acceptable but require wiping `~/.rowboat/storage`
and a note in the commit message.

View file

@ -109,7 +109,9 @@ Long-form docs for specific features. Read the relevant file before making chang
| Feature | Doc |
|---------|-----|
| Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` |
| Calls (video mode) — one hands-free call engine with four presets (voice / video / share screen / practice coaching), device-derived surfaces (full-screen ⇄ floating popout), frame pipeline, prompts catalog | `apps/x/VIDEO_MODE.md` |
| Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` |
| Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` |
## Common Tasks

1
apps/x/.gitignore vendored
View file

@ -1,2 +1,3 @@
node_modules/
test-fixtures/
*.tsbuildinfo

Binary file not shown.

View file

@ -92,6 +92,8 @@ All in `apps/renderer/src/lib/analytics.ts`:
- `chat_message_sent``{ voice_input, voice_output, search_enabled }`
- `oauth_connected` / `oauth_disconnected``{ provider }`
- `voice_input_started` — no properties
- `call_started``{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`)
- `call_turn_latency``{ endpoint_to_submit_ms, submit_to_speak_ms, speak_to_audio_ms, total_ms }` — voice-to-voice latency breakdown for one call turn (utterance accepted → submitted → first TTS speak → audio playing)
- `search_executed``{ types: string[] }`
- `note_exported``{ format }`

243
apps/x/VIDEO_MODE.md Normal file
View file

@ -0,0 +1,243 @@
# Calls (Video Mode) — Deep Dive
Calls let the user talk to the assistant hands-free while it *sees* them
(webcam) and their screen (screen share). There is ONE call engine —
continuous listening, auto-submitted utterances, forced read-aloud TTS, frame
capture — entered through four presets that differ only in starting devices.
This doc covers the product flow, the technical pipeline, and the LLM prompt
surface with exact pointers.
## Product flow
The composer has a **call split-button** (`chat-input-with-mentions.tsx`).
The main click is the "work together" default — preset `share`: screen
sharing ON, camera OFF, floating pill, so the user keeps working while the
assistant watches along (the button tooltip discloses the screen share). The
chevron menu holds the deviations. While a call is live the button turns red
and ends it.
| Preset | Starting devices | First surface |
|--------|------------------|---------------|
| `share` — main click | screen on, camera off | floating pill |
| `voice` — "Voice call" | camera off, screen off | floating mascot pill |
| `video` — "Video call" | camera on | full-screen call |
| `practice` — "Practice session" | camera on, + coaching persona | full-screen call |
**One surface rule** (`callSurface` in `App.tsx`): full screen and screen
sharing are mutually exclusive in both directions — a full-screen call covers
the screen, so sharing it would show the call itself.
- sharing → floating popout, always (pill = working)
- not sharing → full screen unless `callMinimized` (full screen = facing
each other)
- expanding the pill auto-STOPS any share; minimizing the full-screen call
auto-STARTS one (the pill exists to work together) — presenting from full
screen likewise collapses to the pill
- the camera toggle never changes the surface: turning it on from the pill
puts your video IN the pill; expanding is its own explicit action
**Screen-share consent** is three-layered: a toast the moment any share
starts ("Your screen is being shared… [Stop sharing]"), a persistent
"Sharing screen" badge on the pill, and macOS's purple recording indicator.
If the auto-share fails (Screen Recording permission not granted) the call
starts anyway as a voice call, with a toast linking to System Settings.
Practice/coaching is always an explicit choice — expanding to full screen
never turns the coach on.
In-call controls (identical bar on both surfaces): camera toggle (silhouette
avatar while off, no webcam frames captured), screen share toggle, mascot ⇄
"R" letter avatar, end call. While the assistant is thinking or speaking, a
red **Stop** button appears on the mascot tile — it silences TTS instantly,
skips queued voice segments, and aborts the run if it's still generating
(stopping a run from anywhere, including the composer, also silences TTS). Captions of the in-progress utterance and the
assistant's spoken line run along the bottom. Typing in the composer still
works mid-call; frames ride along with typed messages too.
Outside calls the composer keeps exactly one voice affordance: the **mic
button** (push-to-talk dictation, untouched). Spoken responses exist only
inside calls (forced full read-aloud, off on hang-up). The old video
dropdown, talking-head toggle, read-aloud headphones toggle, and summary/full
TTS dropdown are all retired — a per-message "read aloud" action on assistant
messages is the planned replacement for text-in/voice-out.
The call button is disabled unless both voice input (Deepgram) and voice
output (TTS) are configured. `call_started` (with `preset`) is captured in
PostHog — the adoption metric for this feature.
**Popout mechanics**: a small always-on-top frameless window (camera tile
when on + mascot tile, live caption, control bar) floating over every app —
including Rowboat. Control-bar actions round-trip `video:popoutAction`
main → `video:popout-action` → app window, which owns the mic/camera/capture;
`expand` also refocuses the app window (handled in main).
## Frame pipeline
`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source
(stream → offscreen `<video>` → canvas JPEG → ring buffer):
- Cadence: 1 fps (`CAPTURE_INTERVAL_MS`, line 20); ring buffer ~2 min.
- Webcam: 512px wide, JPEG q0.65, max **12 frames/message** (lines 21, 31).
- Screen: 1280px wide (text legibility), JPEG q0.7, max **4 frames/message**
(lines 24, 32).
- `collectFrames()` drains frames buffered since the last send, evenly
sampled down to the caps, always keeping the newest; grabs one final frame
at the moment of send. Falls back to the single latest frame for
rapid-fire messages.
`App.tsx` `handlePromptSubmit` attaches the drained frames (whenever a call
is live) to the outgoing message as `UserImagePart`s and sets
`composition.videoMode` when the camera or screen is active, plus
`composition.coachMode` during a practice session. Frames also become
`isVideoFrame` display attachments (filmstrip in the transcript —
`chat-message-attachments.tsx`; history hydration in
`lib/run-to-conversation.ts`).
## Message schema & model encoding
- `packages/shared/src/message.ts:51``UserImagePart`: inline base64
(`data`, `mediaType`), `source: 'camera' | 'screen'`, `capturedAt`. Unlike
file attachments (path references read via the `LLMParse` tool), image
parts go to the model as real multimodal image parts.
- `packages/core/src/agents/runtime.ts` `convertFromMessages` (~line 1013):
emits a context line (frame counts + time span), then labeled groups —
a `"Webcam frames (oldest to newest):"` text part before camera images and
a `"Screen-share frames (oldest to newest):"` text part before screen
images — so the model never confuses the user with their screen.
- Frames stay inline in history (no pruning) deliberately: pruning would
bust provider prefix caching every turn and cost more than it saves.
- The auto-permission classifier stringifies + truncates content to ~3KB per
message, so inline base64 can't blow up its prompt.
## Hands-free voice loop
`apps/renderer/src/hooks/useVoiceMode.ts`:
- `startContinuous(onUtterance)` (line 404): push-to-talk params but with
`endpointing=1800` (line 25) so thinking pauses don't cut the user off,
plus `utterance_end_ms=2000` (line 38) as a second end-of-speech signal.
**Gotcha:** Deepgram's `speech_final` usually arrives on a result with an
EMPTY transcript — empty finals must reach the endpoint check or
utterances never complete (see the NOTE in `ws.onmessage`).
- `setPaused(true)` (line 414) while the assistant thinks/speaks: drops mic
audio (so TTS is never transcribed back), discards half-heard buffer,
sends Deepgram KeepAlives every 5s. `App.tsx` drives this from
`activeIsProcessing || tts.state !== 'idle'`.
- Mid-call socket drops reconnect after 1s; the offline audio backlog is
capped (~30s).
Call lifecycle lives in `App.tsx` `startCall(preset)` / `endCall()`:
entering a call saves/forces TTS settings, cancels any push-to-talk
recording, and starts the continuous loop; ending restores everything.
Push-to-talk is disabled while a call owns the mic.
## Popout window
- The popout window keeps the Dock icon alive: it uses
`setVisibleOnAllWorkspaces(true)` WITHOUT `visibleOnFullScreen` — that flag
turns the app into a macOS "agent" app and hides its Dock icon while the
window exists (looks like Rowboat vanished). Trade-off: the popout doesn't
hover over other apps' fullscreen Spaces.
- Shown iff the derived `callSurface === 'popout'` (effect in `App.tsx`).
Renderer asks `video:setPopout {show}`; main creates a frameless,
`alwaysOnTop` ('floating'), all-workspaces BrowserWindow at the top-right
of the primary display, loading the renderer bundle with `#video-popout`
(`apps/renderer/src/main.tsx` branches on the hash →
`components/video-popout.tsx`).
- Call state streams over the `video:popout-state` push channel; main caches
the last payload and replays it on popout load. Shown with
`showInactive()` so it never steals focus.
- The popout captures its **own** camera preview (MediaStreams can't cross
windows) and synthesizes the mascot mouth level (no audio in that window).
- `video:popoutAction` relays control-bar actions to the app window, matched
only by real app-window URLs — `getAllWindows()` also contains hidden
utility windows (PDF export) that must not be shown or messaged.
## Permissions
- Camera: `voice:ensureCameraAccess` settles the macOS TCC prompt before
`getUserMedia` (same pattern as the mic). `NSCameraUsageDescription` is in
`forge.config.cjs` `extendInfo`.
- Screen: `getDisplayMedia` is auto-approved with the primary screen by
`setDisplayMediaRequestHandler` in `main.ts` (no picker);
`meeting:checkScreenPermission` registers the app in macOS Screen
Recording settings on first use.
## LLM prompts catalog
| Prompt | Where |
|--------|-------|
| `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) |
| `# Practice Session (Coach Mode)` system section — coaching persona: specific/actionable feedback after each take, one-sentence interjections mid-flow, structured debrief on wrap-up | `composeSystemInstructions`, gated on `coachMode` (directly after the video section) |
| "Driving the app" paragraph in the video-mode section — on calls, prefer app-navigation read-view/open-item (show while telling) over describing or squinting at frames | same `# Video Mode` section; full action docs in the `app-navigation` skill (`application/assistant/skills/app-navigation/skill.ts`) |
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts` (`convertFromMessages`) |
| `videoMode` / `coachMode` composition overrides (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts` (`CompositionOverrides`); set from `App.tsx` `sendConfig` |
Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are
reused untouched — calls set `voiceInput` per utterance and force
`voiceOutput: 'full'`.
## Driving the app on a call
The assistant can drive the Rowboat UI itself via the extended
`app-navigation` builtin ("app driver"): `open-view` (any main view),
`read-view` (returns the emails / background agents / chat-history data the
view renders — and the renderer simultaneously navigates there so the user
watches it happen), and `open-item` (a specific email thread, note,
background agent, or past chat, deep-linked on screen). Data comes from the
same core functions the UI's IPC handlers use (`listImportantThreads` /
`searchThreads`, background-task `listTasks`, the sessions container) — no
OCR of screen frames. The renderer applies results via
`applyAppNavigation` in App.tsx, fed from BOTH event paths: the legacy
`runs:events` ref-poll AND a watcher over the session-chat conversation (the
turn runtime does not emit legacy run events — miss this and navigation
silently no-ops while the tool reports success). Session switches seed the
watcher so replaying history never navigates. During a call, visible
navigations also collapse the full-screen call to the pill and focus the app
window (`app:focusMainWindow`) so the user actually sees the screen change.
Card labels live in `lib/chat-conversation.ts`. The call prompt and the
`app-navigation` skill teach the show-while-telling pattern: read-view →
speak the highlights → open-item when the user picks one.
## Latency
Voice-to-voice latency (user stops talking → assistant audio) is engineered
at four points; the `call_turn_latency` PostHog event measures the real
distribution (utterance → submit → first speak → audio playing):
- **Smart endpointing** (`useVoiceMode.ts`): Deepgram endpoints at 600ms and
the client decides — a transcript ending in terminal punctuation fires
immediately (~600ms after last word); a mid-thought trail holds another
1.2s (resumed speech cancels the hold). Complete sentences turn around
~1.2s faster than the old fixed 1800ms endpoint.
- **Streaming TTS** (`voice:synthesizeStreamStart``voice:tts-chunk`
MediaSource playback in `useVoiceTTS.ts`): the first segment of an idle
queue plays from the first MP3 chunk instead of after the full body
(ElevenLabs `/stream`, flash model). Follow-up segments keep the gapless
full-body prefetch path. Falls back to non-streaming on any failure.
- **Early clause speech** (`turn-view.ts` `applyOverlay`): a still-open
`<voice>` block ≥60 chars emits its last complete clause immediately, so
speech starts while the rest of the sentence generates.
- **Acknowledgment cue** (`lib/call-sounds.ts`): a soft blip the instant an
utterance is accepted — perceived latency matters as much as measured.
## Cost notes
Webcam frames ≈ 250350 tokens each (≤12/message ≈ 34k); screen frames ≈
1.52k tokens each (≤4/message ≈ 68k). History keeps frames inline, so long
sessions grow but stay prefix-cached. First lever if cost bites: drop to one
screen frame per message unless the screen changed.
## Known limitations
- Turn-taking is strict — no barge-in (would need echo cancellation against
TTS output).
- Frame sampling, not video: motion between frames is invisible (the prompt
tells the model not to claim otherwise).
- Vocal-delivery feedback is limited: Deepgram reduces speech to text, so
"energy" coaching leans on visual cues.
- Screen share always captures the primary display (no window/display
picker yet).
- The full-screen call covers the chat; there's no in-call transcript drawer.
- The "attach camera frames to typed chat without a call" combination (the
old video+chat mode) was cut in the call-model simplification; if analytics
show demand, it should return as an attachment chip, not a mode.

View file

@ -199,6 +199,7 @@ module.exports = {
],
extendInfo: {
NSAudioCaptureUsageDescription: 'Rowboat needs access to system audio to transcribe meetings from other apps (Zoom, Meet, etc.)',
NSCameraUsageDescription: 'Rowboat uses your camera in video chat mode so the assistant can see you and give feedback (e.g. pitch practice).',
},
osxSign: {
batchCodesignCalls: true,

View file

@ -1,6 +1,6 @@
import { BrowserWindow } from 'electron';
import { ipc } from '@x/shared';
import { browserViewManager, type BrowserState } from './view.js';
import { browserViewManager, type BrowserState, type HttpAuthRequest } from './view.js';
type IPCChannels = ipc.IPCChannels;
@ -20,6 +20,7 @@ type BrowserHandlers = {
'browser:forward': InvokeHandler<'browser:forward'>;
'browser:reload': InvokeHandler<'browser:reload'>;
'browser:getState': InvokeHandler<'browser:getState'>;
'browser:httpAuthResponse': InvokeHandler<'browser:httpAuthResponse'>;
};
/**
@ -62,6 +63,9 @@ export const browserIpcHandlers: BrowserHandlers = {
'browser:getState': async () => {
return browserViewManager.getState();
},
'browser:httpAuthResponse': async (_event, args) => {
return browserViewManager.respondToHttpAuth(args);
},
};
/**
@ -70,12 +74,26 @@ export const browserIpcHandlers: BrowserHandlers = {
* window is created so the manager has a window to attach to.
*/
export function setupBrowserEventForwarding(): void {
browserViewManager.on('state-updated', (state: BrowserState) => {
const windows = BrowserWindow.getAllWindows();
for (const win of windows) {
if (!win.isDestroyed() && win.webContents) {
win.webContents.send('browser:didUpdateState', state);
}
// Only send to app windows, never to OAuth/SSO popup windows created by
// page window.open() — those render untrusted web content, and browsing
// state / auth-challenge metadata must not cross into them.
const broadcast = (channel: string, payload: unknown) => {
for (const win of BrowserWindow.getAllWindows()) {
if (win.isDestroyed() || !win.webContents) continue;
if (browserViewManager.isPopupWindow(win)) continue;
win.webContents.send(channel, payload);
}
};
browserViewManager.on('state-updated', (state: BrowserState) => {
broadcast('browser:didUpdateState', state);
});
browserViewManager.on('http-auth-request', (request: HttpAuthRequest) => {
broadcast('browser:httpAuthRequest', request);
});
browserViewManager.on('http-auth-resolved', (requestId: string) => {
broadcast('browser:httpAuthResolved', { requestId });
});
}

View file

@ -1,11 +1,12 @@
import { randomUUID } from 'node:crypto';
import { EventEmitter } from 'node:events';
import { BrowserWindow, WebContentsView, session, shell, type Session } from 'electron';
import { BrowserWindow, WebContentsView, session, shell, type Session, type WebContents } from 'electron';
import type {
BrowserPageElement,
BrowserPageSnapshot,
BrowserState,
BrowserTabState,
HttpAuthRequest,
} from '@x/shared/dist/browser-control.js';
import { normalizeNavigationTarget } from './navigation.js';
import {
@ -20,7 +21,7 @@ import {
type RawBrowserPageSnapshot,
} from './page-scripts.js';
export type { BrowserPageSnapshot, BrowserState, BrowserTabState };
export type { BrowserPageSnapshot, BrowserState, BrowserTabState, HttpAuthRequest };
/**
* Embedded browser pane implementation.
@ -36,13 +37,33 @@ export type { BrowserPageSnapshot, BrowserState, BrowserTabState };
export const BROWSER_PARTITION = 'persist:rowboat-browser';
// Claims Chrome 130 on macOS — close enough to recent stable for OAuth servers
// that sniff the UA looking for "real browser" shapes.
const SPOOF_UA =
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36';
// Spoof a real Chrome UA so OAuth servers don't reject the embedded browser.
// The Chrome major version is derived from the running Chromium at startup:
// pinning a fixed version goes stale as Electron upgrades, and Chromium keeps
// emitting Sec-CH-UA client hints with the *real* version — a UA/client-hint
// version mismatch is a classic bot-detection signal (Google sign-in,
// Cloudflare). Minor version is frozen at 0.0.0, exactly like real Chrome's
// reduced UA. The platform token matches the actual OS for the same reason.
function getChromeMajorVersion(): number {
const major = Number.parseInt(process.versions.chrome ?? '', 10);
return Number.isFinite(major) && major > 0 ? major : 130;
}
function buildChromeUserAgent(): string {
const platformToken =
process.platform === 'darwin'
? 'Macintosh; Intel Mac OS X 10_15_7'
: process.platform === 'win32'
? 'Windows NT 10.0; Win64; x64'
: 'X11; Linux x86_64';
return `Mozilla/5.0 (${platformToken}) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${getChromeMajorVersion()}.0.0.0 Safari/537.36`;
}
const SPOOF_UA = buildChromeUserAgent();
const HOME_URL = 'https://www.google.com';
const NAVIGATION_TIMEOUT_MS = 10000;
const HTTP_AUTH_TIMEOUT_MS = 120000;
const POST_ACTION_IDLE_MS = 400;
const POST_ACTION_MAX_ELEMENTS = 25;
const POST_ACTION_MAX_TEXT_LENGTH = 4000;
@ -68,6 +89,13 @@ type CachedSnapshot = {
elements: Array<{ index: number; selector: string }>;
};
type PendingHttpAuth = {
callback: (username?: string, password?: string) => void;
timer: NodeJS.Timeout;
// The webContents that raised the challenge, so its teardown can cancel it.
webContents: WebContents;
};
const EMPTY_STATE: BrowserState = {
activeTabId: null,
tabs: [],
@ -109,6 +137,12 @@ export class BrowserViewManager extends EventEmitter {
private visible = false;
private bounds: BrowserBounds = { x: 0, y: 0, width: 0, height: 0 };
private snapshotCache = new Map<string, CachedSnapshot>();
private pendingHttpAuth = new Map<string, PendingHttpAuth>();
// Child windows created by page window.open() (OAuth/SSO popups). Tracked so
// they can be closed when the host window goes away — otherwise an orphaned
// popup keeps BrowserWindow.getAllWindows() non-empty and, on macOS, blocks
// the app from reopening via the Dock (see main.ts 'activate' handler).
private popupWindows = new Set<BrowserWindow>();
private cleanupWindowListeners: (() => void) | null = null;
attach(window: BrowserWindow): void {
@ -137,6 +171,7 @@ export class BrowserViewManager extends EventEmitter {
if (this.window !== window) return;
const tabs = [...this.tabs.values()];
const popups = [...this.popupWindows];
this.cleanupWindowListeners = null;
this.window = null;
this.browserSession = null;
@ -150,6 +185,14 @@ export class BrowserViewManager extends EventEmitter {
this.attachedTabId = null;
this.visible = false;
this.snapshotCache.clear();
for (const requestId of [...this.pendingHttpAuth.keys()]) {
this.finishHttpAuth(requestId);
}
// Close any OAuth/SSO popups so they don't outlive the app window.
for (const popup of popups) {
if (!popup.isDestroyed()) popup.close();
}
this.popupWindows.clear();
};
hostWebContents.on('did-start-loading', handleDidStartLoading);
@ -171,6 +214,36 @@ export class BrowserViewManager extends EventEmitter {
if (this.browserSession) return this.browserSession;
const browserSession = session.fromPartition(BROWSER_PARTITION);
browserSession.setUserAgent(SPOOF_UA);
// Electron's Sec-CH-UA client hints only carry the "Chromium" brand;
// real Chrome also sends "Google Chrome". Some sign-in flows (notably
// Google's) distinguish the two, so rewrite the brand list to match what
// Chrome sends. Both the low-entropy header (`sec-ch-ua`, major versions)
// and the high-entropy one (`sec-ch-ua-full-version-list`, requested via
// Accept-CH and carrying full versions) must be rewritten together — a
// header that claims "Google Chrome" alongside one that doesn't is a
// stronger bot signal than the original. Only headers Chromium already
// attached are rewritten — none are added. (navigator.userAgentData JS
// brands still report only Chromium; there is no reliable hook to spoof
// that under sandbox+contextIsolation, and header-based detection is the
// common case.)
const chromeMajor = getChromeMajorVersion();
const chromeFull = process.versions.chrome ?? `${chromeMajor}.0.0.0`;
const brandLists: Record<string, string> = {
'sec-ch-ua': `"Chromium";v="${chromeMajor}", "Google Chrome";v="${chromeMajor}", "Not-A.Brand";v="99"`,
'sec-ch-ua-full-version-list': `"Chromium";v="${chromeFull}", "Google Chrome";v="${chromeFull}", "Not-A.Brand";v="99.0.0.0"`,
};
browserSession.webRequest.onBeforeSendHeaders((details, callback) => {
const requestHeaders = details.requestHeaders;
for (const name of Object.keys(requestHeaders)) {
const replacement = brandLists[name.toLowerCase()];
if (replacement !== undefined) {
requestHeaders[name] = replacement;
}
}
callback({ requestHeaders });
});
this.browserSession = browserSession;
return browserSession;
}
@ -196,17 +269,34 @@ export class BrowserViewManager extends EventEmitter {
return /^https?:\/\//i.test(url) || url === 'about:blank';
}
/**
* webPreferences shared by browser tabs and OAuth popups. Kept in one place
* so the security-sensitive popup surface can never drift from tabs.
*/
private browserWebPreferences(): Electron.WebPreferences {
return {
session: this.getSession(),
contextIsolation: true,
sandbox: true,
nodeIntegration: false,
// Chromium's built-in PDFium viewer, so PDFs render inline instead
// of showing a blank page.
plugins: true,
// Remove the WebAuthn API from the embedded browser only. Electron ships
// the API but not Chrome's authenticator UI (Touch ID sheet, QR/phone
// hybrid), so passkey challenges hang forever on "Verifying it's you...".
// With the API absent, sites feature-detect it and fall back to
// password/other verification. Scoped here (not app-wide) so the app's
// own renderer keeps WebAuthn.
disableBlinkFeatures: 'WebAuth',
};
}
private createView(): WebContentsView {
const view = new WebContentsView({
webPreferences: {
session: this.getSession(),
contextIsolation: true,
sandbox: true,
nodeIntegration: false,
},
webPreferences: this.browserWebPreferences(),
});
view.webContents.setUserAgent(SPOOF_UA);
return view;
}
@ -266,16 +356,145 @@ export class BrowserViewManager extends EventEmitter {
});
wc.on('page-title-updated', this.emitState.bind(this));
wc.setWindowOpenHandler(({ url }) => {
if (this.isEmbeddedTabUrl(url)) {
void this.newTab(url);
} else {
void shell.openExternal(url);
this.wireWindowPolicy(wc);
}
/**
* Window-open, popup, and HTTP-auth wiring shared by tabs and popups.
*/
private wireWindowPolicy(wc: WebContents): void {
wc.setWindowOpenHandler((details) => this.handleWindowOpen(details));
wc.on('did-create-window', (child) => this.wirePopupWindow(child));
this.wireHttpAuth(wc);
}
/**
* Shared window.open / target=_blank policy for tabs and popups.
*
* An open that hands a handle back to the opener must become a real child
* window so window.opener / postMessage survive this is how OAuth/SSO
* popups (Google, Microsoft, Plaid, ...) return their result; denying them
* also makes sites report "popup blocked". Those are: a sized popup
* (disposition 'new-window'), a *named* window.open(url, 'name') (non-empty
* frameName), or a scripted blank window the opener will populate
* (about:blank). A nameless target=_blank link (foreground-tab, empty
* frameName) has no opener contract and opens as a tab, matching browser
* behavior. Non-web schemes go to the system handler.
*
* Residual gap: a nameless, featureless window.open(url) is indistinguishable
* from a _blank link (both foreground-tab + empty frameName) and opens as a
* tab, losing its opener rare for OAuth, which virtually always names or
* sizes its popup.
*/
private handleWindowOpen(details: Electron.HandlerDetails): Electron.WindowOpenHandlerResponse {
const { url, disposition, frameName } = details;
if (this.isEmbeddedTabUrl(url)) {
const needsOpener =
disposition === 'new-window' || frameName !== '' || url === 'about:blank';
if (needsOpener) {
return {
action: 'allow',
overrideBrowserWindowOptions: {
autoHideMenuBar: true,
webPreferences: this.browserWebPreferences(),
},
};
}
void this.newTab(url);
return { action: 'deny' };
}
void shell.openExternal(url);
return { action: 'deny' };
}
private wirePopupWindow(child: BrowserWindow): void {
this.popupWindows.add(child);
child.once('closed', () => this.popupWindows.delete(child));
this.wireWindowPolicy(child.webContents);
}
/** True if `win` is an OAuth/SSO popup created by page window.open(). */
isPopupWindow(win: BrowserWindow): boolean {
return this.popupWindows.has(win);
}
/**
* HTTP basic/proxy auth. Chromium's default is to cancel the challenge, so
* 401-protected sites and authenticating proxies dead-end. When the browser
* pane is on screen to answer, forward the challenge to it as a credential
* prompt (cancelled after a timeout if unanswered). When the pane is closed
* e.g. agent-driven navigation don't preventDefault, so Chromium cancels
* immediately and the 401 page is readable rather than hanging.
*/
private wireHttpAuth(wc: WebContents): void {
wc.on('login', (event, _details, authInfo, callback) => {
if (!this.visible || !this.window) return;
event.preventDefault();
const requestId = randomUUID();
const timer = setTimeout(() => {
this.finishHttpAuth(requestId);
}, HTTP_AUTH_TIMEOUT_MS);
this.pendingHttpAuth.set(requestId, { callback, timer, webContents: wc });
// If the challenging contents dies before an answer, resolve now so the
// native callback and timer don't leak (backstop for paths other than
// destroyTab, which cancels explicitly before removeAllListeners()).
wc.once('destroyed', () => this.finishHttpAuth(requestId));
const request: HttpAuthRequest = {
requestId,
host: authInfo.host,
isProxy: authInfo.isProxy,
...(authInfo.realm ? { realm: authInfo.realm } : {}),
};
this.emit('http-auth-request', request);
});
}
/**
* Resolve a pending auth challenge. `username === undefined` cancels it; an
* empty-string username is a valid submission (token-style Basic auth).
* Always notifies the renderer so a dialog it may still be showing (e.g.
* after a timeout or tab close) is pruned.
*/
private finishHttpAuth(requestId: string, username?: string, password?: string): boolean {
const pending = this.pendingHttpAuth.get(requestId);
if (!pending) return false;
this.pendingHttpAuth.delete(requestId);
clearTimeout(pending.timer);
try {
if (username == null) {
pending.callback();
} else {
pending.callback(username, password ?? '');
}
} catch {
// The challenged webContents may already be destroyed.
}
this.emit('http-auth-resolved', requestId);
return true;
}
private cancelHttpAuthForWebContents(wc: WebContents): void {
const ids: string[] = [];
for (const [requestId, pending] of this.pendingHttpAuth) {
if (pending.webContents === wc) ids.push(requestId);
}
for (const requestId of ids) {
this.finishHttpAuth(requestId);
}
}
respondToHttpAuth(input: {
requestId: string;
username?: string;
password?: string;
}): { ok: boolean } {
return { ok: this.finishHttpAuth(input.requestId, input.username, input.password) };
}
private snapshotTabState(tab: BrowserTab): BrowserTabState {
const wc = tab.view.webContents;
return {
@ -364,6 +583,10 @@ export class BrowserViewManager extends EventEmitter {
private destroyTab(tab: BrowserTab): void {
this.invalidateSnapshot(tab.id);
// Cancel any auth challenge this tab raised before we drop its listeners,
// so the native callback + timer don't leak and the renderer prunes its
// dialog (removeAllListeners() below would kill the 'destroyed' backstop).
this.cancelHttpAuthForWebContents(tab.view.webContents);
tab.view.webContents.removeAllListeners();
if (!tab.view.webContents.isDestroyed()) {
tab.view.webContents.close();

View file

@ -1,7 +1,8 @@
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app } from 'electron';
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app, screen } from 'electron';
import { ipc } from '@x/shared';
import path from 'node:path';
import os from 'node:os';
import { fileURLToPath } from 'node:url';
import {
connectProvider,
disconnectProvider,
@ -24,6 +25,8 @@ const execFileAsync = promisify(execFile);
import { RunEvent } from '@x/shared/dist/runs.js';
import { ServiceEvent } from '@x/shared/dist/service-events.js';
import type { SessionBusEvent } from '@x/shared/dist/sessions.js';
import type { ISessions, EmitterSessionBus } from '@x/core/dist/sessions/index.js';
import container from '@x/core/dist/di/container.js';
import { listOnboardingModels } from '@x/core/dist/models/models-dev.js';
import { testModelConnection, listModelsForProvider, generateOneShot } from '@x/core/dist/models/models.js';
@ -49,6 +52,8 @@ import type { CodeSession } from '@x/shared/dist/code-sessions.js';
import { invalidateCopilotInstructionsCache } from '@x/core/dist/application/assistant/instructions.js';
import { triggerSync as triggerGranolaSync } from '@x/core/dist/knowledge/granola/sync.js';
import { ISlackConfigRepo } from '@x/core/dist/slack/repo.js';
import { IChannelsConfigRepo } from '@x/core/dist/channels/repo.js';
import { applyChannelsConfig, getChannelsStatus, logoutWhatsApp, subscribeChannelsStatus } from '@x/core/dist/channels/service.js';
import { runAgentSlack, getAgentSlackCliStatus, AgentSlackRunError } from '@x/core/dist/slack/agent-slack-exec.js';
import { knowledgeSourcesRepo } from '@x/core/dist/knowledge/sources/repo.js';
import { rankSlackHomeMessages } from '@x/core/dist/knowledge/sources/rank_slack_home.js';
@ -66,6 +71,9 @@ import { IAgentScheduleRepo } from '@x/core/dist/agent-schedule/repo.js';
import { IAgentScheduleStateRepo } from '@x/core/dist/agent-schedule/state-repo.js';
import { triggerRun as triggerAgentScheduleRun } from '@x/core/dist/agent-schedule/runner.js';
import { search } from '@x/core/dist/search/search.js';
import { resolveMeetingPrep } from '@x/core/dist/knowledge/meeting_prep.js';
import { readPrepNoteForEvent } from '@x/core/dist/knowledge/meeting_prep_brief.js';
import { invalidateKnowledgeIndex } from '@x/core/dist/knowledge/knowledge_index.js';
import { versionHistory, voice } from '@x/core';
import { classifySchedule, processRowboatInstruction } from '@x/core/dist/knowledge/inline_tasks.js';
import { getBillingInfo } from '@x/core/dist/billing/billing.js';
@ -73,7 +81,7 @@ import { summarizeMeeting } from '@x/core/dist/knowledge/summarize_meeting.js';
import { getAccessToken } from '@x/core/dist/auth/tokens.js';
import { getRowboatConfig } from '@x/core/dist/config/rowboat.js';
import { runLiveNoteAgent } from '@x/core/dist/knowledge/live-note/runner.js';
import { listImportantThreads, listEverythingElseThreads, saveMessageBodyHeight, triggerSync as triggerGmailSync, sendThreadReply, archiveThread, trashThread, markThreadRead, getAccountEmail, getAccountName, getConnectionStatus as getGmailConnectionStatus } from '@x/core/dist/knowledge/sync_gmail.js';
import { listImportantThreads, listEverythingElseThreads, saveMessageBodyHeight, triggerSync as triggerGmailSync, sendThreadReply, saveThreadDraft, deleteThreadDraft, listDraftThreads, searchThreads, archiveThread, trashThread, markThreadRead, downloadAttachment, getAccountEmail, getAccountName, getConnectionStatus as getGmailConnectionStatus } from '@x/core/dist/knowledge/sync_gmail.js';
import { searchContacts as searchGmailContacts, warmContactIndex } from '@x/core/dist/knowledge/gmail_contacts.js';
import { searchSentContacts, warmSentContacts } from '@x/core/dist/knowledge/gmail_sent_contacts.js';
import { getGoogleDocsConnectionStatus, importGoogleDoc, syncGoogleDocDown, syncGoogleDocUp, getGoogleDocLink } from '@x/core/dist/knowledge/google_docs.js';
@ -380,9 +388,36 @@ type InvokeHandlers = {
[K in InvokeChannels]: InvokeHandler<K>;
};
// In-flight streaming TTS requests, keyed by renderer-chosen requestId.
const activeTtsStreams = new Map<string, AbortController>();
// Video-mode popout window (shown for the whole duration of a screen share,
// floating over every app including Rowboat itself) and the last call state
// pushed by the main window — replayed to the popout when it finishes loading.
let videoPopoutWin: BrowserWindow | null = null;
let lastVideoPopoutState: {
ttsState: 'idle' | 'synthesizing' | 'speaking';
status: 'listening' | 'thinking' | 'speaking' | null;
cameraOn: boolean;
screenSharing: boolean;
interimText: string | null;
} | null = null;
// Match only real app windows — getAllWindows() can also contain the popout
// itself and hidden utility windows (e.g. PDF-export renderers), which must
// not be shown, focused, or sent app events.
function findMainAppWindow(): BrowserWindow | undefined {
return BrowserWindow.getAllWindows().find((w) => {
if (w === videoPopoutWin || w.isDestroyed()) return false;
const url = w.webContents.getURL();
const isAppWindow = url.startsWith('app://') || url.startsWith('http://localhost');
return isAppWindow && !url.includes('#video-popout');
});
}
/**
* Register all IPC handlers with type safety and runtime validation
*
*
* This function ensures:
* 1. All invoke channels have handlers (exhaustiveness checking)
* 2. Handler signatures match channel definitions
@ -511,7 +546,25 @@ function queueChange(relPath: string): void {
/**
* Handle workspace change event from core watcher
*/
function touchesKnowledge(event: z.infer<typeof workspaceShared.WorkspaceChangeEvent>): boolean {
const hit = (p: string | undefined) => typeof p === 'string' && p.startsWith('knowledge/');
switch (event.type) {
case 'created':
case 'changed':
case 'deleted':
return hit(event.path);
case 'moved':
return hit(event.from) || hit(event.to);
case 'bulkChanged':
return !event.paths || event.paths.some(hit);
default:
return false;
}
}
function handleWorkspaceChange(event: z.infer<typeof workspaceShared.WorkspaceChangeEvent>): void {
// Any knowledge-base change drops the cached index so the next read rebuilds.
if (touchesKnowledge(event)) invalidateKnowledgeIndex();
// Debounce 'changed' events, emit others immediately
if (event.type === 'changed' && event.path) {
queueChange(event.path);
@ -617,6 +670,39 @@ export async function startRunsWatcher(): Promise<void> {
});
}
// New runtime: session bus → renderer windows (session-design.md §10).
function emitSessionEvent(event: SessionBusEvent): void {
const windows = BrowserWindow.getAllWindows();
for (const win of windows) {
if (!win.isDestroyed() && win.webContents) {
win.webContents.send('sessions:events', event);
}
}
}
// Mobile channels: status changes (QR pairing, connect/disconnect) → renderer.
let channelsWatcher: (() => void) | null = null;
export function startChannelsWatcher(): void {
if (channelsWatcher) return;
channelsWatcher = subscribeChannelsStatus((status) => {
const windows = BrowserWindow.getAllWindows();
for (const win of windows) {
if (!win.isDestroyed() && win.webContents) {
win.webContents.send('channels:status', status);
}
}
});
}
let sessionsWatcher: (() => void) | null = null;
export function startSessionsWatcher(): void {
if (sessionsWatcher) {
return;
}
const sessionBus = container.resolve<EmitterSessionBus>('sessionBus');
sessionsWatcher = sessionBus.subscribe((event) => emitSessionEvent(event));
}
let servicesWatcher: (() => void) | null = null;
export async function startServicesWatcher(): Promise<void> {
if (servicesWatcher) {
@ -744,6 +830,18 @@ export function setupIpcHandlers() {
'gmail:sendReply': async (_event, args) => {
return sendThreadReply(args);
},
'gmail:saveDraft': async (_event, args) => {
return saveThreadDraft(args);
},
'gmail:deleteDraft': async (_event, args) => {
return deleteThreadDraft(args.draftId);
},
'gmail:getDrafts': async () => {
return listDraftThreads();
},
'gmail:search': async (_event, args) => {
return searchThreads(args.query, { limit: args.limit });
},
'gmail:getConnectionStatus': async () => {
return getGmailConnectionStatus();
},
@ -760,7 +858,10 @@ export function setupIpcHandlers() {
return trashThread(args.threadId);
},
'gmail:markThreadRead': async (_event, args) => {
return markThreadRead(args.threadId);
return markThreadRead(args.threadId, args.read);
},
'gmail:downloadAttachment': async (_event, args) => {
return downloadAttachment(args);
},
'gmail:saveMessageHeight': async (_event, args) => {
saveMessageBodyHeight(args.threadId, args.messageId, args.height);
@ -824,6 +925,82 @@ export function setupIpcHandlers() {
await runsCore.deleteRun(args.runId);
return { success: true };
},
// ── New runtime: sessions + turns ─────────────────────────
// Thin pass-throughs to the sessions service. sendMessage returns the
// turnId immediately; the turn advances in the background and the
// renderer reconciles via the sessions:events feed. Input-routing calls
// settle with that advance's outcome (the renderer fire-and-forgets).
'sessions:create': async (_event, args) => {
const sessionId = await container.resolve<ISessions>('sessions').createSession(args);
return { sessionId };
},
'sessions:list': async () => {
return { sessions: container.resolve<ISessions>('sessions').listSessions() };
},
'sessions:get': async (_event, args) => {
return container.resolve<ISessions>('sessions').getSession(args.sessionId);
},
'sessions:getTurn': async (_event, args) => {
return container.resolve<ISessions>('sessions').getTurn(args.turnId);
},
'sessions:sendMessage': async (_event, args) => {
return container.resolve<ISessions>('sessions').sendMessage(args.sessionId, args.input, args.config);
},
'sessions:respondToPermission': async (_event, args) => {
await container.resolve<ISessions>('sessions').respondToPermission(args.turnId, args.toolCallId, args.decision, args.metadata);
return { success: true };
},
'sessions:respondToAskHuman': async (_event, args) => {
await container.resolve<ISessions>('sessions').respondToAskHuman(args.turnId, args.toolCallId, args.answer);
return { success: true };
},
'sessions:stopTurn': async (_event, args) => {
await container.resolve<ISessions>('sessions').stopTurn(args.turnId, args.reason);
return { success: true };
},
'sessions:resumeTurn': async (_event, args) => {
await container.resolve<ISessions>('sessions').resumeTurn(args.sessionId);
return { success: true };
},
'sessions:setTitle': async (_event, args) => {
await container.resolve<ISessions>('sessions').setTitle(args.sessionId, args.title);
return { success: true };
},
'sessions:delete': async (_event, args) => {
await container.resolve<ISessions>('sessions').deleteSession(args.sessionId);
return { success: true };
},
'sessions:downloadLog': async (event, args) => {
// Concatenate the session's turn logs into one JSONL for debugging.
const sessions = container.resolve<ISessions>('sessions');
const state = await sessions.getSession(args.sessionId);
const win = BrowserWindow.fromWebContents(event.sender);
const result = await dialog.showSaveDialog(win!, {
defaultPath: `${args.sessionId}.jsonl.log`,
filters: [
{ name: 'Chat Log', extensions: ['log'] },
{ name: 'JSONL', extensions: ['jsonl'] },
{ name: 'All Files', extensions: ['*'] },
],
});
if (result.canceled || !result.filePath) {
return { success: false };
}
try {
const lines: string[] = [];
for (const ref of state.turns) {
const turn = await sessions.getTurn(ref.turnId);
for (const turnEvent of turn.events) {
lines.push(JSON.stringify(turnEvent));
}
}
await fs.writeFile(result.filePath, lines.join('\n') + '\n');
return { success: true };
} catch (err) {
const message = err instanceof Error ? err.message : 'Failed to download chat log';
return { success: false, error: message };
}
},
'runs:downloadLog': async (event, args) => {
const runFileName = `${args.runId}.jsonl`;
if (path.basename(runFileName) !== runFileName) {
@ -1079,6 +1256,22 @@ export function setupIpcHandlers() {
return { success: true };
},
// ── Mobile channels (WhatsApp / Telegram bridge) ─────────────
'channels:getConfig': async () => {
return container.resolve<IChannelsConfigRepo>('channelsConfigRepo').getConfig();
},
'channels:setConfig': async (_event, args) => {
await container.resolve<IChannelsConfigRepo>('channelsConfigRepo').setConfig(args);
await applyChannelsConfig(args);
return { success: true };
},
'channels:getStatus': async () => {
return getChannelsStatus();
},
'channels:whatsappLogout': async () => {
await logoutWhatsApp();
return { success: true };
},
'slack:getConfig': async () => {
const repo = container.resolve<ISlackConfigRepo>('slackConfigRepo');
const config = await repo.getConfig();
@ -1595,6 +1788,11 @@ export function setupIpcHandlers() {
const notes = await summarizeMeeting(args.transcript, args.meetingStartTime, args.calendarEventJson);
return { notes };
},
'meeting-prep:resolve': async (_event, args) => {
const result = await resolveMeetingPrep(args.attendees);
const prepNote = args.eventId ? await readPrepNoteForEvent(args.eventId) : null;
return { ...result, prepNote };
},
'inline-task:classifySchedule': async (_event, args) => {
const schedule = await classifySchedule(args.instruction);
return { schedule };
@ -1608,6 +1806,51 @@ export function setupIpcHandlers() {
'voice:synthesize': async (_event, args) => {
return voice.synthesizeSpeech(args.text);
},
'voice:synthesizeStreamStart': async (event, args) => {
const { requestId, text } = args;
const sender = event.sender;
const controller = new AbortController();
activeTtsStreams.set(requestId, controller);
// Fire-and-forget: chunks are pushed to the renderer as they arrive so
// playback can begin immediately; the invoke returns once started.
void voice
.synthesizeSpeechStream(
text,
(chunk) => {
if (!sender.isDestroyed()) {
sender.send('voice:tts-chunk', {
requestId,
chunkBase64: chunk.toString('base64'),
done: false,
});
}
},
controller.signal,
)
.then(() => {
if (!sender.isDestroyed()) {
sender.send('voice:tts-chunk', { requestId, done: true });
}
})
.catch((err: unknown) => {
if (!sender.isDestroyed() && !controller.signal.aborted) {
sender.send('voice:tts-chunk', {
requestId,
done: true,
error: err instanceof Error ? err.message : String(err),
});
}
})
.finally(() => {
activeTtsStreams.delete(requestId);
});
return { ok: true };
},
'voice:synthesizeStreamCancel': async (_event, args) => {
activeTtsStreams.get(args.requestId)?.abort();
activeTtsStreams.delete(args.requestId);
return {};
},
'voice:ensureMicAccess': async () => {
if (process.platform !== 'darwin') return { granted: true };
const status = systemPreferences.getMediaAccessStatus('microphone');
@ -1626,6 +1869,113 @@ export function setupIpcHandlers() {
return { granted: false };
}
},
'voice:ensureCameraAccess': async () => {
if (process.platform !== 'darwin') return { granted: true };
const status = systemPreferences.getMediaAccessStatus('camera');
console.log('[video] Camera permission status:', status);
if (status === 'granted') return { granted: true };
// Same flow as the microphone: settle the native TCC prompt before the
// renderer's getUserMedia so the first video click doesn't silently fail.
try {
const granted = await systemPreferences.askForMediaAccess('camera');
console.log('[video] Camera permission after prompt:', granted);
return { granted };
} catch {
return { granted: false };
}
},
'video:setPopout': async (_event, args) => {
if (!args.show) {
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) videoPopoutWin.destroy();
videoPopoutWin = null;
return {};
}
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) return {};
const workArea = screen.getPrimaryDisplay().workArea;
const width = 340;
const height = 184;
const ipcDir = path.dirname(fileURLToPath(import.meta.url));
const preloadPath = app.isPackaged
? path.join(ipcDir, '../preload/dist/preload.js')
: path.join(ipcDir, '../../../preload/dist/preload.js');
const win = new BrowserWindow({
width,
height,
x: workArea.x + workArea.width - width - 24,
y: workArea.y + 24,
frame: false,
resizable: false,
alwaysOnTop: true,
skipTaskbar: true,
show: false,
hasShadow: true,
backgroundColor: '#171717',
webPreferences: {
nodeIntegration: false,
contextIsolation: true,
sandbox: true,
preload: preloadPath,
},
});
// Float above other apps on every workspace. Deliberately NOT
// `visibleOnFullScreen: true`: on macOS that flag hides the app's Dock
// icon for as long as such a window exists (the app becomes an
// "agent" app), which reads as Rowboat having vanished. The trade-off
// is the popout won't hover over other apps' fullscreen Spaces.
win.setAlwaysOnTop(true, 'floating');
win.setVisibleOnAllWorkspaces(true);
win.webContents.once('did-finish-load', () => {
if (lastVideoPopoutState) {
win.webContents.send('video:popout-state', lastVideoPopoutState);
}
// showInactive: appearing must not steal focus from the app the user
// switched to — that would immediately re-hide the popout.
if (!win.isDestroyed()) win.showInactive();
});
win.on('closed', () => {
if (videoPopoutWin === win) videoPopoutWin = null;
});
videoPopoutWin = win;
if (app.isPackaged) {
win.loadURL('app://-/index.html#video-popout');
} else {
win.loadURL('http://localhost:5173/#video-popout');
}
return {};
},
'video:popoutState': async (_event, args) => {
lastVideoPopoutState = args;
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) {
videoPopoutWin.webContents.send('video:popout-state', args);
}
return {};
},
'app:focusMainWindow': async () => {
const main = findMainAppWindow();
if (main) {
if (main.isMinimized()) main.restore();
main.show();
main.focus();
}
return {};
},
'video:getPopoutState': async () => {
return { state: lastVideoPopoutState };
},
'video:popoutAction': async (_event, args) => {
// Relay a popout control-bar action to the app window, which owns the
// call (mic, camera, screen capture) and executes it there. 'expand'
// additionally brings the app window back to the foreground.
const main = findMainAppWindow();
if (args.action === 'expand' && main) {
if (main.isMinimized()) main.restore();
main.show();
main.focus();
}
main?.webContents.send('video:popout-action', args);
return {};
},
// Live-note handlers
'live-note:run': async (_event, args) => {
const result = await runLiveNoteAgent(args.filePath, 'manual', args.context);

View file

@ -2,7 +2,8 @@ import { app, BrowserWindow, desktopCapturer, protocol, net, shell, session, typ
import path from "node:path";
import {
setupIpcHandlers,
startRunsWatcher,
startRunsWatcher, startSessionsWatcher,
startChannelsWatcher,
startCodeSessionStatusWatcher,
startServicesWatcher,
startLiveNoteAgentWatcher,
@ -25,8 +26,10 @@ import { init as initEmailLabeling } from "@x/core/dist/knowledge/label_emails.j
import { init as initNoteTagging } from "@x/core/dist/knowledge/tag_notes.js";
import { init as initInlineTasks } from "@x/core/dist/knowledge/inline_tasks.js";
import { init as initAgentRunner } from "@x/core/dist/agent-schedule/runner.js";
import { init as initChannels } from "@x/core/dist/channels/service.js";
import { init as initAgentNotes } from "@x/core/dist/knowledge/agent_notes.js";
import { init as initCalendarNotifications } from "@x/core/dist/knowledge/notify_calendar_meetings.js";
import { init as initMeetingPrep } from "@x/core/dist/knowledge/meeting_prep_scheduler.js";
import { init as initLiveNoteScheduler } from "@x/core/dist/knowledge/live-note/scheduler.js";
import { init as initEventProcessor, registerConsumer } from "@x/core/dist/events/init.js";
import { liveNoteEventConsumer } from "@x/core/dist/knowledge/live-note/event-consumer.js";
@ -36,6 +39,7 @@ import { init as initAppsServer, shutdown as shutdownAppsServer } from "@x/core/
import { registerAppsHostApi } from "@x/core/dist/apps/host-api.js";
import { shutdown as shutdownAnalytics } from "@x/core/dist/analytics/posthog.js";
import { identifyIfSignedIn } from "@x/core/dist/analytics/identify.js";
import { migrateRuns } from "@x/core/dist/migrations/runs/migrate.js";
import { initConfigs } from "@x/core/dist/config/initConfigs.js";
import { getAgentSlackCliStatus } from "@x/core/dist/slack/agent-slack-exec.js";
@ -45,6 +49,7 @@ import { execFileSync } from "node:child_process";
import { init as initChromeSync } from "@x/core/dist/knowledge/chrome-extension/server/server.js";
import container, { registerBrowserControlService, registerNotificationService } from "@x/core/dist/di/container.js";
import type { CodeModeManager } from "@x/core/dist/code-mode/acp/manager.js";
import type { ISessions } from "@x/core/dist/sessions/index.js";
import { browserViewManager, BROWSER_PARTITION } from "./browser/view.js";
import { setupBrowserEventForwarding } from "./browser/ipc.js";
import { ElectronBrowserControlService } from "./browser/control-service.js";
@ -388,6 +393,37 @@ app.whenReady().then(async () => {
// start runs watcher
startRunsWatcher();
// One-time: port legacy runs/*.jsonl into the new turn/session runtime.
// Must run BEFORE the session index is built so migrated sessions are picked
// up by the startup scan. Fully defensive — never blocks boot.
try {
const migration = migrateRuns();
if (migration.scanned > 0) {
console.log(
`[runs-migration] migrated ${migration.migratedTurns} turn(s) across ` +
`${migration.migratedSessions} session(s) from ${migration.scanned} run(s) ` +
`(${migration.skipped} skipped, ${migration.failed.length} failed)`,
);
for (const failure of migration.failed) {
console.warn(`[runs-migration] left in place (failed): ${failure.file}${failure.error}`);
}
}
} catch (error) {
console.error('[runs-migration] pass failed:', error);
}
// New runtime: build the in-memory session index (startup scan) before the
// renderer can list sessions, then forward the session bus to windows.
await container.resolve<ISessions>('sessions').initialize();
startSessionsWatcher();
// Mobile channels (WhatsApp/Telegram bridge): needs the session index, so
// start after initialize(). Failures must never block boot.
startChannelsWatcher();
initChannels().catch((error) => {
console.error('[Channels] Failed to start mobile channels:', error);
});
// start code-session status tracker (derives working/needs-you/idle + notifications)
startCodeSessionStatusWatcher();
@ -451,6 +487,9 @@ app.whenReady().then(async () => {
// start calendar meeting notification service (fires 1-minute warnings)
initCalendarNotifications();
// start meeting prep scheduler (generates prep notes ~6h before a meeting)
void initMeetingPrep();
// start chrome extension sync server
initChromeSync();

View file

@ -6,7 +6,9 @@
"dev": "vite",
"build": "tsc -b && vite build",
"lint": "eslint .",
"preview": "vite preview"
"preview": "vite preview",
"test": "vitest run",
"test:watch": "vitest"
},
"dependencies": {
"@codemirror/language": "^6.12.3",
@ -83,6 +85,9 @@
},
"devDependencies": {
"@eslint/js": "^9.39.1",
"@testing-library/dom": "^10.4.1",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.2",
"@types/node": "^24.10.1",
"@types/react": "^19.2.5",
"@types/react-dom": "^19.2.3",
@ -91,9 +96,11 @@
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.4.24",
"globals": "^16.5.0",
"jsdom": "^29.1.1",
"tw-animate-css": "^1.4.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.4"
"vite": "^7.2.4",
"vitest": "catalog:"
}
}

View file

@ -0,0 +1,62 @@
/**
* Regenerates the bundled product-tour narration clips.
*
* Parses the TOUR_STEPS texts straight out of product-tour.tsx (so the code
* stays the single source of truth), synthesizes each one via @x/core's
* synthesizeSpeech (Rowboat proxy when signed in, direct ElevenLabs otherwise,
* using the voice id configured there / in ~/.rowboat/config/elevenlabs.json),
* and writes MP3s to src/assets/tour/<step-id>.mp3.
*
* Run whenever a step's narration text or the tour voice changes:
* cd apps/x && npm run deps # script imports core's built output
* node apps/renderer/scripts/generate-tour-audio.mjs
*
* Pass step ids to regenerate only those clips (e.g. to re-roll one whose
* synthesis came out glitchy):
* node apps/renderer/scripts/generate-tour-audio.mjs welcome done
*/
import { mkdir, readFile, writeFile } from 'node:fs/promises'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
const here = path.dirname(fileURLToPath(import.meta.url))
const tourSource = path.join(here, '../src/components/product-tour.tsx')
const outDir = path.join(here, '../src/assets/tour')
const corePath = path.join(here, '../../../packages/core/dist/voice/voice.js')
const { synthesizeSpeech } = await import(corePath)
const src = await readFile(tourSource, 'utf8')
const start = src.indexOf('const TOUR_STEPS')
const end = src.indexOf('\n]', start)
if (start === -1 || end === -1) throw new Error('Could not locate TOUR_STEPS in product-tour.tsx')
const block = src.slice(start, end)
const steps = []
// voiceText, when present, is the spoken variant of the bubble text.
const re = /id: '([^']+)'[\s\S]*?text:\s*("[^"]*"|'[^']*')(?:,\s*voiceText:\s*("[^"]*"|'[^']*'))?/g
for (let m; (m = re.exec(block)); ) {
// The captures are JS string literals from our own source; evaluate them
// to resolve the quoting.
steps.push({ id: m[1], text: new Function(`return ${m[3] ?? m[2]}`)() })
}
if (steps.length === 0) throw new Error('Parsed zero tour steps — regex out of sync with product-tour.tsx?')
console.log(`Parsed ${steps.length} tour steps`)
const only = process.argv.slice(2)
if (only.length > 0) {
const unknown = only.filter((id) => !steps.some((s) => s.id === id))
if (unknown.length > 0) throw new Error(`Unknown step ids: ${unknown.join(', ')}`)
steps.splice(0, steps.length, ...steps.filter((s) => only.includes(s.id)))
console.log(`Regenerating only: ${only.join(', ')}`)
}
await mkdir(outDir, { recursive: true })
for (const step of steps) {
process.stdout.write(`synthesizing ${step.id}... `)
const { audioBase64 } = await synthesizeSpeech(step.text)
const file = path.join(outDir, `${step.id}.mp3`)
await writeFile(file, Buffer.from(audioBase64, 'base64'))
console.log(`${(audioBase64.length * 0.75 / 1024).toFixed(0)} KB`)
}
console.log(`Done — ${steps.length} clips in ${path.relative(process.cwd(), outDir)}`)

View file

@ -169,6 +169,25 @@
color: var(--gm-placeholder);
}
.gmail-search-clear {
display: inline-flex;
align-items: center;
justify-content: center;
width: 20px;
height: 20px;
border: none;
border-radius: 4px;
background: transparent;
color: var(--gm-text-muted);
cursor: pointer;
transition: background 120ms ease, color 120ms ease;
}
.gmail-search-clear:hover {
background: var(--gm-icon-hover-bg);
color: var(--gm-text);
}
.gmail-icon-button {
display: inline-flex;
align-items: center;
@ -205,6 +224,46 @@
flex-direction: column;
}
/* Native list virtualization: offscreen rows skip layout and paint entirely.
Applied only to rows without a mounted ThreadDetail (those hold iframes and
composers, which must keep rendering while offscreen). */
.gmail-row-group-cv {
content-visibility: auto;
contain-intrinsic-size: auto 40px;
}
/* While the list is scrolling, rows ignore the pointer so hover restyles and
prefetch timers don't compete with frame rendering. */
.gmail-shell[data-scrolling] .gmail-row-shell {
pointer-events: none;
}
/* Archived/trashed rows slide out and collapse before removal. Removing the
class (failed action) snaps the row back. */
.gmail-row-group-leaving {
overflow: hidden;
pointer-events: none;
animation: gmail-row-leave 160ms ease-in forwards;
}
@keyframes gmail-row-leave {
0% {
opacity: 1;
transform: translateX(0);
max-height: 48px;
}
60% {
opacity: 0;
transform: translateX(32px);
max-height: 48px;
}
100% {
opacity: 0;
transform: translateX(32px);
max-height: 0;
}
}
.gmail-list-header {
position: sticky;
top: 0;
@ -319,6 +378,20 @@
background: var(--gm-bg-row-selected-hover);
}
/* The j/k keyboard cursor. Declared after the hover rules (same specificity)
so the focus ring survives hovering the focused row. */
.gmail-row-focused,
.gmail-row-focused:hover {
background: var(--gm-bg-row-hover);
box-shadow: inset 0 0 0 1px var(--gm-accent);
}
.gmail-row-selected.gmail-row-focused,
.gmail-row-selected.gmail-row-focused:hover {
background: var(--gm-bg-row-selected);
box-shadow: inset 2px 0 0 var(--gm-accent), inset 0 0 0 1px var(--gm-accent);
}
.gmail-row-unread {
color: var(--gm-text);
}
@ -394,12 +467,50 @@
border-top: 1px solid var(--gm-border);
border-bottom: 1px solid var(--gm-border);
box-shadow: inset 2px 0 0 var(--gm-accent);
/* Replays whenever the detail is shown hidden details are display: none,
so un-hiding restarts the animation. */
animation: gmail-detail-open 140ms ease-out;
}
.gmail-detail-hidden {
display: none;
}
@keyframes gmail-detail-open {
from {
opacity: 0;
transform: translateY(-6px);
}
to {
opacity: 1;
transform: none;
}
}
/* The inline reply/forward composer pops in from below the thread. */
.gmail-compose-inline {
animation: gmail-compose-open 140ms ease-out;
}
@keyframes gmail-compose-open {
from {
opacity: 0;
transform: translateY(8px);
}
to {
opacity: 1;
transform: none;
}
}
@media (prefers-reduced-motion: reduce) {
.gmail-detail-inline,
.gmail-compose-inline,
.gmail-row-group-leaving {
animation: none;
}
}
.gmail-detail-toolbar {
display: flex;
align-items: center;
@ -1017,6 +1128,17 @@
var(--rowboat-shadow);
}
.dark .rowboat-code-chat-input {
border-color: color-mix(in oklab, var(--border) 76%, transparent);
background: #000000;
box-shadow: none;
}
.dark .rowboat-code-chat-input:focus-within {
border-color: color-mix(in oklab, var(--ring) 70%, var(--border) 30%);
box-shadow: 0 0 0 1px color-mix(in oklab, var(--ring) 30%, transparent);
}
[data-slot="message-content"] {
line-height: 1.62;
}

File diff suppressed because it is too large Load diff

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View file

@ -43,6 +43,7 @@ export const Conversation = ({
const contentRef = useRef<HTMLDivElement | null>(null);
const scrollRef = useRef<HTMLDivElement | null>(null);
const spacerRef = useRef<HTMLDivElement | null>(null);
const stickToBottomRef = useRef(true);
const [isAtBottom, setIsAtBottom] = useState(true);
const updateBottomState = useCallback(() => {
@ -50,7 +51,9 @@ export const Conversation = ({
if (!container) return;
const distanceFromBottom =
container.scrollHeight - container.scrollTop - container.clientHeight;
setIsAtBottom(distanceFromBottom <= BOTTOM_THRESHOLD_PX);
const atBottom = distanceFromBottom <= BOTTOM_THRESHOLD_PX;
stickToBottomRef.current = atBottom;
setIsAtBottom(atBottom);
}, []);
const applyAnchorLayout = useCallback(
@ -131,7 +134,12 @@ export const Conversation = ({
cancelAnimationFrame(rafId);
}
rafId = requestAnimationFrame(() => {
const shouldStick = !anchorMessageId && stickToBottomRef.current;
applyAnchorLayout(false);
if (shouldStick) {
container.scrollTop = container.scrollHeight;
updateBottomState();
}
});
};
@ -178,6 +186,7 @@ export const Conversation = ({
const container = scrollRef.current;
if (!container) return;
container.scrollTop = container.scrollHeight;
stickToBottomRef.current = true;
updateBottomState();
}, [updateBottomState]);

View file

@ -6,9 +6,7 @@ import {
Pencil, Check, PanelRightClose, PanelRightOpen, Sparkles,
Code2, FolderOpen, LayoutTemplate,
} from 'lucide-react'
import type { z } from 'zod'
import type { BackgroundTask, BackgroundTaskSummary, Triggers } from '@x/shared/dist/background-task.js'
import type { Run } from '@x/shared/dist/runs.js'
import { Button } from '@/components/ui/button'
import { Switch } from '@/components/ui/switch'
import { Input } from '@/components/ui/input'
@ -17,7 +15,7 @@ import { useBackgroundTaskAgentStatus } from '@/hooks/use-bg-task-agent-status'
import { formatRelativeTime } from '@/lib/relative-time'
import { toast } from '@/lib/toast'
import type { ConversationItem } from '@/lib/chat-conversation'
import { runLogToConversation } from '@/lib/run-to-conversation'
import { fetchAgentRunTranscript, type AgentRunTranscript } from '@/lib/agent-transcript'
import { CompactConversation } from '@/components/compact-conversation'
import { RichMarkdownViewer } from '@/components/rich-markdown-viewer'
import { HtmlFileViewer } from '@/components/html-file-viewer'
@ -970,10 +968,9 @@ function SetupTab({
// Runs history tab — list + drill-down transcript view
//
// Source of truth: `bg-tasks/<slug>/runs.log` — a plain-text file with one
// runId per line (newest first). The actual transcripts live at the global
// `$WorkDir/runs/<runId>.jsonl`, so this tab fetches runIds via the bg-task
// IPC, then loads each Run through the standard `runs:fetch`. No bg-task-
// specific transcript path or schema needed.
// turn id per line (newest first). Transcripts live in the turn runtime's
// storage; this tab fetches ids via the bg-task IPC, then loads each through
// the shared agent-transcript loader (turn-first, legacy-run fallback).
// ---------------------------------------------------------------------------
interface RunRowSummary {
@ -984,27 +981,6 @@ interface RunRowSummary {
error?: string
}
// Pull the bits we want to display for a row out of a full Run's event log.
function summarizeRun(run: z.infer<typeof Run>): RunRowSummary {
const out: RunRowSummary = { runId: run.id, createdAt: run.createdAt, trigger: run.subUseCase }
for (const event of run.log) {
if (event.type === 'error' && typeof event.error === 'string') {
out.error = event.error
} else if (event.type === 'message' && event.message?.role === 'assistant') {
const content = event.message.content
if (typeof content === 'string') {
out.summary = content
} else if (Array.isArray(content)) {
const text = content
.filter((p) => p.type === 'text')
.map((p) => ('text' in p ? p.text : ''))
.join('')
if (text) out.summary = text
}
}
}
return out
}
function RunsHistoryTab({ slug, task }: { slug: string; task: BackgroundTask }) {
const [rows, setRows] = useState<RunRowSummary[]>([])
@ -1016,19 +992,25 @@ function RunsHistoryTab({ slug, task }: { slug: string; task: BackgroundTask })
setLoading(true)
try {
const { runIds } = await window.ipc.invoke('bg-task:listRunIds', { slug, limit: 100 })
// Fetch each Run in parallel via the canonical IPC. Runs whose
// jsonl no longer exists (deleted manually, never written, …) are
// dropped silently.
// Fetch transcripts in parallel (turn-first, legacy-run
// fallback). Ids whose files no longer exist keep a bare row so
// the user knows the run happened.
const settled = await Promise.allSettled(
runIds.map(runId => window.ipc.invoke('runs:fetch', { runId }))
runIds.map(runId => fetchAgentRunTranscript(runId))
)
const next: RunRowSummary[] = []
for (let i = 0; i < settled.length; i++) {
const r = settled[i]
if (r.status === 'fulfilled' && r.value) {
next.push(summarizeRun(r.value))
if (r.status === 'fulfilled') {
const t = r.value
next.push({
runId: t.id,
...(t.createdAt === undefined ? {} : { createdAt: t.createdAt }),
...(t.trigger === undefined ? {} : { trigger: t.trigger }),
...(t.summary === undefined ? {} : { summary: t.summary }),
...(t.error === undefined ? {} : { error: t.error }),
})
} else {
// Keep the row visible with just the id so the user knows it exists.
next.push({ runId: runIds[i] })
}
}
@ -1134,7 +1116,7 @@ function RunTranscriptView({
isInFlight: boolean
onBack: () => void
}) {
const [run, setRun] = useState<z.infer<typeof Run> | null>(null)
const [transcript, setTranscript] = useState<AgentRunTranscript | null>(null)
const [loading, setLoading] = useState(true)
const [error, setError] = useState<string | null>(null)
@ -1144,15 +1126,13 @@ function RunTranscriptView({
setError(null)
void (async () => {
try {
// Bg-task transcripts now live at the global runs/ location —
// same path resolution as every other run, no special handling.
const r = await window.ipc.invoke('runs:fetch', { runId })
const t = await fetchAgentRunTranscript(runId)
if (cancelled) return
setRun(r)
setTranscript(t)
} catch (err) {
if (cancelled) return
setError(err instanceof Error ? err.message : String(err))
setRun(null)
setTranscript(null)
} finally {
if (!cancelled) setLoading(false)
}
@ -1160,8 +1140,8 @@ function RunTranscriptView({
return () => { cancelled = true }
}, [runId])
const summary = run ? summarizeRun(run) : undefined
const items: ConversationItem[] = run ? runLogToConversation(run.log) : []
const summary = transcript ?? undefined
const items: ConversationItem[] = transcript?.items ?? []
return (
<div className="flex flex-1 flex-col overflow-hidden">
@ -1221,10 +1201,10 @@ function RunTranscriptView({
Couldn&apos;t load transcript: {error}
</div>
)}
{run && !loading && items.length === 0 && (
{transcript && !loading && items.length === 0 && (
<p className="text-xs italic text-muted-foreground">No messages or tool calls recorded.</p>
)}
{run && !loading && items.length > 0 && (
{transcript && !loading && items.length > 0 && (
<CompactConversation items={items} />
)}
</div>

View file

@ -1,7 +1,19 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { ArrowLeft, ArrowRight, Loader2, Plus, RotateCw, X } from 'lucide-react'
import type { HttpAuthRequest } from '@x/shared/dist/browser-control.js'
import { TabBar } from '@/components/tab-bar'
import { Button } from '@/components/ui/button'
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog'
import { Input } from '@/components/ui/input'
import { cn } from '@/lib/utils'
/**
@ -86,9 +98,80 @@ const getBrowserTabTitle = (tab: BrowserTabState) => {
}
}
/**
* Credential prompt for HTTP basic/proxy auth challenges raised by pages in
* the embedded browser. Rendered as a regular app dialog: BrowserPane already
* hides the native WebContentsView whenever a dialog overlay is open, so the
* prompt is never obscured by the page that triggered it.
*/
function BrowserHttpAuthDialog({
request,
onSubmit,
onCancel,
}: {
request: HttpAuthRequest
onSubmit: (username: string, password: string) => void
onCancel: () => void
}) {
const [username, setUsername] = useState('')
const [password, setPassword] = useState('')
// Basic auth allows an empty username (token-style `curl -u :TOKEN`), so the
// only invalid submission is fully empty. The server decides the rest.
const canSubmit = username.length > 0 || password.length > 0
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault()
if (!canSubmit) return
onSubmit(username, password)
}
return (
<Dialog open onOpenChange={(open) => { if (!open) onCancel() }}>
<DialogContent className="w-[min(24rem,calc(100%-2rem))] max-w-sm">
<DialogHeader>
<DialogTitle>Sign in</DialogTitle>
<DialogDescription>
{request.isProxy
? `The proxy ${request.host} requires a username and password.`
: `${request.host} requires a username and password.`}
{request.realm ? ` (${request.realm})` : ''}
</DialogDescription>
</DialogHeader>
<form onSubmit={handleSubmit} className="flex flex-col gap-3">
<Input
autoFocus
value={username}
onChange={(e) => setUsername(e.target.value)}
placeholder="Username"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
<Input
type="password"
value={password}
onChange={(e) => setPassword(e.target.value)}
placeholder="Password"
/>
<DialogFooter>
<Button type="button" variant="outline" onClick={onCancel}>
Cancel
</Button>
<Button type="submit" disabled={!canSubmit}>
Sign in
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
)
}
export function BrowserPane({ onClose, forceHidden = false }: BrowserPaneProps) {
const [state, setState] = useState<BrowserState>(EMPTY_STATE)
const [addressValue, setAddressValue] = useState('')
const [authQueue, setAuthQueue] = useState<HttpAuthRequest[]>([])
const activeTabIdRef = useRef<string | null>(null)
const addressFocusedRef = useRef(false)
@ -121,6 +204,51 @@ export function BrowserPane({ onClose, forceHidden = false }: BrowserPaneProps)
return cleanup
}, [applyState])
// Mirror of authQueue for the unmount handler, which must read the latest
// queue without re-subscribing on every change.
const authQueueRef = useRef<HttpAuthRequest[]>([])
useEffect(() => {
authQueueRef.current = authQueue
}, [authQueue])
useEffect(() => {
const offRequest = window.ipc.on('browser:httpAuthRequest', (incoming) => {
setAuthQueue((queue) => [...queue, incoming as HttpAuthRequest])
})
// Main resolved a challenge on its own (timeout, or its tab/window was
// destroyed) — drop the corresponding dialog so it can't linger over an
// unrelated page with a submit that would no-op.
const offResolved = window.ipc.on('browser:httpAuthResolved', (incoming) => {
const { requestId } = incoming as { requestId: string }
setAuthQueue((queue) => queue.filter((request) => request.requestId !== requestId))
})
return () => {
offRequest()
offResolved()
// Cancel anything still pending so the main-process login callbacks and
// timers are freed immediately instead of waiting out the timeout.
for (const request of authQueueRef.current) {
void window.ipc.invoke('browser:httpAuthResponse', { requestId: request.requestId })
}
}
}, [])
const respondToAuth = useCallback(
(requestId: string, credentials: { username: string; password: string } | null) => {
setAuthQueue((queue) => queue.filter((request) => request.requestId !== requestId))
// Omit username to cancel; include it (even empty) to submit.
void window.ipc.invoke(
'browser:httpAuthResponse',
credentials
? { requestId, username: credentials.username, password: credentials.password }
: { requestId },
)
},
[],
)
const activeAuthRequest = authQueue[0] ?? null
const setViewVisible = useCallback((visible: boolean) => {
if (viewVisibleRef.current === visible) return
viewVisibleRef.current = visible
@ -420,6 +548,17 @@ export function BrowserPane({ onClose, forceHidden = false }: BrowserPaneProps)
className="relative min-h-0 min-w-0 flex-1"
data-browser-viewport
/>
{activeAuthRequest && (
<BrowserHttpAuthDialog
key={activeAuthRequest.requestId}
request={activeAuthRequest}
onSubmit={(username, password) =>
respondToAuth(activeAuthRequest.requestId, { username, password })
}
onCancel={() => respondToAuth(activeAuthRequest.requestId, null)}
/>
)}
</div>
)
}

View file

@ -15,16 +15,19 @@ import {
FolderCog,
FolderOpen,
Globe,
Headphones,
ImagePlus,
LoaderIcon,
Lock,
Mic,
MoreHorizontal,
Phone,
PhoneOff,
Plus,
Presentation,
ShieldCheck,
Square,
Terminal,
Video,
X,
} from 'lucide-react'
@ -210,6 +213,18 @@ function compactWorkDirPath(path: string) {
return path.replace(/^\/Users\/[^/]+/, '~')
}
// Call presets: front doors into the same call engine, differing only in
// starting devices. 'share' is the call button's main click — the "work
// together" default (screen shared, camera off, floating pill). The chevron
// menu holds the deviations.
export type CallPreset = 'voice' | 'video' | 'share' | 'practice'
const CALL_PRESET_MENU: Array<{ preset: CallPreset; label: string; description: string; Icon: typeof Phone }> = [
{ preset: 'voice', label: 'Voice call', description: 'Just talk — nothing is shared, the mascot hovers while you work', Icon: AudioLines },
{ preset: 'video', label: 'Video call', description: 'Camera on, face to face — it sees your expressions', Icon: Video },
{ preset: 'practice', label: 'Practice session', description: 'Rehearse a pitch or interview with live coaching', Icon: Presentation },
]
interface ChatInputInnerProps {
onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void
onStop?: () => void
@ -223,18 +238,20 @@ interface ChatInputInnerProps {
onDraftChange?: (text: string) => void
isRecording?: boolean
recordingText?: string
recordingState?: 'connecting' | 'listening'
recordingState?: 'connecting' | 'listening' | 'stopping'
/** Live mic amplitude history (RMS per frame) driving the recording waveform. */
audioLevelsRef?: React.MutableRefObject<number[]>
onStartRecording?: () => void
onSubmitRecording?: () => void
onSubmitRecording?: () => void | Promise<void>
onCancelRecording?: () => void
voiceAvailable?: boolean
ttsAvailable?: boolean
ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void
/** A call is live (hands-free voice loop + spoken responses). */
inCall?: boolean
/** Start a call with the given preset's device defaults. */
onStartCall?: (preset: CallPreset) => void
onEndCall?: () => void
/** Calls need both voice input (STT) and voice output (TTS) configured. */
callAvailable?: boolean
/** Fired when the user picks a different model in the dropdown (only when no run exists yet). */
onSelectedModelChange?: (model: SelectedModel | null) => void
/** Work directory for this chat (per-chat). Null when none is set. */
@ -262,16 +279,16 @@ function ChatInputInner({
onDraftChange,
isRecording,
recordingText,
recordingState,
audioLevelsRef,
onStartRecording,
onSubmitRecording,
onCancelRecording,
voiceAvailable,
ttsAvailable,
ttsEnabled,
ttsMode,
onToggleTts,
onTtsModeChange,
inCall,
onStartCall,
onEndCall,
callAvailable,
onSelectedModelChange,
workDir = null,
onWorkDirChange,
@ -341,22 +358,15 @@ function ChatInputInner({
}
})
// When a run exists, freeze the dropdown to the run's resolved model+provider.
// Sessions runtime: model and permission mode are per-message turn config,
// so nothing is frozen for an existing chat — the picker stays live.
useEffect(() => {
if (!runId) {
setLockedModel(null)
setPermissionMode('auto')
return
}
let cancelled = false
window.ipc.invoke('runs:fetch', { runId }).then((run) => {
if (cancelled) return
if (run.provider && run.model) {
setLockedModel({ provider: run.provider, model: run.model })
}
setPermissionMode(run.permissionMode ?? 'manual')
}).catch(() => { /* legacy run or fetch failure — leave unlocked */ })
return () => { cancelled = true }
setLockedModel(null)
}, [runId])
useEffect(() => {
@ -758,7 +768,7 @@ function ChatInputInner({
const currentWorkDirPath = effectiveWorkDir ? compactWorkDirPath(effectiveWorkDir) : ''
return (
<div className="rowboat-chat-input rounded-lg border border-border bg-background shadow-none">
<div data-tour-id="chat-composer" className="rowboat-chat-input rounded-lg border border-border bg-background shadow-none">
{attachments.length > 0 && (
<div className="flex flex-wrap gap-2 px-4 pb-1 pt-3">
{attachments.map((attachment) => {
@ -829,23 +839,33 @@ function ChatInputInner({
>
<X className="h-4 w-4" />
</button>
{/* Audio-reactive waveform only the transcribed words are intentionally
not shown while recording; they're still captured and submitted. */}
<div className="flex flex-1 items-center overflow-hidden">
<div className="flex min-w-0 flex-1 flex-col gap-1 overflow-hidden">
<VoiceWaveform audioLevelsRef={audioLevelsRef} />
<div
className={cn(
'min-h-5 truncate text-sm leading-5',
recordingText?.trim() ? 'text-foreground' : 'text-muted-foreground'
)}
>
{recordingText?.trim() || (recordingState === 'stopping' ? 'Finalizing...' : 'Listening...')}
</div>
</div>
<Button
size="icon"
onClick={onSubmitRecording}
disabled={!recordingText?.trim()}
disabled={recordingState === 'stopping'}
className={cn(
'h-7 w-7 shrink-0 rounded-full transition-all',
recordingText?.trim()
recordingState !== 'stopping'
? 'bg-primary text-primary-foreground hover:bg-primary/90'
: 'bg-muted text-muted-foreground'
)}
>
<ArrowUp className="h-4 w-4" />
{recordingState === 'stopping' ? (
<LoaderIcon className="h-4 w-4 animate-spin" />
) : (
<ArrowUp className="h-4 w-4" />
)}
</Button>
</div>
) : (
@ -1267,48 +1287,66 @@ function ChatInputInner({
</DropdownMenuContent>
</DropdownMenu>
) : null}
{onToggleTts && ttsAvailable && (
{onStartCall && (
<div className="flex shrink-0 items-center">
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
<TooltipTrigger asChild>
<button
type="button"
onClick={onToggleTts}
onClick={() => {
if (inCall) {
onEndCall?.()
} else if (callAvailable) {
onStartCall('share')
}
}}
className={cn(
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
ttsEnabled
? 'text-foreground hover:bg-muted'
: 'text-muted-foreground hover:bg-muted hover:text-foreground'
'flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
inCall
? 'bg-red-600 text-white hover:bg-red-500'
: callAvailable
? 'text-muted-foreground hover:bg-muted hover:text-foreground'
: 'cursor-default text-muted-foreground/40'
)}
aria-label={ttsEnabled ? 'Disable voice output' : 'Enable voice output'}
aria-label={inCall ? 'End call' : 'Start a call'}
>
<Headphones className="h-4 w-4" />
{!ttsEnabled && (
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
</span>
)}
{inCall ? <PhoneOff className="h-4 w-4" /> : <Phone className="h-4 w-4" />}
</button>
</TooltipTrigger>
<TooltipContent side="top">
{ttsEnabled ? 'Voice output on' : 'Voice output off'}
{inCall
? 'End call'
: callAvailable
? 'Start a call — it sees your screen while you talk it through'
: 'Calls need voice input and output configured'}
</TooltipContent>
</Tooltip>
{ttsEnabled && onTtsModeChange && (
{!inCall && (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<button
type="button"
className="flex h-7 w-4 shrink-0 items-center justify-center text-muted-foreground transition-colors hover:text-foreground"
aria-label="Call options"
>
<ChevronDown className="h-3 w-3" />
</button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuRadioGroup value={ttsMode ?? 'summary'} onValueChange={(v) => onTtsModeChange(v as 'summary' | 'full')}>
<DropdownMenuRadioItem value="summary">Speak summary</DropdownMenuRadioItem>
<DropdownMenuRadioItem value="full">Speak full response</DropdownMenuRadioItem>
</DropdownMenuRadioGroup>
<DropdownMenuContent align="end" className="w-72">
{CALL_PRESET_MENU.map(({ preset, label, description, Icon }) => (
<DropdownMenuItem
key={preset}
disabled={!callAvailable}
onSelect={() => onStartCall(preset)}
className="items-start gap-3 py-2"
>
<Icon className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" />
<span className="min-w-0">
<span className="block text-sm font-medium leading-tight">{label}</span>
<span className="block pt-0.5 text-xs leading-tight text-muted-foreground">{description}</span>
</span>
</DropdownMenuItem>
))}
</DropdownMenuContent>
</DropdownMenu>
)}
@ -1477,17 +1515,16 @@ export interface ChatInputWithMentionsProps {
onDraftChange?: (text: string) => void
isRecording?: boolean
recordingText?: string
recordingState?: 'connecting' | 'listening'
recordingState?: 'connecting' | 'listening' | 'stopping'
audioLevelsRef?: React.MutableRefObject<number[]>
onStartRecording?: () => void
onSubmitRecording?: () => void
onSubmitRecording?: () => void | Promise<void>
onCancelRecording?: () => void
voiceAvailable?: boolean
ttsAvailable?: boolean
ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void
inCall?: boolean
onStartCall?: (preset: CallPreset) => void
onEndCall?: () => void
callAvailable?: boolean
onSelectedModelChange?: (model: SelectedModel | null) => void
workDir?: string | null
onWorkDirChange?: (value: string | null) => void
@ -1517,11 +1554,10 @@ export function ChatInputWithMentions({
onSubmitRecording,
onCancelRecording,
voiceAvailable,
ttsAvailable,
ttsEnabled,
ttsMode,
onToggleTts,
onTtsModeChange,
inCall,
onStartCall,
onEndCall,
callAvailable,
onSelectedModelChange,
workDir,
onWorkDirChange,
@ -1548,11 +1584,10 @@ export function ChatInputWithMentions({
onSubmitRecording={onSubmitRecording}
onCancelRecording={onCancelRecording}
voiceAvailable={voiceAvailable}
ttsAvailable={ttsAvailable}
ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={onToggleTts}
onTtsModeChange={onTtsModeChange}
inCall={inCall}
onStartCall={onStartCall}
onEndCall={onEndCall}
callAvailable={callAvailable}
onSelectedModelChange={onSelectedModelChange}
workDir={workDir}
onWorkDirChange={onWorkDirChange}

View file

@ -91,11 +91,28 @@ interface ChatMessageAttachmentsProps {
export function ChatMessageAttachments({ attachments, className }: ChatMessageAttachmentsProps) {
if (attachments.length === 0) return null
const imageAttachments = attachments.filter((attachment) => isImageMime(attachment.mimeType))
const fileAttachments = attachments.filter((attachment) => !isImageMime(attachment.mimeType))
const videoFrames = attachments.filter((attachment) => attachment.isVideoFrame)
const imageAttachments = attachments.filter(
(attachment) => !attachment.isVideoFrame && isImageMime(attachment.mimeType)
)
const fileAttachments = attachments.filter(
(attachment) => !attachment.isVideoFrame && !isImageMime(attachment.mimeType)
)
return (
<div className={cn('flex flex-col items-end gap-2', className)}>
{videoFrames.length > 0 && (
<div className="flex max-w-[340px] flex-wrap justify-end gap-1">
{videoFrames.map((frame, index) => (
<img
key={`frame-${index}`}
src={frame.thumbnailUrl}
alt={`Camera frame ${index + 1}`}
className="h-12 w-auto rounded-md border border-border/60 bg-muted object-cover"
/>
))}
</div>
)}
{imageAttachments.length > 0 && (
<div className="flex flex-wrap justify-end gap-2">
{imageAttachments.map((attachment, index) => (

View file

@ -37,7 +37,7 @@ import { MarkdownPreOverride } from '@/components/ai-elements/markdown-code-over
import { defaultRemarkPlugins } from 'streamdown'
import remarkBreaks from 'remark-breaks'
import { type ChatTab } from '@/components/tab-bar'
import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions'
import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions'
import { ChatMessageAttachments } from '@/components/chat-message-attachments'
import { useSidebar } from '@/components/ui/sidebar'
import { wikiLabel } from '@/lib/wiki-links'
@ -142,6 +142,10 @@ interface ChatSidebarProps {
chatTabStates?: Record<string, ChatTabViewState>
viewportAnchors?: Record<string, ChatViewportAnchorState>
isProcessing: boolean
// Actively working (sessions runtime). When provided, drives the shimmer
// instead of isProcessing so waiting on a permission/ask-human doesn't
// render a "Thinking…" under the card.
isThinking?: boolean
isStopping?: boolean
onStop?: () => void
onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void
@ -177,17 +181,16 @@ interface ChatSidebarProps {
// Voice / TTS props
isRecording?: boolean
recordingText?: string
recordingState?: 'connecting' | 'listening'
recordingState?: 'connecting' | 'listening' | 'stopping'
audioLevelsRef?: React.MutableRefObject<number[]>
onStartRecording?: () => void
onSubmitRecording?: () => void
onSubmitRecording?: () => void | Promise<void>
onCancelRecording?: () => void
voiceAvailable?: boolean
ttsAvailable?: boolean
ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void
inCall?: boolean
onStartCall?: (preset: CallPreset) => void
onEndCall?: () => void
callAvailable?: boolean
onComposioConnected?: (toolkitSlug: string) => void
}
@ -211,6 +214,7 @@ export function ChatSidebar({
chatTabStates = {},
viewportAnchors = {},
isProcessing,
isThinking,
isStopping,
onStop,
onSubmit,
@ -246,11 +250,10 @@ export function ChatSidebar({
onSubmitRecording,
onCancelRecording,
voiceAvailable,
ttsAvailable,
ttsEnabled,
ttsMode,
onToggleTts,
onTtsModeChange,
inCall,
onStartCall,
onEndCall,
callAvailable,
onComposioConnected,
}: ChatSidebarProps) {
const { state: sidebarState } = useSidebar()
@ -373,7 +376,14 @@ export function ChatSidebar({
}
try {
const result = await window.ipc.invoke('runs:downloadLog', { runId: activeRunId })
// Session-first (new runtime); legacy runs fallback covers old
// background tabs until stage 7 removes the runs runtime.
let result: { success: boolean; error?: string }
try {
result = await window.ipc.invoke('sessions:downloadLog', { sessionId: activeRunId })
} catch {
result = await window.ipc.invoke('runs:downloadLog', { runId: activeRunId })
}
if (result.success) {
toast.success('Chat log saved')
} else if (result.error) {
@ -725,7 +735,7 @@ export function ChatSidebar({
onApproveSession={() => onPermissionResponse(permRequest.toolCall.toolCallId, permRequest.subflow, 'approve', 'session')}
onApproveAlways={() => onPermissionResponse(permRequest.toolCall.toolCallId, permRequest.subflow, 'approve', 'always')}
onDeny={() => onPermissionResponse(permRequest.toolCall.toolCallId, permRequest.subflow, 'deny')}
isProcessing={isActive && isProcessing}
isProcessing={isActive && (isThinking ?? isProcessing)}
response={response}
/>
)}
@ -742,7 +752,7 @@ export function ChatSidebar({
key={request.toolCallId}
query={request.query}
onResponse={(response) => onAskHumanResponse(request.toolCallId, request.subflow, response)}
isProcessing={isActive && isProcessing}
isProcessing={isActive && (isThinking ?? isProcessing)}
/>
))}
@ -754,7 +764,7 @@ export function ChatSidebar({
</Message>
)}
{isActive && isProcessing && !tabState.currentAssistantMessage && (
{isActive && (isThinking ?? isProcessing) && !tabState.currentAssistantMessage && (
<Message from="assistant">
<MessageContent>
<Shimmer duration={1}>Thinking...</Shimmer>
@ -813,11 +823,10 @@ export function ChatSidebar({
onSubmitRecording={isActive ? onSubmitRecording : undefined}
onCancelRecording={isActive ? onCancelRecording : undefined}
voiceAvailable={isActive && voiceAvailable}
ttsAvailable={isActive && ttsAvailable}
ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={isActive ? onToggleTts : undefined}
onTtsModeChange={isActive ? onTtsModeChange : undefined}
inCall={inCall}
onStartCall={isActive ? onStartCall : undefined}
onEndCall={isActive ? onEndCall : undefined}
callAvailable={callAvailable}
/>
</div>
)

View file

@ -29,6 +29,12 @@ const darkHighlight = HighlightStyle.define([
])
export function cmBaseExtensions(isDark: boolean): Extension[] {
const bg = isDark ? '#0f1117' : '#ffffff'
const panelBg = isDark ? '#151821' : '#f6f8fa'
const text = isDark ? '#d4d4d8' : '#24292f'
const muted = isDark ? '#7d8590' : '#6e7781'
const border = isDark ? '#2f3542' : '#d0d7de'
return [
lineNumbers(),
bracketMatching(),
@ -39,18 +45,62 @@ export function cmBaseExtensions(isDark: boolean): Extension[] {
EditorView.theme(
{
'&': {
backgroundColor: 'transparent',
backgroundColor: bg,
color: text,
fontSize: '12px',
height: '100%',
},
'.cm-editor': {
backgroundColor: bg,
color: text,
},
'.cm-content': {
caretColor: text,
},
'.cm-scroller': {
fontFamily: 'ui-monospace, SFMono-Regular, Menlo, monospace',
overflow: 'auto',
},
'.cm-line': {
color: text,
},
'.cm-gutters': {
backgroundColor: 'transparent',
border: 'none',
color: isDark ? '#6b7280' : '#9ca3af',
backgroundColor: panelBg,
borderRight: `1px solid ${border}`,
color: muted,
},
'.cm-activeLine': {
backgroundColor: isDark ? 'rgba(110, 118, 129, 0.16)' : 'rgba(175, 184, 193, 0.18)',
},
'.cm-activeLineGutter': {
backgroundColor: isDark ? 'rgba(110, 118, 129, 0.16)' : 'rgba(175, 184, 193, 0.18)',
},
'.cm-selectionBackground, &.cm-focused .cm-selectionBackground, .cm-content ::selection': {
backgroundColor: isDark ? 'rgba(88, 166, 255, 0.32)' : 'rgba(9, 105, 218, 0.22)',
},
'.cm-panels, .cm-panel': {
backgroundColor: panelBg,
color: text,
borderColor: border,
},
'.cm-mergeView': {
backgroundColor: bg,
color: text,
},
'.cm-mergeViewEditors': {
backgroundColor: bg,
},
'.cm-mergeView .cm-editor': {
borderColor: border,
},
'.cm-changedLine': {
backgroundColor: isDark ? 'rgba(56, 139, 253, 0.14)' : 'rgba(9, 105, 218, 0.08)',
},
'.cm-deletedChunk': {
backgroundColor: isDark ? 'rgba(248, 81, 73, 0.14)' : 'rgba(255, 235, 233, 0.95)',
},
'.cm-insertedLine, .cm-insertedChunk': {
backgroundColor: isDark ? 'rgba(63, 185, 80, 0.14)' : 'rgba(234, 255, 234, 0.95)',
},
'&.cm-focused': { outline: 'none' },
// GitHub-style expander bar for folded unchanged regions (@codemirror/merge).

View file

@ -1,5 +1,5 @@
import { useEffect, useRef, useState } from 'react'
import { ArrowUp, FileText, Loader2, LoaderIcon, Plus, Square, Terminal, X } from 'lucide-react'
import { useEffect, useRef, useState, type DragEvent, type MutableRefObject } from 'react'
import { ArrowUp, FileText, Loader2, LoaderIcon, Mic, Plus, Square, Terminal, X } from 'lucide-react'
import type { CodeSession, CodeSessionStatus } from '@x/shared/src/code-sessions.js'
import { cn } from '@/lib/utils'
import { toast } from 'sonner'
@ -15,9 +15,54 @@ import { CodeRunPermissionRequest, CodingRunTimeline } from '@/components/coding
import { PermissionRequest } from '@/components/ai-elements/permission-request'
import { AskHumanRequest } from '@/components/ai-elements/ask-human-request'
import { WebSearchResult } from '@/components/ai-elements/web-search-result'
import { useVoiceMode } from '@/hooks/useVoiceMode'
import { useCodeChat, isDirectTurn, isChatToolCall, isChatErrorMessage, type CodeChatItem } from './use-code-chat'
const AGENT_LABEL: Record<string, string> = { claude: 'Claude Code', codex: 'Codex' }
const WAVE_BAR_WIDTH = 3
const WAVE_BAR_GAP = 2
const WAVE_BAR_MIN = 1.5
const WAVE_BAR_MAX = 18
function VoiceWaveform({ audioLevelsRef }: { audioLevelsRef: MutableRefObject<number[]> }) {
const [bars, setBars] = useState<number[]>([])
useEffect(() => {
let raf = 0
let lastSig = ''
const tick = () => {
const levels = audioLevelsRef.current
const next = levels.length > 48 ? levels.slice(levels.length - 48) : levels
const sig = `${next.length}:${next.length ? next[next.length - 1] : 0}`
if (sig !== lastSig) {
lastSig = sig
setBars(next.slice())
}
raf = requestAnimationFrame(tick)
}
raf = requestAnimationFrame(tick)
return () => cancelAnimationFrame(raf)
}, [audioLevelsRef])
return (
<div className="flex h-5 w-full items-center overflow-hidden" style={{ gap: WAVE_BAR_GAP }}>
{bars.map((level, i) => {
const amp = Math.min(1, Math.max(0, level)) ** 0.8
return (
<span
key={i}
className="shrink-0 rounded-full bg-primary"
style={{
width: WAVE_BAR_WIDTH,
height: WAVE_BAR_MIN + amp * (WAVE_BAR_MAX - WAVE_BAR_MIN),
transition: 'height 90ms linear',
}}
/>
)
})}
</div>
)
}
function RowboatToolCall({ item, onOpenDiff }: { item: ToolCall; onOpenDiff: (path: string) => void }) {
const [open, setOpen] = useState(false)
@ -97,20 +142,30 @@ export function CodeChat({
session,
status,
onOpenDiff,
voiceAvailable = false,
}: {
session: CodeSession
status: CodeSessionStatus
onOpenDiff: (path: string) => void
voiceAvailable?: boolean
}) {
const {
items, liveText, isProcessing, pendingPermission, pendingToolPermissions, pendingAskHumans,
items, liveText, isProcessing, compactionStatus, contextUsage,
pendingPermission, pendingToolPermissions, pendingAskHumans,
loading, send, stop, resolvePermission, respondToToolPermission, respondToAskHuman,
} = useCodeChat(session)
const [draft, setDraft] = useState('')
const [stopping, setStopping] = useState(false)
const textareaRef = useRef<HTMLTextAreaElement>(null)
const voice = useVoiceMode()
const voiceWarmup = voice.warmup
const busy = isProcessing || status === 'working' || status === 'needs-you'
const recording = voice.state !== 'idle'
const recordingStopping = voice.state === 'submitting'
const contextUsedPercent = contextUsage
? Math.min(100, Math.round((contextUsage.used / contextUsage.size) * 100))
: null
// Attached file PATHS — like dragging a file into the Claude Code CLI, the
// agent receives paths and reads the files itself with its own tools.
const [attachments, setAttachments] = useState<string[]>([])
@ -119,6 +174,7 @@ export function CodeChat({
setDraft('')
setAttachments([])
setStopping(false)
voice.cancel()
textareaRef.current?.focus()
}, [session.id])
@ -126,6 +182,10 @@ export function CodeChat({
if (!busy) setStopping(false)
}, [busy])
useEffect(() => {
if (voiceAvailable) voiceWarmup()
}, [voiceAvailable, voiceWarmup])
const addAttachments = (paths: string[]) => {
const cleaned = paths.filter(Boolean)
if (cleaned.length === 0) return
@ -141,7 +201,7 @@ export function CodeChat({
textareaRef.current?.focus()
}
const handleDrop = (e: React.DragEvent) => {
const handleDrop = (e: DragEvent) => {
if (!e.dataTransfer?.files?.length) return
e.preventDefault()
const paths = Array.from(e.dataTransfer.files)
@ -175,6 +235,27 @@ export function CodeChat({
await stop()
}
const handleStartRecording = () => {
if (busy) return
void voice.start()
}
const handleSubmitRecording = async () => {
if (!recording || recordingStopping) return
const text = await voice.submit()
if (!text) return
const result = await send(text)
if (!result.ok && result.error) {
toast.error(result.error)
setDraft(text)
}
}
const handleCancelRecording = () => {
voice.cancel()
textareaRef.current?.focus()
}
const basename = (p: string) => p.split(/[\\/]/).pop() || p
return (
@ -188,7 +269,10 @@ export function CodeChat({
<Terminal className="size-3.5 shrink-0 text-muted-foreground" />
<div className="min-w-0 flex-1">
<div className="truncate text-sm font-medium">{session.title}</div>
<div className="text-[11px] text-muted-foreground">{AGENT_LABEL[session.agent]} direct</div>
<div className="text-[11px] text-muted-foreground">
{AGENT_LABEL[session.agent]} direct
{contextUsedPercent != null ? ` · ${contextUsedPercent}% context used` : ''}
</div>
</div>
</div>
@ -239,9 +323,19 @@ export function CodeChat({
/>
))}
{busy && !pendingPermission && pendingToolPermissions.size === 0 && pendingAskHumans.size === 0 && (
<Shimmer className="text-sm">
{stopping ? 'Stopping…' : `${AGENT_LABEL[session.agent]} is working…`}
</Shimmer>
compactionStatus === 'stalled' ? (
<div className="text-sm text-amber-600">
Context compaction is taking longer than expected. You can stop and retry in a fresh session.
</div>
) : (
<Shimmer className="text-sm">
{stopping
? 'Stopping…'
: compactionStatus === 'running'
? 'Compacting context…'
: `${AGENT_LABEL[session.agent]} is working…`}
</Shimmer>
)
)}
</ConversationContent>
<ConversationScrollButton />
@ -249,8 +343,8 @@ export function CodeChat({
{/* Composer mirrors the assistant chat input's look (rounded card,
borderless textarea, round primary send / destructive stop). */}
<div className="p-3">
<div className="rowboat-chat-input mx-auto w-full max-w-3xl rounded-lg border border-border bg-background shadow-none">
<div className="bg-background p-3 dark:bg-black">
<div className="rowboat-chat-input rowboat-code-chat-input mx-auto w-full max-w-3xl rounded-lg border border-border bg-background shadow-none">
{attachments.length > 0 && (
<div className="flex flex-wrap gap-2 px-4 pb-1 pt-3">
{attachments.map((p) => (
@ -273,22 +367,58 @@ export function CodeChat({
))}
</div>
)}
<div className="px-4 pb-2 pt-4">
<Textarea
ref={textareaRef}
value={draft}
onChange={(e) => setDraft(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault()
void handleSend()
}
}}
placeholder="Type your message..."
className="max-h-40 min-h-[24px] w-full resize-none border-0 bg-transparent p-0 text-sm shadow-none outline-none focus-visible:ring-0"
rows={2}
/>
</div>
{recording ? (
<div className="flex items-center gap-3 px-4 py-3">
<button
type="button"
onClick={handleCancelRecording}
className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full text-muted-foreground transition-colors hover:bg-muted hover:text-foreground"
aria-label="Cancel recording"
>
<X className="h-4 w-4" />
</button>
<div className="flex min-w-0 flex-1 flex-col gap-1 overflow-hidden">
<VoiceWaveform audioLevelsRef={voice.audioLevelsRef} />
<div className={cn('min-h-5 truncate text-sm leading-5', voice.interimText.trim() ? 'text-foreground' : 'text-muted-foreground')}>
{voice.interimText.trim() || (recordingStopping ? 'Finalizing...' : 'Listening...')}
</div>
</div>
<Button
size="icon"
onClick={() => void handleSubmitRecording()}
disabled={recordingStopping}
className={cn(
'h-7 w-7 shrink-0 rounded-full transition-all',
recordingStopping
? 'bg-muted text-muted-foreground'
: 'bg-primary text-primary-foreground hover:bg-primary/90',
)}
>
{recordingStopping ? (
<LoaderIcon className="h-4 w-4 animate-spin" />
) : (
<ArrowUp className="h-4 w-4" />
)}
</Button>
</div>
) : (
<div className="px-4 pb-2 pt-4">
<Textarea
ref={textareaRef}
value={draft}
onChange={(e) => setDraft(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault()
void handleSend()
}
}}
placeholder="Type your message..."
className="max-h-40 min-h-[24px] w-full resize-none border-0 bg-transparent p-0 text-sm shadow-none outline-none focus-visible:ring-0"
rows={2}
/>
</div>
)}
<div className="flex items-center gap-2 px-3 pb-3">
<Tooltip>
<TooltipTrigger asChild>
@ -303,6 +433,22 @@ export function CodeChat({
</TooltipTrigger>
<TooltipContent side="top">Attach files the agent reads them from disk (or drag & drop)</TooltipContent>
</Tooltip>
{voiceAvailable && (
<Tooltip>
<TooltipTrigger asChild>
<button
type="button"
onClick={handleStartRecording}
disabled={busy || recording}
className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full text-muted-foreground transition-colors hover:bg-muted hover:text-foreground disabled:cursor-not-allowed disabled:opacity-50"
aria-label="Voice input"
>
<Mic className="h-4 w-4" />
</button>
</TooltipTrigger>
<TooltipContent side="top">Voice input</TooltipContent>
</Tooltip>
)}
<span className="flex min-w-0 items-center gap-1.5 text-xs text-muted-foreground">
<Terminal className="size-3.5 shrink-0" />
<span className="truncate">Direct straight to {AGENT_LABEL[session.agent]}</span>

View file

@ -338,12 +338,14 @@ export function CodeView({
{terminalOpen ? <ChevronDown className="size-3.5" /> : <ChevronUp className="size-3.5" />}
</button>
{terminalOpen && (
<div style={{ height: terminalHeight }}>
<TerminalPane
key={selectedSession.id}
terminalId={selectedSession.id}
cwd={selectedSession.cwd}
/>
<div className="bg-background pb-3 dark:bg-black" style={{ height: terminalHeight + 12 }}>
<div className="h-full min-h-0">
<TerminalPane
key={selectedSession.id}
terminalId={selectedSession.id}
cwd={selectedSession.cwd}
/>
</div>
</div>
)}
</div>

View file

@ -78,8 +78,8 @@ export function NewSessionDialog({
}) {
const [agentStatus, setAgentStatus] = useState<{ claude: AgentStatus; codex: AgentStatus } | null>(null)
const [agent, setAgent] = useState<CodingAgent>('claude')
// Rowboat drives by default — direct CLI access is the power-user opt-in.
const [mode, setMode] = useState<CodeSessionMode>('rowboat')
// Direct drive by default; Rowboat orchestration remains an opt-in per session.
const [mode, setMode] = useState<CodeSessionMode>('direct')
const [policy, setPolicy] = useState<ApprovalPolicy>('auto-approve-reads')
const [isolation, setIsolation] = useState<'in-repo' | 'worktree'>('in-repo')
const [title, setTitle] = useState('')
@ -101,7 +101,7 @@ export function NewSessionDialog({
setTitle('')
setCreating(false)
setIsolation('in-repo')
setMode('rowboat')
setMode('direct')
setModelKey('default')
setAgentModel('default')
setAgentEffort('default')

View file

@ -6,7 +6,7 @@ import { useTheme } from '@/contexts/theme-context'
// xterm color schemes tuned to the app's light/dark backgrounds.
const DARK_THEME = {
background: '#1b1b1f',
background: '#000000',
foreground: '#d4d4d8',
cursor: '#d4d4d8',
selectionBackground: 'rgba(120, 140, 255, 0.3)',
@ -106,5 +106,11 @@ export function TerminalPane({ terminalId, cwd }: { terminalId: string; cwd: str
if (term) term.options.theme = resolvedTheme === 'dark' ? DARK_THEME : LIGHT_THEME
}, [resolvedTheme])
return <div ref={containerRef} className="h-full w-full overflow-hidden px-2 pt-1" />
return (
<div
ref={containerRef}
className="h-full w-full overflow-hidden px-2 pt-1"
style={{ backgroundColor: resolvedTheme === 'dark' ? '#000000' : '#ffffff' }}
/>
)
}

View file

@ -41,6 +41,10 @@ export interface PendingCodePermission {
const DIRECT_PREFIX = 'direct-'
const STRUCTURAL_EVENTS = new Set(['tool_call', 'tool_call_update', 'plan', 'permission'])
const COMPACTION_TITLE = 'Compacting context'
const COMPACTION_STALLED_MS = 90_000
export type CompactionStatus = 'idle' | 'running' | 'stalled'
function messageText(content: unknown): string {
if (typeof content === 'string') return content
@ -62,6 +66,8 @@ export function useCodeChat(session: CodeSession | null) {
const [items, setItems] = useState<CodeChatItem[]>([])
const [liveText, setLiveText] = useState('')
const [isProcessing, setIsProcessing] = useState(false)
const [compactionStatus, setCompactionStatus] = useState<CompactionStatus>('idle')
const [contextUsage, setContextUsage] = useState<{ used: number; size: number } | null>(null)
const [pendingPermission, setPendingPermission] = useState<PendingCodePermission | null>(null)
// Rowboat-mode copilot gates, same as the main chat: pre-tool-call permission
// requests and ask-human questions. Keyed by toolCallId.
@ -69,6 +75,7 @@ export function useCodeChat(session: CodeSession | null) {
const [pendingAskHumans, setPendingAskHumans] = useState<Map<string, z.infer<typeof AskHumanRequestEvent>>>(new Map())
const [loading, setLoading] = useState(false)
const seenMessageIdsRef = useRef<Set<string>>(new Set())
const compactionToolIdRef = useRef<string | null>(null)
const applyCodeRunEvent = useCallback((toolCallId: string, event: CodeRunEvent) => {
if (toolCallId.startsWith(DIRECT_PREFIX)) {
@ -99,6 +106,9 @@ export function useCodeChat(session: CodeSession | null) {
if (!sessionId) {
setItems([])
setLiveText('')
setCompactionStatus('idle')
setContextUsage(null)
compactionToolIdRef.current = null
setPendingPermission(null)
return
}
@ -106,6 +116,9 @@ export function useCodeChat(session: CodeSession | null) {
setLoading(true)
setItems([])
setLiveText('')
setCompactionStatus('idle')
setContextUsage(null)
compactionToolIdRef.current = null
setPendingPermission(null)
setPendingToolPermissions(new Map())
setPendingAskHumans(new Map())
@ -217,6 +230,12 @@ export function useCodeChat(session: CodeSession | null) {
return () => { cancelled = true }
}, [sessionId])
useEffect(() => {
if (compactionStatus !== 'running') return
const timer = window.setTimeout(() => setCompactionStatus('stalled'), COMPACTION_STALLED_MS)
return () => window.clearTimeout(timer)
}, [compactionStatus])
// Live event stream.
useEffect(() => {
if (!sessionId) return
@ -230,6 +249,8 @@ export function useCodeChat(session: CodeSession | null) {
break
case 'run-processing-end':
setIsProcessing(false)
setCompactionStatus('idle')
compactionToolIdRef.current = null
setPendingPermission(null)
// Anything still streaming that never landed as a message (e.g. the
// turn errored) is flushed so the text isn't lost.
@ -247,6 +268,8 @@ export function useCodeChat(session: CodeSession | null) {
break
case 'run-stopped':
setIsProcessing(false)
setCompactionStatus('idle')
compactionToolIdRef.current = null
setPendingPermission(null)
setPendingToolPermissions(new Map())
setPendingAskHumans(new Map())
@ -348,6 +371,19 @@ export function useCodeChat(session: CodeSession | null) {
break
case 'code-run-event': {
setIsProcessing(true)
if (event.event.type === 'usage') {
setContextUsage({ used: event.event.used, size: event.event.size })
}
if (event.event.type === 'tool_call' && event.event.title === COMPACTION_TITLE) {
compactionToolIdRef.current = event.event.id ?? null
setCompactionStatus('running')
}
if (event.event.type === 'tool_call_update'
&& event.event.id != null
&& event.event.id === compactionToolIdRef.current) {
compactionToolIdRef.current = null
setCompactionStatus('idle')
}
if (event.event.type === 'message' && event.event.role === 'agent' && event.toolCallId.startsWith(DIRECT_PREFIX)) {
const text = event.event.text
setLiveText((prev) => prev + text)
@ -460,6 +496,8 @@ export function useCodeChat(session: CodeSession | null) {
items,
liveText,
isProcessing,
compactionStatus,
contextUsage,
pendingPermission,
pendingToolPermissions,
pendingAskHumans,

View file

@ -9,6 +9,7 @@ import {
Pencil,
Search,
ShieldQuestion,
Sparkles,
Terminal,
Trash2,
Wrench,
@ -87,7 +88,8 @@ export function reduceEvents(events: CodeRunEvent[]): Row[] {
return rows
}
function toolKindIcon(kind?: string) {
function toolKindIcon(kind?: string, title?: string) {
if (title === 'Compacting context') return <Sparkles className="size-3.5 shrink-0 text-muted-foreground" />
switch (kind) {
case 'read': return <Eye className="size-3.5 shrink-0 text-muted-foreground" />
case 'edit': return <Pencil className="size-3.5 shrink-0 text-muted-foreground" />
@ -137,7 +139,7 @@ export function CodingRunTimeline({
{running
? <Loader className="size-3.5 shrink-0 animate-spin text-muted-foreground" />
: <CheckCircle2 className="size-3.5 shrink-0 text-green-600" />}
{toolKindIcon(row.toolKind)}
{toolKindIcon(row.toolKind, row.title)}
<span className="truncate text-foreground/90">{row.title ?? row.toolKind ?? 'Tool call'}</span>
</div>
{row.diffs.length > 0 && (

File diff suppressed because it is too large Load diff

View file

@ -11,11 +11,9 @@ import {
ChevronDown, ChevronRight,
} from 'lucide-react'
import { LiveNoteSchema, type LiveNote, type Triggers } from '@x/shared/dist/live-note.js'
import type { Run } from '@x/shared/dist/runs.js'
import type z from 'zod'
import { useLiveNoteAgentStatus } from '@/hooks/use-live-note-agent-status'
import { formatRelativeTime } from '@/lib/relative-time'
import { runLogToConversation } from '@/lib/run-to-conversation'
import { fetchAgentRunTranscript, type AgentRunTranscript } from '@/lib/agent-transcript'
import { CompactConversation } from '@/components/compact-conversation'
export type OpenLiveNotePanelDetail = {
@ -661,7 +659,7 @@ function SectionRegion({ label, children }: { label?: string; children: React.Re
}
function LastRunTab({ live }: { live: LiveNote }) {
const [run, setRun] = useState<z.infer<typeof Run> | null>(null)
const [transcript, setTranscript] = useState<AgentRunTranscript | null>(null)
const [loadingRun, setLoadingRun] = useState(false)
const [fetchError, setFetchError] = useState<string | null>(null)
@ -669,7 +667,7 @@ function LastRunTab({ live }: { live: LiveNote }) {
useEffect(() => {
if (!runId) {
setRun(null)
setTranscript(null)
setFetchError(null)
setLoadingRun(false)
return
@ -679,13 +677,13 @@ function LastRunTab({ live }: { live: LiveNote }) {
setFetchError(null)
void (async () => {
try {
const r = await window.ipc.invoke('runs:fetch', { runId })
const t = await fetchAgentRunTranscript(runId)
if (cancelled) return
setRun(r)
setTranscript(t)
} catch (err) {
if (cancelled) return
setFetchError(err instanceof Error ? err.message : String(err))
setRun(null)
setTranscript(null)
} finally {
if (!cancelled) setLoadingRun(false)
}
@ -704,7 +702,7 @@ function LastRunTab({ live }: { live: LiveNote }) {
}
const isError = !!live.lastRunError
const items = run ? runLogToConversation(run.log) : []
const items = transcript?.items ?? []
return (
<div className="flex-1 overflow-auto px-4 py-4 space-y-4">
@ -751,10 +749,10 @@ function LastRunTab({ live }: { live: LiveNote }) {
Couldn't load transcript: {fetchError}
</div>
)}
{run && !loadingRun && items.length === 0 && (
{transcript && !loadingRun && items.length === 0 && (
<p className="text-xs italic text-muted-foreground">No messages or tool calls recorded.</p>
)}
{run && !loadingRun && items.length > 0 && (
{transcript && !loadingRun && items.length > 0 && (
<CompactConversation items={items} />
)}
</div>

View file

@ -1,6 +1,7 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { createPortal } from 'react-dom'
import { Calendar, ChevronDown, Clock, ExternalLink, Loader2, MapPin, Mic, Square, UserRound, UsersRound, Video, X } from 'lucide-react'
import { Calendar, ChevronDown, ChevronRight, Clock, ExternalLink, FileText, Loader2, MapPin, Mic, Sparkles, Square, UserPlus, UserRound, UsersRound, Video, X } from 'lucide-react'
import { Streamdown } from 'streamdown'
import { Button } from '@/components/ui/button'
import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover'
@ -14,6 +15,39 @@ const MEETINGS_ROOT = 'knowledge/Meetings'
const CALENDAR_DIR = 'calendar_sync'
const UPCOMING_MAX_DAYS = 4 // today + next 3
declare global {
interface Window {
__pendingMeetingPrepCreate?: { prompt: string }
}
}
// Mirrors the `meeting-prep:resolve` IPC response shape.
type PrepNote = {
path: string
name: string
role?: string
organization?: string
markdown: string
}
type PrepAttendee = {
label: string
email?: string
displayName?: string
note: PrepNote | null
}
type PrepOrg = {
path: string
name: string
markdown: string
}
type PrepResult = {
attendees: PrepAttendee[]
organizations: PrepOrg[]
prepNote: { path: string; brief: string } | null
matchedCount: number
unmatchedCount: number
}
type MeetingNoteRow = {
path: string
name: string
@ -358,7 +392,178 @@ function parseDescriptionParts(value: string): DescriptionPart[] {
}).filter((part) => part.text.length > 0)
}
function UpcomingEvents() {
// Hand the unmatched attendee off to the Copilot to research + create a note.
function requestCreateNote(attendee: PrepAttendee, meetingSummary: string) {
const who = attendee.displayName || attendee.label
const email = attendee.email ? ` <${attendee.email}>` : ''
window.__pendingMeetingPrepCreate = {
prompt: `Create a person note in my knowledge base for ${who}${email}. They're attending my "${meetingSummary}" meeting. Pull together what you know about them from my emails, past meetings, and calendar.`,
}
window.dispatchEvent(new Event('meeting-prep:create-note'))
}
// One note row (used for both people and organizations): a clickable row that
// navigates to the note. The markdown is NOT rendered inline in the card.
function PrepNoteRow({ title, subtitle, path, onOpenNote }: {
title: string
subtitle?: string
path: string
onOpenNote: (path: string) => void
}) {
return (
<button
type="button"
onClick={() => onOpenNote(path)}
title={`Open ${title}`}
className="flex w-full items-center gap-3 border-b px-5 py-1.5 text-left transition-colors last:border-b-0 hover:bg-muted/50"
>
<span className="flex min-w-0 flex-1 flex-col">
<span className="truncate text-[12.5px] font-semibold text-foreground">{title}</span>
{subtitle ? <span className="truncate text-[11px] text-muted-foreground">{subtitle}</span> : null}
</span>
<ChevronRight className="size-3.5 shrink-0 text-muted-foreground" />
</button>
)
}
function PrepAttendeeNote({ attendee, onOpenNote }: { attendee: PrepAttendee; onOpenNote: (path: string) => void }) {
const note = attendee.note
if (!note) return null
const subtitle = [note.role, note.organization].filter(Boolean).join(' · ')
return <PrepNoteRow title={note.name} subtitle={subtitle || undefined} path={note.path} onOpenNote={onOpenNote} />
}
function PrepUnmatchedSection({ attendees, meetingSummary }: { attendees: PrepAttendee[]; meetingSummary: string }) {
const [open, setOpen] = useState(false)
if (attendees.length === 0) return null
return (
<div className="border-t bg-muted/20">
<button
type="button"
onClick={() => setOpen((v) => !v)}
className="flex w-full items-center gap-3 px-5 py-2.5 text-left transition-colors hover:bg-muted/40"
>
{open ? <ChevronDown className="size-4 shrink-0 text-muted-foreground" /> : <ChevronRight className="size-4 shrink-0 text-muted-foreground" />}
<span className="text-xs font-medium text-muted-foreground">
{attendees.length} {attendees.length === 1 ? 'other' : 'others'} no notes yet
</span>
</button>
{open ? (
<div className="flex flex-col gap-1 px-5 pb-3 pl-12">
{attendees.map((att, idx) => (
<div key={`${att.email ?? att.label}-${idx}`} className="flex items-center justify-between gap-3 py-1">
<span className="min-w-0 truncate text-sm text-foreground">{att.label}</span>
<button
type="button"
onClick={() => requestCreateNote(att, meetingSummary)}
className="inline-flex shrink-0 items-center gap-1.5 rounded-md border bg-background px-2 py-1 text-xs font-medium text-foreground transition-colors hover:bg-accent"
>
<UserPlus className="size-3.5" />
Create note
</button>
</div>
))}
</div>
) : null}
</div>
)
}
// Inline prep for a single event: resolves the attendees against the knowledge
// base and renders their notes directly beneath the event row. Re-resolves when
// a person note changes (e.g. after "Create note") so it stays fresh.
function InlineMeetingPrep({ event, onOpenNote }: { event: UpcomingEvent; onOpenNote: (path: string) => void }) {
const [prep, setPrep] = useState<PrepResult | null>(null)
const [refreshTick, setRefreshTick] = useState(0)
useEffect(() => {
let cancelled = false
const run = async () => {
try {
const attendees = event.attendees.map((a) => ({ email: a.email, displayName: a.displayName, self: a.self }))
const result = await window.ipc.invoke('meeting-prep:resolve', { attendees, eventId: event.id })
if (!cancelled) setPrep(result)
} catch (err) {
console.error('Meeting prep failed:', err)
if (!cancelled) setPrep(null)
}
}
void run()
return () => {
cancelled = true
}
}, [event.id, refreshTick])
// Refresh when a People note is created/changed so newly-created notes appear.
useEffect(() => {
const isPeoplePath = (p: string | undefined) =>
typeof p === 'string' && p.startsWith('knowledge/People/')
const cleanup = window.ipc.on('workspace:didChange', (e) => {
switch (e.type) {
case 'created':
case 'changed':
case 'deleted':
if (isPeoplePath(e.path)) setRefreshTick((t) => t + 1)
break
case 'moved':
if (isPeoplePath(e.from) || isPeoplePath(e.to)) setRefreshTick((t) => t + 1)
break
case 'bulkChanged':
if (!e.paths || e.paths.some(isPeoplePath)) setRefreshTick((t) => t + 1)
break
}
})
return cleanup
}, [])
if (!prep || prep.attendees.length === 0) return null
const matched = prep.attendees.filter((a) => a.note)
const unmatched = prep.attendees.filter((a) => !a.note)
return (
<div className="bg-muted/10">
{prep.prepNote && prep.prepNote.brief ? (
<div className="border-b px-5 pb-3 pt-3">
<Streamdown className="prose prose-sm dark:prose-invert max-w-none text-foreground/90 [&>*:first-child]:mt-0 [&>*:last-child]:mb-0 [&_p]:my-1 [&_ul]:my-1 [&_ol]:my-1 [&_li]:my-0.5 [&_p]:text-[12.5px] [&_li]:text-[12.5px]">
{prep.prepNote.brief}
</Streamdown>
<button
type="button"
onClick={() => onOpenNote(prep.prepNote!.path)}
className="mt-2 inline-flex items-center gap-1.5 text-xs font-medium text-muted-foreground transition-colors hover:text-foreground"
>
<FileText className="size-3.5" />
Open full prep
</button>
</div>
) : null}
<div className="flex items-center gap-1.5 px-5 pb-1 pt-2.5">
<UsersRound className="size-3.5 text-muted-foreground" />
<span className="text-[11px] font-medium uppercase tracking-wider text-muted-foreground">People</span>
</div>
{matched.map((att, idx) => (
<PrepAttendeeNote key={att.note!.path + idx} attendee={att} onOpenNote={onOpenNote} />
))}
<PrepUnmatchedSection attendees={unmatched} meetingSummary={event.summary} />
{prep.organizations.length > 0 ? (
<>
<div className="px-5 pb-1 pt-2.5">
<span className="text-[11px] font-medium uppercase tracking-wider text-muted-foreground">
{prep.organizations.length === 1 ? 'Company' : 'Companies'}
</span>
</div>
{prep.organizations.map((org) => (
<PrepNoteRow key={org.path} title={org.name} subtitle="Organization" path={org.path} onOpenNote={onOpenNote} />
))}
</>
) : null}
</div>
)
}
function UpcomingEvents({ onOpenNote }: { onOpenNote: (path: string) => void }) {
const [events, setEvents] = useState<UpcomingEvent[]>([])
const [loading, setLoading] = useState(true)
const [error, setError] = useState<string | null>(null)
@ -487,6 +692,20 @@ function UpcomingEvents() {
return selectVisibleDays(window)
}, [events])
// The next meeting that's worth prepping for — soonest timed event with at
// least one other attendee that hasn't ended. Its row gets inline prep.
// `events` is sorted (all-day first, then by start), so `find` returns it.
const prepEventId = useMemo(() => {
const nowMs = Date.now()
const candidate = events.find((ev) => {
if (ev.isAllDay) return false
if (ev.attendees.every((a) => a.self)) return false
const endMs = ev.end ? ev.end.getTime() : ev.start.getTime() + 30 * 60 * 1000
return endMs > nowMs
})
return candidate?.id ?? null
}, [events])
const totalVisible = visibleDays.reduce((s, d) => s + d.events.length, 0)
const now = new Date()
const todayKey = localDateKey(now)
@ -532,6 +751,8 @@ function UpcomingEvents() {
key={day.dateKey}
day={day}
isToday={day.dateKey === todayKey}
prepEventId={prepEventId}
onOpenNote={onOpenNote}
/>
))}
</div>
@ -542,7 +763,7 @@ function UpcomingEvents() {
)
}
function UpcomingDayCard({ day, isToday }: { day: DayGroup; isToday: boolean }) {
function UpcomingDayCard({ day, isToday, prepEventId, onOpenNote }: { day: DayGroup; isToday: boolean; prepEventId: string | null; onOpenNote: (path: string) => void }) {
const dayNum = day.date.getDate()
const month = day.date.toLocaleDateString([], { month: 'short' })
const weekday = day.date.toLocaleDateString([], { weekday: 'short' })
@ -573,7 +794,13 @@ function UpcomingDayCard({ day, isToday }: { day: DayGroup; isToday: boolean })
</div>
) : (
day.events.map((ev, idx) => (
<UpcomingEventItem key={ev.id} event={ev} isLast={idx === count - 1} />
<UpcomingEventItem
key={ev.id}
event={ev}
isLast={idx === count - 1}
isPrepTarget={ev.id === prepEventId}
onOpenNote={onOpenNote}
/>
))
)}
</div>
@ -588,14 +815,20 @@ function NowBadge() {
)
}
function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: boolean }) {
function UpcomingEventItem({ event, isLast, isPrepTarget, onOpenNote }: { event: UpcomingEvent; isLast: boolean; isPrepTarget: boolean; onOpenNote: (path: string) => void }) {
const [open, setOpen] = useState(false)
// The next meeting auto-expands its prep; any other meeting with attendees
// can be expanded on demand via the Prep toggle (resolves lazily on open).
const prepEligible = !event.isAllDay && event.attendees.some((a) => !a.self)
const [prepOpen, setPrepOpen] = useState(isPrepTarget)
const showPrep = prepEligible && prepOpen
const isNow = isEventNow(event)
const platform = meetingPlatformLabel(event.conferenceLink)
const subtitle = platform ?? event.location
const titleAndLocation = event.location ? `${event.summary} · ${event.location}` : event.summary
return (
<div className={cn(!isLast && 'border-b')}>
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>
<div
@ -604,7 +837,7 @@ function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: bo
title={titleAndLocation}
className={cn(
'group flex w-full cursor-pointer items-center gap-4 px-5 py-3 text-left transition-colors',
!isLast && 'border-b',
showPrep && 'border-b',
isNow ? 'bg-muted' : 'hover:bg-muted/50',
)}
>
@ -625,7 +858,24 @@ function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: bo
</span>
) : null}
</span>
<div className="shrink-0">
<div className="flex shrink-0 items-center gap-2">
{prepEligible ? (
<button
type="button"
onClick={(e) => { e.stopPropagation(); setPrepOpen((v) => !v) }}
onMouseDown={(e) => e.stopPropagation()}
aria-expanded={prepOpen}
title={prepOpen ? 'Hide prep' : 'Show meeting prep'}
className={cn(
'inline-flex items-center gap-1.5 rounded-md border px-2.5 py-1.5 text-xs font-medium transition-colors',
prepOpen ? 'bg-accent text-foreground' : 'bg-background text-foreground hover:bg-accent',
)}
>
<Sparkles className="size-3.5" />
Prep
<ChevronDown className={cn('size-3 transition-transform', prepOpen && 'rotate-180')} />
</button>
) : null}
{event.conferenceLink ? (
<SplitJoinButton
onJoinAndNotes={() => triggerMeetingCapture(event, true)}
@ -647,6 +897,8 @@ function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: bo
</PopoverTrigger>
<EventDetailsPopover event={event} onClose={() => setOpen(false)} />
</Popover>
{showPrep ? <InlineMeetingPrep event={event} onOpenNote={onOpenNote} /> : null}
</div>
)
}
@ -917,6 +1169,9 @@ export function MeetingsView({ onOpenNote, onTakeMeetingNotes, meetingState, mee
const rows = entries
.filter((entry) => entry.kind === 'file' && entry.name.endsWith('.md'))
// Generated prep notes live under Meetings/prep/ — they're upcoming
// prep, not past meeting notes, so keep them out of this table.
.filter((entry) => !entry.path.startsWith(`${MEETINGS_ROOT}/prep/`))
.map((entry) => {
const relative = entry.path.slice(`${MEETINGS_ROOT}/`.length)
const parts = relative.split('/')
@ -1017,7 +1272,7 @@ export function MeetingsView({ onOpenNote, onTakeMeetingNotes, meetingState, mee
</p>
</div>
<div className="flex-1 overflow-auto">
<UpcomingEvents />
<UpcomingEvents onOpenNote={onOpenNote} />
<div className="p-6">
{loading ? (
<div className="flex items-center justify-center py-10">

View file

@ -0,0 +1,878 @@
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
import { createPortal } from 'react-dom'
import { X } from 'lucide-react'
import { Button } from '@/components/ui/button'
import { TalkingHead, type MascotHat } from '@/components/talking-head'
import { AgentsFleet, MascotVignette, type TourVignetteKind } from '@/components/tour-vignettes'
import { TourSounds } from '@/lib/tour-sounds'
import type { TTSState } from '@/hooks/useVoiceTTS'
import { cn } from '@/lib/utils'
import tourClipWelcome from '@/assets/tour/welcome.mp3'
import tourClipHome from '@/assets/tour/home.mp3'
import tourClipEmail from '@/assets/tour/email.mp3'
import tourClipMeetings from '@/assets/tour/meetings.mp3'
import tourClipCode from '@/assets/tour/code.mp3'
import tourClipKnowledge from '@/assets/tour/knowledge.mp3'
import tourClipAgents from '@/assets/tour/agents.mp3'
import tourClipWorkspaces from '@/assets/tour/workspaces.mp3'
import tourClipChats from '@/assets/tour/chats.mp3'
import tourClipComposer from '@/assets/tour/composer.mp3'
import tourClipDone from '@/assets/tour/done.mp3'
export type TourNavTarget =
| 'home'
| 'email'
| 'meetings'
| 'code'
| 'knowledge'
| 'agents'
| 'workspaces'
type TourStep = {
id: string
/** Matches a [data-tour-id] element. Steps whose target is absent are skipped. */
targetId?: string
/** View to open when the step starts, via App's navigation handlers. */
navigate?: TourNavTarget
/** Costume the mascot wears at this stop. */
hat?: MascotHat
/** Looping animation staged around the mascot while it presents this stop. */
vignette?: TourVignetteKind
title: string
text: string
/** Spoken narration, when it should differ from the bubble text. */
voiceText?: string
}
const TOUR_STEPS: TourStep[] = [
{
id: 'welcome',
title: 'All aboard! ⚓',
text: "I'm your captain for the next minute. The lights are down and the water's in — let me row you across Rowboat, one stop at a time. Use Next or your arrow keys.",
voiceText: "I'm your captain for the next minute. The lights are down and the water's in — let me row you across Rowboat, one stop at a time.",
},
{
id: 'home',
targetId: 'nav-home',
navigate: 'home',
title: 'First stop: Home',
text: 'Home is your landing spot — a quick overview of what needs your attention to get you back into the flow.',
},
{
id: 'email',
targetId: 'nav-email',
navigate: 'email',
hat: 'mailcap',
vignette: 'email',
title: 'Email',
text: 'Read and triage your inbox right here. Rowboat can summarize threads, label messages, and help you draft replies.',
},
{
id: 'meetings',
targetId: 'nav-meetings',
navigate: 'meetings',
hat: 'headphones',
vignette: 'meetings',
title: 'Meetings',
text: 'Record or join meetings, and get transcripts and notes automatically — prep briefs show up before your calls, too.',
},
{
id: 'code',
targetId: 'nav-code',
navigate: 'code',
hat: 'hardhat',
title: 'Code',
text: 'The Code section runs coding agents on your projects — point one at a folder and drive it from a chat.',
},
{
id: 'knowledge',
targetId: 'nav-knowledge',
navigate: 'knowledge',
hat: 'gradcap',
vignette: 'brain',
title: 'Brain',
text: "Brain is your knowledge base — notes, files, and everything Rowboat learns for you, all connected and searchable.",
},
{
id: 'agents',
targetId: 'nav-agents',
navigate: 'agents',
hat: 'captain',
vignette: 'agents',
title: 'Background agents',
text: 'Background agents work on schedules — they keep your Brain fresh and take care of recurring tasks while you row elsewhere.',
},
{
id: 'workspaces',
targetId: 'nav-workspaces',
navigate: 'workspaces',
hat: 'explorer',
title: 'Workspaces',
text: 'Workspaces hold your project folders and files, so related work stays docked together.',
},
{
id: 'chats',
targetId: 'nav-chats',
title: 'Chats',
text: 'Your recent conversations live here — pick any of them back up right where you left off.',
},
{
id: 'composer',
targetId: 'chat-composer',
title: 'Talk to Rowboat',
text: 'And this is where we talk! Type, dictate with the mic, or turn on voice output — tap my face button and Ill read replies out loud myself.',
},
{
id: 'done',
hat: 'party',
title: "Land ho! 🎉",
text: "That's the whole bay — and there's my wake to prove it. Take this voyage again anytime from the bottom of the sidebar. Happy rowing!",
},
]
// Pre-recorded narration bundled with the app (no TTS API call, works
// offline/signed-out). Regenerate with scripts/generate-tour-audio.mjs after
// editing any step's text. Steps without a clip fall back to live TTS.
const TOUR_CLIPS: Record<string, string> = {
welcome: tourClipWelcome,
home: tourClipHome,
email: tourClipEmail,
meetings: tourClipMeetings,
code: tourClipCode,
knowledge: tourClipKnowledge,
agents: tourClipAgents,
workspaces: tourClipWorkspaces,
chats: tourClipChats,
composer: tourClipComposer,
done: tourClipDone,
}
const MASCOT_SIZE = 120
const VIEWPORT_MARGIN = 16
const BUBBLE_WIDTH = 288
const TARGET_RESOLVE_TIMEOUT_MS = 1500
const ZOOM_SCALE = 1.05
const GLIDE_EASING = 'cubic-bezier(0.45, 0, 0.2, 1)'
type Pt = { x: number; y: number }
type Rect = { left: number; top: number; width: number; height: number }
type Spot = Rect & { round: boolean }
function clamp(v: number, min: number, max: number): number {
return Math.max(min, Math.min(max, v))
}
function easeInOutCubic(t: number): number {
return t < 0.5 ? 4 * t * t * t : 1 - Math.pow(-2 * t + 2, 3) / 2
}
function quadPoint(p0: Pt, c: Pt, p1: Pt, t: number): Pt {
const u = 1 - t
return {
x: u * u * p0.x + 2 * u * t * c.x + t * t * p1.x,
y: u * u * p0.y + 2 * u * t * c.y + t * t * p1.y,
}
}
function quadPathLength(d: string): number {
const p = document.createElementNS('http://www.w3.org/2000/svg', 'path')
p.setAttribute('d', d)
return p.getTotalLength()
}
// A data-tour-id can legitimately appear on several elements (e.g. the chat
// composer renders in both the full-screen chat and the side pane) — pick the
// one that is actually laid out.
function findTourTarget(targetId: string): HTMLElement | null {
const nodes = document.querySelectorAll<HTMLElement>(`[data-tour-id="${targetId}"]`)
for (const el of nodes) {
const rect = el.getBoundingClientRect()
if (rect.width > 0 && rect.height > 0) return el
}
return null
}
function mascotDestForCenter(): Pt {
return {
x: window.innerWidth / 2 - MASCOT_SIZE / 2 - BUBBLE_WIDTH / 2,
y: window.innerHeight / 2 - MASCOT_SIZE / 2,
}
}
function mascotDestForRect(rect: Rect): Pt {
let x: number
let y: number
const fitsRight = rect.left + rect.width + MASCOT_SIZE + BUBBLE_WIDTH + VIEWPORT_MARGIN * 3 < window.innerWidth
if (fitsRight) {
x = rect.left + rect.width + 20
y = rect.top + rect.height / 2 - MASCOT_SIZE / 2
} else if (rect.top > MASCOT_SIZE + VIEWPORT_MARGIN * 2) {
x = rect.left + rect.width / 2 - MASCOT_SIZE / 2
y = rect.top - MASCOT_SIZE - 20
} else {
x = rect.left - MASCOT_SIZE - 20
y = rect.top + rect.height / 2 - MASCOT_SIZE / 2
}
return {
x: clamp(x, VIEWPORT_MARGIN, window.innerWidth - MASCOT_SIZE - VIEWPORT_MARGIN),
y: clamp(y, VIEWPORT_MARGIN, window.innerHeight - MASCOT_SIZE - VIEWPORT_MARGIN),
}
}
function spotForRect(rect: Rect): Spot {
return {
left: rect.left - 6,
top: rect.top - 6,
width: rect.width + 12,
height: rect.height + 12,
round: false,
}
}
function spotForMascot(dest: Pt): Spot {
return {
left: dest.x - 26,
top: dest.y - 18,
width: MASCOT_SIZE + 52,
height: MASCOT_SIZE + 44,
round: true,
}
}
type ProductTourProps = {
onClose: () => void
onNavigate: (target: TourNavTarget) => void
ttsAvailable: boolean
ttsState: TTSState
speak: (text: string) => void
speakUrl: (url: string) => void
cancelSpeech: () => void
getLevel: () => number
}
/**
* The Grand Voyage: a mascot-guided walkthrough where the app dims to a
* night-time bay, the boat rows curved routes between [data-tour-id] anchors
* (leaving a dotted wake behind it), a spotlight and gentle camera zoom reveal
* each section, and a parchment mini-map charts progress. Narrated with TTS
* lip sync when available; ends in confetti.
*
* Rendered through a portal to <body> so the camera zoom applied to the app
* shell never transforms the tour's own fixed-position layers.
*/
export function ProductTour({
onClose,
onNavigate,
ttsAvailable,
ttsState,
speak,
speakUrl,
cancelSpeech,
getLevel,
}: ProductTourProps) {
const [stepIndex, setStepIndex] = useState(0)
const [arrived, setArrived] = useState(false)
const [rowing, setRowing] = useState(false)
const [flipped, setFlipped] = useState(false)
const [bubbleSide, setBubbleSide] = useState<'left' | 'right'>('right')
const [spot, setSpot] = useState<Spot | null>(null)
const [wakes, setWakes] = useState<{ id: number; d: string }[]>([])
const [activeWake, setActiveWake] = useState<{ d: string; len: number } | null>(null)
const [confettiOn, setConfettiOn] = useState(false)
const [resizeNonce, setResizeNonce] = useState(0)
const reducedMotion = useMemo(
() => window.matchMedia('(prefers-reduced-motion: reduce)').matches,
[]
)
// Mascot position is animated by mutating the container's transform directly
// (60fps travel without re-rendering React); posRef is the source of truth.
const mascotElRef = useRef<HTMLDivElement>(null)
const posRef = useRef<Pt>({
x: window.innerWidth / 2 - MASCOT_SIZE / 2,
y: window.innerHeight + 60,
})
const travelRafRef = useRef(0)
const wakePathElRef = useRef<SVGPathElement>(null)
const wakeIdRef = useRef(0)
const curveSideRef = useRef(1)
const lastSplashRef = useRef(0)
const directionRef = useRef(1)
const enteredStepRef = useRef(-1)
const stepIndexRef = useRef(stepIndex)
// Camera zoom state applied to the app shell (outside the portal)
const shellRef = useRef<HTMLElement | null>(null)
const zoomRef = useRef<{ ox: number; oy: number; s: number } | null>(null)
const soundsRef = useRef<TourSounds | null>(null)
const onCloseRef = useRef(onClose)
const onNavigateRef = useRef(onNavigate)
const speakRef = useRef(speak)
const speakUrlRef = useRef(speakUrl)
const cancelSpeechRef = useRef(cancelSpeech)
const ttsAvailableRef = useRef(ttsAvailable)
// Keep latest callbacks/state in refs so the step effect and key handlers
// stay stable. Runs before the step effect below (effect order = call order).
useEffect(() => {
stepIndexRef.current = stepIndex
onCloseRef.current = onClose
onNavigateRef.current = onNavigate
speakRef.current = speak
speakUrlRef.current = speakUrl
cancelSpeechRef.current = cancelSpeech
ttsAvailableRef.current = ttsAvailable
})
useLayoutEffect(() => {
if (mascotElRef.current) {
mascotElRef.current.style.transform = `translate(${posRef.current.x}px, ${posRef.current.y}px)`
}
soundsRef.current = new TourSounds()
return () => {
soundsRef.current?.dispose()
soundsRef.current = null
}
}, [])
// Grab the app shell for the camera zoom; restore it when the tour ends
useEffect(() => {
const shell = document.querySelector<HTMLElement>('.rowboat-shell')
shellRef.current = shell
return () => {
if (shell) {
shell.style.transform = ''
shell.style.transformOrigin = ''
shell.style.transition = ''
}
shellRef.current = null
zoomRef.current = null
}
}, [])
const applyZoom = useCallback((origin: { ox: number; oy: number } | null) => {
const shell = shellRef.current
if (!shell || reducedMotion) return
if (origin) {
// transform-origin transitions too, so moving between targets pans
// smoothly instead of jumping when the origin changes
shell.style.transition = `transform 0.9s ${GLIDE_EASING}, transform-origin 0.9s ${GLIDE_EASING}`
shell.style.transformOrigin = `${origin.ox}px ${origin.oy}px`
shell.style.transform = `scale(${ZOOM_SCALE})`
zoomRef.current = { ox: origin.ox, oy: origin.oy, s: ZOOM_SCALE }
} else {
shell.style.transition = `transform 0.9s ${GLIDE_EASING}, transform-origin 0.9s ${GLIDE_EASING}`
shell.style.transform = 'scale(1)'
zoomRef.current = null
}
}, [reducedMotion])
// Where the element will sit on screen once this step's zoom settles:
// undo the current zoom mathematically, then apply the upcoming one (whose
// origin is the target's own center, so the center never moves).
const displayedRect = useCallback((el: HTMLElement, willZoom: boolean): { rect: Rect; origin: { ox: number; oy: number } } => {
const m = el.getBoundingClientRect()
const z = zoomRef.current
let cx = m.left + m.width / 2
let cy = m.top + m.height / 2
let w = m.width
let h = m.height
if (z) {
cx = z.ox + (cx - z.ox) / z.s
cy = z.oy + (cy - z.oy) / z.s
w /= z.s
h /= z.s
}
const s = willZoom && !reducedMotion ? ZOOM_SCALE : 1
return {
rect: { left: cx - (w * s) / 2, top: cy - (h * s) / 2, width: w * s, height: h * s },
origin: { ox: cx, oy: cy },
}
}, [reducedMotion])
const cancelTravel = useCallback(() => {
cancelAnimationFrame(travelRafRef.current)
}, [])
const moveMascot = useCallback((p: Pt) => {
posRef.current = p
if (mascotElRef.current) {
mascotElRef.current.style.transform = `translate(${p.x}px, ${p.y}px)`
}
}, [])
// Row along a curved path from the current position, drawing the wake as we
// go and splashing the oar; commits the wake as a dotted trail on arrival.
const startTravel = useCallback((dest: Pt, onArrive: () => void) => {
cancelTravel()
const from = { ...posRef.current }
const dist = Math.hypot(dest.x - from.x, dest.y - from.y)
if (dist < 6 || reducedMotion) {
moveMascot(dest)
setRowing(false)
onArrive()
return
}
const dur = clamp(dist * 1.1, 550, 1500)
curveSideRef.current = -curveSideRef.current
const mx = (from.x + dest.x) / 2
const my = (from.y + dest.y) / 2
const nx = -(dest.y - from.y) / dist
const ny = (dest.x - from.x) / dist
const mag = Math.min(140, dist * 0.3) * curveSideRef.current
const c = { x: mx + nx * mag, y: my + ny * mag }
// Wake follows the stern (bottom-center of the mascot box)
const sternX = MASCOT_SIZE / 2
const sternY = MASCOT_SIZE * 0.82
const d = `M ${from.x + sternX} ${from.y + sternY} Q ${c.x + sternX} ${c.y + sternY} ${dest.x + sternX} ${dest.y + sternY}`
const len = quadPathLength(d)
setActiveWake({ d, len })
setFlipped(dest.x < from.x - 4)
setRowing(true)
lastSplashRef.current = 0
const t0 = performance.now()
const frame = (now: number) => {
const raw = Math.min(1, (now - t0) / dur)
const t = easeInOutCubic(raw)
moveMascot(quadPoint(from, c, dest, t))
if (wakePathElRef.current) {
wakePathElRef.current.style.strokeDashoffset = String(len * (1 - t))
}
if (now - lastSplashRef.current > 420) {
lastSplashRef.current = now
soundsRef.current?.splash()
}
if (raw < 1) {
travelRafRef.current = requestAnimationFrame(frame)
} else {
setRowing(false)
setActiveWake(null)
setWakes((ws) => [...ws, { id: wakeIdRef.current++, d }])
onArrive()
}
}
travelRafRef.current = requestAnimationFrame(frame)
}, [cancelTravel, moveMascot, reducedMotion])
const playDing = useCallback(() => soundsRef.current?.ding(), [])
const finish = useCallback(() => {
cancelSpeechRef.current()
onCloseRef.current()
}, [])
const goTo = useCallback((index: number, direction: 1 | -1) => {
directionRef.current = direction
if (index < 0) return
if (index >= TOUR_STEPS.length) {
finish()
return
}
// Silence the current step's narration right away — not on arrival —
// so it can't talk over (or into) the next step's speech.
cancelSpeechRef.current()
setStepIndex(index)
}, [finish])
// Stop any in-flight narration when the tour unmounts
useEffect(() => () => cancelSpeechRef.current(), [])
useEffect(() => {
const handleResize = () => setResizeNonce((n) => n + 1)
window.addEventListener('resize', handleResize)
return () => window.removeEventListener('resize', handleResize)
}, [])
// Enter the current step: navigate, wait for its anchor, aim the spotlight
// and camera, row over, then narrate. Re-runs on resize to re-anchor.
useEffect(() => {
const step = TOUR_STEPS[stepIndex]
const entering = enteredStepRef.current !== stepIndex
let cancelled = false
if (entering && step.navigate) {
onNavigateRef.current(step.navigate)
}
const settle = (dest: Pt, side: 'left' | 'right', spotlight: Spot, origin: { ox: number; oy: number } | null) => {
applyZoom(origin)
setSpot(spotlight)
setBubbleSide(side)
if (!entering) {
// Resize while already at this step: jump, keep the bubble up
cancelTravel()
moveMascot(dest)
return
}
enteredStepRef.current = stepIndex
setArrived(false)
startTravel(dest, () => {
if (cancelled) return
setArrived(true)
soundsRef.current?.bump()
cancelSpeechRef.current()
const clip = TOUR_CLIPS[step.id]
if (clip) {
speakUrlRef.current(clip)
} else if (ttsAvailableRef.current) {
speakRef.current(step.voiceText ?? step.text)
}
if (stepIndex === TOUR_STEPS.length - 1) {
setConfettiOn(true)
soundsRef.current?.fanfare()
}
})
}
if (!step.targetId) {
const dest = mascotDestForCenter()
applyZoom(null)
settle(dest, 'right', spotForMascot(dest), null)
return () => {
cancelled = true
cancelTravel()
}
}
const startedAt = performance.now()
let pollRaf = 0
const attempt = () => {
if (cancelled) return
const el = findTourTarget(step.targetId!)
if (el) {
const { rect, origin } = displayedRect(el, true)
const dest = mascotDestForRect(rect)
const side: 'left' | 'right' = dest.x + MASCOT_SIZE / 2 < window.innerWidth / 2 ? 'right' : 'left'
settle(dest, side, spotForRect(rect), origin)
return
}
if (performance.now() - startedAt < TARGET_RESOLVE_TIMEOUT_MS) {
pollRaf = requestAnimationFrame(attempt)
} else if (entering) {
// Anchor never appeared (feature disabled / pane closed) — skip past it
goTo(stepIndex + directionRef.current, directionRef.current as 1 | -1)
}
}
attempt()
return () => {
cancelled = true
cancelAnimationFrame(pollRaf)
cancelTravel()
}
}, [stepIndex, resizeNonce, goTo, applyZoom, displayedRect, startTravel, cancelTravel, moveMascot])
useEffect(() => {
const handleKeyDown = (e: KeyboardEvent) => {
if (e.key === 'Escape') {
e.preventDefault()
finish()
} else if (e.key === 'ArrowRight' || e.key === 'Enter') {
e.preventDefault()
goTo(stepIndexRef.current + 1, 1)
} else if (e.key === 'ArrowLeft') {
e.preventDefault()
goTo(stepIndexRef.current - 1, -1)
}
}
window.addEventListener('keydown', handleKeyDown)
return () => window.removeEventListener('keydown', handleKeyDown)
}, [finish, goTo])
const step = TOUR_STEPS[stepIndex]
const isFirst = stepIndex === 0
const isLast = stepIndex === TOUR_STEPS.length - 1
return createPortal(
<>
<style>{`
@keyframes tour-bubble-in {
0% { opacity: 0; transform: translateY(6px) scale(0.97); }
100% { opacity: 1; transform: translateY(0) scale(1); }
}
@keyframes tour-wave-drift {
from { transform: translateX(0); }
to { transform: translateX(-50%); }
}
@keyframes tour-stamp-in {
0% { opacity: 0; transform: scale(2); }
100% { opacity: 1; transform: scale(1); }
}
`}</style>
{/* night falls: dim everything except the spotlight cutout */}
{spot && (
<div
className="pointer-events-none fixed z-[64]"
style={{
left: spot.left,
top: spot.top,
width: spot.width,
height: spot.height,
borderRadius: spot.round ? 9999 : 14,
boxShadow: '0 0 0 200vmax rgba(7, 14, 26, 0.52)',
transition: `all 0.9s ${GLIDE_EASING}`,
}}
/>
)}
<TourWater />
{/* wake trails: the committed dotted route + the wake being drawn now */}
<svg className="pointer-events-none fixed inset-0 z-[66] h-full w-full">
{wakes.map((w) => (
<path
key={w.id}
d={w.d}
fill="none"
stroke="#9CCBEA"
strokeWidth={4}
strokeLinecap="round"
strokeDasharray="1 11"
opacity={0.75}
/>
))}
{activeWake && (
<path
ref={wakePathElRef}
d={activeWake.d}
fill="none"
stroke="#BFE0F5"
strokeWidth={5}
strokeLinecap="round"
strokeDasharray={activeWake.len}
strokeDashoffset={activeWake.len}
opacity={0.8}
/>
)}
</svg>
<TourMiniMap total={TOUR_STEPS.length} current={stepIndex} arrived={arrived} />
{/* the boat (position driven imperatively during travel) */}
<div
ref={mascotElRef}
className="fixed left-0 top-0 z-[70]"
style={{ width: MASCOT_SIZE, pointerEvents: 'none' }}
>
{arrived && !reducedMotion && step.vignette && step.vignette !== 'agents' && (
<MascotVignette kind={step.vignette} playDing={playDing} />
)}
<div style={{ transform: flipped ? 'scaleX(-1)' : undefined, transition: 'transform 0.35s ease-in-out' }}>
<TalkingHead ttsState={ttsState} getLevel={getLevel} size={MASCOT_SIZE} hat={step.hat} rowing={rowing} />
</div>
{arrived && (
<div
key={step.id}
className={cn(
'pointer-events-auto absolute top-0 z-10 rounded-xl border border-border bg-popover p-4 text-popover-foreground shadow-lg',
bubbleSide === 'right' ? 'left-full ml-3' : 'right-full mr-3'
)}
style={{
width: BUBBLE_WIDTH,
animation: 'tour-bubble-in 0.25s ease-out',
}}
>
<button
type="button"
onClick={finish}
className="absolute right-2 top-2 flex h-6 w-6 items-center justify-center rounded-md text-muted-foreground transition-colors hover:bg-muted hover:text-foreground"
aria-label="End tour"
>
<X className="h-3.5 w-3.5" />
</button>
<p className="pr-6 text-sm font-semibold">{step.title}</p>
<p className="mt-1.5 text-sm text-muted-foreground">{step.text}</p>
<div className="mt-3 flex items-center justify-between">
<span className="text-xs tabular-nums text-muted-foreground">
{stepIndex + 1} / {TOUR_STEPS.length}
</span>
<div className="flex items-center gap-1.5">
{!isFirst && (
<Button variant="ghost" size="sm" className="h-7 px-2.5" onClick={() => goTo(stepIndex - 1, -1)}>
Back
</Button>
)}
<Button
size="sm"
className="h-7 px-3"
onClick={() => (isLast ? finish() : goTo(stepIndex + 1, 1))}
>
{isLast ? 'Done' : 'Next'}
</Button>
</div>
</div>
</div>
)}
</div>
{arrived && !reducedMotion && step.vignette === 'agents' && <AgentsFleet />}
{confettiOn && !reducedMotion && <ConfettiBurst />}
</>,
document.body
)
}
/** Animated translucent waves lapping at the bottom of the screen. */
function TourWater() {
const back = useMemo(() => wavePath(2400, 96, 8, 22), [])
const front = useMemo(() => wavePath(2400, 96, 12, 30), [])
return (
<div className="pointer-events-none fixed inset-x-0 bottom-0 z-[65] h-24 overflow-hidden" aria-hidden="true">
<svg
className="absolute bottom-0 left-0 h-full"
style={{ width: '200%', animation: 'tour-wave-drift 11s linear infinite' }}
viewBox="0 0 2400 96"
preserveAspectRatio="none"
>
<path d={back} fill="#5F9BC9" opacity={0.3} />
</svg>
<svg
className="absolute bottom-0 left-0 h-full"
style={{ width: '200%', animation: 'tour-wave-drift 7s linear infinite reverse' }}
viewBox="0 0 2400 96"
preserveAspectRatio="none"
>
<path d={front} fill="#8FB6D9" opacity={0.35} />
</svg>
</div>
)
}
// Periodic wave: alternating up/down humps so a -50% translate loops seamlessly
// (both hump counts are even, so half the width is a whole number of periods).
function wavePath(width: number, height: number, humps: number, amp: number): string {
const yTop = 34
const seg = width / humps
let d = `M 0 ${yTop}`
for (let i = 0; i < humps; i++) {
d += ` q ${seg / 2} ${i % 2 === 0 ? -amp : amp} ${seg} 0`
}
d += ` L ${width} ${height} L 0 ${height} Z`
return d
}
/** Parchment chart in the corner: islands per stop, dotted route, boat marker. */
function TourMiniMap({ total, current, arrived }: { total: number; current: number; arrived: boolean }) {
const MW = 184
const MH = 96
const points = useMemo(
() =>
Array.from({ length: total }, (_, i) => {
const t = total === 1 ? 0 : i / (total - 1)
return {
x: 14 + (MW - 28) * t,
y: MH / 2 + 4 + Math.sin(t * Math.PI * 1.5 + 0.6) * (MH * 0.26),
}
}),
[total]
)
const route = points.map((p, i) => (i === 0 ? `M ${p.x} ${p.y}` : `L ${p.x} ${p.y}`)).join(' ')
const boat = points[clamp(current, 0, total - 1)]
return (
<div
className="fixed bottom-5 left-5 z-[71] rounded-lg border-2 border-[#8A6B3D]/60 bg-[#F4E9CE] px-2 pb-1.5 pt-2 shadow-xl"
style={{ transform: 'rotate(-1.2deg)' }}
aria-hidden="true"
>
<p className="mb-0.5 px-1 text-[9px] font-semibold uppercase tracking-[0.22em] text-[#6B5138]">
Voyage chart
</p>
<svg width={MW} height={MH} viewBox={`0 0 ${MW} ${MH}`}>
<path d={route} fill="none" stroke="#8A6B3D" strokeWidth={1.5} strokeDasharray="3 4" opacity={0.65} />
{points.map((p, i) => {
const visited = i < current || (i === current && arrived)
return (
<g key={i}>
<ellipse cx={p.x} cy={p.y} rx={7} ry={5} fill="#DFC896" stroke="#8A6B3D" strokeWidth={1.5} />
{visited && (
<g style={{ animation: 'tour-stamp-in 0.3s ease-out backwards' }}>
<line x1={p.x - 1} y1={p.y - 13} x2={p.x - 1} y2={p.y - 3} stroke="#7A4A21" strokeWidth={1.5} />
<path d={`M ${p.x - 1} ${p.y - 13} L ${p.x + 6} ${p.y - 10.5} L ${p.x - 1} ${p.y - 8} Z`} fill="#D9534F" />
</g>
)}
</g>
)
})}
{/* boat marker glides between islands in step with the real mascot */}
<g style={{ transform: `translate(${boat.x}px, ${boat.y - 7}px)`, transition: `transform 0.9s ${GLIDE_EASING}` }}>
<path d="M -7 0 Q 0 4 7 0 Q 4 6 0 6 Q -4 6 -7 0 Z" fill="#54402F" stroke="#3E2E24" strokeWidth={1} />
<circle cx={0} cy={-3} r={3} fill="#E8E9F5" stroke="#17171B" strokeWidth={1} />
</g>
</svg>
</div>
)
}
/** Two confetti cannons firing from the bottom corners, canvas-driven. */
function ConfettiBurst() {
const canvasRef = useRef<HTMLCanvasElement>(null)
useEffect(() => {
const canvas = canvasRef.current
const ctx = canvas?.getContext('2d')
if (!canvas || !ctx) return
const w = window.innerWidth
const h = window.innerHeight
const dpr = window.devicePixelRatio || 1
canvas.width = w * dpr
canvas.height = h * dpr
ctx.scale(dpr, dpr)
const colors = ['#5B8DEF', '#F2B8BE', '#FFD166', '#7AC74F', '#8FB6D9', '#F2699C']
const parts = Array.from({ length: 150 }, (_, i) => {
const fromLeft = i % 2 === 0
return {
x: fromLeft ? 24 : w - 24,
y: h - 40,
vx: (fromLeft ? 1 : -1) * (2.5 + Math.random() * 6.5),
vy: -(9 + Math.random() * 8),
size: 5 + Math.random() * 5,
color: colors[i % colors.length],
rot: Math.random() * Math.PI,
vr: (Math.random() - 0.5) * 0.3,
}
})
let raf = 0
const t0 = performance.now()
const frame = (now: number) => {
ctx.clearRect(0, 0, w, h)
for (const p of parts) {
p.vy += 0.18
p.vx *= 0.99
p.x += p.vx
p.y += p.vy
p.rot += p.vr
ctx.save()
ctx.translate(p.x, p.y)
ctx.rotate(p.rot)
ctx.fillStyle = p.color
ctx.fillRect(-p.size / 2, -p.size / 3, p.size, p.size * 0.66)
ctx.restore()
}
if (now - t0 < 3200) {
raf = requestAnimationFrame(frame)
} else {
ctx.clearRect(0, 0, w, h)
}
}
raf = requestAnimationFrame(frame)
return () => cancelAnimationFrame(raf)
}, [])
return (
<canvas
ref={canvasRef}
className="pointer-events-none fixed inset-0 z-[72]"
style={{ width: '100vw', height: '100vh' }}
/>
)
}

View file

@ -2,7 +2,7 @@
import * as React from "react"
import { useState, useEffect, useCallback, useMemo } from "react"
import { Server, Key, Shield, Palette, Monitor, Sun, Moon, Loader2, CheckCircle2, Plus, X, Wrench, Search, ChevronRight, Link2, Tags, Mail, BookOpen, User, Plug, HelpCircle, MessageCircle, Bug, Terminal, AlertTriangle, RefreshCw, PanelRight, Bell } from "lucide-react"
import { Server, Key, Shield, Palette, Monitor, Sun, Moon, Loader2, CheckCircle2, Plus, X, Wrench, Search, ChevronRight, Link2, Tags, Mail, BookOpen, User, Plug, HelpCircle, MessageCircle, Bug, Terminal, AlertTriangle, RefreshCw, PanelRight, Bell, Smartphone } from "lucide-react"
import {
Dialog,
@ -25,10 +25,11 @@ import { useTheme } from "@/contexts/theme-context"
import { toast } from "sonner"
import { AccountSettings } from "@/components/settings/account-settings"
import { ConnectedAccountsSettings } from "@/components/settings/connected-accounts-settings"
import { MobileChannelsSettings } from "@/components/settings/mobile-channels-settings"
import type { ApprovalPolicy } from "@x/shared/src/code-mode.js"
import { startProvisioning, useProvisioning, enabledOptimistic, type AgentStatus, type CodeModeAgentStatus } from "@/lib/code-mode-provisioning"
type ConfigTab = "account" | "connections" | "models" | "mcp" | "security" | "code-mode" | "appearance" | "notifications" | "note-tagging" | "help"
type ConfigTab = "account" | "connections" | "mobile" | "models" | "mcp" | "security" | "code-mode" | "appearance" | "notifications" | "note-tagging" | "help"
interface TabConfig {
id: ConfigTab
@ -51,6 +52,12 @@ const tabs: TabConfig[] = [
icon: Plug,
description: "Manage accounts and tools",
},
{
id: "mobile",
label: "Mobile",
icon: Smartphone,
description: "Chat with Rowboat from WhatsApp or Telegram",
},
{
id: "models",
label: "Models",
@ -2216,7 +2223,7 @@ export function SettingsDialog({ children, defaultTab = "account", open: control
</div>
{/* Content */}
<div className={cn("flex-1 p-4 min-h-0", (activeTab === "models" || activeTab === "connections" || activeTab === "account" || activeTab === "code-mode" || activeTab === "notifications") ? "overflow-y-auto" : activeTab === "note-tagging" ? "overflow-hidden flex flex-col" : "overflow-hidden")}>
<div className={cn("flex-1 p-4 min-h-0", (activeTab === "models" || activeTab === "connections" || activeTab === "mobile" || activeTab === "account" || activeTab === "code-mode" || activeTab === "notifications") ? "overflow-y-auto" : activeTab === "note-tagging" ? "overflow-hidden flex flex-col" : "overflow-hidden")}>
{activeTab === "account" ? (
<AccountSettings dialogOpen={open} />
) : activeTab === "connections" ? (
@ -2231,6 +2238,8 @@ export function SettingsDialog({ children, defaultTab = "account", open: control
<ToolsLibrarySettings dialogOpen={open} rowboatConnected={rowboatConnected} />
</div>
</div>
) : activeTab === "mobile" ? (
<MobileChannelsSettings dialogOpen={open} />
) : activeTab === "models" ? (
rowboatConnected
? <RowboatModelSettings dialogOpen={open} />

View file

@ -0,0 +1,287 @@
"use client"
import { useCallback, useEffect, useState } from "react"
import type { z } from "zod"
import { Loader2, MessageCircle, Send, Smartphone } from "lucide-react"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Separator } from "@/components/ui/separator"
import { Switch } from "@/components/ui/switch"
import { toast } from "sonner"
import type { ChannelsConfig, ChannelsStatus } from "@x/shared/src/channels.js"
type Config = z.infer<typeof ChannelsConfig>
type Status = z.infer<typeof ChannelsStatus>
// Comma/newline separated entries; each entry keeps digits only, so a
// formatted number like "+1 (415) 555-1234" survives as one entry instead of
// being shattered at the spaces.
function parseIdList(draft: string): string[] {
return draft
.split(/[,;\n]+/)
.map((s) => s.replace(/\D/g, ""))
.filter(Boolean)
}
export function MobileChannelsSettings({ dialogOpen }: { dialogOpen: boolean }) {
const [config, setConfig] = useState<Config | null>(null)
const [status, setStatus] = useState<Status | null>(null)
const [tokenDraft, setTokenDraft] = useState("")
const [waAllowDraft, setWaAllowDraft] = useState("")
const [tgAllowDraft, setTgAllowDraft] = useState("")
const [saving, setSaving] = useState(false)
useEffect(() => {
if (!dialogOpen) return
let cancelled = false
void (async () => {
try {
const [cfg, st] = await Promise.all([
window.ipc.invoke("channels:getConfig", null),
window.ipc.invoke("channels:getStatus", null),
])
if (cancelled) return
setConfig(cfg)
setStatus(st)
setTokenDraft(cfg.telegram.botToken)
setWaAllowDraft(cfg.whatsapp.allowFrom.join(", "))
setTgAllowDraft(cfg.telegram.allowFrom.join(", "))
} catch {
if (!cancelled) toast.error("Failed to load mobile channel settings")
}
})()
const unsubscribe = window.ipc.on("channels:status", (st) => {
setStatus(st)
})
return () => {
cancelled = true
unsubscribe()
}
}, [dialogOpen])
const save = useCallback(async (next: Config) => {
setConfig(next)
setSaving(true)
try {
await window.ipc.invoke("channels:setConfig", next)
} catch (error) {
toast.error(error instanceof Error ? error.message : "Failed to save channel settings")
} finally {
setSaving(false)
}
}, [])
if (!config) {
return (
<div className="flex items-center justify-center py-8 text-muted-foreground">
<Loader2 className="size-4 animate-spin" />
</div>
)
}
const wa = status?.whatsapp
const tg = status?.telegram
return (
<div className="space-y-6">
<div className="flex items-start gap-2.5 rounded-md bg-muted/50 px-3 py-2.5">
<Smartphone className="size-4 mt-0.5 shrink-0 text-muted-foreground" />
<p className="text-xs text-muted-foreground">
Chat with Rowboat from your phone. Send <span className="font-mono">help</span> for
commands (<span className="font-mono">list</span>, <span className="font-mono">resume 2</span>,{" "}
<span className="font-mono">new</span>, <span className="font-mono">stop</span>) anything
else is a message to the current chat. Your computer must be on with Rowboat running.
</p>
</div>
{/* WhatsApp */}
<div className="space-y-3">
<div className="flex items-center justify-between">
<div className="flex items-center gap-2.5">
<div className="flex size-8 items-center justify-center rounded-md bg-muted">
<MessageCircle className="size-4" />
</div>
<div className="flex flex-col">
<span className="text-sm font-medium">WhatsApp</span>
<span className="text-xs text-muted-foreground">
{wa?.state === "connected"
? `Linked as +${wa.self ?? "?"} — message yourself to use it`
: wa?.state === "qr"
? "Scan the QR below with your phone"
: wa?.state === "starting"
? "Connecting…"
: wa?.state === "error"
? wa.error ?? "Error"
: "Links your own WhatsApp as a companion device"}
</span>
</div>
</div>
<div className="flex items-center gap-2">
{wa?.state === "connected" && (
<Button
variant="outline"
size="sm"
className="h-7 px-3 text-xs"
onClick={() => {
void window.ipc.invoke("channels:whatsappLogout", null).catch(() => {
toast.error("Failed to unlink WhatsApp")
})
}}
>
Unlink
</Button>
)}
<Switch
checked={config.whatsapp.enabled}
disabled={saving}
onCheckedChange={(enabled) =>
void save({ ...config, whatsapp: { ...config.whatsapp, enabled } })
}
/>
</div>
</div>
{config.whatsapp.enabled && wa?.state === "qr" && wa.qrDataUrl && (
<div className="flex items-center gap-4 rounded-md border p-3">
<img src={wa.qrDataUrl} alt="WhatsApp pairing QR" className="size-40 rounded" />
<div className="text-xs text-muted-foreground space-y-1">
<p className="font-medium text-foreground">Link Rowboat to WhatsApp</p>
<p>1. Open WhatsApp on your phone</p>
<p>2. Settings Linked Devices Link a Device</p>
<p>3. Scan this code</p>
<p className="pt-1">
Then message <span className="font-medium">yourself</span> (your own contact) to talk
to Rowboat.
</p>
</div>
</div>
)}
{config.whatsapp.enabled && (
<div className="space-y-1.5">
<label className="text-xs font-medium">Additional allowed numbers</label>
<div className="flex gap-2">
<Input
value={waAllowDraft}
onChange={(e) => setWaAllowDraft(e.target.value)}
placeholder="e.g. 14155551234, 919876543210"
className="h-8 text-xs"
/>
<Button
variant="outline"
size="sm"
className="h-8 px-3 text-xs"
disabled={saving}
onClick={() =>
void save({
...config,
whatsapp: { ...config.whatsapp, allowFrom: parseIdList(waAllowDraft) },
})
}
>
Save
</Button>
</div>
<p className="text-xs text-muted-foreground">
Your own number (self-chat) is always allowed. Digits only, with country code.
</p>
</div>
)}
</div>
<Separator />
{/* Telegram */}
<div className="space-y-3">
<div className="flex items-center justify-between">
<div className="flex items-center gap-2.5">
<div className="flex size-8 items-center justify-center rounded-md bg-muted">
<Send className="size-4" />
</div>
<div className="flex flex-col">
<span className="text-sm font-medium">Telegram</span>
<span className="text-xs text-muted-foreground">
{tg?.state === "polling"
? `Listening${tg.botUsername ? ` as @${tg.botUsername}` : ""}`
: tg?.state === "starting"
? "Connecting…"
: tg?.state === "error"
? tg.error ?? "Error"
: "Uses your own bot — create one with @BotFather"}
</span>
</div>
</div>
<Switch
checked={config.telegram.enabled}
disabled={saving}
onCheckedChange={(enabled) =>
void save({ ...config, telegram: { ...config.telegram, enabled } })
}
/>
</div>
{config.telegram.enabled && (
<>
<div className="space-y-1.5">
<label className="text-xs font-medium">Bot token</label>
<div className="flex gap-2">
<Input
type="password"
value={tokenDraft}
onChange={(e) => setTokenDraft(e.target.value)}
placeholder="123456789:AAF…"
className="h-8 text-xs font-mono"
/>
<Button
variant="outline"
size="sm"
className="h-8 px-3 text-xs"
disabled={saving}
onClick={() =>
void save({
...config,
telegram: { ...config.telegram, botToken: tokenDraft.trim() },
})
}
>
Save
</Button>
</div>
<p className="text-xs text-muted-foreground">
Message @BotFather on Telegram /newbot paste the token here.
</p>
</div>
<div className="space-y-1.5">
<label className="text-xs font-medium">Allowed chat IDs</label>
<div className="flex gap-2">
<Input
value={tgAllowDraft}
onChange={(e) => setTgAllowDraft(e.target.value)}
placeholder="e.g. 123456789"
className="h-8 text-xs"
/>
<Button
variant="outline"
size="sm"
className="h-8 px-3 text-xs"
disabled={saving}
onClick={() =>
void save({
...config,
telegram: { ...config.telegram, allowFrom: parseIdList(tgAllowDraft) },
})
}
>
Save
</Button>
</div>
<p className="text-xs text-muted-foreground">
Message your bot once it replies with your chat ID to add here.
</p>
</div>
</>
)}
</div>
</div>
)
}

View file

@ -60,6 +60,7 @@ import {
} from "@/components/ui/tooltip"
import { cn } from "@/lib/utils"
import { SettingsDialog } from "@/components/settings-dialog"
import { MascotFaceIcon } from "@/components/talking-head"
import { extractConferenceLink } from "@/lib/calendar-event"
import { useBilling } from "@/hooks/useBilling"
import { toast } from "@/lib/toast"
@ -172,6 +173,8 @@ type SidebarContentPanelProps = {
onNewChat?: () => void
onToggleBrowser?: () => void
onVoiceNoteCreated?: (path: string) => void
/** Starts the mascot-guided product tour. */
onStartTour?: () => void
/** Which primary destination is currently active, for nav highlighting. */
activeNav?: 'home' | 'email' | 'meetings' | 'code' | 'knowledge' | 'agents' | 'apps' | 'workspaces' | null
/** Live meeting recording state, so the recording row can show its indicator/stop. */
@ -421,6 +424,7 @@ export function SidebarContentPanel({
onNewChat,
onToggleBrowser,
onVoiceNoteCreated,
onStartTour,
activeNav,
meetingRecordingState = 'idle',
recordingMeetingSource = null,
@ -698,13 +702,14 @@ export function SidebarContentPanel({
<SidebarGroupContent>
<SidebarMenu>
<SidebarMenuItem>
<SidebarMenuButton isActive={activeNav === 'home'} onClick={onOpenHome}>
<SidebarMenuButton data-tour-id="nav-home" isActive={activeNav === 'home'} onClick={onOpenHome}>
<Home className="size-4 shrink-0" />
<span className="flex-1 truncate">Home</span>
</SidebarMenuButton>
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-email"
isActive={activeNav === 'email'}
onClick={() => onOpenEmail?.()}
className={previewEmail ? 'h-auto py-1.5' : undefined}
@ -727,6 +732,7 @@ export function SidebarContentPanel({
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-meetings"
isActive={activeNav === 'meetings'}
onClick={onOpenMeetings}
className={meetingSublabel ? 'h-auto py-1.5' : undefined}
@ -805,7 +811,7 @@ export function SidebarContentPanel({
</SidebarMenuItem>
{codeModeEnabled && (
<SidebarMenuItem>
<SidebarMenuButton isActive={activeNav === 'code'} onClick={onOpenCode}>
<SidebarMenuButton data-tour-id="nav-code" isActive={activeNav === 'code'} onClick={onOpenCode}>
<Code2 className="size-4 shrink-0" />
<span className="flex-1 truncate">Code</span>
</SidebarMenuButton>
@ -813,6 +819,7 @@ export function SidebarContentPanel({
)}
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-knowledge"
isActive={activeNav === 'knowledge'}
onClick={() => knowledgeActions.openKnowledgeView()}
className={knowledgeUpdatedLabel ? 'h-auto py-1.5' : undefined}
@ -833,6 +840,7 @@ export function SidebarContentPanel({
<SidebarMenu>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-agents"
isActive={activeNav === 'agents'}
onClick={onOpenBgTasks}
className={bgAgentsLabel ? 'h-auto py-1.5' : undefined}
@ -857,11 +865,12 @@ export function SidebarContentPanel({
onClick={onOpenApps}
>
<LayoutGrid className="size-4 shrink-0 text-muted-foreground" />
<span className="flex-1 truncate text-muted-foreground">Mini Apps</span>
<span className="flex-1 truncate text-muted-foreground">Apps</span>
</SidebarMenuButton>
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-workspaces"
isActive={activeNav === 'workspaces'}
onClick={() => knowledgeActions.openWorkspaceAt()}
className="h-auto py-1.5"
@ -886,6 +895,7 @@ export function SidebarContentPanel({
<SidebarGroupContent>
<button
type="button"
data-tour-id="nav-chats"
onClick={() => setChatsExpanded((v) => !v)}
className="flex w-full items-center gap-1.5 px-3 py-1 text-[10.5px] font-semibold uppercase tracking-wider text-muted-foreground"
>
@ -1027,6 +1037,15 @@ export function SidebarContentPanel({
</AlertDialog>
)}
</div>
{onStartTour && (
<button
onClick={onStartTour}
className="flex w-full items-center gap-2 rounded-md px-2 py-1 text-xs text-sidebar-foreground/70 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground transition-colors"
>
<MascotFaceIcon className="size-4" />
<span>Take a tour</span>
</button>
)}
<SettingsDialog>
<button className="flex w-full items-center gap-2 rounded-md px-2 py-1 text-xs text-sidebar-foreground/70 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground transition-colors">
<Settings className="size-4" />

View file

@ -0,0 +1,487 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { X } from 'lucide-react'
import type { TTSState } from '@/hooks/useVoiceTTS'
import { cn } from '@/lib/utils'
const POSITION_STORAGE_KEY = 'talking-head-position'
// Must match the overlay's `bottom-28 right-8` anchor classes.
const ANCHOR_RIGHT_PX = 32
const ANCHOR_BOTTOM_PX = 112
const VIEWPORT_MARGIN_PX = 8
// Palette pulled from the mascot artwork: pale lavender body, dark walnut boat.
const BODY_FILL = '#E8E9F5'
const BODY_STROKE = '#17171B'
const CHEEK_FILL = '#F2B8BE'
const BOAT_DARK = '#3E2E24'
const BOAT_MID = '#54402F'
const BOAT_LIGHT = '#6B5138'
const MOUTH_FILL = '#2A1E19'
export type MascotHat =
| 'mailcap'
| 'headphones'
| 'hardhat'
| 'gradcap'
| 'captain'
| 'explorer'
| 'party'
type TalkingHeadProps = {
ttsState: TTSState
getLevel: () => number
size?: number
/** Costume piece drawn on the head (used by the product tour). */
hat?: MascotHat
/** Paddle hard: fast bobbing + oar strokes, e.g. while traveling in the tour. */
rowing?: boolean
}
/**
* The Rowboat mascot as an animated inline SVG: a round pale character sitting
* in a wooden rowboat holding an oar. The mouth is driven every animation
* frame from the live TTS audio level; eyes blink on a randomized timer.
*/
export function TalkingHead({ ttsState, getLevel, size = 160, hat, rowing = false }: TalkingHeadProps) {
const mouthOpenRef = useRef<SVGEllipseElement>(null)
const mouthSmileRef = useRef<SVGPathElement>(null)
const oarRef = useRef<SVGGElement>(null)
const smoothedRef = useRef(0)
const [blinking, setBlinking] = useState(false)
const speaking = ttsState === 'speaking'
const thinking = ttsState === 'synthesizing'
// Lip sync + oar paddle loop. Writes SVG attributes directly to avoid
// re-rendering React at 60fps. Stops itself once speech has ended and the
// mouth has settled closed; restarts when `speaking` flips this effect.
useEffect(() => {
let raf = 0
let t = 0
const tick = () => {
const target = speaking ? getLevel() : 0
const prev = smoothedRef.current
// Fast attack, slower decay reads as natural mouth movement
const smoothed = target > prev ? prev + (target - prev) * 0.5 : prev + (target - prev) * 0.2
const settled = !speaking && !rowing && smoothed < 0.005
smoothedRef.current = settled ? 0 : smoothed
const open = settled ? 0 : Math.min(1, smoothed * 1.6)
const mouthOpen = mouthOpenRef.current
const mouthSmile = mouthSmileRef.current
if (mouthOpen && mouthSmile) {
if (open > 0.06) {
mouthOpen.setAttribute('rx', String(6.5 + open * 4))
mouthOpen.setAttribute('ry', String(1.5 + open * 9))
mouthOpen.style.opacity = '1'
mouthSmile.style.opacity = '0'
} else {
mouthOpen.style.opacity = '0'
mouthSmile.style.opacity = '1'
}
}
const oar = oarRef.current
if (oar) {
if (rowing) {
t += 0.14
const angle = Math.sin(t) * 13
oar.setAttribute('transform', `rotate(${angle.toFixed(2)} 128 118)`)
} else if (speaking) {
t += 0.045
const angle = Math.sin(t) * 7
oar.setAttribute('transform', `rotate(${angle.toFixed(2)} 128 118)`)
} else {
oar.setAttribute('transform', 'rotate(0 128 118)')
}
}
if (!settled) {
raf = requestAnimationFrame(tick)
}
}
raf = requestAnimationFrame(tick)
return () => cancelAnimationFrame(raf)
}, [speaking, rowing, getLevel])
// Randomized blinking
useEffect(() => {
let timeout: ReturnType<typeof setTimeout>
let cancelled = false
const scheduleBlink = () => {
timeout = setTimeout(() => {
if (cancelled) return
setBlinking(true)
setTimeout(() => {
if (cancelled) return
setBlinking(false)
scheduleBlink()
}, 140)
}, 2400 + Math.random() * 2600)
}
scheduleBlink()
return () => {
cancelled = true
clearTimeout(timeout)
}
}, [])
return (
<div
className="talking-head-bob relative select-none"
style={{
width: size,
height: size,
animationDuration: rowing ? '0.8s' : speaking ? '1.6s' : '3.2s',
}}
>
<style>{`
@keyframes talking-head-bob {
0%, 100% { transform: translateY(0) rotate(-1.6deg); }
50% { transform: translateY(-4px) rotate(1.6deg); }
}
.talking-head-bob {
animation-name: talking-head-bob;
animation-timing-function: ease-in-out;
animation-iteration-count: infinite;
}
@keyframes talking-head-ripple {
0% { transform: scale(0.6); opacity: 0.5; }
100% { transform: scale(1.25); opacity: 0; }
}
.talking-head-ripple {
transform-origin: center;
transform-box: fill-box;
animation: talking-head-ripple 2.6s ease-out infinite;
}
@keyframes talking-head-bubble {
0%, 100% { opacity: 0.25; transform: translateY(0); }
50% { opacity: 1; transform: translateY(-2px); }
}
.talking-head-bubble {
animation: talking-head-bubble 1.2s ease-in-out infinite;
}
`}</style>
<svg viewBox="0 0 200 190" width={size} height={size} aria-hidden="true">
{/* water ripples under the boat */}
<g>
<ellipse className="talking-head-ripple" cx="100" cy="168" rx="62" ry="9" fill="none" stroke="#8FB6D9" strokeWidth="2" style={{ animationDelay: '0s' }} />
<ellipse className="talking-head-ripple" cx="100" cy="168" rx="62" ry="9" fill="none" stroke="#8FB6D9" strokeWidth="2" style={{ animationDelay: '1.3s' }} />
<ellipse cx="100" cy="168" rx="52" ry="7" fill="#8FB6D9" opacity="0.18" />
</g>
{/* thinking bubbles while synthesizing */}
{thinking && (
<g fill={BODY_STROKE} opacity="0.75">
<circle className="talking-head-bubble" cx="146" cy="34" r="3" style={{ animationDelay: '0s' }} />
<circle className="talking-head-bubble" cx="157" cy="26" r="4.2" style={{ animationDelay: '0.2s' }} />
<circle className="talking-head-bubble" cx="170" cy="16" r="5.4" style={{ animationDelay: '0.4s' }} />
</g>
)}
{/* character: head + body blob */}
<g>
<path
d="M 100 22
C 129 22 148 43 148 68
C 148 82 141 93 131 100
C 141 107 147 117 148 128
L 52 128
C 53 115 60 105 69 99
C 59 92 52 81 52 68
C 52 43 71 22 100 22 Z"
fill={BODY_FILL}
stroke={BODY_STROKE}
strokeWidth="5"
strokeLinejoin="round"
/>
{/* eyes */}
<g style={{ transform: thinking ? 'translateY(-2.5px)' : undefined, transition: 'transform 0.3s' }}>
<ellipse
cx="84" cy="64" rx="5" ry={blinking ? 0.8 : 7}
fill={BODY_STROKE}
style={{ transition: 'ry 0.06s' }}
/>
<ellipse
cx="116" cy="64" rx="5" ry={blinking ? 0.8 : 7}
fill={BODY_STROKE}
style={{ transition: 'ry 0.06s' }}
/>
<circle cx="86" cy="61" r="1.6" fill="#FFFFFF" opacity={blinking ? 0 : 0.9} />
<circle cx="118" cy="61" r="1.6" fill="#FFFFFF" opacity={blinking ? 0 : 0.9} />
</g>
{/* cheeks */}
<ellipse cx="72" cy="76" rx="6.5" ry="4" fill={CHEEK_FILL} opacity="0.85" />
<ellipse cx="128" cy="76" rx="6.5" ry="4" fill={CHEEK_FILL} opacity="0.85" />
{/* mouth: smile when quiet, open ellipse driven by audio level */}
<path
ref={mouthSmileRef}
d="M 91 80 Q 100 88 109 80"
fill="none"
stroke={BODY_STROKE}
strokeWidth="4"
strokeLinecap="round"
/>
<ellipse
ref={mouthOpenRef}
cx="100" cy="84" rx="7" ry="2"
fill={MOUTH_FILL}
stroke={BODY_STROKE}
strokeWidth="3"
style={{ opacity: 0 }}
/>
{hat && <MascotHatArt hat={hat} />}
</g>
{/* oar (rotates while speaking) */}
<g ref={oarRef}>
<line x1="158" y1="88" x2="88" y2="152" stroke={BODY_STROKE} strokeWidth="12" strokeLinecap="round" />
<line x1="158" y1="88" x2="88" y2="152" stroke={BOAT_MID} strokeWidth="7" strokeLinecap="round" />
<path
d="M 84 148 L 56 170 C 52 173 52 178 57 178 L 90 165 Z"
fill={BOAT_DARK}
stroke={BODY_STROKE}
strokeWidth="4"
strokeLinejoin="round"
/>
</g>
{/* hand resting over the oar */}
<ellipse cx="121" cy="120" rx="10" ry="8" fill={BODY_FILL} stroke={BODY_STROKE} strokeWidth="4" />
{/* boat hull (drawn last so it overlaps the body) */}
<g>
<path
d="M 30 120
C 50 132 150 132 170 120
C 168 142 152 160 100 160
C 48 160 32 142 30 120 Z"
fill={BOAT_MID}
stroke={BODY_STROKE}
strokeWidth="5"
strokeLinejoin="round"
/>
{/* plank lines */}
<path d="M 36 133 C 60 143 140 143 164 133" fill="none" stroke={BOAT_DARK} strokeWidth="3" strokeLinecap="round" />
<path d="M 44 145 C 66 153 134 153 156 145" fill="none" stroke={BOAT_DARK} strokeWidth="3" strokeLinecap="round" />
{/* gunwale highlight */}
<path d="M 33 121 C 52 131 148 131 167 121" fill="none" stroke={BOAT_LIGHT} strokeWidth="4" strokeLinecap="round" />
</g>
</svg>
</div>
)
}
type TalkingHeadOverlayProps = {
ttsState: TTSState
getLevel: () => number
onDismiss?: () => void
}
// Keep the widget fully on-screen relative to its bottom-right CSS anchor.
// Falls back to the default render size when the element isn't mounted yet.
function clampPositionToViewport(pos: { x: number; y: number }, el: HTMLDivElement | null): { x: number; y: number } {
const width = el?.offsetWidth ?? 160
const height = el?.offsetHeight ?? 160
const baseLeft = window.innerWidth - ANCHOR_RIGHT_PX - width
const baseTop = window.innerHeight - ANCHOR_BOTTOM_PX - height
const clamp = (v: number, min: number, max: number) => Math.max(min, Math.min(max, v))
return {
x: clamp(pos.x, VIEWPORT_MARGIN_PX - baseLeft, window.innerWidth - VIEWPORT_MARGIN_PX - width - baseLeft),
y: clamp(pos.y, VIEWPORT_MARGIN_PX - baseTop, window.innerHeight - VIEWPORT_MARGIN_PX - height - baseTop),
}
}
function loadStoredPosition(): { x: number; y: number } {
try {
const raw = localStorage.getItem(POSITION_STORAGE_KEY)
if (raw) {
const parsed = JSON.parse(raw)
if (typeof parsed?.x === 'number' && typeof parsed?.y === 'number') {
return parsed
}
}
} catch {
// ignore corrupt stored position
}
return { x: 0, y: 0 }
}
/**
* Floating, draggable widget that hosts the talking head. Anchored to the
* bottom-right of the window (above the composer) and offset by a persisted
* drag position, so it hovers over whatever view is active.
*/
export function TalkingHeadOverlay({ ttsState, getLevel, onDismiss }: TalkingHeadOverlayProps) {
// Clamp the stored offset at init so a stale position (e.g. saved on a
// bigger window) can't leave the widget stranded off-screen.
const [offset, setOffset] = useState(() => clampPositionToViewport(loadStoredPosition(), null))
const [dragging, setDragging] = useState(false)
const dragStartRef = useRef<{ pointerX: number; pointerY: number; x: number; y: number } | null>(null)
const containerRef = useRef<HTMLDivElement>(null)
const clampToViewport = useCallback(
(pos: { x: number; y: number }) => clampPositionToViewport(pos, containerRef.current),
[]
)
// Re-clamp when the window shrinks
useEffect(() => {
const handleResize = () => setOffset(prev => clampToViewport(prev))
window.addEventListener('resize', handleResize)
return () => window.removeEventListener('resize', handleResize)
}, [clampToViewport])
const handlePointerDown = useCallback((e: React.PointerEvent<HTMLDivElement>) => {
if (e.button !== 0) return
dragStartRef.current = { pointerX: e.clientX, pointerY: e.clientY, x: offset.x, y: offset.y }
setDragging(true)
e.currentTarget.setPointerCapture(e.pointerId)
}, [offset])
const handlePointerMove = useCallback((e: React.PointerEvent<HTMLDivElement>) => {
const start = dragStartRef.current
if (!start) return
setOffset({
x: start.x + (e.clientX - start.pointerX),
y: start.y + (e.clientY - start.pointerY),
})
}, [])
const handlePointerUp = useCallback((e: React.PointerEvent<HTMLDivElement>) => {
if (!dragStartRef.current) return
dragStartRef.current = null
setDragging(false)
setOffset(prev => clampToViewport(prev))
e.currentTarget.releasePointerCapture(e.pointerId)
}, [clampToViewport])
useEffect(() => {
if (dragging) return
try {
localStorage.setItem(POSITION_STORAGE_KEY, JSON.stringify(offset))
} catch {
// best-effort persistence
}
}, [offset, dragging])
return (
<div
ref={containerRef}
className={cn(
'group fixed bottom-28 right-8 z-50 touch-none',
dragging ? 'cursor-grabbing' : 'cursor-grab'
)}
style={{
transform: `translate(${offset.x}px, ${offset.y}px)`,
// Constant value so the entrance animation runs once on mount and
// never restarts (re-applying it after a drag would replay the pop).
animation: 'talking-head-pop 0.35s cubic-bezier(0.34, 1.56, 0.64, 1)',
}}
onPointerDown={handlePointerDown}
onPointerMove={handlePointerMove}
onPointerUp={handlePointerUp}
onPointerCancel={handlePointerUp}
role="img"
aria-label="Rowboat talking head"
>
<style>{`
@keyframes talking-head-pop {
0% { opacity: 0; scale: 0.4; }
100% { opacity: 1; scale: 1; }
}
`}</style>
{onDismiss && (
<button
type="button"
onPointerDown={(e) => e.stopPropagation()}
onClick={onDismiss}
className="absolute -right-1 -top-1 z-10 flex h-5 w-5 items-center justify-center rounded-full border border-border bg-background text-muted-foreground opacity-0 shadow-sm transition-opacity hover:text-foreground group-hover:opacity-100"
aria-label="Hide talking head"
>
<X className="h-3 w-3" />
</button>
)}
<TalkingHead ttsState={ttsState} getLevel={getLevel} />
</div>
)
}
/** Costume pieces for the product tour, drawn on the mascot's head. */
function MascotHatArt({ hat }: { hat: MascotHat }) {
const outline = { stroke: BODY_STROKE, strokeWidth: 4, strokeLinejoin: 'round' as const }
switch (hat) {
case 'mailcap':
return (
<g>
<path d="M 68 36 Q 100 4 132 36 Q 100 26 68 36 Z" fill="#4A7DDB" {...outline} />
<path d="M 126 33 Q 148 31 150 39 Q 132 42 124 39 Z" fill="#3D68B8" {...outline} />
</g>
)
case 'headphones':
return (
<g>
<path d="M 64 58 Q 100 2 136 58" fill="none" stroke={BODY_STROKE} strokeWidth="11" strokeLinecap="round" />
<path d="M 64 58 Q 100 2 136 58" fill="none" stroke="#4B5563" strokeWidth="6" strokeLinecap="round" />
<ellipse cx="64" cy="61" rx="8" ry="11" fill="#4B5563" {...outline} />
<ellipse cx="136" cy="61" rx="8" ry="11" fill="#4B5563" {...outline} />
</g>
)
case 'hardhat':
return (
<g>
<path d="M 66 38 Q 100 0 134 38 Z" fill="#F6C445" {...outline} />
<path d="M 96 10 Q 94 22 96 36 L 106 36 Q 107 22 105 9 Z" fill="#E0AC2C" stroke="none" />
<path d="M 56 38 L 144 38" fill="none" stroke={BODY_STROKE} strokeWidth="9" strokeLinecap="round" />
<path d="M 58 38 L 142 38" fill="none" stroke="#F6C445" strokeWidth="5" strokeLinecap="round" />
</g>
)
case 'gradcap':
return (
<g>
<path d="M 80 28 Q 100 42 120 28 L 120 38 Q 100 50 80 38 Z" fill="#232838" {...outline} />
<path d="M 100 2 L 144 20 L 100 38 L 56 20 Z" fill="#2E3450" {...outline} />
<path d="M 100 20 Q 124 26 136 34" fill="none" stroke="#E8B94A" strokeWidth="3" strokeLinecap="round" />
<circle cx="137" cy="38" r="4" fill="#E8B94A" stroke={BODY_STROKE} strokeWidth="3" />
</g>
)
case 'captain':
return (
<g>
<path d="M 68 36 Q 100 -2 132 36 Q 100 28 68 36 Z" fill="#F4F5F9" {...outline} />
<path d="M 66 36 Q 100 46 134 36 L 134 42 Q 100 52 66 42 Z" fill="#22262E" {...outline} />
<path d="M 122 42 Q 146 42 148 48 Q 130 52 120 48 Z" fill="#22262E" {...outline} />
<circle cx="100" cy="38" r="4.5" fill="#E8B94A" stroke={BODY_STROKE} strokeWidth="3" />
</g>
)
case 'explorer':
return (
<g>
<ellipse cx="100" cy="35" rx="46" ry="9" fill="#C9A46B" {...outline} />
<path d="M 72 34 Q 100 4 128 34 Z" fill="#C9A46B" {...outline} />
<path d="M 76 30 Q 100 38 124 30" fill="none" stroke="#8A6B3D" strokeWidth="4" strokeLinecap="round" />
</g>
)
case 'party':
return (
<g>
<path d="M 100 -2 L 84 34 L 116 34 Z" fill="#F2699C" {...outline} />
<path d="M 92 16 L 111 22 M 88 26 L 114 31" fill="none" stroke="#FFD166" strokeWidth="3.5" strokeLinecap="round" />
<circle cx="100" cy="0" r="5" fill="#FFD166" stroke={BODY_STROKE} strokeWidth="3" />
</g>
)
}
}
/** Small static mascot face used as the toolbar toggle icon. */
export function MascotFaceIcon({ className }: { className?: string }) {
return (
<svg viewBox="0 0 24 24" width="16" height="16" className={className} aria-hidden="true">
<circle cx="12" cy="12" r="9.5" fill="none" stroke="currentColor" strokeWidth="1.8" />
<ellipse cx="8.6" cy="10.5" rx="1.3" ry="1.8" fill="currentColor" />
<ellipse cx="15.4" cy="10.5" rx="1.3" ry="1.8" fill="currentColor" />
<path d="M 9 14.5 Q 12 17 15 14.5" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" />
</svg>
)
}

View file

@ -0,0 +1,262 @@
import { useEffect } from 'react'
export type MascotVignetteKind = 'email' | 'meetings' | 'brain'
export type TourVignetteKind = MascotVignetteKind | 'agents'
/**
* Little looping "shows" staged around the mascot while it presents a section
* during the product tour. Purely decorative: everything is pointer-events-none
* and rendered on the tour's own layers, never inside the section's real UI.
*/
export function MascotVignette({ kind, playDing }: { kind: MascotVignetteKind; playDing?: () => void }) {
// One round of dings as the first envelopes land, then let the loop run silent
useEffect(() => {
if (kind !== 'email' || !playDing) return
const timers = [900, 1900, 2900].map((ms) => setTimeout(playDing, ms))
return () => timers.forEach(clearTimeout)
}, [kind, playDing])
return (
<div className="pointer-events-none absolute left-1/2 top-0 z-0 -translate-x-1/2" aria-hidden="true">
<style>{`
@keyframes tour-env-left {
0% { opacity: 0; transform: translate(-150px, -130px) rotate(-26deg); }
10% { opacity: 1; }
52% { opacity: 1; transform: translate(0px, 0px) rotate(5deg); }
64% { opacity: 0; transform: translate(2px, 12px) rotate(5deg); }
100% { opacity: 0; transform: translate(2px, 12px) rotate(5deg); }
}
@keyframes tour-env-right {
0% { opacity: 0; transform: translate(150px, -140px) rotate(26deg); }
10% { opacity: 1; }
52% { opacity: 1; transform: translate(0px, 0px) rotate(-5deg); }
64% { opacity: 0; transform: translate(-2px, 12px) rotate(-5deg); }
100% { opacity: 0; transform: translate(-2px, 12px) rotate(-5deg); }
}
@keyframes tour-badge-pulse {
0%, 100% { transform: scale(1); }
50% { transform: scale(1.15); }
}
@keyframes tour-note-line {
0% { width: 0; }
18% { width: var(--w); }
80% { width: var(--w); opacity: 1; }
92%, 100% { width: var(--w); opacity: 0; }
}
@keyframes tour-quill-bob {
0% { transform: translate(4px, 26px) rotate(-8deg); }
25% { transform: translate(46px, 30px) rotate(4deg); }
50% { transform: translate(6px, 40px) rotate(-8deg); }
75% { transform: translate(48px, 44px) rotate(4deg); }
100% { transform: translate(4px, 26px) rotate(-8deg); }
}
@keyframes tour-voice-bar {
0%, 100% { transform: scaleY(0.35); }
50% { transform: scaleY(1); }
}
@keyframes tour-orb {
0% { opacity: 0; transform: translate(var(--from-x), var(--from-y)) scale(0.5); }
15% { opacity: 0.9; }
70% { opacity: 0.9; transform: translate(0, 0) scale(1); }
85%, 100% { opacity: 0; transform: translate(0, 6px) scale(0.3); }
}
@keyframes tour-node-pulse {
0%, 100% { opacity: 0.55; }
50% { opacity: 1; }
}
@keyframes tour-edge-draw {
from { stroke-dashoffset: 60; }
to { stroke-dashoffset: 0; }
}
`}</style>
{kind === 'email' && (
<div className="relative" style={{ width: 240, height: 150 }}>
{[
{ anim: 'tour-env-left', delay: 0, x: 78, y: 96 },
{ anim: 'tour-env-right', delay: 1.0, x: 128, y: 102 },
{ anim: 'tour-env-left', delay: 2.0, x: 104, y: 92 },
{ anim: 'tour-env-right', delay: 3.0, x: 90, y: 104 },
].map((e, i) => (
<div
key={i}
className="absolute"
style={{ left: e.x, top: e.y, animation: `${e.anim} 4s ease-in-out ${e.delay}s infinite both` }}
>
<svg width="34" height="24" viewBox="0 0 34 24">
<rect x="1.5" y="1.5" width="31" height="21" rx="3" fill="#FFF8E7" stroke="#17171B" strokeWidth="2.5" />
<path d="M 2 4 L 17 14 L 32 4" fill="none" stroke="#17171B" strokeWidth="2.5" strokeLinejoin="round" />
</svg>
</div>
))}
<div
className="absolute"
style={{ left: 148, top: 78, animation: 'tour-badge-pulse 2s ease-in-out infinite' }}
>
<svg width="22" height="22" viewBox="0 0 22 22">
<circle cx="11" cy="11" r="9.5" fill="#3FA95C" stroke="#17171B" strokeWidth="2.5" />
<path d="M 6.5 11.5 L 9.5 14.5 L 15.5 8" fill="none" stroke="#FFFFFF" strokeWidth="2.5" strokeLinecap="round" strokeLinejoin="round" />
</svg>
</div>
</div>
)}
{kind === 'meetings' && (
<div className="relative" style={{ width: 220, height: 160, top: -110 }}>
{/* someone's talking */}
<div className="absolute left-1/2 flex -translate-x-1/2 items-end gap-1" style={{ top: 0, height: 26 }}>
{[0.9, 1.1, 0.7, 1.3, 0.8].map((dur, i) => (
<div
key={i}
className="w-1.5 origin-bottom rounded-full bg-[#8FB6D9]"
style={{ height: 22, animation: `tour-voice-bar ${dur}s ease-in-out ${i * 0.12}s infinite` }}
/>
))}
</div>
{/* ...and the notepad writes itself */}
<div
className="absolute left-1/2 rounded-md border-2 border-[#17171B] bg-[#FFFDF6] shadow-md"
style={{ top: 36, width: 96, height: 104, transform: 'translateX(-50%) rotate(-3deg)' }}
>
<div className="mx-2 mt-2 h-1.5 rounded bg-[#D9534F]/70" />
{[64, 52, 60, 44].map((w, i) => (
<div
key={i}
className="ml-2 mt-3 h-1.5 rounded bg-[#9AA1AE]"
style={{ ['--w' as string]: `${w}px`, animation: `tour-note-line 5s ease-out ${i * 1.1}s infinite both` }}
/>
))}
<svg
className="absolute left-0 top-0"
width="26"
height="26"
viewBox="0 0 26 26"
style={{ animation: 'tour-quill-bob 5s ease-in-out infinite' }}
>
<path d="M 4 22 L 10 12 Q 14 4 22 2 Q 18 10 12 14 Z" fill="#6B5138" stroke="#17171B" strokeWidth="2" strokeLinejoin="round" />
</svg>
</div>
</div>
)}
{kind === 'brain' && (
<div className="relative" style={{ width: 260, height: 180, top: -128 }}>
{/* constellation assembling above the head */}
<svg className="absolute left-1/2 -translate-x-1/2" width="140" height="80" viewBox="0 0 140 80" style={{ top: 0 }}>
{[
'M 22 58 L 52 24',
'M 52 24 L 88 40',
'M 88 40 L 120 18',
'M 88 40 L 70 66',
].map((d, i) => (
<path
key={i}
d={d}
fill="none"
stroke="#9CCBEA"
strokeWidth="2"
strokeDasharray="60"
style={{ animation: `tour-edge-draw 1.2s ease-out ${0.4 + i * 0.35}s both` }}
/>
))}
{[
[22, 58], [52, 24], [88, 40], [120, 18], [70, 66],
].map(([cx, cy], i) => (
<circle
key={i}
cx={cx}
cy={cy}
r={5}
fill="#BFE0F5"
stroke="#17171B"
strokeWidth="2"
style={{ animation: `tour-node-pulse 2.4s ease-in-out ${i * 0.3}s infinite` }}
/>
))}
</svg>
{/* thought-orbs drifting into the head */}
{[
{ fx: '-120px', fy: '-30px', delay: 0 },
{ fx: '120px', fy: '-40px', delay: 1.1 },
{ fx: '-90px', fy: '50px', delay: 2.2 },
{ fx: '110px', fy: '46px', delay: 3.3 },
].map((o, i) => (
<div
key={i}
className="absolute rounded-full"
style={{
left: 122,
top: 118,
width: 16,
height: 16,
background: 'radial-gradient(circle, #FFF3B8 20%, #FFD166 60%, transparent 75%)',
['--from-x' as string]: o.fx,
['--from-y' as string]: o.fy,
animation: `tour-orb 4.4s ease-in-out ${o.delay}s infinite both`,
}}
/>
))}
</div>
)}
</div>
)
}
/**
* Background-agents vignette: a fleet of tiny mascots rowing across the water
* at the bottom of the screen while the big one takes a break.
*/
export function AgentsFleet() {
return (
<div className="pointer-events-none fixed inset-x-0 bottom-0 z-[67] h-28 overflow-hidden" aria-hidden="true">
<style>{`
@keyframes tour-fleet-right {
from { transform: translateX(-18vw); }
to { transform: translateX(112vw); }
}
@keyframes tour-fleet-left {
from { transform: translateX(112vw); }
to { transform: translateX(-18vw); }
}
@keyframes tour-fleet-bob {
0%, 100% { transform: translateY(0) rotate(-2deg); }
50% { transform: translateY(-3px) rotate(2deg); }
}
`}</style>
{[
{ size: 46, bottom: 4, dur: 16, delay: 0, dir: 'tour-fleet-right' },
{ size: 34, bottom: 22, dur: 22, delay: -8, dir: 'tour-fleet-left' },
{ size: 28, bottom: 14, dur: 27, delay: -3, dir: 'tour-fleet-right' },
].map((b, i) => (
<div
key={i}
className="absolute left-0"
style={{ bottom: b.bottom, animation: `${b.dir} ${b.dur}s linear ${b.delay}s infinite` }}
>
<div style={{ animation: 'tour-fleet-bob 1.4s ease-in-out infinite', transform: b.dir === 'tour-fleet-left' ? 'scaleX(-1)' : undefined }}>
<MiniRower size={b.size} flipped={b.dir === 'tour-fleet-left'} />
</div>
</div>
))}
</div>
)
}
function MiniRower({ size, flipped }: { size: number; flipped: boolean }) {
return (
<svg
width={size}
height={size * 0.75}
viewBox="0 0 60 45"
style={{ transform: flipped ? 'scaleX(-1)' : undefined }}
>
<circle cx="27" cy="13" r="10" fill="#E8E9F5" stroke="#17171B" strokeWidth="3" />
<circle cx="23.5" cy="11.5" r="1.4" fill="#17171B" />
<circle cx="30.5" cy="11.5" r="1.4" fill="#17171B" />
<path d="M 23 16 Q 27 19 31 16" fill="none" stroke="#17171B" strokeWidth="1.6" strokeLinecap="round" />
<line x1="40" y1="14" x2="20" y2="38" stroke="#17171B" strokeWidth="4.5" strokeLinecap="round" />
<line x1="40" y1="14" x2="20" y2="38" stroke="#54402F" strokeWidth="2.5" strokeLinecap="round" />
<path d="M 7 25 C 17 31 43 31 53 25 C 51 37 43 42 30 42 C 17 42 9 37 7 25 Z" fill="#54402F" stroke="#17171B" strokeWidth="3" strokeLinejoin="round" />
</svg>
)
}

View file

@ -0,0 +1,238 @@
import { useEffect, useRef, useState } from 'react'
import { Minimize2, MonitorUp, PhoneOff, Presentation, Square, User, Video, VideoOff } from 'lucide-react'
import { MascotFaceIcon, TalkingHead } from '@/components/talking-head'
import type { TTSState } from '@/hooks/useVoiceTTS'
import { cn } from '@/lib/utils'
export type VideoCallStatus = 'listening' | 'thinking' | 'speaking'
interface VideoCallViewProps {
/** Live camera stream from useVideoMode — attached to the user's tile. */
streamRef: React.MutableRefObject<MediaStream | null>
cameraOn: boolean
onToggleCamera: () => void
/** Starting a share collapses this view into the floating popout (the
* surface is derived from devices see App.tsx). */
onToggleScreenShare: () => void
/** Practice preset: the assistant is coaching this session. */
practiceMode?: boolean
/** Shrink to the floating pill without touching any devices. */
onMinimize: () => void
/** Stop the assistant: silence speech and abort the run if still going. */
onInterrupt: () => void
ttsState: TTSState
/** Live TTS output level — drives the mascot's mouth animation. */
getTtsLevel: () => number
status: VideoCallStatus
/** Live transcript of the user's in-progress utterance. */
interimText?: string
/** The assistant line currently being spoken aloud. */
assistantCaption?: string
onLeave: () => void
}
const STATUS_DISPLAY: Record<VideoCallStatus, { label: string; dotClass: string }> = {
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
}
/**
* Full-screen call surface: a Meet-style two-tile layout with the user's
* webcam on one side and the mascot as the other participant. Shown only
* while the camera is on with no screen share (the derived-surface rule in
* App.tsx) sharing or muting the camera moves the call into the floating
* popout. The mascot animates with the assistant's speech; dismissing it
* swaps in a Meet-style letter avatar ("R"). Live captions run along the
* bottom.
*/
export function VideoCallView({
streamRef,
cameraOn,
onToggleCamera,
onToggleScreenShare,
practiceMode,
onMinimize,
onInterrupt,
ttsState,
getTtsLevel,
status,
interimText,
assistantCaption,
onLeave,
}: VideoCallViewProps) {
const videoRef = useRef<HTMLVideoElement | null>(null)
const [mascotVisible, setMascotVisible] = useState(true)
useEffect(() => {
if (!cameraOn) return
const videoEl = videoRef.current
if (!videoEl) return
videoEl.srcObject = streamRef.current
videoEl.play().catch(() => {})
return () => {
videoEl.srcObject = null
}
}, [streamRef, cameraOn])
const userSpeaking = status === 'listening' && Boolean(interimText)
const assistantSpeaking = ttsState === 'speaking'
const caption = assistantSpeaking && assistantCaption
? { who: 'Rowboat', text: assistantCaption }
: interimText
? { who: 'You', text: interimText }
: null
return (
<div className="fixed inset-0 z-[100] flex flex-col bg-neutral-950">
{practiceMode && (
<span className="absolute left-4 top-4 z-10 flex items-center gap-1.5 rounded-full bg-violet-600/90 px-3 py-1 text-xs font-medium text-white">
<Presentation className="h-3.5 w-3.5" />
Practice session
</span>
)}
<button
type="button"
onClick={onMinimize}
className="absolute right-4 top-4 z-10 flex h-8 w-8 items-center justify-center rounded-full bg-neutral-800 text-white/80 transition-colors hover:bg-neutral-700 hover:text-white"
aria-label="Minimize call (shares your screen)"
title="Minimize — shares your screen so it can help you work"
>
<Minimize2 className="h-4 w-4" />
</button>
{/* Participant tiles */}
<div className="grid min-h-0 flex-1 grid-cols-1 gap-3 p-4 pb-2 md:grid-cols-2">
{/* User */}
<div
className={cn(
'relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
userSpeaking && 'ring-2 ring-green-500/80'
)}
>
{cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-full w-full object-cover"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<span className="flex h-40 w-40 items-center justify-center rounded-full bg-neutral-700 text-neutral-400" aria-label="Camera off">
<User className="h-20 w-20" />
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
You
</span>
</div>
{/* Assistant */}
<div
className={cn(
'group relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
assistantSpeaking && 'ring-2 ring-sky-400/80'
)}
>
{mascotVisible ? (
<TalkingHead ttsState={ttsState} getLevel={getTtsLevel} size={220} />
) : (
<span
className="flex h-40 w-40 items-center justify-center rounded-full bg-sky-600 text-7xl font-medium text-white"
aria-label="Rowboat"
>
R
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
Rowboat
</span>
{status !== 'listening' && (
<button
type="button"
onClick={onInterrupt}
className="absolute bottom-3 right-3 flex items-center gap-1.5 rounded-md bg-red-600/90 px-2.5 py-1 text-sm font-medium text-white transition-colors hover:bg-red-500"
aria-label="Stop the assistant"
title={status === 'speaking' ? 'Stop speaking' : 'Stop responding'}
>
<Square className="h-3 w-3 fill-current" />
Stop
</button>
)}
<button
type="button"
onClick={() => setMascotVisible((v) => !v)}
className="absolute right-3 top-3 rounded-md bg-black/50 px-2 py-1 text-xs text-white/80 opacity-0 transition-opacity hover:text-white group-hover:opacity-100"
>
{mascotVisible ? 'Hide mascot' : 'Show mascot'}
</button>
</div>
</div>
{/* Captions */}
<div className="flex h-14 items-center justify-center px-6">
{caption && (
<div className="max-w-3xl truncate rounded-lg bg-black/60 px-4 py-2 text-sm text-white/90">
<span className="mr-2 font-semibold text-white">{caption.who}:</span>
{caption.text}
</div>
)}
</div>
{/* Control bar */}
<div className="flex items-center justify-center gap-4 pb-5">
<span className="flex items-center gap-2 rounded-full bg-neutral-800 px-3 py-1.5 text-xs font-medium text-white/90">
<span className={cn('block h-2 w-2 rounded-full', STATUS_DISPLAY[status].dotClass)} />
{STATUS_DISPLAY[status].label}
</span>
<button
type="button"
onClick={onToggleCamera}
className={cn(
'flex h-10 w-10 items-center justify-center rounded-full transition-colors',
cameraOn
? 'bg-neutral-800 text-white/90 hover:bg-neutral-700'
: 'bg-red-600 text-white hover:bg-red-500'
)}
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
>
{cameraOn ? <Video className="h-5 w-5" /> : <VideoOff className="h-5 w-5" />}
</button>
<button
type="button"
onClick={onToggleScreenShare}
className="flex h-10 w-10 items-center justify-center rounded-full bg-neutral-800 text-white/90 transition-colors hover:bg-neutral-700"
aria-label="Present your screen"
title="Present your screen"
>
<MonitorUp className="h-5 w-5" />
</button>
<button
type="button"
onClick={() => setMascotVisible((v) => !v)}
className="relative flex h-10 w-10 items-center justify-center rounded-full bg-neutral-800 text-white/90 transition-colors hover:bg-neutral-700"
aria-label={mascotVisible ? 'Hide mascot' : 'Show mascot'}
>
<MascotFaceIcon />
{!mascotVisible && (
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
<span className="block h-[1.5px] w-6 -rotate-45 rounded-full bg-white/80" />
</span>
)}
</button>
<button
type="button"
onClick={onLeave}
className="flex h-10 w-14 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
aria-label="End call"
>
<PhoneOff className="h-5 w-5" />
</button>
</div>
</div>
)
}

View file

@ -0,0 +1,207 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { Maximize2, MonitorUp, PhoneOff, Square, User, Video, VideoOff } from 'lucide-react'
import { TalkingHead } from '@/components/talking-head'
type PopoutState = {
ttsState: 'idle' | 'synthesizing' | 'speaking'
status: 'listening' | 'thinking' | 'speaking' | null
cameraOn: boolean
screenSharing: boolean
interimText: string | null
}
const STATUS_DISPLAY: Record<NonNullable<PopoutState['status']>, { label: string; dotClass: string }> = {
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
}
const dragRegion = { WebkitAppRegion: 'drag' } as React.CSSProperties
const noDragRegion = { WebkitAppRegion: 'no-drag' } as React.CSSProperties
/**
* Content of the always-on-top popout window shown for the whole duration of
* a screen share (Meet-style floating mini-call) it floats over every app,
* including Rowboat itself, and is the call's control surface while sharing:
* camera toggle, share toggle, end-call. Rendered in its own BrowserWindow
* (see `video:setPopout` in the main process); call state streams in over
* the `video:popout-state` push channel and control actions round-trip back
* through `video:popoutAction`. Captures its own webcam feed MediaStreams
* can't cross windows.
*/
export function VideoPopout() {
// Camera defaults OFF: guessing "on" would flash the user's video for a
// beat before the real state arrives — which reads as a bug. The true
// state is fetched immediately below.
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: false, screenSharing: false, interimText: null })
const videoRef = useRef<HTMLVideoElement | null>(null)
useEffect(() => {
const cleanup = window.ipc.on('video:popout-state', (next) => setState(next))
// The main process replays the cached state on did-finish-load, but that
// can race this listener's registration — fetch it explicitly too.
window.ipc
.invoke('video:getPopoutState', null)
.then(({ state: cached }) => {
if (cached) setState(cached)
})
.catch(() => {})
return cleanup
}, [])
// Own camera feed, following the main window's camera-on/off state.
useEffect(() => {
if (!state.cameraOn) return
let stream: MediaStream | null = null
let cancelled = false
navigator.mediaDevices
.getUserMedia({ video: { width: { ideal: 640 }, facingMode: 'user' }, audio: false })
.then((s) => {
if (cancelled) {
s.getTracks().forEach((t) => t.stop())
return
}
stream = s
if (videoRef.current) {
videoRef.current.srcObject = s
videoRef.current.play().catch(() => {})
}
})
.catch((err) => console.error('[popout] camera failed:', err))
return () => {
cancelled = true
stream?.getTracks().forEach((t) => t.stop())
if (videoRef.current) videoRef.current.srcObject = null
}
}, [state.cameraOn])
// The popout has no TTS audio pipeline — synthesize a plausible mouth level
// so the mascot still animates while the assistant speaks in the main window.
const getLevel = useCallback(() => 0.45 + 0.35 * Math.sin(performance.now() / 90), [])
const sendAction = useCallback((action: 'toggle-camera' | 'toggle-share' | 'stop-speaking' | 'end-call' | 'expand') => {
void window.ipc.invoke('video:popoutAction', { action }).catch(() => {})
}, [])
const statusDisplay = state.status ? STATUS_DISPLAY[state.status] : null
return (
<div
className="relative flex h-screen w-screen select-none flex-col gap-1.5 bg-neutral-900 p-1.5"
style={dragRegion}
>
<div className="flex min-h-0 flex-1 gap-1.5">
<div className="relative flex-1 overflow-hidden rounded-lg bg-neutral-800">
{state.cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-full w-full object-cover"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<div className="flex h-full w-full items-center justify-center">
<span className="flex h-12 w-12 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
<User className="h-6 w-6" />
</span>
</div>
)}
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
You
</span>
{/* Persistent consent badge the user must always be able to see
at a glance that their screen is going out. */}
{state.screenSharing && (
<span className="absolute left-1.5 top-1.5 flex items-center gap-1 rounded-full bg-sky-600/90 px-1.5 py-0.5 text-[10px] font-medium text-white">
<span className="block h-1.5 w-1.5 animate-pulse rounded-full bg-white" />
Sharing screen
</span>
)}
</div>
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-lg bg-neutral-800">
<TalkingHead ttsState={state.ttsState} getLevel={getLevel} size={84} />
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
Rowboat
</span>
{statusDisplay && (
<span className="absolute right-1.5 top-1.5 flex items-center gap-1 rounded-full bg-black/50 px-1.5 py-0.5 text-[10px] font-medium text-white">
<span className={`block h-1.5 w-1.5 rounded-full ${statusDisplay.dotClass}`} />
{statusDisplay.label}
</span>
)}
{(state.status === 'speaking' || state.status === 'thinking') && (
<button
type="button"
onClick={() => sendAction('stop-speaking')}
className="absolute bottom-1 right-1.5 flex items-center gap-1 rounded bg-red-600/90 px-1.5 py-0.5 text-[10px] font-medium text-white hover:bg-red-500"
style={noDragRegion}
aria-label="Stop the assistant"
title={state.status === 'speaking' ? 'Stop speaking' : 'Stop responding'}
>
<Square className="h-2.5 w-2.5 fill-current" />
Stop
</button>
)}
</div>
{/* Live caption of the in-progress utterance, floating over the tiles */}
{state.interimText && (
<div className="pointer-events-none absolute inset-x-1.5 bottom-9 flex justify-center">
<span className="max-w-full truncate rounded bg-black/70 px-1.5 py-0.5 text-[10px] text-white/90">
{state.interimText}
</span>
</div>
)}
</div>
{/* Control bar — actions execute in the main app window */}
<div className="flex h-7 shrink-0 items-center justify-center gap-2" style={noDragRegion}>
<button
type="button"
onClick={() => sendAction('toggle-camera')}
className={`flex h-6 w-6 items-center justify-center rounded-full transition-colors ${
state.cameraOn
? 'bg-neutral-700 text-white/90 hover:bg-neutral-600'
: 'bg-red-600 text-white hover:bg-red-500'
}`}
aria-label={state.cameraOn ? 'Turn off camera' : 'Turn on camera'}
title={state.cameraOn ? 'Turn off camera' : 'Turn on camera'}
>
{state.cameraOn ? <Video className="h-3.5 w-3.5" /> : <VideoOff className="h-3.5 w-3.5" />}
</button>
<button
type="button"
onClick={() => sendAction('toggle-share')}
className={`flex h-6 w-6 items-center justify-center rounded-full transition-colors ${
state.screenSharing
? 'bg-sky-600 text-white hover:bg-sky-500'
: 'bg-neutral-700 text-white/90 hover:bg-neutral-600'
}`}
aria-label={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
title={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
>
<MonitorUp className="h-3.5 w-3.5" />
</button>
<button
type="button"
onClick={() => sendAction('end-call')}
className="flex h-6 w-8 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
aria-label="End call"
title="End call"
>
<PhoneOff className="h-3.5 w-3.5" />
</button>
<button
type="button"
onClick={() => sendAction('expand')}
className="flex h-6 w-6 items-center justify-center rounded-full bg-neutral-700 text-white/90 transition-colors hover:bg-neutral-600"
aria-label="Expand to full screen (stops screen sharing)"
title="Expand to full screen (stops sharing)"
>
<Maximize2 className="h-3.5 w-3.5" />
</button>
</div>
</div>
)
}

View file

@ -1,6 +1,7 @@
import { useCallback, useRef, useState } from 'react';
import { toast } from 'sonner';
import { buildDeepgramListenUrl } from '@/lib/deepgram-listen-url';
import { finalizeDeepgramStream } from '@/lib/deepgram-finalize';
import { useRowboatAccount } from '@/hooks/useRowboatAccount';
export type MeetingTranscriptionState = 'idle' | 'connecting' | 'recording' | 'stopping';
@ -196,6 +197,25 @@ export function useMeetingTranscription(onAutoStop?: () => void) {
}, 1000);
}, [writeTranscriptToFile]);
const stopInputCapture = useCallback(() => {
if (processorRef.current) {
processorRef.current.disconnect();
processorRef.current = null;
}
if (audioCtxRef.current) {
audioCtxRef.current.close();
audioCtxRef.current = null;
}
if (micStreamRef.current) {
micStreamRef.current.getTracks().forEach(t => t.stop());
micStreamRef.current = null;
}
if (systemStreamRef.current) {
systemStreamRef.current.getTracks().forEach(t => t.stop());
systemStreamRef.current = null;
}
}, []);
const cleanup = useCallback(() => {
if (writeTimerRef.current) {
clearTimeout(writeTimerRef.current);
@ -213,28 +233,13 @@ export function useMeetingTranscription(onAutoStop?: () => void) {
clearInterval(trackPollingRef.current);
trackPollingRef.current = null;
}
if (processorRef.current) {
processorRef.current.disconnect();
processorRef.current = null;
}
if (audioCtxRef.current) {
audioCtxRef.current.close();
audioCtxRef.current = null;
}
if (micStreamRef.current) {
micStreamRef.current.getTracks().forEach(t => t.stop());
micStreamRef.current = null;
}
if (systemStreamRef.current) {
systemStreamRef.current.getTracks().forEach(t => t.stop());
systemStreamRef.current = null;
}
stopInputCapture();
if (wsRef.current) {
wsRef.current.onclose = null;
wsRef.current.close();
wsRef.current = null;
}
}, []);
}, [stopInputCapture]);
const start = useCallback(async (calendarEvent?: CalendarEventMeta): Promise<string | null> => {
if (state !== 'idle') return null;
@ -551,12 +556,14 @@ export function useMeetingTranscription(onAutoStop?: () => void) {
if (state !== 'recording') return;
setState('stopping');
stopInputCapture();
await finalizeDeepgramStream(wsRef.current, 2200);
cleanup();
interimRef.current = new Map();
await writeTranscriptToFile();
interimRef.current = new Map();
setState('idle');
}, [state, cleanup, writeTranscriptToFile]);
}, [state, cleanup, stopInputCapture, writeTranscriptToFile]);
return { state, start, stop };
}

View file

@ -0,0 +1,123 @@
import { StrictMode } from 'react'
import { act, renderHook, waitFor } from '@testing-library/react'
import { describe, expect, it } from 'vitest'
import type { SessionBusEvent } from '@x/shared/src/sessions.js'
import type { SessionsClient } from '@/lib/session-chat/client'
import type { SessionFeedListener } from '@/lib/session-chat/feed'
import {
assistantText,
completed,
completedTurnLog,
created,
requested,
sessionState,
turnCompleted,
user,
} from '@/lib/session-chat/test-fixtures'
import { isChatMessage } from '@/lib/chat-conversation'
import { useSessionChat } from './useSessionChat'
const S1 = 'sess-1'
function makeDeps() {
const calls: Array<{ method: string; args: unknown[] }> = []
let emit: SessionFeedListener = () => undefined
let unsubscribed = 0
const sessions = new Map([[S1, sessionState(S1, ['turn-1'])]])
const turns = new Map([['turn-1', completedTurnLog('turn-1', S1, 'q1', 'a1')]])
const client: SessionsClient = {
create: async () => ({ sessionId: 'x' }),
list: async () => ({ sessions: [] }),
get: async (sessionId) => {
const state = sessions.get(sessionId)
if (!state) throw new Error('session not found')
return state
},
getTurn: async (turnId) => ({ turnId, events: turns.get(turnId) ?? [] }),
sendMessage: async (...args) => {
calls.push({ method: 'sendMessage', args })
return { turnId: 'turn-2' }
},
respondToPermission: async (...args) => {
calls.push({ method: 'respondToPermission', args })
},
respondToAskHuman: async (...args) => {
calls.push({ method: 'respondToAskHuman', args })
},
stopTurn: async (...args) => {
calls.push({ method: 'stopTurn', args })
},
resumeTurn: async () => undefined,
setTitle: async () => undefined,
delete: async () => undefined,
}
return {
deps: {
client,
subscribeFeed: (listener: SessionFeedListener) => {
emit = listener
return () => {
unsubscribed += 1
}
},
},
calls,
emit: (event: SessionBusEvent) => emit(event),
getUnsubscribed: () => unsubscribed,
}
}
describe('useSessionChat', () => {
it('seeds from the session, follows live events, and routes actions', async () => {
const { deps, calls, emit } = makeDeps()
const { result } = renderHook(() => useSessionChat(S1, deps), { wrapper: StrictMode })
await waitFor(() => {
expect(result.current.latestTurnId).toBe('turn-1')
})
expect(
result.current.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['q1', 'a1'])
// A new turn streams in over the feed.
act(() => {
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: created('turn-2', S1, user('q2')) })
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: requested('turn-2', 0) })
emit({
kind: 'turn-event',
sessionId: S1,
turnId: 'turn-2',
event: { type: 'text_delta', turnId: 'turn-2', modelCallIndex: 0, delta: 'a2…' },
})
})
expect(result.current.latestTurnId).toBe('turn-2')
expect(result.current.chatState?.currentAssistantMessage).toBe('a2…')
expect(result.current.chatState?.isProcessing).toBe(true)
act(() => {
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: completed('turn-2', 0, assistantText('a2')) })
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: turnCompleted('turn-2', 'a2') })
})
expect(result.current.chatState?.isProcessing).toBe(false)
await act(async () => {
await result.current.respondToPermission('tc1', 'deny')
await result.current.stop()
})
expect(calls).toEqual([
{ method: 'respondToPermission', args: ['turn-2', 'tc1', 'deny', undefined] },
{ method: 'stopTurn', args: ['turn-2'] },
])
})
it('unsubscribes from the feed on unmount (StrictMode double-mounts included)', async () => {
const { deps, getUnsubscribed } = makeDeps()
const { unmount } = renderHook(() => useSessionChat(S1, deps), {
wrapper: StrictMode,
})
unmount()
// StrictMode's simulated cleanup plus the real unmount: every subscribe
// was matched by an unsubscribe.
expect(getUnsubscribed()).toBe(2)
})
})

View file

@ -0,0 +1,36 @@
import { useEffect, useMemo, useState, useSyncExternalStore } from 'react'
import { ipcSessionsClient } from '@/lib/session-chat/client'
import { subscribeSessionFeed } from '@/lib/session-chat/feed'
import { SessionChatStore, type SessionChatStoreDeps } from '@/lib/session-chat/store'
const defaultDeps: SessionChatStoreDeps = {
client: ipcSessionsClient,
subscribeFeed: subscribeSessionFeed,
}
// Thin subscription over SessionChatStore — all logic (seeding, feed events,
// reducer, overlay, action routing) lives in the store, which is unit-tested
// without React. `deps` is injectable for tests.
export function useSessionChat(
sessionId: string | null,
deps: SessionChatStoreDeps = defaultDeps,
) {
const [store] = useState(() => new SessionChatStore(deps))
useEffect(() => store.connect(), [store])
useEffect(() => {
void store.setSession(sessionId)
}, [store, sessionId])
const snapshot = useSyncExternalStore(store.subscribe, store.getSnapshot)
return useMemo(
() => ({
...snapshot,
sendMessage: store.sendMessage,
respondToPermission: store.respondToPermission,
answerAskHuman: store.answerAskHuman,
stop: store.stop,
}),
[snapshot, store],
)
}
export type SessionChat = ReturnType<typeof useSessionChat>

View file

@ -0,0 +1,33 @@
import { useEffect, useMemo, useState, useSyncExternalStore } from 'react'
import { ipcSessionsClient } from '@/lib/session-chat/client'
import { subscribeSessionFeed } from '@/lib/session-chat/feed'
import { SessionListStore, type SessionChatStoreDeps } from '@/lib/session-chat/store'
const defaultDeps: SessionChatStoreDeps = {
client: ipcSessionsClient,
subscribeFeed: subscribeSessionFeed,
}
// The session list (chat history sidebar): seeded from sessions:list, kept
// current by index-changed feed events. Logic lives in SessionListStore.
export function useSessions(deps: SessionChatStoreDeps = defaultDeps) {
const [store] = useState(() => new SessionListStore(deps))
useEffect(() => {
const disconnect = store.connect()
void store.load()
return disconnect
}, [store])
const snapshot = useSyncExternalStore(store.subscribe, store.getSnapshot)
const client = deps.client
return useMemo(
() => ({
...snapshot,
createSession: (input: { title?: string } = {}) => client.create(input),
deleteSession: (sessionId: string) => client.delete(sessionId),
setTitle: (sessionId: string, title: string) => client.setTitle(sessionId, title),
}),
[snapshot, client],
)
}
export type Sessions = ReturnType<typeof useSessions>

View file

@ -0,0 +1,312 @@
import { useCallback, useEffect, useRef, useState } from 'react';
export type VideoModeState = 'idle' | 'starting' | 'live';
export type ScreenShareState = 'idle' | 'starting' | 'live';
export interface CapturedVideoFrame {
/** base64-encoded JPEG bytes (no data: prefix) — shape of the UserImagePart wire format */
data: string;
mediaType: string;
capturedAt: string; // ISO timestamp
/** data: URL of the same frame, for direct display in the transcript */
dataUrl: string;
source: 'camera' | 'screen';
}
// Frames are grabbed once per second — dense enough to catch expression and
// posture changes while the user talks. Per message we attach at most
// MAX_*_FRAMES_PER_MESSAGE frames, evenly sampled across the window since the
// last send, so long monologues don't balloon the request.
const CAPTURE_INTERVAL_MS = 1000;
const MAX_CAMERA_FRAMES_PER_MESSAGE = 12;
// Screen frames are ~4x the resolution (and tokens) of camera frames, and the
// latest view matters far more than the trajectory — keep the cap small.
const MAX_SCREEN_FRAMES_PER_MESSAGE = 4;
// Rolling buffer bound (~2 minutes). The buffer only needs to cover the gap
// between two sends; anything older is stale context anyway.
const MAX_BUFFERED_FRAMES = 120;
// Downscale targets. 512px wide JPEG keeps a webcam frame around 20-40KB —
// cheap enough to inline a dozen per message as multimodal image parts.
// Screen captures keep 1280px so on-screen text stays legible to the model.
const CAMERA_FRAME_WIDTH = 512;
const SCREEN_FRAME_WIDTH = 1280;
const CAMERA_JPEG_QUALITY = 0.65;
const SCREEN_JPEG_QUALITY = 0.7;
interface BufferedFrame {
dataUrl: string;
capturedAt: string;
ts: number;
}
// One capture pipeline: stream → offscreen <video> → canvas JPEG → ring buffer.
interface CapturePipe {
stream: MediaStream | null;
videoEl: HTMLVideoElement | null;
canvas: HTMLCanvasElement | null;
interval: ReturnType<typeof setInterval> | null;
frames: BufferedFrame[];
lastCollectTs: number;
}
const emptyPipe = (): CapturePipe => ({
stream: null,
videoEl: null,
canvas: null,
interval: null,
frames: [],
lastCollectTs: 0,
});
function capturePipeFrame(pipe: CapturePipe, width: number, quality: number) {
const videoEl = pipe.videoEl;
if (!videoEl || videoEl.readyState < 2 || videoEl.videoWidth === 0) return;
if (!pipe.canvas) {
pipe.canvas = document.createElement('canvas');
}
const canvas = pipe.canvas;
const scale = Math.min(1, width / videoEl.videoWidth);
canvas.width = Math.round(videoEl.videoWidth * scale);
canvas.height = Math.round(videoEl.videoHeight * scale);
const ctx = canvas.getContext('2d');
if (!ctx) return;
ctx.drawImage(videoEl, 0, 0, canvas.width, canvas.height);
const dataUrl = canvas.toDataURL('image/jpeg', quality);
// A near-empty data URL means the frame was blank (source still warming up)
if (dataUrl.length < 100) return;
pipe.frames.push({ dataUrl, capturedAt: new Date().toISOString(), ts: Date.now() });
if (pipe.frames.length > MAX_BUFFERED_FRAMES) {
pipe.frames.splice(0, pipe.frames.length - MAX_BUFFERED_FRAMES);
}
}
function attachPipeSource(pipe: CapturePipe, stream: MediaStream, grab: () => void) {
pipe.stream = stream;
// Offscreen <video> that feeds the capture canvas; any visible preview
// attaches to the same MediaStream separately.
const videoEl = document.createElement('video');
videoEl.muted = true;
videoEl.playsInline = true;
videoEl.srcObject = stream;
pipe.videoEl = videoEl;
videoEl.play().catch(() => {});
// First frame as soon as the source delivers data, then steady-state cadence.
videoEl.addEventListener('loadeddata', () => grab(), { once: true });
pipe.interval = setInterval(grab, CAPTURE_INTERVAL_MS);
}
function teardownPipe(pipe: CapturePipe) {
if (pipe.interval) {
clearInterval(pipe.interval);
pipe.interval = null;
}
if (pipe.videoEl) {
pipe.videoEl.srcObject = null;
pipe.videoEl = null;
}
if (pipe.stream) {
pipe.stream.getTracks().forEach((t) => t.stop());
pipe.stream = null;
}
pipe.frames = [];
pipe.lastCollectTs = 0;
}
/**
* Drain frames captured since the previous collection, evenly sampled down to
* `max` (always keeping the newest). Falls back to the single most recent
* frame when nothing new accumulated (rapid-fire messages), so every message
* carries at least one frame once the source has warmed up.
*/
function drainPipe(pipe: CapturePipe, max: number, source: CapturedVideoFrame['source']): CapturedVideoFrame[] {
const all = pipe.frames;
if (all.length === 0) return [];
let window_ = all.filter((f) => f.ts > pipe.lastCollectTs);
if (window_.length === 0) {
window_ = [all[all.length - 1]];
}
pipe.lastCollectTs = window_[window_.length - 1].ts;
let sampled: BufferedFrame[];
if (window_.length <= max) {
sampled = window_;
} else {
sampled = [];
const step = (window_.length - 1) / (max - 1);
for (let i = 0; i < max; i++) {
sampled.push(window_[Math.round(i * step)]);
}
}
return sampled.map((f) => ({
data: f.dataUrl.slice(f.dataUrl.indexOf(',') + 1),
mediaType: 'image/jpeg',
capturedAt: f.capturedAt,
dataUrl: f.dataUrl,
source,
}));
}
export function useVideoMode() {
const [state, setState] = useState<VideoModeState>('idle');
const [screenState, setScreenState] = useState<ScreenShareState>('idle');
// Camera can be turned off mid-session (Meet-style) while the mode — and
// any screen share — keeps running. Resets to on for the next session.
const [cameraOn, setCameraOn] = useState(true);
const cameraPipeRef = useRef<CapturePipe>(emptyPipe());
const screenPipeRef = useRef<CapturePipe>(emptyPipe());
// Stable stream refs for preview components (<video srcObject>).
const streamRef = useRef<MediaStream | null>(null);
const screenStreamRef = useRef<MediaStream | null>(null);
const stateRef = useRef<VideoModeState>('idle');
stateRef.current = state;
const screenStateRef = useRef<ScreenShareState>('idle');
screenStateRef.current = screenState;
const captureCameraFrame = useCallback(() => {
capturePipeFrame(cameraPipeRef.current, CAMERA_FRAME_WIDTH, CAMERA_JPEG_QUALITY);
}, []);
const captureScreenFrame = useCallback(() => {
capturePipeFrame(screenPipeRef.current, SCREEN_FRAME_WIDTH, SCREEN_JPEG_QUALITY);
}, []);
const stopScreenShare = useCallback(() => {
teardownPipe(screenPipeRef.current);
screenStreamRef.current = null;
setScreenState('idle');
}, []);
const stop = useCallback(() => {
teardownPipe(cameraPipeRef.current);
streamRef.current = null;
setState('idle');
setCameraOn(true);
stopScreenShare();
}, [stopScreenShare]);
// Acquire the webcam and start its capture pipeline. Shared by start()
// and by re-enabling the camera mid-session.
const acquireCamera = useCallback(async (): Promise<boolean> => {
// Settle the macOS TCC camera permission before getUserMedia, same as
// voice mode does for the mic — otherwise the first click silently
// fails while the native prompt is still up.
const access = await window.ipc
.invoke('voice:ensureCameraAccess', null)
.catch(() => ({ granted: true }));
if (!access.granted) {
console.error('[video] Camera access denied');
return false;
}
let stream: MediaStream | null = null;
try {
stream = await navigator.mediaDevices.getUserMedia({
video: { width: { ideal: 1280 }, height: { ideal: 720 }, facingMode: 'user' },
audio: false,
});
} catch (err) {
console.error('[video] Camera access failed:', err);
return false;
}
streamRef.current = stream;
attachPipeSource(cameraPipeRef.current, stream, captureCameraFrame);
return true;
}, [captureCameraFrame]);
/**
* Turn the camera off/on without leaving video mode (Meet-style). While
* off, no webcam frames are captured or attached; screen-share frames
* (if presenting) keep flowing.
*/
const setCameraEnabled = useCallback(async (enabled: boolean): Promise<boolean> => {
if (stateRef.current !== 'live') return false;
if (enabled) {
const ok = await acquireCamera();
if (ok) setCameraOn(true);
return ok;
}
teardownPipe(cameraPipeRef.current);
streamRef.current = null;
setCameraOn(false);
return true;
}, [acquireCamera]);
/**
* Start video mode. `camera: false` starts a camera-less session (voice
* call / screen-share-only) the mode is live so frames can flow from
* other sources, and the camera can be enabled later via setCameraEnabled.
*/
const start = useCallback(async ({ camera = true }: { camera?: boolean } = {}): Promise<boolean> => {
if (stateRef.current !== 'idle') return true;
setState('starting');
if (camera) {
const ok = await acquireCamera();
if (!ok) {
setState('idle');
return false;
}
}
setCameraOn(camera);
setState('live');
return true;
}, [acquireCamera]);
/**
* Share the screen. The main process auto-approves getDisplayMedia with
* the primary screen (see setDisplayMediaRequestHandler in main.ts), so
* no source picker appears. Returns false if capture couldn't start
* (usually the macOS Screen Recording permission).
*/
const startScreenShare = useCallback(async (): Promise<boolean> => {
if (screenStateRef.current !== 'idle') return true;
setScreenState('starting');
// Surfaces the macOS Screen Recording permission state and, on first
// use, registers the app in System Settings (same flow meetings use).
await window.ipc.invoke('meeting:checkScreenPermission', null).catch(() => null);
let stream: MediaStream | null = null;
try {
stream = await navigator.mediaDevices.getDisplayMedia({
video: { frameRate: { ideal: 5 } },
audio: false,
});
} catch (err) {
console.error('[video] Screen share failed:', err);
setScreenState('idle');
return false;
}
screenStreamRef.current = stream;
// The capture can end outside our UI (display unplugged, OS revokes) —
// tear down cleanly so the UI doesn't show a dead share.
stream.getVideoTracks()[0]?.addEventListener('ended', () => stopScreenShare(), { once: true });
attachPipeSource(screenPipeRef.current, stream, captureScreenFrame);
setScreenState('live');
return true;
}, [captureScreenFrame, stopScreenShare]);
/**
* Drain webcam + screen-share frames buffered since the last send, tagged
* by source. Webcam frames come first, then screen frames.
*/
const collectFrames = useCallback((): CapturedVideoFrame[] => {
if (stateRef.current !== 'live') return [];
// Grab a frame right now so the message always includes the moment of send.
captureCameraFrame();
const frames = drainPipe(cameraPipeRef.current, MAX_CAMERA_FRAMES_PER_MESSAGE, 'camera');
if (screenStateRef.current === 'live') {
captureScreenFrame();
frames.push(...drainPipe(screenPipeRef.current, MAX_SCREEN_FRAMES_PER_MESSAGE, 'screen'));
}
return frames;
}, [captureCameraFrame, captureScreenFrame]);
// Release the camera/screen if the component unmounts with video mode on.
useEffect(() => stop, [stop]);
return { state, screenState, cameraOn, streamRef, screenStreamRef, start, stop, startScreenShare, stopScreenShare, setCameraEnabled, collectFrames };
}

View file

@ -1,10 +1,11 @@
import { useCallback, useRef, useState } from 'react';
import { buildDeepgramListenUrl } from '@/lib/deepgram-listen-url';
import { finalizeDeepgramStream } from '@/lib/deepgram-finalize';
import { useRowboatAccount } from '@/hooks/useRowboatAccount';
import posthog from 'posthog-js';
import * as analytics from '@/lib/analytics';
export type VoiceState = 'idle' | 'connecting' | 'listening';
export type VoiceState = 'idle' | 'connecting' | 'listening' | 'submitting';
const DEEPGRAM_PARAMS = new URLSearchParams({
model: 'nova-3',
@ -18,7 +19,35 @@ const DEEPGRAM_PARAMS = new URLSearchParams({
endpointing: '100',
no_delay: 'true',
});
const DEEPGRAM_LISTEN_URL = `wss://api.deepgram.com/v1/listen?${DEEPGRAM_PARAMS.toString()}`;
// Hands-free (continuous) mode: Deepgram's endpoint fires FAST (600ms of
// silence) and we apply smart hold logic on our side — if the transcript
// already reads as a complete thought (terminal punctuation) the utterance
// fires immediately, otherwise we hold INCOMPLETE_HOLD_MS longer in case the
// user was mid-thought. Net effect: complete sentences turn around ~1.2s
// faster than the old fixed 1800ms endpoint, while thinking pauses still get
// the same total grace (~1.8s).
const CONTINUOUS_ENDPOINTING_MS = 600;
const INCOMPLETE_HOLD_MS = 1200;
// While the mic is paused (assistant speaking), keep the idle Deepgram socket
// alive — it closes after ~10s without audio otherwise.
const KEEPALIVE_INTERVAL_MS = 5000;
// Deepgram punctuates finals (punctuate=true) — a transcript ending in
// terminal punctuation (optionally inside a closing quote/paren) is treated
// as a complete thought.
const COMPLETE_THOUGHT_RE = /[.!?…]["')\]]*\s*$/;
function deepgramParams(continuous: boolean): URLSearchParams {
if (!continuous) return DEEPGRAM_PARAMS;
const params = new URLSearchParams(DEEPGRAM_PARAMS);
params.set('endpointing', String(CONTINUOUS_ENDPOINTING_MS));
// Second end-of-speech signal: speech_final can be missed (it often rides
// on a result with an empty transcript, or never fires when background
// noise keeps the endpointer engaged). UtteranceEnd is word-timing based
// and arrives as its own message type, so we listen for both.
params.set('utterance_end_ms', '1000');
return params;
}
// Cap on retained per-frame amplitude samples (~64ms/frame ⇒ ~5 min of history).
// The waveform only ever displays the most recent window, so older samples are dropped.
@ -52,6 +81,13 @@ export function useVoiceMode() {
const audioLevelsRef = useRef<number[]>([]);
// Running peak amplitude for the waveform auto-gain (see PEAK_DECAY/MIN_PEAK).
const audioPeakRef = useRef(0);
// Hands-free mode: invoked with each completed utterance (speech_final).
const continuousCbRef = useRef<((text: string) => void) | null>(null);
// While true (assistant is speaking), mic audio is dropped instead of streamed.
const pausedRef = useRef(false);
const keepAliveTimerRef = useRef<ReturnType<typeof setInterval> | null>(null);
// Pending mid-thought hold (smart endpointing) — see maybeEndUtterance.
const holdTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// Refresh cached auth details (called on warmup, not on mic click)
const refreshAuth = useCallback(async () => {
@ -70,9 +106,44 @@ export function useVoiceMode() {
}
}, [refreshRowboatAccount]);
// Hands-free mode: flush the accumulated utterance to the callback.
// Both end-of-speech signals may fire for the same utterance — the second
// finds an empty buffer and is a no-op.
const fireContinuousUtterance = useCallback(() => {
if (holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
if (!continuousCbRef.current || pausedRef.current) return;
const utterance = transcriptBufferRef.current.trim();
transcriptBufferRef.current = '';
interimRef.current = '';
setInterimText('');
if (utterance) continuousCbRef.current(utterance);
}, []);
// Smart endpoint: Deepgram's endpoint fires fast (600ms). If the
// transcript reads as a complete thought, hand it off immediately; if it
// trails off mid-sentence ("so what I want is…"), hold a little longer —
// resumed speech cancels the hold and the utterance keeps growing.
const maybeEndUtterance = useCallback(() => {
if (!continuousCbRef.current || pausedRef.current) return;
const buffered = transcriptBufferRef.current.trim();
if (!buffered) return;
if (COMPLETE_THOUGHT_RE.test(buffered)) {
fireContinuousUtterance();
return;
}
if (holdTimerRef.current) clearTimeout(holdTimerRef.current);
holdTimerRef.current = setTimeout(() => {
holdTimerRef.current = null;
fireContinuousUtterance();
}, INCOMPLETE_HOLD_MS);
}, [fireContinuousUtterance]);
// Create and connect a Deepgram WebSocket using cached auth.
// Starts the connection and returns immediately (does not wait for open).
const connectWs = useCallback(async () => {
const connectWs = useCallback(async (continuous = false) => {
if (wsRef.current && (wsRef.current.readyState === WebSocket.OPEN || wsRef.current.readyState === WebSocket.CONNECTING)) return;
// Refresh auth if we don't have it cached yet
@ -81,12 +152,13 @@ export function useVoiceMode() {
}
if (!cachedAuth) return;
const params = deepgramParams(continuous);
let ws: WebSocket;
if (cachedAuth.type === 'rowboat') {
const listenUrl = buildDeepgramListenUrl(cachedAuth.url, DEEPGRAM_PARAMS);
const listenUrl = buildDeepgramListenUrl(cachedAuth.url, params);
ws = new WebSocket(listenUrl, ['bearer', cachedAuth.token]);
} else {
ws = new WebSocket(DEEPGRAM_LISTEN_URL, ['token', cachedAuth.apiKey]);
ws = new WebSocket(`wss://api.deepgram.com/v1/listen?${params.toString()}`, ['token', cachedAuth.apiKey]);
}
wsRef.current = ws;
@ -102,16 +174,43 @@ export function useVoiceMode() {
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
if (!data.channel?.alternatives?.[0]) return;
// Hands-free mode: word-timing based end-of-speech marker.
if (data.type === 'UtteranceEnd') {
maybeEndUtterance();
return;
}
if (!data.channel?.alternatives?.[0]) return;
const transcript = data.channel.alternatives[0].transcript;
if (!transcript) return;
// The user resumed speaking — cancel any pending mid-thought hold
// so the utterance keeps growing instead of firing under them.
if (transcript && holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
if (data.is_final) {
transcriptBufferRef.current += (transcriptBufferRef.current ? ' ' : '') + transcript;
interimRef.current = '';
setInterimText(transcriptBufferRef.current);
} else {
// NOTE: the endpoint marker (speech_final) usually arrives on a
// result whose transcript is EMPTY — the silence after the user
// stops talking. Empty finals must still reach the speech_final
// check below or hands-free utterances never complete.
if (transcript) {
transcriptBufferRef.current += (transcriptBufferRef.current ? ' ' : '') + transcript;
interimRef.current = '';
}
// Hands-free mode: an endpoint may complete the utterance —
// immediately for complete thoughts, after a short hold for
// mid-sentence trails.
if (continuousCbRef.current && data.speech_final) {
maybeEndUtterance();
return;
}
if (transcript) {
setInterimText(transcriptBufferRef.current);
}
} else if (transcript) {
interimRef.current = transcript;
setInterimText(transcriptBufferRef.current + (transcriptBufferRef.current ? ' ' : '') + transcript);
}
@ -126,11 +225,57 @@ export function useVoiceMode() {
ws.onclose = () => {
console.log('[voice] WebSocket closed');
wsRef.current = null;
// A hands-free call is long-lived — if the socket drops while the
// call is still on, reconnect instead of silently going deaf.
if (continuousCbRef.current) {
setTimeout(() => {
if (continuousCbRef.current && !wsRef.current) {
void connectWs(true);
}
}, 1000);
}
};
}, [refreshAuth]);
}, [refreshAuth, maybeEndUtterance]);
// Stop audio capture and close WS
const stopAudioCapture = useCallback(() => {
const waitForWsOpen = useCallback(async (timeoutMs = 1500): Promise<boolean> => {
const ws = wsRef.current;
if (!ws) return false;
if (ws.readyState === WebSocket.OPEN) return true;
if (ws.readyState !== WebSocket.CONNECTING) return false;
return new Promise<boolean>((resolve) => {
let done = false;
let timeout: ReturnType<typeof setTimeout>;
const finish = (ok: boolean) => {
if (done) return;
done = true;
clearTimeout(timeout);
ws.removeEventListener('open', onOpen);
ws.removeEventListener('error', onError);
ws.removeEventListener('close', onClose);
resolve(ok);
};
const onOpen = () => finish(true);
const onError = () => finish(false);
const onClose = () => finish(false);
timeout = setTimeout(() => finish(false), timeoutMs);
ws.addEventListener('open', onOpen);
ws.addEventListener('error', onError);
ws.addEventListener('close', onClose);
});
}, []);
const flushBufferedAudio = useCallback(() => {
const ws = wsRef.current;
if (!ws || ws.readyState !== WebSocket.OPEN) return;
const buffered = audioBufferRef.current;
audioBufferRef.current = [];
for (const chunk of buffered) {
ws.send(chunk);
}
}, []);
const stopInputCapture = useCallback(() => {
if (processorRef.current) {
processorRef.current.disconnect();
processorRef.current = null;
@ -143,11 +288,26 @@ export function useVoiceMode() {
mediaStreamRef.current.getTracks().forEach(t => t.stop());
mediaStreamRef.current = null;
}
}, []);
// Stop audio capture and close WS
const stopAudioCapture = useCallback(() => {
stopInputCapture();
if (wsRef.current) {
wsRef.current.onclose = null;
wsRef.current.close();
wsRef.current = null;
}
continuousCbRef.current = null;
pausedRef.current = false;
if (holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
if (keepAliveTimerRef.current) {
clearInterval(keepAliveTimerRef.current);
keepAliveTimerRef.current = null;
}
audioBufferRef.current = [];
audioLevelsRef.current = [];
audioPeakRef.current = 0;
@ -155,9 +315,9 @@ export function useVoiceMode() {
transcriptBufferRef.current = '';
interimRef.current = '';
setState('idle');
}, []);
}, [stopInputCapture]);
const start = useCallback(async () => {
const start = useCallback(async (continuous = false) => {
if (state !== 'idle') return;
transcriptBufferRef.current = '';
@ -192,7 +352,7 @@ export function useVoiceMode() {
console.error('Microphone access denied:', err);
return null;
}),
connectWs(),
connectWs(continuous),
]);
if (!stream) {
@ -216,6 +376,9 @@ export function useVoiceMode() {
processorRef.current = processor;
processor.onaudioprocess = (e) => {
// Paused (assistant is speaking in a call): drop mic audio so the
// assistant's own TTS never gets transcribed back at it.
if (pausedRef.current) return;
const float32 = e.inputBuffer.getChannelData(0);
const int16 = new Int16Array(float32.length);
let sumSquares = 0;
@ -240,8 +403,13 @@ export function useVoiceMode() {
if (wsRef.current?.readyState === WebSocket.OPEN) {
wsRef.current.send(buffer);
} else {
// WebSocket still connecting — buffer the audio
// WebSocket still connecting (or reconnecting mid-call) —
// buffer the audio, bounded so an unreachable server during a
// long call can't grow it without limit (~30s at 64ms/chunk).
audioBufferRef.current.push(buffer);
if (audioBufferRef.current.length > 500) {
audioBufferRef.current.shift();
}
}
};
@ -250,25 +418,71 @@ export function useVoiceMode() {
}, [state, connectWs, stopAudioCapture]);
/** Stop recording and return the full transcript (finalized + any current interim) */
const submit = useCallback((): string => {
const submit = useCallback(async (): Promise<string> => {
setState('submitting');
stopInputCapture();
if (wsRef.current?.readyState === WebSocket.CONNECTING) {
await waitForWsOpen();
}
flushBufferedAudio();
await finalizeDeepgramStream(wsRef.current);
let text = transcriptBufferRef.current;
if (interimRef.current) {
text += (text ? ' ' : '') + interimRef.current;
}
text = text.trim();
stopAudioCapture();
return text;
}, [stopAudioCapture]);
}, [flushBufferedAudio, stopAudioCapture, stopInputCapture, waitForWsOpen]);
/** Cancel recording without returning transcript */
const cancel = useCallback(() => {
stopAudioCapture();
}, [stopAudioCapture]);
/**
* Hands-free (call) mode: listen continuously and invoke `onUtterance`
* with each completed utterance. Runs until cancel()/stop.
*/
const startContinuous = useCallback(async (onUtterance: (text: string) => void) => {
continuousCbRef.current = onUtterance;
await start(true);
}, [start]);
/**
* Mute/unmute the continuous stream (used while the assistant is
* thinking/speaking). Keeps the Deepgram socket alive with KeepAlives and
* discards any half-heard utterance from before the pause.
*/
const setPaused = useCallback((paused: boolean) => {
if (pausedRef.current === paused) return;
pausedRef.current = paused;
if (paused) {
if (holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
transcriptBufferRef.current = '';
interimRef.current = '';
setInterimText('');
keepAliveTimerRef.current = setInterval(() => {
if (wsRef.current?.readyState === WebSocket.OPEN) {
wsRef.current.send(JSON.stringify({ type: 'KeepAlive' }));
}
}, KEEPALIVE_INTERVAL_MS);
} else if (keepAliveTimerRef.current) {
clearInterval(keepAliveTimerRef.current);
keepAliveTimerRef.current = null;
}
}, []);
/** Pre-cache auth details so mic click skips IPC round-trips */
const warmup = useCallback(() => {
refreshAuth().catch(() => {});
}, [refreshAuth]);
return { state, interimText, audioLevelsRef, start, submit, cancel, warmup };
return { state, interimText, audioLevelsRef, start, submit, cancel, warmup, startContinuous, setPaused };
}

View file

@ -1,4 +1,4 @@
import { useCallback, useRef, useState } from 'react';
import { useCallback, useEffect, useRef, useState } from 'react';
export type TTSState = 'idle' | 'synthesizing' | 'speaking';
@ -14,14 +14,23 @@ function synthesize(text: string): Promise<SynthesizedAudio> {
);
}
function playAudio(dataUrl: string, audioRef: React.MutableRefObject<HTMLAudioElement | null>): Promise<void> {
function playAudio(
dataUrl: string,
audioRef: React.MutableRefObject<HTMLAudioElement | null>,
onAudioElement?: (audio: HTMLAudioElement) => void
): Promise<void> {
return new Promise<void>((resolve, reject) => {
const audio = new Audio(dataUrl);
audioRef.current = audio;
onAudioElement?.(audio);
audio.onended = () => {
console.log('[tts] audio ended');
resolve();
};
// pause() (from cancel) must settle this promise too, or the queue
// loop stays parked on it forever. Natural end also fires 'pause'
// just before 'ended'; double-resolve is harmless.
audio.onpause = () => resolve();
audio.onerror = (e) => {
console.error('[tts] audio error:', e);
reject(new Error('Audio playback failed'));
@ -35,47 +44,282 @@ function playAudio(dataUrl: string, audioRef: React.MutableRefObject<HTMLAudioEl
});
}
/** A queue entry: text to synthesize, or a ready-to-play audio URL (e.g. a bundled clip). */
type QueueItem = { text: string } | { url: string };
type TtsChunkMsg = { requestId: string; chunkBase64?: string; done: boolean; error?: string };
export function useVoiceTTS() {
const [state, setState] = useState<TTSState>('idle');
const audioRef = useRef<HTMLAudioElement | null>(null);
const queueRef = useRef<string[]>([]);
const queueRef = useRef<QueueItem[]>([]);
const processingRef = useRef(false);
// Pre-fetched audio ready to play immediately
const prefetchedRef = useRef<Promise<SynthesizedAudio> | null>(null);
// Streaming synthesis: per-request chunk handlers + the in-flight request
// id (so cancel() can abort the main-process fetch).
const streamHandlersRef = useRef<Map<string, (msg: TtsChunkMsg) => void>>(new Map());
const activeStreamIdRef = useRef<string | null>(null);
// Bumped by cancel(). A queue loop that awaited across a cancel sees a
// stale generation and exits instead of playing audio that was cancelled
// while still synthesizing (which would overlap the next utterance).
const generationRef = useRef(0);
// Web Audio analyser tap for lip-sync (talking head)
const audioCtxRef = useRef<AudioContext | null>(null);
const analyserRef = useRef<AnalyserNode | null>(null);
const levelBufferRef = useRef<Uint8Array<ArrayBuffer> | null>(null);
// Route playback through an AnalyserNode so consumers can read the live
// output level. If Web Audio wiring fails, the element still plays directly.
const connectAnalyser = useCallback((audio: HTMLAudioElement) => {
try {
let ctx = audioCtxRef.current;
if (!ctx) {
ctx = new AudioContext();
audioCtxRef.current = ctx;
const analyser = ctx.createAnalyser();
analyser.fftSize = 512;
analyser.smoothingTimeConstant = 0.5;
analyser.connect(ctx.destination);
analyserRef.current = analyser;
}
if (ctx.state === 'suspended') {
void ctx.resume();
}
const source = ctx.createMediaElementSource(audio);
source.connect(analyserRef.current!);
// Detach once this chunk is done (ended, cancelled via pause, or
// failed) so source nodes don't accumulate over a long session.
const disconnect = () => {
try {
source.disconnect();
} catch {
// already disconnected
}
};
audio.addEventListener('ended', disconnect, { once: true });
audio.addEventListener('pause', disconnect, { once: true });
audio.addEventListener('error', disconnect, { once: true });
} catch (err) {
console.error('[tts] analyser hookup failed:', err);
}
}, []);
// Current output level, 0..1. Safe to call every animation frame.
// Release the audio graph when the owning component unmounts
useEffect(() => () => {
audioCtxRef.current?.close().catch(() => {});
audioCtxRef.current = null;
analyserRef.current = null;
}, []);
// Route streaming TTS chunks to whichever request is waiting for them.
useEffect(() => {
return window.ipc.on('voice:tts-chunk', (msg) => {
streamHandlersRef.current.get(msg.requestId)?.(msg);
});
}, []);
/**
* Streaming synthesis + playback via MediaSource: audio starts on the
* first chunk instead of after the full body. Rejects (for caller
* fallback to non-streaming synth) if the stream fails before any audio
* arrived; resolves when playback finishes.
*/
const streamSynthesizeAndPlay = useCallback((text: string, onStarted: () => void): Promise<void> => {
return new Promise<void>((resolve, reject) => {
if (typeof MediaSource === 'undefined' || !MediaSource.isTypeSupported('audio/mpeg')) {
reject(new Error('MSE audio/mpeg unsupported'));
return;
}
const requestId = `tts-${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
const mediaSource = new MediaSource();
const audio = new Audio();
audio.src = URL.createObjectURL(mediaSource);
audioRef.current = audio;
connectAnalyser(audio);
activeStreamIdRef.current = requestId;
let sourceBuffer: SourceBuffer | null = null;
const pending: Uint8Array[] = [];
let streamDone = false;
let gotAudio = false;
let settled = false;
const cleanup = () => {
streamHandlersRef.current.delete(requestId);
if (activeStreamIdRef.current === requestId) activeStreamIdRef.current = null;
URL.revokeObjectURL(audio.src);
};
const finish = (err?: Error) => {
if (settled) return;
settled = true;
cleanup();
if (err) reject(err);
else resolve();
};
// Drain pending chunks into the SourceBuffer one at a time
// (appendBuffer is async; only one append may be in flight).
const pump = () => {
if (!sourceBuffer || sourceBuffer.updating || settled) return;
const chunk = pending.shift();
if (chunk) {
try {
sourceBuffer.appendBuffer(chunk as BufferSource);
} catch (e) {
finish(e as Error);
}
return;
}
if (streamDone && mediaSource.readyState === 'open') {
try {
mediaSource.endOfStream();
} catch { /* already ended */ }
}
};
mediaSource.addEventListener('sourceopen', () => {
try {
sourceBuffer = mediaSource.addSourceBuffer('audio/mpeg');
} catch (e) {
finish(e as Error);
return;
}
sourceBuffer.addEventListener('updateend', pump);
pump();
}, { once: true });
streamHandlersRef.current.set(requestId, (msg) => {
if (msg.error && !gotAudio) {
streamDone = true;
finish(new Error(msg.error));
return;
}
if (msg.chunkBase64) {
gotAudio = true;
const bin = atob(msg.chunkBase64);
const bytes = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
pending.push(bytes);
pump();
}
if (msg.done) {
streamDone = true;
pump();
}
});
audio.addEventListener('playing', () => onStarted(), { once: true });
audio.onended = () => finish();
// pause() (from cancel) must settle this promise too; natural end
// also fires 'pause' just before 'ended'; double-settle is a no-op.
audio.onpause = () => finish();
audio.onerror = () => finish(new Error('stream playback failed'));
window.ipc
.invoke('voice:synthesizeStreamStart', { requestId, text })
.then((res) => {
if (!res.ok) finish(new Error(res.error || 'stream start failed'));
})
.catch((e) => finish(e as Error));
// Starts as soon as the first appended data is decodable.
audio.play().catch(() => { /* surfaced via onerror / chunk error */ });
// Nothing arrived at all — bail so the caller can fall back.
setTimeout(() => {
if (!gotAudio && !settled) finish(new Error('stream timeout'));
}, 10_000);
});
}, [connectAnalyser]);
const getLevel = useCallback((): number => {
const analyser = analyserRef.current;
if (!analyser) return 0;
let buffer = levelBufferRef.current;
if (!buffer || buffer.length !== analyser.fftSize) {
buffer = new Uint8Array(analyser.fftSize);
levelBufferRef.current = buffer;
}
analyser.getByteTimeDomainData(buffer);
let sum = 0;
for (let i = 0; i < buffer.length; i++) {
const d = (buffer[i] - 128) / 128;
sum += d * d;
}
const rms = Math.sqrt(sum / buffer.length);
return Math.min(1, rms * 4);
}, []);
const processQueue = useCallback(async () => {
if (processingRef.current) return;
processingRef.current = true;
const gen = generationRef.current;
// Kick off full-body pre-fetch for the next queued text while the
// current one plays — keeps sentence-to-sentence playback gapless.
const prefetchNext = () => {
const next = queueRef.current[0];
if (next && 'text' in next && next.text.trim() && !prefetchedRef.current) {
console.log('[tts] pre-fetching next:', next.text.substring(0, 80));
prefetchedRef.current = synthesize(next.text);
}
};
while (queueRef.current.length > 0) {
const text = queueRef.current.shift()!;
if (!text.trim()) continue;
const item = queueRef.current.shift()!;
if ('text' in item && !item.text.trim()) continue;
// Cold start (nothing playing, nothing pre-fetched): stream the
// synthesis so audio begins on the first chunk instead of after
// the full body — this is where first-response latency lives.
if ('text' in item && !prefetchedRef.current) {
setState('synthesizing');
console.log('[tts] stream-synthesizing:', item.text.substring(0, 80));
try {
await streamSynthesizeAndPlay(item.text, () => {
if (generationRef.current !== gen) return;
setState('speaking');
prefetchNext();
});
if (generationRef.current !== gen) return;
continue;
} catch (err) {
if (generationRef.current !== gen) return;
console.error('[tts] stream failed, falling back to full synth:', err);
// fall through to the non-streaming path below
}
}
try {
// Use pre-fetched result if available, otherwise synthesize now
// Pre-recorded URL plays as-is; text uses the pre-fetched
// result if available, otherwise synthesizes now.
let audioPromise: Promise<SynthesizedAudio>;
if (prefetchedRef.current) {
if ('url' in item) {
audioPromise = Promise.resolve({ dataUrl: item.url });
} else if (prefetchedRef.current) {
console.log('[tts] using pre-fetched audio');
audioPromise = prefetchedRef.current;
prefetchedRef.current = null;
} else {
setState('synthesizing');
console.log('[tts] synthesizing:', text.substring(0, 80));
audioPromise = synthesize(text);
console.log('[tts] synthesizing:', item.text.substring(0, 80));
audioPromise = synthesize(item.text);
}
const audio = await audioPromise;
// Cancelled while synthesizing — cancel() already reset all
// state (and a new loop may be running), so just bail.
if (generationRef.current !== gen) return;
setState('speaking');
// Kick off pre-fetch for next chunk while this one plays
const nextText = queueRef.current[0];
if (nextText?.trim()) {
console.log('[tts] pre-fetching next:', nextText.substring(0, 80));
prefetchedRef.current = synthesize(nextText);
}
prefetchNext();
await playAudio(audio.dataUrl, audioRef);
await playAudio(audio.dataUrl, audioRef, connectAnalyser);
if (generationRef.current !== gen) return;
} catch (err) {
if (generationRef.current !== gen) return;
console.error('[tts] error:', err);
prefetchedRef.current = null;
}
@ -85,17 +329,33 @@ export function useVoiceTTS() {
prefetchedRef.current = null;
processingRef.current = false;
setState('idle');
}, []);
}, [connectAnalyser, streamSynthesizeAndPlay]);
const speak = useCallback((text: string) => {
console.log('[tts] speak() called:', text.substring(0, 80));
queueRef.current.push(text);
queueRef.current.push({ text });
processQueue();
}, [processQueue]);
// Play a pre-recorded clip (e.g. bundled tour narration) through the same
// queue, so lip-sync levels, state, and cancel() all work unchanged.
const speakUrl = useCallback((url: string) => {
console.log('[tts] speakUrl() called:', url.substring(0, 120));
queueRef.current.push({ url });
processQueue();
}, [processQueue]);
const cancel = useCallback(() => {
generationRef.current++;
queueRef.current = [];
prefetchedRef.current = null;
// Abort any in-flight streaming synthesis in the main process.
if (activeStreamIdRef.current) {
void window.ipc
.invoke('voice:synthesizeStreamCancel', { requestId: activeStreamIdRef.current })
.catch(() => {});
activeStreamIdRef.current = null;
}
if (audioRef.current) {
audioRef.current.pause();
audioRef.current = null;
@ -104,5 +364,5 @@ export function useVoiceTTS() {
setState('idle');
}, []);
return { state, speak, cancel };
return { state, speak, speakUrl, cancel, getLevel };
}

View file

@ -0,0 +1,28 @@
import { describe, expect, it } from 'vitest'
import { turnToTranscript } from './agent-transcript'
import { completedTurnLog, created, requested, user } from './session-chat/test-fixtures'
describe('turnToTranscript', () => {
it('maps a completed turn to id/createdAt/summary/items', () => {
const events = completedTurnLog('turn-1', 'sess-1', 'do the thing', 'all done')
const t = turnToTranscript('turn-1', events)
expect(t.id).toBe('turn-1')
expect(t.createdAt).toBeDefined()
expect(t.summary).toBe('all done')
expect(t.error).toBeUndefined()
expect(t.trigger).toBeUndefined()
expect(t.items.length).toBeGreaterThan(0)
})
it('surfaces turn_failed as error with no summary', () => {
const events = [
created('turn-2', 'sess-1', user('go')),
requested('turn-2', 0),
{ type: 'model_call_failed' as const, turnId: 'turn-2', ts: '2026-07-02T10:00:00Z', modelCallIndex: 0, error: 'boom' },
{ type: 'turn_failed' as const, turnId: 'turn-2', ts: '2026-07-02T10:00:00Z', error: 'boom', usage: {} },
]
const t = turnToTranscript('turn-2', events)
expect(t.error).toBe('boom')
expect(t.summary).toBeUndefined()
})
})

View file

@ -0,0 +1,89 @@
import type { z } from 'zod'
import type { Run } from '@x/shared/src/runs.js'
import { reduceTurn, type TurnEvent } from '@x/shared/src/turns.js'
import type { ConversationItem } from '@/lib/chat-conversation'
import { runLogToConversation } from '@/lib/run-to-conversation'
import { buildTurnConversation } from '@/lib/session-chat/turn-view'
// Unified read model for a headless agent run's transcript, whether the id
// is a turn (new runtime) or a legacy run (pre-turn-runtime files under
// $WorkDir/runs/). Used by the background-tasks and live-note history views.
export interface AgentRunTranscript {
id: string
createdAt?: string
// Legacy runs only (subUseCase); turns carry the trigger inside the
// message text instead.
trigger?: string
summary?: string
error?: string
items: ConversationItem[]
}
export function turnToTranscript(
turnId: string,
events: Array<z.infer<typeof TurnEvent>>,
): AgentRunTranscript {
const state = reduceTurn(events)
const out: AgentRunTranscript = {
id: turnId,
createdAt: state.definition.ts,
items: buildTurnConversation(state),
}
if (state.terminal?.type === 'turn_failed') {
out.error = state.terminal.error
}
for (let i = state.modelCalls.length - 1; i >= 0; i--) {
const response = state.modelCalls[i].response
if (!response) continue
const content = response.content
const text =
typeof content === 'string'
? content
: content.map((p) => (p.type === 'text' ? p.text : '')).join('')
if (text) {
out.summary = text
break
}
}
return out
}
export function runToTranscript(run: z.infer<typeof Run>): AgentRunTranscript {
const out: AgentRunTranscript = {
id: run.id,
createdAt: run.createdAt,
trigger: run.subUseCase,
items: runLogToConversation(run.log),
}
for (const event of run.log) {
if (event.type === 'error' && typeof event.error === 'string') {
out.error = event.error
} else if (event.type === 'message' && event.message?.role === 'assistant') {
const content = event.message.content
if (typeof content === 'string') {
out.summary = content
} else if (Array.isArray(content)) {
const text = content
.filter((p) => p.type === 'text')
.map((p) => ('text' in p ? p.text : ''))
.join('')
if (text) out.summary = text
}
}
}
return out
}
// Turn-first with a legacy-run fallback so histories recorded before the
// turn-runtime migration stay readable. The fallback goes away with the
// runs runtime (stage 7).
export async function fetchAgentRunTranscript(id: string): Promise<AgentRunTranscript> {
try {
const turn = await window.ipc.invoke('sessions:getTurn', { turnId: id })
return turnToTranscript(id, turn.events)
} catch {
const run = await window.ipc.invoke('runs:fetch', { runId: id })
return runToTranscript(run)
}
}

View file

@ -69,6 +69,26 @@ export function voiceInputStarted() {
posthog.capture('voice_input_started')
}
export function callStarted(preset: 'voice' | 'video' | 'share' | 'practice') {
posthog.capture('call_started', { preset })
}
// Voice-to-voice latency breakdown for one call turn (all milliseconds):
// utterance accepted → message submitted → first TTS speak() → audio playing.
export function callTurnLatency(props: {
endpointToSubmitMs: number
submitToSpeakMs: number
speakToAudioMs: number
totalMs: number
}) {
posthog.capture('call_turn_latency', {
endpoint_to_submit_ms: Math.round(props.endpointToSubmitMs),
submit_to_speak_ms: Math.round(props.submitToSpeakMs),
speak_to_audio_ms: Math.round(props.speakToAudioMs),
total_ms: Math.round(props.totalMs),
})
}
export function searchExecuted(types: string[]) {
posthog.capture('search_executed', { types })
}

View file

@ -0,0 +1,30 @@
// Tiny synthesized UI sounds for calls — no audio assets, one lazy context.
let ctx: AudioContext | null = null
/**
* Soft rising blip played the instant an utterance is accepted sub-second
* acknowledgment makes the (still ongoing) model turn feel responsive
* instead of dead air.
*/
export function playAckCue() {
try {
if (!ctx) ctx = new AudioContext()
if (ctx.state === 'suspended') void ctx.resume()
const t = ctx.currentTime
const osc = ctx.createOscillator()
const gain = ctx.createGain()
osc.type = 'sine'
osc.frequency.setValueAtTime(880, t)
osc.frequency.exponentialRampToValueAtTime(1320, t + 0.08)
gain.gain.setValueAtTime(0.0001, t)
gain.gain.exponentialRampToValueAtTime(0.08, t + 0.015)
gain.gain.exponentialRampToValueAtTime(0.0001, t + 0.12)
osc.connect(gain)
gain.connect(ctx.destination)
osc.start(t)
osc.stop(t + 0.13)
} catch {
// cosmetic — never let a sound failure affect the call
}
}

View file

@ -10,6 +10,9 @@ export interface MessageAttachment {
mimeType: string
size?: number
thumbnailUrl?: string
/** Live webcam frame from video chat mode rendered as a compact filmstrip.
* Carries no path; thumbnailUrl holds the frame as a data: URL. */
isVideoFrame?: boolean
}
export interface ChatMessage {
@ -198,6 +201,22 @@ const summarizeFilterUpdates = (updates: Record<string, unknown>): string => {
return parts.length > 0 ? parts.join(', ') : 'Updated view'
}
const APP_VIEW_LABELS: Record<string, string> = {
home: 'home',
email: 'email',
meetings: 'meetings',
'live-notes': 'live notes',
'bg-tasks': 'background agents',
'chat-history': 'chat history',
knowledge: 'knowledge',
workspace: 'workspace',
code: 'code',
bases: 'bases',
graph: 'graph',
}
const appViewLabel = (view: unknown): string => APP_VIEW_LABELS[view as string] ?? String(view ?? 'view')
export const getAppActionCardData = (tool: ToolCall): AppActionCardData | null => {
if (tool.name !== 'app-navigation') return null
const result = tool.result as Record<string, unknown> | undefined
@ -209,8 +228,10 @@ export const getAppActionCardData = (tool: ToolCall): AppActionCardData | null =
const action = input.action as string
switch (action) {
case 'open-note': return { action, label: `Opening ${(input.path as string || '').split('/').pop()?.replace(/\.md$/, '') || 'note'}...` }
case 'open-view': return { action, label: `Opening ${input.view} view...` }
case 'open-view': return { action, label: `Opening ${appViewLabel(input.view)}...` }
case 'open-app': return { action, label: `Opening ${input.appId || 'app'}...` }
case 'read-view': return { action, label: `Reading ${appViewLabel(input.view)}...` }
case 'open-item': return { action, label: 'Opening...' }
case 'update-base-view': return { action, label: 'Updating view...' }
case 'create-base': return { action, label: `Creating "${input.name}"...` }
case 'get-base-state': return null // renders as normal tool block
@ -225,9 +246,33 @@ export const getAppActionCardData = (tool: ToolCall): AppActionCardData | null =
return { action: 'open-note', label: `Opened ${name}` }
}
case 'open-view':
return { action: 'open-view', label: `Opened ${result.view} view` }
return { action: 'open-view', label: `Opened ${appViewLabel(result.view)}` }
case 'open-app':
return { action: 'open-app', label: `Opened ${result.appName || result.appId || 'app'}` }
case 'read-view': {
const counted =
(result.threads as unknown[] | undefined)?.length ??
(result.agents as unknown[] | undefined)?.length ??
(result.sessions as unknown[] | undefined)?.length
return {
action: 'read-view',
label: counted !== undefined
? `Read ${appViewLabel(result.view)} (${counted} item${counted === 1 ? '' : 's'})`
: `Read ${appViewLabel(result.view)}`,
}
}
case 'open-item': {
switch (result.kind) {
case 'email-thread': return { action: 'open-item', label: 'Opened email thread' }
case 'note': {
const name = (result.path as string || '').split('/').pop()?.replace(/\.md$/, '') || 'note'
return { action: 'open-item', label: `Opened ${name}` }
}
case 'bg-task': return { action: 'open-item', label: `Opened agent "${result.taskName}"` }
case 'session': return { action: 'open-item', label: 'Opened chat' }
default: return { action: 'open-item', label: 'Opened item' }
}
}
case 'update-base-view':
return {
action: 'update-base-view',

View file

@ -0,0 +1,33 @@
export async function finalizeDeepgramStream(ws: WebSocket | null, timeoutMs = 1800): Promise<void> {
if (!ws || ws.readyState !== WebSocket.OPEN) return;
await new Promise<void>((resolve) => {
let done = false;
let timeout: ReturnType<typeof setTimeout>;
const finish = () => {
if (done) return;
done = true;
clearTimeout(timeout);
ws.removeEventListener('message', onMessage);
resolve();
};
timeout = setTimeout(finish, timeoutMs);
const onMessage = (event: MessageEvent) => {
try {
const data = JSON.parse(event.data);
if (data?.is_final || data?.speech_final || data?.type === 'UtteranceEnd') {
finish();
}
} catch {
// Ignore non-JSON control frames.
}
};
ws.addEventListener('message', onMessage);
try {
ws.send(JSON.stringify({ type: 'Finalize' }));
} catch {
finish();
}
});
}

View file

@ -41,6 +41,9 @@ export function runLogToConversation(log: RunLog): ConversationItem[] {
filename?: string
mimeType?: string
size?: number
data?: string
mediaType?: string
source?: string
toolCallId?: string
toolName?: string
arguments?: unknown
@ -52,13 +55,24 @@ export function runLogToConversation(log: RunLog): ConversationItem[] {
.join('')
const attachmentParts = parts.filter((p) => p.type === 'attachment' && p.path)
if (attachmentParts.length > 0) {
msgAttachments = attachmentParts.map((p) => ({
path: p.path!,
filename: p.filename || p.path!.split('/').pop() || p.path!,
mimeType: p.mimeType || 'application/octet-stream',
size: p.size,
}))
// Video-mode webcam frames — inline base64 image parts, shown as a filmstrip
const imageParts = parts.filter((p) => p.type === 'image' && p.data)
if (attachmentParts.length > 0 || imageParts.length > 0) {
msgAttachments = [
...attachmentParts.map((p) => ({
path: p.path!,
filename: p.filename || p.path!.split('/').pop() || p.path!,
mimeType: p.mimeType || 'application/octet-stream',
size: p.size,
})),
...imageParts.map((p, index) => ({
path: '',
filename: `${p.source === 'screen' ? 'screen' : 'camera'}-frame-${index + 1}.jpg`,
mimeType: p.mediaType || 'image/jpeg',
thumbnailUrl: `data:${p.mediaType || 'image/jpeg'};base64,${p.data}`,
isVideoFrame: true,
})),
]
}
if (msg.role === 'assistant') {

View file

@ -0,0 +1,70 @@
import type { z } from 'zod'
import type { UserMessage } from '@x/shared/src/message.js'
import type { SessionIndexEntry, SessionState } from '@x/shared/src/sessions.js'
import type { JsonValue, RequestedAgent, TurnEvent } from '@x/shared/src/turns.js'
// Narrow, injectable surface over the sessions IPC channels so stores are
// testable with a plain fake instead of a window.ipc stub.
export interface SendMessageConfig {
agent: z.infer<typeof RequestedAgent>
autoPermission?: boolean
maxModelCalls?: number
}
export interface SessionsClient {
create(input: { title?: string }): Promise<{ sessionId: string }>
list(): Promise<{ sessions: SessionIndexEntry[] }>
get(sessionId: string): Promise<SessionState>
getTurn(turnId: string): Promise<{ turnId: string; events: Array<z.infer<typeof TurnEvent>> }>
sendMessage(
sessionId: string,
input: z.infer<typeof UserMessage>,
config: SendMessageConfig,
): Promise<{ turnId: string }>
respondToPermission(
turnId: string,
toolCallId: string,
decision: 'allow' | 'deny',
metadata?: JsonValue,
): Promise<void>
respondToAskHuman(turnId: string, toolCallId: string, answer: string): Promise<void>
stopTurn(turnId: string, reason?: string): Promise<void>
resumeTurn(sessionId: string): Promise<void>
setTitle(sessionId: string, title: string): Promise<void>
delete(sessionId: string): Promise<void>
}
export const ipcSessionsClient: SessionsClient = {
create: (input) => window.ipc.invoke('sessions:create', input),
list: () => window.ipc.invoke('sessions:list', {}),
get: (sessionId) => window.ipc.invoke('sessions:get', { sessionId }),
getTurn: (turnId) => window.ipc.invoke('sessions:getTurn', { turnId }),
sendMessage: (sessionId, input, config) =>
window.ipc.invoke('sessions:sendMessage', { sessionId, input, config }),
respondToPermission: async (turnId, toolCallId, decision, metadata) => {
await window.ipc.invoke('sessions:respondToPermission', {
turnId,
toolCallId,
decision,
...(metadata === undefined ? {} : { metadata }),
})
},
respondToAskHuman: async (turnId, toolCallId, answer) => {
await window.ipc.invoke('sessions:respondToAskHuman', { turnId, toolCallId, answer })
},
stopTurn: async (turnId, reason) => {
await window.ipc.invoke('sessions:stopTurn', {
turnId,
...(reason === undefined ? {} : { reason }),
})
},
resumeTurn: async (sessionId) => {
await window.ipc.invoke('sessions:resumeTurn', { sessionId })
},
setTitle: async (sessionId, title) => {
await window.ipc.invoke('sessions:setTitle', { sessionId, title })
},
delete: async (sessionId) => {
await window.ipc.invoke('sessions:delete', { sessionId })
},
}

View file

@ -0,0 +1,52 @@
import { describe, expect, it } from 'vitest'
import type { SessionBusEvent } from '@x/shared/src/sessions.js'
import { createSessionFeed, type SessionFeedListener } from './feed'
const event: SessionBusEvent = {
kind: 'index-changed',
sessionId: 's1',
entry: null,
}
describe('createSessionFeed', () => {
it('starts the source lazily on first subscribe and fans events out', () => {
let sourceStarted = 0
let push: SessionFeedListener = () => undefined
const feed = createSessionFeed((listener) => {
sourceStarted += 1
push = listener
return () => undefined
})
expect(sourceStarted).toBe(0)
const seenA: SessionBusEvent[] = []
const seenB: SessionBusEvent[] = []
feed.subscribe((e) => seenA.push(e))
feed.subscribe((e) => seenB.push(e))
expect(sourceStarted).toBe(1) // one shared IPC listener
push(event)
expect(seenA).toEqual([event])
expect(seenB).toEqual([event])
})
it('unsubscribes cleanly and isolates a throwing listener', () => {
let push: SessionFeedListener = () => undefined
const feed = createSessionFeed((listener) => {
push = listener
return () => undefined
})
const seen: SessionBusEvent[] = []
feed.subscribe(() => {
throw new Error('bad subscriber')
})
const unsubscribe = feed.subscribe((e) => seen.push(e))
push(event)
expect(seen).toEqual([event])
unsubscribe()
push(event)
expect(seen).toEqual([event])
})
})

View file

@ -0,0 +1,42 @@
import type { SessionBusEvent } from '@x/shared/src/sessions.js'
export type SessionFeedListener = (event: SessionBusEvent) => void
export type SessionFeedSource = (listener: SessionFeedListener) => () => void
// One shared consumer of the sessions:events push channel; stores tap this
// fan-out instead of each opening their own IPC listener. Factory so tests
// can drive a fake source.
export function createSessionFeed(source: SessionFeedSource) {
const listeners = new Set<SessionFeedListener>()
let detach: (() => void) | null = null
const ensureStarted = () => {
if (detach) return
detach = source((event) => {
// Copy so (un)subscribing during dispatch is safe.
for (const listener of [...listeners]) {
try {
listener(event)
} catch {
// A misbehaving subscriber must never break the feed.
}
}
})
}
return {
subscribe(listener: SessionFeedListener): () => void {
ensureStarted()
listeners.add(listener)
return () => {
listeners.delete(listener)
}
},
}
}
const appFeed = createSessionFeed((listener) => window.ipc.on('sessions:events', listener))
export function subscribeSessionFeed(listener: SessionFeedListener): () => void {
return appFeed.subscribe(listener)
}

View file

@ -0,0 +1,314 @@
import { describe, expect, it } from 'vitest'
import type { SessionBusEvent, SessionState } from '@x/shared/src/sessions.js'
import { isChatMessage } from '@/lib/chat-conversation'
import type { SessionsClient } from './client'
import type { SessionFeedListener } from './feed'
import { SessionChatStore, SessionListStore } from './store'
import {
assistantText,
completed,
completedTurnLog,
created,
indexEntry,
requested,
sessionState,
turnCompleted,
user,
type TEvent,
} from './test-fixtures'
const S1 = 'sess-1'
class FakeClient implements SessionsClient {
sessions = new Map<string, SessionState>()
turns = new Map<string, TEvent[]>()
calls: Array<{ method: string; args: unknown[] }> = []
// When set, get() defers until the returned resolver is invoked.
deferredGet: (() => void) | null = null
private record(method: string, ...args: unknown[]) {
this.calls.push({ method, args })
}
async create(input: { title?: string }) {
this.record('create', input)
return { sessionId: 'new-session' }
}
async list() {
this.record('list')
return { sessions: [...this.sessions.keys()].map((id) => indexEntry(id)) }
}
async get(sessionId: string) {
this.record('get', sessionId)
if (this.deferredGet === null) {
const state = this.sessions.get(sessionId)
if (!state) throw new Error(`session not found: ${sessionId}`)
return state
}
return new Promise<SessionState>((resolve, reject) => {
this.deferredGet = () => {
const state = this.sessions.get(sessionId)
if (state) resolve(state)
else reject(new Error(`session not found: ${sessionId}`))
}
})
}
async getTurn(turnId: string) {
this.record('getTurn', turnId)
const events = this.turns.get(turnId)
if (!events) throw new Error(`turn not found: ${turnId}`)
return { turnId, events }
}
async sendMessage(sessionId: string, input: unknown, config: unknown) {
this.record('sendMessage', sessionId, input, config)
return { turnId: 'turn-next' }
}
async respondToPermission(...args: unknown[]) {
this.record('respondToPermission', ...args)
}
async respondToAskHuman(...args: unknown[]) {
this.record('respondToAskHuman', ...args)
}
async stopTurn(...args: unknown[]) {
this.record('stopTurn', ...args)
}
async resumeTurn(...args: unknown[]) {
this.record('resumeTurn', ...args)
}
async setTitle(...args: unknown[]) {
this.record('setTitle', ...args)
}
async delete(...args: unknown[]) {
this.record('delete', ...args)
}
}
function makeStore() {
const client = new FakeClient()
let emit: SessionFeedListener = () => undefined
let subscribed = 0
let unsubscribed = 0
const store = new SessionChatStore({
client,
subscribeFeed: (listener) => {
subscribed += 1
emit = listener
return () => {
unsubscribed += 1
emit = () => undefined
}
},
})
const disconnect = store.connect()
return {
client,
store,
disconnect,
emit: (event: SessionBusEvent) => emit(event),
getSubscribed: () => subscribed,
getUnsubscribed: () => unsubscribed,
}
}
function turnEvent(
sessionId: string,
turnId: string,
event:
| TEvent
| { type: 'text_delta' | 'reasoning_delta'; turnId: string; modelCallIndex: number; delta: string },
): SessionBusEvent {
return { kind: 'turn-event', sessionId, turnId, event }
}
describe('SessionChatStore', () => {
it('seeds a session: prior turns frozen, latest live, conversation composed', async () => {
const { client, store } = makeStore()
client.sessions.set(S1, sessionState(S1, ['turn-1', 'turn-2']))
client.turns.set('turn-1', completedTurnLog('turn-1', S1, 'first?', 'first answer'))
client.turns.set('turn-2', completedTurnLog('turn-2', S1, 'second?', 'second answer'))
await store.setSession(S1)
const snapshot = store.getSnapshot()
expect(snapshot.loading).toBe(false)
expect(snapshot.latestTurnId).toBe('turn-2')
expect(
snapshot.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['first?', 'first answer', 'second?', 'second answer'])
expect(snapshot.chatState?.isProcessing).toBe(false)
})
it('applies live durable events through the shared reducer', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1, user('go'))))
emit(turnEvent(S1, 'turn-1', requested('turn-1', 0)))
expect(store.getSnapshot().chatState?.isThinking).toBe(true)
emit(turnEvent(S1, 'turn-1', completed('turn-1', 0, assistantText('done'))))
emit(turnEvent(S1, 'turn-1', turnCompleted('turn-1')))
const snapshot = store.getSnapshot()
expect(snapshot.latestTurnId).toBe('turn-1')
expect(snapshot.chatState?.isProcessing).toBe(false)
expect(
snapshot.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['go', 'done'])
})
it('accumulates text deltas and clears them on the canonical response', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1, user('go'))))
emit(turnEvent(S1, 'turn-1', requested('turn-1', 0)))
emit(turnEvent(S1, 'turn-1', { type: 'text_delta', turnId: 'turn-1', modelCallIndex: 0, delta: 'he' }))
emit(turnEvent(S1, 'turn-1', { type: 'text_delta', turnId: 'turn-1', modelCallIndex: 0, delta: 'y' }))
expect(store.getSnapshot().chatState?.currentAssistantMessage).toBe('hey')
emit(turnEvent(S1, 'turn-1', completed('turn-1', 0, assistantText('hey'))))
expect(store.getSnapshot().chatState?.currentAssistantMessage).toBe('')
})
it('freezes the previous latest turn when a new turn starts', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, ['turn-1']))
client.turns.set('turn-1', completedTurnLog('turn-1', S1, 'q1', 'a1'))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-2', created('turn-2', S1, user('q2'))))
const snapshot = store.getSnapshot()
expect(snapshot.latestTurnId).toBe('turn-2')
expect(
snapshot.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['q1', 'a1', 'q2'])
})
it('ignores events for other sessions', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent('other-session', 'turn-x', created('turn-x', 'other-session')))
expect(store.getSnapshot().chatState?.conversation).toEqual([])
})
it('reconciles an unknown mid-turn event by refetching the turn', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
// The feed attached mid-turn: we never saw turn_created for turn-9.
client.turns.set('turn-9', [
created('turn-9', S1, user('missed')),
requested('turn-9', 0),
])
emit(turnEvent(S1, 'turn-9', completed('turn-9', 0, assistantText('caught up'))))
await Promise.resolve()
await Promise.resolve()
expect(store.getSnapshot().latestTurnId).toBe('turn-9')
})
it('routes actions against the latest turn', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1)))
await store.sendMessage(user('next'), { agent: { agentId: 'copilot' } })
await store.respondToPermission('tc1', 'allow')
await store.answerAskHuman('ah1', '42')
await store.stop()
expect(client.calls.filter((c) => c.method !== 'get' && c.method !== 'getTurn')).toEqual([
{ method: 'sendMessage', args: [S1, user('next'), { agent: { agentId: 'copilot' } }] },
{ method: 'respondToPermission', args: ['turn-1', 'tc1', 'allow', undefined] },
{ method: 'respondToAskHuman', args: ['turn-1', 'ah1', '42'] },
{ method: 'stopTurn', args: ['turn-1'] },
])
})
it('rejects sendMessage without an active session', async () => {
const { store } = makeStore()
await expect(
store.sendMessage(user('x'), { agent: { agentId: 'copilot' } }),
).rejects.toThrowError('No active session')
})
it('drops stale loads after a session switch', async () => {
const { client, store } = makeStore()
client.sessions.set(S1, sessionState(S1, ['turn-1']))
client.turns.set('turn-1', completedTurnLog('turn-1', S1, 'old', 'old answer'))
client.sessions.set('sess-2', sessionState('sess-2', []))
client.deferredGet = () => undefined // arm deferral
const first = store.setSession(S1)
const release = client.deferredGet
client.deferredGet = null
const second = store.setSession('sess-2')
release?.() // S1's get resolves after the switch
await Promise.all([first, second])
expect(store.getSnapshot().sessionId).toBe('sess-2')
expect(store.getSnapshot().chatState?.conversation).toEqual([])
})
it('surfaces load errors and disconnects its feed subscription', async () => {
const { store, disconnect, getUnsubscribed } = makeStore()
await store.setSession('missing-session')
expect(store.getSnapshot().error).toMatch(/session not found/)
disconnect()
expect(getUnsubscribed()).toBe(1)
void store
})
it('survives a StrictMode-style connect -> cleanup -> connect cycle', async () => {
// React StrictMode runs every effect's mount/cleanup/mount cycle in dev;
// the feed must re-attach or the store goes permanently deaf (the bug
// this test pins: a constructor-made subscription torn down by the first
// cleanup was never restored, leaving the chat stuck at "Thinking…").
const { client, store, disconnect, emit, getSubscribed } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
disconnect()
const disconnect2 = store.connect()
expect(getSubscribed()).toBe(2)
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1, user('go'))))
emit(turnEvent(S1, 'turn-1', requested('turn-1', 0)))
emit(turnEvent(S1, 'turn-1', completed('turn-1', 0, assistantText('done'))))
emit(turnEvent(S1, 'turn-1', turnCompleted('turn-1')))
expect(store.getSnapshot().chatState?.isProcessing).toBe(false)
expect(
store.getSnapshot().chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['go', 'done'])
disconnect2()
})
})
describe('SessionListStore', () => {
it('loads, applies index updates and deletions, and sorts by updatedAt', async () => {
const client = new FakeClient()
client.sessions.set('a', sessionState('a', []))
client.sessions.set('b', sessionState('b', []))
let emit: SessionFeedListener = () => undefined
const store = new SessionListStore({
client,
subscribeFeed: (listener) => {
emit = listener
return () => undefined
},
})
store.connect()
await store.load()
expect(store.getSnapshot().sessions.map((s) => s.sessionId).sort()).toEqual(['a', 'b'])
emit({
kind: 'index-changed',
sessionId: 'c',
entry: indexEntry('c', { updatedAt: '2026-07-03T00:00:00Z' }),
})
expect(store.getSnapshot().sessions[0].sessionId).toBe('c')
emit({ kind: 'index-changed', sessionId: 'a', entry: null })
expect(store.getSnapshot().sessions.map((s) => s.sessionId)).toEqual(['c', 'b'])
})
})

View file

@ -0,0 +1,314 @@
import type { z } from 'zod'
import type { UserMessage } from '@x/shared/src/message.js'
import type { SessionBusEvent, SessionIndexEntry } from '@x/shared/src/sessions.js'
import {
reduceTurn,
type JsonValue,
type TurnEvent,
type TurnState,
type TurnStreamEvent,
} from '@x/shared/src/turns.js'
import type { SendMessageConfig, SessionsClient } from './client'
import type { SessionFeedListener } from './feed'
import {
applyOverlay,
buildSessionChatState,
emptyOverlay,
type LiveOverlay,
type SessionChatState,
} from './turn-view'
type TEvent = z.infer<typeof TurnEvent>
export interface SessionChatSnapshot {
sessionId: string | null
chatState: SessionChatState | null
latestTurnId: string | null
loading: boolean
error: string | null
}
export interface SessionChatStoreDeps {
client: SessionsClient
subscribeFeed: (listener: SessionFeedListener) => () => void
}
// Framework-agnostic controller for one active session's chat. Owns all the
// logic (seeding via getSession/getTurn, applying live feed events with the
// shared reducer, the ephemeral overlay, action routing); the useSessionChat
// hook is a thin useSyncExternalStore subscription over it.
export class SessionChatStore {
private readonly client: SessionsClient
private readonly subscribeFeed: (listener: SessionFeedListener) => () => void
private feedDisconnect: (() => void) | null = null
private readonly listeners = new Set<() => void>()
private sessionId: string | null = null
// Settled earlier turns, reduced once and frozen.
private priorTurns: TurnState[] = []
// The latest turn's raw event log; re-reduced on each durable event.
private latestEvents: TEvent[] | null = null
private overlay: LiveOverlay = emptyOverlay()
private loading = false
private error: string | null = null
// Guards stale async loads after a session switch.
private generation = 0
private snapshot: SessionChatSnapshot = {
sessionId: null,
chatState: null,
latestTurnId: null,
loading: false,
error: null,
}
constructor(deps: SessionChatStoreDeps) {
this.client = deps.client
this.subscribeFeed = deps.subscribeFeed
}
// Feed attachment is effect-managed and idempotent so React StrictMode's
// mount -> cleanup -> mount cycle re-attaches cleanly (a constructor-made
// subscription would be torn down by the first cleanup and never restored).
connect(): () => void {
if (!this.feedDisconnect) {
this.feedDisconnect = this.subscribeFeed(this.onFeedEvent)
}
return () => {
this.feedDisconnect?.()
this.feedDisconnect = null
}
}
subscribe = (onChange: () => void): (() => void) => {
this.listeners.add(onChange)
return () => {
this.listeners.delete(onChange)
}
}
getSnapshot = (): SessionChatSnapshot => this.snapshot
async setSession(sessionId: string | null): Promise<void> {
if (sessionId === this.sessionId) return
this.generation += 1
const generation = this.generation
this.sessionId = sessionId
this.priorTurns = []
this.latestEvents = null
this.overlay = emptyOverlay()
this.error = null
this.loading = sessionId !== null
this.emit()
if (sessionId === null) return
try {
const state = await this.client.get(sessionId)
const turns = await Promise.all(
state.turns.map((ref) => this.client.getTurn(ref.turnId)),
)
if (generation !== this.generation) return
const reduced = turns.map((turn) => reduceTurn(turn.events))
this.priorTurns = reduced.slice(0, -1)
this.latestEvents = turns.length > 0 ? turns[turns.length - 1].events : null
this.loading = false
this.emit()
} catch (error) {
if (generation !== this.generation) return
console.error('[session-chat] failed to load session', sessionId, error)
this.loading = false
this.error = error instanceof Error ? error.message : String(error)
this.emit()
}
}
private onFeedEvent: SessionFeedListener = (event: SessionBusEvent) => {
if (event.kind !== 'turn-event' || event.sessionId !== this.sessionId) return
const turnEvent = event.event
if (isDurable(turnEvent)) {
if (turnEvent.type === 'turn_created') {
// A new turn started for this session: freeze the previous latest.
this.freezeLatest()
this.latestEvents = [turnEvent]
this.overlay = emptyOverlay()
} else if (this.latestEvents && this.latestEvents[0].turnId === turnEvent.turnId) {
this.latestEvents.push(turnEvent)
} else {
// An event for a turn we haven't seen (missed turn_created, e.g. the
// feed attached mid-turn): reconcile by refetching that turn.
void this.reloadTurn(event.turnId)
return
}
}
this.overlay = applyOverlay(this.overlay, turnEvent)
this.emit()
}
private freezeLatest(): void {
if (!this.latestEvents) return
try {
this.priorTurns = [...this.priorTurns, reduceTurn(this.latestEvents)]
} catch {
// A turn we can't reduce is dropped from history rather than wedging
// the whole conversation.
}
this.latestEvents = null
}
private async reloadTurn(turnId: string): Promise<void> {
const generation = this.generation
try {
const turn = await this.client.getTurn(turnId)
if (generation !== this.generation) return
if (this.latestEvents && this.latestEvents[0].turnId !== turnId) {
this.freezeLatest()
}
this.latestEvents = turn.events
this.emit()
} catch (error) {
// The next snapshot-worthy event will retry.
console.error('[session-chat] failed to reload turn', turnId, error)
}
}
// ── Actions ─────────────────────────────────────────────────────────────
sendMessage = async (
input: z.infer<typeof UserMessage>,
config: SendMessageConfig,
): Promise<{ turnId: string }> => {
if (!this.sessionId) throw new Error('No active session')
return this.client.sendMessage(this.sessionId, input, config)
}
respondToPermission = async (
toolCallId: string,
decision: 'allow' | 'deny',
metadata?: JsonValue,
): Promise<void> => {
const turnId = this.snapshot.latestTurnId
if (!turnId) return
await this.client.respondToPermission(turnId, toolCallId, decision, metadata)
}
answerAskHuman = async (toolCallId: string, answer: string): Promise<void> => {
const turnId = this.snapshot.latestTurnId
if (!turnId) return
await this.client.respondToAskHuman(turnId, toolCallId, answer)
}
stop = async (): Promise<void> => {
const turnId = this.snapshot.latestTurnId
if (!turnId) return
await this.client.stopTurn(turnId)
}
// ── Derivation ──────────────────────────────────────────────────────────
private emit(): void {
this.snapshot = this.derive()
for (const listener of [...this.listeners]) {
listener()
}
}
private derive(): SessionChatSnapshot {
let turns = this.priorTurns
let error = this.error
if (this.latestEvents) {
try {
turns = [...this.priorTurns, reduceTurn(this.latestEvents)]
} catch (reduceError) {
error =
reduceError instanceof Error ? reduceError.message : String(reduceError)
}
}
const latest = turns[turns.length - 1]
return {
sessionId: this.sessionId,
chatState:
this.sessionId !== null && !this.loading
? buildSessionChatState(turns, this.overlay)
: null,
latestTurnId: latest?.definition.turnId ?? null,
loading: this.loading,
error,
}
}
}
function isDurable(event: TurnStreamEvent): event is TEvent {
return event.type !== 'text_delta' && event.type !== 'reasoning_delta'
}
// ---------------------------------------------------------------------------
// Session list store
// ---------------------------------------------------------------------------
export interface SessionListSnapshot {
sessions: SessionIndexEntry[]
loading: boolean
}
export class SessionListStore {
private readonly client: SessionsClient
private readonly subscribeFeed: (listener: SessionFeedListener) => () => void
private feedDisconnect: (() => void) | null = null
private readonly listeners = new Set<() => void>()
private entries = new Map<string, SessionIndexEntry>()
private loading = true
private snapshot: SessionListSnapshot = { sessions: [], loading: true }
constructor(deps: SessionChatStoreDeps) {
this.client = deps.client
this.subscribeFeed = deps.subscribeFeed
}
connect(): () => void {
if (!this.feedDisconnect) {
this.feedDisconnect = this.subscribeFeed(this.onFeedEvent)
}
return () => {
this.feedDisconnect?.()
this.feedDisconnect = null
}
}
subscribe = (onChange: () => void): (() => void) => {
this.listeners.add(onChange)
return () => {
this.listeners.delete(onChange)
}
}
getSnapshot = (): SessionListSnapshot => this.snapshot
async load(): Promise<void> {
const { sessions } = await this.client.list()
this.entries = new Map(sessions.map((entry) => [entry.sessionId, entry]))
this.loading = false
this.emit()
}
private onFeedEvent: SessionFeedListener = (event: SessionBusEvent) => {
if (event.kind !== 'index-changed') return
if (event.entry === null) {
this.entries.delete(event.sessionId)
} else {
this.entries.set(event.sessionId, event.entry)
}
this.emit()
}
private emit(): void {
this.snapshot = {
sessions: [...this.entries.values()].sort((a, b) =>
b.updatedAt.localeCompare(a.updatedAt),
),
loading: this.loading,
}
for (const listener of [...this.listeners]) {
listener()
}
}
}

View file

@ -0,0 +1,171 @@
import type { z } from 'zod'
import type { SessionIndexEntry, SessionState } from '@x/shared/src/sessions.js'
import type { ResolvedAgent, TurnEvent } from '@x/shared/src/turns.js'
// Compact turn-event builders for renderer tests. Sequences must satisfy the
// shared reducer's invariants (reduceTurn is what the store runs on them).
export type TEvent = z.infer<typeof TurnEvent>
export const TS = '2026-07-02T10:00:00Z'
export const FIXTURE_AGENT: z.infer<typeof ResolvedAgent> = {
agentId: 'copilot',
systemPrompt: 'SYS',
model: { provider: 'openai', model: 'gpt-fixture' },
tools: [],
}
export function user(text: string) {
return { role: 'user' as const, content: text }
}
export function assistantText(text: string) {
return { role: 'assistant' as const, content: text }
}
export function toolCallPart(id: string, name: string, args: unknown = {}) {
return { type: 'tool-call' as const, toolCallId: id, toolName: name, arguments: args }
}
export function created(
turnId: string,
sessionId: string,
input: ReturnType<typeof user> = user('hello'),
): TEvent {
return {
type: 'turn_created',
schemaVersion: 1,
turnId,
ts: TS,
sessionId,
agent: { requested: { agentId: 'copilot' }, resolved: FIXTURE_AGENT },
context: [],
input,
config: { autoPermission: false, humanAvailable: true, maxModelCalls: 20 },
}
}
export function requested(
turnId: string,
index: number,
refs: string[] = ['input'],
): TEvent {
return {
type: 'model_call_requested',
turnId,
ts: TS,
modelCallIndex: index,
request: {
messages: refs,
parameters: {},
},
}
}
export function completed(
turnId: string,
index: number,
message: { role: 'assistant'; content: unknown },
): TEvent {
return {
type: 'model_call_completed',
turnId,
ts: TS,
modelCallIndex: index,
message: message as never,
finishReason: 'stop',
usage: {},
}
}
export function invocation(turnId: string, toolCallId: string, name: string): TEvent {
return {
type: 'tool_invocation_requested',
turnId,
ts: TS,
toolCallId,
toolId: `builtin:${name}`,
toolName: name,
execution: 'sync',
input: {},
}
}
export function toolResult(
turnId: string,
toolCallId: string,
name: string,
output: unknown = 'ok',
): TEvent {
return {
type: 'tool_result',
turnId,
ts: TS,
toolCallId,
toolName: name,
source: 'sync',
result: { output: output as never, isError: false },
}
}
export function turnCompleted(turnId: string, text = 'done'): TEvent {
return {
type: 'turn_completed',
turnId,
ts: TS,
output: assistantText(text),
finishReason: 'stop',
usage: {},
}
}
// A settled single-response turn.
export function completedTurnLog(
turnId: string,
sessionId: string,
question: string,
answer: string,
): TEvent[] {
return [
created(turnId, sessionId, user(question)),
requested(turnId, 0),
completed(turnId, 0, assistantText(answer)),
turnCompleted(turnId, answer),
]
}
export function indexEntry(
sessionId: string,
overrides: Partial<SessionIndexEntry> = {},
): SessionIndexEntry {
return {
sessionId,
title: `Session ${sessionId}`,
createdAt: TS,
updatedAt: TS,
turnCount: 1,
latestTurnStatus: 'completed',
...overrides,
}
}
export function sessionState(
sessionId: string,
turnIds: string[],
): SessionState {
return {
definition: { type: 'session_created', schemaVersion: 1, sessionId, ts: TS },
title: `Session ${sessionId}`,
turns: turnIds.map((turnId, i) => ({
turnId,
sessionSeq: i + 1,
agentId: 'copilot',
model: FIXTURE_AGENT.model,
ts: TS,
})),
latestTurnId: turnIds[turnIds.length - 1],
createdAt: TS,
updatedAt: TS,
}
}

View file

@ -0,0 +1,333 @@
import { describe, expect, it } from 'vitest'
import { reduceTurn } from '@x/shared/src/turns.js'
import { isChatMessage, isErrorMessage, isToolCall } from '@/lib/chat-conversation'
import {
applyOverlay,
buildSessionChatState,
buildTurnConversation,
emptyOverlay,
} from './turn-view'
import {
TS,
assistantText,
completed,
completedTurnLog,
created,
invocation,
requested,
toolCallPart,
toolResult,
turnCompleted,
user,
type TEvent,
} from './test-fixtures'
const T1 = 'turn-1'
const S1 = 'sess-1'
function assistantCalls(...parts: Array<ReturnType<typeof toolCallPart>>) {
return { role: 'assistant' as const, content: parts }
}
describe('applyOverlay', () => {
it('accumulates text and reasoning deltas', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, { type: 'text_delta', turnId: T1, modelCallIndex: 0, delta: 'he' })
overlay = applyOverlay(overlay, { type: 'text_delta', turnId: T1, modelCallIndex: 0, delta: 'y' })
overlay = applyOverlay(overlay, {
type: 'reasoning_delta',
turnId: T1,
modelCallIndex: 0,
delta: 'hmm',
})
expect(overlay.text).toBe('hey')
expect(overlay.reasoning).toBe('hmm')
})
it('clears text/reasoning when the canonical model response arrives', () => {
let overlay = { ...emptyOverlay(), text: 'streaming', reasoning: 'thinking' }
overlay = applyOverlay(overlay, completed(T1, 0, assistantText('final')))
expect(overlay.text).toBe('')
expect(overlay.reasoning).toBe('')
})
it('accumulates tool-output progress chunks and drops them on the terminal result', () => {
let overlay = emptyOverlay()
const progress = (chunk: string): TEvent => ({
type: 'tool_progress',
turnId: T1,
ts: TS,
toolCallId: 'tc1',
source: 'sync',
progress: { kind: 'tool-output', chunk },
})
overlay = applyOverlay(overlay, progress('$ ls\n'))
overlay = applyOverlay(overlay, progress('README.md\n'))
expect(overlay.toolOutput.tc1).toBe('$ ls\nREADME.md\n')
overlay = applyOverlay(overlay, toolResult(T1, 'tc1', 'executeCommand'))
expect(overlay.toolOutput.tc1).toBeUndefined()
})
it('ignores non-output progress payloads', () => {
const overlay = applyOverlay(emptyOverlay(), {
type: 'tool_progress',
turnId: T1,
ts: TS,
toolCallId: 'tc1',
source: 'sync',
progress: { pct: 50 },
})
expect(overlay.toolOutput).toEqual({})
})
})
describe('voice output', () => {
const delta = (d: string): Parameters<typeof applyOverlay>[1] =>
({ type: 'text_delta', turnId: T1, modelCallIndex: 0, delta: d })
it('extracts completed <voice> blocks across split deltas', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('Sure! <voi'))
expect(overlay.voiceSegments).toEqual([])
overlay = applyOverlay(overlay, delta('ce>hello there</voice> and <voice>bye'))
expect(overlay.voiceSegments).toEqual(['hello there'])
overlay = applyOverlay(overlay, delta('</voice> done'))
expect(overlay.voiceSegments).toEqual(['hello there', 'bye'])
})
it('emits an early clause from a long open block, then the remainder on close', () => {
let overlay = emptyOverlay()
const longClause = 'Okay so the first thing I would look at here is the error message,'
overlay = applyOverlay(overlay, delta(`<voice>${longClause} because`))
// Open block crossed the early-speech threshold at a clause boundary.
expect(overlay.voiceSegments).toEqual([longClause])
overlay = applyOverlay(overlay, delta(' it tells you the root cause.</voice>'))
// Remainder only — the early clause is not repeated.
expect(overlay.voiceSegments).toEqual([longClause, 'because it tells you the root cause.'])
})
it('does not emit early clauses from short open blocks', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('<voice>Sure, one sec'))
expect(overlay.voiceSegments).toEqual([])
overlay = applyOverlay(overlay, delta('.</voice>'))
expect(overlay.voiceSegments).toEqual(['Sure, one sec.'])
})
it('keeps segments but resets the scan on model_call_completed', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('<voice>one</voice>'))
overlay = applyOverlay(overlay, completed(T1, 0, assistantText('<voice>one</voice>')))
expect(overlay.voiceSegments).toEqual(['one'])
expect(overlay.text).toBe('')
overlay = applyOverlay(overlay, delta('<voice>two</voice>'))
expect(overlay.voiceSegments).toEqual(['one', 'two'])
})
it('strips voice tags from the streaming message and exposes segments on state', () => {
const turn = reduceTurn([created(T1, S1), requested(T1, 0)])
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('Plan: <voice>speak this</voice> rest'))
const state = buildSessionChatState([turn], overlay)
expect(state.currentAssistantMessage).toBe('Plan: speak this rest')
expect(state.voiceSegments).toEqual(['speak this'])
})
it('strips voice tags from persisted assistant messages', () => {
const state = reduceTurn(
completedTurnLog(T1, S1, 'q', 'Sure. <voice>Here you go.</voice> Done.'),
)
const items = buildTurnConversation(state)
const assistant = items.find((i) => isChatMessage(i) && i.role === 'assistant')
expect(isChatMessage(assistant!) && assistant.content).toBe('Sure. Here you go. Done.')
})
})
describe('buildTurnConversation', () => {
it('maps user input, assistant text, and settled tool calls', () => {
const call = assistantCalls(toolCallPart('tc1', 'echo', { x: 1 }))
const state = reduceTurn([
created(T1, S1, user('run it')),
requested(T1, 0),
completed(T1, 0, call),
invocation(T1, 'tc1', 'echo'),
toolResult(T1, 'tc1', 'echo', { echoed: true }),
requested(T1, 1, [
'assistant:0',
'toolResult:tc1',
]),
completed(T1, 1, assistantText('all done')),
turnCompleted(T1, 'all done'),
])
const items = buildTurnConversation(state)
expect(items.map((i) => (isToolCall(i) ? `tool:${i.name}:${i.status}` : isChatMessage(i) ? `${i.role}` : 'x'))).toEqual([
'user',
'tool:echo:completed',
'assistant',
])
const tool = items.find(isToolCall)
expect(tool?.result).toEqual({ echoed: true })
expect(tool?.input).toEqual({ x: 1 })
})
it('marks permission-pending tools pending and running tools running', () => {
const state = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('p1', 'executeCommand'), toolCallPart('r1', 'echo'))),
{
type: 'tool_permission_required',
turnId: T1,
ts: TS,
toolCallId: 'p1',
toolName: 'executeCommand',
request: { kind: 'command', commandNames: ['rm'] },
},
invocation(T1, 'r1', 'echo'),
])
const items = buildTurnConversation(state).filter(isToolCall)
expect(items.map((i) => i.status)).toEqual(['pending', 'running'])
})
it('renders user attachments and a failed turn as an error item', () => {
const input = {
role: 'user' as const,
content: [
{ type: 'text' as const, text: 'see file' },
{ type: 'attachment' as const, path: '/tmp/a.png', filename: 'a.png', mimeType: 'image/png' },
],
}
const state = reduceTurn([
created(T1, S1, input as never),
requested(T1, 0),
{ type: 'model_call_failed', turnId: T1, ts: TS, modelCallIndex: 0, error: 'boom' },
{ type: 'turn_failed', turnId: T1, ts: TS, error: 'boom', usage: {} },
])
const items = buildTurnConversation(state)
expect(isChatMessage(items[0]) && items[0].attachments?.[0].filename).toBe('a.png')
expect(isErrorMessage(items[1]) && items[1].message).toBe('boom')
})
})
describe('buildSessionChatState', () => {
it('composes the conversation across turns and derives processing flags', () => {
const prior = reduceTurn(completedTurnLog('turn-1', S1, 'first?', 'first answer'))
const latest = reduceTurn([
created('turn-2', S1, user('second?')),
requested('turn-2', 0),
])
const state = buildSessionChatState([prior, latest], emptyOverlay())
expect(
state.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['first?', 'first answer', 'second?'])
expect(state.isProcessing).toBe(true) // latest turn idle = actively working
expect(state.isThinking).toBe(true)
})
it('is settled (not processing) when the latest turn is terminal', () => {
const turn = reduceTurn(completedTurnLog(T1, S1, 'q', 'a'))
const state = buildSessionChatState([turn], emptyOverlay())
expect(state.isProcessing).toBe(false)
expect(state.isThinking).toBe(false)
})
it('exposes pending permissions as request events; waiting is not thinking', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('p1', 'executeCommand', { command: 'rm -rf /' }))),
{
type: 'tool_permission_required',
turnId: T1,
ts: TS,
toolCallId: 'p1',
toolName: 'executeCommand',
request: { kind: 'command', commandNames: ['rm'] },
},
{
type: 'turn_suspended',
turnId: T1,
ts: TS,
pendingPermissions: [
{ toolCallId: 'p1', toolName: 'executeCommand', request: { kind: 'command', commandNames: ['rm'] } },
],
pendingAsyncTools: [],
usage: {},
},
])
const state = buildSessionChatState([turn], emptyOverlay())
const request = state.allPermissionRequests.get('p1')
expect(request).toMatchObject({
type: 'tool-permission-request',
toolCall: { toolCallId: 'p1', toolName: 'executeCommand' },
permission: { kind: 'command', commandNames: ['rm'] },
})
expect(state.isProcessing).toBe(true)
expect(state.isThinking).toBe(false)
})
it('maps human decisions to responses and classifier decisions to auto-decisions', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('h1', 'echo'), toolCallPart('c1', 'echo'))),
{ type: 'tool_permission_required', turnId: T1, ts: TS, toolCallId: 'h1', toolName: 'echo', request: { kind: 'command', commandNames: ['x'] } },
{ type: 'tool_permission_required', turnId: T1, ts: TS, toolCallId: 'c1', toolName: 'echo', request: { kind: 'command', commandNames: ['y'] } },
{ type: 'tool_permission_resolved', turnId: T1, ts: TS, toolCallId: 'h1', decision: 'allow', source: 'human' },
{ type: 'tool_permission_resolved', turnId: T1, ts: TS, toolCallId: 'c1', decision: 'deny', source: 'classifier', reason: 'risky' },
{ type: 'tool_result', turnId: T1, ts: TS, toolCallId: 'c1', toolName: 'echo', source: 'runtime', result: { output: 'denied', isError: true } },
invocation(T1, 'h1', 'echo'),
toolResult(T1, 'h1', 'echo'),
])
const state = buildSessionChatState([turn], emptyOverlay())
expect(state.permissionResponses.get('h1')).toBe('approve')
expect(state.autoPermissionDecisions.get('c1')).toMatchObject({
decision: 'deny',
reason: 'risky',
})
})
it('exposes pending ask-human calls with question and options', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('ah1', 'ask-human', { question: 'Deploy?', options: ['Yes', 'No'] }))),
{
type: 'tool_invocation_requested',
turnId: T1,
ts: TS,
toolCallId: 'ah1',
toolId: 'builtin:ask-human',
toolName: 'ask-human',
execution: 'async',
input: { question: 'Deploy?', options: ['Yes', 'No'] },
},
])
const state = buildSessionChatState([turn], emptyOverlay())
expect(state.pendingAskHumanRequests.get('ah1')).toMatchObject({
query: 'Deploy?',
options: ['Yes', 'No'],
})
expect(state.isThinking).toBe(false) // waiting on the human, no shimmer
})
it('stitches live tool output onto the matching tool item', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('tc1', 'executeCommand'))),
invocation(T1, 'tc1', 'executeCommand'),
])
const overlay = { ...emptyOverlay(), toolOutput: { tc1: 'partial output' } }
const state = buildSessionChatState([turn], overlay)
const tool = state.conversation.filter(isToolCall)[0]
expect(tool.streamingOutput).toBe('partial output')
})
it('surfaces streaming text as currentAssistantMessage', () => {
const turn = reduceTurn([created(T1, S1), requested(T1, 0)])
const state = buildSessionChatState([turn], { ...emptyOverlay(), text: 'typing…' })
expect(state.currentAssistantMessage).toBe('typing…')
})
})

View file

@ -0,0 +1,395 @@
import type { z } from 'zod'
import type {
AskHumanRequestEvent,
ToolPermissionAutoDecisionEvent,
ToolPermissionMetadata,
ToolPermissionRequestEvent,
} from '@x/shared/src/runs.js'
import {
deriveTurnStatus,
outstandingAsyncTools,
outstandingPermissions,
type ToolCallState,
type TurnState,
type TurnStreamEvent,
} from '@x/shared/src/turns.js'
import type {
ChatMessage,
ConversationItem,
ErrorMessage,
MessageAttachment,
PermissionResponse,
ToolCall,
} from '@/lib/chat-conversation'
// Pure derivations from reduced turn state (+ the ephemeral live overlay) to
// the view shapes the existing chat components already consume. No IPC, no
// React — everything here is unit-testable with plain state values.
// ---------------------------------------------------------------------------
// Live overlay: ephemeral streaming buffers (turn-runtime-design.md §14.4).
// ---------------------------------------------------------------------------
export type LiveOverlay = {
text: string
reasoning: string
toolOutput: Record<string, string>
// Speakable segments seen while streaming, in order, monotonically growing
// for the lifetime of the overlay (i.e. one active turn). Usually the
// contents of completed <voice>…</voice> blocks, but a long still-open
// block may emit an early clause (see EARLY_SPEECH_MIN_CHARS) so speech
// can start before the sentence finishes generating. Consumers speak
// segments beyond what they've already spoken; the overlay reset on turn
// switch starts a fresh list.
voiceSegments: string[]
// Scan cursor into `text` — everything before it has been checked for
// complete voice blocks.
voiceScanIndex: number
// Chars of the currently-open voice block's content already emitted as an
// early clause — the block's remainder (on close) excludes them.
voicePartialConsumed: number
}
export const emptyOverlay = (): LiveOverlay => ({
text: '',
reasoning: '',
toolOutput: {},
voiceSegments: [],
voiceScanIndex: 0,
voicePartialConsumed: 0,
})
// The model emits <voice>…</voice> around speakable text when voice output
// is enabled; tags are never shown to the user.
export function stripVoiceTags(text: string): string {
return text.replace(/<\/?voice>/g, '')
}
const VOICE_BLOCK = /<voice>([\s\S]*?)<\/voice>/g
const VOICE_OPEN_TAG = '<voice>'
// Early speech: once an open block has this many unconsumed chars, its last
// complete clause is emitted immediately instead of waiting for </voice> —
// TTS starts on the first clause while the rest of the sentence generates.
const EARLY_SPEECH_MIN_CHARS = 60
// ...but never emit a fragment shorter than this (prosody suffers).
const EARLY_SPEECH_MIN_EMIT = 30
// Clause boundaries (punctuation, optionally inside closing quote/paren,
// followed by whitespace or end-of-buffer).
const CLAUSE_BOUNDARY = /[,;:.!?…—]["')\]]*(?=\s|$)/g
// Accumulates deltas; canonical durable events supersede the buffers (the
// committed transcript now contains what was streaming).
export function applyOverlay(overlay: LiveOverlay, event: TurnStreamEvent): LiveOverlay {
switch (event.type) {
case 'text_delta': {
const text = overlay.text + event.delta
// Extract complete voice blocks past the scan cursor. Incomplete
// blocks (opening tag seen, closing not yet) stay unconsumed until a
// later delta completes them. The first complete block may have had an
// early clause emitted while it was open — skip those chars.
const segments: string[] = []
let scanIndex = overlay.voiceScanIndex
let partialConsumed = overlay.voicePartialConsumed
VOICE_BLOCK.lastIndex = scanIndex
for (let m = VOICE_BLOCK.exec(text); m; m = VOICE_BLOCK.exec(text)) {
const content = m[1].slice(partialConsumed).trim()
partialConsumed = 0
if (content) segments.push(content)
scanIndex = m.index + m[0].length
}
// Early speech: if a voice block is still open and has accumulated a
// long unconsumed run, emit its last complete clause now — speech can
// start while the rest of the sentence is still generating.
const openIdx = text.indexOf(VOICE_OPEN_TAG, scanIndex)
if (openIdx !== -1) {
const unconsumed = text.slice(openIdx + VOICE_OPEN_TAG.length + partialConsumed)
if (unconsumed.length >= EARLY_SPEECH_MIN_CHARS) {
let lastBoundaryEnd = -1
CLAUSE_BOUNDARY.lastIndex = 0
for (let b = CLAUSE_BOUNDARY.exec(unconsumed); b; b = CLAUSE_BOUNDARY.exec(unconsumed)) {
lastBoundaryEnd = b.index + b[0].length
}
if (lastBoundaryEnd >= EARLY_SPEECH_MIN_EMIT) {
const clause = unconsumed.slice(0, lastBoundaryEnd).trim()
if (clause) segments.push(clause)
partialConsumed += lastBoundaryEnd
}
}
} else {
// No open block — any partial bookkeeping belongs to a block that
// has since closed.
partialConsumed = 0
}
return {
...overlay,
text,
...(segments.length > 0
? { voiceSegments: [...overlay.voiceSegments, ...segments] }
: {}),
voiceScanIndex: scanIndex,
voicePartialConsumed: partialConsumed,
}
}
case 'reasoning_delta':
return { ...overlay, reasoning: overlay.reasoning + event.delta }
case 'model_call_completed':
return { ...overlay, text: '', reasoning: '', voiceScanIndex: 0, voicePartialConsumed: 0 }
case 'tool_progress': {
const progress = event.progress
if (
progress &&
typeof progress === 'object' &&
!Array.isArray(progress) &&
(progress as { kind?: unknown }).kind === 'tool-output' &&
typeof (progress as { chunk?: unknown }).chunk === 'string'
) {
const chunk = (progress as { chunk: string }).chunk
return {
...overlay,
toolOutput: {
...overlay.toolOutput,
[event.toolCallId]: (overlay.toolOutput[event.toolCallId] ?? '') + chunk,
},
}
}
return overlay
}
case 'tool_result': {
if (!(event.toolCallId in overlay.toolOutput)) return overlay
const toolOutput = { ...overlay.toolOutput }
delete toolOutput[event.toolCallId]
return { ...overlay, toolOutput }
}
default:
return overlay
}
}
// ---------------------------------------------------------------------------
// Conversation items
// ---------------------------------------------------------------------------
type UserContent = TurnState['definition']['input']['content']
function extractText(content: UserContent): string {
if (typeof content === 'string') return content
return content
.map((part) => (part.type === 'text' ? part.text : ''))
.filter(Boolean)
.join('\n')
}
function extractAttachments(content: UserContent): MessageAttachment[] | undefined {
if (typeof content === 'string') return undefined
const attachments = content.flatMap((part) =>
part.type === 'attachment'
? [
{
path: part.path,
filename: part.filename,
mimeType: part.mimeType,
...(part.size === undefined ? {} : { size: part.size }),
},
]
: [],
)
return attachments.length > 0 ? attachments : undefined
}
function toolStatus(tc: ToolCallState): ToolCall['status'] {
if (tc.result) return 'completed'
if (tc.permission && !tc.permission.resolved) return 'pending'
return 'running'
}
// One turn's contribution to the conversation: the user input, then per
// completed model call its text and tool calls (with live status/results).
export function buildTurnConversation(state: TurnState): ConversationItem[] {
const items: ConversationItem[] = []
const turnId = state.definition.turnId
let seq = 0
const ts = () => Date.parse(state.definition.ts) + seq++
const userText = extractText(state.definition.input.content)
const attachments = extractAttachments(state.definition.input.content)
if (userText || attachments) {
items.push({
id: `${turnId}:user`,
role: 'user',
content: userText,
timestamp: ts(),
...(attachments ? { attachments } : {}),
} satisfies ChatMessage)
}
const toolCallsById = new Map(state.toolCalls.map((tc) => [tc.toolCallId, tc]))
for (const call of state.modelCalls) {
if (call.response === undefined) continue
const content = call.response.content
// Voice tags are model-facing markup, never shown (parity with the
// legacy path's display-time strip).
const text = stripVoiceTags(
typeof content === 'string'
? content
: content
.map((part) => (part.type === 'text' ? part.text : ''))
.filter(Boolean)
.join('\n'),
)
if (text) {
items.push({
id: `${turnId}:a${call.index}`,
role: 'assistant',
content: text,
timestamp: ts(),
} satisfies ChatMessage)
}
if (Array.isArray(content)) {
for (const part of content) {
if (part.type !== 'tool-call') continue
const tc = toolCallsById.get(part.toolCallId)
items.push({
id: part.toolCallId,
name: part.toolName,
input: part.arguments as ToolCall['input'],
...(tc?.result ? { result: tc.result.result.output as ToolCall['result'] } : {}),
status: tc ? toolStatus(tc) : 'running',
timestamp: ts(),
} satisfies ToolCall)
}
}
}
if (state.terminal?.type === 'turn_failed') {
items.push({
id: `${turnId}:error`,
kind: 'error',
message: state.terminal.error,
timestamp: ts(),
} satisfies ErrorMessage)
}
return items
}
// ---------------------------------------------------------------------------
// Session chat state (superset of the ChatTabViewState fields)
// ---------------------------------------------------------------------------
type PermMeta = z.infer<typeof ToolPermissionMetadata>
export type SessionChatState = {
conversation: ConversationItem[]
currentAssistantMessage: string
// See LiveOverlay.voiceSegments.
voiceSegments: string[]
pendingAskHumanRequests: Map<string, z.infer<typeof AskHumanRequestEvent>>
allPermissionRequests: Map<string, z.infer<typeof ToolPermissionRequestEvent>>
permissionResponses: Map<string, PermissionResponse>
autoPermissionDecisions: Map<string, z.infer<typeof ToolPermissionAutoDecisionEvent>>
// Composer blocked / Stop shown until the latest turn settles (waiting on a
// permission or ask-human still counts as processing).
isProcessing: boolean
// Actively working (non-terminal, nothing waiting on the user) — drives the
// "Thinking…" shimmer; deliberately false under permission/ask-human cards.
isThinking: boolean
}
function toolCallPartOf(tc: ToolCallState) {
return {
type: 'tool-call' as const,
toolCallId: tc.toolCallId,
toolName: tc.toolName,
arguments: tc.input,
}
}
// Compose the whole session's conversation: prior (settled) turns plus the
// latest turn, with the live overlay stitched onto the latest turn's items.
export function buildSessionChatState(
turns: TurnState[],
overlay: LiveOverlay,
): SessionChatState {
const conversation: ConversationItem[] = []
for (const turn of turns) {
conversation.push(...buildTurnConversation(turn))
}
for (let i = 0; i < conversation.length; i++) {
const item = conversation[i]
if ('name' in item && overlay.toolOutput[item.id]) {
conversation[i] = { ...item, streamingOutput: overlay.toolOutput[item.id] }
}
}
const latest = turns[turns.length - 1]
const status = latest ? deriveTurnStatus(latest) : undefined
const latestTurnId = latest?.definition.turnId ?? ''
const allPermissionRequests = new Map<string, z.infer<typeof ToolPermissionRequestEvent>>()
const permissionResponses = new Map<string, PermissionResponse>()
const autoPermissionDecisions = new Map<
string,
z.infer<typeof ToolPermissionAutoDecisionEvent>
>()
const pendingAskHumanRequests = new Map<string, z.infer<typeof AskHumanRequestEvent>>()
if (latest) {
for (const tc of outstandingPermissions(latest)) {
allPermissionRequests.set(tc.toolCallId, {
runId: latestTurnId,
type: 'tool-permission-request',
subflow: [],
toolCall: toolCallPartOf(tc),
permission: tc.permission?.required.request as PermMeta,
})
}
for (const tc of latest.toolCalls) {
const resolved = tc.permission?.resolved
if (!resolved) continue
if (resolved.source === 'human') {
permissionResponses.set(tc.toolCallId, resolved.decision === 'allow' ? 'approve' : 'deny')
} else if (resolved.source === 'classifier') {
autoPermissionDecisions.set(tc.toolCallId, {
runId: latestTurnId,
type: 'tool-permission-auto-decision',
subflow: [],
toolCallId: tc.toolCallId,
toolCall: toolCallPartOf(tc),
permission: tc.permission?.required.request as PermMeta,
decision: resolved.decision,
reason: resolved.reason ?? '',
})
}
}
for (const tc of outstandingAsyncTools(latest)) {
if (tc.toolName !== 'ask-human') continue
const input = (tc.input ?? {}) as { question?: unknown; options?: unknown }
pendingAskHumanRequests.set(tc.toolCallId, {
runId: latestTurnId,
type: 'ask-human-request',
toolCallId: tc.toolCallId,
subflow: [],
query: typeof input.question === 'string' ? input.question : '',
...(Array.isArray(input.options) && input.options.every((o) => typeof o === 'string')
? { options: input.options }
: {}),
})
}
}
const settled = status === 'completed' || status === 'failed' || status === 'cancelled'
return {
conversation,
currentAssistantMessage: stripVoiceTags(overlay.text),
voiceSegments: overlay.voiceSegments,
pendingAskHumanRequests,
allPermissionRequests,
permissionResponses,
autoPermissionDecisions,
isProcessing: latest !== undefined && !settled,
isThinking: status === 'idle',
}
}

View file

@ -0,0 +1,109 @@
/**
* Tiny synthesized sound effects for the product tour oar splashes, dock
* bumps, and an arrival fanfare. Everything is generated with Web Audio
* oscillators/noise so no audio assets are needed. All methods fail silently
* if audio is unavailable.
*/
export class TourSounds {
private ctx: AudioContext | null = null
private master: GainNode | null = null
private ensure(): AudioContext | null {
try {
if (!this.ctx) {
this.ctx = new AudioContext()
this.master = this.ctx.createGain()
this.master.gain.value = 0.5
this.master.connect(this.ctx.destination)
}
if (this.ctx.state === 'suspended') void this.ctx.resume()
return this.ctx
} catch {
return null
}
}
/** Short filtered-noise burst with a falling pitch — an oar dipping. */
splash() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const dur = 0.22
const noise = ctx.createBufferSource()
const buffer = ctx.createBuffer(1, Math.ceil(ctx.sampleRate * dur), ctx.sampleRate)
const data = buffer.getChannelData(0)
for (let i = 0; i < data.length; i++) data[i] = Math.random() * 2 - 1
noise.buffer = buffer
const filter = ctx.createBiquadFilter()
filter.type = 'bandpass'
filter.Q.value = 1.2
filter.frequency.setValueAtTime(1600, ctx.currentTime)
filter.frequency.exponentialRampToValueAtTime(350, ctx.currentTime + dur)
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.14, ctx.currentTime)
gain.gain.exponentialRampToValueAtTime(0.001, ctx.currentTime + dur)
noise.connect(filter).connect(gain).connect(this.master)
noise.start()
noise.stop(ctx.currentTime + dur)
}
/** Soft low thump — the boat nudging a dock. */
bump() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const osc = ctx.createOscillator()
osc.type = 'sine'
osc.frequency.setValueAtTime(150, ctx.currentTime)
osc.frequency.exponentialRampToValueAtTime(70, ctx.currentTime + 0.18)
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.2, ctx.currentTime)
gain.gain.exponentialRampToValueAtTime(0.001, ctx.currentTime + 0.25)
osc.connect(gain).connect(this.master)
osc.start()
osc.stop(ctx.currentTime + 0.3)
}
/** Gentle high blip — an email landing in the boat. */
ding() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const osc = ctx.createOscillator()
osc.type = 'sine'
osc.frequency.setValueAtTime(880, ctx.currentTime)
osc.frequency.exponentialRampToValueAtTime(1320, ctx.currentTime + 0.05)
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.08, ctx.currentTime)
gain.gain.exponentialRampToValueAtTime(0.001, ctx.currentTime + 0.3)
osc.connect(gain).connect(this.master)
osc.start()
osc.stop(ctx.currentTime + 0.35)
}
/** Little four-note arpeggio for the tour finale. */
fanfare() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const notes = [523.25, 659.25, 783.99, 1046.5] // C5 E5 G5 C6
notes.forEach((freq, i) => {
const start = ctx.currentTime + i * 0.13
const osc = ctx.createOscillator()
osc.type = 'triangle'
osc.frequency.value = freq
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.0001, start)
gain.gain.exponentialRampToValueAtTime(0.16, start + 0.02)
gain.gain.exponentialRampToValueAtTime(0.001, start + (i === notes.length - 1 ? 0.7 : 0.3))
osc.connect(gain).connect(this.master!)
osc.start(start)
osc.stop(start + 0.8)
})
}
dispose() {
this.ctx?.close().catch(() => {})
this.ctx = null
this.master = null
}
}

View file

@ -6,6 +6,7 @@ import { PostHogProvider } from 'posthog-js/react'
import type { CaptureResult } from 'posthog-js'
import { ThemeProvider } from '@/contexts/theme-context'
import { configureAnalyticsContext } from './lib/analytics'
import { VideoPopout } from '@/components/video-popout'
// Fetch the stable installation ID from main so renderer + main share one
// PostHog distinct_id. Falls back to PostHog's auto-generated anonymous ID
@ -57,4 +58,14 @@ async function bootstrap() {
// The loaded callback applies api_url/app_version once PostHog has initialized.
}
bootstrap()
// The video-mode popout window loads the same bundle with a hash route and
// renders only the mini-call UI — no analytics or app bootstrap needed.
if (window.location.hash === '#video-popout') {
createRoot(document.getElementById('root')!).render(
<StrictMode>
<VideoPopout />
</StrictMode>,
)
} else {
bootstrap()
}

View file

@ -0,0 +1 @@
import '@testing-library/jest-dom/vitest'

View file

@ -1,5 +1,5 @@
import path from "path"
import { defineConfig } from 'vite'
import { defineConfig } from 'vitest/config'
import react from '@vitejs/plugin-react'
import tailwindcss from '@tailwindcss/vite'
@ -18,4 +18,10 @@ export default defineConfig({
build: {
outDir: 'dist',
},
test: {
environment: 'jsdom',
setupFiles: ['./src/test/setup.ts'],
include: ['src/**/*.test.{ts,tsx}'],
css: false,
},
})

View file

@ -12,7 +12,11 @@
"deps": "npm run shared && npm run core && npm run preload",
"main": "wait-on http://localhost:5173 && cd apps/main && npm run build && npm run start",
"lint": "eslint .",
"lint:fix": "eslint . --fix"
"lint:fix": "eslint . --fix",
"test": "npm run shared && npm run test:shared && npm run test:core && npm run test:renderer",
"test:shared": "cd packages/shared && npm test",
"test:core": "cd packages/core && npm test",
"test:renderer": "cd apps/renderer && npm test"
},
"devDependencies": {
"@eslint/js": "^9.39.2",
@ -26,4 +30,4 @@
"typescript-eslint": "^8.50.1",
"wait-on": "^9.0.3"
}
}
}

View file

@ -0,0 +1,561 @@
# Session Layer Technical Specification
Status: design complete for the session layer v1. No implementation exists
yet.
This document specifies the session layer that sits above the turn runtime
defined in `turn-runtime-design.md`. That document is assumed context; this
one does not restate turn semantics.
## 1. Goals
The session layer must:
1. Own conversations composed of ordered turns.
2. Persist each session as one append-only JSONL file with the same
validation discipline as turn files.
3. Enforce one active turn per session.
4. Assemble each turn's context as a reference to the previous turn.
5. Maintain an in-memory index for listing, sorting, and filtering sessions,
updated write-through and rebuilt by scanning at startup.
6. Route external inputs — permission decisions, ask-human answers, async
tool results — to the correct turn through dedicated APIs.
7. Forward live turn events to the renderer over IPC.
8. Provide headless standalone turns outside any session.
## 2. Non-goals (v1)
- Queued user messages. The committed future shape is in section 12.1.
- Steering / mid-turn message injection (section 12.4).
- Session-scoped permission grants ("always allow for this chat"); every
applicable tool call prompts in v1 (section 12.2).
- Context compaction; a viable mechanism sketch is recorded in section 12.3.
V1 behavior on context overflow is the turn-level model failure.
- LLM auto-titling (section 12.6).
- A persisted index cache; startup always scans (section 12.5).
- Cross-process coordination. A single main process is enforced.
- Data migration from the current runs system. Old conversations are not
converted; the old code path remains readable until it is deleted.
- Session list pagination. The index is in-memory and shipped whole.
## 3. Terminology
A **session** is a durable, ordered chain of turns plus presentation
metadata (title). Conversation content lives exclusively in turn files; the
session file stores turn references with denormalized metadata.
The **index** is an in-memory projection over all session files, used for
the session list UI. It is never a source of truth.
A **standalone turn** is a turn with `sessionId: null`, created outside any
session by headless callers. Standalone turns do not appear in the index.
## 4. Storage design
### 4.1 File location
Session files live under:
```text
WorkDir/storage/sessions/YYYY/MM/DD/<sessionId>.jsonl
```
Session IDs come from the existing
`IMonotonicallyIncreasingIdGenerator`, and the repository derives the
date-partitioned path from the ID exactly as the turn repository does,
including format validation and path-traversal rejection.
### 4.2 File rules
Identical discipline to turn files:
- The first line is always `session_created` with `schemaVersion: 1`.
- Every event contains `sessionId` and an ISO timestamp `ts`.
- Physical line order is authoritative.
- Reads validate every line strictly; any malformed line makes the session
corrupt; no truncation, repair, or skipping.
- Unknown schema versions and unknown event types fail loudly. Future
additive event types (queueing, grants, compaction) arrive as a schema
version bump; the reducer will accept old and new versions and write the
newest.
- Appends are awaited but not explicitly `fsync`ed.
### 4.3 Repository contract
```ts
interface ISessionRepo {
create(event: SessionCreated): Promise<void>;
read(sessionId: string): Promise<SessionEvent[]>;
append(sessionId: string, events: SessionEvent[]): Promise<void>;
withLock<T>(sessionId: string, fn: () => Promise<T>): Promise<T>;
listSessionIds(): Promise<string[]>;
delete(sessionId: string): Promise<void>;
}
```
- `create` fails if the file exists.
- `listSessionIds` enumerates the partition directories for the startup
scan.
- `delete` removes the session file only. Turn files referenced by the
session are left in place as harmless orphans (see section 9, deletion).
- `withLock` is in-process per-session exclusion, mirroring the turn repo.
## 5. Event schemas
All session event schemas live in `@x/shared` alongside the turn schemas.
```ts
interface BaseSessionEvent {
sessionId: string;
ts: string;
}
interface SessionCreated extends BaseSessionEvent {
type: "session_created";
schemaVersion: 1;
title?: string;
}
interface SessionTurnAppended extends BaseSessionEvent {
type: "turn_appended";
turnId: string;
sessionSeq: number; // 1-based position of the turn in the session
agentId: string;
model: ModelDescriptor; // resolved provider/model for the turn
}
interface SessionTitleChanged extends BaseSessionEvent {
type: "title_changed";
title: string;
}
type SessionEvent =
| SessionCreated
| SessionTurnAppended
| SessionTitleChanged;
```
`turn_appended` deliberately denormalizes `agentId` and `model` from the
turn so the index can fold from session files without opening turn files.
The turn file remains authoritative for the turn's actual configuration.
The session file never mirrors turn outcomes. Turn lifecycle facts live only
in turn files; deriving "is this session busy/suspended/failed" reads the
latest turn (section 8).
## 6. Session reducer
`@x/shared` owns one pure reducer shared by core and renderer:
```ts
function reduceSession(events: SessionEvent[]): SessionState;
interface SessionState {
definition: SessionCreated;
title?: string;
turns: Array<{
turnId: string;
sessionSeq: number;
agentId: string;
model: ModelDescriptor;
ts: string;
}>;
latestTurnId?: string;
createdAt: string; // definition.ts
updatedAt: string; // ts of the last event
}
```
Invariants (violations throw, as with the turn reducer):
- `session_created` is present, first, and unique.
- All event `sessionId` values match.
- `sessionSeq` is strictly increasing starting at 1, with no gaps.
- `turnId` values are unique.
- Unsupported schema versions and unknown event types fail loudly.
## 7. Write ordering and consistency
Per user message, the session layer performs, in order:
1. `turnRuntime.createTurn(...)` — the turn file is created.
2. `sessionRepo.append(turn_appended)` — the session references the turn.
3. `advanceTurn(...)` — execution begins.
Rules:
- A crash between steps 1 and 2 leaves an orphan turn file: unreferenced,
never advanced, and benign. Turns are only ever found by reference, so an
orphan is invisible. V1 does not garbage-collect orphans.
- The reverse order is forbidden: a `turn_appended` referencing a turn file
that was never created would be a dangling reference, which is corruption.
- Step 2 precedes step 3 so that a turn that is executing is always already
referenced by its session.
- Session-file appends happen under the session lock; turn-file appends are
the turn runtime's concern.
## 8. In-memory index
### 8.1 Shape
```ts
interface SessionIndexEntry {
sessionId: string;
title?: string;
createdAt: string;
updatedAt: string;
turnCount: number;
lastAgentId?: string;
lastModel?: ModelDescriptor;
latestTurnId?: string;
latestTurnStatus:
| "none" // session has no turns yet
| "completed"
| "failed"
| "cancelled"
| "suspended" // durable suspension: pending permissions/async tools
| "idle"; // non-terminal, not suspended: interrupted by a crash
}
```
`latestTurnStatus` is derived from the latest turn's reduced state, using
the same derivation everywhere: terminal event kind if present, else
`suspended` if a suspension with outstanding work is the resting state, else
`idle`. Whether a turn is *actively processing right now* is not in the
index; it is ephemeral bus state (`turn-processing-start/end`), per the turn
specification.
### 8.2 Startup scan
1. `listSessionIds()`.
2. For each session: read and reduce the session file, producing the entry's
session-derived fields.
3. For each session with turns: read and reduce the latest turn file only,
producing `latestTurnStatus`.
4. Publish the completed index to the renderer.
A corrupt session file or corrupt latest-turn file does not abort startup:
the entry is surfaced in an errored state (identifiable in the UI, excluded
from normal interaction) and the scan continues.
### 8.3 Maintenance
- Every session mutation updates the entry in the same code path that
appends to the session file (write-through), then publishes a
`session-index-changed` event on the application bus.
- When an `advanceTurn` outcome settles, the session layer updates
`latestTurnStatus` and publishes `session-index-changed`.
- There is no filesystem watcher. Out-of-band edits to session or turn files
while the app runs are unsupported; offline changes are reconciled by the
next startup scan.
- The main process enforces single-instance via
`app.requestSingleInstanceLock()`; all locking in both layers is
in-process.
## 9. Sessions API
```ts
interface SendMessageConfig {
agent: RequestedAgent; // agent id + optional model override
autoPermission?: boolean; // default false
maxModelCalls?: number; // default per turn spec
}
interface ISessions {
createSession(input?: { title?: string }): Promise<string>;
listSessions(): SessionIndexEntry[];
getSession(sessionId: string): Promise<SessionState>;
getTurn(turnId: string): Promise<Turn>; // passthrough to turn runtime
sendMessage(
sessionId: string,
input: UserMessage,
config: SendMessageConfig,
): Promise<{ turnId: string }>;
respondToPermission(
turnId: string,
toolCallId: string,
decision: "allow" | "deny",
metadata?: JsonValue,
): Promise<void>;
respondToAskHuman(
turnId: string,
toolCallId: string,
answer: string,
): Promise<void>;
deliverAsyncToolResult(
turnId: string,
toolCallId: string,
result: ToolResultData,
): Promise<void>;
stopTurn(turnId: string, reason?: string): Promise<void>;
resumeTurn(sessionId: string): Promise<void>;
setTitle(sessionId: string, title: string): Promise<void>;
deleteSession(sessionId: string): Promise<void>;
}
```
### 9.1 sendMessage
Under the per-session lock:
1. Read and reduce the session.
2. If the session has turns, read and reduce the latest turn. If it is
non-terminal — running, suspended, or idle — reject with a typed
`TurnNotSettledError`. There is no implicit queueing, steering, or
cancel-and-replace, and no implicit routing to a pending ask-human.
3. Build context: `[]` (inline, empty) for the first turn, else
`{ previousTurnId: latestTurnId }`.
4. Create the turn: config from `SendMessageConfig` lands on the turn
(`humanAvailable: true` always, for session turns). Sessions store no
configuration; every turn is self-describing.
5. Append `turn_appended` with the next `sessionSeq` and denormalized
agent/model.
6. If the session has no title, append `title_changed` derived from the
truncated first user message.
7. Start `advanceTurn` in the background; consume its events (section 10).
8. Return `{ turnId }` immediately. The renderer follows progress through
events, not the return value.
Continuation after a failed or exhausted (`code: "model-call-limit"`) turn
is just `sendMessage`: failed turns are terminal, and the new turn's context
reference includes the failed turn's structurally complete transcript.
### 9.2 External inputs
`respondToPermission`, `respondToAskHuman`, and `deliverAsyncToolResult`
each translate to one `advanceTurn(turnId, input)` call with the
corresponding `TurnExternalInput`. `respondToAskHuman` is the dedicated
endpoint for the `ask-human` tool — a thin wrapper over
`async_tool_result` — and is deliberately separate from `sendMessage`.
Validation (unknown call, already-resolved call, terminal turn) is the turn
runtime's job; the session layer passes its errors through.
### 9.3 stopTurn and resumeTurn
- `stopTurn` cancels via the turn runtime: aborting the active invocation's
signal if one is running, else advancing a suspended turn with a `cancel`
input.
- `resumeTurn` re-enters the latest turn with no input — the turn spec's
recovery entry point — for turns left `idle` by a crash. There is no
automatic resume sweep at startup: recovery re-issues interrupted model
calls, so resumption must be an explicit user action. Suspended turns need
no resumption; they advance when their inputs arrive.
### 9.4 Deletion
`deleteSession` removes the session file and the index entry, and publishes
`session-index-changed`. Referenced turn files are retained as orphans:
turns are only discoverable by reference, so orphaned files are inert.
Deleting an entity's file is not a violation of append-only discipline,
which governs mutation of live logs, not their removal.
## 10. Event forwarding and live UI
1. For every `advanceTurn` it initiates, the session layer consumes
`TurnExecution.events` and forwards each `TurnStreamEvent` — tagged with
`sessionId` — through the application bus to renderer windows over one
IPC channel (`sessions:events`).
2. When `outcome` settles, the session layer updates the index entry and
publishes `session-index-changed`.
3. The renderer follows the turn spec's historical/live pattern: fetch
turns via `getTurn`, run the shared `reduceTurn` per turn, compose the
session timeline turn-by-turn (each turn renders its input and its own
activity; the referenced prefix is never re-rendered from context),
append live durable events and re-reduce, and keep text/reasoning deltas
in an ephemeral overlay cleared by canonical responses.
4. Pending approvals and ask-human prompts render from the suspended turn's
reduced state, so they survive restarts without any session-layer
bookkeeping.
## 11. Headless standalone turns
A helper covers the non-session callers (background tasks, live notes,
knowledge pipelines, scheduled agents):
```ts
function runHeadlessTurn(input: {
agent: RequestedAgent;
context?: ConversationMessage[]; // inline; defaults to []
input: UserMessage;
maxModelCalls?: number;
signal?: AbortSignal;
}): Promise<TurnOutcome>;
```
- `sessionId: null`, `autoPermission: true`, `humanAvailable: false`.
- Creates the turn, advances to the first settled outcome, and returns it.
- Standalone turns never appear in the index; callers keep their own turn
IDs if they need history.
## 12. Deferred designs with committed shapes
These are not implemented in v1. Their shapes are recorded so v1 decisions
stay compatible; each arrives as a session schema version bump.
### 12.1 Queued messages
```ts
{ type: "message_queued", queueId, message, ts }
{ type: "queued_message_replaced", queueId, message, ts } // edit
{ type: "queued_message_removed", queueId, ts } // cancel
// promotion: turn_appended gains consumedQueueIds?: string[]
```
The reducer derives the pending queue by supersession. Promotion rules
(collapse-into-one-turn vs FIFO, behavior after failed turns) are decided
together with steering.
### 12.2 Session permission grants
```ts
{ type: "permission_grant_added", grantId, toolId, ts }
{ type: "permission_grant_removed", grantId, ts }
```
The injected `IPermissionChecker` consults a session-keyed grants view
before answering `required: true`. V1 grants would be blanket per-toolId;
argument-pattern matchers are a separate, security-sensitive project.
### 12.3 Compaction (mechanism sketch)
Compaction requires zero turn-schema change. A session-level compaction
event records `{ compactionId, summary, firstKeptTurnId }`; the next turn
after a compaction uses **inline** context (summary message + kept
transcript), which restarts the reference chain and bounds resolution depth
by construction. Trigger policy and summarizer design are unspecified.
### 12.4 Steering
Rides the queue events with a different promotion rule, and additionally
requires a turn-level `steer` external input (a turn schema bump) with an
injection boundary after tool-batch completion. Recorded here so the session
design does not foreclose it.
### 12.5 Persisted index cache
If the startup scan ever gets slow, a single cache file keyed by file
mtimes can be added. It is a rebuildable cache, never a source of truth:
missing, stale, or invalid means rebuild from session files.
### 12.6 Auto-titling
An LLM-generated title replacing the truncated-first-message default,
appended as an ordinary `title_changed` event.
## 13. Required test scenarios
All tests use the in-memory/mocked turn runtime and repo fakes.
### 13.1 Reducer
- Valid event sequences reduce to expected state.
- Every invariant violation throws: missing/duplicate `session_created`,
mismatched sessionId, non-monotonic or gapped `sessionSeq`, duplicate
turnIds, unknown type/version.
- Title folding: default, explicit changes, last-wins.
### 13.2 Repository
- Date-partitioned paths, ID validation, create-if-absent.
- Strict line validation on read; corrupt files rejected whole.
- `listSessionIds` enumeration across partitions.
- Deletion removes only the session file.
### 13.3 sendMessage
- First turn: inline `[]` context, `sessionSeq: 1`, default title appended.
- Subsequent turns: context references the latest turn; seq increments.
- Rejection with `TurnNotSettledError` when the latest turn is running,
suspended, or idle — and success after it settles.
- Continuation after failed and model-call-limit turns.
- Concurrent sendMessage calls serialize under the session lock; exactly
one wins, the other rejects.
- Config lands on the turn; the session file stores only denormalized
agent/model on `turn_appended`.
### 13.4 Ordering and crash simulation
- Simulated crash between `createTurn` and `turn_appended`: orphan turn
file, session unchanged, retry produces a fresh turn.
- `turn_appended` is present before the first advance begins.
### 13.5 External inputs
- Permission decision, ask-human answer, and async result each advance the
correct turn with the correct input type.
- Turn-runtime rejections (unknown call, terminal turn) pass through.
- `sendMessage` never routes to ask-human.
### 13.6 Index
- Startup fold matches write-through state for the same history.
- Latest-turn status derivation for every status value.
- Corrupt session file yields an errored entry without aborting the scan.
- Mutations publish `session-index-changed`; deletion removes the entry.
### 13.7 Event forwarding
- Forwarded events are tagged with sessionId and arrive in order.
- Outcome settlement updates `latestTurnStatus`.
### 13.8 Headless
- Standalone turns: `sessionId: null`, auto permission, human unavailable,
absent from the index.
## 14. Suggested module layout
```text
apps/x/packages/shared/src/sessions.ts # event schemas, reducer, index types
apps/x/packages/core/src/sessions/
sessions.ts # ISessions implementation
repo.ts # ISessionRepo contract
fs-repo.ts # filesystem implementation
index.ts # in-memory index + startup scan
headless.ts # runHeadlessTurn
```
Awilix registration mirrors the turn runtime: singleton scope, PROXY
constructor injection, no container resolution from inside the classes.
## 15. Integration sequence
The rollout is staged as commits on one branch (squash-merge acceptable);
old and new stacks coexist briefly but never share state, and no data is
migrated:
1. `@x/shared`: turn + session schemas and reducers.
2. Turn runtime + fs turn repo (unit tests only, wired to nothing).
3. Session layer + index (unit tests only).
4. Bridges: agent resolver, context resolver, tool runner, permission
checker/classifier — adapted from the `new-runtime` reference
implementation where applicable.
5. IPC (`sessions:*`) + renderer swap: Copilot chat UI moves to the
sessions API.
6. Headless callers move to `runHeadlessTurn`.
7. Delete the old runs runtime.
## 16. Implementation acceptance criteria
The session layer is implementation-complete only when:
1. Session event schemas and `reduceSession` live in `@x/shared` and are
consumed unchanged by core and renderer.
2. Session files follow the partitioned append-only JSONL layout with
strict validation.
3. Turn-file-first write ordering is enforced; orphan turns are benign.
4. `sendMessage` rejects non-terminal latest turns with a typed error; no
implicit queueing, steering, or ask-human routing exists.
5. Ask-human answers flow only through the dedicated endpoint.
6. The index is write-through with a startup scan, no watcher, and no
persisted cache; single-instance is enforced.
7. Deletion removes only the session file.
8. Headless callers run standalone turns and appear nowhere in the index.
9. All required test scenarios pass with mocked dependencies.

File diff suppressed because it is too large Load diff

View file

@ -8,7 +8,9 @@
"build": "rm -rf dist && tsc -p tsconfig.build.json",
"dev": "tsc -w -p tsconfig.build.json",
"test": "vitest run",
"test:watch": "vitest"
"test:watch": "vitest",
"inspect-turn": "node dist/turns/inspect-cli.js",
"inspect": "node dist/turns/inspect-cli.js"
},
"dependencies": {
"@agentclientprotocol/claude-agent-acp": "^0.39.0",
@ -28,6 +30,7 @@
"@x/shared": "workspace:*",
"ai": "^5.0.133",
"awilix": "^12.0.5",
"baileys": "7.0.0-rc13",
"chokidar": "^4.0.3",
"cors": "^2.8.6",
"cron-parser": "^5.5.0",
@ -43,6 +46,7 @@
"papaparse": "^5.5.3",
"pdf-parse": "^2.4.5",
"posthog-node": "^4.18.0",
"qrcode": "^1.5.4",
"react": "^19.2.3",
"xlsx": "^0.18.5",
"yaml": "^2.8.2",
@ -54,6 +58,7 @@
"@types/node": "^25.0.3",
"@types/papaparse": "^5.5.2",
"@types/pdf-parse": "^1.1.5",
"@types/qrcode": "^1.5.6",
"vitest": "catalog:"
}
}

View file

@ -2,13 +2,10 @@ import { CronExpressionParser } from "cron-parser";
import container from "../di/container.js";
import { IAgentScheduleRepo } from "./repo.js";
import { IAgentScheduleStateRepo } from "./state-repo.js";
import { IRunsRepo } from "../runs/repo.js";
import { IAgentRuntime } from "../agents/runtime.js";
import { IMonotonicallyIncreasingIdGenerator } from "../application/lib/id-gen.js";
import { AgentScheduleConfig, AgentScheduleEntry } from "@x/shared/dist/agent-schedule.js";
import { AgentScheduleState, AgentScheduleStateEntry } from "@x/shared/dist/agent-schedule-state.js";
import { MessageEvent } from "@x/shared/dist/runs.js";
import { createRun } from "../runs/runs.js";
import { startHeadlessAgent } from "../agents/headless-app.js";
import { withUseCase } from "../analytics/use_case.js";
import z from "zod";
const DEFAULT_STARTING_MESSAGE = "go";
@ -147,10 +144,7 @@ function shouldRunNow(
async function runAgent(
agentName: string,
entry: z.infer<typeof AgentScheduleEntry>,
stateRepo: IAgentScheduleStateRepo,
runsRepo: IRunsRepo,
agentRuntime: IAgentRuntime,
idGenerator: IMonotonicallyIncreasingIdGenerator
stateRepo: IAgentScheduleStateRepo
): Promise<void> {
console.log(`[AgentRunner] Starting agent: ${agentName}`);
@ -163,31 +157,24 @@ async function runAgent(
});
try {
// Create a new run via core (resolves agent + default model+provider).
const run = await createRun({
agentId: agentName,
useCase: 'copilot_chat',
subUseCase: 'scheduled',
});
console.log(`[AgentRunner] Created run ${run.id} for agent ${agentName}`);
// Add the starting message as a user message
// Start a standalone turn (resolves agent + default model/provider).
// Fire-and-forget like the old trigger(): the schedule state machine
// below runs on wall-clock; completion is observed only for logging.
// The signal caps a hung turn at the same TIMEOUT_MS the state-based
// watchdog uses.
const startingMessage = entry.startingMessage ?? DEFAULT_STARTING_MESSAGE;
const messageEvent: z.infer<typeof MessageEvent> = {
runId: run.id,
type: "message",
messageId: await idGenerator.next(),
message: {
role: "user",
content: startingMessage,
},
subflow: [],
};
await runsRepo.appendEvents(run.id, [messageEvent]);
console.log(`[AgentRunner] Sent starting message to agent ${agentName}: "${startingMessage}"`);
// Trigger the run
await agentRuntime.trigger(run.id);
const handle = await withUseCase(
{ useCase: 'copilot_chat', subUseCase: 'scheduled' },
() => startHeadlessAgent({
agentId: agentName,
message: startingMessage,
signal: AbortSignal.timeout(TIMEOUT_MS),
}),
);
console.log(`[AgentRunner] Started turn ${handle.turnId} for agent ${agentName}: "${startingMessage}"`);
void handle.done
.then((result) => console.log(`[AgentRunner] Turn ${handle.turnId} (${agentName}) settled: ${result.outcome.status}`))
.catch((error) => console.error(`[AgentRunner] Turn ${handle.turnId} (${agentName}) errored:`, error instanceof Error ? error.message : error));
// Calculate next run time
const nextRunAt = calculateNextRunAt(entry.schedule);
@ -264,9 +251,6 @@ async function checkForTimeouts(
async function pollAndRun(): Promise<void> {
const scheduleRepo = container.resolve<IAgentScheduleRepo>("agentScheduleRepo");
const stateRepo = container.resolve<IAgentScheduleStateRepo>("agentScheduleStateRepo");
const runsRepo = container.resolve<IRunsRepo>("runsRepo");
const agentRuntime = container.resolve<IAgentRuntime>("agentRuntime");
const idGenerator = container.resolve<IMonotonicallyIncreasingIdGenerator>("idGenerator");
// Load config and state
let config: z.infer<typeof AgentScheduleConfig>;
@ -314,7 +298,7 @@ async function pollAndRun(): Promise<void> {
if (shouldRunNow(entry, agentState)) {
// Run agent (don't await - let it run in background)
runAgent(agentName, entry, stateRepo, runsRepo, agentRuntime, idGenerator).catch((error) => {
runAgent(agentName, entry, stateRepo).catch((error) => {
console.error(`[AgentRunner] Unhandled error in runAgent for ${agentName}:`, error);
});
}

View file

@ -0,0 +1,25 @@
import container from "../di/container.js";
import {
type HeadlessAgentHandle,
type HeadlessAgentOptions,
type HeadlessAgentResult,
type IHeadlessAgentRunner,
} from "./headless.js";
function runner(): IHeadlessAgentRunner {
return container.resolve<IHeadlessAgentRunner>("headlessAgentRunner");
}
export function startHeadlessAgent(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentHandle> {
return runner().start(options);
}
export function runHeadlessAgent(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentResult & { turnId: string }> {
return runner().run(options);
}
export { toolInputPaths } from "./headless.js";

View file

@ -0,0 +1,300 @@
import { describe, expect, it, vi } from "vitest";
import type { z } from "zod";
import { reduceTurn, type TurnEvent } from "@x/shared/dist/turns.js";
import type {
CreateTurnInput,
ITurnRuntime,
TurnExecution,
TurnOutcome,
} from "../turns/api.js";
import {
HeadlessRunError,
HeadlessAgentRunner,
assistantText,
lastAssistantText,
toolInputPaths,
} from "./headless.js";
import type { IDefaultModelResolver } from "../models/default-model-resolver.js";
type TEvent = z.infer<typeof TurnEvent>;
const TURN_ID = "2026-07-02T10-00-00Z-0000001-000";
const TS = "2026-07-02T10:00:00Z";
const echoTool = {
toolId: "builtin:file-editText",
name: "file-editText",
description: "Edit",
inputSchema: {},
execution: "sync" as const,
requiresHuman: false,
};
function turnLog(opts: {
responseText?: string;
toolCalls?: Array<{ id: string; name: string; input: unknown; invoked: boolean }>;
failed?: string;
}): TEvent[] {
const events: TEvent[] = [
{
type: "turn_created",
schemaVersion: 1,
turnId: TURN_ID,
ts: TS,
sessionId: null,
agent: {
requested: { agentId: "worker" },
resolved: {
agentId: "worker",
systemPrompt: "SYS",
model: { provider: "fake", model: "m" },
tools: [echoTool],
},
},
context: [],
input: { role: "user", content: "go" },
config: { autoPermission: true, humanAvailable: false, maxModelCalls: 20 },
},
{
type: "model_call_requested",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
request: { messages: ["input"], parameters: {} },
},
];
const toolCalls = opts.toolCalls ?? [];
if (toolCalls.length > 0) {
events.push({
type: "model_call_completed",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
message: {
role: "assistant",
content: toolCalls.map((tc) => ({
type: "tool-call" as const,
toolCallId: tc.id,
toolName: tc.name,
arguments: tc.input as never,
})),
},
finishReason: "tool-calls",
usage: {},
});
for (const tc of toolCalls) {
if (!tc.invoked) continue;
events.push({
type: "tool_invocation_requested",
turnId: TURN_ID,
ts: TS,
toolCallId: tc.id,
toolId: "builtin:" + tc.name,
toolName: tc.name,
execution: "sync",
input: tc.input as never,
});
events.push({
type: "tool_result",
turnId: TURN_ID,
ts: TS,
toolCallId: tc.id,
toolName: tc.name,
source: "sync",
result: { output: "ok", isError: false },
});
}
}
if (opts.failed) {
events.push({
type: "model_call_failed",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
error: opts.failed,
});
events.push({ type: "turn_failed", turnId: TURN_ID, ts: TS, error: opts.failed, usage: {} });
} else if (opts.responseText !== undefined && toolCalls.length === 0) {
events.push({
type: "model_call_completed",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
message: { role: "assistant", content: opts.responseText },
finishReason: "stop",
usage: {},
});
events.push({
type: "turn_completed",
turnId: TURN_ID,
ts: TS,
output: { role: "assistant", content: opts.responseText },
finishReason: "stop",
usage: {},
});
}
return events;
}
class FakeRuntime implements ITurnRuntime {
createInputs: CreateTurnInput[] = [];
constructor(
private readonly log: TEvent[],
private readonly outcome: TurnOutcome,
) {}
async createTurn(input: CreateTurnInput): Promise<string> {
this.createInputs.push(input);
return TURN_ID;
}
advanceTurn(): TurnExecution {
return {
events: (async function* () {})(),
outcome: Promise.resolve(this.outcome),
};
}
async getTurn() {
return { turnId: TURN_ID, events: this.log };
}
}
const completedOutcome: TurnOutcome = {
status: "completed",
output: { role: "assistant", content: "done" },
finishReason: "stop",
usage: {},
};
function createRunner(
runtime: ITurnRuntime,
resolve = vi.fn(async () => ({
provider: "default-provider",
model: "default-model",
})),
): { runner: HeadlessAgentRunner; resolve: IDefaultModelResolver["resolve"] } {
return {
runner: new HeadlessAgentRunner({
turnRuntime: runtime,
defaultModelResolver: { resolve },
}),
resolve,
};
}
describe("headless agent helpers", () => {
it("assistantText handles string and parts content", () => {
expect(assistantText({ role: "assistant", content: "hi" })).toBe("hi");
expect(
assistantText({
role: "assistant",
content: [
{ type: "reasoning", text: "hmm" },
{ type: "text", text: "a" },
{ type: "text", text: "b" },
],
}),
).toBe("ab");
expect(assistantText({ role: "assistant", content: "" })).toBeNull();
});
it("lastAssistantText returns the final assistant text in the transcript", () => {
const state = reduceTurn(turnLog({ responseText: "the summary" }));
expect(lastAssistantText(state)).toBe("the summary");
expect(lastAssistantText(reduceTurn(turnLog({ failed: "boom" })))).toBeNull();
});
it("toolInputPaths collects paths of invoked calls only", () => {
const state = reduceTurn(
turnLog({
toolCalls: [
{ id: "a", name: "file-editText", input: { path: "notes/x.md" }, invoked: true },
{ id: "b", name: "file-editText", input: { path: "notes/y.md" }, invoked: true },
{ id: "c", name: "file-writeText", input: { path: "notes/z.md" }, invoked: true },
{ id: "d", name: "file-editText", input: { path: "never-ran.md" }, invoked: false },
],
}),
);
expect(toolInputPaths(state, ["file-editText"])).toEqual(
new Set(["notes/x.md", "notes/y.md"]),
);
expect(toolInputPaths(state, ["file-writeText"])).toEqual(new Set(["notes/z.md"]));
});
});
describe("HeadlessAgentRunner", () => {
it("creates a standalone auto-permission turn and returns the id before settling", async () => {
const runtime = new FakeRuntime(turnLog({ responseText: "the summary" }), completedOutcome);
const { runner, resolve } = createRunner(runtime);
const handle = await runner.start({
agentId: "worker",
message: "go",
model: "m",
provider: "fake",
});
expect(handle.turnId).toBe(TURN_ID);
expect(resolve).not.toHaveBeenCalled();
expect(runtime.createInputs[0]).toMatchObject({
agent: { agentId: "worker", overrides: { model: { provider: "fake", model: "m" } } },
sessionId: null,
context: [],
input: { role: "user", content: "go" },
config: { autoPermission: true, humanAvailable: false },
});
const result = await handle.done;
expect(result.outcome.status).toBe("completed");
expect(result.summary).toBe("the summary");
});
it("omits the model override when neither model nor provider is set", async () => {
const runtime = new FakeRuntime(turnLog({ responseText: "ok" }), completedOutcome);
const { runner, resolve } = createRunner(runtime);
await runner.start({ agentId: "worker", message: "go" });
expect(resolve).not.toHaveBeenCalled();
expect(runtime.createInputs[0].agent.overrides).toBeUndefined();
});
it("uses an injected default only for a partial model override", async () => {
const runtime = new FakeRuntime(turnLog({ responseText: "ok" }), completedOutcome);
const { runner, resolve } = createRunner(runtime);
await runner.start({
agentId: "worker",
message: "go",
model: "custom-model",
});
expect(resolve).toHaveBeenCalledOnce();
expect(runtime.createInputs[0].agent.overrides).toEqual({
model: { provider: "default-provider", model: "custom-model" },
});
});
it("throwOnError rejects done with HeadlessRunError on failed outcomes", async () => {
const runtime = new FakeRuntime(turnLog({ failed: "provider exploded" }), {
status: "failed",
error: "provider exploded",
usage: {},
});
const { runner } = createRunner(runtime);
const handle = await runner.start({
agentId: "worker",
message: "go",
throwOnError: true,
});
await expect(handle.done).rejects.toThrowError(HeadlessRunError);
await expect(handle.done).rejects.toThrowError("provider exploded");
});
it("without throwOnError a failed outcome resolves (old wait semantics)", async () => {
const runtime = new FakeRuntime(turnLog({ failed: "boom" }), {
status: "failed",
error: "boom",
usage: {},
});
const { runner } = createRunner(runtime);
const handle = await runner.start({ agentId: "worker", message: "go" });
const result = await handle.done;
expect(result.outcome.status).toBe("failed");
expect(result.summary).toBeNull();
});
});

View file

@ -0,0 +1,182 @@
import type { z } from "zod";
import type { AssistantMessage } from "@x/shared/dist/message.js";
import {
reduceTurn,
type TurnState,
} from "@x/shared/dist/turns.js";
import type { ITurnRuntime, TurnOutcome } from "../turns/api.js";
import type { IDefaultModelResolver } from "../models/default-model-resolver.js";
// Drop-in replacement for the old headless runs pattern
// (createRun → createMessage → waitForRunCompletion → extractAgentResponse):
// one standalone turn per invocation (sessionId null, automatic permissions,
// no human). HeadlessAgentRunner.start returns the turn id immediately (callers
// record it in pointer files / bus events before completion); `done` settles
// with the outcome, the reduced turn state, and the final assistant text.
export class HeadlessRunError extends Error {
constructor(
message: string,
readonly turnId: string,
readonly outcome: TurnOutcome,
) {
super(message);
this.name = "HeadlessRunError";
}
}
export interface HeadlessAgentOptions {
agentId: string;
message: string;
// Model id; when set without provider, the app-default provider applies.
model?: string;
provider?: string;
maxModelCalls?: number;
signal?: AbortSignal;
// Old waitForRunCompletion({ throwOnError: true }) semantics: `done`
// rejects with HeadlessRunError unless the turn completes.
throwOnError?: boolean;
}
export interface HeadlessAgentResult {
outcome: TurnOutcome;
state: TurnState;
// Last assistant text in the transcript (old extractAgentResponse).
summary: string | null;
}
export interface HeadlessAgentHandle {
turnId: string;
done: Promise<HeadlessAgentResult>;
}
export interface HeadlessAgentRunnerDependencies {
turnRuntime: ITurnRuntime;
defaultModelResolver: IDefaultModelResolver;
}
export interface IHeadlessAgentRunner {
start(options: HeadlessAgentOptions): Promise<HeadlessAgentHandle>;
run(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentResult & { turnId: string }>;
}
export function assistantText(
message: z.infer<typeof AssistantMessage>,
): string | null {
const content = message.content;
if (typeof content === "string") {
return content || null;
}
const text = content
.map((part) => (part.type === "text" ? part.text : ""))
.join("");
return text || null;
}
export function lastAssistantText(state: TurnState): string | null {
for (let i = state.modelCalls.length - 1; i >= 0; i--) {
const response = state.modelCalls[i].response;
if (response) {
const text = assistantText(response);
if (text) {
return text;
}
}
}
return null;
}
// Paths passed to the given file tools across the turn — replaces the old
// pattern of subscribing to the run bus for tool-invocation events. Only
// actually-invoked calls count (denied/unknown calls never ran).
export function toolInputPaths(
state: TurnState,
toolNames: string[],
): Set<string> {
const paths = new Set<string>();
for (const toolCall of state.toolCalls) {
if (!toolNames.includes(toolCall.toolName) || !toolCall.invocation) {
continue;
}
const input = toolCall.input as { path?: unknown } | null | undefined;
if (input && typeof input === "object" && typeof input.path === "string") {
paths.add(input.path);
}
}
return paths;
}
export class HeadlessAgentRunner implements IHeadlessAgentRunner {
private readonly turnRuntime: ITurnRuntime;
private readonly defaultModelResolver: IDefaultModelResolver;
constructor({
turnRuntime,
defaultModelResolver,
}: HeadlessAgentRunnerDependencies) {
this.turnRuntime = turnRuntime;
this.defaultModelResolver = defaultModelResolver;
}
async start(options: HeadlessAgentOptions): Promise<HeadlessAgentHandle> {
let modelOverride: { provider: string; model: string } | undefined;
if (options.model && options.provider) {
modelOverride = { provider: options.provider, model: options.model };
} else if (options.model || options.provider) {
const defaults = await this.defaultModelResolver.resolve();
modelOverride = {
provider: options.provider ?? defaults.provider,
model: options.model ?? defaults.model,
};
}
const turnId = await this.turnRuntime.createTurn({
agent: {
agentId: options.agentId,
...(modelOverride ? { overrides: { model: modelOverride } } : {}),
},
sessionId: null,
context: [],
input: { role: "user", content: options.message },
config: {
autoPermission: true,
humanAvailable: false,
...(options.maxModelCalls === undefined
? {}
: { maxModelCalls: options.maxModelCalls }),
},
});
const execution = this.turnRuntime.advanceTurn(turnId, undefined, {
signal: options.signal,
});
const done = execution.outcome.then(async (outcome) => {
const state = reduceTurn(
(await this.turnRuntime.getTurn(turnId)).events,
);
if (options.throwOnError && outcome.status !== "completed") {
throw new HeadlessRunError(
outcome.status === "failed"
? outcome.error
: `turn ${outcome.status}`,
turnId,
outcome,
);
}
return { outcome, state, summary: lastAssistantText(state) };
});
// The handle may be used fire-and-forget; rejections surface when awaited.
done.catch(() => undefined);
return { turnId, done };
}
async run(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentResult & { turnId: string }> {
const handle = await this.start(options);
const result = await handle.done;
return { turnId: handle.turnId, ...result };
}
}

View file

@ -114,7 +114,7 @@ function filePermissionTargets(toolName: string, args: Record<string, unknown>):
}
}
async function getToolPermissionMetadata(
export async function getToolPermissionMetadata(
toolCall: z.infer<typeof ToolCallPart>,
underlyingTool: z.infer<typeof ToolAttachment>,
sessionAllowedCommands: Set<string>,
@ -172,7 +172,7 @@ async function getToolPermissionMetadata(
};
}
function loadUserWorkDir(runId: string): string | null {
export function loadUserWorkDir(runId: string): string | null {
try {
const file = workDirConfigFile(runId);
if (!fs.existsSync(file)) return null;
@ -185,7 +185,7 @@ function loadUserWorkDir(runId: string): string | null {
}
}
function loadAgentNotesContext(): string | null {
export function loadAgentNotesContext(): string | null {
const sections: string[] = [];
const userFile = path.join(AGENT_NOTES_DIR, 'user.md');
@ -327,6 +327,129 @@ If Middle pane state is note, the supplied path and content are available so you
If Middle pane state is browser, only the URL and page title are supplied; the page content itself is NOT included. If you need the page content to answer, use the browser tools available to you to read the page. The user may or may not be talking about this page. Only reference or act on this page when the user's message clearly relates to it, such as "this page", "this article", "what I'm looking at", "this site", or "summarize this". For unrelated questions, ignore this page entirely and answer normally. Do not mention that you can see the browser unless it is relevant to the answer.`;
export interface ComposeSystemInstructionsInput {
instructions: string;
agentNotesContext: string | null;
userWorkDir: string | null;
voiceInput: boolean;
voiceOutput: 'summary' | 'full' | null;
searchEnabled: boolean;
codeMode: 'claude' | 'codex' | null;
codeCwd: string | null;
// Optional so legacy callers (old streamAgent path) are unaffected.
videoMode?: boolean;
coachMode?: boolean;
}
// System-prompt assembly, extracted verbatim from streamAgent so the new turn
// runtime's agent resolver composes byte-identical prompts. Pure: callers
// load agent notes / work dir themselves.
export function composeSystemInstructions({
instructions,
agentNotesContext,
userWorkDir,
voiceInput,
voiceOutput,
searchEnabled,
codeMode,
codeCwd,
videoMode,
coachMode,
}: ComposeSystemInstructionsInput): string {
let instructionsWithDateTime = `${instructions}\n\n${USER_CONTEXT_SYSTEM_INSTRUCTIONS}`;
if (agentNotesContext) {
instructionsWithDateTime += `\n\n${agentNotesContext}`;
}
if (userWorkDir) {
instructionsWithDateTime += `\n\n# User Work Directory
The user has chosen the following directory as their current **work directory**:
\`${userWorkDir}\`
Treat this as the **default location** for file operations whenever the user refers to files generically:
- "list the files", "show me what's in here", "what's the latest report" list or look in the work directory.
- "save this", "export it", "write that to a file" write the output into the work directory unless the user names another location.
- "open the file I was just working on", "the doc from earlier" assume the work directory first.
Use absolute paths rooted at this directory with the \`file-*\` tools. For example, list with \`file-list({ path: "${userWorkDir}" })\`, read text with \`file-readText\`, and write text with \`file-writeText\`. For PDFs, Office docs, images, scanned docs, and other non-text files, use \`parseFile\` or \`LLMParse\` with the absolute path; you do NOT need to copy the file into the workspace first.
**Exceptions these ALWAYS take precedence over the work directory default:**
1. **Knowledge base questions.** If the user asks about anything in the knowledge graph (notes, people, organizations, projects, topics) or paths starting with \`knowledge/\`, use file tools against \`knowledge/\` as documented above. Do NOT redirect those into the work directory.
2. **Explicit paths.** If the user names a different directory or gives an absolute/relative path (e.g. "in ~/Downloads", "from /tmp/foo", "the Desktop"), honor that path exactly and ignore the work-directory default for that request.
3. **Workspace-specific operations.** Anything that obviously belongs in the Rowboat workspace (config files, MCP servers, agent schedules, etc.) stays in the workspace, not the work directory.
Do not announce the work directory unless it's relevant. Just use it.`;
}
if (voiceInput) {
instructionsWithDateTime += `\n\n# Voice Input\nThe user's message was transcribed from speech. Be aware that:\n- There may be transcription errors. Silently correct obvious ones (e.g. homophones, misheard words). If an error is genuinely ambiguous, briefly mention your interpretation (e.g. "I'm assuming you meant X").\n- Spoken messages are often long-winded. The user may ramble, repeat themselves, or correct something they said earlier in the same message. Focus on their final intent, not every word verbatim.`;
}
if (videoMode) {
instructionsWithDateTime += `\n\n# Video Mode (Live Camera)
The user has turned on video mode: their webcam is on, and their messages arrive with a series of live webcam frames (ordered oldest to newest) captured while they were speaking or typing. The frames are a live view of the user themselves not a document or file to analyze.
How to use the frames:
- Use them for what visual awareness genuinely adds: the user's expressions, body language, posture, gestures, eye contact with the camera, visible energy, anything they hold up to the camera, and their surroundings when relevant.
- Compare across frames to notice change over time within a message (e.g. increasingly slouched, started smiling, looked away for most of the message).
- When the user practices something performative (a pitch, presentation, interview, talk) or asks for delivery feedback, give specific, actionable coaching grounded in what you actually see cite concrete observations ("in the last few frames you looked down and away from the camera") rather than generic advice. Cover posture, eye contact, facial expressiveness, gesturing, and visible energy.
- If they show something to the camera (an object, document, whiteboard), read or describe it and respond accordingly.
Driving the app:
- You can control the Rowboat app the user is looking at via the app-navigation tool (load the app-navigation skill first): open views, READ a view's contents as data (emails, background agents, chat history), and open specific items on their screen.
- When the user asks about anything that lives inside Rowboat ("what emails do I have?", "what agents are running?", "open the one from Arjun"), prefer driving over describing: read-view shows the view on their screen while returning its data, then answer out loud briefly; open-item when they pick one. Narrate as you act ("pulling up your inbox…"). Reading a view's data beats squinting at screen-share frames — it's exact.
Screen sharing:
- The user may also share their screen. Screen-share frames arrive in a separately labeled group after the webcam frames; they show the user's screen, not the user.
- When screen frames are present, treat them as the primary subject: the user is usually asking about, or working on, what's visible there. Read the screen carefully code, documents, error messages, UI state and help with it concretely.
- The LAST screen frame is the most current view of their screen; earlier ones show how it changed while they spoke.
- Frames are downscaled captures: small text may be hard to read. If something crucial is illegible, say what you need ("zoom into the error message" / "make that panel bigger") rather than guessing.
Etiquette:
- Do not narrate or list what you see in every response. Bring up visual observations only when relevant to the user's request or clearly worth mentioning.
- Never comment on the user's physical appearance, attractiveness, or personal attributes visual feedback is strictly about delivery, expression, and body language.
- Frames are periodic snapshots, not continuous video; moments between frames are missing. Don't claim certainty about motion you couldn't have seen.`;
}
if (coachMode) {
instructionsWithDateTime += `\n\n# Practice Session (Coach Mode)
The user started a practice session: they are rehearsing something performative a pitch, presentation, interview answer, or talk and want live coaching. You are their coach for this session.
How to coach:
- Watch and listen for delivery: pacing, filler words, rambling, structure, and (from the webcam frames) posture, eye contact with the camera, facial expressiveness, gesturing, and visible energy.
- After each take or answer, give brief, specific, actionable feedback: 2-3 concrete observations, quoting what you actually saw or heard ("you looked down during the ask", "the opening 'um, so, basically' undercuts your hook"). Then let them go again.
- If they are clearly mid-flow, keep any interjection to one short sentence or stay silent and save it for the break.
- Ask what they're practicing for at the start if it isn't obvious (investor pitch? interview? conference talk?) and tailor feedback to that audience.
- When they say they're done or wrap up, give a structured debrief: what's working, the top 3 things to improve, and one concrete drill or reframe to try next time.
- Be encouraging but honest vague praise wastes their rehearsal time. Never comment on physical appearance; delivery, expression, and body language only.`;
}
if (voiceOutput === 'summary') {
instructionsWithDateTime += `\n\n# Voice Output (MANDATORY — READ THIS FIRST)\nThe user has voice output enabled. THIS IS YOUR #1 PRIORITY: you MUST start your response with <voice></voice> tags. If your response does not begin with <voice> tags, the user will hear nothing — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. Do not start with markdown, headings, or any other text. The literal first characters of your response must be "<voice>".\n2. Place ALL <voice> tags at the BEGINNING of your response, before any detailed content. Do NOT intersperse <voice> tags throughout the response.\n3. Wrap EACH spoken sentence in its own separate <voice> tag so it can be spoken incrementally. Do NOT wrap everything in a single <voice> block.\n4. Use voice as a TL;DR and navigation aid — do NOT read the entire response aloud.\n5. After all <voice> tags, you may include detailed written content (markdown, tables, code, etc.) that will be shown visually but not spoken.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things: the Q2 roadmap timeline, hiring for the backend role, and the client demo next week.</voice>\n<voice>I've pulled out the key details and action items below — the demo prep notes are at the end.</voice>\n\n## Meeting with Alex — March 11\n### Roadmap\n- Agreed to push Q2 launch to April 15...\n(detailed written content continues)\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You have five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you requested and Taylor flagged a contract issue.</voice>\n<voice>There's also a warm intro from a VC partner connecting you with someone at a prospective customer.</voice>\n<voice>I've drafted responses for three of them. The details and drafts are below.</voice>\n\n(email blocks, tables, and detailed content follow)\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a pretty packed day — seven meetings starting with standup at 9.</voice>\n<voice>The big ones are your investor call at 11, lunch with a partner from your lead VC at 12:30, and a customer call at 4.</voice>\n<voice>Your only free block for deep work is 2:30 to 4.</voice>\n\n(calendar block with full event details follows)\n\nExample 4 — User asks: "draft an email to Sam with our metrics"\n\n<voice>Done — I've drafted the email to Sam with your latest WAU and churn numbers.</voice>\n<voice>Take a look at the draft below and send it when you're ready.</voice>\n\n(email block with draft follows)\n\nREMEMBER: If you do not start with <voice> tags, the user hears silence. Always speak first, then write.`;
} else if (voiceOutput === 'full') {
instructionsWithDateTime += `\n\n# Voice Output — Full Read-Aloud (MANDATORY — READ THIS FIRST)\nThe user wants your ENTIRE response spoken aloud. THIS IS YOUR #1 PRIORITY: every single sentence must be wrapped in <voice></voice> tags. If you write anything outside <voice> tags, the user will not hear it — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. The literal first characters of your response must be "<voice>".\n2. Wrap EACH sentence in its own separate <voice> tag so it can be spoken incrementally.\n3. Write your response in a natural, conversational style suitable for listening — no markdown headings, bullet points, or formatting symbols. Use plain spoken language.\n4. Structure the content as if you are speaking to the user directly. Use transitions like "first", "also", "one more thing" instead of visual formatting.\n5. EVERY sentence MUST be inside a <voice> tag. Do not leave ANY content outside <voice> tags. If it's not in a <voice> tag, the user cannot hear it.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things.</voice>\n<voice>First, you discussed the Q2 roadmap timeline and agreed to push the launch to April.</voice>\n<voice>Second, you talked about hiring for the backend role — Alex will send over two candidates by Friday.</voice>\n<voice>And lastly, the client demo is next week on Thursday at 2pm, and you're handling the intro slides.</voice>\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You've got five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you asked for, and Taylor flagged a contract issue that needs your sign-off.</voice>\n<voice>There's a warm intro from a VC partner connecting you with an engineering lead at a potential customer.</voice>\n<voice>And someone from a prospective client wants to confirm your API tier before your call this afternoon.</voice>\n<voice>I've drafted replies for three of them — the metrics update, the intro, and the API question.</voice>\n<voice>The only one I left for you is Taylor's contract redline, since that needs your judgment on the liability cap.</voice>\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a packed day — seven meetings starting with standup at 9.</voice>\n<voice>The highlights are your investor call at 11, lunch with a VC partner at 12:30, and a customer call at 4.</voice>\n<voice>Your only open block for deep work is 2:30 to 4, so plan accordingly.</voice>\n<voice>Oh, and your 1-on-1 with your co-founder is at 5:30 — that's a walking meeting.</voice>\n\nExample 4 — User asks: "how are our metrics looking?"\n\n<voice>Metrics are looking strong this week.</voice>\n<voice>You hit 2,573 weekly active users, which is up 12% week over week.</voice>\n<voice>That means you've crossed the 2,500 milestone — worth calling out in your next investor update.</voice>\n<voice>Churn is down to 4.1%, improving month over month.</voice>\n<voice>The trailing 8-week compound growth rate is about 10%.</voice>\n\nREMEMBER: Start with <voice> immediately. No preamble, no markdown before it. Speak first.`;
}
if (searchEnabled) {
instructionsWithDateTime += `\n\n# Search\nThe user has requested a search. Use the web-search tool to answer their query.`;
}
if (codeMode) {
const agentDisplay = codeMode === 'claude' ? 'Claude Code' : 'Codex';
instructionsWithDateTime += `\n\n# Code Mode (Active) — Agent: ${agentDisplay}
The user has turned on **code mode** and the composer chip is set to **${agentDisplay}** (\`${codeMode}\`). For EVERY coding task this turn, use **${agentDisplay}**, and narrate that agent ("Using ${agentDisplay} to …").
The chip is the single source of truth for which agent runs:
- Do NOT carry over a different agent from earlier in this thread even if a previous run used the other agent, use **${agentDisplay}** now.
- Do NOT switch agents based on an in-chat text request ("use codex", "switch to claude"). The agent only changes when the user toggles the chip; if they ask in chat, tell them to toggle the chip.
**How to run coding work call the \`code_agent_run\` tool** with:
- \`agent\`: \`${codeMode}\` (always — match the chip).
- \`cwd\`: ${codeCwd ? `\`${codeCwd}\` (always — this coding session is pinned to that directory; never use another path)` : `the absolute project/working directory (resolve it per the code-with-agents skill — a path the user named, the "# User Work Directory" block, or ask once)`}.
- \`prompt\`: a clear, self-contained coding instruction.
The tool runs the agent on-device and streams its tool calls, file diffs, and plan into the chat; any action needing approval surfaces as an inline permission card, so you do NOT pre-confirm with an in-chat "reply yes". This chat keeps ONE persistent agent session, so follow-up coding requests automatically resume with full context just call \`code_agent_run\` again. Do NOT shell out to \`acpx\` or \`executeCommand\` for coding, and do NOT fall back to your own file tools.
If the user's message is clearly NOT a coding request (small talk, an unrelated question), answer directly without invoking the coding agent. Code mode signals readiness, not that every message must route through the agent.`;
}
return instructionsWithDateTime;
}
export interface IAgentRuntime {
trigger(runId: string): Promise<void>;
}
@ -861,15 +984,27 @@ export function convertFromMessages(messages: z.infer<typeof Message>[]): ModelM
providerOptions,
});
} else {
// New content parts array — collapse to text for LLM
// New content parts array — collapse text/attachments to text
// for the LLM; inline image parts (video-mode webcam and
// screen-share frames) are passed through as real multimodal
// image parts, grouped under labeled text headers so the
// model knows which images show the user vs their screen.
const textSegments: string[] = userMessageContextPrefix ? [userMessageContextPrefix] : [];
const attachmentLines: string[] = [];
type EncodedImagePart = { type: "image"; image: string; mediaType: string };
const cameraParts: EncodedImagePart[] = [];
const screenParts: EncodedImagePart[] = [];
const frameTimes: string[] = [];
for (const part of msg.content) {
if (part.type === "attachment") {
const sizeStr = part.size ? `, ${formatBytes(part.size)}` : '';
const lineStr = part.lineNumber ? ` (line ${part.lineNumber})` : '';
attachmentLines.push(`- ${part.filename} (${part.mimeType}${sizeStr}) at ${part.path}${lineStr}`);
} else if (part.type === "image") {
const target = part.source === "screen" ? screenParts : cameraParts;
target.push({ type: "image", image: part.data, mediaType: part.mediaType });
if (part.capturedAt) frameTimes.push(part.capturedAt);
} else {
textSegments.push(part.text);
}
@ -883,11 +1018,38 @@ export function convertFromMessages(messages: z.infer<typeof Message>[]): ModelM
}
}
result.push({
role: "user",
content: textSegments.join("\n"),
providerOptions,
});
const imageCount = cameraParts.length + screenParts.length;
if (imageCount > 0) {
const span = frameTimes.length >= 2
? ` spanning ${frameTimes[0]} to ${frameTimes[frameTimes.length - 1]}`
: frameTimes.length === 1
? ` captured at ${frameTimes[0]}`
: '';
const kinds: string[] = [];
if (cameraParts.length > 0) kinds.push(`${cameraParts.length} live webcam frame${cameraParts.length === 1 ? '' : 's'} of the user`);
if (screenParts.length > 0) kinds.push(`${screenParts.length} frame${screenParts.length === 1 ? '' : 's'} of the user's shared screen`);
textSegments.push(`[Video mode: ${kinds.join(' and ')} attached below, each group oldest to newest,${span ? span + ',' : ''} recorded while they composed this message.]`);
const content: Array<{ type: "text"; text: string } | EncodedImagePart> = [
{ type: "text", text: textSegments.join("\n") },
];
if (cameraParts.length > 0) {
content.push({ type: "text", text: "Webcam frames (oldest to newest):" }, ...cameraParts);
}
if (screenParts.length > 0) {
content.push({ type: "text", text: "Screen-share frames (oldest to newest):" }, ...screenParts);
}
result.push({
role: "user",
content,
providerOptions,
});
} else {
result.push({
role: "user",
content: textSegments.join("\n"),
providerOptions,
});
}
}
break;
}
@ -1423,70 +1585,17 @@ export async function* streamAgent({
loopLogger.log('running llm turn');
// stream agent response and build message
const messageBuilder = new StreamStepMessageBuilder();
let instructionsWithDateTime = `${agent.instructions}\n\n${USER_CONTEXT_SYSTEM_INSTRUCTIONS}`;
// Inject Agent Notes context for copilot
if (state.agentName === 'copilot' || state.agentName === 'rowboatx') {
const agentNotesContext = loadAgentNotesContext();
if (agentNotesContext) {
instructionsWithDateTime += `\n\n${agentNotesContext}`;
}
const userWorkDir = loadUserWorkDir(runId);
if (userWorkDir) {
loopLogger.log('injecting user work directory', userWorkDir);
instructionsWithDateTime += `\n\n# User Work Directory
The user has chosen the following directory as their current **work directory**:
\`${userWorkDir}\`
Treat this as the **default location** for file operations whenever the user refers to files generically:
- "list the files", "show me what's in here", "what's the latest report" list or look in the work directory.
- "save this", "export it", "write that to a file" write the output into the work directory unless the user names another location.
- "open the file I was just working on", "the doc from earlier" assume the work directory first.
Use absolute paths rooted at this directory with the \`file-*\` tools. For example, list with \`file-list({ path: "${userWorkDir}" })\`, read text with \`file-readText\`, and write text with \`file-writeText\`. For PDFs, Office docs, images, scanned docs, and other non-text files, use \`parseFile\` or \`LLMParse\` with the absolute path; you do NOT need to copy the file into the workspace first.
**Exceptions these ALWAYS take precedence over the work directory default:**
1. **Knowledge base questions.** If the user asks about anything in the knowledge graph (notes, people, organizations, projects, topics) or paths starting with \`knowledge/\`, use file tools against \`knowledge/\` as documented above. Do NOT redirect those into the work directory.
2. **Explicit paths.** If the user names a different directory or gives an absolute/relative path (e.g. "in ~/Downloads", "from /tmp/foo", "the Desktop"), honor that path exactly and ignore the work-directory default for that request.
3. **Workspace-specific operations.** Anything that obviously belongs in the Rowboat workspace (config files, MCP servers, agent schedules, etc.) stays in the workspace, not the work directory.
Do not announce the work directory unless it's relevant. Just use it.`;
}
}
if (voiceInput) {
loopLogger.log('voice input enabled, injecting voice input prompt');
instructionsWithDateTime += `\n\n# Voice Input\nThe user's message was transcribed from speech. Be aware that:\n- There may be transcription errors. Silently correct obvious ones (e.g. homophones, misheard words). If an error is genuinely ambiguous, briefly mention your interpretation (e.g. "I'm assuming you meant X").\n- Spoken messages are often long-winded. The user may ramble, repeat themselves, or correct something they said earlier in the same message. Focus on their final intent, not every word verbatim.`;
}
if (voiceOutput === 'summary') {
loopLogger.log('voice output enabled (summary mode), injecting voice output prompt');
instructionsWithDateTime += `\n\n# Voice Output (MANDATORY — READ THIS FIRST)\nThe user has voice output enabled. THIS IS YOUR #1 PRIORITY: you MUST start your response with <voice></voice> tags. If your response does not begin with <voice> tags, the user will hear nothing — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. Do not start with markdown, headings, or any other text. The literal first characters of your response must be "<voice>".\n2. Place ALL <voice> tags at the BEGINNING of your response, before any detailed content. Do NOT intersperse <voice> tags throughout the response.\n3. Wrap EACH spoken sentence in its own separate <voice> tag so it can be spoken incrementally. Do NOT wrap everything in a single <voice> block.\n4. Use voice as a TL;DR and navigation aid — do NOT read the entire response aloud.\n5. After all <voice> tags, you may include detailed written content (markdown, tables, code, etc.) that will be shown visually but not spoken.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things: the Q2 roadmap timeline, hiring for the backend role, and the client demo next week.</voice>\n<voice>I've pulled out the key details and action items below — the demo prep notes are at the end.</voice>\n\n## Meeting with Alex — March 11\n### Roadmap\n- Agreed to push Q2 launch to April 15...\n(detailed written content continues)\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You have five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you requested and Taylor flagged a contract issue.</voice>\n<voice>There's also a warm intro from a VC partner connecting you with someone at a prospective customer.</voice>\n<voice>I've drafted responses for three of them. The details and drafts are below.</voice>\n\n(email blocks, tables, and detailed content follow)\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a pretty packed day — seven meetings starting with standup at 9.</voice>\n<voice>The big ones are your investor call at 11, lunch with a partner from your lead VC at 12:30, and a customer call at 4.</voice>\n<voice>Your only free block for deep work is 2:30 to 4.</voice>\n\n(calendar block with full event details follows)\n\nExample 4 — User asks: "draft an email to Sam with our metrics"\n\n<voice>Done — I've drafted the email to Sam with your latest WAU and churn numbers.</voice>\n<voice>Take a look at the draft below and send it when you're ready.</voice>\n\n(email block with draft follows)\n\nREMEMBER: If you do not start with <voice> tags, the user hears silence. Always speak first, then write.`;
} else if (voiceOutput === 'full') {
loopLogger.log('voice output enabled (full mode), injecting voice output prompt');
instructionsWithDateTime += `\n\n# Voice Output — Full Read-Aloud (MANDATORY — READ THIS FIRST)\nThe user wants your ENTIRE response spoken aloud. THIS IS YOUR #1 PRIORITY: every single sentence must be wrapped in <voice></voice> tags. If you write anything outside <voice> tags, the user will not hear it — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. The literal first characters of your response must be "<voice>".\n2. Wrap EACH sentence in its own separate <voice> tag so it can be spoken incrementally.\n3. Write your response in a natural, conversational style suitable for listening — no markdown headings, bullet points, or formatting symbols. Use plain spoken language.\n4. Structure the content as if you are speaking to the user directly. Use transitions like "first", "also", "one more thing" instead of visual formatting.\n5. EVERY sentence MUST be inside a <voice> tag. Do not leave ANY content outside <voice> tags. If it's not in a <voice> tag, the user cannot hear it.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things.</voice>\n<voice>First, you discussed the Q2 roadmap timeline and agreed to push the launch to April.</voice>\n<voice>Second, you talked about hiring for the backend role — Alex will send over two candidates by Friday.</voice>\n<voice>And lastly, the client demo is next week on Thursday at 2pm, and you're handling the intro slides.</voice>\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You've got five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you asked for, and Taylor flagged a contract issue that needs your sign-off.</voice>\n<voice>There's a warm intro from a VC partner connecting you with an engineering lead at a potential customer.</voice>\n<voice>And someone from a prospective client wants to confirm your API tier before your call this afternoon.</voice>\n<voice>I've drafted replies for three of them — the metrics update, the intro, and the API question.</voice>\n<voice>The only one I left for you is Taylor's contract redline, since that needs your judgment on the liability cap.</voice>\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a packed day — seven meetings starting with standup at 9.</voice>\n<voice>The highlights are your investor call at 11, lunch with a VC partner at 12:30, and a customer call at 4.</voice>\n<voice>Your only open block for deep work is 2:30 to 4, so plan accordingly.</voice>\n<voice>Oh, and your 1-on-1 with your co-founder is at 5:30 — that's a walking meeting.</voice>\n\nExample 4 — User asks: "how are our metrics looking?"\n\n<voice>Metrics are looking strong this week.</voice>\n<voice>You hit 2,573 weekly active users, which is up 12% week over week.</voice>\n<voice>That means you've crossed the 2,500 milestone — worth calling out in your next investor update.</voice>\n<voice>Churn is down to 4.1%, improving month over month.</voice>\n<voice>The trailing 8-week compound growth rate is about 10%.</voice>\n\nREMEMBER: Start with <voice> immediately. No preamble, no markdown before it. Speak first.`;
}
if (searchEnabled) {
loopLogger.log('search enabled, injecting search prompt');
instructionsWithDateTime += `\n\n# Search\nThe user has requested a search. Use the web-search tool to answer their query.`;
}
if (codeMode) {
loopLogger.log('code mode enabled, injecting coding-agent context', codeMode);
const agentDisplay = codeMode === 'claude' ? 'Claude Code' : 'Codex';
instructionsWithDateTime += `\n\n# Code Mode (Active) — Agent: ${agentDisplay}
The user has turned on **code mode** and the composer chip is set to **${agentDisplay}** (\`${codeMode}\`). For EVERY coding task this turn, use **${agentDisplay}**, and narrate that agent ("Using ${agentDisplay} to …").
The chip is the single source of truth for which agent runs:
- Do NOT carry over a different agent from earlier in this thread even if a previous run used the other agent, use **${agentDisplay}** now.
- Do NOT switch agents based on an in-chat text request ("use codex", "switch to claude"). The agent only changes when the user toggles the chip; if they ask in chat, tell them to toggle the chip.
**How to run coding work call the \`code_agent_run\` tool** with:
- \`agent\`: \`${codeMode}\` (always — match the chip).
- \`cwd\`: ${codeCwd ? `\`${codeCwd}\` (always — this coding session is pinned to that directory; never use another path)` : `the absolute project/working directory (resolve it per the code-with-agents skill — a path the user named, the "# User Work Directory" block, or ask once)`}.
- \`prompt\`: a clear, self-contained coding instruction.
The tool runs the agent on-device and streams its tool calls, file diffs, and plan into the chat; any action needing approval surfaces as an inline permission card, so you do NOT pre-confirm with an in-chat "reply yes". This chat keeps ONE persistent agent session, so follow-up coding requests automatically resume with full context just call \`code_agent_run\` again. Do NOT shell out to \`acpx\` or \`executeCommand\` for coding, and do NOT fall back to your own file tools.
If the user's message is clearly NOT a coding request (small talk, an unrelated question), answer directly without invoking the coding agent. Code mode signals readiness, not that every message must route through the agent.`;
}
const composeCopilotContext = state.agentName === 'copilot' || state.agentName === 'rowboatx';
const instructionsWithDateTime = composeSystemInstructions({
instructions: agent.instructions,
agentNotesContext: composeCopilotContext ? loadAgentNotesContext() : null,
userWorkDir: composeCopilotContext ? loadUserWorkDir(runId) : null,
voiceInput,
voiceOutput,
searchEnabled,
codeMode,
codeCwd,
});
let streamError: string | null = null;
for await (const event of streamLlm(
model,

View file

@ -1,6 +1,6 @@
import { AsyncLocalStorage } from 'node:async_hooks';
export type UseCase = 'copilot_chat' | 'live_note_agent' | 'background_task_agent' | 'meeting_note' | 'knowledge_sync' | 'code_session' | 'app_llm_generate' | 'app_copilot_run';
export type UseCase = 'copilot_chat' | 'live_note_agent' | 'background_task_agent' | 'meeting_note' | 'meeting_prep' | 'knowledge_sync' | 'code_session' | 'app_llm_generate' | 'app_copilot_run';
export interface UseCaseContext {
useCase: UseCase;

View file

@ -114,7 +114,7 @@ ${codeModeEnabled
? `**Code with Agents:** When users ask you to write code, build a project, create a script, fix a bug, or do any software development task — **including simple things like "create a .c file" or "write a hello-world in Python"** — your FIRST action MUST be \`loadSkill('code-with-agents')\`. Do NOT reach for \`executeCommand\` (PowerShell / bash / shell) or any workspace file tool to do code work yourself before loading this skill. The skill decides whether to delegate to Claude Code / Codex (via acpx) or hand control back to you, and it presents the user a one-click choice when needed. Paths outside the Rowboat workspace root (e.g. \`G:/...\`, \`~/projects/...\`) are NORMAL for coding tasks — do NOT raise "outside workspace" concerns or fall back to your own tools.`
: `**Code with Agents (disabled):** Code mode is currently OFF in the user's settings. Do NOT load \`code-with-agents\` and do NOT call acpx. Handle coding requests yourself with your normal tools if you can. After answering, add a final line letting the user know they can delegate coding to Claude Code or Codex by enabling Code Mode in Settings → Code Mode.`}
**App Control:** When users ask you to open notes, show the bases or graph view, filter or search notes, or manage saved views, load the \`app-navigation\` skill first. It provides structured guidance for navigating the app UI and controlling the knowledge base view.
**App Control (drive the app):** You can drive the Rowboat UI the user is looking at open any view (email, meetings, background agents, chat history, knowledge, workspace, code, bases, graph), READ what a view contains (\`read-view\` returns emails / background agents / past chats as data while showing the view on screen), and open specific items (an email thread, a note, an agent, a past chat). When users ask to open, show, find, or ask about anything that lives inside Rowboat, load the \`app-navigation\` skill first — it documents the show-while-telling pattern. This matters most on calls: navigate so the user sees what you see, then answer briefly.
**Background Tasks (Self-Running Work):** Rowboat can run *background tasks* persistent instructions the agent fires on a schedule and/or in response to incoming emails / calendar events. A bg-task either maintains a snapshot in its \`index.md\` (digest, dashboard, rolling summary) or performs a recurring side-effect (send a Slack message, draft an email, post to a webhook, call an API). This is the flagship surface for *anything recurring*.
@ -295,7 +295,7 @@ ${runtimeContextPrompt}
- \`addMcpServer\`, \`listMcpServers\`, \`listMcpTools\`, \`executeMcpTool\` - MCP server management and execution
- \`loadSkill\` - Skill loading
${slackToolsLine}- \`web-search\` - Search the web. Returns rich results with full text, highlights, and metadata. The \`category\` parameter defaults to \`general\` (full web search) — only use a specific category like \`news\`, \`company\`, \`research paper\` etc. when the query is clearly about that type. For everyday queries (weather, restaurants, prices, how-to), use \`general\`.
- \`app-navigation\` - Control the app UI: open notes, switch views, filter/search the knowledge base, manage saved views. **Load the \`app-navigation\` skill before using this tool.**
- \`app-navigation\` - Drive the app UI: open any view, read a view's contents (emails / background agents / chat history), open specific items (email thread, note, agent, past chat), filter/search the knowledge base, manage saved views. **Load the \`app-navigation\` skill before using this tool.**
- \`browser-control\` - Control the embedded browser pane: open sites, inspect the live page, switch tabs, and interact with indexed page elements. **Load the \`browser-control\` skill before using this tool.**
- \`save-to-memory\` - Save observations about the user to the agent memory system. Use this proactively during conversations.
${composioToolsLine}

View file

@ -1,82 +1,91 @@
export const skill = String.raw`
# App Navigation Skill
# App Driving Skill
You have access to the **app-navigation** tool which lets you control the Rowboat UI directly opening notes, switching views, filtering the knowledge base, and creating saved views.
You have the **app-navigation** tool: you can DRIVE the Rowboat app the user
is looking at open any view, read what a view contains, open specific items
(an email thread, a note, a background agent, a past chat), filter the
knowledge base, and manage saved views. Navigation happens on the USER'S
screen: when you open something, they watch it open.
## The core pattern: show while telling
When the user asks about something that lives inside Rowboat ("what emails do
I have?", "what background agents are running?", "open the note about Acme"),
don't answer blind. Drive:
1. **read-view** the relevant view this returns the actual data AND
navigates the user's screen to that view at the same time.
2. Answer from the returned data, concisely.
3. If they ask about one item ("open the one from Arjun"), **open-item** it
it appears on their screen and summarize what's in it if useful.
This matters most during a call: the user is talking to you hands-free and
watching the screen. Navigate so they see what you see, and keep spoken
answers short.
## Actions
### read-view read a view's contents (and show it)
Returns the same data the view renders; the app simultaneously navigates to
that view so the user sees it.
- ` + "`view: \"email\"`" + ` → latest important inbox threads: ` + "`{ threadId, subject, from, date, unread, summary }`" + `.
Pass ` + "`query`" + ` to search instead (sender name, subject words e.g. ` + "`query: \"from Arjun\"`" + ` or just ` + "`\"Arjun\"`" + `).
- ` + "`view: \"bg-tasks\"`" + ` → background agents: ` + "`{ name, slug, active, triggers, lastRunAt, lastRunSummary, lastRunError }`" + `.
- ` + "`view: \"chat-history\"`" + ` → past chats: ` + "`{ sessionId, title, updatedAt, turnCount }`" + `.
- ` + "`limit`" + ` (optional, default 15).
For notes, meetings, and live notes use the ` + "`file-*`" + ` tools (they are
markdown files in the workspace) and then open-note / open-item to show them.
### open-item open one specific thing on screen
- ` + "`kind: \"email-thread\"`" + ` + ` + "`threadId`" + ` (from read-view email)
- ` + "`kind: \"note\"`" + ` + ` + "`path`" + `
- ` + "`kind: \"bg-task\"`" + ` + ` + "`taskName`" + ` (from read-view bg-tasks; validated against real tasks)
- ` + "`kind: \"session\"`" + ` + ` + "`sessionId`" + ` (from read-view chat-history)
### open-view just switch the screen
` + "`view`" + `: ` + "`home | email | meetings | live-notes | bg-tasks | chat-history | knowledge | workspace | code | bases | graph`" + `
Use when the user asks to "go to"/"show" a view without a question to answer.
### open-note
Open a specific knowledge file in the editor pane.
Open a knowledge file in the editor. ` + "`path`" + `: full workspace-relative path
(e.g. ` + "`knowledge/People/John Smith.md`" + `). Use ` + "`file-grep`" + ` first if unsure
of the exact path.
**When to use:** When the user asks to see, open, or view a specific note (e.g., "open John's note", "show me the Acme project page").
### update-base-view / get-base-state / create-base
Knowledge-base table control (unchanged):
- ` + "`update-base-view`" + `: ` + "`filters`" + ` (` + "`set/add/remove/clear`" + ` of ` + "`{category, value}`" + `),
` + "`sort`" + ` (` + "`{field, dir}`" + `), ` + "`search`" + `. **Never pass ` + "`columns`" + ` unless the user
explicitly asks to change columns** it overrides their layout.
- ` + "`get-base-state`" + `: available filter categories/values and note count.
- ` + "`create-base`" + `: save the current view configuration under ` + "`name`" + `.
**Parameters:**
- ` + "`path`" + `: Full workspace-relative path (e.g., ` + "`knowledge/People/John Smith.md`" + `)
## Worked examples
**Tips:**
- Use ` + "`file-grep`" + ` first to find the exact path if you're unsure of the filename.
- Always pass the full ` + "`knowledge/...`" + ` path, not just the filename.
**"What emails do I have?"** (on a call)
1. ` + "`app-navigation({ action: \"read-view\", view: \"email\" })`" + ` — email view opens on their screen.
2. Speak the highlights: "You've got six new ones — the ones that matter are from Arjun about the deck and from Stripe about billing."
### open-view
Switch the UI to the graph or bases view.
**"Open the one from Arjun."**
1. Find Arjun's thread in the data you already have (or ` + "`read-view`" + ` with ` + "`query: \"Arjun\"`" + `).
2. ` + "`app-navigation({ action: \"open-item\", kind: \"email-thread\", threadId: \"...\" })`" + `
3. "It's open — he's asking whether Thursday works for the pitch review."
**When to use:** When the user asks to see the knowledge graph, view all notes, or open the bases/table view.
**"What background agents do I have?"**
1. ` + "`app-navigation({ action: \"read-view\", view: \"bg-tasks\" })`" + `
2. "Three: the inbox summarizer ran an hour ago, the meeting-prep agent is active, and the Linear digest failed its last run — want me to open that one?"
**Parameters:**
- ` + "`view`" + `: ` + "`\"graph\"`" + ` or ` + "`\"bases\"`" + `
**"Show me all active customers"**
1. ` + "`get-base-state`" + ` to see available categories, then
2. ` + "`update-base-view`" + ` with ` + "`filters.set: [{ category: \"relationship\", value: \"customer\" }]`" + `
### update-base-view
Change filters, columns, sort order, or search in the bases (table) view.
**When to use:** When the user asks to find, filter, sort, or search notes. Examples: "show me all active customers", "filter by topic=hiring", "sort by name", "search for pricing".
**Parameters:**
- ` + "`filters`" + `: Object with ` + "`set`" + `, ` + "`add`" + `, ` + "`remove`" + `, or ` + "`clear`" + ` each takes an array of ` + "`{ category, value }`" + ` pairs.
- ` + "`set`" + `: Replace ALL current filters with these.
- ` + "`add`" + `: Append filters without removing existing ones.
- ` + "`remove`" + `: Remove specific filters.
- ` + "`clear: true`" + `: Remove all filters.
- ` + "`columns`" + `: Object with ` + "`set`" + `, ` + "`add`" + `, or ` + "`remove`" + ` each takes an array of column names (frontmatter keys).
- ` + "`sort`" + `: ` + "`{ field, dir }`" + ` where dir is ` + "`\"asc\"`" + ` or ` + "`\"desc\"`" + `.
- ` + "`search`" + `: Free-text search string.
**Tips:**
- If unsure what categories/values are available, call ` + "`get-base-state`" + ` first.
- For "show me X", prefer ` + "`filters.set`" + ` to start fresh rather than ` + "`filters.add`" + `.
- Categories come from frontmatter keys (e.g., relationship, status, topic, type).
- **CRITICAL: Do NOT pass ` + "`columns`" + ` unless the user explicitly asks to show/hide specific columns.** Omit the ` + "`columns`" + ` parameter entirely when only filtering, sorting, or searching. Passing ` + "`columns`" + ` will override the user's current column layout and can make the view appear empty.
### get-base-state
Retrieve information about what's in the knowledge base available filter categories, values, and note count.
**When to use:** When you need to know what properties exist before filtering, or when the user asks "what can I filter by?", "how many notes are there?", etc.
**Parameters:**
- ` + "`base_name`" + ` (optional): Name of a saved base to inspect.
### create-base
Save the current view configuration as a named base.
**When to use:** When the user asks to save a filtered view, create a saved search, or says "save this as [name]".
**Parameters:**
- ` + "`name`" + `: Human-readable name for the base.
## Workflow Example
1. User: "Show me all people who are customers"
2. First, check what properties are available:
` + "`app-navigation({ action: \"get-base-state\" })`" + `
3. Apply filters based on the available properties:
` + "`app-navigation({ action: \"update-base-view\", filters: { set: [{ category: \"relationship\", value: \"customer\" }] } })`" + `
4. If the user wants to save it:
` + "`app-navigation({ action: \"create-base\", name: \"Customers\" })`" + `
## Important Notes
- The ` + "`update-base-view`" + ` action will automatically navigate to the bases view if the user isn't already there.
- ` + "`open-note`" + ` validates that the file exists before navigating.
- Filter categories and values come from frontmatter in knowledge files.
- **Never send ` + "`columns`" + ` or ` + "`sort`" + ` with ` + "`update-base-view`" + ` unless the user specifically asks to change them.** Only pass the parameters you intend to change omitted parameters are left untouched.
## Notes
- read-view/open-view/open-item change what the user is looking at that is
the point, but don't bounce their screen around needlessly; navigate when
it serves the question.
- open-note and open-item validate the target exists before navigating.
- update-base-view auto-navigates to the bases view.
`;
export default skill;

View file

@ -24,6 +24,9 @@ import { ICodeModeConfigRepo } from "../../code-mode/repo.js";
import type { ApprovalPolicy } from "@x/shared/dist/code-mode.js";
import type { ICodeProjectsRepo } from "../../code-mode/projects/repo.js";
import * as gitService from "../../code-mode/git/service.js";
import { listImportantThreads, searchThreads } from "../../knowledge/sync_gmail.js";
import { listTasks as listBackgroundTasks } from "../../background-tasks/fileops.js";
import type { ISessions } from "../../sessions/api.js";
// Inputs for the bg-task builtin tools. Reuse the canonical schema field
// descriptions; only `triggers` gets a tighter contextual override (the
@ -1068,15 +1071,24 @@ export const BuiltinTools: z.infer<typeof BuiltinToolsSchema> = {
// ============================================================================
'app-navigation': {
description: 'Control the app UI - navigate to notes, switch views, filter/search the knowledge base, and manage saved views.',
description: 'Drive the Rowboat app UI: navigate to any view, read what a view contains (emails, background agents, chat history), open specific items (an email thread, a note, an agent, a past chat), filter/search the knowledge base, and manage saved views. Use it to SHOW the user things while telling them — navigation happens on their screen.',
inputSchema: z.object({
action: z.enum(["open-note", "open-view", "open-app", "update-base-view", "get-base-state", "create-base"]).describe("The navigation action to perform"),
action: z.enum(["open-note", "open-view", "open-app", "read-view", "open-item", "update-base-view", "get-base-state", "create-base"]).describe("The navigation action to perform"),
// open-note
path: z.string().optional().describe("Knowledge file path for open-note, e.g. knowledge/People/John.md"),
// open-app
appId: z.string().optional().describe("Mini App id (folder name under ~/.rowboat/apps) for open-app — opens it in the middle pane under Mini Apps."),
// open-view
view: z.enum(["bases", "graph"]).optional().describe("Which view to open (for open-view action)"),
appId: z.string().optional().describe("App folder slug under ~/.rowboat/apps (for open-app) — opens the app in the middle pane."),
// open-view / read-view
view: z.enum(["home", "email", "meetings", "live-notes", "bg-tasks", "chat-history", "knowledge", "workspace", "code", "bases", "graph"]).optional().describe("Which view to open (open-view) or read (read-view; supported for read: email, bg-tasks, chat-history)"),
// read-view (email)
query: z.string().optional().describe("For read-view on email: search query (sender name, subject words, etc.). Omit to list the latest important inbox threads."),
limit: z.number().int().min(1).max(50).optional().describe("For read-view: max items to return (default 15)"),
// open-item
kind: z.enum(["email-thread", "note", "bg-task", "session"]).optional().describe("What to open (for open-item)"),
threadId: z.string().optional().describe("Gmail thread id (open-item kind=email-thread; get it from read-view email)"),
taskName: z.string().optional().describe("Background task/agent name (open-item kind=bg-task; get it from read-view bg-tasks)"),
sessionId: z.string().optional().describe("Chat session id (open-item kind=session; get it from read-view chat-history)"),
// update-base-view
filters: z.object({
set: z.array(z.object({ category: z.string(), value: z.string() })).optional().describe("Replace all filters with these"),
@ -1136,6 +1148,110 @@ export const BuiltinTools: z.infer<typeof BuiltinToolsSchema> = {
return { success: true, action: 'open-app', appId, appName };
}
case 'read-view': {
// Returns the same data the view renders, so the assistant
// can answer precisely — and the renderer navigates to the
// view at the same time so the user SEES what's being read.
const view = input.view as string;
const limit = (input.limit as number | undefined) ?? 15;
try {
switch (view) {
case 'email': {
const query = (input.query as string | undefined)?.trim();
const result = query
? await searchThreads(query, { limit })
: listImportantThreads({ limit });
const threads = (result.threads ?? []).slice(0, limit).map((t) => ({
threadId: t.threadId,
subject: t.subject ?? '(no subject)',
from: t.from ?? '',
date: t.date ?? '',
unread: t.unread ?? false,
summary: t.summary ? t.summary.slice(0, 200) : undefined,
}));
return { success: true, action: 'read-view', view, query, threads };
}
case 'bg-tasks': {
const { items } = await listBackgroundTasks({ limit });
const agents = items.map((t) => ({
name: t.name,
slug: t.slug,
active: t.active,
triggers: t.triggers,
lastRunAt: t.lastRunAt,
lastRunSummary: t.lastRunSummary ? t.lastRunSummary.slice(0, 200) : undefined,
lastRunError: t.lastRunError ? t.lastRunError.slice(0, 200) : undefined,
}));
return { success: true, action: 'read-view', view, agents };
}
case 'chat-history': {
const sessions = container.resolve<ISessions>('sessions')
.listSessions()
.slice(0, limit)
.map((s) => ({
sessionId: s.sessionId,
title: s.title ?? '(untitled)',
updatedAt: s.updatedAt,
turnCount: s.turnCount,
}));
return { success: true, action: 'read-view', view, sessions };
}
default:
return {
success: false,
error: `read-view supports: email, bg-tasks, chat-history. For notes/meetings/live-notes use the file-* tools (they are files under the workspace); for other views use open-view and describe what you need.`,
};
}
} catch (error) {
return {
success: false,
error: error instanceof Error ? error.message : `Failed to read ${view}`,
};
}
}
case 'open-item': {
const kind = input.kind as string;
switch (kind) {
case 'email-thread': {
const threadId = input.threadId as string | undefined;
if (!threadId) return { success: false, error: 'threadId is required for kind=email-thread' };
return { success: true, action: 'open-item', kind, threadId };
}
case 'note': {
const filePath = input.path as string | undefined;
if (!filePath) return { success: false, error: 'path is required for kind=note' };
const result = await files.exists(filePath);
if (!result.exists) return { success: false, error: `File not found: ${filePath}` };
return { success: true, action: 'open-item', kind, path: filePath };
}
case 'bg-task': {
const taskName = input.taskName as string | undefined;
if (!taskName) return { success: false, error: 'taskName is required for kind=bg-task' };
// Validate (and canonicalize) against the real task list.
const { items: tasks } = await listBackgroundTasks({});
const match = tasks.find(
(t) => t.name === taskName || t.slug === taskName
|| t.name.toLowerCase() === taskName.toLowerCase(),
);
if (!match) {
return {
success: false,
error: `No background task named "${taskName}". Known tasks: ${tasks.map((t) => t.name).join(', ') || '(none)'}`,
};
}
return { success: true, action: 'open-item', kind, taskName: match.name };
}
case 'session': {
const sessionId = input.sessionId as string | undefined;
if (!sessionId) return { success: false, error: 'sessionId is required for kind=session' };
return { success: true, action: 'open-item', kind, sessionId };
}
default:
return { success: false, error: `Unknown item kind: ${kind}` };
}
}
case 'update-base-view': {
const updates: Record<string, unknown> = {};
if (input.filters) updates.filters = input.filters;

View file

@ -1,9 +1,8 @@
import type { BackgroundTask, BackgroundTaskTriggerType } from '@x/shared/dist/background-task.js';
import { PrefixLogger } from '@x/shared/dist/prefix-logger.js';
import { fetchTask, patchTask, prependRunId } from './fileops.js';
import { createRun, createMessage } from '../runs/runs.js';
import { getBackgroundTaskAgentModel } from '../models/defaults.js';
import { extractAgentResponse, waitForRunCompletion } from '../agents/utils.js';
import { startHeadlessAgent } from '../agents/headless-app.js';
import { buildTriggerBlock } from '../agents/build-trigger-block.js';
import { backgroundTaskBus } from './bus.js';
import { withUseCase } from '../analytics/use_case.js';
@ -139,20 +138,25 @@ export async function runBackgroundTask(
}
const model = task.model || await getBackgroundTaskAgentModel();
const agentRun = await createRun({
agentId: 'background-task-agent',
model,
...(task.provider ? { provider: task.provider } : {}),
useCase: 'background_task_agent',
// Granular trigger as analytics sub-use-case — matches live-note's
// pattern at runner.ts:149.
subUseCase: trigger,
});
// Establish the use-case context for the whole turn so every tool the
// agent calls (notably notify-user) reads `background_task_agent` via
// getCurrentUseCase(); the AsyncLocalStorage context set here flows
// through the turn's async execution chain.
const handle = await withUseCase(
{ useCase: 'background_task_agent', subUseCase: trigger },
() => startHeadlessAgent({
agentId: 'background-task-agent',
message: buildMessage(slug, task, trigger, context, codeProject),
model,
...(task.provider ? { provider: task.provider } : {}),
throwOnError: true,
}),
);
const runId = agentRun.id;
// Record this run in the task's runs.log pointer file (newest first).
// The transcript itself lives at the global $WorkDir/runs/<runId>.jsonl
// — runs.log is just an index that ties runIds to this task.
const runId = handle.turnId;
// Record this turn in the task's runs.log pointer file (newest first).
// The transcript itself lives at $WorkDir/storage/turns/YYYY/MM/DD/
// — runs.log is just an index that ties turn ids to this task.
await prependRunId(slug, runId);
const startedAt = new Date().toISOString();
@ -184,20 +188,7 @@ export async function runBackgroundTask(
});
try {
// Establish the use-case context for the whole run so every tool the
// agent calls (notably notify-user) reads `background_task_agent` via
// getCurrentUseCase(). createMessage synchronously fires
// agentRuntime.trigger(), so the detached run loop — and the tool
// calls within it — inherit this AsyncLocalStorage context. (The
// runtime's own enterUseCase runs inside an async generator and
// doesn't reliably propagate to tool execution, so we set it here at
// the trigger point instead.)
await withUseCase(
{ useCase: 'background_task_agent', subUseCase: trigger },
() => createMessage(runId, buildMessage(slug, task, trigger, context, codeProject)),
);
await waitForRunCompletion(runId, { throwOnError: true });
const summary = await extractAgentResponse(runId);
const { summary } = await handle.done;
// Success — bump cycle anchor, refresh summary, clear any prior error.
await patchTask(slug, {

View file

@ -0,0 +1,278 @@
import { describe, expect, it, vi } from "vitest";
import type { SessionIndexEntry } from "@x/shared/dist/sessions.js";
import type { TurnStreamEvent } from "@x/shared/dist/turns.js";
import { EmitterSessionBus } from "../sessions/bus.js";
import { TurnInputError } from "../turns/api.js";
import type { ISessions } from "../sessions/api.js";
import { ChannelBridge, type ModelChoice } from "./bridge.js";
const SENDER = "test:1";
function entry(overrides: Partial<SessionIndexEntry>): SessionIndexEntry {
return {
sessionId: "s1",
createdAt: "2026-07-01T00:00:00Z",
updatedAt: "2026-07-01T00:00:00Z",
turnCount: 1,
latestTurnStatus: "completed",
...overrides,
};
}
function completedEvent(turnId: string, text: string): TurnStreamEvent {
return {
type: "turn_completed",
turnId,
ts: "2026-07-01T00:00:00Z",
output: { role: "assistant", content: text },
finishReason: "stop",
usage: {},
} as unknown as TurnStreamEvent;
}
function askEvent(turnId: string, question: string, options?: string[]): TurnStreamEvent {
return {
type: "turn_suspended",
turnId,
ts: "2026-07-01T00:00:00Z",
pendingPermissions: [],
pendingAsyncTools: [
{
toolCallId: "call_1",
toolId: "builtin:ask-human",
toolName: "ask-human",
input: { question, ...(options ? { options } : {}) },
},
],
usage: {},
} as unknown as TurnStreamEvent;
}
interface Harness {
bridge: ChannelBridge;
bus: EmitterSessionBus;
replies: string[];
reply: (text: string) => Promise<void>;
sessions: {
createSession: ReturnType<typeof vi.fn>;
sendMessage: ReturnType<typeof vi.fn>;
respondToAskHuman: ReturnType<typeof vi.fn>;
stopTurn: ReturnType<typeof vi.fn>;
getTurn: ReturnType<typeof vi.fn>;
listSessions: ReturnType<typeof vi.fn>;
};
listModels: ReturnType<typeof vi.fn>;
publish: (turnId: string, event: TurnStreamEvent) => void;
}
const MODELS: ModelChoice[] = [
{ provider: "anthropic", model: "claude-fable-5", label: "Fable 5 — Anthropic" },
{ provider: "anthropic", model: "claude-sonnet-5", label: "Sonnet 5 — Anthropic" },
{ provider: "openai", model: "gpt-5", label: "GPT-5 — OpenAI" },
];
function harness(entries: SessionIndexEntry[] = []): Harness {
const bus = new EmitterSessionBus();
const publish = (turnId: string, event: TurnStreamEvent) =>
bus.publish({ kind: "turn-event", sessionId: "s1", turnId, event });
const sessions = {
createSession: vi.fn(async () => "s1"),
sendMessage: vi.fn(async () => ({ turnId: "t1" })),
respondToAskHuman: vi.fn(async () => undefined),
stopTurn: vi.fn(async () => undefined),
getTurn: vi.fn(async () => ({ turnId: "t1", events: [] })),
listSessions: vi.fn(() => entries),
};
const listModels = vi.fn(async () => MODELS);
const bridge = new ChannelBridge({
sessions: sessions as unknown as ISessions,
sessionBus: bus,
listModels,
});
const replies: string[] = [];
const reply = async (text: string) => {
replies.push(text);
};
return { bridge, bus, replies, reply, sessions, listModels, publish };
}
// Settle the turn as soon as sendMessage is called: the watcher subscribes
// before sendMessage, so a synchronous publish lands in its buffer — the
// exact race the buffering exists for.
function settleOnSend(h: Harness, event: TurnStreamEvent, turnId = "t1"): void {
h.sessions.sendMessage.mockImplementation(async () => {
h.publish(turnId, event);
return { turnId };
});
}
describe("ChannelBridge commands", () => {
it("replies with help text", async () => {
const h = harness();
await h.bridge.handleInbound(SENDER, "help", h.reply);
expect(h.replies).toHaveLength(1);
expect(h.replies[0]).toContain("Rowboat commands");
});
it("lists recent sessions newest-first and resumes by index", async () => {
const h = harness([
entry({ sessionId: "old", title: "Old chat", updatedAt: "2026-07-01T00:00:00Z" }),
entry({ sessionId: "new", title: "New chat", updatedAt: "2026-07-02T00:00:00Z" }),
]);
await h.bridge.handleInbound(SENDER, "list", h.reply);
expect(h.replies[0]).toContain("1. New chat");
expect(h.replies[0]).toContain("2. Old chat");
await h.bridge.handleInbound(SENDER, "resume 2", h.reply);
expect(h.replies[1]).toContain('Resumed "Old chat"');
settleOnSend(h, completedEvent("t1", "done"));
await h.bridge.handleInbound(SENDER, "hello again", h.reply);
expect(h.sessions.sendMessage).toHaveBeenCalledWith(
"old",
expect.objectContaining({ content: "hello again" }),
expect.objectContaining({ autoPermission: true }),
);
});
it("rejects resume with an out-of-range index", async () => {
const h = harness([entry({ sessionId: "s1", title: "Only chat" })]);
await h.bridge.handleInbound(SENDER, "resume 5", h.reply);
expect(h.replies[0]).toContain("No chat #5");
});
});
describe("ChannelBridge model selection", () => {
it("lists models and applies a by-index selection to later turns", async () => {
const h = harness();
await h.bridge.handleInbound(SENDER, "model", h.reply);
expect(h.replies[0]).toContain("1. Fable 5 — Anthropic");
expect(h.replies[0]).toContain("app default");
await h.bridge.handleInbound(SENDER, "model 3", h.reply);
expect(h.replies[1]).toContain("GPT-5");
settleOnSend(h, completedEvent("t1", "done"));
await h.bridge.handleInbound(SENDER, "hello", h.reply);
expect(h.sessions.sendMessage).toHaveBeenCalledWith(
"s1",
expect.anything(),
expect.objectContaining({
agent: {
agentId: "copilot",
overrides: { model: { provider: "openai", model: "gpt-5" } },
},
}),
);
});
it("selects by unique name match and resets on 'model default'", async () => {
const h = harness();
await h.bridge.handleInbound(SENDER, "model fable", h.reply);
expect(h.replies[0]).toContain("Fable 5");
settleOnSend(h, completedEvent("t1", "done"));
await h.bridge.handleInbound(SENDER, "hi", h.reply);
expect(h.sessions.sendMessage).toHaveBeenCalledWith(
"s1",
expect.anything(),
expect.objectContaining({
agent: expect.objectContaining({
overrides: { model: { provider: "anthropic", model: "claude-fable-5" } },
}),
}),
);
await h.bridge.handleInbound(SENDER, "model default", h.reply);
settleOnSend(h, completedEvent("t2", "done"), "t2");
await h.bridge.handleInbound(SENDER, "hi again", h.reply);
expect(h.sessions.sendMessage).toHaveBeenLastCalledWith(
"s1",
expect.anything(),
expect.objectContaining({ agent: { agentId: "copilot" } }),
);
});
it("asks for disambiguation on an ambiguous name and rejects unknown ones", async () => {
const h = harness();
await h.bridge.handleInbound(SENDER, "model claude", h.reply);
expect(h.replies[0]).toContain("matches 2 models");
await h.bridge.handleInbound(SENDER, "model llama", h.reply);
expect(h.replies[1]).toContain('No model matching "llama"');
});
});
describe("ChannelBridge message flow", () => {
it("creates a session, acks, and delivers the completed text", async () => {
const h = harness();
settleOnSend(h, completedEvent("t1", "The answer is 42."));
await h.bridge.handleInbound(SENDER, "what is the answer?", h.reply);
expect(h.sessions.createSession).toHaveBeenCalledOnce();
expect(h.replies[0]).toContain("Working on it");
expect(h.replies[1]).toBe("The answer is 42.");
});
it("delivers a failed turn as an error reply", async () => {
const h = harness();
settleOnSend(h, {
type: "turn_failed",
turnId: "t1",
ts: "2026-07-01T00:00:00Z",
error: "model exploded",
usage: {},
} as unknown as TurnStreamEvent);
await h.bridge.handleInbound(SENDER, "do a thing", h.reply);
expect(h.replies[1]).toContain("model exploded");
});
it("relays the ask-human question text and options, then routes the answer", async () => {
const h = harness();
settleOnSend(h, askEvent("t1", "Which lane?", ["fast", "slow"]));
await h.bridge.handleInbound(SENDER, "start task", h.reply);
expect(h.replies[1]).toContain("Which lane?");
expect(h.replies[1]).toContain("1. fast");
// The answer resolves the ask; the turn then completes.
h.sessions.respondToAskHuman.mockImplementation(async () => {
h.publish("t1", completedEvent("t1", "Took the fast lane."));
});
await h.bridge.handleInbound(SENDER, "fast", h.reply);
expect(h.sessions.respondToAskHuman).toHaveBeenCalledWith("t1", "call_1", "fast");
expect(h.replies[3]).toBe("Took the fast lane.");
});
it("falls back to a normal message when the ask was already answered elsewhere", async () => {
const h = harness();
settleOnSend(h, askEvent("t1", "Which lane?"));
await h.bridge.handleInbound(SENDER, "start task", h.reply);
h.sessions.respondToAskHuman.mockRejectedValue(
new TurnInputError("no pending async tool call call_1"),
);
settleOnSend(h, completedEvent("t2", "Handled as a new message."), "t2");
await h.bridge.handleInbound(SENDER, "actually do this instead", h.reply);
expect(h.sessions.sendMessage).toHaveBeenLastCalledWith(
"s1",
expect.objectContaining({ content: "actually do this instead" }),
expect.anything(),
);
expect(h.replies.at(-1)).toBe("Handled as a new message.");
});
it("reports busy while a turn is in flight", async () => {
const h = harness();
let releaseTurn!: () => void;
h.sessions.sendMessage.mockImplementation(async () => {
releaseTurn = () => h.publish("t1", completedEvent("t1", "finally"));
return { turnId: "t1" };
});
const first = h.bridge.handleInbound(SENDER, "slow task", h.reply);
await new Promise((resolve) => setTimeout(resolve, 0));
await h.bridge.handleInbound(SENDER, "impatient follow-up", h.reply);
expect(h.replies.some((r) => r.includes("Still working"))).toBe(true);
releaseTurn();
await first;
expect(h.replies.at(-1)).toBe("finally");
});
});

View file

@ -0,0 +1,540 @@
import type { SessionIndexEntry } from "@x/shared/dist/sessions.js";
import { reduceTurn, type TurnStreamEvent } from "@x/shared/dist/turns.js";
import { assistantText, lastAssistantText } from "../agents/headless.js";
import { TurnInputError } from "../turns/api.js";
import { ASK_HUMAN_TOOL } from "../turns/bridges/real-agent-resolver.js";
import { TurnNotSettledError, type ISessions } from "../sessions/api.js";
import type { EmitterSessionBus } from "../sessions/bus.js";
// Transport-agnostic command layer: inbound texts from a messaging channel
// are parsed into commands (list / resume / new / stop / status) or forwarded
// into a regular chat session; the turn's final assistant text is sent back
// through the transport's reply callback. Turns run with autoPermission and
// show up live in the desktop UI like any other session.
const AGENT_ID = "copilot";
const TURN_TIMEOUT_MS = 30 * 60 * 1000;
const LIST_LIMIT = 10;
// Telegram caps messages at 4096 chars; WhatsApp is far higher. Long replies
// are chunked, then truncated — the desktop app has the full text.
const REPLY_CHUNK_SIZE = 3500;
const MAX_REPLY_CHUNKS = 3;
const ASK_HUMAN_TOOL_ID = `builtin:${ASK_HUMAN_TOOL}`;
const HELP_TEXT = [
"🤖 Rowboat commands:",
"• list — recent chats",
"• resume N — continue chat N from the list",
"• new [message] — start a fresh chat",
"• model [N or name] — pick the model (\"model default\" resets)",
"• status — current chat and what it's doing",
"• stop — cancel the running task",
"",
"Anything else is sent to the current chat.",
].join("\n");
const MODEL_LIST_LIMIT = 20;
export type ReplyFn = (text: string) => Promise<void>;
export interface ModelChoice {
provider: string;
model: string;
label: string;
}
interface SenderState {
activeSessionId: string | null;
// sessionIds as last shown by `list` (1-based indexing for `resume N`).
lastList: string[];
// Choices as last shown by `model` (1-based indexing for `model N`).
lastModels: ModelChoice[];
// Per-sender override passed on every turn; null = app default model.
model: { provider: string; model: string } | null;
pendingAsk: { turnId: string; toolCallId: string } | null;
busy: boolean;
}
type Settled =
| { kind: "completed"; text: string | null }
| { kind: "failed"; error: string }
| { kind: "cancelled" }
| { kind: "ask_human"; toolCallId: string; query: string; options?: string[] }
| { kind: "suspended" }
| { kind: "timeout" };
function settleOf(event: TurnStreamEvent): Settled | null {
switch (event.type) {
case "turn_completed":
return { kind: "completed", text: assistantText(event.output) };
case "turn_failed":
return { kind: "failed", error: event.error };
case "turn_cancelled":
return { kind: "cancelled" };
case "turn_suspended": {
const ask = event.pendingAsyncTools.find(
(t) => t.toolId === ASK_HUMAN_TOOL_ID || t.toolName === ASK_HUMAN_TOOL,
);
if (ask) {
const input = ask.input as { question?: unknown; options?: unknown } | null;
const query =
typeof input?.question === "string" && input.question
? input.question
: "The agent needs your input.";
const options = Array.isArray(input?.options)
? input.options.filter((o): o is string => typeof o === "string")
: undefined;
return { kind: "ask_human", toolCallId: ask.toolCallId, query, options };
}
// Other async tools settle on their own and the turn resumes;
// keep waiting. Pending permissions need the desktop.
if (event.pendingAsyncTools.length === 0 && event.pendingPermissions.length > 0) {
return { kind: "suspended" };
}
return null;
}
default:
return null;
}
}
function relativeTime(iso: string): string {
const then = Date.parse(iso);
if (!Number.isFinite(then)) return "";
const diffSec = Math.round((Date.now() - then) / 1000);
if (diffSec < 60) return "just now";
const diffMin = Math.round(diffSec / 60);
if (diffMin < 60) return `${diffMin}m ago`;
const diffHr = Math.round(diffMin / 60);
if (diffHr < 24) return `${diffHr}h ago`;
return `${Math.round(diffHr / 24)}d ago`;
}
function chunkReply(text: string): string[] {
if (text.length <= REPLY_CHUNK_SIZE) return [text];
const parts: string[] = [];
let rest = text;
while (rest.length > 0 && parts.length < MAX_REPLY_CHUNKS) {
parts.push(rest.slice(0, REPLY_CHUNK_SIZE));
rest = rest.slice(REPLY_CHUNK_SIZE);
}
if (rest.length > 0) {
parts[parts.length - 1] += "\n… (truncated — open Rowboat for the full reply)";
}
return parts;
}
interface TurnWatcher {
waitFor(turnId: string, timeoutMs: number): Promise<Settled>;
dispose(): void;
}
export class ChannelBridge {
private senders = new Map<string, SenderState>();
constructor(
private readonly deps: {
sessions: ISessions;
sessionBus: EmitterSessionBus;
listModels: () => Promise<ModelChoice[]>;
},
) {}
async handleInbound(senderKey: string, text: string, reply: ReplyFn): Promise<void> {
const trimmed = text.trim();
if (!trimmed) return;
const state = this.senderState(senderKey);
const lower = trimmed.toLowerCase();
try {
if (lower === "help" || lower === "?") {
await reply(HELP_TEXT);
return;
}
if (lower === "list" || lower === "chats") {
await reply(this.renderList(state));
return;
}
const resume = /^(?:resume|open)\s+(\d+)$/.exec(lower);
if (resume) {
await reply(this.resumeSession(state, Number(resume[1])));
return;
}
if (lower === "model" || lower === "models") {
await reply(await this.renderModelList(state));
return;
}
const model = /^model\s+(.+)$/i.exec(trimmed);
if (model) {
await reply(await this.selectModel(state, model[1].trim()));
return;
}
if (lower === "status") {
await reply(this.renderStatus(state));
return;
}
if (lower === "stop") {
await reply(await this.stopActive(state));
return;
}
if (lower === "new") {
state.activeSessionId = null;
state.pendingAsk = null;
await reply("🆕 Fresh chat — send your first message.");
return;
}
const newWithText = /^new\s+([\s\S]+)$/i.exec(trimmed);
if (newWithText) {
state.activeSessionId = null;
state.pendingAsk = null;
await this.runMessage(state, newWithText[1].trim(), reply);
return;
}
await this.runMessage(state, trimmed, reply);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
await reply(`${message}`).catch(() => undefined);
}
}
private senderState(senderKey: string): SenderState {
let state = this.senders.get(senderKey);
if (!state) {
state = {
activeSessionId: null,
lastList: [],
lastModels: [],
model: null,
pendingAsk: null,
busy: false,
};
this.senders.set(senderKey, state);
}
return state;
}
private isCurrentModel(state: SenderState, choice: ModelChoice): boolean {
return (
state.model?.provider === choice.provider && state.model?.model === choice.model
);
}
private async renderModelList(state: SenderState): Promise<string> {
const choices = await this.deps.listModels();
if (choices.length === 0) {
return "No models available — configure one in Rowboat → Settings → Models.";
}
state.lastModels = choices;
const shown = choices.slice(0, MODEL_LIST_LIMIT);
const lines = shown.map((c, i) => {
const current = this.isCurrentModel(state, c) ? " ← current" : "";
return `${i + 1}. ${c.label}${current}`;
});
if (choices.length > shown.length) {
lines.push(`… and ${choices.length - shown.length} more — pick by name.`);
}
return [
`Models${state.model ? "" : " (using app default)"}:`,
...lines,
"",
`Reply "model N" or "model <name>" to switch, "model default" to reset.`,
].join("\n");
}
private async selectModel(state: SenderState, arg: string): Promise<string> {
const lower = arg.toLowerCase();
if (lower === "default" || lower === "reset") {
state.model = null;
return "✅ Using the app default model.";
}
if (state.lastModels.length === 0) {
state.lastModels = await this.deps.listModels();
}
let choice: ModelChoice | undefined;
if (/^\d+$/.test(lower)) {
choice = state.lastModels[Number(lower) - 1];
if (!choice) {
return `No model #${lower} — send "model" to see the list.`;
}
} else {
const matches = state.lastModels.filter(
(c) =>
c.label.toLowerCase().includes(lower) ||
c.model.toLowerCase().includes(lower),
);
if (matches.length === 0) {
return `No model matching "${arg}" — send "model" to see the list.`;
}
if (matches.length > 1) {
const preview = matches.slice(0, 5).map((c) => `${c.label}`);
return [`"${arg}" matches ${matches.length} models:`, ...preview, "", "Be more specific."].join("\n");
}
choice = matches[0];
}
state.model = { provider: choice.provider, model: choice.model };
return `✅ Model set to ${choice.label} for your chats from here.`;
}
private sessionEntry(sessionId: string): SessionIndexEntry | undefined {
return this.deps.sessions.listSessions().find((e) => e.sessionId === sessionId);
}
private recentSessions(): SessionIndexEntry[] {
return this.deps.sessions
.listSessions()
.filter((e) => !e.error)
.sort((a, b) => b.updatedAt.localeCompare(a.updatedAt))
.slice(0, LIST_LIMIT);
}
private renderList(state: SenderState): string {
const entries = this.recentSessions();
if (entries.length === 0) {
return "No chats yet — just send a message to start one.";
}
state.lastList = entries.map((e) => e.sessionId);
const lines = entries.map((e, i) => {
const marker =
e.latestTurnStatus === "suspended" ? " ⚠️" :
e.latestTurnStatus === "idle" ? " ⏳" : "";
const active = e.sessionId === state.activeSessionId ? " ← current" : "";
return `${i + 1}. ${e.title ?? "Untitled"}${marker} (${relativeTime(e.updatedAt)})${active}`;
});
return [
"Recent chats:",
...lines,
"",
`Reply "resume N" to continue one.`,
].join("\n");
}
private resumeSession(state: SenderState, index: number): string {
if (state.lastList.length === 0) {
state.lastList = this.recentSessions().map((e) => e.sessionId);
}
const sessionId = state.lastList[index - 1];
if (!sessionId) {
return `No chat #${index} — send "list" to see recent chats.`;
}
state.activeSessionId = sessionId;
state.pendingAsk = null;
const entry = this.sessionEntry(sessionId);
return `▶️ Resumed "${entry?.title ?? "Untitled"}" — send a message to continue.`;
}
private renderStatus(state: SenderState): string {
if (!state.activeSessionId) {
return "No current chat — your next message starts a new one.";
}
const entry = this.sessionEntry(state.activeSessionId);
if (!entry) return "Current chat no longer exists — send a message to start fresh.";
const status = state.busy
? "working"
: entry.latestTurnStatus === "suspended"
? "waiting on input"
: entry.latestTurnStatus;
return `Current chat: "${entry.title ?? "Untitled"}" — ${status}.`;
}
private async stopActive(state: SenderState): Promise<string> {
state.pendingAsk = null;
if (!state.activeSessionId) return "Nothing to stop.";
const entry = this.sessionEntry(state.activeSessionId);
if (!entry?.latestTurnId) return "Nothing to stop.";
if (
entry.latestTurnStatus === "completed" ||
entry.latestTurnStatus === "failed" ||
entry.latestTurnStatus === "cancelled"
) {
return "Nothing running in the current chat.";
}
await this.deps.sessions.stopTurn(entry.latestTurnId, "stopped from mobile channel");
return "🛑 Stop requested.";
}
private async runMessage(state: SenderState, text: string, reply: ReplyFn): Promise<void> {
if (state.busy) {
await reply('⏳ Still working on the previous message — send "stop" to cancel it.');
return;
}
state.busy = true;
try {
await reply("⏳ Working on it…");
if (state.pendingAsk) {
const ask = state.pendingAsk;
state.pendingAsk = null;
const answered = await this.answerAsk(state, ask, text, reply);
if (answered) return;
// The ask was already resolved elsewhere (e.g. answered in the
// desktop UI) or the turn is terminal — treat the text as a
// normal message instead of discarding it.
}
await this.sendToSession(state, text, reply);
} catch (error) {
if (error instanceof TurnNotSettledError) {
await reply(
'⏳ That chat is still working on something — send "stop" to cancel it, or "new" to start a fresh chat.',
);
return;
}
throw error;
} finally {
state.busy = false;
}
}
private async sendToSession(state: SenderState, text: string, reply: ReplyFn): Promise<void> {
// Subscribe before advancing so no settle event can slip past.
const watcher = this.watchBus();
try {
if (!state.activeSessionId) {
state.activeSessionId = await this.deps.sessions.createSession();
}
const sent = await this.deps.sessions.sendMessage(
state.activeSessionId,
{ role: "user", content: text },
{
agent: {
agentId: AGENT_ID,
...(state.model ? { overrides: { model: state.model } } : {}),
},
autoPermission: true,
},
);
const settled = await watcher.waitFor(sent.turnId, TURN_TIMEOUT_MS);
await this.deliverSettled(state, sent.turnId, settled, reply);
} finally {
watcher.dispose();
}
}
// Returns false when the ask was stale (already answered on the desktop /
// turn terminal) — the caller then routes the text as a normal message.
private async answerAsk(
state: SenderState,
ask: { turnId: string; toolCallId: string },
text: string,
reply: ReplyFn,
): Promise<boolean> {
const watcher = this.watchBus();
try {
const settledPromise = watcher.waitFor(ask.turnId, TURN_TIMEOUT_MS);
// respondToAskHuman resolves only when the whole advance settles,
// so it must not be awaited ahead of the watcher (that would
// bypass TURN_TIMEOUT_MS). Race instead: its rejection (stale
// ask) must beat the 30-minute timeout; its success defers to the
// settle event.
const settled = await Promise.race([
settledPromise,
this.deps.sessions
.respondToAskHuman(ask.turnId, ask.toolCallId, text)
.then(() => settledPromise),
]);
await this.deliverSettled(state, ask.turnId, settled, reply);
return true;
} catch (error) {
if (error instanceof TurnInputError) return false;
throw error;
} finally {
watcher.dispose();
}
}
private async deliverSettled(
state: SenderState,
turnId: string,
settled: Settled,
reply: ReplyFn,
): Promise<void> {
switch (settled.kind) {
case "completed": {
let text = settled.text;
if (!text) {
// Rare: final message had no text parts; recover the last
// assistant text from the persisted turn.
try {
const turn = await this.deps.sessions.getTurn(turnId);
text = lastAssistantText(reduceTurn(turn.events));
} catch {
text = null;
}
}
for (const chunk of chunkReply(text ?? "✅ Done (no text reply).")) {
await reply(chunk);
}
return;
}
case "failed":
await reply(`❌ Task failed: ${settled.error}`);
return;
case "cancelled":
await reply("🛑 Stopped.");
return;
case "ask_human": {
state.pendingAsk = { turnId, toolCallId: settled.toolCallId };
const lines = [`${settled.query}`];
if (settled.options?.length) {
lines.push(...settled.options.map((o, i) => `${i + 1}. ${o}`));
}
lines.push("", "Reply with your answer.");
await reply(lines.join("\n"));
return;
}
case "suspended":
await reply(
"⚠️ The agent is waiting for a permission approval — open Rowboat on your desktop to continue.",
);
return;
case "timeout":
await reply(
"⏱️ Still running after 30 minutes — check the desktop app for progress.",
);
return;
}
}
// Buffers settle-relevant events (≈1 per turn) from the moment of
// subscription so a settle firing between advance-start and waitFor() is
// never lost — without retaining the per-token delta stream of every
// concurrent session. One watcher per in-flight message.
private watchBus(): TurnWatcher {
const buffered: Array<{ turnId: string; settled: Settled }> = [];
let waiter: { turnId: string; resolve: (settled: Settled) => void } | null = null;
let cancelTimer: (() => void) | null = null;
const unsubscribe = this.deps.sessionBus.subscribe((event) => {
if (event.kind !== "turn-event") return;
const settled = settleOf(event.event);
if (!settled) return;
if (waiter) {
if (event.turnId === waiter.turnId) waiter.resolve(settled);
return;
}
buffered.push({ turnId: event.turnId, settled });
});
return {
waitFor: (turnId: string, timeoutMs: number): Promise<Settled> =>
new Promise<Settled>((resolve) => {
const hit = buffered.find((b) => b.turnId === turnId);
if (hit) {
resolve(hit.settled);
return;
}
buffered.length = 0;
const timer = setTimeout(() => resolve({ kind: "timeout" }), timeoutMs);
cancelTimer = () => clearTimeout(timer);
waiter = {
turnId,
resolve: (settled) => {
clearTimeout(timer);
resolve(settled);
},
};
}),
dispose: () => {
unsubscribe();
cancelTimer?.();
},
};
}
}

View file

@ -0,0 +1,40 @@
import fs from 'node:fs/promises';
import path from 'node:path';
import type { z } from 'zod';
import { WorkDir } from '../config/config.js';
import { ChannelsConfig, DEFAULT_CHANNELS_CONFIG } from '@x/shared/dist/channels.js';
export interface IChannelsConfigRepo {
getConfig(): Promise<z.infer<typeof ChannelsConfig>>;
setConfig(config: z.infer<typeof ChannelsConfig>): Promise<void>;
}
export class FSChannelsConfigRepo implements IChannelsConfigRepo {
private readonly configPath = path.join(WorkDir, 'config', 'channels.json');
constructor() {
this.ensureConfigFile();
}
private async ensureConfigFile(): Promise<void> {
try {
await fs.access(this.configPath);
} catch {
await fs.writeFile(this.configPath, JSON.stringify(DEFAULT_CHANNELS_CONFIG, null, 2));
}
}
async getConfig(): Promise<z.infer<typeof ChannelsConfig>> {
try {
const content = await fs.readFile(this.configPath, 'utf8');
return ChannelsConfig.parse(JSON.parse(content));
} catch {
return DEFAULT_CHANNELS_CONFIG;
}
}
async setConfig(config: z.infer<typeof ChannelsConfig>): Promise<void> {
const validated = ChannelsConfig.parse(config);
await fs.writeFile(this.configPath, JSON.stringify(validated, null, 2));
}
}

View file

@ -0,0 +1,251 @@
import path from "node:path";
import fs from "node:fs/promises";
import type { z } from "zod";
import type { ChannelsConfig, ChannelsStatus } from "@x/shared/dist/channels.js";
import container from "../di/container.js";
import { WorkDir } from "../config/config.js";
import type { ISessions } from "../sessions/api.js";
import type { EmitterSessionBus } from "../sessions/bus.js";
import { isSignedIn } from "../account/account.js";
import { listGatewayModels } from "../models/gateway.js";
import { listOnboardingModels } from "../models/models-dev.js";
import { ChannelBridge, type ModelChoice } from "./bridge.js";
import type { IChannelsConfigRepo } from "./repo.js";
import { TelegramTransport } from "./transports/telegram.js";
// Type-only: the real module (which pulls the ~9MB baileys dependency) is
// loaded dynamically in startWhatsApp, so boot pays nothing while disabled.
import type { WhatsAppTransport } from "./transports/whatsapp.js";
// Lifecycle owner for the mobile channels: reads config, runs the enabled
// transports against one shared ChannelBridge, and fans status out to the
// renderer (QR pairing, connection state). init() from main after
// sessions.initialize(); applyChannelsConfig() on every settings save.
type Config = z.infer<typeof ChannelsConfig>;
type Status = z.infer<typeof ChannelsStatus>;
const WHATSAPP_AUTH_DIR = path.join(WorkDir, "channels", "whatsapp-auth");
const TELEGRAM_STATE_FILE = path.join(WorkDir, "channels", "telegram-state.json");
let bridge: ChannelBridge | null = null;
let whatsapp: WhatsAppTransport | null = null;
let telegram: TelegramTransport | null = null;
const status: Status = {
whatsapp: { state: "disabled" },
telegram: { state: "disabled" },
};
const statusListeners = new Set<(status: Status) => void>();
// Serializes apply/logout so a fast settings double-save can't interleave
// transport teardown and startup. enqueue() recovers the chain before adding
// a step — a rejected step must fail only its own caller, never poison every
// later settings save.
let lifecycle: Promise<void> = Promise.resolve();
function enqueue(step: () => Promise<void>): Promise<void> {
const run = lifecycle.catch(() => undefined).then(step);
lifecycle = run.catch(() => undefined);
return run;
}
function notifyStatus(): void {
const snapshot = structuredClone(status);
for (const listener of statusListeners) {
try {
listener(snapshot);
} catch {
// observers must never affect the channels
}
}
}
function setWhatsAppStatus(next: Status["whatsapp"]): void {
status.whatsapp = next;
notifyStatus();
}
function setTelegramStatus(next: Status["telegram"]): void {
status.telegram = next;
notifyStatus();
}
export function getChannelsStatus(): Status {
return structuredClone(status);
}
export function subscribeChannelsStatus(listener: (status: Status) => void): () => void {
statusListeners.add(listener);
return () => statusListeners.delete(listener);
}
// Same catalog the desktop model picker uses (models:list IPC).
async function listBridgeModels(): Promise<ModelChoice[]> {
const catalog = (await isSignedIn())
? await listGatewayModels()
: await listOnboardingModels();
return catalog.providers.flatMap((provider) =>
provider.models.map((m) => ({
provider: provider.id,
model: m.id,
label: `${m.name ?? m.id}${provider.name}`,
})),
);
}
function ensureBridge(): ChannelBridge {
if (!bridge) {
bridge = new ChannelBridge({
sessions: container.resolve<ISessions>("sessions"),
sessionBus: container.resolve<EmitterSessionBus>("sessionBus"),
listModels: listBridgeModels,
});
}
return bridge;
}
async function stopWhatsApp(): Promise<void> {
if (!whatsapp) return;
const stopping = whatsapp;
whatsapp = null;
await stopping.stop().catch(() => undefined);
setWhatsAppStatus({ state: "disabled" });
}
function stopTelegram(): void {
if (!telegram) return;
const stopping = telegram;
telegram = null;
stopping.stop();
setTelegramStatus({ state: "disabled" });
}
// Invalidates pending async QR renders whenever a newer status lands.
let qrSeq = 0;
async function startWhatsApp(config: Config["whatsapp"]): Promise<void> {
if (!config.enabled) {
setWhatsAppStatus({ state: "disabled" });
return;
}
const channelBridge = ensureBridge();
const [{ WhatsAppTransport: Transport }, QRCode] = await Promise.all([
import("./transports/whatsapp.js"),
import("qrcode").then((m) => m.default),
]);
const transport = new Transport({
authDir: WHATSAPP_AUTH_DIR,
allowFrom: config.allowFrom,
onInbound: (senderKey, chatJid, text) => {
// Route replies through whichever transport is current at send
// time — the originating instance may have been replaced by a
// settings save while the turn was running.
const reply = async (replyText: string) => {
const current = whatsapp;
if (!current) throw new Error("WhatsApp channel is disabled");
await current.send(chatJid, replyText);
};
void channelBridge.handleInbound(senderKey, text, reply);
},
onStatus: (update) => {
if (whatsapp !== transport) return; // superseded instance
if (update.state === "qr" && update.qr) {
const seq = ++qrSeq;
// Render the pairing QR main-side so the renderer just shows
// an <img>; the raw pairing string never leaves core.
QRCode.toDataURL(update.qr, { margin: 1, width: 256 })
.then((qrDataUrl) => {
if (whatsapp === transport && seq === qrSeq) {
setWhatsAppStatus({ state: "qr", qrDataUrl });
}
})
.catch(() => {
if (whatsapp === transport && seq === qrSeq) {
setWhatsAppStatus({ state: "error", error: "Failed to render pairing QR" });
}
});
return;
}
qrSeq++;
setWhatsAppStatus({
state: update.state,
...(update.self ? { self: update.self } : {}),
...(update.error ? { error: update.error } : {}),
});
},
});
whatsapp = transport;
transport.start().catch((error) => {
if (whatsapp !== transport) return;
setWhatsAppStatus({
state: "error",
error: error instanceof Error ? error.message : String(error),
});
});
}
function startTelegram(config: Config["telegram"]): void {
if (!config.enabled) {
setTelegramStatus({ state: "disabled" });
return;
}
if (!config.botToken) {
setTelegramStatus({ state: "error", error: "Bot token missing — create one with @BotFather" });
return;
}
const channelBridge = ensureBridge();
const transport = new TelegramTransport({
botToken: config.botToken,
allowFrom: config.allowFrom,
stateFile: TELEGRAM_STATE_FILE,
onInbound: (senderKey, chatId, text) => {
const reply = async (replyText: string) => {
const current = telegram;
if (!current) throw new Error("Telegram channel is disabled");
await current.send(chatId, replyText);
};
void channelBridge.handleInbound(senderKey, text, reply);
},
onStatus: (update) => {
if (telegram !== transport) return; // superseded instance
setTelegramStatus(update);
},
});
telegram = transport;
void transport.start();
}
export function applyChannelsConfig(config: Config): Promise<void> {
return enqueue(async () => {
await stopWhatsApp();
stopTelegram();
await startWhatsApp(config.whatsapp);
startTelegram(config.telegram);
});
}
// Unlink the WhatsApp device and, if the channel is still enabled, restart it
// so a fresh pairing QR appears. Telegram is left untouched.
export function logoutWhatsApp(): Promise<void> {
return enqueue(async () => {
if (whatsapp) {
const out = whatsapp;
whatsapp = null;
await out.logout().catch(() => undefined);
} else {
await fs.rm(WHATSAPP_AUTH_DIR, { recursive: true, force: true }).catch(() => undefined);
}
const config = await container
.resolve<IChannelsConfigRepo>("channelsConfigRepo")
.getConfig();
await startWhatsApp(config.whatsapp);
});
}
export async function init(): Promise<void> {
const config = await container
.resolve<IChannelsConfigRepo>("channelsConfigRepo")
.getConfig();
await applyChannelsConfig(config);
}

View file

@ -0,0 +1,218 @@
import fs from "node:fs/promises";
import path from "node:path";
import type { z } from "zod";
import type { TelegramChannelStatus } from "@x/shared/dist/channels.js";
// Telegram Bot API transport. Deliberately dependency-free: the Bot API is
// plain HTTPS — getUpdates long polling (outbound connection, works behind
// NAT) plus sendMessage. The user supplies their own bot token (@BotFather).
//
// The getUpdates offset is persisted to disk after each processed batch:
// Telegram only marks updates confirmed when a LATER getUpdates call passes a
// higher offset, so without persistence every transport restart (app relaunch
// or settings save) would redeliver — and re-execute — the last batch.
const POLL_TIMEOUT_S = 50;
const RETRY_DELAY_MS = 5000;
const MAX_RETRY_DELAY_MS = 60_000;
type Status = z.infer<typeof TelegramChannelStatus>;
class TelegramApiError extends Error {
constructor(
message: string,
readonly code?: number,
) {
super(message);
this.name = "TelegramApiError";
}
}
// 401 = token revoked/invalid, 404 = bot deleted / malformed token. Retrying
// these forever would hammer the API and show a misleading "polling" status.
function isTerminal(error: unknown): boolean {
return error instanceof TelegramApiError && (error.code === 401 || error.code === 404);
}
interface TelegramUpdate {
update_id: number;
message?: {
message_id: number;
text?: string;
chat: { id: number; type: string };
from?: { id: number; is_bot?: boolean };
};
}
export interface TelegramTransportOptions {
botToken: string;
allowFrom: string[];
// JSON file holding { offset } across restarts.
stateFile: string;
// chatId is the address to reply to; the caller owns reply routing.
onInbound: (senderKey: string, chatId: string, text: string) => void;
onStatus: (status: Status) => void;
}
export class TelegramTransport {
private abort: AbortController | null = null;
private stopped = false;
private offset = 0;
private botUsername: string | undefined;
constructor(private readonly opts: TelegramTransportOptions) {}
async start(): Promise<void> {
this.stopped = false;
this.opts.onStatus({ state: "starting" });
void this.run();
}
stop(): void {
this.stopped = true;
this.abort?.abort();
this.opts.onStatus({ state: "disabled" });
}
private async call(method: string, body?: unknown, signal?: AbortSignal): Promise<unknown> {
const res = await fetch(`https://api.telegram.org/bot${this.opts.botToken}/${method}`, {
method: "POST",
headers: { "content-type": "application/json" },
body: body === undefined ? undefined : JSON.stringify(body),
...(signal ? { signal } : {}),
});
const payload = (await res.json()) as {
ok: boolean;
result?: unknown;
description?: string;
error_code?: number;
};
if (!payload.ok) {
throw new TelegramApiError(
payload.description ?? `Telegram API error (${method})`,
payload.error_code,
);
}
return payload.result;
}
private async loadOffset(): Promise<void> {
try {
const raw = await fs.readFile(this.opts.stateFile, "utf8");
const parsed = JSON.parse(raw) as { offset?: unknown };
if (typeof parsed.offset === "number" && Number.isFinite(parsed.offset)) {
this.offset = parsed.offset;
}
} catch {
// first run or unreadable state — start from 0
}
}
private async saveOffset(): Promise<void> {
try {
await fs.mkdir(path.dirname(this.opts.stateFile), { recursive: true });
await fs.writeFile(this.opts.stateFile, JSON.stringify({ offset: this.offset }));
} catch {
// best effort — worst case is one redelivered batch after restart
}
}
private async sleep(ms: number): Promise<void> {
await new Promise((resolve) => setTimeout(resolve, ms));
}
private async run(): Promise<void> {
await this.loadOffset();
// Validate the token, retrying transient failures (the app often
// starts at login before the network is up). Only a definitive
// API rejection is terminal.
let delay = RETRY_DELAY_MS;
while (!this.stopped) {
try {
const me = (await this.call("getMe")) as { username?: string };
if (this.stopped) return;
this.botUsername = me.username;
this.opts.onStatus({ state: "polling", botUsername: me.username });
break;
} catch (error) {
if (this.stopped) return;
const message = error instanceof Error ? error.message : String(error);
if (isTerminal(error)) {
this.opts.onStatus({
state: "error",
error: `Bot token rejected (${message}) — create a new token with @BotFather.`,
});
return;
}
this.opts.onStatus({ state: "error", error: message });
await this.sleep(delay);
delay = Math.min(delay * 2, MAX_RETRY_DELAY_MS);
}
}
delay = RETRY_DELAY_MS;
let healthy = true;
while (!this.stopped) {
this.abort = new AbortController();
try {
const updates = (await this.call(
"getUpdates",
{
timeout: POLL_TIMEOUT_S,
offset: this.offset,
allowed_updates: ["message"],
},
this.abort.signal,
)) as TelegramUpdate[];
for (const update of updates) {
this.offset = update.update_id + 1;
this.handleUpdate(update);
}
if (updates.length > 0) {
await this.saveOffset();
}
if (!healthy) {
// Restore the healthy status only after a successful poll.
healthy = true;
this.opts.onStatus({ state: "polling", botUsername: this.botUsername });
}
delay = RETRY_DELAY_MS;
} catch (error) {
if (this.stopped) return;
const message = error instanceof Error ? error.message : String(error);
if (isTerminal(error)) {
this.opts.onStatus({
state: "error",
error: `Bot token rejected (${message}) — create a new token with @BotFather.`,
});
return;
}
healthy = false;
this.opts.onStatus({ state: "error", error: message });
await this.sleep(delay);
delay = Math.min(delay * 2, MAX_RETRY_DELAY_MS);
}
}
}
private handleUpdate(update: TelegramUpdate): void {
const message = update.message;
if (!message?.text || message.from?.is_bot) return;
// DMs only: group chats would let any member drive the bridge.
if (message.chat.type !== "private") return;
const chatId = String(message.chat.id);
if (!this.opts.allowFrom.includes(chatId)) {
void this.send(
chatId,
`⛔ Not authorized. Your chat ID is ${chatId} — add it under Rowboat → Settings → Mobile to pair this chat.`,
).catch(() => undefined);
return;
}
this.opts.onInbound(`telegram:${chatId}`, chatId, message.text);
}
async send(chatId: string, text: string): Promise<void> {
await this.call("sendMessage", { chat_id: chatId, text });
}
}

View file

@ -0,0 +1,219 @@
import fs from "node:fs/promises";
import makeWASocket, {
DisconnectReason,
areJidsSameUser,
isJidGroup,
jidDecode,
useMultiFileAuthState,
} from "baileys";
// WhatsApp transport via Baileys: the app links to the user's own WhatsApp
// account as a linked device (QR pairing, same as WhatsApp Web) over an
// outbound WebSocket — no server, no port forwarding.
//
// Access model: the linked account's own self-chat ("message yourself") is
// always allowed; other senders must be explicitly allowlisted by phone
// number. Group chats are ignored entirely.
type WASocket = ReturnType<typeof makeWASocket>;
const RECONNECT_DELAY_MS = 3000;
// Marks bridge-sent messages. In the self-chat our own replies come back on
// messages.upsert like any other message; the marker (plus sent-id tracking)
// keeps the bridge from answering itself in a loop.
const REPLY_MARKER = "🤖 ";
export interface WhatsAppTransportStatus {
state: "starting" | "qr" | "connected" | "error" | "disabled";
qr?: string;
self?: string;
error?: string;
}
export interface WhatsAppTransportOptions {
authDir: string;
allowFrom: string[];
// chatJid is the address to reply to; the caller owns reply routing so a
// reply can go through whichever transport instance is current by then.
onInbound: (senderKey: string, chatJid: string, text: string) => void;
onStatus: (status: WhatsAppTransportStatus) => void;
}
interface TextishMessage {
conversation?: unknown;
extendedTextMessage?: { text?: unknown };
ephemeralMessage?: { message?: TextishMessage };
}
interface InboundWAMessage {
key?: {
remoteJid?: string | null;
// Phone-number JID when remoteJid is a LID (anonymized) JID.
remoteJidAlt?: string | null;
fromMe?: boolean | null;
id?: string | null;
};
message?: unknown;
}
function messageText(message: unknown): string | null {
if (!message || typeof message !== "object") return null;
const m = message as TextishMessage;
const unwrapped = m.ephemeralMessage?.message ?? m;
const text: unknown = unwrapped.conversation ?? unwrapped.extendedTextMessage?.text;
return typeof text === "string" && text ? text : null;
}
export class WhatsAppTransport {
private sock: WASocket | null = null;
private stopped = false;
// Bumped on every connect/stop/logout; handlers close over their own
// generation and go inert the moment they are superseded, so a stop()
// racing an await inside connect() cannot leave a zombie socket
// processing messages alongside its replacement.
private generation = 0;
private sentIds = new Set<string>();
constructor(private readonly opts: WhatsAppTransportOptions) {}
async start(): Promise<void> {
this.stopped = false;
this.opts.onStatus({ state: "starting" });
await this.connect();
}
async stop(): Promise<void> {
this.stopped = true;
this.generation++;
try {
this.sock?.end(undefined);
} catch {
// already closed
}
this.sock = null;
this.opts.onStatus({ state: "disabled" });
}
// Unlink this device: invalidates the pairing on the phone and clears
// local credentials so the next start shows a fresh QR.
async logout(): Promise<void> {
this.stopped = true;
this.generation++;
try {
await this.sock?.logout();
} catch {
// best effort — clearing creds below is what actually unpairs us
}
this.sock = null;
await fs.rm(this.opts.authDir, { recursive: true, force: true });
this.opts.onStatus({ state: "disabled" });
}
private async connect(): Promise<void> {
if (this.stopped) return;
const generation = ++this.generation;
const { state, saveCreds } = await useMultiFileAuthState(this.opts.authDir);
if (this.stopped || generation !== this.generation) return;
const sock = makeWASocket({
auth: state,
syncFullHistory: false,
markOnlineOnConnect: false,
});
this.sock = sock;
const isCurrent = () =>
!this.stopped && generation === this.generation && this.sock === sock;
sock.ev.on("creds.update", saveCreds);
sock.ev.on("connection.update", (update) => {
if (!isCurrent()) return;
if (update.qr) {
this.opts.onStatus({ state: "qr", qr: update.qr });
}
if (update.connection === "open") {
const self = jidDecode(sock.user?.id ?? "")?.user;
this.opts.onStatus({ state: "connected", ...(self ? { self } : {}) });
}
if (update.connection === "close") {
const statusCode = (update.lastDisconnect?.error as { output?: { statusCode?: number } } | undefined)
?.output?.statusCode;
if (statusCode === DisconnectReason.loggedOut) {
// Unlinked from the phone; stale creds would loop forever.
void fs.rm(this.opts.authDir, { recursive: true, force: true });
this.opts.onStatus({
state: "error",
error: "Logged out from the phone — toggle WhatsApp off and on to pair again.",
});
return;
}
setTimeout(() => {
if (!isCurrent()) return;
this.connect().catch((error) => {
this.opts.onStatus({
state: "error",
error: error instanceof Error ? error.message : String(error),
});
});
}, RECONNECT_DELAY_MS);
}
});
sock.ev.on("messages.upsert", ({ messages, type }) => {
if (!isCurrent() || type !== "notify") return;
for (const msg of messages) {
this.handleMessage(sock, msg);
}
});
}
private handleMessage(sock: WASocket, msg: InboundWAMessage): void {
const jid: string | undefined = msg.key?.remoteJid ?? undefined;
if (!jid || isJidGroup(jid) || jid === "status@broadcast") return;
const messageId: string | undefined = msg.key?.id ?? undefined;
if (messageId && this.sentIds.has(messageId)) return;
const text = messageText(msg.message);
if (!text || text.startsWith(REPLY_MARKER)) return;
// LID-addressed chats put the anonymized id in remoteJid and (when
// the server supplies it) the real phone-number JID in remoteJidAlt.
// Identity checks must consider both.
const altJid: string | undefined = msg.key?.remoteJidAlt ?? undefined;
const chatJids = altJid ? [jid, altJid] : [jid];
const user = sock.user as { id?: string; lid?: string } | undefined;
const selfIds = [user?.id, user?.lid].filter((v): v is string => Boolean(v));
const isSelfChat = chatJids.some((j) =>
selfIds.some((selfId) => areJidsSameUser(j, selfId)),
);
const senderNumbers = chatJids.flatMap((j) => {
const decoded = jidDecode(j)?.user;
return decoded ? [decoded] : [];
});
// Self-chat is the owner by definition. Anyone else must be
// allowlisted — this bridge is remote control over the desktop agent.
if (!isSelfChat) {
if (msg.key?.fromMe) return;
if (!senderNumbers.some((n) => this.opts.allowFrom.includes(n))) return;
}
// Prefer the phone number (altJid decodes to it when present) as the
// stable sender identity.
const senderId = altJid
? (jidDecode(altJid)?.user ?? senderNumbers[0] ?? jid)
: (senderNumbers[0] ?? jid);
this.opts.onInbound(`whatsapp:${senderId}`, jid, text);
}
async send(jid: string, text: string): Promise<void> {
const sock = this.sock;
if (!sock) throw new Error("WhatsApp is not connected");
const sent = await sock.sendMessage(jid, { text: `${REPLY_MARKER}${text}` });
const id = sent?.key?.id;
if (id) {
this.sentIds.add(id);
if (this.sentIds.size > 500) {
this.sentIds = new Set(Array.from(this.sentIds).slice(-250));
}
}
}
}

View file

@ -139,6 +139,8 @@ function toEvent(update: SessionUpdate): CodeRunEvent {
priority: e.priority ?? undefined,
})),
};
case 'usage_update':
return { type: 'usage', used: update.used, size: update.size };
default:
return { type: 'other', sessionUpdate: update.sessionUpdate };
}

View file

@ -44,6 +44,17 @@ function ensureDefaultConfigs() {
configured: false
}, null, 2));
}
// Create gmail_sync.json with the default onboarding email count if it
// doesn't exist, so the "how many emails to backfill" setting is
// discoverable and editable. Keep the default in sync with
// DEFAULT_MAX_EMAILS in gmail_sync_config.ts.
const gmailSyncConfig = path.join(WorkDir, "config", "gmail_sync.json");
if (!fs.existsSync(gmailSyncConfig)) {
fs.writeFileSync(gmailSyncConfig, JSON.stringify({
maxEmails: 500
}, null, 2));
}
}
ensureDirs();

Some files were not shown because too many files have changed in this diff Show more