apunkt/rowboat
1
0
Fork 0
mirror of https://github.com/rowboatlabs/rowboat.git synced 2026-07-12 21:02:17 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'dev' into feat/out-of-credits-warning

Browse source
This commit is contained in:
PRAKHAR PANDEY 2026-07-06 13:54:20 +05:30 • committed by GitHub
commit c75d55ed31
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
224 changed files with 32863 additions and 2702 deletions
Download patch file Download diff file Expand all files Collapse all files

1
apps/x/.gitignore vendored
View file

@ -1,2 +1,3 @@
node_modules/
test-fixtures/
*.tsbuildinfo

BIN
apps/x/.pnpm-store/v11/index.db Normal file
View file

Binary file not shown.

2
apps/x/ANALYTICS.md
View file

@ -92,6 +92,8 @@ All in `apps/renderer/src/lib/analytics.ts`:
- `chat_message_sent` — `{ voice_input, voice_output, search_enabled }`
- `oauth_connected` / `oauth_disconnected` — `{ provider }`
- `voice_input_started` — no properties
- `call_started` — `{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`)
- `call_turn_latency` — `{ endpoint_to_submit_ms, submit_to_speak_ms, speak_to_audio_ms, total_ms }` — voice-to-voice latency breakdown for one call turn (utterance accepted → submitted → first TTS speak → audio playing)
- `search_executed` — `{ types: string[] }`
- `note_exported` — `{ format }`

253
apps/x/VIDEO_MODE.md Normal file
View file

@ -0,0 +1,253 @@
# Calls (Video Mode) — Deep Dive
Calls let the user talk to the assistant hands-free while it *sees* them
(webcam) and their screen (screen share). There is ONE call engine —
continuous listening, auto-submitted utterances, forced read-aloud TTS, frame
capture — entered through four presets that differ only in starting devices.
This doc covers the product flow, the technical pipeline, and the LLM prompt
surface with exact pointers.
## Product flow
The composer has a **call split-button** (`chat-input-with-mentions.tsx`).
The main click is the "work together" default — preset `share`: screen
sharing ON, camera OFF, floating pill, so the user keeps working while the
assistant watches along (the button tooltip discloses the screen share). The
chevron menu holds the deviations. While a call is live the button turns red
and ends it.
| Preset | Starting devices | First surface |
|--------|------------------|---------------|
| `share` — main click | screen on, camera off | floating pill |
| `voice` — "Voice call" | camera off, screen off | floating mascot pill |
| `video` — "Video call" | camera on | full-screen call |
| `practice` — "Practice session" | camera on, + coaching persona | full-screen call |
**One surface rule** (`callSurface` in `App.tsx`): full screen and screen
sharing are mutually exclusive in both directions — a full-screen call covers
the screen, so sharing it would show the call itself.
- sharing → floating popout, always (pill = working)
- not sharing → full screen unless `callMinimized` (full screen = facing
each other)
- expanding the pill auto-STOPS any share; minimizing the full-screen call
auto-STARTS one (the pill exists to work together) — presenting from full
screen likewise collapses to the pill
- the camera toggle never changes the surface: turning it on from the pill
puts your video IN the pill; expanding is its own explicit action
**Screen-share consent** is three-layered: a toast the moment any share
starts ("Your screen is being shared… [Stop sharing]"), a persistent
"Sharing screen" badge on the pill, and macOS's purple recording indicator.
If the auto-share fails (Screen Recording permission not granted) the call
starts anyway as a voice call, with a toast linking to System Settings.
Practice/coaching is always an explicit choice — expanding to full screen
never turns the coach on.
In-call controls (identical bar on both surfaces): mic mute, camera toggle
(silhouette avatar while off, no webcam frames captured), screen share
toggle, mascot ⇄ "R" letter avatar, end call. **Mute is a full input
pause**, not just audio — mic audio stops reaching Deepgram
(`useVoiceMode.setPaused`, OR'd with the automatic thinking/speaking pause)
AND camera/screen frame capture stops (`useVideoMode.setCapturePaused`;
`collectFrames()` returns nothing while muted, so typed messages carry no
frames either), letting the user talk to someone in the room without the
assistant listening in. Devices stay acquired for instant unmute (camera
light and macOS share indicator stay on — the pill's share badge switches to
"Sharing paused"), the status chip shows "Muted" instead of "Listening",
and assistant output is unaffected (in-flight speech keeps playing; Stop
handles that). Mute resets to off at call start/end. While the assistant is thinking or speaking, a
red **Stop** button appears on the mascot tile — it silences TTS instantly,
skips queued voice segments, and aborts the run if it's still generating
(stopping a run from anywhere, including the composer, also silences TTS). Captions of the in-progress utterance and the
assistant's spoken line run along the bottom. Typing in the composer still
works mid-call; frames ride along with typed messages too.
Outside calls the composer keeps exactly one voice affordance: the **mic
button** (push-to-talk dictation, untouched). Spoken responses exist only
inside calls (forced full read-aloud, off on hang-up). The old video
dropdown, talking-head toggle, read-aloud headphones toggle, and summary/full
TTS dropdown are all retired — a per-message "read aloud" action on assistant
messages is the planned replacement for text-in/voice-out.
The call button is disabled unless both voice input (Deepgram) and voice
output (TTS) are configured. `call_started` (with `preset`) is captured in
PostHog — the adoption metric for this feature.
**Popout mechanics**: a small always-on-top frameless window (camera tile
when on + mascot tile, live caption, control bar) floating over every app —
including Rowboat. Control-bar actions round-trip `video:popoutAction` →
main → `video:popout-action` → app window, which owns the mic/camera/capture;
`expand` also refocuses the app window (handled in main).
## Frame pipeline
`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source
(stream → offscreen `<video>` → canvas JPEG → ring buffer):
- Cadence: 1 fps (`CAPTURE_INTERVAL_MS`, line 20); ring buffer ~2 min.
- Webcam: 512px wide, JPEG q0.65, max **12 frames/message** (lines 21, 31).
- Screen: 1280px wide (text legibility), JPEG q0.7, max **4 frames/message**
(lines 24, 32).
- `collectFrames()` drains frames buffered since the last send, evenly
sampled down to the caps, always keeping the newest; grabs one final frame
at the moment of send. Falls back to the single latest frame for
rapid-fire messages.
`App.tsx` `handlePromptSubmit` attaches the drained frames (whenever a call
is live) to the outgoing message as `UserImagePart`s and sets
`composition.videoMode` when the camera or screen is active, plus
`composition.coachMode` during a practice session. Frames also become
`isVideoFrame` display attachments (filmstrip in the transcript —
`chat-message-attachments.tsx`; history hydration in
`lib/run-to-conversation.ts`).
## Message schema & model encoding
- `packages/shared/src/message.ts:51` — `UserImagePart`: inline base64
(`data`, `mediaType`), `source: 'camera' | 'screen'`, `capturedAt`. Unlike
file attachments (path references read via the `LLMParse` tool), image
parts go to the model as real multimodal image parts.
- `packages/core/src/agents/runtime.ts` `convertFromMessages` (~line 1013):
emits a context line (frame counts + time span), then labeled groups —
a `"Webcam frames (oldest to newest):"` text part before camera images and
a `"Screen-share frames (oldest to newest):"` text part before screen
images — so the model never confuses the user with their screen.
- Frames stay inline in history (no pruning) deliberately: pruning would
bust provider prefix caching every turn and cost more than it saves.
- The auto-permission classifier stringifies + truncates content to ~3KB per
message, so inline base64 can't blow up its prompt.
## Hands-free voice loop
`apps/renderer/src/hooks/useVoiceMode.ts`:
- `startContinuous(onUtterance)` (line 404): push-to-talk params but with
`endpointing=1800` (line 25) so thinking pauses don't cut the user off,
plus `utterance_end_ms=2000` (line 38) as a second end-of-speech signal.
**Gotcha:** Deepgram's `speech_final` usually arrives on a result with an
EMPTY transcript — empty finals must reach the endpoint check or
utterances never complete (see the NOTE in `ws.onmessage`).
- `setPaused(true)` (line 414) while the assistant thinks/speaks: drops mic
audio (so TTS is never transcribed back), discards half-heard buffer,
sends Deepgram KeepAlives every 5s. `App.tsx` drives this from
`activeIsProcessing || tts.state !== 'idle'`.
- Mid-call socket drops reconnect after 1s; the offline audio backlog is
capped (~30s).
Call lifecycle lives in `App.tsx` `startCall(preset)` / `endCall()`:
entering a call saves/forces TTS settings, cancels any push-to-talk
recording, and starts the continuous loop; ending restores everything.
Push-to-talk is disabled while a call owns the mic.
## Popout window
- The popout window keeps the Dock icon alive: it uses
`setVisibleOnAllWorkspaces(true)` WITHOUT `visibleOnFullScreen` — that flag
turns the app into a macOS "agent" app and hides its Dock icon while the
window exists (looks like Rowboat vanished). Trade-off: the popout doesn't
hover over other apps' fullscreen Spaces.
- Shown iff the derived `callSurface === 'popout'` (effect in `App.tsx`).
Renderer asks `video:setPopout {show}`; main creates a frameless,
`alwaysOnTop` ('floating'), all-workspaces BrowserWindow at the top-right
of the primary display, loading the renderer bundle with `#video-popout`
(`apps/renderer/src/main.tsx` branches on the hash →
`components/video-popout.tsx`).
- Call state streams over the `video:popout-state` push channel; main caches
the last payload and replays it on popout load. Shown with
`showInactive()` so it never steals focus.
- The popout captures its **own** camera preview (MediaStreams can't cross
windows) and synthesizes the mascot mouth level (no audio in that window).
- `video:popoutAction` relays control-bar actions to the app window, matched
only by real app-window URLs — `getAllWindows()` also contains hidden
utility windows (PDF export) that must not be shown or messaged.
## Permissions
- Camera: `voice:ensureCameraAccess` settles the macOS TCC prompt before
`getUserMedia` (same pattern as the mic). `NSCameraUsageDescription` is in
`forge.config.cjs` `extendInfo`.
- Screen: `getDisplayMedia` is auto-approved with the primary screen by
`setDisplayMediaRequestHandler` in `main.ts` (no picker);
`meeting:checkScreenPermission` registers the app in macOS Screen
Recording settings on first use.
## LLM prompts catalog
| Prompt | Where |
|--------|-------|
| `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) |
| `# Practice Session (Coach Mode)` system section — coaching persona: specific/actionable feedback after each take, one-sentence interjections mid-flow, structured debrief on wrap-up | `composeSystemInstructions`, gated on `coachMode` (directly after the video section) |
| "Driving the app" paragraph in the video-mode section — on calls, prefer app-navigation read-view/open-item (show while telling) over describing or squinting at frames | same `# Video Mode` section; full action docs in the `app-navigation` skill (`application/assistant/skills/app-navigation/skill.ts`) |
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts` (`convertFromMessages`) |
| `videoMode` / `coachMode` composition overrides (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts` (`CompositionOverrides`); set from `App.tsx` `sendConfig` |
Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are
reused untouched — calls set `voiceInput` per utterance and force
`voiceOutput: 'full'`.
## Driving the app on a call
The assistant can drive the Rowboat UI itself via the extended
`app-navigation` builtin ("app driver"): `open-view` (any main view),
`read-view` (returns the emails / background agents / chat-history data the
view renders — and the renderer simultaneously navigates there so the user
watches it happen), and `open-item` (a specific email thread, note,
background agent, or past chat, deep-linked on screen). Data comes from the
same core functions the UI's IPC handlers use (`listImportantThreads` /
`searchThreads`, background-task `listTasks`, the sessions container) — no
OCR of screen frames. The renderer applies results via
`applyAppNavigation` in App.tsx, fed from BOTH event paths: the legacy
`runs:events` ref-poll AND a watcher over the session-chat conversation (the
turn runtime does not emit legacy run events — miss this and navigation
silently no-ops while the tool reports success). Session switches seed the
watcher so replaying history never navigates. During a call, visible
navigations also collapse the full-screen call to the pill and focus the app
window (`app:focusMainWindow`) so the user actually sees the screen change.
Card labels live in `lib/chat-conversation.ts`. The call prompt and the
`app-navigation` skill teach the show-while-telling pattern: read-view →
speak the highlights → open-item when the user picks one.
## Latency
Voice-to-voice latency (user stops talking → assistant audio) is engineered
at four points; the `call_turn_latency` PostHog event measures the real
distribution (utterance → submit → first speak → audio playing):
- **Smart endpointing** (`useVoiceMode.ts`): Deepgram endpoints at 600ms and
the client decides — a transcript ending in terminal punctuation fires
immediately (~600ms after last word); a mid-thought trail holds another
1.2s (resumed speech cancels the hold). Complete sentences turn around
~1.2s faster than the old fixed 1800ms endpoint.
- **Streaming TTS** (`voice:synthesizeStreamStart` → `voice:tts-chunk` →
MediaSource playback in `useVoiceTTS.ts`): the first segment of an idle
queue plays from the first MP3 chunk instead of after the full body
(ElevenLabs `/stream`, flash model). Follow-up segments keep the gapless
full-body prefetch path. Falls back to non-streaming on any failure.
- **Early clause speech** (`turn-view.ts` `applyOverlay`): a still-open
`<voice>` block ≥60 chars emits its last complete clause immediately, so
speech starts while the rest of the sentence generates.
- **Acknowledgment cue** (`lib/call-sounds.ts`): a soft blip the instant an
utterance is accepted — perceived latency matters as much as measured.
## Cost notes
Webcam frames ≈ 250–350 tokens each (≤12/message ≈ 3–4k); screen frames ≈
1.5–2k tokens each (≤4/message ≈ 6–8k). History keeps frames inline, so long
sessions grow but stay prefix-cached. First lever if cost bites: drop to one
screen frame per message unless the screen changed.
## Known limitations
- Turn-taking is strict — no barge-in (would need echo cancellation against
TTS output).
- Frame sampling, not video: motion between frames is invisible (the prompt
tells the model not to claim otherwise).
- Vocal-delivery feedback is limited: Deepgram reduces speech to text, so
"energy" coaching leans on visual cues.
- Screen share always captures the primary display (no window/display
picker yet).
- The full-screen call covers the chat; there's no in-call transcript drawer.
- The "attach camera frames to typed chat without a call" combination (the
old video+chat mode) was cut in the call-model simplification; if analytics
show demand, it should return as an attachment chip, not a mode.

23
apps/x/apps/main/bundle.mjs
View file

@ -11,6 +11,7 @@
import * as esbuild from 'esbuild';
import { readFile } from 'node:fs/promises';
import { execSync } from 'node:child_process';
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
@ -66,6 +67,28 @@ for (const dir of fs.readdirSync(prebuildsDir)) {
const helper = path.join(prebuildsDir, dir, 'spawn-helper');
if (fs.existsSync(helper)) fs.chmodSync(helper, 0o755);
}
// Self-heal: node-pty ships prebuilt binaries only for darwin/win32, so on any
// host whose prebuild is absent (notably Linux) the staged package has no loadable
// pty.node and the app crashes on launch. Compile the native module for the host
// platform+arch if needed and stage it under prebuilds/<platform>-<arch>/, where
// node-pty's loader looks first. Keeps dev and CI working without a manual node-gyp
// step (the CI workflow's explicit build is the fast path; this is the safety net).
const hostTriple = `${process.platform}-${process.arch}`;
const stagedBinary = path.join(prebuildsDir, hostTriple, 'pty.node');
if (!fs.existsSync(stagedBinary)) {
const builtBinary = path.join(ptySrc, 'build', 'Release', 'pty.node');
if (!fs.existsSync(builtBinary)) {
console.log(`node-pty: no prebuilt binary for ${hostTriple}; compiling with node-gyp…`);
execSync('npx node-gyp rebuild', { cwd: ptySrc, stdio: 'inherit' });
}
if (!fs.existsSync(builtBinary)) {
throw new Error(`node-pty: failed to produce a native binary for ${hostTriple}`);
}
fs.mkdirSync(path.dirname(stagedBinary), { recursive: true });
fs.copyFileSync(builtBinary, stagedBinary);
console.log(`✅ node-pty: staged ${hostTriple}/pty.node`);
}
console.log('✅ node-pty staged in .package/node_modules');
// Bundle the vendored agent-slack CLI into a single self-contained script next

189
apps/x/apps/main/forge.config.cjs
View file

@ -5,6 +5,12 @@
const path = require('path');
const pkg = require('./package.json');
// The Arch Linux (pacman) package is meant only for local builds on an Arch host
// with makepkg. It already self-skips elsewhere (maker-pacman checks for makepkg),
// but CI sets ROWBOAT_SKIP_PACMAN=1 to disable it explicitly — GitHub runners are
// Ubuntu and shouldn't attempt to ship an Arch package.
const SKIP_PACMAN = process.env.ROWBOAT_SKIP_PACMAN === '1';
// Stage the ACP coding-adapters (@agentclientprotocol/*-acp) and their full
// production dependency closure into the packaged app.
//
@ -14,53 +20,88 @@ const pkg = require('./package.json');
// strips the workspace node_modules (see `ignore` below). Without this, packaged
// builds throw `Cannot find module '@agentclientprotocol/...'`.
//
// Why we reconstruct a nested tree instead of copying node_modules: pnpm's store is a
// Why we reconstruct the tree instead of copying node_modules: pnpm's store is a
// symlink farm that legitimately holds multiple versions of the same package (e.g.
// @agentclientprotocol/sdk 0.21 for claude vs 0.22 for codex). We rebuild an npm-style
// nested node_modules — dereferencing symlinks and nesting on version conflict — which
// resolves correctly regardless of pnpm layout.
// node_modules — dereferencing symlinks — that resolves correctly regardless of pnpm
// layout. We HOIST every package to the top-level node_modules and only nest a package
// under its requirer on a genuine version conflict. Hoisting (vs. always nesting) keeps
// the tree shallow: without it, transitive chains like codex-acp → open → wsl-utils →
// is-wsl → is-inside-container → is-docker nest 5+ deep and produce ~260-char paths that
// break the Windows Squirrel/nuget maker's MAX_PATH limit. Node resolution stays correct
// because the top-level node_modules is an ancestor of every staged file, so a hoisted
// package resolves for all requirers and a conflicting version shadows it via nesting.
// verifyAcpStaging() below asserts this held for every dependency edge.
//
// What we DON'T bundle: the agents' native engines (claude / codex, ~200 MB each, shipped
// as platform-specific packages). Those are PROVISIONED on demand into
// ~/.rowboat/engines/<agent>/<version>/ and the adapters are pointed at them via
// CLAUDE_CODE_EXECUTABLE / CODEX_PATH (see packages/core/src/code-mode/acp/). Skipping
// them keeps each OS installer ~400 MB smaller while code mode stays fully functional.
// Shared by stageAcpAdapters and verifyAcpStaging so staging and verification use
// identical resolution semantics.
const ACP_ADAPTERS = [
'@agentclientprotocol/claude-agent-acp',
'@agentclientprotocol/codex-acp',
];
// The native engines, shipped as platform packages. Provisioned on demand
// (see header comment), so they're excluded from staging.
const isAcpNativeEngine = (key) =>
/^@anthropic-ai\/claude-agent-sdk-(win32|darwin|linux)/.test(key) || // native claude
/^@openai\/codex-(win32|darwin|linux)/.test(key); // native codex
// Resolve a dependency's real directory by walking node_modules the way Node does,
// looking for the package DIRECTORY. We deliberately do NOT use
// require.resolve(`${key}/package.json`): that throws for packages whose `exports`
// map doesn't expose package.json (e.g. @anthropic-ai/claude-agent-sdk), which would
// silently drop them and their subtrees. realpathSync dereferences pnpm's symlinks.
// Returns null for deps not installed for this OS (platform-optional binaries).
const acpRealDirOf = (key, fromDir) => {
const fs = require('fs');
let dir = fromDir;
for (;;) {
const cand = path.join(dir, 'node_modules', ...key.split('/'));
if (fs.existsSync(path.join(cand, 'package.json'))) return fs.realpathSync(cand);
const parent = path.dirname(dir);
if (parent === dir) return null;
dir = parent;
}
};
function stageAcpAdapters(mainDir, destNodeModules) {
const fs = require('fs');
const ADAPTERS = [
'@agentclientprotocol/claude-agent-acp',
'@agentclientprotocol/codex-acp',
];
// The native engines, shipped as platform packages. Provisioned on demand instead
// (see comment above), so they're excluded from staging.
const isNativeEngine = (key) =>
/^@anthropic-ai\/claude-agent-sdk-(win32|darwin|linux)/.test(key) || // native claude
/^@openai\/codex-(win32|darwin|linux)/.test(key); // native codex
// Resolve a dependency's real directory by walking node_modules the way Node does,
// looking for the package DIRECTORY. We deliberately do NOT use
// require.resolve(`${key}/package.json`): that throws for packages whose `exports`
// map doesn't expose package.json (e.g. @anthropic-ai/claude-agent-sdk), which would
// silently drop them and their subtrees. realpathSync dereferences pnpm's symlinks.
// Returns null for deps not installed for this OS (platform-optional binaries).
const realDirOf = (key, fromDir) => {
let dir = fromDir;
for (;;) {
const cand = path.join(dir, 'node_modules', ...key.split('/'));
if (fs.existsSync(path.join(cand, 'package.json'))) return fs.realpathSync(cand);
const parent = path.dirname(dir);
if (parent === dir) return null;
dir = parent;
}
};
let copied = 0;
const skippedEngines = new Set();
const install = (srcDir, key, destNM, chain) => {
const destDir = path.join(destNM, ...key.split('/'));
if (fs.existsSync(destDir)) return; // already placed at this exact location
// srcRealDir -> the staged directory whose content represents it. Lets
// verifyAcpStaging map every source package to where it landed.
const placements = new Map();
// package key -> version placed at the TOP-LEVEL node_modules. We hoist every
// package to the top level and only nest a package under its requirer when a
// DIFFERENT version is already hoisted there. See the header comment for why
// (a shallow tree stays under Windows' MAX_PATH).
const rootHoisted = new Map();
const install = (srcDir, key, parentNM, chain) => {
if (chain.has(srcDir)) return; // dependency cycle — resolves to ancestor copy
const pj = JSON.parse(fs.readFileSync(path.join(srcDir, 'package.json'), 'utf8'));
const version = pj.version;
const hoisted = rootHoisted.get(key);
let destNM;
if (hoisted === undefined) {
destNM = destNodeModules; // first sighting → hoist to the top level
rootHoisted.set(key, version);
} else if (hoisted === version) {
// identical version already hoisted at root → reuse it (its subtree is
// already staged); just record where this srcDir resolves to.
placements.set(srcDir, path.join(destNodeModules, ...key.split('/')));
return;
} else {
destNM = parentNM; // genuine version conflict → nest under requirer
}
const destDir = path.join(destNM, ...key.split('/'));
placements.set(srcDir, destDir);
if (fs.existsSync(destDir)) return; // already placed at this exact location
fs.mkdirSync(path.dirname(destDir), { recursive: true });
fs.cpSync(srcDir, destDir, {
recursive: true,
@ -68,18 +109,17 @@ function stageAcpAdapters(mainDir, destNodeModules) {
filter: (s) => path.basename(s) !== 'node_modules', // deps handled by recursion
});
copied++;
const pj = JSON.parse(fs.readFileSync(path.join(srcDir, 'package.json'), 'utf8'));
const deps = { ...pj.dependencies, ...pj.optionalDependencies };
const nextChain = new Set(chain).add(srcDir);
for (const depKey of Object.keys(deps)) {
if (isNativeEngine(depKey)) { skippedEngines.add(depKey); continue; }
const depDir = realDirOf(depKey, srcDir);
if (isAcpNativeEngine(depKey)) { skippedEngines.add(depKey); continue; }
const depDir = acpRealDirOf(depKey, srcDir);
if (depDir) install(depDir, depKey, path.join(destDir, 'node_modules'), nextChain);
}
};
for (const key of ADAPTERS) {
const srcDir = realDirOf(key, mainDir);
for (const key of ACP_ADAPTERS) {
const srcDir = acpRealDirOf(key, mainDir);
if (!srcDir) {
throw new Error(`ACP adapter '${key}' is not installed in ${mainDir} — run pnpm install`);
}
@ -88,7 +128,64 @@ function stageAcpAdapters(mainDir, destNodeModules) {
if (skippedEngines.size) {
console.log(` (skipped native engines — provisioned on demand: ${[...skippedEngines].join(', ')})`);
}
return copied;
return { copied, placements };
}
// Fail the build LOUDLY if hoisting misplaced anything. Re-walk the source dependency
// closure and assert that every (package → dependency) edge resolves, in the STAGED
// tree, to the SAME version it resolves to in the SOURCE pnpm tree. This converts a
// silent runtime "Cannot find module" (or a wrong-version resolution from a botched
// hoist) into an immediate build failure. Expectations are derived from the source
// tree — nothing is hardcoded — so it keeps working as the dependency set changes.
function verifyAcpStaging(mainDir, placements) {
const fs = require('fs');
const versionAt = (dir) =>
JSON.parse(fs.readFileSync(path.join(dir, 'package.json'), 'utf8')).version;
// Resolve `key`'s staged version as seen from `fromStagedDir`, via Node's own
// upward node_modules walk. Reads package.json directly (not require.resolve, whose
// `${key}/package.json` subpath some exports maps block).
const stagedVersionOf = (key, fromStagedDir) => {
let dir = fromStagedDir;
for (;;) {
const cand = path.join(dir, 'node_modules', ...key.split('/'));
if (fs.existsSync(path.join(cand, 'package.json'))) return versionAt(cand);
const parent = path.dirname(dir);
if (parent === dir) return null;
dir = parent;
}
};
const errors = [];
const visited = new Set();
const walk = (srcDir) => {
if (visited.has(srcDir)) return;
visited.add(srcDir);
const pj = JSON.parse(fs.readFileSync(path.join(srcDir, 'package.json'), 'utf8'));
const stagedDir = placements.get(srcDir);
if (!stagedDir) { errors.push(`not staged: ${pj.name}@${pj.version}`); return; }
const deps = { ...pj.dependencies, ...pj.optionalDependencies };
for (const depKey of Object.keys(deps)) {
if (isAcpNativeEngine(depKey)) continue;
const depSrc = acpRealDirOf(depKey, srcDir);
if (!depSrc) continue; // platform-optional / not installed for this OS
const want = versionAt(depSrc);
const got = stagedVersionOf(depKey, stagedDir);
if (got === null) {
errors.push(`${pj.name} → ${depKey}: unresolved in staged tree (expected ${want})`);
} else if (got !== want) {
errors.push(`${pj.name} → ${depKey}: staged resolves ${got}, source resolves ${want}`);
}
walk(depSrc);
}
};
for (const key of ACP_ADAPTERS) {
const srcDir = acpRealDirOf(key, mainDir);
if (srcDir) walk(srcDir);
}
if (errors.length) {
throw new Error(
`ACP staging verification failed — the staged tree resolves differently than source:\n - ${errors.join('\n - ')}`
);
}
}
module.exports = {
@ -102,6 +199,7 @@ module.exports = {
],
extendInfo: {
NSAudioCaptureUsageDescription: 'Rowboat needs access to system audio to transcribe meetings from other apps (Zoom, Meet, etc.)',
NSCameraUsageDescription: 'Rowboat uses your camera in video chat mode so the assistant can see you and give feedback (e.g. pitch practice).',
},
osxSign: {
batchCodesignCalls: true,
@ -178,7 +276,8 @@ module.exports = {
}
}
},
{
// Arch Linux package — local-only; disabled in CI via ROWBOAT_SKIP_PACMAN.
...(SKIP_PACMAN ? [] : [{
name: require.resolve('./makers/maker-pacman.cjs'),
platforms: ['linux'],
config: {
@ -192,7 +291,7 @@ module.exports = {
icon: path.join(__dirname, 'icons/icon.png'),
mimeType: ['x-scheme-handler/rowboat'],
}
},
}]),
{
name: '@electron-forge/maker-zip',
platform: ["darwin", "win32", "linux"],
@ -291,10 +390,12 @@ module.exports = {
// must be copied in explicitly. See stageAcpAdapters() above for the why.
console.log('Staging ACP adapters...');
const acpDest = path.join(packageDir, 'acp', 'node_modules');
const staged = stageAcpAdapters(__dirname, acpDest);
console.log(`✅ Staged ${staged} ACP adapter packages into .package/acp/node_modules`);
const { copied: staged, placements } = stageAcpAdapters(__dirname, acpDest);
// Assert the hoisted tree resolves identically to source before shipping it.
verifyAcpStaging(__dirname, placements);
console.log(`✅ Staged ${staged} ACP adapter packages into .package/acp/node_modules (resolution verified)`);
console.log('✅ All assets staged in .package/');
},
}
};
};

32
apps/x/apps/main/src/browser/ipc.ts
View file

@ -1,6 +1,6 @@
import { BrowserWindow } from 'electron';
import { ipc } from '@x/shared';
import { browserViewManager, type BrowserState } from './view.js';
import { browserViewManager, type BrowserState, type HttpAuthRequest } from './view.js';
type IPCChannels = ipc.IPCChannels;
@ -20,6 +20,7 @@ type BrowserHandlers = {
'browser:forward': InvokeHandler<'browser:forward'>;
'browser:reload': InvokeHandler<'browser:reload'>;
'browser:getState': InvokeHandler<'browser:getState'>;
'browser:httpAuthResponse': InvokeHandler<'browser:httpAuthResponse'>;
};
/**
@ -62,6 +63,9 @@ export const browserIpcHandlers: BrowserHandlers = {
'browser:getState': async () => {
return browserViewManager.getState();
},
'browser:httpAuthResponse': async (_event, args) => {
return browserViewManager.respondToHttpAuth(args);
},
};
/**
@ -70,12 +74,26 @@ export const browserIpcHandlers: BrowserHandlers = {
* window is created so the manager has a window to attach to.
*/
export function setupBrowserEventForwarding(): void {
browserViewManager.on('state-updated', (state: BrowserState) => {
const windows = BrowserWindow.getAllWindows();
for (const win of windows) {
if (!win.isDestroyed() && win.webContents) {
win.webContents.send('browser:didUpdateState', state);
}
// Only send to app windows, never to OAuth/SSO popup windows created by
// page window.open() — those render untrusted web content, and browsing
// state / auth-challenge metadata must not cross into them.
const broadcast = (channel: string, payload: unknown) => {
for (const win of BrowserWindow.getAllWindows()) {
if (win.isDestroyed() || !win.webContents) continue;
if (browserViewManager.isPopupWindow(win)) continue;
win.webContents.send(channel, payload);
}
};
browserViewManager.on('state-updated', (state: BrowserState) => {
broadcast('browser:didUpdateState', state);
});
browserViewManager.on('http-auth-request', (request: HttpAuthRequest) => {
broadcast('browser:httpAuthRequest', request);
});
browserViewManager.on('http-auth-resolved', (requestId: string) => {
broadcast('browser:httpAuthResolved', { requestId });
});
}

259
apps/x/apps/main/src/browser/view.ts
View file

@ -1,11 +1,12 @@
import { randomUUID } from 'node:crypto';
import { EventEmitter } from 'node:events';
import { BrowserWindow, WebContentsView, session, shell, type Session } from 'electron';
import { BrowserWindow, WebContentsView, session, shell, type Session, type WebContents } from 'electron';
import type {
BrowserPageElement,
BrowserPageSnapshot,
BrowserState,
BrowserTabState,
HttpAuthRequest,
} from '@x/shared/dist/browser-control.js';
import { normalizeNavigationTarget } from './navigation.js';
import {
@ -20,7 +21,7 @@ import {
type RawBrowserPageSnapshot,
} from './page-scripts.js';
export type { BrowserPageSnapshot, BrowserState, BrowserTabState };
export type { BrowserPageSnapshot, BrowserState, BrowserTabState, HttpAuthRequest };
/**
* Embedded browser pane implementation.
@ -36,13 +37,33 @@ export type { BrowserPageSnapshot, BrowserState, BrowserTabState };
export const BROWSER_PARTITION = 'persist:rowboat-browser';
// Claims Chrome 130 on macOS — close enough to recent stable for OAuth servers
// that sniff the UA looking for "real browser" shapes.
const SPOOF_UA =
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36';
// Spoof a real Chrome UA so OAuth servers don't reject the embedded browser.
// The Chrome major version is derived from the running Chromium at startup:
// pinning a fixed version goes stale as Electron upgrades, and Chromium keeps
// emitting Sec-CH-UA client hints with the *real* version — a UA/client-hint
// version mismatch is a classic bot-detection signal (Google sign-in,
// Cloudflare). Minor version is frozen at 0.0.0, exactly like real Chrome's
// reduced UA. The platform token matches the actual OS for the same reason.
function getChromeMajorVersion(): number {
const major = Number.parseInt(process.versions.chrome ?? '', 10);
return Number.isFinite(major) && major > 0 ? major : 130;
}
function buildChromeUserAgent(): string {
const platformToken =
process.platform === 'darwin'
? 'Macintosh; Intel Mac OS X 10_15_7'
: process.platform === 'win32'
? 'Windows NT 10.0; Win64; x64'
: 'X11; Linux x86_64';
return `Mozilla/5.0 (${platformToken}) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/${getChromeMajorVersion()}.0.0.0 Safari/537.36`;
}
const SPOOF_UA = buildChromeUserAgent();
const HOME_URL = 'https://www.google.com';
const NAVIGATION_TIMEOUT_MS = 10000;
const HTTP_AUTH_TIMEOUT_MS = 120000;
const POST_ACTION_IDLE_MS = 400;
const POST_ACTION_MAX_ELEMENTS = 25;
const POST_ACTION_MAX_TEXT_LENGTH = 4000;
@ -68,6 +89,13 @@ type CachedSnapshot = {
elements: Array<{ index: number; selector: string }>;
};
type PendingHttpAuth = {
callback: (username?: string, password?: string) => void;
timer: NodeJS.Timeout;
// The webContents that raised the challenge, so its teardown can cancel it.
webContents: WebContents;
};
const EMPTY_STATE: BrowserState = {
activeTabId: null,
tabs: [],
@ -109,6 +137,12 @@ export class BrowserViewManager extends EventEmitter {
private visible = false;
private bounds: BrowserBounds = { x: 0, y: 0, width: 0, height: 0 };
private snapshotCache = new Map<string, CachedSnapshot>();
private pendingHttpAuth = new Map<string, PendingHttpAuth>();
// Child windows created by page window.open() (OAuth/SSO popups). Tracked so
// they can be closed when the host window goes away — otherwise an orphaned
// popup keeps BrowserWindow.getAllWindows() non-empty and, on macOS, blocks
// the app from reopening via the Dock (see main.ts 'activate' handler).
private popupWindows = new Set<BrowserWindow>();
private cleanupWindowListeners: (() => void) | null = null;
attach(window: BrowserWindow): void {
@ -137,6 +171,7 @@ export class BrowserViewManager extends EventEmitter {
if (this.window !== window) return;
const tabs = [...this.tabs.values()];
const popups = [...this.popupWindows];
this.cleanupWindowListeners = null;
this.window = null;
this.browserSession = null;
@ -150,6 +185,14 @@ export class BrowserViewManager extends EventEmitter {
this.attachedTabId = null;
this.visible = false;
this.snapshotCache.clear();
for (const requestId of [...this.pendingHttpAuth.keys()]) {
this.finishHttpAuth(requestId);
}
// Close any OAuth/SSO popups so they don't outlive the app window.
for (const popup of popups) {
if (!popup.isDestroyed()) popup.close();
}
this.popupWindows.clear();
};
hostWebContents.on('did-start-loading', handleDidStartLoading);
@ -171,6 +214,36 @@ export class BrowserViewManager extends EventEmitter {
if (this.browserSession) return this.browserSession;
const browserSession = session.fromPartition(BROWSER_PARTITION);
browserSession.setUserAgent(SPOOF_UA);
// Electron's Sec-CH-UA client hints only carry the "Chromium" brand;
// real Chrome also sends "Google Chrome". Some sign-in flows (notably
// Google's) distinguish the two, so rewrite the brand list to match what
// Chrome sends. Both the low-entropy header (`sec-ch-ua`, major versions)
// and the high-entropy one (`sec-ch-ua-full-version-list`, requested via
// Accept-CH and carrying full versions) must be rewritten together — a
// header that claims "Google Chrome" alongside one that doesn't is a
// stronger bot signal than the original. Only headers Chromium already
// attached are rewritten — none are added. (navigator.userAgentData JS
// brands still report only Chromium; there is no reliable hook to spoof
// that under sandbox+contextIsolation, and header-based detection is the
// common case.)
const chromeMajor = getChromeMajorVersion();
const chromeFull = process.versions.chrome ?? `${chromeMajor}.0.0.0`;
const brandLists: Record<string, string> = {
'sec-ch-ua': `"Chromium";v="${chromeMajor}", "Google Chrome";v="${chromeMajor}", "Not-A.Brand";v="99"`,
'sec-ch-ua-full-version-list': `"Chromium";v="${chromeFull}", "Google Chrome";v="${chromeFull}", "Not-A.Brand";v="99.0.0.0"`,
};
browserSession.webRequest.onBeforeSendHeaders((details, callback) => {
const requestHeaders = details.requestHeaders;
for (const name of Object.keys(requestHeaders)) {
const replacement = brandLists[name.toLowerCase()];
if (replacement !== undefined) {
requestHeaders[name] = replacement;
}
}
callback({ requestHeaders });
});
this.browserSession = browserSession;
return browserSession;
}
@ -196,17 +269,34 @@ export class BrowserViewManager extends EventEmitter {
return /^https?:\/\//i.test(url) || url === 'about:blank';
}
/**
* webPreferences shared by browser tabs and OAuth popups. Kept in one place
* so the security-sensitive popup surface can never drift from tabs.
*/
private browserWebPreferences(): Electron.WebPreferences {
return {
session: this.getSession(),
contextIsolation: true,
sandbox: true,
nodeIntegration: false,
// Chromium's built-in PDFium viewer, so PDFs render inline instead
// of showing a blank page.
plugins: true,
// Remove the WebAuthn API from the embedded browser only. Electron ships
// the API but not Chrome's authenticator UI (Touch ID sheet, QR/phone
// hybrid), so passkey challenges hang forever on "Verifying it's you...".
// With the API absent, sites feature-detect it and fall back to
// password/other verification. Scoped here (not app-wide) so the app's
// own renderer keeps WebAuthn.
disableBlinkFeatures: 'WebAuth',
};
}
private createView(): WebContentsView {
const view = new WebContentsView({
webPreferences: {
session: this.getSession(),
contextIsolation: true,
sandbox: true,
nodeIntegration: false,
},
webPreferences: this.browserWebPreferences(),
});
view.webContents.setUserAgent(SPOOF_UA);
return view;
}
@ -266,16 +356,145 @@ export class BrowserViewManager extends EventEmitter {
});
wc.on('page-title-updated', this.emitState.bind(this));
wc.setWindowOpenHandler(({ url }) => {
if (this.isEmbeddedTabUrl(url)) {
void this.newTab(url);
} else {
void shell.openExternal(url);
this.wireWindowPolicy(wc);
}
/**
* Window-open, popup, and HTTP-auth wiring shared by tabs and popups.
*/
private wireWindowPolicy(wc: WebContents): void {
wc.setWindowOpenHandler((details) => this.handleWindowOpen(details));
wc.on('did-create-window', (child) => this.wirePopupWindow(child));
this.wireHttpAuth(wc);
}
/**
* Shared window.open / target=_blank policy for tabs and popups.
*
* An open that hands a handle back to the opener must become a real child
* window so window.opener / postMessage survive — this is how OAuth/SSO
* popups (Google, Microsoft, Plaid, ...) return their result; denying them
* also makes sites report "popup blocked". Those are: a sized popup
* (disposition 'new-window'), a *named* window.open(url, 'name') (non-empty
* frameName), or a scripted blank window the opener will populate
* (about:blank). A nameless target=_blank link (foreground-tab, empty
* frameName) has no opener contract and opens as a tab, matching browser
* behavior. Non-web schemes go to the system handler.
*
* Residual gap: a nameless, featureless window.open(url) is indistinguishable
* from a _blank link (both foreground-tab + empty frameName) and opens as a
* tab, losing its opener — rare for OAuth, which virtually always names or
* sizes its popup.
*/
private handleWindowOpen(details: Electron.HandlerDetails): Electron.WindowOpenHandlerResponse {
const { url, disposition, frameName } = details;
if (this.isEmbeddedTabUrl(url)) {
const needsOpener =
disposition === 'new-window' || frameName !== '' || url === 'about:blank';
if (needsOpener) {
return {
action: 'allow',
overrideBrowserWindowOptions: {
autoHideMenuBar: true,
webPreferences: this.browserWebPreferences(),
},
};
}
void this.newTab(url);
return { action: 'deny' };
}
void shell.openExternal(url);
return { action: 'deny' };
}
private wirePopupWindow(child: BrowserWindow): void {
this.popupWindows.add(child);
child.once('closed', () => this.popupWindows.delete(child));
this.wireWindowPolicy(child.webContents);
}
/** True if `win` is an OAuth/SSO popup created by page window.open(). */
isPopupWindow(win: BrowserWindow): boolean {
return this.popupWindows.has(win);
}
/**
* HTTP basic/proxy auth. Chromium's default is to cancel the challenge, so
* 401-protected sites and authenticating proxies dead-end. When the browser
* pane is on screen to answer, forward the challenge to it as a credential
* prompt (cancelled after a timeout if unanswered). When the pane is closed
* — e.g. agent-driven navigation — don't preventDefault, so Chromium cancels
* immediately and the 401 page is readable rather than hanging.
*/
private wireHttpAuth(wc: WebContents): void {
wc.on('login', (event, _details, authInfo, callback) => {
if (!this.visible || !this.window) return;
event.preventDefault();
const requestId = randomUUID();
const timer = setTimeout(() => {
this.finishHttpAuth(requestId);
}, HTTP_AUTH_TIMEOUT_MS);
this.pendingHttpAuth.set(requestId, { callback, timer, webContents: wc });
// If the challenging contents dies before an answer, resolve now so the
// native callback and timer don't leak (backstop for paths other than
// destroyTab, which cancels explicitly before removeAllListeners()).
wc.once('destroyed', () => this.finishHttpAuth(requestId));
const request: HttpAuthRequest = {
requestId,
host: authInfo.host,
isProxy: authInfo.isProxy,
...(authInfo.realm ? { realm: authInfo.realm } : {}),
};
this.emit('http-auth-request', request);
});
}
/**
* Resolve a pending auth challenge. `username === undefined` cancels it; an
* empty-string username is a valid submission (token-style Basic auth).
* Always notifies the renderer so a dialog it may still be showing (e.g.
* after a timeout or tab close) is pruned.
*/
private finishHttpAuth(requestId: string, username?: string, password?: string): boolean {
const pending = this.pendingHttpAuth.get(requestId);
if (!pending) return false;
this.pendingHttpAuth.delete(requestId);
clearTimeout(pending.timer);
try {
if (username == null) {
pending.callback();
} else {
pending.callback(username, password ?? '');
}
} catch {
// The challenged webContents may already be destroyed.
}
this.emit('http-auth-resolved', requestId);
return true;
}
private cancelHttpAuthForWebContents(wc: WebContents): void {
const ids: string[] = [];
for (const [requestId, pending] of this.pendingHttpAuth) {
if (pending.webContents === wc) ids.push(requestId);
}
for (const requestId of ids) {
this.finishHttpAuth(requestId);
}
}
respondToHttpAuth(input: {
requestId: string;
username?: string;
password?: string;
}): { ok: boolean } {
return { ok: this.finishHttpAuth(input.requestId, input.username, input.password) };
}
private snapshotTabState(tab: BrowserTab): BrowserTabState {
const wc = tab.view.webContents;
return {
@ -364,6 +583,10 @@ export class BrowserViewManager extends EventEmitter {
private destroyTab(tab: BrowserTab): void {
this.invalidateSnapshot(tab.id);
// Cancel any auth challenge this tab raised before we drop its listeners,
// so the native callback + timer don't leak and the renderer prunes its
// dialog (removeAllListeners() below would kill the 'destroyed' backstop).
this.cancelHttpAuthForWebContents(tab.view.webContents);
tab.view.webContents.removeAllListeners();
if (!tab.view.webContents.isDestroyed()) {
tab.view.webContents.close();

383
apps/x/apps/main/src/ipc.ts
View file

@ -1,7 +1,8 @@
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app } from 'electron';
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app, screen } from 'electron';
import { ipc } from '@x/shared';
import path from 'node:path';
import os from 'node:os';
import { fileURLToPath } from 'node:url';
import {
connectProvider,
disconnectProvider,
@ -24,6 +25,8 @@ const execFileAsync = promisify(execFile);
import { RunEvent } from '@x/shared/dist/runs.js';
import { ServiceEvent } from '@x/shared/dist/service-events.js';
import type { SessionBusEvent } from '@x/shared/dist/sessions.js';
import type { ISessions, EmitterSessionBus } from '@x/core/dist/sessions/index.js';
import container from '@x/core/dist/di/container.js';
import { listOnboardingModels } from '@x/core/dist/models/models-dev.js';
import { testModelConnection, listModelsForProvider, generateOneShot } from '@x/core/dist/models/models.js';
@ -49,6 +52,8 @@ import type { CodeSession } from '@x/shared/dist/code-sessions.js';
import { invalidateCopilotInstructionsCache } from '@x/core/dist/application/assistant/instructions.js';
import { triggerSync as triggerGranolaSync } from '@x/core/dist/knowledge/granola/sync.js';
import { ISlackConfigRepo } from '@x/core/dist/slack/repo.js';
import { IChannelsConfigRepo } from '@x/core/dist/channels/repo.js';
import { applyChannelsConfig, getChannelsStatus, logoutWhatsApp, subscribeChannelsStatus } from '@x/core/dist/channels/service.js';
import { runAgentSlack, getAgentSlackCliStatus, AgentSlackRunError } from '@x/core/dist/slack/agent-slack-exec.js';
import { knowledgeSourcesRepo } from '@x/core/dist/knowledge/sources/repo.js';
import { rankSlackHomeMessages } from '@x/core/dist/knowledge/sources/rank_slack_home.js';
@ -62,6 +67,9 @@ import { IAgentScheduleRepo } from '@x/core/dist/agent-schedule/repo.js';
import { IAgentScheduleStateRepo } from '@x/core/dist/agent-schedule/state-repo.js';
import { triggerRun as triggerAgentScheduleRun } from '@x/core/dist/agent-schedule/runner.js';
import { search } from '@x/core/dist/search/search.js';
import { resolveMeetingPrep } from '@x/core/dist/knowledge/meeting_prep.js';
import { readPrepNoteForEvent } from '@x/core/dist/knowledge/meeting_prep_brief.js';
import { invalidateKnowledgeIndex } from '@x/core/dist/knowledge/knowledge_index.js';
import { versionHistory, voice } from '@x/core';
import { classifySchedule, processRowboatInstruction } from '@x/core/dist/knowledge/inline_tasks.js';
import { getBillingInfo } from '@x/core/dist/billing/billing.js';
@ -69,7 +77,7 @@ import { summarizeMeeting } from '@x/core/dist/knowledge/summarize_meeting.js';
import { getAccessToken } from '@x/core/dist/auth/tokens.js';
import { getRowboatConfig } from '@x/core/dist/config/rowboat.js';
import { runLiveNoteAgent } from '@x/core/dist/knowledge/live-note/runner.js';
import { listImportantThreads, listEverythingElseThreads, saveMessageBodyHeight, triggerSync as triggerGmailSync, sendThreadReply, archiveThread, trashThread, markThreadRead, getAccountEmail, getAccountName, getConnectionStatus as getGmailConnectionStatus } from '@x/core/dist/knowledge/sync_gmail.js';
import { listImportantThreads, listEverythingElseThreads, saveMessageBodyHeight, triggerSync as triggerGmailSync, sendThreadReply, saveThreadDraft, deleteThreadDraft, listDraftThreads, searchThreads, archiveThread, trashThread, markThreadRead, downloadAttachment, getAccountEmail, getAccountName, getConnectionStatus as getGmailConnectionStatus, setThreadImportance } from '@x/core/dist/knowledge/sync_gmail.js';
import { searchContacts as searchGmailContacts, warmContactIndex } from '@x/core/dist/knowledge/gmail_contacts.js';
import { searchSentContacts, warmSentContacts } from '@x/core/dist/knowledge/gmail_sent_contacts.js';
import { getGoogleDocsConnectionStatus, importGoogleDoc, syncGoogleDocDown, syncGoogleDocUp, getGoogleDocLink } from '@x/core/dist/knowledge/google_docs.js';
@ -376,9 +384,37 @@ type InvokeHandlers = {
[K in InvokeChannels]: InvokeHandler<K>;
};
// In-flight streaming TTS requests, keyed by renderer-chosen requestId.
const activeTtsStreams = new Map<string, AbortController>();
// Video-mode popout window (shown for the whole duration of a screen share,
// floating over every app including Rowboat itself) and the last call state
// pushed by the main window — replayed to the popout when it finishes loading.
let videoPopoutWin: BrowserWindow | null = null;
let lastVideoPopoutState: {
ttsState: 'idle' | 'synthesizing' | 'speaking';
status: 'listening' | 'thinking' | 'speaking' | null;
cameraOn: boolean;
micMuted: boolean;
screenSharing: boolean;
interimText: string | null;
} | null = null;
// Match only real app windows — getAllWindows() can also contain the popout
// itself and hidden utility windows (e.g. PDF-export renderers), which must
// not be shown, focused, or sent app events.
function findMainAppWindow(): BrowserWindow | undefined {
return BrowserWindow.getAllWindows().find((w) => {
if (w === videoPopoutWin || w.isDestroyed()) return false;
const url = w.webContents.getURL();
const isAppWindow = url.startsWith('app://') || url.startsWith('http://localhost');
return isAppWindow && !url.includes('#video-popout');
});
}
/**
* Register all IPC handlers with type safety and runtime validation
*
*
* This function ensures:
* 1. All invoke channels have handlers (exhaustiveness checking)
* 2. Handler signatures match channel definitions
@ -507,7 +543,25 @@ function queueChange(relPath: string): void {
/**
* Handle workspace change event from core watcher
*/
function touchesKnowledge(event: z.infer<typeof workspaceShared.WorkspaceChangeEvent>): boolean {
const hit = (p: string | undefined) => typeof p === 'string' && p.startsWith('knowledge/');
switch (event.type) {
case 'created':
case 'changed':
case 'deleted':
return hit(event.path);
case 'moved':
return hit(event.from) || hit(event.to);
case 'bulkChanged':
return !event.paths || event.paths.some(hit);
default:
return false;
}
}
function handleWorkspaceChange(event: z.infer<typeof workspaceShared.WorkspaceChangeEvent>): void {
// Any knowledge-base change drops the cached index so the next read rebuilds.
if (touchesKnowledge(event)) invalidateKnowledgeIndex();
// Debounce 'changed' events, emit others immediately
if (event.type === 'changed' && event.path) {
queueChange(event.path);
@ -569,6 +623,9 @@ function emitServiceEvent(event: z.infer<typeof ServiceEvent>): void {
}
export function emitOAuthEvent(event: { provider: string; success: boolean; error?: string; userId?: string }): void {
// Native connection status (e.g. Google) is baked into the Copilot system
// prompt, so any OAuth state change must rebuild it.
invalidateCopilotInstructionsCache();
const windows = BrowserWindow.getAllWindows();
for (const win of windows) {
if (!win.isDestroyed() && win.webContents) {
@ -613,6 +670,52 @@ export async function startRunsWatcher(): Promise<void> {
});
}
// New runtime: session bus → renderer windows (session-design.md §10).
function emitSessionEvent(event: SessionBusEvent): void {
const windows = BrowserWindow.getAllWindows();
for (const win of windows) {
if (!win.isDestroyed() && win.webContents) {
win.webContents.send('sessions:events', event);
}
}
}
// Mobile channels: status changes (QR pairing, connect/disconnect) → renderer.
let channelsWatcher: (() => void) | null = null;
export function startChannelsWatcher(): void {
if (channelsWatcher) return;
channelsWatcher = subscribeChannelsStatus((status) => {
const windows = BrowserWindow.getAllWindows();
for (const win of windows) {
if (!win.isDestroyed() && win.webContents) {
win.webContents.send('channels:status', status);
}
}
});
}
let sessionsWatcher: (() => void) | null = null;
export function startSessionsWatcher(): void {
if (sessionsWatcher) {
return;
}
const sessionBus = container.resolve<EmitterSessionBus>('sessionBus');
sessionsWatcher = sessionBus.subscribe((event) => emitSessionEvent(event));
}
// The renderer window is created before the session-index startup scan
// finishes, so an early sessions:list could observe a partially built index
// (the scan runs oldest-first — exactly the newest chats would be missing).
// sessions:list awaits this deferred; main.ts resolves it when the scan
// settles (success or failure, so the list never hangs).
let resolveSessionsIndexReady: () => void;
const sessionsIndexReady = new Promise<void>((resolve) => {
resolveSessionsIndexReady = resolve;
});
export function markSessionsIndexReady(): void {
resolveSessionsIndexReady();
}
let servicesWatcher: (() => void) | null = null;
export async function startServicesWatcher(): Promise<void> {
if (servicesWatcher) {
@ -740,6 +843,18 @@ export function setupIpcHandlers() {
'gmail:sendReply': async (_event, args) => {
return sendThreadReply(args);
},
'gmail:saveDraft': async (_event, args) => {
return saveThreadDraft(args);
},
'gmail:deleteDraft': async (_event, args) => {
return deleteThreadDraft(args.draftId);
},
'gmail:getDrafts': async () => {
return listDraftThreads();
},
'gmail:search': async (_event, args) => {
return searchThreads(args.query, { limit: args.limit });
},
'gmail:getConnectionStatus': async () => {
return getGmailConnectionStatus();
},
@ -749,6 +864,10 @@ export function setupIpcHandlers() {
'gmail:getAccountName': async () => {
return { name: await getAccountName() };
},
'gmail:setImportance': async (_event, args) => {
const result = setThreadImportance(args.threadId, args.importance);
return { ok: result.success, previous: result.previous, error: result.error };
},
'gmail:archiveThread': async (_event, args) => {
return archiveThread(args.threadId);
},
@ -756,7 +875,10 @@ export function setupIpcHandlers() {
return trashThread(args.threadId);
},
'gmail:markThreadRead': async (_event, args) => {
return markThreadRead(args.threadId);
return markThreadRead(args.threadId, args.read);
},
'gmail:downloadAttachment': async (_event, args) => {
return downloadAttachment(args);
},
'gmail:saveMessageHeight': async (_event, args) => {
saveMessageBodyHeight(args.threadId, args.messageId, args.height);
@ -813,10 +935,90 @@ export function setupIpcHandlers() {
'runs:list': async (_event, args) => {
return runsCore.listRuns(args.cursor);
},
'runs:listByWorkDir': async (_event, args) => {
return runsCore.listRunsByWorkDir(args.dir);
},
'runs:delete': async (_event, args) => {
await runsCore.deleteRun(args.runId);
return { success: true };
},
// ── New runtime: sessions + turns ─────────────────────────
// Thin pass-throughs to the sessions service. sendMessage returns the
// turnId immediately; the turn advances in the background and the
// renderer reconciles via the sessions:events feed. Input-routing calls
// settle with that advance's outcome (the renderer fire-and-forgets).
'sessions:create': async (_event, args) => {
const sessionId = await container.resolve<ISessions>('sessions').createSession(args);
return { sessionId };
},
'sessions:list': async () => {
await sessionsIndexReady;
return { sessions: container.resolve<ISessions>('sessions').listSessions() };
},
'sessions:get': async (_event, args) => {
return container.resolve<ISessions>('sessions').getSession(args.sessionId);
},
'sessions:getTurn': async (_event, args) => {
return container.resolve<ISessions>('sessions').getTurn(args.turnId);
},
'sessions:sendMessage': async (_event, args) => {
return container.resolve<ISessions>('sessions').sendMessage(args.sessionId, args.input, args.config);
},
'sessions:respondToPermission': async (_event, args) => {
await container.resolve<ISessions>('sessions').respondToPermission(args.turnId, args.toolCallId, args.decision, args.metadata);
return { success: true };
},
'sessions:respondToAskHuman': async (_event, args) => {
await container.resolve<ISessions>('sessions').respondToAskHuman(args.turnId, args.toolCallId, args.answer);
return { success: true };
},
'sessions:stopTurn': async (_event, args) => {
await container.resolve<ISessions>('sessions').stopTurn(args.turnId, args.reason);
return { success: true };
},
'sessions:resumeTurn': async (_event, args) => {
await container.resolve<ISessions>('sessions').resumeTurn(args.sessionId);
return { success: true };
},
'sessions:setTitle': async (_event, args) => {
await container.resolve<ISessions>('sessions').setTitle(args.sessionId, args.title);
return { success: true };
},
'sessions:delete': async (_event, args) => {
await container.resolve<ISessions>('sessions').deleteSession(args.sessionId);
return { success: true };
},
'sessions:downloadLog': async (event, args) => {
// Concatenate the session's turn logs into one JSONL for debugging.
const sessions = container.resolve<ISessions>('sessions');
const state = await sessions.getSession(args.sessionId);
const win = BrowserWindow.fromWebContents(event.sender);
const result = await dialog.showSaveDialog(win!, {
defaultPath: `${args.sessionId}.jsonl.log`,
filters: [
{ name: 'Chat Log', extensions: ['log'] },
{ name: 'JSONL', extensions: ['jsonl'] },
{ name: 'All Files', extensions: ['*'] },
],
});
if (result.canceled || !result.filePath) {
return { success: false };
}
try {
const lines: string[] = [];
for (const ref of state.turns) {
const turn = await sessions.getTurn(ref.turnId);
for (const turnEvent of turn.events) {
lines.push(JSON.stringify(turnEvent));
}
}
await fs.writeFile(result.filePath, lines.join('\n') + '\n');
return { success: true };
} catch (err) {
const message = err instanceof Error ? err.message : 'Failed to download chat log';
return { success: false, error: message };
}
},
'runs:downloadLog': async (event, args) => {
const runFileName = `${args.runId}.jsonl`;
if (path.basename(runFileName) !== runFileName) {
@ -1072,6 +1274,22 @@ export function setupIpcHandlers() {
return { success: true };
},
// ── Mobile channels (WhatsApp / Telegram bridge) ─────────────
'channels:getConfig': async () => {
return container.resolve<IChannelsConfigRepo>('channelsConfigRepo').getConfig();
},
'channels:setConfig': async (_event, args) => {
await container.resolve<IChannelsConfigRepo>('channelsConfigRepo').setConfig(args);
await applyChannelsConfig(args);
return { success: true };
},
'channels:getStatus': async () => {
return getChannelsStatus();
},
'channels:whatsappLogout': async () => {
await logoutWhatsApp();
return { success: true };
},
'slack:getConfig': async () => {
const repo = container.resolve<ISlackConfigRepo>('slackConfigRepo');
const config = await repo.getConfig();
@ -1541,6 +1759,11 @@ export function setupIpcHandlers() {
const notes = await summarizeMeeting(args.transcript, args.meetingStartTime, args.calendarEventJson);
return { notes };
},
'meeting-prep:resolve': async (_event, args) => {
const result = await resolveMeetingPrep(args.attendees);
const prepNote = args.eventId ? await readPrepNoteForEvent(args.eventId) : null;
return { ...result, prepNote };
},
'inline-task:classifySchedule': async (_event, args) => {
const schedule = await classifySchedule(args.instruction);
return { schedule };
@ -1554,6 +1777,51 @@ export function setupIpcHandlers() {
'voice:synthesize': async (_event, args) => {
return voice.synthesizeSpeech(args.text);
},
'voice:synthesizeStreamStart': async (event, args) => {
const { requestId, text } = args;
const sender = event.sender;
const controller = new AbortController();
activeTtsStreams.set(requestId, controller);
// Fire-and-forget: chunks are pushed to the renderer as they arrive so
// playback can begin immediately; the invoke returns once started.
void voice
.synthesizeSpeechStream(
text,
(chunk) => {
if (!sender.isDestroyed()) {
sender.send('voice:tts-chunk', {
requestId,
chunkBase64: chunk.toString('base64'),
done: false,
});
}
},
controller.signal,
)
.then(() => {
if (!sender.isDestroyed()) {
sender.send('voice:tts-chunk', { requestId, done: true });
}
})
.catch((err: unknown) => {
if (!sender.isDestroyed() && !controller.signal.aborted) {
sender.send('voice:tts-chunk', {
requestId,
done: true,
error: err instanceof Error ? err.message : String(err),
});
}
})
.finally(() => {
activeTtsStreams.delete(requestId);
});
return { ok: true };
},
'voice:synthesizeStreamCancel': async (_event, args) => {
activeTtsStreams.get(args.requestId)?.abort();
activeTtsStreams.delete(args.requestId);
return {};
},
'voice:ensureMicAccess': async () => {
if (process.platform !== 'darwin') return { granted: true };
const status = systemPreferences.getMediaAccessStatus('microphone');
@ -1572,6 +1840,113 @@ export function setupIpcHandlers() {
return { granted: false };
}
},
'voice:ensureCameraAccess': async () => {
if (process.platform !== 'darwin') return { granted: true };
const status = systemPreferences.getMediaAccessStatus('camera');
console.log('[video] Camera permission status:', status);
if (status === 'granted') return { granted: true };
// Same flow as the microphone: settle the native TCC prompt before the
// renderer's getUserMedia so the first video click doesn't silently fail.
try {
const granted = await systemPreferences.askForMediaAccess('camera');
console.log('[video] Camera permission after prompt:', granted);
return { granted };
} catch {
return { granted: false };
}
},
'video:setPopout': async (_event, args) => {
if (!args.show) {
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) videoPopoutWin.destroy();
videoPopoutWin = null;
return {};
}
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) return {};
const workArea = screen.getPrimaryDisplay().workArea;
const width = 340;
const height = 184;
const ipcDir = path.dirname(fileURLToPath(import.meta.url));
const preloadPath = app.isPackaged
? path.join(ipcDir, '../preload/dist/preload.js')
: path.join(ipcDir, '../../../preload/dist/preload.js');
const win = new BrowserWindow({
width,
height,
x: workArea.x + workArea.width - width - 24,
y: workArea.y + 24,
frame: false,
resizable: false,
alwaysOnTop: true,
skipTaskbar: true,
show: false,
hasShadow: true,
backgroundColor: '#171717',
webPreferences: {
nodeIntegration: false,
contextIsolation: true,
sandbox: true,
preload: preloadPath,
},
});
// Float above other apps on every workspace. Deliberately NOT
// `visibleOnFullScreen: true`: on macOS that flag hides the app's Dock
// icon for as long as such a window exists (the app becomes an
// "agent" app), which reads as Rowboat having vanished. The trade-off
// is the popout won't hover over other apps' fullscreen Spaces.
win.setAlwaysOnTop(true, 'floating');
win.setVisibleOnAllWorkspaces(true);
win.webContents.once('did-finish-load', () => {
if (lastVideoPopoutState) {
win.webContents.send('video:popout-state', lastVideoPopoutState);
}
// showInactive: appearing must not steal focus from the app the user
// switched to — that would immediately re-hide the popout.
if (!win.isDestroyed()) win.showInactive();
});
win.on('closed', () => {
if (videoPopoutWin === win) videoPopoutWin = null;
});
videoPopoutWin = win;
if (app.isPackaged) {
win.loadURL('app://-/index.html#video-popout');
} else {
win.loadURL('http://localhost:5173/#video-popout');
}
return {};
},
'video:popoutState': async (_event, args) => {
lastVideoPopoutState = args;
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) {
videoPopoutWin.webContents.send('video:popout-state', args);
}
return {};
},
'app:focusMainWindow': async () => {
const main = findMainAppWindow();
if (main) {
if (main.isMinimized()) main.restore();
main.show();
main.focus();
}
return {};
},
'video:getPopoutState': async () => {
return { state: lastVideoPopoutState };
},
'video:popoutAction': async (_event, args) => {
// Relay a popout control-bar action to the app window, which owns the
// call (mic, camera, screen capture) and executes it there. 'expand'
// additionally brings the app window back to the foreground.
const main = findMainAppWindow();
if (args.action === 'expand' && main) {
if (main.isMinimized()) main.restore();
main.show();
main.focus();
}
main?.webContents.send('video:popout-action', args);
return {};
},
// Live-note handlers
'live-note:run': async (_event, args) => {
const result = await runLiveNoteAgent(args.filePath, 'manual', args.context);

90
apps/x/apps/main/src/main.ts
View file

@ -2,7 +2,8 @@ import { app, BrowserWindow, desktopCapturer, protocol, net, shell, session, typ
import path from "node:path";
import {
setupIpcHandlers,
startRunsWatcher,
startRunsWatcher, startSessionsWatcher, markSessionsIndexReady,
startChannelsWatcher,
startCodeSessionStatusWatcher,
startServicesWatcher,
startLiveNoteAgentWatcher,
@ -25,8 +26,10 @@ import { init as initEmailLabeling } from "@x/core/dist/knowledge/label_emails.j
import { init as initNoteTagging } from "@x/core/dist/knowledge/tag_notes.js";
import { init as initInlineTasks } from "@x/core/dist/knowledge/inline_tasks.js";
import { init as initAgentRunner } from "@x/core/dist/agent-schedule/runner.js";
import { init as initChannels } from "@x/core/dist/channels/service.js";
import { init as initAgentNotes } from "@x/core/dist/knowledge/agent_notes.js";
import { init as initCalendarNotifications } from "@x/core/dist/knowledge/notify_calendar_meetings.js";
import { init as initMeetingPrep } from "@x/core/dist/knowledge/meeting_prep_scheduler.js";
import { init as initLiveNoteScheduler } from "@x/core/dist/knowledge/live-note/scheduler.js";
import { init as initEventProcessor, registerConsumer } from "@x/core/dist/events/init.js";
import { liveNoteEventConsumer } from "@x/core/dist/knowledge/live-note/event-consumer.js";
@ -35,6 +38,7 @@ import { backgroundTaskEventConsumer } from "@x/core/dist/background-tasks/event
import { init as initLocalSites, shutdown as shutdownLocalSites } from "@x/core/dist/local-sites/server.js";
import { shutdown as shutdownAnalytics } from "@x/core/dist/analytics/posthog.js";
import { identifyIfSignedIn } from "@x/core/dist/analytics/identify.js";
import { migrateRuns } from "@x/core/dist/migrations/runs/migrate.js";
import { initConfigs } from "@x/core/dist/config/initConfigs.js";
import { getAgentSlackCliStatus } from "@x/core/dist/slack/agent-slack-exec.js";
@ -44,6 +48,7 @@ import { execFileSync } from "node:child_process";
import { init as initChromeSync } from "@x/core/dist/knowledge/chrome-extension/server/server.js";
import container, { registerBrowserControlService, registerNotificationService } from "@x/core/dist/di/container.js";
import type { CodeModeManager } from "@x/core/dist/code-mode/acp/manager.js";
import type { ISessions } from "@x/core/dist/sessions/index.js";
import { browserViewManager, BROWSER_PARTITION } from "./browser/view.js";
import { setupBrowserEventForwarding } from "./browser/ipc.js";
import { ElectronBrowserControlService } from "./browser/control-service.js";
@ -56,6 +61,11 @@ import {
} from "./deeplink.js";
import { disconnectGoogleIfScopesStale } from "./oauth-handler.js";
// Captured as early as possible so it reflects actual process start. Used to
// gate grace-eligible notifications (e.g. the burst of background-task
// completions a reopen replays) — see ElectronNotificationService.
const APP_LAUNCHED_AT = Date.now();
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
@ -211,6 +221,38 @@ function configureSessionPermissions(targetSession: Session): void {
});
}
// Wire Ctrl/Cmd + (+ / − / 0) to zoom the renderer in/out/reset.
// The app sets no application menu, so the default menu's zoom roles aren't
// available — and on Linux the menu bar is suppressed by the frameless
// `hiddenInset` title bar — so handle the accelerators directly here.
// `event.preventDefault()` stops the keystroke from leaking into the editor.
function setupZoomShortcuts(win: BrowserWindow) {
const ZOOM_STEP = 0.5; // zoom-level units (factor = 1.2 ^ level, ~9.5% per step)
const MIN_ZOOM_LEVEL = -3;
const MAX_ZOOM_LEVEL = 3;
const wc = win.webContents;
wc.on("before-input-event", (event, input) => {
if (input.type !== "keyDown") return;
// Cmd on macOS, Ctrl elsewhere.
if (!(process.platform === "darwin" ? input.meta : input.control)) return;
// input.key is the produced character: "+"/"=" share a physical key (as do
// "-"/"_"), and numpad +/- produce the same characters, so this covers both.
const key = input.key;
if (key === "+" || key === "=") {
wc.setZoomLevel(Math.min(wc.getZoomLevel() + ZOOM_STEP, MAX_ZOOM_LEVEL));
event.preventDefault();
} else if (key === "-" || key === "_") {
wc.setZoomLevel(Math.max(wc.getZoomLevel() - ZOOM_STEP, MIN_ZOOM_LEVEL));
event.preventDefault();
} else if (key === "0") {
wc.setZoomLevel(0);
event.preventDefault();
}
});
}
function createWindow() {
const win = new BrowserWindow({
width: 1280,
@ -287,6 +329,9 @@ function createWindow() {
// The WebContentsView is created lazily on first `browser:setVisible`.
browserViewManager.attach(win);
// Cmd/Ctrl + (+ / − / 0) zoom shortcuts for the renderer UI.
setupZoomShortcuts(win);
if (app.isPackaged) {
win.loadURL("app://-/index.html");
} else {
@ -330,7 +375,7 @@ app.whenReady().then(async () => {
});
registerBrowserControlService(new ElectronBrowserControlService());
registerNotificationService(new ElectronNotificationService());
registerNotificationService(new ElectronNotificationService(APP_LAUNCHED_AT));
setupIpcHandlers();
setupBrowserEventForwarding();
@ -347,6 +392,44 @@ app.whenReady().then(async () => {
// start runs watcher
startRunsWatcher();
// One-time: port legacy runs/*.jsonl into the new turn/session runtime.
// Must run BEFORE the session index is built so migrated sessions are picked
// up by the startup scan. Fully defensive — never blocks boot.
try {
const migration = migrateRuns();
if (migration.scanned > 0) {
console.log(
`[runs-migration] migrated ${migration.migratedTurns} turn(s) across ` +
`${migration.migratedSessions} session(s) from ${migration.scanned} run(s) ` +
`(${migration.skipped} skipped, ${migration.failed.length} failed)`,
);
for (const failure of migration.failed) {
console.warn(`[runs-migration] left in place (failed): ${failure.file} — ${failure.error}`);
}
}
} catch (error) {
console.error('[runs-migration] pass failed:', error);
}
// New runtime: build the in-memory session index (startup scan), then
// forward the session bus to windows. The renderer window is already up and
// may have called sessions:list — that handler blocks on
// markSessionsIndexReady, which must fire even if the scan throws so the
// list never hangs.
try {
await container.resolve<ISessions>('sessions').initialize();
} finally {
markSessionsIndexReady();
}
startSessionsWatcher();
// Mobile channels (WhatsApp/Telegram bridge): needs the session index, so
// start after initialize(). Failures must never block boot.
startChannelsWatcher();
initChannels().catch((error) => {
console.error('[Channels] Failed to start mobile channels:', error);
});
// start code-session status tracker (derives working/needs-you/idle + notifications)
startCodeSessionStatusWatcher();
@ -410,6 +493,9 @@ app.whenReady().then(async () => {
// start calendar meeting notification service (fires 1-minute warnings)
initCalendarNotifications();
// start meeting prep scheduler (generates prep notes ~6h before a meeting)
void initMeetingPrep();
// start chrome extension sync server
initChromeSync();

19
apps/x/apps/main/src/notification/electron-notification-service.ts
View file

@ -1,5 +1,6 @@
import { BrowserWindow, Notification, shell } from "electron";
import type { INotificationService, NotifyInput } from "@x/core/dist/application/notification/service.js";
import { shouldSuppressDuringStartupGrace } from "@x/core/dist/application/notification/service.js";
import { dispatchUrl } from "../deeplink.js";
const HTTP_URL = /^https?:\/\//i;
@ -11,11 +12,27 @@ export class ElectronNotificationService implements INotificationService {
// gets dropped and macOS clicks just focus the app silently.
private active = new Set<Notification>();
// Timestamp the app launched, used to gate grace-eligible notifications (see
// NotifyInput.suppressDuringStartupGrace). Captured by the caller in main.ts
// so it reflects process start rather than whenever the first notify fires.
private readonly launchedAt: number;
constructor(launchedAt: number = Date.now()) {
this.launchedAt = launchedAt;
}
isSupported(): boolean {
return Notification.isSupported();
}
notify({ title = "Rowboat", message, link, actionLabel, secondaryActions, onlyWhenBackground }: NotifyInput): void {
notify({ title = "Rowboat", message, link, actionLabel, secondaryActions, onlyWhenBackground, suppressDuringStartupGrace }: NotifyInput): void {
// Startup grace: a reopen replays every background task that completed
// while the app was closed, so grace-eligible notifications fired in the
// first moments after launch are dropped to avoid a notification flood.
if (shouldSuppressDuringStartupGrace({ suppressDuringStartupGrace }, this.launchedAt)) {
return;
}
// Ambient notifications are suppressed while the app is in the
// foreground — the user is already looking at it. A window counts as
// foreground only if it's actually focused (minimized / other-space

11
apps/x/apps/renderer/package.json
View file

@ -6,7 +6,9 @@
"dev": "vite",
"build": "tsc -b && vite build",
"lint": "eslint .",
"preview": "vite preview"
"preview": "vite preview",
"test": "vitest run",
"test:watch": "vitest"
},
"dependencies": {
"@codemirror/language": "^6.12.3",
@ -83,6 +85,9 @@
},
"devDependencies": {
"@eslint/js": "^9.39.1",
"@testing-library/dom": "^10.4.1",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.2",
"@types/node": "^24.10.1",
"@types/react": "^19.2.5",
"@types/react-dom": "^19.2.3",
@ -91,9 +96,11 @@
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.4.24",
"globals": "^16.5.0",
"jsdom": "^29.1.1",
"tw-animate-css": "^1.4.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.4"
"vite": "^7.2.4",
"vitest": "catalog:"
}
}

62
apps/x/apps/renderer/scripts/generate-tour-audio.mjs Normal file
View file

@ -0,0 +1,62 @@
/**
* Regenerates the bundled product-tour narration clips.
*
* Parses the TOUR_STEPS texts straight out of product-tour.tsx (so the code
* stays the single source of truth), synthesizes each one via @x/core's
* synthesizeSpeech (Rowboat proxy when signed in, direct ElevenLabs otherwise,
* using the voice id configured there / in ~/.rowboat/config/elevenlabs.json),
* and writes MP3s to src/assets/tour/<step-id>.mp3.
*
* Run whenever a step's narration text or the tour voice changes:
* cd apps/x && npm run deps # script imports core's built output
* node apps/renderer/scripts/generate-tour-audio.mjs
*
* Pass step ids to regenerate only those clips (e.g. to re-roll one whose
* synthesis came out glitchy):
* node apps/renderer/scripts/generate-tour-audio.mjs welcome done
*/
import { mkdir, readFile, writeFile } from 'node:fs/promises'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
const here = path.dirname(fileURLToPath(import.meta.url))
const tourSource = path.join(here, '../src/components/product-tour.tsx')
const outDir = path.join(here, '../src/assets/tour')
const corePath = path.join(here, '../../../packages/core/dist/voice/voice.js')
const { synthesizeSpeech } = await import(corePath)
const src = await readFile(tourSource, 'utf8')
const start = src.indexOf('const TOUR_STEPS')
const end = src.indexOf('\n]', start)
if (start === -1 || end === -1) throw new Error('Could not locate TOUR_STEPS in product-tour.tsx')
const block = src.slice(start, end)
const steps = []
// voiceText, when present, is the spoken variant of the bubble text.
const re = /id: '([^']+)'[\s\S]*?text:\s*("[^"]*"|'[^']*')(?:,\s*voiceText:\s*("[^"]*"|'[^']*'))?/g
for (let m; (m = re.exec(block)); ) {
// The captures are JS string literals from our own source; evaluate them
// to resolve the quoting.
steps.push({ id: m[1], text: new Function(`return ${m[3] ?? m[2]}`)() })
}
if (steps.length === 0) throw new Error('Parsed zero tour steps — regex out of sync with product-tour.tsx?')
console.log(`Parsed ${steps.length} tour steps`)
const only = process.argv.slice(2)
if (only.length > 0) {
const unknown = only.filter((id) => !steps.some((s) => s.id === id))
if (unknown.length > 0) throw new Error(`Unknown step ids: ${unknown.join(', ')}`)
steps.splice(0, steps.length, ...steps.filter((s) => only.includes(s.id)))
console.log(`Regenerating only: ${only.join(', ')}`)
}
await mkdir(outDir, { recursive: true })
for (const step of steps) {
process.stdout.write(`synthesizing ${step.id}... `)
const { audioBase64 } = await synthesizeSpeech(step.text)
const file = path.join(outDir, `${step.id}.mp3`)
await writeFile(file, Buffer.from(audioBase64, 'base64'))
console.log(`${(audioBase64.length * 0.75 / 1024).toFixed(0)} KB`)
}
console.log(`Done — ${steps.length} clips in ${path.relative(process.cwd(), outDir)}`)

891
apps/x/apps/renderer/src/App.css
View file

File diff suppressed because it is too large Load diff

1108
apps/x/apps/renderer/src/App.tsx
View file

File diff suppressed because it is too large Load diff

BIN
apps/x/apps/renderer/src/assets/tour/agents.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/chats.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/code.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/composer.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/done.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/email.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/home.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/knowledge.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/meetings.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/welcome.mp3 Normal file
View file

Binary file not shown.

BIN
apps/x/apps/renderer/src/assets/tour/workspaces.mp3 Normal file
View file

Binary file not shown.

11
apps/x/apps/renderer/src/components/ai-elements/conversation.tsx
View file

@ -43,6 +43,7 @@ export const Conversation = ({
const contentRef = useRef<HTMLDivElement | null>(null);
const scrollRef = useRef<HTMLDivElement | null>(null);
const spacerRef = useRef<HTMLDivElement | null>(null);
const stickToBottomRef = useRef(true);
const [isAtBottom, setIsAtBottom] = useState(true);
const updateBottomState = useCallback(() => {
@ -50,7 +51,9 @@ export const Conversation = ({
if (!container) return;
const distanceFromBottom =
container.scrollHeight - container.scrollTop - container.clientHeight;
setIsAtBottom(distanceFromBottom <= BOTTOM_THRESHOLD_PX);
const atBottom = distanceFromBottom <= BOTTOM_THRESHOLD_PX;
stickToBottomRef.current = atBottom;
setIsAtBottom(atBottom);
}, []);
const applyAnchorLayout = useCallback(
@ -131,7 +134,12 @@ export const Conversation = ({
cancelAnimationFrame(rafId);
}
rafId = requestAnimationFrame(() => {
const shouldStick = !anchorMessageId && stickToBottomRef.current;
applyAnchorLayout(false);
if (shouldStick) {
container.scrollTop = container.scrollHeight;
updateBottomState();
}
});
};
@ -178,6 +186,7 @@ export const Conversation = ({
const container = scrollRef.current;
if (!container) return;
container.scrollTop = container.scrollHeight;
stickToBottomRef.current = true;
updateBottomState();
}, [updateBottomState]);

4
apps/x/apps/renderer/src/components/ai-elements/tool.tsx
View file

@ -22,7 +22,7 @@ import {
import { type ComponentProps, type ReactNode, isValidElement, useState } from "react";
import { AnimatePresence, motion } from "motion/react";
import type { ToolCall, ToolGroup as ToolGroupType } from "@/lib/chat-conversation";
import { getToolActionsSummary, getToolDisplayName, getToolGroupSummary, toToolState } from "@/lib/chat-conversation";
import { getToolActionsSummary, getToolDisplayName, getToolErrorText, getToolGroupSummary, toToolState } from "@/lib/chat-conversation";
const formatToolValue = (value: unknown) => {
if (typeof value === "string") return value;
@ -329,7 +329,7 @@ export const ToolGroupComponent = ({ group, isToolOpen, onToolOpenChange }: Tool
<ToolTabbedContent
input={tool.input as ToolUIPart["input"]}
output={tool.result as ToolUIPart["output"]}
errorText={tool.status === 'error' ? 'Tool error' : undefined}
errorText={getToolErrorText(tool)}
/>
</ToolContent>
</Tool>

70
apps/x/apps/renderer/src/components/bg-tasks-view.tsx
View file

@ -6,9 +6,7 @@ import {
Pencil, Check, PanelRightClose, PanelRightOpen, Sparkles,
Code2, FolderOpen, LayoutTemplate,
} from 'lucide-react'
import type { z } from 'zod'
import type { BackgroundTask, BackgroundTaskSummary, Triggers } from '@x/shared/dist/background-task.js'
import type { Run } from '@x/shared/dist/runs.js'
import { Button } from '@/components/ui/button'
import { Switch } from '@/components/ui/switch'
import { Input } from '@/components/ui/input'
@ -17,7 +15,7 @@ import { useBackgroundTaskAgentStatus } from '@/hooks/use-bg-task-agent-status'
import { formatRelativeTime } from '@/lib/relative-time'
import { toast } from '@/lib/toast'
import type { ConversationItem } from '@/lib/chat-conversation'
import { runLogToConversation } from '@/lib/run-to-conversation'
import { fetchAgentRunTranscript, type AgentRunTranscript } from '@/lib/agent-transcript'
import { CompactConversation } from '@/components/compact-conversation'
import { RichMarkdownViewer } from '@/components/rich-markdown-viewer'
import { HtmlFileViewer } from '@/components/html-file-viewer'
@ -970,10 +968,9 @@ function SetupTab({
// Runs history tab — list + drill-down transcript view
//
// Source of truth: `bg-tasks/<slug>/runs.log` — a plain-text file with one
// runId per line (newest first). The actual transcripts live at the global
// `$WorkDir/runs/<runId>.jsonl`, so this tab fetches runIds via the bg-task
// IPC, then loads each Run through the standard `runs:fetch`. No bg-task-
// specific transcript path or schema needed.
// turn id per line (newest first). Transcripts live in the turn runtime's
// storage; this tab fetches ids via the bg-task IPC, then loads each through
// the shared agent-transcript loader (turn-first, legacy-run fallback).
// ---------------------------------------------------------------------------
interface RunRowSummary {
@ -984,27 +981,6 @@ interface RunRowSummary {
error?: string
}
// Pull the bits we want to display for a row out of a full Run's event log.
function summarizeRun(run: z.infer<typeof Run>): RunRowSummary {
const out: RunRowSummary = { runId: run.id, createdAt: run.createdAt, trigger: run.subUseCase }
for (const event of run.log) {
if (event.type === 'error' && typeof event.error === 'string') {
out.error = event.error
} else if (event.type === 'message' && event.message?.role === 'assistant') {
const content = event.message.content
if (typeof content === 'string') {
out.summary = content
} else if (Array.isArray(content)) {
const text = content
.filter((p) => p.type === 'text')
.map((p) => ('text' in p ? p.text : ''))
.join('')
if (text) out.summary = text
}
}
}
return out
}
function RunsHistoryTab({ slug, task }: { slug: string; task: BackgroundTask }) {
const [rows, setRows] = useState<RunRowSummary[]>([])
@ -1016,19 +992,25 @@ function RunsHistoryTab({ slug, task }: { slug: string; task: BackgroundTask })
setLoading(true)
try {
const { runIds } = await window.ipc.invoke('bg-task:listRunIds', { slug, limit: 100 })
// Fetch each Run in parallel via the canonical IPC. Runs whose
// jsonl no longer exists (deleted manually, never written, …) are
// dropped silently.
// Fetch transcripts in parallel (turn-first, legacy-run
// fallback). Ids whose files no longer exist keep a bare row so
// the user knows the run happened.
const settled = await Promise.allSettled(
runIds.map(runId => window.ipc.invoke('runs:fetch', { runId }))
runIds.map(runId => fetchAgentRunTranscript(runId))
)
const next: RunRowSummary[] = []
for (let i = 0; i < settled.length; i++) {
const r = settled[i]
if (r.status === 'fulfilled' && r.value) {
next.push(summarizeRun(r.value))
if (r.status === 'fulfilled') {
const t = r.value
next.push({
runId: t.id,
...(t.createdAt === undefined ? {} : { createdAt: t.createdAt }),
...(t.trigger === undefined ? {} : { trigger: t.trigger }),
...(t.summary === undefined ? {} : { summary: t.summary }),
...(t.error === undefined ? {} : { error: t.error }),
})
} else {
// Keep the row visible with just the id so the user knows it exists.
next.push({ runId: runIds[i] })
}
}
@ -1134,7 +1116,7 @@ function RunTranscriptView({
isInFlight: boolean
onBack: () => void
}) {
const [run, setRun] = useState<z.infer<typeof Run> | null>(null)
const [transcript, setTranscript] = useState<AgentRunTranscript | null>(null)
const [loading, setLoading] = useState(true)
const [error, setError] = useState<string | null>(null)
@ -1144,15 +1126,13 @@ function RunTranscriptView({
setError(null)
void (async () => {
try {
// Bg-task transcripts now live at the global runs/ location —
// same path resolution as every other run, no special handling.
const r = await window.ipc.invoke('runs:fetch', { runId })
const t = await fetchAgentRunTranscript(runId)
if (cancelled) return
setRun(r)
setTranscript(t)
} catch (err) {
if (cancelled) return
setError(err instanceof Error ? err.message : String(err))
setRun(null)
setTranscript(null)
} finally {
if (!cancelled) setLoading(false)
}
@ -1160,8 +1140,8 @@ function RunTranscriptView({
return () => { cancelled = true }
}, [runId])
const summary = run ? summarizeRun(run) : undefined
const items: ConversationItem[] = run ? runLogToConversation(run.log) : []
const summary = transcript ?? undefined
const items: ConversationItem[] = transcript?.items ?? []
return (
<div className="flex flex-1 flex-col overflow-hidden">
@ -1221,10 +1201,10 @@ function RunTranscriptView({
Couldn&apos;t load transcript: {error}
</div>
)}
{run && !loading && items.length === 0 && (
{transcript && !loading && items.length === 0 && (
<p className="text-xs italic text-muted-foreground">No messages or tool calls recorded.</p>
)}
{run && !loading && items.length > 0 && (
{transcript && !loading && items.length > 0 && (
<CompactConversation items={items} />
)}
</div>

139
apps/x/apps/renderer/src/components/browser-pane/BrowserPane.tsx
View file

@ -1,7 +1,19 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { ArrowLeft, ArrowRight, Loader2, Plus, RotateCw, X } from 'lucide-react'
import type { HttpAuthRequest } from '@x/shared/dist/browser-control.js'
import { TabBar } from '@/components/tab-bar'
import { Button } from '@/components/ui/button'
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from '@/components/ui/dialog'
import { Input } from '@/components/ui/input'
import { cn } from '@/lib/utils'
/**
@ -86,9 +98,80 @@ const getBrowserTabTitle = (tab: BrowserTabState) => {
}
}
/**
* Credential prompt for HTTP basic/proxy auth challenges raised by pages in
* the embedded browser. Rendered as a regular app dialog: BrowserPane already
* hides the native WebContentsView whenever a dialog overlay is open, so the
* prompt is never obscured by the page that triggered it.
*/
function BrowserHttpAuthDialog({
request,
onSubmit,
onCancel,
}: {
request: HttpAuthRequest
onSubmit: (username: string, password: string) => void
onCancel: () => void
}) {
const [username, setUsername] = useState('')
const [password, setPassword] = useState('')
// Basic auth allows an empty username (token-style `curl -u :TOKEN`), so the
// only invalid submission is fully empty. The server decides the rest.
const canSubmit = username.length > 0 || password.length > 0
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault()
if (!canSubmit) return
onSubmit(username, password)
}
return (
<Dialog open onOpenChange={(open) => { if (!open) onCancel() }}>
<DialogContent className="w-[min(24rem,calc(100%-2rem))] max-w-sm">
<DialogHeader>
<DialogTitle>Sign in</DialogTitle>
<DialogDescription>
{request.isProxy
? `The proxy ${request.host} requires a username and password.`
: `${request.host} requires a username and password.`}
{request.realm ? ` (${request.realm})` : ''}
</DialogDescription>
</DialogHeader>
<form onSubmit={handleSubmit} className="flex flex-col gap-3">
<Input
autoFocus
value={username}
onChange={(e) => setUsername(e.target.value)}
placeholder="Username"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
<Input
type="password"
value={password}
onChange={(e) => setPassword(e.target.value)}
placeholder="Password"
/>
<DialogFooter>
<Button type="button" variant="outline" onClick={onCancel}>
Cancel
</Button>
<Button type="submit" disabled={!canSubmit}>
Sign in
</Button>
</DialogFooter>
</form>
</DialogContent>
</Dialog>
)
}
export function BrowserPane({ onClose, forceHidden = false }: BrowserPaneProps) {
const [state, setState] = useState<BrowserState>(EMPTY_STATE)
const [addressValue, setAddressValue] = useState('')
const [authQueue, setAuthQueue] = useState<HttpAuthRequest[]>([])
const activeTabIdRef = useRef<string | null>(null)
const addressFocusedRef = useRef(false)
@ -121,6 +204,51 @@ export function BrowserPane({ onClose, forceHidden = false }: BrowserPaneProps)
return cleanup
}, [applyState])
// Mirror of authQueue for the unmount handler, which must read the latest
// queue without re-subscribing on every change.
const authQueueRef = useRef<HttpAuthRequest[]>([])
useEffect(() => {
authQueueRef.current = authQueue
}, [authQueue])
useEffect(() => {
const offRequest = window.ipc.on('browser:httpAuthRequest', (incoming) => {
setAuthQueue((queue) => [...queue, incoming as HttpAuthRequest])
})
// Main resolved a challenge on its own (timeout, or its tab/window was
// destroyed) — drop the corresponding dialog so it can't linger over an
// unrelated page with a submit that would no-op.
const offResolved = window.ipc.on('browser:httpAuthResolved', (incoming) => {
const { requestId } = incoming as { requestId: string }
setAuthQueue((queue) => queue.filter((request) => request.requestId !== requestId))
})
return () => {
offRequest()
offResolved()
// Cancel anything still pending so the main-process login callbacks and
// timers are freed immediately instead of waiting out the timeout.
for (const request of authQueueRef.current) {
void window.ipc.invoke('browser:httpAuthResponse', { requestId: request.requestId })
}
}
}, [])
const respondToAuth = useCallback(
(requestId: string, credentials: { username: string; password: string } | null) => {
setAuthQueue((queue) => queue.filter((request) => request.requestId !== requestId))
// Omit username to cancel; include it (even empty) to submit.
void window.ipc.invoke(
'browser:httpAuthResponse',
credentials
? { requestId, username: credentials.username, password: credentials.password }
: { requestId },
)
},
[],
)
const activeAuthRequest = authQueue[0] ?? null
const setViewVisible = useCallback((visible: boolean) => {
if (viewVisibleRef.current === visible) return
viewVisibleRef.current = visible
@ -420,6 +548,17 @@ export function BrowserPane({ onClose, forceHidden = false }: BrowserPaneProps)
className="relative min-h-0 min-w-0 flex-1"
data-browser-viewport
/>
{activeAuthRequest && (
<BrowserHttpAuthDialog
key={activeAuthRequest.requestId}
request={activeAuthRequest}
onSubmit={(username, password) =>
respondToAuth(activeAuthRequest.requestId, { username, password })
}
onCancel={() => respondToAuth(activeAuthRequest.requestId, null)}
/>
)}
</div>
)
}

55
apps/x/apps/renderer/src/components/chat-empty-state.tsx
View file

@ -1,19 +1,9 @@
import { ArrowUpRight, Bot, Mail, MessageSquare, Sparkles, Telescope } from 'lucide-react'
import { Bot, Mail, Sparkles, Telescope } from 'lucide-react'
import { cn } from '@/lib/utils'
import { formatRelativeTime } from '@/lib/relative-time'
import { ToolConnectionsCard } from '@/components/tool-connections-card'
export interface ChatEmptyStateRun {
id: string
title?: string
createdAt: string
}
interface ChatEmptyStateProps {
recentRuns?: ChatEmptyStateRun[]
onSelectRun?: (runId: string) => void
onOpenChatHistory?: () => void
/** Fill the composer with a starter prompt (does not submit). */
onPickPrompt: (prompt: string) => void
/** Use a wider column — for the full-screen chat where the narrow column looks cramped. */
@ -27,13 +17,10 @@ const SUGGESTED_ACTIONS: { icon: typeof Mail; title: string; sub: string; prompt
]
/**
* Empty-state body for the chat surface: greeting, recent chats, and starter
* action cards. Shown in both the side-pane copilot and full-screen chat.
* Empty-state body for the chat surface: greeting and starter action cards.
* Shown in both the side-pane copilot and full-screen chat.
*/
export function ChatEmptyState({
recentRuns = [],
onSelectRun,
onOpenChatHistory,
onPickPrompt,
wide = false,
}: ChatEmptyStateProps) {
@ -45,45 +32,13 @@ export function ChatEmptyState({
</div>
<div>
<div className="text-base font-semibold tracking-tight">What are we working on?</div>
<div className="text-xs text-muted-foreground">Ask anything, or pick up where you left off.</div>
<div className="text-xs text-muted-foreground">Ask anything, or start with a suggestion.</div>
</div>
</div>
{recentRuns.length > 0 && (
<div>
<div className="flex items-center px-1 pb-2 text-[10.5px] font-semibold uppercase tracking-wider text-muted-foreground">
<span className="flex-1">Recent chats</span>
{onOpenChatHistory && (
<button
type="button"
onClick={onOpenChatHistory}
className="inline-flex items-center gap-0.5 text-[11px] font-medium normal-case tracking-normal text-primary hover:underline"
>
View all
<ArrowUpRight className="size-3" />
</button>
)}
</div>
<div className="flex flex-col gap-0.5">
{recentRuns.slice(0, 4).map((run) => (
<button
key={run.id}
type="button"
onClick={() => onSelectRun?.(run.id)}
className="flex items-center gap-2.5 rounded-md px-2.5 py-2 text-left hover:bg-accent"
>
<MessageSquare className="size-3.5 shrink-0 text-muted-foreground" />
<span className="min-w-0 flex-1 truncate text-[13px]">{run.title || '(Untitled chat)'}</span>
<span className="shrink-0 text-[11px] text-muted-foreground">{formatRelativeTime(run.createdAt)}</span>
</button>
))}
</div>
</div>
)}
<div>
<div className="px-1 pb-2 text-[10.5px] font-semibold uppercase tracking-wider text-muted-foreground">
{recentRuns.length > 0 ? 'Or start fresh' : 'Get started'}
Get started
</div>
<div className="flex flex-col gap-2">
{SUGGESTED_ACTIONS.map((action) => (

278
apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx
View file

@ -1,4 +1,4 @@
import { useCallback, useEffect, useLayoutEffect, useRef, useState } from 'react'
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
import { Tooltip, TooltipContent, TooltipTrigger } from '@/components/ui/tooltip'
import {
ArrowUp,
@ -15,16 +15,19 @@ import {
FolderCog,
FolderOpen,
Globe,
Headphones,
ImagePlus,
LoaderIcon,
Lock,
Mic,
MoreHorizontal,
Phone,
PhoneOff,
Plus,
Presentation,
ShieldCheck,
Square,
Terminal,
Video,
X,
} from 'lucide-react'
@ -210,6 +213,18 @@ function compactWorkDirPath(path: string) {
return path.replace(/^\/Users\/[^/]+/, '~')
}
// Call presets: front doors into the same call engine, differing only in
// starting devices. 'share' is the call button's main click — the "work
// together" default (screen shared, camera off, floating pill). The chevron
// menu holds the deviations.
export type CallPreset = 'voice' | 'video' | 'share' | 'practice'
const CALL_PRESET_MENU: Array<{ preset: CallPreset; label: string; description: string; Icon: typeof Phone }> = [
{ preset: 'voice', label: 'Voice call', description: 'Just talk — nothing is shared, the mascot hovers while you work', Icon: AudioLines },
{ preset: 'video', label: 'Video call', description: 'Camera on, face to face — it sees your expressions', Icon: Video },
{ preset: 'practice', label: 'Practice session', description: 'Rehearse a pitch or interview with live coaching', Icon: Presentation },
]
interface ChatInputInnerProps {
onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void
onStop?: () => void
@ -223,18 +238,20 @@ interface ChatInputInnerProps {
onDraftChange?: (text: string) => void
isRecording?: boolean
recordingText?: string
recordingState?: 'connecting' | 'listening'
recordingState?: 'connecting' | 'listening' | 'stopping'
/** Live mic amplitude history (RMS per frame) driving the recording waveform. */
audioLevelsRef?: React.MutableRefObject<number[]>
onStartRecording?: () => void
onSubmitRecording?: () => void
onSubmitRecording?: () => void | Promise<void>
onCancelRecording?: () => void
voiceAvailable?: boolean
ttsAvailable?: boolean
ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void
/** A call is live (hands-free voice loop + spoken responses). */
inCall?: boolean
/** Start a call with the given preset's device defaults. */
onStartCall?: (preset: CallPreset) => void
onEndCall?: () => void
/** Calls need both voice input (STT) and voice output (TTS) configured. */
callAvailable?: boolean
/** Fired when the user picks a different model in the dropdown (only when no run exists yet). */
onSelectedModelChange?: (model: SelectedModel | null) => void
/** Work directory for this chat (per-chat). Null when none is set. */
@ -262,16 +279,16 @@ function ChatInputInner({
onDraftChange,
isRecording,
recordingText,
recordingState,
audioLevelsRef,
onStartRecording,
onSubmitRecording,
onCancelRecording,
voiceAvailable,
ttsAvailable,
ttsEnabled,
ttsMode,
onToggleTts,
onTtsModeChange,
inCall,
onStartCall,
onEndCall,
callAvailable,
onSelectedModelChange,
workDir = null,
onWorkDirChange,
@ -286,6 +303,11 @@ function ChatInputInner({
const [configuredModels, setConfiguredModels] = useState<ConfiguredModel[]>([])
const [activeModelKey, setActiveModelKey] = useState('')
// The effective runtime default (what a run actually uses when the user
// hasn't picked a model) — shown in the picker instead of guessing from
// list order, which can disagree with the real default.
const [defaultModel, setDefaultModel] = useState<ConfiguredModel | null>(null)
const loadModelConfigEpoch = useRef(0)
const [lockedModel, setLockedModel] = useState<SelectedModel | null>(null)
const [searchEnabled, setSearchEnabled] = useState(false)
const [searchAvailable, setSearchAvailable] = useState(false)
@ -341,22 +363,15 @@ function ChatInputInner({
}
})
// When a run exists, freeze the dropdown to the run's resolved model+provider.
// Sessions runtime: model and permission mode are per-message turn config,
// so nothing is frozen for an existing chat — the picker stays live.
useEffect(() => {
if (!runId) {
setLockedModel(null)
setPermissionMode('auto')
return
}
let cancelled = false
window.ipc.invoke('runs:fetch', { runId }).then((run) => {
if (cancelled) return
if (run.provider && run.model) {
setLockedModel({ provider: run.provider, model: run.model })
}
setPermissionMode(run.permissionMode ?? 'manual')
}).catch(() => { /* legacy run or fetch failure — leave unlocked */ })
return () => { cancelled = true }
setLockedModel(null)
}, [runId])
useEffect(() => {
@ -388,6 +403,17 @@ function ChatInputInner({
// Load the list of models the user can choose from.
// Signed-in: gateway model list. Signed-out: providers configured in models.json.
const loadModelConfig = useCallback(async () => {
// Concurrent runs race (mount fires one before the sign-in state resolves,
// which fires another) — only the newest run may write state, else a slow
// stale run can clobber the fresh list with an empty one.
const epoch = ++loadModelConfigEpoch.current
try {
const def = await window.ipc.invoke('llm:getDefaultModel', null)
if (loadModelConfigEpoch.current !== epoch) return
setDefaultModel({ provider: def.provider as ProviderName, model: def.model })
} catch {
if (loadModelConfigEpoch.current === epoch) setDefaultModel(null)
}
try {
if (isRowboatConnected) {
const listResult = await window.ipc.invoke('models:list', null)
@ -397,28 +423,63 @@ function ChatInputInner({
const models: ConfiguredModel[] = (rowboatProvider?.models || []).map(
(m: { id: string }) => ({ provider: 'rowboat', model: m.id })
)
if (loadModelConfigEpoch.current !== epoch) return
setConfiguredModels(models)
} else {
const result = await window.ipc.invoke('workspace:readFile', { path: 'config/models.json' })
const parsed = JSON.parse(result.data)
// Offer every model the configured key supports — the same full catalog
// Settings uses for its dropdowns — so BYOK chat matches the signed-in
// gateway picker. The picker is no longer limited to a hand-curated
// config.models list. Providers with no catalog (Ollama, OpenAI-compatible)
// fall back to the model saved in config.
const catalog: Record<string, string[]> = {}
try {
const listResult = await window.ipc.invoke('models:list', null)
for (const p of listResult.providers || []) {
catalog[p.id] = (p.models || []).map((m: { id: string }) => m.id)
}
} catch { /* offline / no catalog — fall back to saved config below */ }
const models: ConfiguredModel[] = []
if (parsed?.providers) {
for (const [flavor, entry] of Object.entries(parsed.providers)) {
const e = entry as Record<string, unknown>
const modelList: string[] = Array.isArray(e.models) ? e.models as string[] : []
const singleModel = typeof e.model === 'string' ? e.model : ''
const allModels = modelList.length > 0 ? modelList : singleModel ? [singleModel] : []
for (const model of allModels) {
if (model) {
models.push({ provider: flavor as ProviderName, model })
}
}
const seen = new Set<string>()
const push = (provider: string, model: string) => {
if (!model) return
const key = `${provider}/${model}`
if (seen.has(key)) return
seen.add(key)
models.push({ provider: provider as ProviderName, model })
}
// List the default provider first so its default model leads the picker.
const defaultFlavor = typeof parsed?.provider?.flavor === 'string' ? parsed.provider.flavor : ''
const flavors = Object.keys(parsed?.providers || {})
.sort((a, b) => (a === defaultFlavor ? -1 : b === defaultFlavor ? 1 : 0))
for (const flavor of flavors) {
const e = (parsed.providers[flavor] || {}) as Record<string, unknown>
const hasKey = typeof e.apiKey === 'string' && (e.apiKey as string).trim().length > 0
const hasBaseURL = typeof e.baseURL === 'string' && (e.baseURL as string).trim().length > 0
if (!hasKey && !hasBaseURL) continue // provider not configured
// The provider's saved default model leads, then the rest of its catalog.
push(flavor, typeof e.model === 'string' ? e.model : '')
const catalogModels = catalog[flavor] || []
if (catalogModels.length > 0) {
for (const m of catalogModels) push(flavor, m)
} else {
// No catalog (local provider) — fall back to whatever is saved.
const saved = Array.isArray(e.models) ? e.models as string[] : []
for (const m of saved) push(flavor, m)
}
}
if (loadModelConfigEpoch.current !== epoch) return
setConfiguredModels(models)
}
} catch {
// No config yet
} catch (err) {
// No config yet — but surface unexpected failures for diagnosis.
console.error('[chat-input] failed to load model list', err)
}
}, [isRowboatConnected])
@ -605,15 +666,25 @@ function ChatInputInner({
checkSearch()
}, [isActive, isRowboatConnected])
// The dropdown's items: always include the effective default so the picker
// is never empty (and never missing the model that actually runs) even
// while the full list is still loading.
const pickerModels = useMemo<ConfiguredModel[]>(() => {
if (!defaultModel) return configuredModels
const defaultKey = `${defaultModel.provider}/${defaultModel.model}`
if (configuredModels.some((m) => `${m.provider}/${m.model}` === defaultKey)) return configuredModels
return [defaultModel, ...configuredModels]
}, [configuredModels, defaultModel])
// Selecting a model affects only the *next* run created from this tab.
// Once a run exists, model is frozen on the run and the dropdown is read-only.
const handleModelChange = useCallback((key: string) => {
if (lockedModel) return
const entry = configuredModels.find((m) => `${m.provider}/${m.model}` === key)
const entry = pickerModels.find((m) => `${m.provider}/${m.model}` === key)
if (!entry) return
setActiveModelKey(key)
onSelectedModelChange?.({ provider: entry.provider, model: entry.model })
}, [configuredModels, lockedModel, onSelectedModelChange])
}, [pickerModels, lockedModel, onSelectedModelChange])
// Restore the tab draft when this input mounts.
useEffect(() => {
@ -726,7 +797,7 @@ function ChatInputInner({
const currentWorkDirPath = effectiveWorkDir ? compactWorkDirPath(effectiveWorkDir) : ''
return (
<div className="rowboat-chat-input rounded-lg border border-border bg-background shadow-none">
<div data-tour-id="chat-composer" className="rowboat-chat-input rounded-lg border border-border bg-background shadow-none">
{attachments.length > 0 && (
<div className="flex flex-wrap gap-2 px-4 pb-1 pt-3">
{attachments.map((attachment) => {
@ -797,23 +868,33 @@ function ChatInputInner({
>
<X className="h-4 w-4" />
</button>
{/* Audio-reactive waveform only — the transcribed words are intentionally
not shown while recording; they're still captured and submitted. */}
<div className="flex flex-1 items-center overflow-hidden">
<div className="flex min-w-0 flex-1 flex-col gap-1 overflow-hidden">
<VoiceWaveform audioLevelsRef={audioLevelsRef} />
<div
className={cn(
'min-h-5 truncate text-sm leading-5',
recordingText?.trim() ? 'text-foreground' : 'text-muted-foreground'
)}
>
{recordingText?.trim() || (recordingState === 'stopping' ? 'Finalizing...' : 'Listening...')}
</div>
</div>
<Button
size="icon"
onClick={onSubmitRecording}
disabled={!recordingText?.trim()}
disabled={recordingState === 'stopping'}
className={cn(
'h-7 w-7 shrink-0 rounded-full transition-all',
recordingText?.trim()
recordingState !== 'stopping'
? 'bg-primary text-primary-foreground hover:bg-primary/90'
: 'bg-muted text-muted-foreground'
)}
>
<ArrowUp className="h-4 w-4" />
{recordingState === 'stopping' ? (
<LoaderIcon className="h-4 w-4 animate-spin" />
) : (
<ArrowUp className="h-4 w-4" />
)}
</Button>
</div>
) : (
@ -1207,7 +1288,7 @@ function ChatInputInner({
{providerDisplayNames[lockedModel.provider] || lockedModel.provider} — fixed for this chat
</TooltipContent>
</Tooltip>
) : configuredModels.length > 0 ? (
) : pickerModels.length > 0 ? (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<button
@ -1215,14 +1296,22 @@ function ChatInputInner({
className="flex h-7 min-w-0 items-center gap-1 rounded-full px-2 text-xs text-muted-foreground transition-colors hover:bg-muted hover:text-foreground"
>
<span className="min-w-0 truncate">
{getSelectedModelDisplayName(configuredModels.find((m) => `${m.provider}/${m.model}` === activeModelKey)?.model || configuredModels[0]?.model || 'Model')}
{getSelectedModelDisplayName(
pickerModels.find((m) => `${m.provider}/${m.model}` === activeModelKey)?.model
|| defaultModel?.model
|| pickerModels[0]?.model
|| 'Model'
)}
</span>
<ChevronDown className="h-3 w-3 shrink-0" />
</button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuRadioGroup value={activeModelKey} onValueChange={handleModelChange}>
{configuredModels.map((m) => {
<DropdownMenuRadioGroup
value={activeModelKey || (defaultModel ? `${defaultModel.provider}/${defaultModel.model}` : '')}
onValueChange={handleModelChange}
>
{pickerModels.map((m) => {
const key = `${m.provider}/${m.model}`
return (
<DropdownMenuRadioItem key={key} value={key}>
@ -1235,48 +1324,66 @@ function ChatInputInner({
</DropdownMenuContent>
</DropdownMenu>
) : null}
{onToggleTts && ttsAvailable && (
{onStartCall && (
<div className="flex shrink-0 items-center">
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
<TooltipTrigger asChild>
<button
type="button"
onClick={onToggleTts}
onClick={() => {
if (inCall) {
onEndCall?.()
} else if (callAvailable) {
onStartCall('share')
}
}}
className={cn(
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
ttsEnabled
? 'text-foreground hover:bg-muted'
: 'text-muted-foreground hover:bg-muted hover:text-foreground'
'flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
inCall
? 'bg-red-600 text-white hover:bg-red-500'
: callAvailable
? 'text-muted-foreground hover:bg-muted hover:text-foreground'
: 'cursor-default text-muted-foreground/40'
)}
aria-label={ttsEnabled ? 'Disable voice output' : 'Enable voice output'}
aria-label={inCall ? 'End call' : 'Start a call'}
>
<Headphones className="h-4 w-4" />
{!ttsEnabled && (
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
</span>
)}
{inCall ? <PhoneOff className="h-4 w-4" /> : <Phone className="h-4 w-4" />}
</button>
</TooltipTrigger>
<TooltipContent side="top">
{ttsEnabled ? 'Voice output on' : 'Voice output off'}
{inCall
? 'End call'
: callAvailable
? 'Start a call — it sees your screen while you talk it through'
: 'Calls need voice input and output configured'}
</TooltipContent>
</Tooltip>
{ttsEnabled && onTtsModeChange && (
{!inCall && (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<button
type="button"
className="flex h-7 w-4 shrink-0 items-center justify-center text-muted-foreground transition-colors hover:text-foreground"
aria-label="Call options"
>
<ChevronDown className="h-3 w-3" />
</button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuRadioGroup value={ttsMode ?? 'summary'} onValueChange={(v) => onTtsModeChange(v as 'summary' | 'full')}>
<DropdownMenuRadioItem value="summary">Speak summary</DropdownMenuRadioItem>
<DropdownMenuRadioItem value="full">Speak full response</DropdownMenuRadioItem>
</DropdownMenuRadioGroup>
<DropdownMenuContent align="end" className="w-72">
{CALL_PRESET_MENU.map(({ preset, label, description, Icon }) => (
<DropdownMenuItem
key={preset}
disabled={!callAvailable}
onSelect={() => onStartCall(preset)}
className="items-start gap-3 py-2"
>
<Icon className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" />
<span className="min-w-0">
<span className="block text-sm font-medium leading-tight">{label}</span>
<span className="block pt-0.5 text-xs leading-tight text-muted-foreground">{description}</span>
</span>
</DropdownMenuItem>
))}
</DropdownMenuContent>
</DropdownMenu>
)}
@ -1445,17 +1552,16 @@ export interface ChatInputWithMentionsProps {
onDraftChange?: (text: string) => void
isRecording?: boolean
recordingText?: string
recordingState?: 'connecting' | 'listening'
recordingState?: 'connecting' | 'listening' | 'stopping'
audioLevelsRef?: React.MutableRefObject<number[]>
onStartRecording?: () => void
onSubmitRecording?: () => void
onSubmitRecording?: () => void | Promise<void>
onCancelRecording?: () => void
voiceAvailable?: boolean
ttsAvailable?: boolean
ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void
inCall?: boolean
onStartCall?: (preset: CallPreset) => void
onEndCall?: () => void
callAvailable?: boolean
onSelectedModelChange?: (model: SelectedModel | null) => void
workDir?: string | null
onWorkDirChange?: (value: string | null) => void
@ -1485,11 +1591,10 @@ export function ChatInputWithMentions({
onSubmitRecording,
onCancelRecording,
voiceAvailable,
ttsAvailable,
ttsEnabled,
ttsMode,
onToggleTts,
onTtsModeChange,
inCall,
onStartCall,
onEndCall,
callAvailable,
onSelectedModelChange,
workDir,
onWorkDirChange,
@ -1516,11 +1621,10 @@ export function ChatInputWithMentions({
onSubmitRecording={onSubmitRecording}
onCancelRecording={onCancelRecording}
voiceAvailable={voiceAvailable}
ttsAvailable={ttsAvailable}
ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={onToggleTts}
onTtsModeChange={onTtsModeChange}
inCall={inCall}
onStartCall={onStartCall}
onEndCall={onEndCall}
callAvailable={callAvailable}
onSelectedModelChange={onSelectedModelChange}
workDir={workDir}
onWorkDirChange={onWorkDirChange}

21
apps/x/apps/renderer/src/components/chat-message-attachments.tsx
View file

@ -91,11 +91,28 @@ interface ChatMessageAttachmentsProps {
export function ChatMessageAttachments({ attachments, className }: ChatMessageAttachmentsProps) {
if (attachments.length === 0) return null
const imageAttachments = attachments.filter((attachment) => isImageMime(attachment.mimeType))
const fileAttachments = attachments.filter((attachment) => !isImageMime(attachment.mimeType))
const videoFrames = attachments.filter((attachment) => attachment.isVideoFrame)
const imageAttachments = attachments.filter(
(attachment) => !attachment.isVideoFrame && isImageMime(attachment.mimeType)
)
const fileAttachments = attachments.filter(
(attachment) => !attachment.isVideoFrame && !isImageMime(attachment.mimeType)
)
return (
<div className={cn('flex flex-col items-end gap-2', className)}>
{videoFrames.length > 0 && (
<div className="flex max-w-[340px] flex-wrap justify-end gap-1">
{videoFrames.map((frame, index) => (
<img
key={`frame-${index}`}
src={frame.thumbnailUrl}
alt={`Camera frame ${index + 1}`}
className="h-12 w-auto rounded-md border border-border/60 bg-muted object-cover"
/>
))}
</div>
)}
{imageAttachments.length > 0 && (
<div className="flex flex-wrap justify-end gap-2">
{imageAttachments.map((attachment, index) => (

61
apps/x/apps/renderer/src/components/chat-sidebar.tsx
View file

@ -37,7 +37,7 @@ import { MarkdownPreOverride } from '@/components/ai-elements/markdown-code-over
import { defaultRemarkPlugins } from 'streamdown'
import remarkBreaks from 'remark-breaks'
import { type ChatTab } from '@/components/tab-bar'
import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions'
import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions'
import { ChatMessageAttachments } from '@/components/chat-message-attachments'
import { useSidebar } from '@/components/ui/sidebar'
import { wikiLabel } from '@/lib/wiki-links'
@ -51,6 +51,7 @@ import {
getWebSearchCardData,
getComposioConnectCardData,
getToolDisplayName,
getToolErrorText,
groupConversationItems,
isChatMessage,
isErrorMessage,
@ -142,6 +143,10 @@ interface ChatSidebarProps {
chatTabStates?: Record<string, ChatTabViewState>
viewportAnchors?: Record<string, ChatViewportAnchorState>
isProcessing: boolean
// Actively working (sessions runtime). When provided, drives the shimmer
// instead of isProcessing so waiting on a permission/ask-human doesn't
// render a "Thinking…" under the card.
isThinking?: boolean
isStopping?: boolean
onStop?: () => void
onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void
@ -177,17 +182,16 @@ interface ChatSidebarProps {
// Voice / TTS props
isRecording?: boolean
recordingText?: string
recordingState?: 'connecting' | 'listening'
recordingState?: 'connecting' | 'listening' | 'stopping'
audioLevelsRef?: React.MutableRefObject<number[]>
onStartRecording?: () => void
onSubmitRecording?: () => void
onSubmitRecording?: () => void | Promise<void>
onCancelRecording?: () => void
voiceAvailable?: boolean
ttsAvailable?: boolean
ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void
inCall?: boolean
onStartCall?: (preset: CallPreset) => void
onEndCall?: () => void
callAvailable?: boolean
onComposioConnected?: (toolkitSlug: string) => void
}
@ -211,6 +215,7 @@ export function ChatSidebar({
chatTabStates = {},
viewportAnchors = {},
isProcessing,
isThinking,
isStopping,
onStop,
onSubmit,
@ -246,11 +251,10 @@ export function ChatSidebar({
onSubmitRecording,
onCancelRecording,
voiceAvailable,
ttsAvailable,
ttsEnabled,
ttsMode,
onToggleTts,
onTtsModeChange,
inCall,
onStartCall,
onEndCall,
callAvailable,
onComposioConnected,
}: ChatSidebarProps) {
const { state: sidebarState } = useSidebar()
@ -373,7 +377,14 @@ export function ChatSidebar({
}
try {
const result = await window.ipc.invoke('runs:downloadLog', { runId: activeRunId })
// Session-first (new runtime); legacy runs fallback covers old
// background tabs until stage 7 removes the runs runtime.
let result: { success: boolean; error?: string }
try {
result = await window.ipc.invoke('sessions:downloadLog', { sessionId: activeRunId })
} catch {
result = await window.ipc.invoke('runs:downloadLog', { runId: activeRunId })
}
if (result.success) {
toast.success('Chat log saved')
} else if (result.error) {
@ -474,7 +485,7 @@ export function ChatSidebar({
)
}
const toolTitle = getToolDisplayName(item)
const errorText = item.status === 'error' ? 'Tool error' : ''
const errorText = getToolErrorText(item)
const output = normalizeToolOutput(item.result, item.status)
const input = normalizeToolInput(item.input)
return (
@ -674,11 +685,6 @@ export function ChatSidebar({
{!tabHasConversation ? (
<ChatEmptyState
wide={isMaximized}
// A pinned coding-session chat must not offer jumping
// to other conversations from the empty state either.
recentRuns={pinnedToCodeSession ? [] : recentRuns}
onSelectRun={pinnedToCodeSession ? undefined : onSelectRun}
onOpenChatHistory={pinnedToCodeSession ? undefined : onOpenChatHistory}
onPickPrompt={setLocalPresetMessage}
/>
) : (
@ -730,7 +736,7 @@ export function ChatSidebar({
onApproveSession={() => onPermissionResponse(permRequest.toolCall.toolCallId, permRequest.subflow, 'approve', 'session')}
onApproveAlways={() => onPermissionResponse(permRequest.toolCall.toolCallId, permRequest.subflow, 'approve', 'always')}
onDeny={() => onPermissionResponse(permRequest.toolCall.toolCallId, permRequest.subflow, 'deny')}
isProcessing={isActive && isProcessing}
isProcessing={isActive && (isThinking ?? isProcessing)}
response={response}
/>
)}
@ -747,7 +753,7 @@ export function ChatSidebar({
key={request.toolCallId}
query={request.query}
onResponse={(response) => onAskHumanResponse(request.toolCallId, request.subflow, response)}
isProcessing={isActive && isProcessing}
isProcessing={isActive && (isThinking ?? isProcessing)}
/>
))}
@ -759,7 +765,7 @@ export function ChatSidebar({
</Message>
)}
{isActive && isProcessing && !tabState.currentAssistantMessage && (
{isActive && (isThinking ?? isProcessing) && !tabState.currentAssistantMessage && (
<Message from="assistant">
<MessageContent>
<Shimmer duration={1}>Thinking...</Shimmer>
@ -818,11 +824,10 @@ export function ChatSidebar({
onSubmitRecording={isActive ? onSubmitRecording : undefined}
onCancelRecording={isActive ? onCancelRecording : undefined}
voiceAvailable={isActive && voiceAvailable}
ttsAvailable={isActive && ttsAvailable}
ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={isActive ? onToggleTts : undefined}
onTtsModeChange={isActive ? onTtsModeChange : undefined}
inCall={inCall}
onStartCall={isActive ? onStartCall : undefined}
onEndCall={isActive ? onEndCall : undefined}
callAvailable={callAvailable}
/>
</div>
)

58
apps/x/apps/renderer/src/components/code/cm.ts
View file

@ -29,6 +29,12 @@ const darkHighlight = HighlightStyle.define([
])
export function cmBaseExtensions(isDark: boolean): Extension[] {
const bg = isDark ? '#0f1117' : '#ffffff'
const panelBg = isDark ? '#151821' : '#f6f8fa'
const text = isDark ? '#d4d4d8' : '#24292f'
const muted = isDark ? '#7d8590' : '#6e7781'
const border = isDark ? '#2f3542' : '#d0d7de'
return [
lineNumbers(),
bracketMatching(),
@ -39,18 +45,62 @@ export function cmBaseExtensions(isDark: boolean): Extension[] {
EditorView.theme(
{
'&': {
backgroundColor: 'transparent',
backgroundColor: bg,
color: text,
fontSize: '12px',
height: '100%',
},
'.cm-editor': {
backgroundColor: bg,
color: text,
},
'.cm-content': {
caretColor: text,
},
'.cm-scroller': {
fontFamily: 'ui-monospace, SFMono-Regular, Menlo, monospace',
overflow: 'auto',
},
'.cm-line': {
color: text,
},
'.cm-gutters': {
backgroundColor: 'transparent',
border: 'none',
color: isDark ? '#6b7280' : '#9ca3af',
backgroundColor: panelBg,
borderRight: `1px solid ${border}`,
color: muted,
},
'.cm-activeLine': {
backgroundColor: isDark ? 'rgba(110, 118, 129, 0.16)' : 'rgba(175, 184, 193, 0.18)',
},
'.cm-activeLineGutter': {
backgroundColor: isDark ? 'rgba(110, 118, 129, 0.16)' : 'rgba(175, 184, 193, 0.18)',
},
'.cm-selectionBackground, &.cm-focused .cm-selectionBackground, .cm-content ::selection': {
backgroundColor: isDark ? 'rgba(88, 166, 255, 0.32)' : 'rgba(9, 105, 218, 0.22)',
},
'.cm-panels, .cm-panel': {
backgroundColor: panelBg,
color: text,
borderColor: border,
},
'.cm-mergeView': {
backgroundColor: bg,
color: text,
},
'.cm-mergeViewEditors': {
backgroundColor: bg,
},
'.cm-mergeView .cm-editor': {
borderColor: border,
},
'.cm-changedLine': {
backgroundColor: isDark ? 'rgba(56, 139, 253, 0.14)' : 'rgba(9, 105, 218, 0.08)',
},
'.cm-deletedChunk': {
backgroundColor: isDark ? 'rgba(248, 81, 73, 0.14)' : 'rgba(255, 235, 233, 0.95)',
},
'.cm-insertedLine, .cm-insertedChunk': {
backgroundColor: isDark ? 'rgba(63, 185, 80, 0.14)' : 'rgba(234, 255, 234, 0.95)',
},
'&.cm-focused': { outline: 'none' },
// GitHub-style expander bar for folded unchanged regions (@codemirror/merge).

202
apps/x/apps/renderer/src/components/code/code-chat.tsx
View file

@ -1,5 +1,5 @@
import { useEffect, useRef, useState } from 'react'
import { ArrowUp, FileText, Loader2, LoaderIcon, Plus, Square, Terminal, X } from 'lucide-react'
import { useEffect, useRef, useState, type DragEvent, type MutableRefObject } from 'react'
import { ArrowUp, FileText, Loader2, LoaderIcon, Mic, Plus, Square, Terminal, X } from 'lucide-react'
import type { CodeSession, CodeSessionStatus } from '@x/shared/src/code-sessions.js'
import { cn } from '@/lib/utils'
import { toast } from 'sonner'
@ -10,14 +10,59 @@ import { Conversation, ConversationContent, ConversationScrollButton } from '@/c
import { MessageResponse } from '@/components/ai-elements/message'
import { Shimmer } from '@/components/ai-elements/shimmer'
import { Tool, ToolContent, ToolHeader } from '@/components/ai-elements/tool'
import { toToolState, getToolDisplayName, getWebSearchCardData, type ToolCall } from '@/lib/chat-conversation'
import { toToolState, getToolDisplayName, getToolErrorText, getWebSearchCardData, type ToolCall } from '@/lib/chat-conversation'
import { CodeRunPermissionRequest, CodingRunTimeline } from '@/components/coding-run'
import { PermissionRequest } from '@/components/ai-elements/permission-request'
import { AskHumanRequest } from '@/components/ai-elements/ask-human-request'
import { WebSearchResult } from '@/components/ai-elements/web-search-result'
import { useVoiceMode } from '@/hooks/useVoiceMode'
import { useCodeChat, isDirectTurn, isChatToolCall, isChatErrorMessage, type CodeChatItem } from './use-code-chat'
const AGENT_LABEL: Record<string, string> = { claude: 'Claude Code', codex: 'Codex' }
const WAVE_BAR_WIDTH = 3
const WAVE_BAR_GAP = 2
const WAVE_BAR_MIN = 1.5
const WAVE_BAR_MAX = 18
function VoiceWaveform({ audioLevelsRef }: { audioLevelsRef: MutableRefObject<number[]> }) {
const [bars, setBars] = useState<number[]>([])
useEffect(() => {
let raf = 0
let lastSig = ''
const tick = () => {
const levels = audioLevelsRef.current
const next = levels.length > 48 ? levels.slice(levels.length - 48) : levels
const sig = `${next.length}:${next.length ? next[next.length - 1] : 0}`
if (sig !== lastSig) {
lastSig = sig
setBars(next.slice())
}
raf = requestAnimationFrame(tick)
}
raf = requestAnimationFrame(tick)
return () => cancelAnimationFrame(raf)
}, [audioLevelsRef])
return (
<div className="flex h-5 w-full items-center overflow-hidden" style={{ gap: WAVE_BAR_GAP }}>
{bars.map((level, i) => {
const amp = Math.min(1, Math.max(0, level)) ** 0.8
return (
<span
key={i}
className="shrink-0 rounded-full bg-primary"
style={{
width: WAVE_BAR_WIDTH,
height: WAVE_BAR_MIN + amp * (WAVE_BAR_MAX - WAVE_BAR_MIN),
transition: 'height 90ms linear',
}}
/>
)
})}
</div>
)
}
function RowboatToolCall({ item, onOpenDiff }: { item: ToolCall; onOpenDiff: (path: string) => void }) {
const [open, setOpen] = useState(false)
@ -39,7 +84,7 @@ function RowboatToolCall({ item, onOpenDiff }: { item: ToolCall; onOpenDiff: (pa
<Tool open={open || item.status === 'running'} onOpenChange={setOpen}>
<ToolHeader title={AGENT_LABEL[agent ?? ''] ?? 'Coding agent'} type="tool-code_agent_run" state={toToolState(item.status)} />
<ToolContent>
<CodingRunTimeline events={item.codeRunEvents ?? []} onOpenDiff={onOpenDiff} />
<CodingRunTimeline events={item.codeRunEvents ?? []} error={getToolErrorText(item)} onOpenDiff={onOpenDiff} />
</ToolContent>
</Tool>
)
@ -97,20 +142,30 @@ export function CodeChat({
session,
status,
onOpenDiff,
voiceAvailable = false,
}: {
session: CodeSession
status: CodeSessionStatus
onOpenDiff: (path: string) => void
voiceAvailable?: boolean
}) {
const {
items, liveText, isProcessing, pendingPermission, pendingToolPermissions, pendingAskHumans,
items, liveText, isProcessing, compactionStatus, contextUsage,
pendingPermission, pendingToolPermissions, pendingAskHumans,
loading, send, stop, resolvePermission, respondToToolPermission, respondToAskHuman,
} = useCodeChat(session)
const [draft, setDraft] = useState('')
const [stopping, setStopping] = useState(false)
const textareaRef = useRef<HTMLTextAreaElement>(null)
const voice = useVoiceMode()
const voiceWarmup = voice.warmup
const busy = isProcessing || status === 'working' || status === 'needs-you'
const recording = voice.state !== 'idle'
const recordingStopping = voice.state === 'submitting'
const contextUsedPercent = contextUsage
? Math.min(100, Math.round((contextUsage.used / contextUsage.size) * 100))
: null
// Attached file PATHS — like dragging a file into the Claude Code CLI, the
// agent receives paths and reads the files itself with its own tools.
const [attachments, setAttachments] = useState<string[]>([])
@ -119,6 +174,7 @@ export function CodeChat({
setDraft('')
setAttachments([])
setStopping(false)
voice.cancel()
textareaRef.current?.focus()
}, [session.id])
@ -126,6 +182,10 @@ export function CodeChat({
if (!busy) setStopping(false)
}, [busy])
useEffect(() => {
if (voiceAvailable) voiceWarmup()
}, [voiceAvailable, voiceWarmup])
const addAttachments = (paths: string[]) => {
const cleaned = paths.filter(Boolean)
if (cleaned.length === 0) return
@ -141,7 +201,7 @@ export function CodeChat({
textareaRef.current?.focus()
}
const handleDrop = (e: React.DragEvent) => {
const handleDrop = (e: DragEvent) => {
if (!e.dataTransfer?.files?.length) return
e.preventDefault()
const paths = Array.from(e.dataTransfer.files)
@ -175,6 +235,27 @@ export function CodeChat({
await stop()
}
const handleStartRecording = () => {
if (busy) return
void voice.start()
}
const handleSubmitRecording = async () => {
if (!recording || recordingStopping) return
const text = await voice.submit()
if (!text) return
const result = await send(text)
if (!result.ok && result.error) {
toast.error(result.error)
setDraft(text)
}
}
const handleCancelRecording = () => {
voice.cancel()
textareaRef.current?.focus()
}
const basename = (p: string) => p.split(/[\\/]/).pop() || p
return (
@ -188,7 +269,10 @@ export function CodeChat({
<Terminal className="size-3.5 shrink-0 text-muted-foreground" />
<div className="min-w-0 flex-1">
<div className="truncate text-sm font-medium">{session.title}</div>
<div className="text-[11px] text-muted-foreground">{AGENT_LABEL[session.agent]} — direct</div>
<div className="text-[11px] text-muted-foreground">
{AGENT_LABEL[session.agent]} — direct
{contextUsedPercent != null ? ` · ${contextUsedPercent}% context used` : ''}
</div>
</div>
</div>
@ -239,9 +323,19 @@ export function CodeChat({
/>
))}
{busy && !pendingPermission && pendingToolPermissions.size === 0 && pendingAskHumans.size === 0 && (
<Shimmer className="text-sm">
{stopping ? 'Stopping…' : `${AGENT_LABEL[session.agent]} is working…`}
</Shimmer>
compactionStatus === 'stalled' ? (
<div className="text-sm text-amber-600">
Context compaction is taking longer than expected. You can stop and retry in a fresh session.
</div>
) : (
<Shimmer className="text-sm">
{stopping
? 'Stopping…'
: compactionStatus === 'running'
? 'Compacting context…'
: `${AGENT_LABEL[session.agent]} is working…`}
</Shimmer>
)
)}
</ConversationContent>
<ConversationScrollButton />
@ -249,8 +343,8 @@ export function CodeChat({
{/* Composer — mirrors the assistant chat input's look (rounded card,
borderless textarea, round primary send / destructive stop). */}
<div className="p-3">
<div className="rowboat-chat-input mx-auto w-full max-w-3xl rounded-lg border border-border bg-background shadow-none">
<div className="bg-background p-3 dark:bg-black">
<div className="rowboat-chat-input rowboat-code-chat-input mx-auto w-full max-w-3xl rounded-lg border border-border bg-background shadow-none">
{attachments.length > 0 && (
<div className="flex flex-wrap gap-2 px-4 pb-1 pt-3">
{attachments.map((p) => (
@ -273,22 +367,58 @@ export function CodeChat({
))}
</div>
)}
<div className="px-4 pb-2 pt-4">
<Textarea
ref={textareaRef}
value={draft}
onChange={(e) => setDraft(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault()
void handleSend()
}
}}
placeholder="Type your message..."
className="max-h-40 min-h-[24px] w-full resize-none border-0 bg-transparent p-0 text-sm shadow-none outline-none focus-visible:ring-0"
rows={2}
/>
</div>
{recording ? (
<div className="flex items-center gap-3 px-4 py-3">
<button
type="button"
onClick={handleCancelRecording}
className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full text-muted-foreground transition-colors hover:bg-muted hover:text-foreground"
aria-label="Cancel recording"
>
<X className="h-4 w-4" />
</button>
<div className="flex min-w-0 flex-1 flex-col gap-1 overflow-hidden">
<VoiceWaveform audioLevelsRef={voice.audioLevelsRef} />
<div className={cn('min-h-5 truncate text-sm leading-5', voice.interimText.trim() ? 'text-foreground' : 'text-muted-foreground')}>
{voice.interimText.trim() || (recordingStopping ? 'Finalizing...' : 'Listening...')}
</div>
</div>
<Button
size="icon"
onClick={() => void handleSubmitRecording()}
disabled={recordingStopping}
className={cn(
'h-7 w-7 shrink-0 rounded-full transition-all',
recordingStopping
? 'bg-muted text-muted-foreground'
: 'bg-primary text-primary-foreground hover:bg-primary/90',
)}
>
{recordingStopping ? (
<LoaderIcon className="h-4 w-4 animate-spin" />
) : (
<ArrowUp className="h-4 w-4" />
)}
</Button>
</div>
) : (
<div className="px-4 pb-2 pt-4">
<Textarea
ref={textareaRef}
value={draft}
onChange={(e) => setDraft(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault()
void handleSend()
}
}}
placeholder="Type your message..."
className="max-h-40 min-h-[24px] w-full resize-none border-0 bg-transparent p-0 text-sm shadow-none outline-none focus-visible:ring-0"
rows={2}
/>
</div>
)}
<div className="flex items-center gap-2 px-3 pb-3">
<Tooltip>
<TooltipTrigger asChild>
@ -303,6 +433,22 @@ export function CodeChat({
</TooltipTrigger>
<TooltipContent side="top">Attach files — the agent reads them from disk (or drag & drop)</TooltipContent>
</Tooltip>
{voiceAvailable && (
<Tooltip>
<TooltipTrigger asChild>
<button
type="button"
onClick={handleStartRecording}
disabled={busy || recording}
className="flex h-7 w-7 shrink-0 items-center justify-center rounded-full text-muted-foreground transition-colors hover:bg-muted hover:text-foreground disabled:cursor-not-allowed disabled:opacity-50"
aria-label="Voice input"
>
<Mic className="h-4 w-4" />
</button>
</TooltipTrigger>
<TooltipContent side="top">Voice input</TooltipContent>
</Tooltip>
)}
<span className="flex min-w-0 items-center gap-1.5 text-xs text-muted-foreground">
<Terminal className="size-3.5 shrink-0" />
<span className="truncate">Direct — straight to {AGENT_LABEL[session.agent]}</span>

14
apps/x/apps/renderer/src/components/code/code-view.tsx
View file

@ -338,12 +338,14 @@ export function CodeView({
{terminalOpen ? <ChevronDown className="size-3.5" /> : <ChevronUp className="size-3.5" />}
</button>
{terminalOpen && (
<div style={{ height: terminalHeight }}>
<TerminalPane
key={selectedSession.id}
terminalId={selectedSession.id}
cwd={selectedSession.cwd}
/>
<div className="bg-background pb-3 dark:bg-black" style={{ height: terminalHeight + 12 }}>
<div className="h-full min-h-0">
<TerminalPane
key={selectedSession.id}
terminalId={selectedSession.id}
cwd={selectedSession.cwd}
/>
</div>
</div>
)}
</div>

6
apps/x/apps/renderer/src/components/code/new-session-dialog.tsx
View file

@ -78,8 +78,8 @@ export function NewSessionDialog({
}) {
const [agentStatus, setAgentStatus] = useState<{ claude: AgentStatus; codex: AgentStatus } | null>(null)
const [agent, setAgent] = useState<CodingAgent>('claude')
// Rowboat drives by default — direct CLI access is the power-user opt-in.
const [mode, setMode] = useState<CodeSessionMode>('rowboat')
// Direct drive by default; Rowboat orchestration remains an opt-in per session.
const [mode, setMode] = useState<CodeSessionMode>('direct')
const [policy, setPolicy] = useState<ApprovalPolicy>('auto-approve-reads')
const [isolation, setIsolation] = useState<'in-repo' | 'worktree'>('in-repo')
const [title, setTitle] = useState('')
@ -101,7 +101,7 @@ export function NewSessionDialog({
setTitle('')
setCreating(false)
setIsolation('in-repo')
setMode('rowboat')
setMode('direct')
setModelKey('default')
setAgentModel('default')
setAgentEffort('default')

10
apps/x/apps/renderer/src/components/code/terminal-pane.tsx
View file

@ -6,7 +6,7 @@ import { useTheme } from '@/contexts/theme-context'
// xterm color schemes tuned to the app's light/dark backgrounds.
const DARK_THEME = {
background: '#1b1b1f',
background: '#000000',
foreground: '#d4d4d8',
cursor: '#d4d4d8',
selectionBackground: 'rgba(120, 140, 255, 0.3)',
@ -106,5 +106,11 @@ export function TerminalPane({ terminalId, cwd }: { terminalId: string; cwd: str
if (term) term.options.theme = resolvedTheme === 'dark' ? DARK_THEME : LIGHT_THEME
}, [resolvedTheme])
return <div ref={containerRef} className="h-full w-full overflow-hidden px-2 pt-1" />
return (
<div
ref={containerRef}
className="h-full w-full overflow-hidden px-2 pt-1"
style={{ backgroundColor: resolvedTheme === 'dark' ? '#000000' : '#ffffff' }}
/>
)
}

38
apps/x/apps/renderer/src/components/code/use-code-chat.ts
View file

@ -41,6 +41,10 @@ export interface PendingCodePermission {
const DIRECT_PREFIX = 'direct-'
const STRUCTURAL_EVENTS = new Set(['tool_call', 'tool_call_update', 'plan', 'permission'])
const COMPACTION_TITLE = 'Compacting context'
const COMPACTION_STALLED_MS = 90_000
export type CompactionStatus = 'idle' | 'running' | 'stalled'
function messageText(content: unknown): string {
if (typeof content === 'string') return content
@ -62,6 +66,8 @@ export function useCodeChat(session: CodeSession | null) {
const [items, setItems] = useState<CodeChatItem[]>([])
const [liveText, setLiveText] = useState('')
const [isProcessing, setIsProcessing] = useState(false)
const [compactionStatus, setCompactionStatus] = useState<CompactionStatus>('idle')
const [contextUsage, setContextUsage] = useState<{ used: number; size: number } | null>(null)
const [pendingPermission, setPendingPermission] = useState<PendingCodePermission | null>(null)
// Rowboat-mode copilot gates, same as the main chat: pre-tool-call permission
// requests and ask-human questions. Keyed by toolCallId.
@ -69,6 +75,7 @@ export function useCodeChat(session: CodeSession | null) {
const [pendingAskHumans, setPendingAskHumans] = useState<Map<string, z.infer<typeof AskHumanRequestEvent>>>(new Map())
const [loading, setLoading] = useState(false)
const seenMessageIdsRef = useRef<Set<string>>(new Set())
const compactionToolIdRef = useRef<string | null>(null)
const applyCodeRunEvent = useCallback((toolCallId: string, event: CodeRunEvent) => {
if (toolCallId.startsWith(DIRECT_PREFIX)) {
@ -99,6 +106,9 @@ export function useCodeChat(session: CodeSession | null) {
if (!sessionId) {
setItems([])
setLiveText('')
setCompactionStatus('idle')
setContextUsage(null)
compactionToolIdRef.current = null
setPendingPermission(null)
return
}
@ -106,6 +116,9 @@ export function useCodeChat(session: CodeSession | null) {
setLoading(true)
setItems([])
setLiveText('')
setCompactionStatus('idle')
setContextUsage(null)
compactionToolIdRef.current = null
setPendingPermission(null)
setPendingToolPermissions(new Map())
setPendingAskHumans(new Map())
@ -217,6 +230,12 @@ export function useCodeChat(session: CodeSession | null) {
return () => { cancelled = true }
}, [sessionId])
useEffect(() => {
if (compactionStatus !== 'running') return
const timer = window.setTimeout(() => setCompactionStatus('stalled'), COMPACTION_STALLED_MS)
return () => window.clearTimeout(timer)
}, [compactionStatus])
// Live event stream.
useEffect(() => {
if (!sessionId) return
@ -230,6 +249,8 @@ export function useCodeChat(session: CodeSession | null) {
break
case 'run-processing-end':
setIsProcessing(false)
setCompactionStatus('idle')
compactionToolIdRef.current = null
setPendingPermission(null)
// Anything still streaming that never landed as a message (e.g. the
// turn errored) is flushed so the text isn't lost.
@ -247,6 +268,8 @@ export function useCodeChat(session: CodeSession | null) {
break
case 'run-stopped':
setIsProcessing(false)
setCompactionStatus('idle')
compactionToolIdRef.current = null
setPendingPermission(null)
setPendingToolPermissions(new Map())
setPendingAskHumans(new Map())
@ -348,6 +371,19 @@ export function useCodeChat(session: CodeSession | null) {
break
case 'code-run-event': {
setIsProcessing(true)
if (event.event.type === 'usage') {
setContextUsage({ used: event.event.used, size: event.event.size })
}
if (event.event.type === 'tool_call' && event.event.title === COMPACTION_TITLE) {
compactionToolIdRef.current = event.event.id ?? null
setCompactionStatus('running')
}
if (event.event.type === 'tool_call_update'
&& event.event.id != null
&& event.event.id === compactionToolIdRef.current) {
compactionToolIdRef.current = null
setCompactionStatus('idle')
}
if (event.event.type === 'message' && event.event.role === 'agent' && event.toolCallId.startsWith(DIRECT_PREFIX)) {
const text = event.event.text
setLiveText((prev) => prev + text)
@ -460,6 +496,8 @@ export function useCodeChat(session: CodeSession | null) {
items,
liveText,
isProcessing,
compactionStatus,
contextUsage,
pendingPermission,
pendingToolPermissions,
pendingAskHumans,

41
apps/x/apps/renderer/src/components/coding-run.tsx
View file

@ -1,5 +1,6 @@
import { useMemo, useState } from 'react'
import {
AlertCircle,
CheckCircle2,
Circle,
CircleDot,
@ -9,6 +10,7 @@ import {
Pencil,
Search,
ShieldQuestion,
Sparkles,
Terminal,
Trash2,
Wrench,
@ -16,7 +18,7 @@ import {
import type { CodeRunEvent, PermissionAsk, PermissionDecision } from '@x/shared/src/code-mode.js'
import { cn } from '@/lib/utils'
import { Tool, ToolContent, ToolHeader } from '@/components/ai-elements/tool'
import { toToolState, type ToolCall } from '@/lib/chat-conversation'
import { getToolErrorText, toToolState, type ToolCall } from '@/lib/chat-conversation'
// ── Timeline reduction ──────────────────────────────────────────────
// The raw ACP stream is a flat list of events; collapse it into ordered rows,
@ -87,7 +89,8 @@ export function reduceEvents(events: CodeRunEvent[]): Row[] {
return rows
}
function toolKindIcon(kind?: string) {
function toolKindIcon(kind?: string, title?: string) {
if (title === 'Compacting context') return <Sparkles className="size-3.5 shrink-0 text-muted-foreground" />
switch (kind) {
case 'read': return <Eye className="size-3.5 shrink-0 text-muted-foreground" />
case 'edit': return <Pencil className="size-3.5 shrink-0 text-muted-foreground" />
@ -109,14 +112,26 @@ const basename = (p: string) => p.split(/[\\/]/).pop() || p
export function CodingRunTimeline({
events,
error,
onOpenDiff,
}: {
events: CodeRunEvent[]
error?: string
// When set, changed-file names become clickable (the Code section opens the diff).
onOpenDiff?: (path: string) => void
}) {
const rows = useMemo(() => reduceEvents(events), [events])
if (rows.length === 0) {
if (error) {
return (
<div className="px-4 py-3">
<div className="flex items-start gap-2 rounded-md border border-destructive/30 bg-destructive/5 px-3 py-2 text-sm text-destructive">
<AlertCircle className="mt-0.5 size-3.5 shrink-0" />
<span className="min-w-0 whitespace-pre-wrap break-words">{error}</span>
</div>
</div>
)
}
return <div className="px-4 py-3 text-xs text-muted-foreground">Starting the agent…</div>
}
return (
@ -131,13 +146,18 @@ export function CodingRunTimeline({
}
if (row.kind === 'tool') {
const running = row.status !== 'completed' && row.status !== 'failed'
const failed = row.status === 'failed'
return (
<div key={row.id} className="flex flex-col gap-1">
<div className="flex items-center gap-2 text-sm">
{running
? <Loader className="size-3.5 shrink-0 animate-spin text-muted-foreground" />
: <CheckCircle2 className="size-3.5 shrink-0 text-green-600" />}
{toolKindIcon(row.toolKind)}
{running ? (
<Loader className="size-3.5 shrink-0 animate-spin text-muted-foreground" />
) : failed ? (
<AlertCircle className="size-3.5 shrink-0 text-destructive" />
) : (
<CheckCircle2 className="size-3.5 shrink-0 text-green-600" />
)}
{toolKindIcon(row.toolKind, row.title)}
<span className="truncate text-foreground/90">{row.title ?? row.toolKind ?? 'Tool call'}</span>
</div>
{row.diffs.length > 0 && (
@ -187,6 +207,12 @@ export function CodingRunTimeline({
</div>
)
})}
{error && (
<div className="flex items-start gap-2 rounded-md border border-destructive/30 bg-destructive/5 px-3 py-2 text-sm text-destructive">
<AlertCircle className="mt-0.5 size-3.5 shrink-0" />
<span className="min-w-0 whitespace-pre-wrap break-words">{error}</span>
</div>
)}
</div>
)
}
@ -256,12 +282,13 @@ export function CodingRunBlock({
(item.result as { agent?: string } | undefined)?.agent ??
(item.input as { agent?: string } | undefined)?.agent
const title = AGENT_LABEL[agent ?? ''] ?? 'Coding agent'
const error = getToolErrorText(item)
return (
<>
<Tool open={open} onOpenChange={onOpenChange}>
<ToolHeader title={title} type="tool-code_agent_run" state={toToolState(item.status)} />
<ToolContent>
<CodingRunTimeline events={item.codeRunEvents ?? []} />
<CodingRunTimeline events={item.codeRunEvents ?? []} error={error} />
</ToolContent>
</Tool>
{item.pendingCodePermission && (

3
apps/x/apps/renderer/src/components/compact-conversation.tsx
View file

@ -7,6 +7,7 @@ import {
isErrorMessage,
isToolCall,
getToolDisplayName,
getToolErrorText,
toToolState,
normalizeToolOutput,
} from '@/lib/chat-conversation'
@ -62,7 +63,7 @@ function CompactToolRow({ tool }: { tool: ToolCall }) {
const [open, setOpen] = useState(false)
const title = getToolDisplayName(tool)
const state = toToolState(tool.status)
const errorText = tool.status === 'error' && typeof tool.result === 'string' ? tool.result : undefined
const errorText = getToolErrorText(tool)
return (
<Tool open={open} onOpenChange={setOpen} className="mb-0 text-xs">
<ToolHeader title={title} type={`tool-${tool.name}` as `tool-${string}`} state={state} />

1716
apps/x/apps/renderer/src/components/email-view.tsx
View file

File diff suppressed because it is too large Load diff

20
apps/x/apps/renderer/src/components/live-note-sidebar.tsx
View file

@ -11,11 +11,9 @@ import {
ChevronDown, ChevronRight,
} from 'lucide-react'
import { LiveNoteSchema, type LiveNote, type Triggers } from '@x/shared/dist/live-note.js'
import type { Run } from '@x/shared/dist/runs.js'
import type z from 'zod'
import { useLiveNoteAgentStatus } from '@/hooks/use-live-note-agent-status'
import { formatRelativeTime } from '@/lib/relative-time'
import { runLogToConversation } from '@/lib/run-to-conversation'
import { fetchAgentRunTranscript, type AgentRunTranscript } from '@/lib/agent-transcript'
import { CompactConversation } from '@/components/compact-conversation'
export type OpenLiveNotePanelDetail = {
@ -661,7 +659,7 @@ function SectionRegion({ label, children }: { label?: string; children: React.Re
}
function LastRunTab({ live }: { live: LiveNote }) {
const [run, setRun] = useState<z.infer<typeof Run> | null>(null)
const [transcript, setTranscript] = useState<AgentRunTranscript | null>(null)
const [loadingRun, setLoadingRun] = useState(false)
const [fetchError, setFetchError] = useState<string | null>(null)
@ -669,7 +667,7 @@ function LastRunTab({ live }: { live: LiveNote }) {
useEffect(() => {
if (!runId) {
setRun(null)
setTranscript(null)
setFetchError(null)
setLoadingRun(false)
return
@ -679,13 +677,13 @@ function LastRunTab({ live }: { live: LiveNote }) {
setFetchError(null)
void (async () => {
try {
const r = await window.ipc.invoke('runs:fetch', { runId })
const t = await fetchAgentRunTranscript(runId)
if (cancelled) return
setRun(r)
setTranscript(t)
} catch (err) {
if (cancelled) return
setFetchError(err instanceof Error ? err.message : String(err))
setRun(null)
setTranscript(null)
} finally {
if (!cancelled) setLoadingRun(false)
}
@ -704,7 +702,7 @@ function LastRunTab({ live }: { live: LiveNote }) {
}
const isError = !!live.lastRunError
const items = run ? runLogToConversation(run.log) : []
const items = transcript?.items ?? []
return (
<div className="flex-1 overflow-auto px-4 py-4 space-y-4">
@ -751,10 +749,10 @@ function LastRunTab({ live }: { live: LiveNote }) {
Couldn't load transcript: {fetchError}
</div>
)}
{run && !loadingRun && items.length === 0 && (
{transcript && !loadingRun && items.length === 0 && (
<p className="text-xs italic text-muted-foreground">No messages or tool calls recorded.</p>
)}
{run && !loadingRun && items.length > 0 && (
{transcript && !loadingRun && items.length > 0 && (
<CompactConversation items={items} />
)}
</div>

271
apps/x/apps/renderer/src/components/meetings-view.tsx
View file

@ -1,6 +1,7 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import { createPortal } from 'react-dom'
import { Calendar, ChevronDown, Clock, ExternalLink, Loader2, MapPin, Mic, Square, UserRound, UsersRound, Video, X } from 'lucide-react'
import { Calendar, ChevronDown, ChevronRight, Clock, ExternalLink, FileText, Loader2, MapPin, Mic, Sparkles, Square, UserPlus, UserRound, UsersRound, Video, X } from 'lucide-react'
import { Streamdown } from 'streamdown'
import { Button } from '@/components/ui/button'
import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover'
@ -14,6 +15,39 @@ const MEETINGS_ROOT = 'knowledge/Meetings'
const CALENDAR_DIR = 'calendar_sync'
const UPCOMING_MAX_DAYS = 4 // today + next 3
declare global {
interface Window {
__pendingMeetingPrepCreate?: { prompt: string }
}
}
// Mirrors the `meeting-prep:resolve` IPC response shape.
type PrepNote = {
path: string
name: string
role?: string
organization?: string
markdown: string
}
type PrepAttendee = {
label: string
email?: string
displayName?: string
note: PrepNote | null
}
type PrepOrg = {
path: string
name: string
markdown: string
}
type PrepResult = {
attendees: PrepAttendee[]
organizations: PrepOrg[]
prepNote: { path: string; brief: string } | null
matchedCount: number
unmatchedCount: number
}
type MeetingNoteRow = {
path: string
name: string
@ -358,7 +392,178 @@ function parseDescriptionParts(value: string): DescriptionPart[] {
}).filter((part) => part.text.length > 0)
}
function UpcomingEvents() {
// Hand the unmatched attendee off to the Copilot to research + create a note.
function requestCreateNote(attendee: PrepAttendee, meetingSummary: string) {
const who = attendee.displayName || attendee.label
const email = attendee.email ? ` <${attendee.email}>` : ''
window.__pendingMeetingPrepCreate = {
prompt: `Create a person note in my knowledge base for ${who}${email}. They're attending my "${meetingSummary}" meeting. Pull together what you know about them from my emails, past meetings, and calendar.`,
}
window.dispatchEvent(new Event('meeting-prep:create-note'))
}
// One note row (used for both people and organizations): a clickable row that
// navigates to the note. The markdown is NOT rendered inline in the card.
function PrepNoteRow({ title, subtitle, path, onOpenNote }: {
title: string
subtitle?: string
path: string
onOpenNote: (path: string) => void
}) {
return (
<button
type="button"
onClick={() => onOpenNote(path)}
title={`Open ${title}`}
className="flex w-full items-center gap-3 border-b px-5 py-1.5 text-left transition-colors last:border-b-0 hover:bg-muted/50"
>
<span className="flex min-w-0 flex-1 flex-col">
<span className="truncate text-[12.5px] font-semibold text-foreground">{title}</span>
{subtitle ? <span className="truncate text-[11px] text-muted-foreground">{subtitle}</span> : null}
</span>
<ChevronRight className="size-3.5 shrink-0 text-muted-foreground" />
</button>
)
}
function PrepAttendeeNote({ attendee, onOpenNote }: { attendee: PrepAttendee; onOpenNote: (path: string) => void }) {
const note = attendee.note
if (!note) return null
const subtitle = [note.role, note.organization].filter(Boolean).join(' · ')
return <PrepNoteRow title={note.name} subtitle={subtitle || undefined} path={note.path} onOpenNote={onOpenNote} />
}
function PrepUnmatchedSection({ attendees, meetingSummary }: { attendees: PrepAttendee[]; meetingSummary: string }) {
const [open, setOpen] = useState(false)
if (attendees.length === 0) return null
return (
<div className="border-t bg-muted/20">
<button
type="button"
onClick={() => setOpen((v) => !v)}
className="flex w-full items-center gap-3 px-5 py-2.5 text-left transition-colors hover:bg-muted/40"
>
{open ? <ChevronDown className="size-4 shrink-0 text-muted-foreground" /> : <ChevronRight className="size-4 shrink-0 text-muted-foreground" />}
<span className="text-xs font-medium text-muted-foreground">
{attendees.length} {attendees.length === 1 ? 'other' : 'others'} — no notes yet
</span>
</button>
{open ? (
<div className="flex flex-col gap-1 px-5 pb-3 pl-12">
{attendees.map((att, idx) => (
<div key={`${att.email ?? att.label}-${idx}`} className="flex items-center justify-between gap-3 py-1">
<span className="min-w-0 truncate text-sm text-foreground">{att.label}</span>
<button
type="button"
onClick={() => requestCreateNote(att, meetingSummary)}
className="inline-flex shrink-0 items-center gap-1.5 rounded-md border bg-background px-2 py-1 text-xs font-medium text-foreground transition-colors hover:bg-accent"
>
<UserPlus className="size-3.5" />
Create note
</button>
</div>
))}
</div>
) : null}
</div>
)
}
// Inline prep for a single event: resolves the attendees against the knowledge
// base and renders their notes directly beneath the event row. Re-resolves when
// a person note changes (e.g. after "Create note") so it stays fresh.
function InlineMeetingPrep({ event, onOpenNote }: { event: UpcomingEvent; onOpenNote: (path: string) => void }) {
const [prep, setPrep] = useState<PrepResult | null>(null)
const [refreshTick, setRefreshTick] = useState(0)
useEffect(() => {
let cancelled = false
const run = async () => {
try {
const attendees = event.attendees.map((a) => ({ email: a.email, displayName: a.displayName, self: a.self }))
const result = await window.ipc.invoke('meeting-prep:resolve', { attendees, eventId: event.id })
if (!cancelled) setPrep(result)
} catch (err) {
console.error('Meeting prep failed:', err)
if (!cancelled) setPrep(null)
}
}
void run()
return () => {
cancelled = true
}
}, [event.id, refreshTick])
// Refresh when a People note is created/changed so newly-created notes appear.
useEffect(() => {
const isPeoplePath = (p: string | undefined) =>
typeof p === 'string' && p.startsWith('knowledge/People/')
const cleanup = window.ipc.on('workspace:didChange', (e) => {
switch (e.type) {
case 'created':
case 'changed':
case 'deleted':
if (isPeoplePath(e.path)) setRefreshTick((t) => t + 1)
break
case 'moved':
if (isPeoplePath(e.from) || isPeoplePath(e.to)) setRefreshTick((t) => t + 1)
break
case 'bulkChanged':
if (!e.paths || e.paths.some(isPeoplePath)) setRefreshTick((t) => t + 1)
break
}
})
return cleanup
}, [])
if (!prep || prep.attendees.length === 0) return null
const matched = prep.attendees.filter((a) => a.note)
const unmatched = prep.attendees.filter((a) => !a.note)
return (
<div className="bg-muted/10">
{prep.prepNote && prep.prepNote.brief ? (
<div className="border-b px-5 pb-3 pt-3">
<Streamdown className="prose prose-sm dark:prose-invert max-w-none text-foreground/90 [&>*:first-child]:mt-0 [&>*:last-child]:mb-0 [&_p]:my-1 [&_ul]:my-1 [&_ol]:my-1 [&_li]:my-0.5 [&_p]:text-[12.5px] [&_li]:text-[12.5px]">
{prep.prepNote.brief}
</Streamdown>
<button
type="button"
onClick={() => onOpenNote(prep.prepNote!.path)}
className="mt-2 inline-flex items-center gap-1.5 text-xs font-medium text-muted-foreground transition-colors hover:text-foreground"
>
<FileText className="size-3.5" />
Open full prep
</button>
</div>
) : null}
<div className="flex items-center gap-1.5 px-5 pb-1 pt-2.5">
<UsersRound className="size-3.5 text-muted-foreground" />
<span className="text-[11px] font-medium uppercase tracking-wider text-muted-foreground">People</span>
</div>
{matched.map((att, idx) => (
<PrepAttendeeNote key={att.note!.path + idx} attendee={att} onOpenNote={onOpenNote} />
))}
<PrepUnmatchedSection attendees={unmatched} meetingSummary={event.summary} />
{prep.organizations.length > 0 ? (
<>
<div className="px-5 pb-1 pt-2.5">
<span className="text-[11px] font-medium uppercase tracking-wider text-muted-foreground">
{prep.organizations.length === 1 ? 'Company' : 'Companies'}
</span>
</div>
{prep.organizations.map((org) => (
<PrepNoteRow key={org.path} title={org.name} subtitle="Organization" path={org.path} onOpenNote={onOpenNote} />
))}
</>
) : null}
</div>
)
}
function UpcomingEvents({ onOpenNote }: { onOpenNote: (path: string) => void }) {
const [events, setEvents] = useState<UpcomingEvent[]>([])
const [loading, setLoading] = useState(true)
const [error, setError] = useState<string | null>(null)
@ -487,6 +692,20 @@ function UpcomingEvents() {
return selectVisibleDays(window)
}, [events])
// The next meeting that's worth prepping for — soonest timed event with at
// least one other attendee that hasn't ended. Its row gets inline prep.
// `events` is sorted (all-day first, then by start), so `find` returns it.
const prepEventId = useMemo(() => {
const nowMs = Date.now()
const candidate = events.find((ev) => {
if (ev.isAllDay) return false
if (ev.attendees.every((a) => a.self)) return false
const endMs = ev.end ? ev.end.getTime() : ev.start.getTime() + 30 * 60 * 1000
return endMs > nowMs
})
return candidate?.id ?? null
}, [events])
const totalVisible = visibleDays.reduce((s, d) => s + d.events.length, 0)
const now = new Date()
const todayKey = localDateKey(now)
@ -532,6 +751,8 @@ function UpcomingEvents() {
key={day.dateKey}
day={day}
isToday={day.dateKey === todayKey}
prepEventId={prepEventId}
onOpenNote={onOpenNote}
/>
))}
</div>
@ -542,7 +763,7 @@ function UpcomingEvents() {
)
}
function UpcomingDayCard({ day, isToday }: { day: DayGroup; isToday: boolean }) {
function UpcomingDayCard({ day, isToday, prepEventId, onOpenNote }: { day: DayGroup; isToday: boolean; prepEventId: string | null; onOpenNote: (path: string) => void }) {
const dayNum = day.date.getDate()
const month = day.date.toLocaleDateString([], { month: 'short' })
const weekday = day.date.toLocaleDateString([], { weekday: 'short' })
@ -573,7 +794,13 @@ function UpcomingDayCard({ day, isToday }: { day: DayGroup; isToday: boolean })
</div>
) : (
day.events.map((ev, idx) => (
<UpcomingEventItem key={ev.id} event={ev} isLast={idx === count - 1} />
<UpcomingEventItem
key={ev.id}
event={ev}
isLast={idx === count - 1}
isPrepTarget={ev.id === prepEventId}
onOpenNote={onOpenNote}
/>
))
)}
</div>
@ -588,14 +815,20 @@ function NowBadge() {
)
}
function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: boolean }) {
function UpcomingEventItem({ event, isLast, isPrepTarget, onOpenNote }: { event: UpcomingEvent; isLast: boolean; isPrepTarget: boolean; onOpenNote: (path: string) => void }) {
const [open, setOpen] = useState(false)
// The next meeting auto-expands its prep; any other meeting with attendees
// can be expanded on demand via the Prep toggle (resolves lazily on open).
const prepEligible = !event.isAllDay && event.attendees.some((a) => !a.self)
const [prepOpen, setPrepOpen] = useState(isPrepTarget)
const showPrep = prepEligible && prepOpen
const isNow = isEventNow(event)
const platform = meetingPlatformLabel(event.conferenceLink)
const subtitle = platform ?? event.location
const titleAndLocation = event.location ? `${event.summary} · ${event.location}` : event.summary
return (
<div className={cn(!isLast && 'border-b')}>
<Popover open={open} onOpenChange={setOpen}>
<PopoverTrigger asChild>
<div
@ -604,7 +837,7 @@ function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: bo
title={titleAndLocation}
className={cn(
'group flex w-full cursor-pointer items-center gap-4 px-5 py-3 text-left transition-colors',
!isLast && 'border-b',
showPrep && 'border-b',
isNow ? 'bg-muted' : 'hover:bg-muted/50',
)}
>
@ -625,7 +858,24 @@ function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: bo
</span>
) : null}
</span>
<div className="shrink-0">
<div className="flex shrink-0 items-center gap-2">
{prepEligible ? (
<button
type="button"
onClick={(e) => { e.stopPropagation(); setPrepOpen((v) => !v) }}
onMouseDown={(e) => e.stopPropagation()}
aria-expanded={prepOpen}
title={prepOpen ? 'Hide prep' : 'Show meeting prep'}
className={cn(
'inline-flex items-center gap-1.5 rounded-md border px-2.5 py-1.5 text-xs font-medium transition-colors',
prepOpen ? 'bg-accent text-foreground' : 'bg-background text-foreground hover:bg-accent',
)}
>
<Sparkles className="size-3.5" />
Prep
<ChevronDown className={cn('size-3 transition-transform', prepOpen && 'rotate-180')} />
</button>
) : null}
{event.conferenceLink ? (
<SplitJoinButton
onJoinAndNotes={() => triggerMeetingCapture(event, true)}
@ -647,6 +897,8 @@ function UpcomingEventItem({ event, isLast }: { event: UpcomingEvent; isLast: bo
</PopoverTrigger>
<EventDetailsPopover event={event} onClose={() => setOpen(false)} />
</Popover>
{showPrep ? <InlineMeetingPrep event={event} onOpenNote={onOpenNote} /> : null}
</div>
)
}
@ -917,6 +1169,9 @@ export function MeetingsView({ onOpenNote, onTakeMeetingNotes, meetingState, mee
const rows = entries
.filter((entry) => entry.kind === 'file' && entry.name.endsWith('.md'))
// Generated prep notes live under Meetings/prep/ — they're upcoming
// prep, not past meeting notes, so keep them out of this table.
.filter((entry) => !entry.path.startsWith(`${MEETINGS_ROOT}/prep/`))
.map((entry) => {
const relative = entry.path.slice(`${MEETINGS_ROOT}/`.length)
const parts = relative.split('/')
@ -1017,7 +1272,7 @@ export function MeetingsView({ onOpenNote, onTakeMeetingNotes, meetingState, mee
</p>
</div>
<div className="flex-1 overflow-auto">
<UpcomingEvents />
<UpcomingEvents onOpenNote={onOpenNote} />
<div className="p-6">
{loading ? (
<div className="flex items-center justify-center py-10">

4
apps/x/apps/renderer/src/components/onboarding-modal.tsx
View file

@ -183,8 +183,8 @@ export function OnboardingModal({ open, onComplete }: OnboardingModalProps) {
// Preferred default models for each provider
const preferredDefaults: Partial<Record<LlmProviderFlavor, string>> = {
openai: "gpt-5.2",
anthropic: "claude-opus-4-6-20260202",
openai: "gpt-5.4",
anthropic: "claude-opus-4-8",
}
// Initialize default models from catalog

3
apps/x/apps/renderer/src/components/onboarding/index.tsx
View file

@ -14,6 +14,7 @@ import { StepIndicator } from "./step-indicator"
import { WelcomeStep } from "./steps/welcome-step"
import { LlmSetupStep } from "./steps/llm-setup-step"
import { ConnectAccountsStep } from "./steps/connect-accounts-step"
import { CodeModeStep } from "./steps/code-mode-step"
import { CompletionStep } from "./steps/completion-step"
interface OnboardingModalProps {
@ -33,6 +34,8 @@ export function OnboardingModal({ open, onComplete }: OnboardingModalProps) {
case 2:
return <ConnectAccountsStep state={state} />
case 3:
return <CodeModeStep state={state} />
case 4:
return <CompletionStep state={state} />
}
}, [state.currentStep, state])

12
apps/x/apps/renderer/src/components/onboarding/step-indicator.tsx
View file

@ -6,14 +6,16 @@ import type { Step, OnboardingPath } from "./use-onboarding-state"
const ROWBOAT_STEPS = [
{ step: 0 as Step, label: "Welcome" },
{ step: 2 as Step, label: "Connect" },
{ step: 3 as Step, label: "Done" },
{ step: 3 as Step, label: "Code" },
{ step: 4 as Step, label: "Done" },
]
const BYOK_STEPS = [
{ step: 0 as Step, label: "Welcome" },
{ step: 1 as Step, label: "Model" },
{ step: 2 as Step, label: "Connect" },
{ step: 3 as Step, label: "Done" },
{ step: 3 as Step, label: "Code" },
{ step: 4 as Step, label: "Done" },
]
interface StepIndicatorProps {
@ -26,7 +28,7 @@ export function StepIndicator({ currentStep, path }: StepIndicatorProps) {
const currentIndex = steps.findIndex(s => s.step === currentStep)
return (
<div className="flex items-center gap-2 mb-8 px-4">
<div className="flex items-center gap-2 mb-20 px-4">
{steps.map((s, i) => (
<React.Fragment key={s.step}>
{i > 0 && (
@ -37,7 +39,7 @@ export function StepIndicator({ currentStep, path }: StepIndicatorProps) {
)}
/>
)}
<div className="flex flex-col items-center gap-1.5">
<div className="relative flex flex-col items-center">
<div
className={cn(
"size-8 rounded-full flex items-center justify-center text-xs font-medium transition-all duration-300",
@ -54,7 +56,7 @@ export function StepIndicator({ currentStep, path }: StepIndicatorProps) {
</div>
<span
className={cn(
"text-[11px] font-medium transition-colors duration-300",
"absolute top-full mt-1.5 whitespace-nowrap text-[11px] font-medium transition-colors duration-300",
i <= currentIndex ? "text-foreground" : "text-muted-foreground"
)}
>

147
apps/x/apps/renderer/src/components/onboarding/steps/code-mode-step.tsx Normal file
View file

@ -0,0 +1,147 @@
import { useState, useEffect, useCallback } from "react"
import { Loader2, ArrowLeft, Terminal, CheckCircle2 } from "lucide-react"
import { Button } from "@/components/ui/button"
import { Switch } from "@/components/ui/switch"
import { startProvisioning, type CodeModeAgentStatus } from "@/lib/code-mode-provisioning"
import type { OnboardingState } from "../use-onboarding-state"
interface CodeModeStepProps {
state: OnboardingState
}
const AGENTS = [
{ key: "claude" as const, name: "Claude Code" },
{ key: "codex" as const, name: "Codex" },
]
export function CodeModeStep({ state }: CodeModeStepProps) {
const { handleNext, handleBack } = state
const [enabled, setEnabled] = useState(false)
const [selected, setSelected] = useState<Record<"claude" | "codex", boolean>>({ claude: false, codex: false })
const [status, setStatus] = useState<CodeModeAgentStatus | null>(null)
const [statusLoading, setStatusLoading] = useState(true)
const [saving, setSaving] = useState(false)
// Reflect what's already set up: pre-select installed agents and turn the master
// switch on if any agent is already there, so returning users don't start from off.
useEffect(() => {
let cancelled = false
;(async () => {
setStatusLoading(true)
try {
const result = await window.ipc.invoke("codeMode:checkAgentStatus", null)
if (cancelled) return
setStatus(result)
const claudeInstalled = result.claude.installed
const codexInstalled = result.codex.installed
if (claudeInstalled || codexInstalled) {
setEnabled(true)
setSelected({ claude: claudeInstalled, codex: codexInstalled })
}
} catch {
if (!cancelled) setStatus(null)
} finally {
if (!cancelled) setStatusLoading(false)
}
})()
return () => { cancelled = true }
}, [])
const onContinue = useCallback(async () => {
if (enabled) {
setSaving(true)
try {
await window.ipc.invoke("codeMode:setConfig", { enabled: true, approvalPolicy: "ask" })
window.dispatchEvent(new Event("code-mode-config-changed"))
} catch {
// Non-fatal — the user can still enable code mode later from Settings.
}
setSaving(false)
// Kick off engine downloads in the BACKGROUND for selected agents that aren't
// installed yet. We deliberately don't block onboarding on the ~200 MB download —
// it keeps running in the main process and its progress shows in Settings → Code Mode.
for (const a of AGENTS) {
if (selected[a.key] && !status?.[a.key].installed) {
startProvisioning(a.key, () => {})
}
}
}
handleNext()
}, [enabled, selected, status, handleNext])
return (
<div className="flex flex-col flex-1">
{/* Title */}
<h2 className="text-3xl font-bold tracking-tight text-center mb-2">
Set Up Code Mode
</h2>
<p className="text-base text-muted-foreground text-center leading-relaxed mb-6 max-w-md mx-auto">
Use your existing Claude Code or Codex subscription inside Rowboat to tackle coding
tasks and unlock far more workflows, all without leaving the app. Make sure Claude Code
and Codex are signed in locally with{" "}
<code className="rounded bg-muted px-1 py-0.5 font-mono text-[13px] text-foreground">claude&nbsp;login</code> or{" "}
<code className="rounded bg-muted px-1 py-0.5 font-mono text-[13px] text-foreground">codex&nbsp;login</code> in your terminal.
</p>
{statusLoading ? (
<div className="flex items-center justify-center py-12">
<Loader2 className="size-6 animate-spin text-muted-foreground" />
</div>
) : (
<div className="space-y-4">
{/* Master enable */}
<div className="rounded-xl border px-4 py-3.5 flex items-start gap-3">
<div className="flex-1 min-w-0">
<div className="text-sm font-medium">Enable code mode</div>
<div className="text-xs text-muted-foreground mt-0.5">
Shows the code mode chip in the composer and lets the assistant delegate to your agents.
</div>
</div>
<Switch checked={enabled} onCheckedChange={setEnabled} disabled={saving} />
</div>
{/* Per-agent selection (revealed when enabled) */}
{enabled && (
<div className="space-y-2">
<span className="text-xs font-semibold text-muted-foreground uppercase tracking-wider">
Agents to set up
</span>
{AGENTS.map((a) => {
const st = status?.[a.key]
const ready = (st?.installed ?? false) && (st?.signedIn ?? false)
return (
<div key={a.key} className="rounded-xl border px-4 py-3 flex items-center gap-3">
<Terminal className="size-4 text-muted-foreground shrink-0" />
<div className="flex-1 min-w-0 text-sm font-medium">{a.name}</div>
{ready && <CheckCircle2 className="size-4 text-green-600 shrink-0" />}
<Switch
checked={selected[a.key]}
onCheckedChange={(v) => setSelected((prev) => ({ ...prev, [a.key]: v }))}
/>
</div>
)
})}
</div>
)}
</div>
)}
{/* Footer */}
<div className="flex flex-col gap-3 mt-8 pt-4 border-t">
<Button onClick={onContinue} size="lg" className="h-12 text-base font-medium" disabled={saving}>
{saving ? <Loader2 className="size-5 animate-spin" /> : "Continue"}
</Button>
<div className="flex items-center justify-between">
<Button variant="ghost" onClick={handleBack} className="gap-1">
<ArrowLeft className="size-4" />
Back
</Button>
<Button variant="ghost" onClick={handleNext} className="text-muted-foreground">
Skip for now
</Button>
</div>
</div>
</div>
)
}

37
apps/x/apps/renderer/src/components/onboarding/steps/completion-step.tsx
View file

@ -13,34 +13,25 @@ export function CompletionStep({ state }: CompletionStepProps) {
return (
<div className="flex flex-col items-center justify-center text-center flex-1">
{/* Animated checkmark */}
{/* Title with checkmark on the right */}
<motion.div
initial={{ scale: 0 }}
animate={{ scale: 1 }}
transition={{ type: "spring", stiffness: 260, damping: 20, delay: 0.1 }}
className="relative mb-8"
>
{/* Pulsing ring */}
<motion.div
initial={{ scale: 0.8, opacity: 0.6 }}
animate={{ scale: 1.5, opacity: 0 }}
transition={{ duration: 1.2, repeat: 2, ease: "easeOut" }}
className="absolute inset-0 rounded-full bg-green-500/20"
/>
<div className="relative size-20 rounded-full bg-green-100 dark:bg-green-900/30 flex items-center justify-center">
<CheckCircle2 className="size-10 text-green-600 dark:text-green-400" />
</div>
</motion.div>
{/* Title */}
<motion.h2
initial={{ opacity: 0, y: 8 }}
animate={{ opacity: 1, y: 0 }}
transition={{ delay: 0.25 }}
className="text-3xl font-bold tracking-tight mb-3"
className="flex items-center gap-3 mb-3"
>
You're All Set!
</motion.h2>
<h2 className="text-3xl font-bold tracking-tight">
You're All Set!
</h2>
<motion.span
initial={{ scale: 0 }}
animate={{ scale: 1 }}
transition={{ type: "spring", stiffness: 260, damping: 20, delay: 0.35 }}
className="shrink-0"
>
<CheckCircle2 className="size-9 text-green-600 dark:text-green-400" />
</motion.span>
</motion.div>
<motion.p
initial={{ opacity: 0 }}

32
apps/x/apps/renderer/src/components/onboarding/steps/welcome-step.tsx
View file

@ -12,15 +12,21 @@ export function WelcomeStep({ state }: WelcomeStepProps) {
return (
<div className="flex flex-col items-center justify-center text-center flex-1">
{/* Logo with ambient glow */}
{/* Logo + main heading on the same level */}
<motion.div
initial={{ opacity: 0, scale: 0.8 }}
animate={{ opacity: 1, scale: 1 }}
transition={{ duration: 0.5, ease: [0.16, 1, 0.3, 1] }}
className="relative mb-8"
initial={{ opacity: 0, y: 8 }}
animate={{ opacity: 1, y: 0 }}
transition={{ delay: 0.2 }}
className="flex items-center gap-4 mb-4"
>
<div className="absolute inset-0 size-16 rounded-2xl bg-primary/10 blur-xl scale-[2.5]" />
<img src="/logo-only.png" alt="Rowboat" className="relative size-16" />
<h1 className="text-3xl font-bold tracking-tight">
Welcome to Rowboat
</h1>
{/* Logo with ambient glow */}
<div className="relative shrink-0">
<div className="absolute inset-0 size-12 rounded-2xl bg-primary/10 blur-xl scale-[2.5]" />
<img src="/logo-only.png" alt="Rowboat" className="relative size-12" />
</div>
</motion.div>
{/* Tagline badge */}
@ -28,21 +34,11 @@ export function WelcomeStep({ state }: WelcomeStepProps) {
initial={{ opacity: 0, y: 6 }}
animate={{ opacity: 1, y: 0 }}
transition={{ delay: 0.15 }}
className="inline-flex items-center gap-2 rounded-full border bg-muted/50 px-3.5 py-1.5 text-xs font-medium text-muted-foreground mb-6"
className="inline-flex items-center gap-2 rounded-full border bg-muted/50 px-3.5 py-1.5 text-xs font-medium text-muted-foreground mb-10"
>
<span className="size-1.5 rounded-full bg-green-500 animate-pulse" />
Your AI coworker, with memory
</motion.div>
{/* Main heading */}
<motion.h1
initial={{ opacity: 0, y: 8 }}
animate={{ opacity: 1, y: 0 }}
transition={{ delay: 0.2 }}
className="text-3xl font-bold tracking-tight mb-3"
>
Welcome to Rowboat
</motion.h1>
<motion.p
initial={{ opacity: 0 }}
animate={{ opacity: 1 }}

26
apps/x/apps/renderer/src/components/onboarding/use-onboarding-state.ts
View file

@ -8,7 +8,7 @@ export interface ProviderState {
isConnecting: boolean
}
export type Step = 0 | 1 | 2 | 3
export type Step = 0 | 1 | 2 | 3 | 4
export type OnboardingPath = 'rowboat' | 'byok' | null
@ -155,8 +155,8 @@ export function useOnboardingState(open: boolean, onComplete: () => void) {
// Preferred default models for each provider
const preferredDefaults: Partial<Record<LlmProviderFlavor, string>> = {
openai: "gpt-5.2",
anthropic: "claude-opus-4-6-20260202",
openai: "gpt-5.4",
anthropic: "claude-opus-4-8",
}
// Initialize default models from catalog
@ -377,8 +377,8 @@ export function useOnboardingState(open: boolean, onComplete: () => void) {
}, [startGoogleCalendarConnect])
// New step flow:
// Rowboat path: 0 (welcome) → 2 (connect) → 3 (done)
// BYOK path: 0 (welcome) → 1 (llm setup) → 2 (connect) → 3 (done)
// Rowboat path: 0 (welcome) → 2 (connect) → 3 (code mode) → 4 (done)
// BYOK path: 0 (welcome) → 1 (llm setup) → 2 (connect) → 3 (code mode) → 4 (done)
const handleNext = useCallback(() => {
if (currentStep === 0) {
if (onboardingPath === 'byok') {
@ -390,6 +390,8 @@ export function useOnboardingState(open: boolean, onComplete: () => void) {
setCurrentStep(2)
} else if (currentStep === 2) {
setCurrentStep(3)
} else if (currentStep === 3) {
setCurrentStep(4)
}
}, [currentStep, onboardingPath])
@ -403,6 +405,8 @@ export function useOnboardingState(open: boolean, onComplete: () => void) {
} else {
setCurrentStep(1)
}
} else if (currentStep === 3) {
setCurrentStep(2)
}
}, [currentStep, onboardingPath])
@ -429,13 +433,17 @@ export function useOnboardingState(open: boolean, onComplete: () => void) {
return false
}
const models: string[] = result.models ?? []
const catalog: string[] = result.models ?? []
const preferred = preferredDefaults[llmProvider]
const model =
(preferred && models.includes(preferred) && preferred) ||
models[0] || activeConfig.model.trim() || ""
(preferred && catalog.includes(preferred) && preferred) ||
catalog[0] || activeConfig.model.trim() || ""
await window.ipc.invoke("models:saveConfig", { provider, model, models })
// `models` is the user's curated assistant-model list (shown in Settings),
// NOT the full provider catalog. Onboarding seeds it with just the selected
// model; users add more from Settings. Persisting the whole catalog here
// rendered every model as a separate assistant-model row.
await window.ipc.invoke("models:saveConfig", { provider, model, models: model ? [model] : [] })
window.dispatchEvent(new Event('models-config-changed'))
setTestState({ status: "success" })
setConnectedFlavors(prev => new Set(prev).add(llmProvider))

878
apps/x/apps/renderer/src/components/product-tour.tsx Normal file
View file

@ -0,0 +1,878 @@
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from 'react'
import { createPortal } from 'react-dom'
import { X } from 'lucide-react'
import { Button } from '@/components/ui/button'
import { TalkingHead, type MascotHat } from '@/components/talking-head'
import { AgentsFleet, MascotVignette, type TourVignetteKind } from '@/components/tour-vignettes'
import { TourSounds } from '@/lib/tour-sounds'
import type { TTSState } from '@/hooks/useVoiceTTS'
import { cn } from '@/lib/utils'
import tourClipWelcome from '@/assets/tour/welcome.mp3'
import tourClipHome from '@/assets/tour/home.mp3'
import tourClipEmail from '@/assets/tour/email.mp3'
import tourClipMeetings from '@/assets/tour/meetings.mp3'
import tourClipCode from '@/assets/tour/code.mp3'
import tourClipKnowledge from '@/assets/tour/knowledge.mp3'
import tourClipAgents from '@/assets/tour/agents.mp3'
import tourClipWorkspaces from '@/assets/tour/workspaces.mp3'
import tourClipChats from '@/assets/tour/chats.mp3'
import tourClipComposer from '@/assets/tour/composer.mp3'
import tourClipDone from '@/assets/tour/done.mp3'
export type TourNavTarget =
| 'home'
| 'email'
| 'meetings'
| 'code'
| 'knowledge'
| 'agents'
| 'workspaces'
type TourStep = {
id: string
/** Matches a [data-tour-id] element. Steps whose target is absent are skipped. */
targetId?: string
/** View to open when the step starts, via App's navigation handlers. */
navigate?: TourNavTarget
/** Costume the mascot wears at this stop. */
hat?: MascotHat
/** Looping animation staged around the mascot while it presents this stop. */
vignette?: TourVignetteKind
title: string
text: string
/** Spoken narration, when it should differ from the bubble text. */
voiceText?: string
}
const TOUR_STEPS: TourStep[] = [
{
id: 'welcome',
title: 'All aboard! ⚓',
text: "I'm your captain for the next minute. The lights are down and the water's in — let me row you across Rowboat, one stop at a time. Use Next or your arrow keys.",
voiceText: "I'm your captain for the next minute. The lights are down and the water's in — let me row you across Rowboat, one stop at a time.",
},
{
id: 'home',
targetId: 'nav-home',
navigate: 'home',
title: 'First stop: Home',
text: 'Home is your landing spot — a quick overview of what needs your attention to get you back into the flow.',
},
{
id: 'email',
targetId: 'nav-email',
navigate: 'email',
hat: 'mailcap',
vignette: 'email',
title: 'Email',
text: 'Read and triage your inbox right here. Rowboat can summarize threads, label messages, and help you draft replies.',
},
{
id: 'meetings',
targetId: 'nav-meetings',
navigate: 'meetings',
hat: 'headphones',
vignette: 'meetings',
title: 'Meetings',
text: 'Record or join meetings, and get transcripts and notes automatically — prep briefs show up before your calls, too.',
},
{
id: 'code',
targetId: 'nav-code',
navigate: 'code',
hat: 'hardhat',
title: 'Code',
text: 'The Code section runs coding agents on your projects — point one at a folder and drive it from a chat.',
},
{
id: 'knowledge',
targetId: 'nav-knowledge',
navigate: 'knowledge',
hat: 'gradcap',
vignette: 'brain',
title: 'Brain',
text: "Brain is your knowledge base — notes, files, and everything Rowboat learns for you, all connected and searchable.",
},
{
id: 'agents',
targetId: 'nav-agents',
navigate: 'agents',
hat: 'captain',
vignette: 'agents',
title: 'Background agents',
text: 'Background agents work on schedules — they keep your Brain fresh and take care of recurring tasks while you row elsewhere.',
},
{
id: 'workspaces',
targetId: 'nav-workspaces',
navigate: 'workspaces',
hat: 'explorer',
title: 'Workspaces',
text: 'Workspaces hold your project folders and files, so related work stays docked together.',
},
{
id: 'chats',
targetId: 'nav-chats',
title: 'Chats',
text: 'Your recent conversations live here — pick any of them back up right where you left off.',
},
{
id: 'composer',
targetId: 'chat-composer',
title: 'Talk to Rowboat',
text: 'And this is where we talk! Type, dictate with the mic, or turn on voice output — tap my face button and I’ll read replies out loud myself.',
},
{
id: 'done',
hat: 'party',
title: "Land ho! 🎉",
text: "That's the whole bay — and there's my wake to prove it. Take this voyage again anytime from the bottom of the sidebar. Happy rowing!",
},
]
// Pre-recorded narration bundled with the app (no TTS API call, works
// offline/signed-out). Regenerate with scripts/generate-tour-audio.mjs after
// editing any step's text. Steps without a clip fall back to live TTS.
const TOUR_CLIPS: Record<string, string> = {
welcome: tourClipWelcome,
home: tourClipHome,
email: tourClipEmail,
meetings: tourClipMeetings,
code: tourClipCode,
knowledge: tourClipKnowledge,
agents: tourClipAgents,
workspaces: tourClipWorkspaces,
chats: tourClipChats,
composer: tourClipComposer,
done: tourClipDone,
}
const MASCOT_SIZE = 120
const VIEWPORT_MARGIN = 16
const BUBBLE_WIDTH = 288
const TARGET_RESOLVE_TIMEOUT_MS = 1500
const ZOOM_SCALE = 1.05
const GLIDE_EASING = 'cubic-bezier(0.45, 0, 0.2, 1)'
type Pt = { x: number; y: number }
type Rect = { left: number; top: number; width: number; height: number }
type Spot = Rect & { round: boolean }
function clamp(v: number, min: number, max: number): number {
return Math.max(min, Math.min(max, v))
}
function easeInOutCubic(t: number): number {
return t < 0.5 ? 4 * t * t * t : 1 - Math.pow(-2 * t + 2, 3) / 2
}
function quadPoint(p0: Pt, c: Pt, p1: Pt, t: number): Pt {
const u = 1 - t
return {
x: u * u * p0.x + 2 * u * t * c.x + t * t * p1.x,
y: u * u * p0.y + 2 * u * t * c.y + t * t * p1.y,
}
}
function quadPathLength(d: string): number {
const p = document.createElementNS('http://www.w3.org/2000/svg', 'path')
p.setAttribute('d', d)
return p.getTotalLength()
}
// A data-tour-id can legitimately appear on several elements (e.g. the chat
// composer renders in both the full-screen chat and the side pane) — pick the
// one that is actually laid out.
function findTourTarget(targetId: string): HTMLElement | null {
const nodes = document.querySelectorAll<HTMLElement>(`[data-tour-id="${targetId}"]`)
for (const el of nodes) {
const rect = el.getBoundingClientRect()
if (rect.width > 0 && rect.height > 0) return el
}
return null
}
function mascotDestForCenter(): Pt {
return {
x: window.innerWidth / 2 - MASCOT_SIZE / 2 - BUBBLE_WIDTH / 2,
y: window.innerHeight / 2 - MASCOT_SIZE / 2,
}
}
function mascotDestForRect(rect: Rect): Pt {
let x: number
let y: number
const fitsRight = rect.left + rect.width + MASCOT_SIZE + BUBBLE_WIDTH + VIEWPORT_MARGIN * 3 < window.innerWidth
if (fitsRight) {
x = rect.left + rect.width + 20
y = rect.top + rect.height / 2 - MASCOT_SIZE / 2
} else if (rect.top > MASCOT_SIZE + VIEWPORT_MARGIN * 2) {
x = rect.left + rect.width / 2 - MASCOT_SIZE / 2
y = rect.top - MASCOT_SIZE - 20
} else {
x = rect.left - MASCOT_SIZE - 20
y = rect.top + rect.height / 2 - MASCOT_SIZE / 2
}
return {
x: clamp(x, VIEWPORT_MARGIN, window.innerWidth - MASCOT_SIZE - VIEWPORT_MARGIN),
y: clamp(y, VIEWPORT_MARGIN, window.innerHeight - MASCOT_SIZE - VIEWPORT_MARGIN),
}
}
function spotForRect(rect: Rect): Spot {
return {
left: rect.left - 6,
top: rect.top - 6,
width: rect.width + 12,
height: rect.height + 12,
round: false,
}
}
function spotForMascot(dest: Pt): Spot {
return {
left: dest.x - 26,
top: dest.y - 18,
width: MASCOT_SIZE + 52,
height: MASCOT_SIZE + 44,
round: true,
}
}
type ProductTourProps = {
onClose: () => void
onNavigate: (target: TourNavTarget) => void
ttsAvailable: boolean
ttsState: TTSState
speak: (text: string) => void
speakUrl: (url: string) => void
cancelSpeech: () => void
getLevel: () => number
}
/**
* The Grand Voyage: a mascot-guided walkthrough where the app dims to a
* night-time bay, the boat rows curved routes between [data-tour-id] anchors
* (leaving a dotted wake behind it), a spotlight and gentle camera zoom reveal
* each section, and a parchment mini-map charts progress. Narrated with TTS
* lip sync when available; ends in confetti.
*
* Rendered through a portal to <body> so the camera zoom applied to the app
* shell never transforms the tour's own fixed-position layers.
*/
export function ProductTour({
onClose,
onNavigate,
ttsAvailable,
ttsState,
speak,
speakUrl,
cancelSpeech,
getLevel,
}: ProductTourProps) {
const [stepIndex, setStepIndex] = useState(0)
const [arrived, setArrived] = useState(false)
const [rowing, setRowing] = useState(false)
const [flipped, setFlipped] = useState(false)
const [bubbleSide, setBubbleSide] = useState<'left' | 'right'>('right')
const [spot, setSpot] = useState<Spot | null>(null)
const [wakes, setWakes] = useState<{ id: number; d: string }[]>([])
const [activeWake, setActiveWake] = useState<{ d: string; len: number } | null>(null)
const [confettiOn, setConfettiOn] = useState(false)
const [resizeNonce, setResizeNonce] = useState(0)
const reducedMotion = useMemo(
() => window.matchMedia('(prefers-reduced-motion: reduce)').matches,
[]
)
// Mascot position is animated by mutating the container's transform directly
// (60fps travel without re-rendering React); posRef is the source of truth.
const mascotElRef = useRef<HTMLDivElement>(null)
const posRef = useRef<Pt>({
x: window.innerWidth / 2 - MASCOT_SIZE / 2,
y: window.innerHeight + 60,
})
const travelRafRef = useRef(0)
const wakePathElRef = useRef<SVGPathElement>(null)
const wakeIdRef = useRef(0)
const curveSideRef = useRef(1)
const lastSplashRef = useRef(0)
const directionRef = useRef(1)
const enteredStepRef = useRef(-1)
const stepIndexRef = useRef(stepIndex)
// Camera zoom state applied to the app shell (outside the portal)
const shellRef = useRef<HTMLElement | null>(null)
const zoomRef = useRef<{ ox: number; oy: number; s: number } | null>(null)
const soundsRef = useRef<TourSounds | null>(null)
const onCloseRef = useRef(onClose)
const onNavigateRef = useRef(onNavigate)
const speakRef = useRef(speak)
const speakUrlRef = useRef(speakUrl)
const cancelSpeechRef = useRef(cancelSpeech)
const ttsAvailableRef = useRef(ttsAvailable)
// Keep latest callbacks/state in refs so the step effect and key handlers
// stay stable. Runs before the step effect below (effect order = call order).
useEffect(() => {
stepIndexRef.current = stepIndex
onCloseRef.current = onClose
onNavigateRef.current = onNavigate
speakRef.current = speak
speakUrlRef.current = speakUrl
cancelSpeechRef.current = cancelSpeech
ttsAvailableRef.current = ttsAvailable
})
useLayoutEffect(() => {
if (mascotElRef.current) {
mascotElRef.current.style.transform = `translate(${posRef.current.x}px, ${posRef.current.y}px)`
}
soundsRef.current = new TourSounds()
return () => {
soundsRef.current?.dispose()
soundsRef.current = null
}
}, [])
// Grab the app shell for the camera zoom; restore it when the tour ends
useEffect(() => {
const shell = document.querySelector<HTMLElement>('.rowboat-shell')
shellRef.current = shell
return () => {
if (shell) {
shell.style.transform = ''
shell.style.transformOrigin = ''
shell.style.transition = ''
}
shellRef.current = null
zoomRef.current = null
}
}, [])
const applyZoom = useCallback((origin: { ox: number; oy: number } | null) => {
const shell = shellRef.current
if (!shell || reducedMotion) return
if (origin) {
// transform-origin transitions too, so moving between targets pans
// smoothly instead of jumping when the origin changes
shell.style.transition = `transform 0.9s ${GLIDE_EASING}, transform-origin 0.9s ${GLIDE_EASING}`
shell.style.transformOrigin = `${origin.ox}px ${origin.oy}px`
shell.style.transform = `scale(${ZOOM_SCALE})`
zoomRef.current = { ox: origin.ox, oy: origin.oy, s: ZOOM_SCALE }
} else {
shell.style.transition = `transform 0.9s ${GLIDE_EASING}, transform-origin 0.9s ${GLIDE_EASING}`
shell.style.transform = 'scale(1)'
zoomRef.current = null
}
}, [reducedMotion])
// Where the element will sit on screen once this step's zoom settles:
// undo the current zoom mathematically, then apply the upcoming one (whose
// origin is the target's own center, so the center never moves).
const displayedRect = useCallback((el: HTMLElement, willZoom: boolean): { rect: Rect; origin: { ox: number; oy: number } } => {
const m = el.getBoundingClientRect()
const z = zoomRef.current
let cx = m.left + m.width / 2
let cy = m.top + m.height / 2
let w = m.width
let h = m.height
if (z) {
cx = z.ox + (cx - z.ox) / z.s
cy = z.oy + (cy - z.oy) / z.s
w /= z.s
h /= z.s
}
const s = willZoom && !reducedMotion ? ZOOM_SCALE : 1
return {
rect: { left: cx - (w * s) / 2, top: cy - (h * s) / 2, width: w * s, height: h * s },
origin: { ox: cx, oy: cy },
}
}, [reducedMotion])
const cancelTravel = useCallback(() => {
cancelAnimationFrame(travelRafRef.current)
}, [])
const moveMascot = useCallback((p: Pt) => {
posRef.current = p
if (mascotElRef.current) {
mascotElRef.current.style.transform = `translate(${p.x}px, ${p.y}px)`
}
}, [])
// Row along a curved path from the current position, drawing the wake as we
// go and splashing the oar; commits the wake as a dotted trail on arrival.
const startTravel = useCallback((dest: Pt, onArrive: () => void) => {
cancelTravel()
const from = { ...posRef.current }
const dist = Math.hypot(dest.x - from.x, dest.y - from.y)
if (dist < 6 || reducedMotion) {
moveMascot(dest)
setRowing(false)
onArrive()
return
}
const dur = clamp(dist * 1.1, 550, 1500)
curveSideRef.current = -curveSideRef.current
const mx = (from.x + dest.x) / 2
const my = (from.y + dest.y) / 2
const nx = -(dest.y - from.y) / dist
const ny = (dest.x - from.x) / dist
const mag = Math.min(140, dist * 0.3) * curveSideRef.current
const c = { x: mx + nx * mag, y: my + ny * mag }
// Wake follows the stern (bottom-center of the mascot box)
const sternX = MASCOT_SIZE / 2
const sternY = MASCOT_SIZE * 0.82
const d = `M ${from.x + sternX} ${from.y + sternY} Q ${c.x + sternX} ${c.y + sternY} ${dest.x + sternX} ${dest.y + sternY}`
const len = quadPathLength(d)
setActiveWake({ d, len })
setFlipped(dest.x < from.x - 4)
setRowing(true)
lastSplashRef.current = 0
const t0 = performance.now()
const frame = (now: number) => {
const raw = Math.min(1, (now - t0) / dur)
const t = easeInOutCubic(raw)
moveMascot(quadPoint(from, c, dest, t))
if (wakePathElRef.current) {
wakePathElRef.current.style.strokeDashoffset = String(len * (1 - t))
}
if (now - lastSplashRef.current > 420) {
lastSplashRef.current = now
soundsRef.current?.splash()
}
if (raw < 1) {
travelRafRef.current = requestAnimationFrame(frame)
} else {
setRowing(false)
setActiveWake(null)
setWakes((ws) => [...ws, { id: wakeIdRef.current++, d }])
onArrive()
}
}
travelRafRef.current = requestAnimationFrame(frame)
}, [cancelTravel, moveMascot, reducedMotion])
const playDing = useCallback(() => soundsRef.current?.ding(), [])
const finish = useCallback(() => {
cancelSpeechRef.current()
onCloseRef.current()
}, [])
const goTo = useCallback((index: number, direction: 1 | -1) => {
directionRef.current = direction
if (index < 0) return
if (index >= TOUR_STEPS.length) {
finish()
return
}
// Silence the current step's narration right away — not on arrival —
// so it can't talk over (or into) the next step's speech.
cancelSpeechRef.current()
setStepIndex(index)
}, [finish])
// Stop any in-flight narration when the tour unmounts
useEffect(() => () => cancelSpeechRef.current(), [])
useEffect(() => {
const handleResize = () => setResizeNonce((n) => n + 1)
window.addEventListener('resize', handleResize)
return () => window.removeEventListener('resize', handleResize)
}, [])
// Enter the current step: navigate, wait for its anchor, aim the spotlight
// and camera, row over, then narrate. Re-runs on resize to re-anchor.
useEffect(() => {
const step = TOUR_STEPS[stepIndex]
const entering = enteredStepRef.current !== stepIndex
let cancelled = false
if (entering && step.navigate) {
onNavigateRef.current(step.navigate)
}
const settle = (dest: Pt, side: 'left' | 'right', spotlight: Spot, origin: { ox: number; oy: number } | null) => {
applyZoom(origin)
setSpot(spotlight)
setBubbleSide(side)
if (!entering) {
// Resize while already at this step: jump, keep the bubble up
cancelTravel()
moveMascot(dest)
return
}
enteredStepRef.current = stepIndex
setArrived(false)
startTravel(dest, () => {
if (cancelled) return
setArrived(true)
soundsRef.current?.bump()
cancelSpeechRef.current()
const clip = TOUR_CLIPS[step.id]
if (clip) {
speakUrlRef.current(clip)
} else if (ttsAvailableRef.current) {
speakRef.current(step.voiceText ?? step.text)
}
if (stepIndex === TOUR_STEPS.length - 1) {
setConfettiOn(true)
soundsRef.current?.fanfare()
}
})
}
if (!step.targetId) {
const dest = mascotDestForCenter()
applyZoom(null)
settle(dest, 'right', spotForMascot(dest), null)
return () => {
cancelled = true
cancelTravel()
}
}
const startedAt = performance.now()
let pollRaf = 0
const attempt = () => {
if (cancelled) return
const el = findTourTarget(step.targetId!)
if (el) {
const { rect, origin } = displayedRect(el, true)
const dest = mascotDestForRect(rect)
const side: 'left' | 'right' = dest.x + MASCOT_SIZE / 2 < window.innerWidth / 2 ? 'right' : 'left'
settle(dest, side, spotForRect(rect), origin)
return
}
if (performance.now() - startedAt < TARGET_RESOLVE_TIMEOUT_MS) {
pollRaf = requestAnimationFrame(attempt)
} else if (entering) {
// Anchor never appeared (feature disabled / pane closed) — skip past it
goTo(stepIndex + directionRef.current, directionRef.current as 1 | -1)
}
}
attempt()
return () => {
cancelled = true
cancelAnimationFrame(pollRaf)
cancelTravel()
}
}, [stepIndex, resizeNonce, goTo, applyZoom, displayedRect, startTravel, cancelTravel, moveMascot])
useEffect(() => {
const handleKeyDown = (e: KeyboardEvent) => {
if (e.key === 'Escape') {
e.preventDefault()
finish()
} else if (e.key === 'ArrowRight' || e.key === 'Enter') {
e.preventDefault()
goTo(stepIndexRef.current + 1, 1)
} else if (e.key === 'ArrowLeft') {
e.preventDefault()
goTo(stepIndexRef.current - 1, -1)
}
}
window.addEventListener('keydown', handleKeyDown)
return () => window.removeEventListener('keydown', handleKeyDown)
}, [finish, goTo])
const step = TOUR_STEPS[stepIndex]
const isFirst = stepIndex === 0
const isLast = stepIndex === TOUR_STEPS.length - 1
return createPortal(
<>
<style>{`
@keyframes tour-bubble-in {
0% { opacity: 0; transform: translateY(6px) scale(0.97); }
100% { opacity: 1; transform: translateY(0) scale(1); }
}
@keyframes tour-wave-drift {
from { transform: translateX(0); }
to { transform: translateX(-50%); }
}
@keyframes tour-stamp-in {
0% { opacity: 0; transform: scale(2); }
100% { opacity: 1; transform: scale(1); }
}
`}</style>
{/* night falls: dim everything except the spotlight cutout */}
{spot && (
<div
className="pointer-events-none fixed z-[64]"
style={{
left: spot.left,
top: spot.top,
width: spot.width,
height: spot.height,
borderRadius: spot.round ? 9999 : 14,
boxShadow: '0 0 0 200vmax rgba(7, 14, 26, 0.52)',
transition: `all 0.9s ${GLIDE_EASING}`,
}}
/>
)}
<TourWater />
{/* wake trails: the committed dotted route + the wake being drawn now */}
<svg className="pointer-events-none fixed inset-0 z-[66] h-full w-full">
{wakes.map((w) => (
<path
key={w.id}
d={w.d}
fill="none"
stroke="#9CCBEA"
strokeWidth={4}
strokeLinecap="round"
strokeDasharray="1 11"
opacity={0.75}
/>
))}
{activeWake && (
<path
ref={wakePathElRef}
d={activeWake.d}
fill="none"
stroke="#BFE0F5"
strokeWidth={5}
strokeLinecap="round"
strokeDasharray={activeWake.len}
strokeDashoffset={activeWake.len}
opacity={0.8}
/>
)}
</svg>
<TourMiniMap total={TOUR_STEPS.length} current={stepIndex} arrived={arrived} />
{/* the boat (position driven imperatively during travel) */}
<div
ref={mascotElRef}
className="fixed left-0 top-0 z-[70]"
style={{ width: MASCOT_SIZE, pointerEvents: 'none' }}
>
{arrived && !reducedMotion && step.vignette && step.vignette !== 'agents' && (
<MascotVignette kind={step.vignette} playDing={playDing} />
)}
<div style={{ transform: flipped ? 'scaleX(-1)' : undefined, transition: 'transform 0.35s ease-in-out' }}>
<TalkingHead ttsState={ttsState} getLevel={getLevel} size={MASCOT_SIZE} hat={step.hat} rowing={rowing} />
</div>
{arrived && (
<div
key={step.id}
className={cn(
'pointer-events-auto absolute top-0 z-10 rounded-xl border border-border bg-popover p-4 text-popover-foreground shadow-lg',
bubbleSide === 'right' ? 'left-full ml-3' : 'right-full mr-3'
)}
style={{
width: BUBBLE_WIDTH,
animation: 'tour-bubble-in 0.25s ease-out',
}}
>
<button
type="button"
onClick={finish}
className="absolute right-2 top-2 flex h-6 w-6 items-center justify-center rounded-md text-muted-foreground transition-colors hover:bg-muted hover:text-foreground"
aria-label="End tour"
>
<X className="h-3.5 w-3.5" />
</button>
<p className="pr-6 text-sm font-semibold">{step.title}</p>
<p className="mt-1.5 text-sm text-muted-foreground">{step.text}</p>
<div className="mt-3 flex items-center justify-between">
<span className="text-xs tabular-nums text-muted-foreground">
{stepIndex + 1} / {TOUR_STEPS.length}
</span>
<div className="flex items-center gap-1.5">
{!isFirst && (
<Button variant="ghost" size="sm" className="h-7 px-2.5" onClick={() => goTo(stepIndex - 1, -1)}>
Back
</Button>
)}
<Button
size="sm"
className="h-7 px-3"
onClick={() => (isLast ? finish() : goTo(stepIndex + 1, 1))}
>
{isLast ? 'Done' : 'Next'}
</Button>
</div>
</div>
</div>
)}
</div>
{arrived && !reducedMotion && step.vignette === 'agents' && <AgentsFleet />}
{confettiOn && !reducedMotion && <ConfettiBurst />}
</>,
document.body
)
}
/** Animated translucent waves lapping at the bottom of the screen. */
function TourWater() {
const back = useMemo(() => wavePath(2400, 96, 8, 22), [])
const front = useMemo(() => wavePath(2400, 96, 12, 30), [])
return (
<div className="pointer-events-none fixed inset-x-0 bottom-0 z-[65] h-24 overflow-hidden" aria-hidden="true">
<svg
className="absolute bottom-0 left-0 h-full"
style={{ width: '200%', animation: 'tour-wave-drift 11s linear infinite' }}
viewBox="0 0 2400 96"
preserveAspectRatio="none"
>
<path d={back} fill="#5F9BC9" opacity={0.3} />
</svg>
<svg
className="absolute bottom-0 left-0 h-full"
style={{ width: '200%', animation: 'tour-wave-drift 7s linear infinite reverse' }}
viewBox="0 0 2400 96"
preserveAspectRatio="none"
>
<path d={front} fill="#8FB6D9" opacity={0.35} />
</svg>
</div>
)
}
// Periodic wave: alternating up/down humps so a -50% translate loops seamlessly
// (both hump counts are even, so half the width is a whole number of periods).
function wavePath(width: number, height: number, humps: number, amp: number): string {
const yTop = 34
const seg = width / humps
let d = `M 0 ${yTop}`
for (let i = 0; i < humps; i++) {
d += ` q ${seg / 2} ${i % 2 === 0 ? -amp : amp} ${seg} 0`
}
d += ` L ${width} ${height} L 0 ${height} Z`
return d
}
/** Parchment chart in the corner: islands per stop, dotted route, boat marker. */
function TourMiniMap({ total, current, arrived }: { total: number; current: number; arrived: boolean }) {
const MW = 184
const MH = 96
const points = useMemo(
() =>
Array.from({ length: total }, (_, i) => {
const t = total === 1 ? 0 : i / (total - 1)
return {
x: 14 + (MW - 28) * t,
y: MH / 2 + 4 + Math.sin(t * Math.PI * 1.5 + 0.6) * (MH * 0.26),
}
}),
[total]
)
const route = points.map((p, i) => (i === 0 ? `M ${p.x} ${p.y}` : `L ${p.x} ${p.y}`)).join(' ')
const boat = points[clamp(current, 0, total - 1)]
return (
<div
className="fixed bottom-5 left-5 z-[71] rounded-lg border-2 border-[#8A6B3D]/60 bg-[#F4E9CE] px-2 pb-1.5 pt-2 shadow-xl"
style={{ transform: 'rotate(-1.2deg)' }}
aria-hidden="true"
>
<p className="mb-0.5 px-1 text-[9px] font-semibold uppercase tracking-[0.22em] text-[#6B5138]">
Voyage chart
</p>
<svg width={MW} height={MH} viewBox={`0 0 ${MW} ${MH}`}>
<path d={route} fill="none" stroke="#8A6B3D" strokeWidth={1.5} strokeDasharray="3 4" opacity={0.65} />
{points.map((p, i) => {
const visited = i < current || (i === current && arrived)
return (
<g key={i}>
<ellipse cx={p.x} cy={p.y} rx={7} ry={5} fill="#DFC896" stroke="#8A6B3D" strokeWidth={1.5} />
{visited && (
<g style={{ animation: 'tour-stamp-in 0.3s ease-out backwards' }}>
<line x1={p.x - 1} y1={p.y - 13} x2={p.x - 1} y2={p.y - 3} stroke="#7A4A21" strokeWidth={1.5} />
<path d={`M ${p.x - 1} ${p.y - 13} L ${p.x + 6} ${p.y - 10.5} L ${p.x - 1} ${p.y - 8} Z`} fill="#D9534F" />
</g>
)}
</g>
)
})}
{/* boat marker glides between islands in step with the real mascot */}
<g style={{ transform: `translate(${boat.x}px, ${boat.y - 7}px)`, transition: `transform 0.9s ${GLIDE_EASING}` }}>
<path d="M -7 0 Q 0 4 7 0 Q 4 6 0 6 Q -4 6 -7 0 Z" fill="#54402F" stroke="#3E2E24" strokeWidth={1} />
<circle cx={0} cy={-3} r={3} fill="#E8E9F5" stroke="#17171B" strokeWidth={1} />
</g>
</svg>
</div>
)
}
/** Two confetti cannons firing from the bottom corners, canvas-driven. */
function ConfettiBurst() {
const canvasRef = useRef<HTMLCanvasElement>(null)
useEffect(() => {
const canvas = canvasRef.current
const ctx = canvas?.getContext('2d')
if (!canvas || !ctx) return
const w = window.innerWidth
const h = window.innerHeight
const dpr = window.devicePixelRatio || 1
canvas.width = w * dpr
canvas.height = h * dpr
ctx.scale(dpr, dpr)
const colors = ['#5B8DEF', '#F2B8BE', '#FFD166', '#7AC74F', '#8FB6D9', '#F2699C']
const parts = Array.from({ length: 150 }, (_, i) => {
const fromLeft = i % 2 === 0
return {
x: fromLeft ? 24 : w - 24,
y: h - 40,
vx: (fromLeft ? 1 : -1) * (2.5 + Math.random() * 6.5),
vy: -(9 + Math.random() * 8),
size: 5 + Math.random() * 5,
color: colors[i % colors.length],
rot: Math.random() * Math.PI,
vr: (Math.random() - 0.5) * 0.3,
}
})
let raf = 0
const t0 = performance.now()
const frame = (now: number) => {
ctx.clearRect(0, 0, w, h)
for (const p of parts) {
p.vy += 0.18
p.vx *= 0.99
p.x += p.vx
p.y += p.vy
p.rot += p.vr
ctx.save()
ctx.translate(p.x, p.y)
ctx.rotate(p.rot)
ctx.fillStyle = p.color
ctx.fillRect(-p.size / 2, -p.size / 3, p.size, p.size * 0.66)
ctx.restore()
}
if (now - t0 < 3200) {
raf = requestAnimationFrame(frame)
} else {
ctx.clearRect(0, 0, w, h)
}
}
raf = requestAnimationFrame(frame)
return () => cancelAnimationFrame(raf)
}, [])
return (
<canvas
ref={canvasRef}
className="pointer-events-none fixed inset-0 z-[72]"
style={{ width: '100vw', height: '100vh' }}
/>
)
}

184
apps/x/apps/renderer/src/components/settings-dialog.tsx
View file

@ -2,7 +2,7 @@
import * as React from "react"
import { useState, useEffect, useCallback, useMemo } from "react"
import { Server, Key, Shield, Palette, Monitor, Sun, Moon, Loader2, CheckCircle2, Plus, X, Wrench, Search, ChevronRight, Link2, Tags, Mail, BookOpen, User, Plug, HelpCircle, MessageCircle, Bug, Terminal, AlertTriangle, RefreshCw, PanelRight, Bell } from "lucide-react"
import { Server, Key, Shield, Palette, Monitor, Sun, Moon, Loader2, CheckCircle2, Plus, X, Wrench, Search, ChevronRight, Link2, Tags, Mail, BookOpen, User, Plug, HelpCircle, MessageCircle, Bug, Terminal, AlertTriangle, RefreshCw, PanelRight, Bell, Smartphone } from "lucide-react"
import {
Dialog,
@ -25,9 +25,11 @@ import { useTheme } from "@/contexts/theme-context"
import { toast } from "sonner"
import { AccountSettings } from "@/components/settings/account-settings"
import { ConnectedAccountsSettings } from "@/components/settings/connected-accounts-settings"
import { MobileChannelsSettings } from "@/components/settings/mobile-channels-settings"
import type { ApprovalPolicy } from "@x/shared/src/code-mode.js"
import { startProvisioning, useProvisioning, enabledOptimistic, type AgentStatus, type CodeModeAgentStatus } from "@/lib/code-mode-provisioning"
type ConfigTab = "account" | "connections" | "models" | "mcp" | "security" | "code-mode" | "appearance" | "notifications" | "note-tagging" | "help"
type ConfigTab = "account" | "connections" | "mobile" | "models" | "mcp" | "security" | "code-mode" | "appearance" | "notifications" | "note-tagging" | "help"
interface TabConfig {
id: ConfigTab
@ -50,6 +52,12 @@ const tabs: TabConfig[] = [
icon: Plug,
description: "Manage accounts and tools",
},
{
id: "mobile",
label: "Mobile",
icon: Smartphone,
description: "Chat with Rowboat from WhatsApp or Telegram",
},
{
id: "models",
label: "Models",
@ -319,8 +327,8 @@ const moreProviders: Array<{ id: LlmProviderFlavor; name: string; description: s
]
const preferredDefaults: Partial<Record<LlmProviderFlavor, string>> = {
openai: "gpt-5.2",
anthropic: "claude-opus-4-6-20260202",
openai: "gpt-5.4",
anthropic: "claude-opus-4-8",
}
const defaultBaseURLs: Partial<Record<LlmProviderFlavor, string>> = {
@ -384,38 +392,6 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) {
[]
)
const updateModelAt = useCallback(
(prov: LlmProviderFlavor, index: number, value: string) => {
setProviderConfigs(prev => {
const models = [...prev[prov].models]
models[index] = value
return { ...prev, [prov]: { ...prev[prov], models } }
})
setTestState({ status: "idle" })
},
[]
)
const addModel = useCallback(
(prov: LlmProviderFlavor) => {
setProviderConfigs(prev => ({
...prev,
[prov]: { ...prev[prov], models: [...prev[prov].models, ""] },
}))
},
[]
)
const removeModel = useCallback(
(prov: LlmProviderFlavor, index: number) => {
setProviderConfigs(prev => {
const models = prev[prov].models.filter((_, i) => i !== index)
return { ...prev, [prov]: { ...prev[prov], models: models.length > 0 ? models : [""] } }
})
setTestState({ status: "idle" })
},
[]
)
// Load current config from file
useEffect(() => {
@ -727,48 +703,29 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) {
</div>
) : (
<div className="space-y-2">
{activeConfig.models.map((model, index) => (
<div key={index} className="group/model relative">
{showModelInput ? (
<Input
value={model}
onChange={(e) => updateModelAt(provider, index, e.target.value)}
placeholder="Enter model"
/>
) : (
<Select
value={model}
onValueChange={(value) => updateModelAt(provider, index, value)}
>
<SelectTrigger>
<SelectValue placeholder="Select a model" />
</SelectTrigger>
<SelectContent>
{modelsForProvider.map((m) => (
<SelectItem key={m.id} value={m.id}>
{m.name || m.id}
</SelectItem>
))}
</SelectContent>
</Select>
)}
{activeConfig.models.length > 1 && (
<button
onClick={() => removeModel(provider, index)}
className="absolute right-8 top-1/2 -translate-y-1/2 flex size-6 items-center justify-center rounded text-muted-foreground opacity-0 transition-opacity hover:text-foreground group-hover/model:opacity-100"
>
<X className="size-3.5" />
</button>
)}
</div>
))}
<button
onClick={() => addModel(provider)}
className="flex items-center gap-1.5 text-xs text-muted-foreground hover:text-foreground transition-colors"
>
<Plus className="size-3.5" />
Add assistant model
</button>
{showModelInput ? (
<Input
value={primaryModel}
onChange={(e) => updateConfig(provider, { models: [e.target.value] })}
placeholder="Enter model"
/>
) : (
<Select
value={primaryModel}
onValueChange={(value) => updateConfig(provider, { models: [value] })}
>
<SelectTrigger>
<SelectValue placeholder="Select a model" />
</SelectTrigger>
<SelectContent>
{modelsForProvider.map((m) => (
<SelectItem key={m.id} value={m.id}>
{m.name || m.id}
</SelectItem>
))}
</SelectContent>
</Select>
)}
</div>
)}
{modelsError && (
@ -1756,66 +1713,6 @@ function NoteTaggingSettings({ dialogOpen }: { dialogOpen: boolean }) {
// --- Code Mode Settings ---
type AgentStatus = { installed: boolean; signedIn: boolean }
type CodeModeAgentStatus = { claude: AgentStatus; codex: AgentStatus }
// Engine provisioning runs in the main process and keeps going even if the Settings
// dialog is closed. Track its state at MODULE level (not in the row component, which
// unmounts on close) so reopening Settings still shows the live % instead of the Enable
// button. A single persistent listener on the progress channel feeds this store.
type ProvState = { pct: number | null; error?: string }
const provStore: Record<string, ProvState | undefined> = {}
// Agents we provisioned this session — used to show "Ready" immediately on success
// without waiting for the async status refresh to round-trip (which caused the row to
// briefly flash the Enable button again).
const enabledOptimistic = new Set<string>()
const provListeners = new Set<() => void>()
let provChannelHooked = false
function notifyProv() { provListeners.forEach((l) => l()) }
function startProvisioning(agent: 'claude' | 'codex', onDone: () => void | Promise<void>): void {
if (provStore[agent] && !provStore[agent]!.error) return // already in flight
provStore[agent] = { pct: null }
notifyProv()
if (!provChannelHooked) {
provChannelHooked = true
window.ipc.on('codeMode:engineProgress', (p) => {
const cur = provStore[p.agent]
if (!cur) return
const pct = p.totalBytes ? Math.floor(((p.receivedBytes ?? 0) / p.totalBytes) * 100) : cur.pct
provStore[p.agent] = { pct }
notifyProv()
})
}
window.ipc.invoke('codeMode:provisionEngine', { agent })
.then((res) => {
if (res.success) {
// Mark installed optimistically so the row shows "Ready" the instant the flag
// clears — don't depend on the async status refresh (which re-renders the parent
// separately and left a window showing the Enable button). loadStatus still runs
// in the background to sync the real status.
enabledOptimistic.add(agent)
provStore[agent] = undefined
void onDone()
} else {
provStore[agent] = { pct: null, error: res.error ?? 'Failed to enable' }
}
})
.catch((e) => { provStore[agent] = { pct: null, error: e instanceof Error ? e.message : 'Failed to enable' } })
.finally(notifyProv)
}
function useProvisioning(agent: string): ProvState | undefined {
const [, force] = useState(0)
useEffect(() => {
const l = () => force((n) => n + 1)
provListeners.add(l)
return () => { provListeners.delete(l) }
}, [])
return provStore[agent]
}
function AgentStatusRow({
name,
agent,
@ -2075,7 +1972,7 @@ function CodeModeSettings({ dialogOpen }: { dialogOpen: boolean }) {
// --- Notification Settings ---
type NotificationCategoryKey = "chat_completion" | "new_email" | "agent_permission"
type NotificationCategoryKey = "chat_completion" | "new_email" | "agent_permission" | "background_task"
const NOTIFICATION_CATEGORIES: { key: NotificationCategoryKey; label: string; description: string }[] = [
{
@ -2093,6 +1990,11 @@ const NOTIFICATION_CATEGORIES: { key: NotificationCategoryKey; label: string; de
label: "Permission requests",
description: "When an agent needs your approval to run a tool. Always shown, even when the app is focused.",
},
{
key: "background_task",
label: "Background agents",
description: "When a background agent you've set up has something to surface. Click to open it on the background tasks page.",
},
]
function NotificationSettings({ dialogOpen }: { dialogOpen: boolean }) {
@ -2321,7 +2223,7 @@ export function SettingsDialog({ children, defaultTab = "account", open: control
</div>
{/* Content */}
<div className={cn("flex-1 p-4 min-h-0", (activeTab === "models" || activeTab === "connections" || activeTab === "account" || activeTab === "code-mode" || activeTab === "notifications") ? "overflow-y-auto" : activeTab === "note-tagging" ? "overflow-hidden flex flex-col" : "overflow-hidden")}>
<div className={cn("flex-1 p-4 min-h-0", (activeTab === "models" || activeTab === "connections" || activeTab === "mobile" || activeTab === "account" || activeTab === "code-mode" || activeTab === "notifications") ? "overflow-y-auto" : activeTab === "note-tagging" ? "overflow-hidden flex flex-col" : "overflow-hidden")}>
{activeTab === "account" ? (
<AccountSettings dialogOpen={open} />
) : activeTab === "connections" ? (
@ -2336,6 +2238,8 @@ export function SettingsDialog({ children, defaultTab = "account", open: control
<ToolsLibrarySettings dialogOpen={open} rowboatConnected={rowboatConnected} />
</div>
</div>
) : activeTab === "mobile" ? (
<MobileChannelsSettings dialogOpen={open} />
) : activeTab === "models" ? (
rowboatConnected
? <RowboatModelSettings dialogOpen={open} />

287
apps/x/apps/renderer/src/components/settings/mobile-channels-settings.tsx Normal file
View file

@ -0,0 +1,287 @@
"use client"
import { useCallback, useEffect, useState } from "react"
import type { z } from "zod"
import { Loader2, MessageCircle, Send, Smartphone } from "lucide-react"
import { Button } from "@/components/ui/button"
import { Input } from "@/components/ui/input"
import { Separator } from "@/components/ui/separator"
import { Switch } from "@/components/ui/switch"
import { toast } from "sonner"
import type { ChannelsConfig, ChannelsStatus } from "@x/shared/src/channels.js"
type Config = z.infer<typeof ChannelsConfig>
type Status = z.infer<typeof ChannelsStatus>
// Comma/newline separated entries; each entry keeps digits only, so a
// formatted number like "+1 (415) 555-1234" survives as one entry instead of
// being shattered at the spaces.
function parseIdList(draft: string): string[] {
return draft
.split(/[,;\n]+/)
.map((s) => s.replace(/\D/g, ""))
.filter(Boolean)
}
export function MobileChannelsSettings({ dialogOpen }: { dialogOpen: boolean }) {
const [config, setConfig] = useState<Config | null>(null)
const [status, setStatus] = useState<Status | null>(null)
const [tokenDraft, setTokenDraft] = useState("")
const [waAllowDraft, setWaAllowDraft] = useState("")
const [tgAllowDraft, setTgAllowDraft] = useState("")
const [saving, setSaving] = useState(false)
useEffect(() => {
if (!dialogOpen) return
let cancelled = false
void (async () => {
try {
const [cfg, st] = await Promise.all([
window.ipc.invoke("channels:getConfig", null),
window.ipc.invoke("channels:getStatus", null),
])
if (cancelled) return
setConfig(cfg)
setStatus(st)
setTokenDraft(cfg.telegram.botToken)
setWaAllowDraft(cfg.whatsapp.allowFrom.join(", "))
setTgAllowDraft(cfg.telegram.allowFrom.join(", "))
} catch {
if (!cancelled) toast.error("Failed to load mobile channel settings")
}
})()
const unsubscribe = window.ipc.on("channels:status", (st) => {
setStatus(st)
})
return () => {
cancelled = true
unsubscribe()
}
}, [dialogOpen])
const save = useCallback(async (next: Config) => {
setConfig(next)
setSaving(true)
try {
await window.ipc.invoke("channels:setConfig", next)
} catch (error) {
toast.error(error instanceof Error ? error.message : "Failed to save channel settings")
} finally {
setSaving(false)
}
}, [])
if (!config) {
return (
<div className="flex items-center justify-center py-8 text-muted-foreground">
<Loader2 className="size-4 animate-spin" />
</div>
)
}
const wa = status?.whatsapp
const tg = status?.telegram
return (
<div className="space-y-6">
<div className="flex items-start gap-2.5 rounded-md bg-muted/50 px-3 py-2.5">
<Smartphone className="size-4 mt-0.5 shrink-0 text-muted-foreground" />
<p className="text-xs text-muted-foreground">
Chat with Rowboat from your phone. Send <span className="font-mono">help</span> for
commands (<span className="font-mono">list</span>, <span className="font-mono">resume 2</span>,{" "}
<span className="font-mono">new</span>, <span className="font-mono">stop</span>) — anything
else is a message to the current chat. Your computer must be on with Rowboat running.
</p>
</div>
{/* WhatsApp */}
<div className="space-y-3">
<div className="flex items-center justify-between">
<div className="flex items-center gap-2.5">
<div className="flex size-8 items-center justify-center rounded-md bg-muted">
<MessageCircle className="size-4" />
</div>
<div className="flex flex-col">
<span className="text-sm font-medium">WhatsApp</span>
<span className="text-xs text-muted-foreground">
{wa?.state === "connected"
? `Linked as +${wa.self ?? "?"} — message yourself to use it`
: wa?.state === "qr"
? "Scan the QR below with your phone"
: wa?.state === "starting"
? "Connecting…"
: wa?.state === "error"
? wa.error ?? "Error"
: "Links your own WhatsApp as a companion device"}
</span>
</div>
</div>
<div className="flex items-center gap-2">
{wa?.state === "connected" && (
<Button
variant="outline"
size="sm"
className="h-7 px-3 text-xs"
onClick={() => {
void window.ipc.invoke("channels:whatsappLogout", null).catch(() => {
toast.error("Failed to unlink WhatsApp")
})
}}
>
Unlink
</Button>
)}
<Switch
checked={config.whatsapp.enabled}
disabled={saving}
onCheckedChange={(enabled) =>
void save({ ...config, whatsapp: { ...config.whatsapp, enabled } })
}
/>
</div>
</div>
{config.whatsapp.enabled && wa?.state === "qr" && wa.qrDataUrl && (
<div className="flex items-center gap-4 rounded-md border p-3">
<img src={wa.qrDataUrl} alt="WhatsApp pairing QR" className="size-40 rounded" />
<div className="text-xs text-muted-foreground space-y-1">
<p className="font-medium text-foreground">Link Rowboat to WhatsApp</p>
<p>1. Open WhatsApp on your phone</p>
<p>2. Settings → Linked Devices → Link a Device</p>
<p>3. Scan this code</p>
<p className="pt-1">
Then message <span className="font-medium">yourself</span> (your own contact) to talk
to Rowboat.
</p>
</div>
</div>
)}
{config.whatsapp.enabled && (
<div className="space-y-1.5">
<label className="text-xs font-medium">Additional allowed numbers</label>
<div className="flex gap-2">
<Input
value={waAllowDraft}
onChange={(e) => setWaAllowDraft(e.target.value)}
placeholder="e.g. 14155551234, 919876543210"
className="h-8 text-xs"
/>
<Button
variant="outline"
size="sm"
className="h-8 px-3 text-xs"
disabled={saving}
onClick={() =>
void save({
...config,
whatsapp: { ...config.whatsapp, allowFrom: parseIdList(waAllowDraft) },
})
}
>
Save
</Button>
</div>
<p className="text-xs text-muted-foreground">
Your own number (self-chat) is always allowed. Digits only, with country code.
</p>
</div>
)}
</div>
<Separator />
{/* Telegram */}
<div className="space-y-3">
<div className="flex items-center justify-between">
<div className="flex items-center gap-2.5">
<div className="flex size-8 items-center justify-center rounded-md bg-muted">
<Send className="size-4" />
</div>
<div className="flex flex-col">
<span className="text-sm font-medium">Telegram</span>
<span className="text-xs text-muted-foreground">
{tg?.state === "polling"
? `Listening${tg.botUsername ? ` as @${tg.botUsername}` : ""}`
: tg?.state === "starting"
? "Connecting…"
: tg?.state === "error"
? tg.error ?? "Error"
: "Uses your own bot — create one with @BotFather"}
</span>
</div>
</div>
<Switch
checked={config.telegram.enabled}
disabled={saving}
onCheckedChange={(enabled) =>
void save({ ...config, telegram: { ...config.telegram, enabled } })
}
/>
</div>
{config.telegram.enabled && (
<>
<div className="space-y-1.5">
<label className="text-xs font-medium">Bot token</label>
<div className="flex gap-2">
<Input
type="password"
value={tokenDraft}
onChange={(e) => setTokenDraft(e.target.value)}
placeholder="123456789:AAF…"
className="h-8 text-xs font-mono"
/>
<Button
variant="outline"
size="sm"
className="h-8 px-3 text-xs"
disabled={saving}
onClick={() =>
void save({
...config,
telegram: { ...config.telegram, botToken: tokenDraft.trim() },
})
}
>
Save
</Button>
</div>
<p className="text-xs text-muted-foreground">
Message @BotFather on Telegram → /newbot → paste the token here.
</p>
</div>
<div className="space-y-1.5">
<label className="text-xs font-medium">Allowed chat IDs</label>
<div className="flex gap-2">
<Input
value={tgAllowDraft}
onChange={(e) => setTgAllowDraft(e.target.value)}
placeholder="e.g. 123456789"
className="h-8 text-xs"
/>
<Button
variant="outline"
size="sm"
className="h-8 px-3 text-xs"
disabled={saving}
onClick={() =>
void save({
...config,
telegram: { ...config.telegram, allowFrom: parseIdList(tgAllowDraft) },
})
}
>
Save
</Button>
</div>
<p className="text-xs text-muted-foreground">
Message your bot once — it replies with your chat ID to add here.
</p>
</div>
</>
)}
</div>
</div>
)
}

23
apps/x/apps/renderer/src/components/sidebar-content.tsx
View file

@ -63,6 +63,7 @@ import {
import { cn } from "@/lib/utils"
import { isOutOfCredits, CREDIT_EXHAUSTED_EVENT, CREDIT_REPLENISHED_EVENT } from "@/lib/credit-status"
import { SettingsDialog } from "@/components/settings-dialog"
import { MascotFaceIcon } from "@/components/talking-head"
import { extractConferenceLink } from "@/lib/calendar-event"
import { useBilling } from "@/hooks/useBilling"
import { toast } from "@/lib/toast"
@ -174,6 +175,8 @@ type SidebarContentPanelProps = {
onNewChat?: () => void
onToggleBrowser?: () => void
onVoiceNoteCreated?: (path: string) => void
/** Starts the mascot-guided product tour. */
onStartTour?: () => void
/** Which primary destination is currently active, for nav highlighting. */
activeNav?: 'home' | 'email' | 'meetings' | 'code' | 'knowledge' | 'agents' | 'workspaces' | null
/** Live meeting recording state, so the recording row can show its indicator/stop. */
@ -422,6 +425,7 @@ export function SidebarContentPanel({
onNewChat,
onToggleBrowser,
onVoiceNoteCreated,
onStartTour,
activeNav,
meetingRecordingState = 'idle',
recordingMeetingSource = null,
@ -747,13 +751,14 @@ export function SidebarContentPanel({
<SidebarGroupContent>
<SidebarMenu>
<SidebarMenuItem>
<SidebarMenuButton isActive={activeNav === 'home'} onClick={onOpenHome}>
<SidebarMenuButton data-tour-id="nav-home" isActive={activeNav === 'home'} onClick={onOpenHome}>
<Home className="size-4 shrink-0" />
<span className="flex-1 truncate">Home</span>
</SidebarMenuButton>
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-email"
isActive={activeNav === 'email'}
onClick={() => onOpenEmail?.()}
className={previewEmail ? 'h-auto py-1.5' : undefined}
@ -776,6 +781,7 @@ export function SidebarContentPanel({
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-meetings"
isActive={activeNav === 'meetings'}
onClick={onOpenMeetings}
className={meetingSublabel ? 'h-auto py-1.5' : undefined}
@ -854,7 +860,7 @@ export function SidebarContentPanel({
</SidebarMenuItem>
{codeModeEnabled && (
<SidebarMenuItem>
<SidebarMenuButton isActive={activeNav === 'code'} onClick={onOpenCode}>
<SidebarMenuButton data-tour-id="nav-code" isActive={activeNav === 'code'} onClick={onOpenCode}>
<Code2 className="size-4 shrink-0" />
<span className="flex-1 truncate">Code</span>
</SidebarMenuButton>
@ -862,6 +868,7 @@ export function SidebarContentPanel({
)}
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-knowledge"
isActive={activeNav === 'knowledge'}
onClick={() => knowledgeActions.openKnowledgeView()}
className={knowledgeUpdatedLabel ? 'h-auto py-1.5' : undefined}
@ -882,6 +889,7 @@ export function SidebarContentPanel({
<SidebarMenu>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-agents"
isActive={activeNav === 'agents'}
onClick={onOpenBgTasks}
className={bgAgentsLabel ? 'h-auto py-1.5' : undefined}
@ -902,6 +910,7 @@ export function SidebarContentPanel({
</SidebarMenuItem>
<SidebarMenuItem>
<SidebarMenuButton
data-tour-id="nav-workspaces"
isActive={activeNav === 'workspaces'}
onClick={() => knowledgeActions.openWorkspaceAt()}
className="h-auto py-1.5"
@ -926,6 +935,7 @@ export function SidebarContentPanel({
<SidebarGroupContent>
<button
type="button"
data-tour-id="nav-chats"
onClick={() => setChatsExpanded((v) => !v)}
className="flex w-full items-center gap-1.5 px-3 py-1 text-[10.5px] font-semibold uppercase tracking-wider text-muted-foreground"
>
@ -1125,6 +1135,15 @@ export function SidebarContentPanel({
</AlertDialog>
)}
</div>
{onStartTour && (
<button
onClick={onStartTour}
className="flex w-full items-center gap-2 rounded-md px-2 py-1 text-xs text-sidebar-foreground/70 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground transition-colors"
>
<MascotFaceIcon className="size-4" />
<span>Take a tour</span>
</button>
)}
<SettingsDialog>
<button className="flex w-full items-center gap-2 rounded-md px-2 py-1 text-xs text-sidebar-foreground/70 hover:bg-sidebar-accent hover:text-sidebar-accent-foreground transition-colors">
<Settings className="size-4" />

487
apps/x/apps/renderer/src/components/talking-head.tsx Normal file
View file

@ -0,0 +1,487 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { X } from 'lucide-react'
import type { TTSState } from '@/hooks/useVoiceTTS'
import { cn } from '@/lib/utils'
const POSITION_STORAGE_KEY = 'talking-head-position'
// Must match the overlay's `bottom-28 right-8` anchor classes.
const ANCHOR_RIGHT_PX = 32
const ANCHOR_BOTTOM_PX = 112
const VIEWPORT_MARGIN_PX = 8
// Palette pulled from the mascot artwork: pale lavender body, dark walnut boat.
const BODY_FILL = '#E8E9F5'
const BODY_STROKE = '#17171B'
const CHEEK_FILL = '#F2B8BE'
const BOAT_DARK = '#3E2E24'
const BOAT_MID = '#54402F'
const BOAT_LIGHT = '#6B5138'
const MOUTH_FILL = '#2A1E19'
export type MascotHat =
| 'mailcap'
| 'headphones'
| 'hardhat'
| 'gradcap'
| 'captain'
| 'explorer'
| 'party'
type TalkingHeadProps = {
ttsState: TTSState
getLevel: () => number
size?: number
/** Costume piece drawn on the head (used by the product tour). */
hat?: MascotHat
/** Paddle hard: fast bobbing + oar strokes, e.g. while traveling in the tour. */
rowing?: boolean
}
/**
* The Rowboat mascot as an animated inline SVG: a round pale character sitting
* in a wooden rowboat holding an oar. The mouth is driven every animation
* frame from the live TTS audio level; eyes blink on a randomized timer.
*/
export function TalkingHead({ ttsState, getLevel, size = 160, hat, rowing = false }: TalkingHeadProps) {
const mouthOpenRef = useRef<SVGEllipseElement>(null)
const mouthSmileRef = useRef<SVGPathElement>(null)
const oarRef = useRef<SVGGElement>(null)
const smoothedRef = useRef(0)
const [blinking, setBlinking] = useState(false)
const speaking = ttsState === 'speaking'
const thinking = ttsState === 'synthesizing'
// Lip sync + oar paddle loop. Writes SVG attributes directly to avoid
// re-rendering React at 60fps. Stops itself once speech has ended and the
// mouth has settled closed; restarts when `speaking` flips this effect.
useEffect(() => {
let raf = 0
let t = 0
const tick = () => {
const target = speaking ? getLevel() : 0
const prev = smoothedRef.current
// Fast attack, slower decay reads as natural mouth movement
const smoothed = target > prev ? prev + (target - prev) * 0.5 : prev + (target - prev) * 0.2
const settled = !speaking && !rowing && smoothed < 0.005
smoothedRef.current = settled ? 0 : smoothed
const open = settled ? 0 : Math.min(1, smoothed * 1.6)
const mouthOpen = mouthOpenRef.current
const mouthSmile = mouthSmileRef.current
if (mouthOpen && mouthSmile) {
if (open > 0.06) {
mouthOpen.setAttribute('rx', String(6.5 + open * 4))
mouthOpen.setAttribute('ry', String(1.5 + open * 9))
mouthOpen.style.opacity = '1'
mouthSmile.style.opacity = '0'
} else {
mouthOpen.style.opacity = '0'
mouthSmile.style.opacity = '1'
}
}
const oar = oarRef.current
if (oar) {
if (rowing) {
t += 0.14
const angle = Math.sin(t) * 13
oar.setAttribute('transform', `rotate(${angle.toFixed(2)} 128 118)`)
} else if (speaking) {
t += 0.045
const angle = Math.sin(t) * 7
oar.setAttribute('transform', `rotate(${angle.toFixed(2)} 128 118)`)
} else {
oar.setAttribute('transform', 'rotate(0 128 118)')
}
}
if (!settled) {
raf = requestAnimationFrame(tick)
}
}
raf = requestAnimationFrame(tick)
return () => cancelAnimationFrame(raf)
}, [speaking, rowing, getLevel])
// Randomized blinking
useEffect(() => {
let timeout: ReturnType<typeof setTimeout>
let cancelled = false
const scheduleBlink = () => {
timeout = setTimeout(() => {
if (cancelled) return
setBlinking(true)
setTimeout(() => {
if (cancelled) return
setBlinking(false)
scheduleBlink()
}, 140)
}, 2400 + Math.random() * 2600)
}
scheduleBlink()
return () => {
cancelled = true
clearTimeout(timeout)
}
}, [])
return (
<div
className="talking-head-bob relative select-none"
style={{
width: size,
height: size,
animationDuration: rowing ? '0.8s' : speaking ? '1.6s' : '3.2s',
}}
>
<style>{`
@keyframes talking-head-bob {
0%, 100% { transform: translateY(0) rotate(-1.6deg); }
50% { transform: translateY(-4px) rotate(1.6deg); }
}
.talking-head-bob {
animation-name: talking-head-bob;
animation-timing-function: ease-in-out;
animation-iteration-count: infinite;
}
@keyframes talking-head-ripple {
0% { transform: scale(0.6); opacity: 0.5; }
100% { transform: scale(1.25); opacity: 0; }
}
.talking-head-ripple {
transform-origin: center;
transform-box: fill-box;
animation: talking-head-ripple 2.6s ease-out infinite;
}
@keyframes talking-head-bubble {
0%, 100% { opacity: 0.25; transform: translateY(0); }
50% { opacity: 1; transform: translateY(-2px); }
}
.talking-head-bubble {
animation: talking-head-bubble 1.2s ease-in-out infinite;
}
`}</style>
<svg viewBox="0 0 200 190" width={size} height={size} aria-hidden="true">
{/* water ripples under the boat */}
<g>
<ellipse className="talking-head-ripple" cx="100" cy="168" rx="62" ry="9" fill="none" stroke="#8FB6D9" strokeWidth="2" style={{ animationDelay: '0s' }} />
<ellipse className="talking-head-ripple" cx="100" cy="168" rx="62" ry="9" fill="none" stroke="#8FB6D9" strokeWidth="2" style={{ animationDelay: '1.3s' }} />
<ellipse cx="100" cy="168" rx="52" ry="7" fill="#8FB6D9" opacity="0.18" />
</g>
{/* thinking bubbles while synthesizing */}
{thinking && (
<g fill={BODY_STROKE} opacity="0.75">
<circle className="talking-head-bubble" cx="146" cy="34" r="3" style={{ animationDelay: '0s' }} />
<circle className="talking-head-bubble" cx="157" cy="26" r="4.2" style={{ animationDelay: '0.2s' }} />
<circle className="talking-head-bubble" cx="170" cy="16" r="5.4" style={{ animationDelay: '0.4s' }} />
</g>
)}
{/* character: head + body blob */}
<g>
<path
d="M 100 22
C 129 22 148 43 148 68
C 148 82 141 93 131 100
C 141 107 147 117 148 128
L 52 128
C 53 115 60 105 69 99
C 59 92 52 81 52 68
C 52 43 71 22 100 22 Z"
fill={BODY_FILL}
stroke={BODY_STROKE}
strokeWidth="5"
strokeLinejoin="round"
/>
{/* eyes */}
<g style={{ transform: thinking ? 'translateY(-2.5px)' : undefined, transition: 'transform 0.3s' }}>
<ellipse
cx="84" cy="64" rx="5" ry={blinking ? 0.8 : 7}
fill={BODY_STROKE}
style={{ transition: 'ry 0.06s' }}
/>
<ellipse
cx="116" cy="64" rx="5" ry={blinking ? 0.8 : 7}
fill={BODY_STROKE}
style={{ transition: 'ry 0.06s' }}
/>
<circle cx="86" cy="61" r="1.6" fill="#FFFFFF" opacity={blinking ? 0 : 0.9} />
<circle cx="118" cy="61" r="1.6" fill="#FFFFFF" opacity={blinking ? 0 : 0.9} />
</g>
{/* cheeks */}
<ellipse cx="72" cy="76" rx="6.5" ry="4" fill={CHEEK_FILL} opacity="0.85" />
<ellipse cx="128" cy="76" rx="6.5" ry="4" fill={CHEEK_FILL} opacity="0.85" />
{/* mouth: smile when quiet, open ellipse driven by audio level */}
<path
ref={mouthSmileRef}
d="M 91 80 Q 100 88 109 80"
fill="none"
stroke={BODY_STROKE}
strokeWidth="4"
strokeLinecap="round"
/>
<ellipse
ref={mouthOpenRef}
cx="100" cy="84" rx="7" ry="2"
fill={MOUTH_FILL}
stroke={BODY_STROKE}
strokeWidth="3"
style={{ opacity: 0 }}
/>
{hat && <MascotHatArt hat={hat} />}
</g>
{/* oar (rotates while speaking) */}
<g ref={oarRef}>
<line x1="158" y1="88" x2="88" y2="152" stroke={BODY_STROKE} strokeWidth="12" strokeLinecap="round" />
<line x1="158" y1="88" x2="88" y2="152" stroke={BOAT_MID} strokeWidth="7" strokeLinecap="round" />
<path
d="M 84 148 L 56 170 C 52 173 52 178 57 178 L 90 165 Z"
fill={BOAT_DARK}
stroke={BODY_STROKE}
strokeWidth="4"
strokeLinejoin="round"
/>
</g>
{/* hand resting over the oar */}
<ellipse cx="121" cy="120" rx="10" ry="8" fill={BODY_FILL} stroke={BODY_STROKE} strokeWidth="4" />
{/* boat hull (drawn last so it overlaps the body) */}
<g>
<path
d="M 30 120
C 50 132 150 132 170 120
C 168 142 152 160 100 160
C 48 160 32 142 30 120 Z"
fill={BOAT_MID}
stroke={BODY_STROKE}
strokeWidth="5"
strokeLinejoin="round"
/>
{/* plank lines */}
<path d="M 36 133 C 60 143 140 143 164 133" fill="none" stroke={BOAT_DARK} strokeWidth="3" strokeLinecap="round" />
<path d="M 44 145 C 66 153 134 153 156 145" fill="none" stroke={BOAT_DARK} strokeWidth="3" strokeLinecap="round" />
{/* gunwale highlight */}
<path d="M 33 121 C 52 131 148 131 167 121" fill="none" stroke={BOAT_LIGHT} strokeWidth="4" strokeLinecap="round" />
</g>
</svg>
</div>
)
}
type TalkingHeadOverlayProps = {
ttsState: TTSState
getLevel: () => number
onDismiss?: () => void
}
// Keep the widget fully on-screen relative to its bottom-right CSS anchor.
// Falls back to the default render size when the element isn't mounted yet.
function clampPositionToViewport(pos: { x: number; y: number }, el: HTMLDivElement | null): { x: number; y: number } {
const width = el?.offsetWidth ?? 160
const height = el?.offsetHeight ?? 160
const baseLeft = window.innerWidth - ANCHOR_RIGHT_PX - width
const baseTop = window.innerHeight - ANCHOR_BOTTOM_PX - height
const clamp = (v: number, min: number, max: number) => Math.max(min, Math.min(max, v))
return {
x: clamp(pos.x, VIEWPORT_MARGIN_PX - baseLeft, window.innerWidth - VIEWPORT_MARGIN_PX - width - baseLeft),
y: clamp(pos.y, VIEWPORT_MARGIN_PX - baseTop, window.innerHeight - VIEWPORT_MARGIN_PX - height - baseTop),
}
}
function loadStoredPosition(): { x: number; y: number } {
try {
const raw = localStorage.getItem(POSITION_STORAGE_KEY)
if (raw) {
const parsed = JSON.parse(raw)
if (typeof parsed?.x === 'number' && typeof parsed?.y === 'number') {
return parsed
}
}
} catch {
// ignore corrupt stored position
}
return { x: 0, y: 0 }
}
/**
* Floating, draggable widget that hosts the talking head. Anchored to the
* bottom-right of the window (above the composer) and offset by a persisted
* drag position, so it hovers over whatever view is active.
*/
export function TalkingHeadOverlay({ ttsState, getLevel, onDismiss }: TalkingHeadOverlayProps) {
// Clamp the stored offset at init so a stale position (e.g. saved on a
// bigger window) can't leave the widget stranded off-screen.
const [offset, setOffset] = useState(() => clampPositionToViewport(loadStoredPosition(), null))
const [dragging, setDragging] = useState(false)
const dragStartRef = useRef<{ pointerX: number; pointerY: number; x: number; y: number } | null>(null)
const containerRef = useRef<HTMLDivElement>(null)
const clampToViewport = useCallback(
(pos: { x: number; y: number }) => clampPositionToViewport(pos, containerRef.current),
[]
)
// Re-clamp when the window shrinks
useEffect(() => {
const handleResize = () => setOffset(prev => clampToViewport(prev))
window.addEventListener('resize', handleResize)
return () => window.removeEventListener('resize', handleResize)
}, [clampToViewport])
const handlePointerDown = useCallback((e: React.PointerEvent<HTMLDivElement>) => {
if (e.button !== 0) return
dragStartRef.current = { pointerX: e.clientX, pointerY: e.clientY, x: offset.x, y: offset.y }
setDragging(true)
e.currentTarget.setPointerCapture(e.pointerId)
}, [offset])
const handlePointerMove = useCallback((e: React.PointerEvent<HTMLDivElement>) => {
const start = dragStartRef.current
if (!start) return
setOffset({
x: start.x + (e.clientX - start.pointerX),
y: start.y + (e.clientY - start.pointerY),
})
}, [])
const handlePointerUp = useCallback((e: React.PointerEvent<HTMLDivElement>) => {
if (!dragStartRef.current) return
dragStartRef.current = null
setDragging(false)
setOffset(prev => clampToViewport(prev))
e.currentTarget.releasePointerCapture(e.pointerId)
}, [clampToViewport])
useEffect(() => {
if (dragging) return
try {
localStorage.setItem(POSITION_STORAGE_KEY, JSON.stringify(offset))
} catch {
// best-effort persistence
}
}, [offset, dragging])
return (
<div
ref={containerRef}
className={cn(
'group fixed bottom-28 right-8 z-50 touch-none',
dragging ? 'cursor-grabbing' : 'cursor-grab'
)}
style={{
transform: `translate(${offset.x}px, ${offset.y}px)`,
// Constant value so the entrance animation runs once on mount and
// never restarts (re-applying it after a drag would replay the pop).
animation: 'talking-head-pop 0.35s cubic-bezier(0.34, 1.56, 0.64, 1)',
}}
onPointerDown={handlePointerDown}
onPointerMove={handlePointerMove}
onPointerUp={handlePointerUp}
onPointerCancel={handlePointerUp}
role="img"
aria-label="Rowboat talking head"
>
<style>{`
@keyframes talking-head-pop {
0% { opacity: 0; scale: 0.4; }
100% { opacity: 1; scale: 1; }
}
`}</style>
{onDismiss && (
<button
type="button"
onPointerDown={(e) => e.stopPropagation()}
onClick={onDismiss}
className="absolute -right-1 -top-1 z-10 flex h-5 w-5 items-center justify-center rounded-full border border-border bg-background text-muted-foreground opacity-0 shadow-sm transition-opacity hover:text-foreground group-hover:opacity-100"
aria-label="Hide talking head"
>
<X className="h-3 w-3" />
</button>
)}
<TalkingHead ttsState={ttsState} getLevel={getLevel} />
</div>
)
}
/** Costume pieces for the product tour, drawn on the mascot's head. */
function MascotHatArt({ hat }: { hat: MascotHat }) {
const outline = { stroke: BODY_STROKE, strokeWidth: 4, strokeLinejoin: 'round' as const }
switch (hat) {
case 'mailcap':
return (
<g>
<path d="M 68 36 Q 100 4 132 36 Q 100 26 68 36 Z" fill="#4A7DDB" {...outline} />
<path d="M 126 33 Q 148 31 150 39 Q 132 42 124 39 Z" fill="#3D68B8" {...outline} />
</g>
)
case 'headphones':
return (
<g>
<path d="M 64 58 Q 100 2 136 58" fill="none" stroke={BODY_STROKE} strokeWidth="11" strokeLinecap="round" />
<path d="M 64 58 Q 100 2 136 58" fill="none" stroke="#4B5563" strokeWidth="6" strokeLinecap="round" />
<ellipse cx="64" cy="61" rx="8" ry="11" fill="#4B5563" {...outline} />
<ellipse cx="136" cy="61" rx="8" ry="11" fill="#4B5563" {...outline} />
</g>
)
case 'hardhat':
return (
<g>
<path d="M 66 38 Q 100 0 134 38 Z" fill="#F6C445" {...outline} />
<path d="M 96 10 Q 94 22 96 36 L 106 36 Q 107 22 105 9 Z" fill="#E0AC2C" stroke="none" />
<path d="M 56 38 L 144 38" fill="none" stroke={BODY_STROKE} strokeWidth="9" strokeLinecap="round" />
<path d="M 58 38 L 142 38" fill="none" stroke="#F6C445" strokeWidth="5" strokeLinecap="round" />
</g>
)
case 'gradcap':
return (
<g>
<path d="M 80 28 Q 100 42 120 28 L 120 38 Q 100 50 80 38 Z" fill="#232838" {...outline} />
<path d="M 100 2 L 144 20 L 100 38 L 56 20 Z" fill="#2E3450" {...outline} />
<path d="M 100 20 Q 124 26 136 34" fill="none" stroke="#E8B94A" strokeWidth="3" strokeLinecap="round" />
<circle cx="137" cy="38" r="4" fill="#E8B94A" stroke={BODY_STROKE} strokeWidth="3" />
</g>
)
case 'captain':
return (
<g>
<path d="M 68 36 Q 100 -2 132 36 Q 100 28 68 36 Z" fill="#F4F5F9" {...outline} />
<path d="M 66 36 Q 100 46 134 36 L 134 42 Q 100 52 66 42 Z" fill="#22262E" {...outline} />
<path d="M 122 42 Q 146 42 148 48 Q 130 52 120 48 Z" fill="#22262E" {...outline} />
<circle cx="100" cy="38" r="4.5" fill="#E8B94A" stroke={BODY_STROKE} strokeWidth="3" />
</g>
)
case 'explorer':
return (
<g>
<ellipse cx="100" cy="35" rx="46" ry="9" fill="#C9A46B" {...outline} />
<path d="M 72 34 Q 100 4 128 34 Z" fill="#C9A46B" {...outline} />
<path d="M 76 30 Q 100 38 124 30" fill="none" stroke="#8A6B3D" strokeWidth="4" strokeLinecap="round" />
</g>
)
case 'party':
return (
<g>
<path d="M 100 -2 L 84 34 L 116 34 Z" fill="#F2699C" {...outline} />
<path d="M 92 16 L 111 22 M 88 26 L 114 31" fill="none" stroke="#FFD166" strokeWidth="3.5" strokeLinecap="round" />
<circle cx="100" cy="0" r="5" fill="#FFD166" stroke={BODY_STROKE} strokeWidth="3" />
</g>
)
}
}
/** Small static mascot face used as the toolbar toggle icon. */
export function MascotFaceIcon({ className }: { className?: string }) {
return (
<svg viewBox="0 0 24 24" width="16" height="16" className={className} aria-hidden="true">
<circle cx="12" cy="12" r="9.5" fill="none" stroke="currentColor" strokeWidth="1.8" />
<ellipse cx="8.6" cy="10.5" rx="1.3" ry="1.8" fill="currentColor" />
<ellipse cx="15.4" cy="10.5" rx="1.3" ry="1.8" fill="currentColor" />
<path d="M 9 14.5 Q 12 17 15 14.5" fill="none" stroke="currentColor" strokeWidth="1.8" strokeLinecap="round" />
</svg>
)
}

262
apps/x/apps/renderer/src/components/tour-vignettes.tsx Normal file
View file

@ -0,0 +1,262 @@
import { useEffect } from 'react'
export type MascotVignetteKind = 'email' | 'meetings' | 'brain'
export type TourVignetteKind = MascotVignetteKind | 'agents'
/**
* Little looping "shows" staged around the mascot while it presents a section
* during the product tour. Purely decorative: everything is pointer-events-none
* and rendered on the tour's own layers, never inside the section's real UI.
*/
export function MascotVignette({ kind, playDing }: { kind: MascotVignetteKind; playDing?: () => void }) {
// One round of dings as the first envelopes land, then let the loop run silent
useEffect(() => {
if (kind !== 'email' || !playDing) return
const timers = [900, 1900, 2900].map((ms) => setTimeout(playDing, ms))
return () => timers.forEach(clearTimeout)
}, [kind, playDing])
return (
<div className="pointer-events-none absolute left-1/2 top-0 z-0 -translate-x-1/2" aria-hidden="true">
<style>{`
@keyframes tour-env-left {
0% { opacity: 0; transform: translate(-150px, -130px) rotate(-26deg); }
10% { opacity: 1; }
52% { opacity: 1; transform: translate(0px, 0px) rotate(5deg); }
64% { opacity: 0; transform: translate(2px, 12px) rotate(5deg); }
100% { opacity: 0; transform: translate(2px, 12px) rotate(5deg); }
}
@keyframes tour-env-right {
0% { opacity: 0; transform: translate(150px, -140px) rotate(26deg); }
10% { opacity: 1; }
52% { opacity: 1; transform: translate(0px, 0px) rotate(-5deg); }
64% { opacity: 0; transform: translate(-2px, 12px) rotate(-5deg); }
100% { opacity: 0; transform: translate(-2px, 12px) rotate(-5deg); }
}
@keyframes tour-badge-pulse {
0%, 100% { transform: scale(1); }
50% { transform: scale(1.15); }
}
@keyframes tour-note-line {
0% { width: 0; }
18% { width: var(--w); }
80% { width: var(--w); opacity: 1; }
92%, 100% { width: var(--w); opacity: 0; }
}
@keyframes tour-quill-bob {
0% { transform: translate(4px, 26px) rotate(-8deg); }
25% { transform: translate(46px, 30px) rotate(4deg); }
50% { transform: translate(6px, 40px) rotate(-8deg); }
75% { transform: translate(48px, 44px) rotate(4deg); }
100% { transform: translate(4px, 26px) rotate(-8deg); }
}
@keyframes tour-voice-bar {
0%, 100% { transform: scaleY(0.35); }
50% { transform: scaleY(1); }
}
@keyframes tour-orb {
0% { opacity: 0; transform: translate(var(--from-x), var(--from-y)) scale(0.5); }
15% { opacity: 0.9; }
70% { opacity: 0.9; transform: translate(0, 0) scale(1); }
85%, 100% { opacity: 0; transform: translate(0, 6px) scale(0.3); }
}
@keyframes tour-node-pulse {
0%, 100% { opacity: 0.55; }
50% { opacity: 1; }
}
@keyframes tour-edge-draw {
from { stroke-dashoffset: 60; }
to { stroke-dashoffset: 0; }
}
`}</style>
{kind === 'email' && (
<div className="relative" style={{ width: 240, height: 150 }}>
{[
{ anim: 'tour-env-left', delay: 0, x: 78, y: 96 },
{ anim: 'tour-env-right', delay: 1.0, x: 128, y: 102 },
{ anim: 'tour-env-left', delay: 2.0, x: 104, y: 92 },
{ anim: 'tour-env-right', delay: 3.0, x: 90, y: 104 },
].map((e, i) => (
<div
key={i}
className="absolute"
style={{ left: e.x, top: e.y, animation: `${e.anim} 4s ease-in-out ${e.delay}s infinite both` }}
>
<svg width="34" height="24" viewBox="0 0 34 24">
<rect x="1.5" y="1.5" width="31" height="21" rx="3" fill="#FFF8E7" stroke="#17171B" strokeWidth="2.5" />
<path d="M 2 4 L 17 14 L 32 4" fill="none" stroke="#17171B" strokeWidth="2.5" strokeLinejoin="round" />
</svg>
</div>
))}
<div
className="absolute"
style={{ left: 148, top: 78, animation: 'tour-badge-pulse 2s ease-in-out infinite' }}
>
<svg width="22" height="22" viewBox="0 0 22 22">
<circle cx="11" cy="11" r="9.5" fill="#3FA95C" stroke="#17171B" strokeWidth="2.5" />
<path d="M 6.5 11.5 L 9.5 14.5 L 15.5 8" fill="none" stroke="#FFFFFF" strokeWidth="2.5" strokeLinecap="round" strokeLinejoin="round" />
</svg>
</div>
</div>
)}
{kind === 'meetings' && (
<div className="relative" style={{ width: 220, height: 160, top: -110 }}>
{/* someone's talking */}
<div className="absolute left-1/2 flex -translate-x-1/2 items-end gap-1" style={{ top: 0, height: 26 }}>
{[0.9, 1.1, 0.7, 1.3, 0.8].map((dur, i) => (
<div
key={i}
className="w-1.5 origin-bottom rounded-full bg-[#8FB6D9]"
style={{ height: 22, animation: `tour-voice-bar ${dur}s ease-in-out ${i * 0.12}s infinite` }}
/>
))}
</div>
{/* ...and the notepad writes itself */}
<div
className="absolute left-1/2 rounded-md border-2 border-[#17171B] bg-[#FFFDF6] shadow-md"
style={{ top: 36, width: 96, height: 104, transform: 'translateX(-50%) rotate(-3deg)' }}
>
<div className="mx-2 mt-2 h-1.5 rounded bg-[#D9534F]/70" />
{[64, 52, 60, 44].map((w, i) => (
<div
key={i}
className="ml-2 mt-3 h-1.5 rounded bg-[#9AA1AE]"
style={{ ['--w' as string]: `${w}px`, animation: `tour-note-line 5s ease-out ${i * 1.1}s infinite both` }}
/>
))}
<svg
className="absolute left-0 top-0"
width="26"
height="26"
viewBox="0 0 26 26"
style={{ animation: 'tour-quill-bob 5s ease-in-out infinite' }}
>
<path d="M 4 22 L 10 12 Q 14 4 22 2 Q 18 10 12 14 Z" fill="#6B5138" stroke="#17171B" strokeWidth="2" strokeLinejoin="round" />
</svg>
</div>
</div>
)}
{kind === 'brain' && (
<div className="relative" style={{ width: 260, height: 180, top: -128 }}>
{/* constellation assembling above the head */}
<svg className="absolute left-1/2 -translate-x-1/2" width="140" height="80" viewBox="0 0 140 80" style={{ top: 0 }}>
{[
'M 22 58 L 52 24',
'M 52 24 L 88 40',
'M 88 40 L 120 18',
'M 88 40 L 70 66',
].map((d, i) => (
<path
key={i}
d={d}
fill="none"
stroke="#9CCBEA"
strokeWidth="2"
strokeDasharray="60"
style={{ animation: `tour-edge-draw 1.2s ease-out ${0.4 + i * 0.35}s both` }}
/>
))}
{[
[22, 58], [52, 24], [88, 40], [120, 18], [70, 66],
].map(([cx, cy], i) => (
<circle
key={i}
cx={cx}
cy={cy}
r={5}
fill="#BFE0F5"
stroke="#17171B"
strokeWidth="2"
style={{ animation: `tour-node-pulse 2.4s ease-in-out ${i * 0.3}s infinite` }}
/>
))}
</svg>
{/* thought-orbs drifting into the head */}
{[
{ fx: '-120px', fy: '-30px', delay: 0 },
{ fx: '120px', fy: '-40px', delay: 1.1 },
{ fx: '-90px', fy: '50px', delay: 2.2 },
{ fx: '110px', fy: '46px', delay: 3.3 },
].map((o, i) => (
<div
key={i}
className="absolute rounded-full"
style={{
left: 122,
top: 118,
width: 16,
height: 16,
background: 'radial-gradient(circle, #FFF3B8 20%, #FFD166 60%, transparent 75%)',
['--from-x' as string]: o.fx,
['--from-y' as string]: o.fy,
animation: `tour-orb 4.4s ease-in-out ${o.delay}s infinite both`,
}}
/>
))}
</div>
)}
</div>
)
}
/**
* Background-agents vignette: a fleet of tiny mascots rowing across the water
* at the bottom of the screen while the big one takes a break.
*/
export function AgentsFleet() {
return (
<div className="pointer-events-none fixed inset-x-0 bottom-0 z-[67] h-28 overflow-hidden" aria-hidden="true">
<style>{`
@keyframes tour-fleet-right {
from { transform: translateX(-18vw); }
to { transform: translateX(112vw); }
}
@keyframes tour-fleet-left {
from { transform: translateX(112vw); }
to { transform: translateX(-18vw); }
}
@keyframes tour-fleet-bob {
0%, 100% { transform: translateY(0) rotate(-2deg); }
50% { transform: translateY(-3px) rotate(2deg); }
}
`}</style>
{[
{ size: 46, bottom: 4, dur: 16, delay: 0, dir: 'tour-fleet-right' },
{ size: 34, bottom: 22, dur: 22, delay: -8, dir: 'tour-fleet-left' },
{ size: 28, bottom: 14, dur: 27, delay: -3, dir: 'tour-fleet-right' },
].map((b, i) => (
<div
key={i}
className="absolute left-0"
style={{ bottom: b.bottom, animation: `${b.dir} ${b.dur}s linear ${b.delay}s infinite` }}
>
<div style={{ animation: 'tour-fleet-bob 1.4s ease-in-out infinite', transform: b.dir === 'tour-fleet-left' ? 'scaleX(-1)' : undefined }}>
<MiniRower size={b.size} flipped={b.dir === 'tour-fleet-left'} />
</div>
</div>
))}
</div>
)
}
function MiniRower({ size, flipped }: { size: number; flipped: boolean }) {
return (
<svg
width={size}
height={size * 0.75}
viewBox="0 0 60 45"
style={{ transform: flipped ? 'scaleX(-1)' : undefined }}
>
<circle cx="27" cy="13" r="10" fill="#E8E9F5" stroke="#17171B" strokeWidth="3" />
<circle cx="23.5" cy="11.5" r="1.4" fill="#17171B" />
<circle cx="30.5" cy="11.5" r="1.4" fill="#17171B" />
<path d="M 23 16 Q 27 19 31 16" fill="none" stroke="#17171B" strokeWidth="1.6" strokeLinecap="round" />
<line x1="40" y1="14" x2="20" y2="38" stroke="#17171B" strokeWidth="4.5" strokeLinecap="round" />
<line x1="40" y1="14" x2="20" y2="38" stroke="#54402F" strokeWidth="2.5" strokeLinecap="round" />
<path d="M 7 25 C 17 31 43 31 53 25 C 51 37 43 42 30 42 C 17 42 9 37 7 25 Z" fill="#54402F" stroke="#17171B" strokeWidth="3" strokeLinejoin="round" />
</svg>
)
}

274
apps/x/apps/renderer/src/components/video-call-view.tsx Normal file
View file

@ -0,0 +1,274 @@
import { useEffect, useRef, useState } from 'react'
import { Mic, MicOff, Minimize2, MonitorUp, PhoneOff, Presentation, Square, User, Video, VideoOff } from 'lucide-react'
import { MascotFaceIcon, TalkingHead } from '@/components/talking-head'
import type { TTSState } from '@/hooks/useVoiceTTS'
import { cn } from '@/lib/utils'
export type VideoCallStatus = 'listening' | 'thinking' | 'speaking'
interface VideoCallViewProps {
/** Live camera stream from useVideoMode — attached to the user's tile. */
streamRef: React.MutableRefObject<MediaStream | null>
cameraOn: boolean
onToggleCamera: () => void
/** User mute = full input pause: no mic audio AND no frame capture. */
micMuted: boolean
onToggleMic: () => void
/** Starting a share collapses this view into the floating popout (the
* surface is derived from devices — see App.tsx). */
onToggleScreenShare: () => void
/** Practice preset: the assistant is coaching this session. */
practiceMode?: boolean
/** Shrink to the floating pill without touching any devices. */
onMinimize: () => void
/** Stop the assistant: silence speech and abort the run if still going. */
onInterrupt: () => void
ttsState: TTSState
/** Live TTS output level — drives the mascot's mouth animation. */
getTtsLevel: () => number
status: VideoCallStatus
/** Live transcript of the user's in-progress utterance. */
interimText?: string
/** The assistant line currently being spoken aloud. */
assistantCaption?: string
onLeave: () => void
}
const STATUS_DISPLAY: Record<VideoCallStatus, { label: string; dotClass: string }> = {
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
}
/**
* Full-screen call surface: a Meet-style two-tile layout with the user's
* webcam on one side and the mascot as the other participant. Shown only
* while the camera is on with no screen share (the derived-surface rule in
* App.tsx) — sharing or muting the camera moves the call into the floating
* popout. The mascot animates with the assistant's speech; dismissing it
* swaps in a Meet-style letter avatar ("R"). Live captions run along the
* bottom.
*/
export function VideoCallView({
streamRef,
cameraOn,
onToggleCamera,
micMuted,
onToggleMic,
onToggleScreenShare,
practiceMode,
onMinimize,
onInterrupt,
ttsState,
getTtsLevel,
status,
interimText,
assistantCaption,
onLeave,
}: VideoCallViewProps) {
const videoRef = useRef<HTMLVideoElement | null>(null)
const [mascotVisible, setMascotVisible] = useState(true)
useEffect(() => {
if (!cameraOn) return
const videoEl = videoRef.current
if (!videoEl) return
videoEl.srcObject = streamRef.current
videoEl.play().catch(() => {})
return () => {
videoEl.srcObject = null
}
}, [streamRef, cameraOn])
const userSpeaking = status === 'listening' && Boolean(interimText)
const assistantSpeaking = ttsState === 'speaking'
const caption = assistantSpeaking && assistantCaption
? { who: 'Rowboat', text: assistantCaption }
: interimText
? { who: 'You', text: interimText }
: null
return (
<div className="fixed inset-0 z-[100] flex flex-col bg-neutral-950">
{practiceMode && (
<span className="absolute left-4 top-4 z-10 flex items-center gap-1.5 rounded-full bg-violet-600/90 px-3 py-1 text-xs font-medium text-white">
<Presentation className="h-3.5 w-3.5" />
Practice session
</span>
)}
<button
type="button"
onClick={onMinimize}
className="absolute right-4 top-4 z-10 flex h-8 w-8 items-center justify-center rounded-full bg-neutral-800 text-white/80 transition-colors hover:bg-neutral-700 hover:text-white"
aria-label="Minimize call (shares your screen)"
title="Minimize — shares your screen so it can help you work"
>
<Minimize2 className="h-4 w-4" />
</button>
{/* Participant tiles */}
<div className="grid min-h-0 flex-1 grid-cols-1 gap-3 p-4 pb-2 md:grid-cols-2">
{/* User */}
<div
className={cn(
'relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
userSpeaking && 'ring-2 ring-green-500/80'
)}
>
{cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-full w-full object-cover"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<span className="flex h-40 w-40 items-center justify-center rounded-full bg-neutral-700 text-neutral-400" aria-label="Camera off">
<User className="h-20 w-20" />
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
You
</span>
{micMuted && (
<span className="absolute bottom-3 right-3 flex items-center gap-1.5 rounded-md bg-red-600/90 px-2 py-0.5 text-sm font-medium text-white">
<MicOff className="h-3.5 w-3.5" />
Muted — nothing is heard or captured
</span>
)}
</div>
{/* Assistant */}
<div
className={cn(
'group relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
assistantSpeaking && 'ring-2 ring-sky-400/80'
)}
>
{mascotVisible ? (
<TalkingHead ttsState={ttsState} getLevel={getTtsLevel} size={220} />
) : (
<span
className="flex h-40 w-40 items-center justify-center rounded-full bg-sky-600 text-7xl font-medium text-white"
aria-label="Rowboat"
>
R
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
Rowboat
</span>
{status !== 'listening' && (
<button
type="button"
onClick={onInterrupt}
className="absolute bottom-3 right-3 flex items-center gap-1.5 rounded-md bg-red-600/90 px-2.5 py-1 text-sm font-medium text-white transition-colors hover:bg-red-500"
aria-label="Stop the assistant"
title={status === 'speaking' ? 'Stop speaking' : 'Stop responding'}
>
<Square className="h-3 w-3 fill-current" />
Stop
</button>
)}
<button
type="button"
onClick={() => setMascotVisible((v) => !v)}
className="absolute right-3 top-3 rounded-md bg-black/50 px-2 py-1 text-xs text-white/80 opacity-0 transition-opacity hover:text-white group-hover:opacity-100"
>
{mascotVisible ? 'Hide mascot' : 'Show mascot'}
</button>
</div>
</div>
{/* Captions */}
<div className="flex h-14 items-center justify-center px-6">
{caption && (
<div className="max-w-3xl truncate rounded-lg bg-black/60 px-4 py-2 text-sm text-white/90">
<span className="mr-2 font-semibold text-white">{caption.who}:</span>
{caption.text}
</div>
)}
</div>
{/* Control bar */}
<div className="flex items-center justify-center gap-4 pb-5">
<span className="flex items-center gap-2 rounded-full bg-neutral-800 px-3 py-1.5 text-xs font-medium text-white/90">
{/* Muted overrides "Listening" — the green pulse would be a lie.
Thinking/speaking still show: output continues while muted. */}
{micMuted && status === 'listening' ? (
<>
<span className="block h-2 w-2 rounded-full bg-red-500" />
Muted
</>
) : (
<>
<span className={cn('block h-2 w-2 rounded-full', STATUS_DISPLAY[status].dotClass)} />
{STATUS_DISPLAY[status].label}
</>
)}
</span>
<button
type="button"
onClick={onToggleMic}
className={cn(
'flex h-10 w-10 items-center justify-center rounded-full transition-colors',
micMuted
? 'bg-red-600 text-white hover:bg-red-500'
: 'bg-neutral-800 text-white/90 hover:bg-neutral-700'
)}
aria-label={micMuted ? 'Unmute' : 'Mute (pauses mic and frame capture)'}
title={micMuted ? 'Unmute' : 'Mute — pauses your mic and all frame capture'}
>
{micMuted ? <MicOff className="h-5 w-5" /> : <Mic className="h-5 w-5" />}
</button>
<button
type="button"
onClick={onToggleCamera}
className={cn(
'flex h-10 w-10 items-center justify-center rounded-full transition-colors',
cameraOn
? 'bg-neutral-800 text-white/90 hover:bg-neutral-700'
: 'bg-red-600 text-white hover:bg-red-500'
)}
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
>
{cameraOn ? <Video className="h-5 w-5" /> : <VideoOff className="h-5 w-5" />}
</button>
<button
type="button"
onClick={onToggleScreenShare}
className="flex h-10 w-10 items-center justify-center rounded-full bg-neutral-800 text-white/90 transition-colors hover:bg-neutral-700"
aria-label="Present your screen"
title="Present your screen"
>
<MonitorUp className="h-5 w-5" />
</button>
<button
type="button"
onClick={() => setMascotVisible((v) => !v)}
className="relative flex h-10 w-10 items-center justify-center rounded-full bg-neutral-800 text-white/90 transition-colors hover:bg-neutral-700"
aria-label={mascotVisible ? 'Hide mascot' : 'Show mascot'}
>
<MascotFaceIcon />
{!mascotVisible && (
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
<span className="block h-[1.5px] w-6 -rotate-45 rounded-full bg-white/80" />
</span>
)}
</button>
<button
type="button"
onClick={onLeave}
className="flex h-10 w-14 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
aria-label="End call"
>
<PhoneOff className="h-5 w-5" />
</button>
</div>
</div>
)
}

239
apps/x/apps/renderer/src/components/video-popout.tsx Normal file
View file

@ -0,0 +1,239 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { Maximize2, Mic, MicOff, MonitorUp, PhoneOff, Square, User, Video, VideoOff } from 'lucide-react'
import { TalkingHead } from '@/components/talking-head'
type PopoutState = {
ttsState: 'idle' | 'synthesizing' | 'speaking'
status: 'listening' | 'thinking' | 'speaking' | null
cameraOn: boolean
/** User mute = full input pause: no mic audio AND no frame capture. */
micMuted: boolean
screenSharing: boolean
interimText: string | null
}
const STATUS_DISPLAY: Record<NonNullable<PopoutState['status']>, { label: string; dotClass: string }> = {
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
}
const dragRegion = { WebkitAppRegion: 'drag' } as React.CSSProperties
const noDragRegion = { WebkitAppRegion: 'no-drag' } as React.CSSProperties
/**
* Content of the always-on-top popout window shown for the whole duration of
* a screen share (Meet-style floating mini-call) — it floats over every app,
* including Rowboat itself, and is the call's control surface while sharing:
* camera toggle, share toggle, end-call. Rendered in its own BrowserWindow
* (see `video:setPopout` in the main process); call state streams in over
* the `video:popout-state` push channel and control actions round-trip back
* through `video:popoutAction`. Captures its own webcam feed — MediaStreams
* can't cross windows.
*/
export function VideoPopout() {
// Camera defaults OFF: guessing "on" would flash the user's video for a
// beat before the real state arrives — which reads as a bug. The true
// state is fetched immediately below.
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: false, micMuted: false, screenSharing: false, interimText: null })
const videoRef = useRef<HTMLVideoElement | null>(null)
useEffect(() => {
const cleanup = window.ipc.on('video:popout-state', (next) => setState(next))
// The main process replays the cached state on did-finish-load, but that
// can race this listener's registration — fetch it explicitly too.
window.ipc
.invoke('video:getPopoutState', null)
.then(({ state: cached }) => {
if (cached) setState(cached)
})
.catch(() => {})
return cleanup
}, [])
// Own camera feed, following the main window's camera-on/off state.
useEffect(() => {
if (!state.cameraOn) return
let stream: MediaStream | null = null
let cancelled = false
navigator.mediaDevices
.getUserMedia({ video: { width: { ideal: 640 }, facingMode: 'user' }, audio: false })
.then((s) => {
if (cancelled) {
s.getTracks().forEach((t) => t.stop())
return
}
stream = s
if (videoRef.current) {
videoRef.current.srcObject = s
videoRef.current.play().catch(() => {})
}
})
.catch((err) => console.error('[popout] camera failed:', err))
return () => {
cancelled = true
stream?.getTracks().forEach((t) => t.stop())
if (videoRef.current) videoRef.current.srcObject = null
}
}, [state.cameraOn])
// The popout has no TTS audio pipeline — synthesize a plausible mouth level
// so the mascot still animates while the assistant speaks in the main window.
const getLevel = useCallback(() => 0.45 + 0.35 * Math.sin(performance.now() / 90), [])
const sendAction = useCallback((action: 'toggle-mic' | 'toggle-camera' | 'toggle-share' | 'stop-speaking' | 'end-call' | 'expand') => {
void window.ipc.invoke('video:popoutAction', { action }).catch(() => {})
}, [])
const statusDisplay = state.status ? STATUS_DISPLAY[state.status] : null
return (
<div
className="relative flex h-screen w-screen select-none flex-col gap-1.5 bg-neutral-900 p-1.5"
style={dragRegion}
>
<div className="flex min-h-0 flex-1 gap-1.5">
<div className="relative flex-1 overflow-hidden rounded-lg bg-neutral-800">
{state.cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-full w-full object-cover"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<div className="flex h-full w-full items-center justify-center">
<span className="flex h-12 w-12 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
<User className="h-6 w-6" />
</span>
</div>
)}
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
You
</span>
{/* Persistent consent badge — the user must always be able to see
at a glance that their screen is going out. Muted pauses frame
capture while keeping the share stream open, so say so. */}
{state.screenSharing && (
<span className="absolute left-1.5 top-1.5 flex items-center gap-1 rounded-full bg-sky-600/90 px-1.5 py-0.5 text-[10px] font-medium text-white">
<span className={`block h-1.5 w-1.5 rounded-full bg-white ${state.micMuted ? '' : 'animate-pulse'}`} />
{state.micMuted ? 'Sharing paused' : 'Sharing screen'}
</span>
)}
{state.micMuted && (
<span className="absolute bottom-1 right-1.5 flex items-center gap-1 rounded bg-red-600/90 px-1.5 py-0.5 text-[10px] font-medium text-white">
<MicOff className="h-2.5 w-2.5" />
Muted
</span>
)}
</div>
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-lg bg-neutral-800">
<TalkingHead ttsState={state.ttsState} getLevel={getLevel} size={84} />
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
Rowboat
</span>
{statusDisplay && (
<span className="absolute right-1.5 top-1.5 flex items-center gap-1 rounded-full bg-black/50 px-1.5 py-0.5 text-[10px] font-medium text-white">
{/* Muted overrides "Listening" — the green pulse would be a lie. */}
{state.micMuted && state.status === 'listening' ? (
<>
<span className="block h-1.5 w-1.5 rounded-full bg-red-500" />
Muted
</>
) : (
<>
<span className={`block h-1.5 w-1.5 rounded-full ${statusDisplay.dotClass}`} />
{statusDisplay.label}
</>
)}
</span>
)}
{(state.status === 'speaking' || state.status === 'thinking') && (
<button
type="button"
onClick={() => sendAction('stop-speaking')}
className="absolute bottom-1 right-1.5 flex items-center gap-1 rounded bg-red-600/90 px-1.5 py-0.5 text-[10px] font-medium text-white hover:bg-red-500"
style={noDragRegion}
aria-label="Stop the assistant"
title={state.status === 'speaking' ? 'Stop speaking' : 'Stop responding'}
>
<Square className="h-2.5 w-2.5 fill-current" />
Stop
</button>
)}
</div>
{/* Live caption of the in-progress utterance, floating over the tiles */}
{state.interimText && (
<div className="pointer-events-none absolute inset-x-1.5 bottom-9 flex justify-center">
<span className="max-w-full truncate rounded bg-black/70 px-1.5 py-0.5 text-[10px] text-white/90">
{state.interimText}
</span>
</div>
)}
</div>
{/* Control bar — actions execute in the main app window */}
<div className="flex h-7 shrink-0 items-center justify-center gap-2" style={noDragRegion}>
<button
type="button"
onClick={() => sendAction('toggle-mic')}
className={`flex h-6 w-6 items-center justify-center rounded-full transition-colors ${
state.micMuted
? 'bg-red-600 text-white hover:bg-red-500'
: 'bg-neutral-700 text-white/90 hover:bg-neutral-600'
}`}
aria-label={state.micMuted ? 'Unmute' : 'Mute (pauses mic and frame capture)'}
title={state.micMuted ? 'Unmute' : 'Mute — pauses your mic and all frame capture'}
>
{state.micMuted ? <MicOff className="h-3.5 w-3.5" /> : <Mic className="h-3.5 w-3.5" />}
</button>
<button
type="button"
onClick={() => sendAction('toggle-camera')}
className={`flex h-6 w-6 items-center justify-center rounded-full transition-colors ${
state.cameraOn
? 'bg-neutral-700 text-white/90 hover:bg-neutral-600'
: 'bg-red-600 text-white hover:bg-red-500'
}`}
aria-label={state.cameraOn ? 'Turn off camera' : 'Turn on camera'}
title={state.cameraOn ? 'Turn off camera' : 'Turn on camera'}
>
{state.cameraOn ? <Video className="h-3.5 w-3.5" /> : <VideoOff className="h-3.5 w-3.5" />}
</button>
<button
type="button"
onClick={() => sendAction('toggle-share')}
className={`flex h-6 w-6 items-center justify-center rounded-full transition-colors ${
state.screenSharing
? 'bg-sky-600 text-white hover:bg-sky-500'
: 'bg-neutral-700 text-white/90 hover:bg-neutral-600'
}`}
aria-label={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
title={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
>
<MonitorUp className="h-3.5 w-3.5" />
</button>
<button
type="button"
onClick={() => sendAction('end-call')}
className="flex h-6 w-8 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
aria-label="End call"
title="End call"
>
<PhoneOff className="h-3.5 w-3.5" />
</button>
<button
type="button"
onClick={() => sendAction('expand')}
className="flex h-6 w-6 items-center justify-center rounded-full bg-neutral-700 text-white/90 transition-colors hover:bg-neutral-600"
aria-label="Expand to full screen (stops screen sharing)"
title="Expand to full screen (stops sharing)"
>
<Maximize2 className="h-3.5 w-3.5" />
</button>
</div>
</div>
)
}

136
apps/x/apps/renderer/src/components/workspace-view.tsx
View file

@ -1,4 +1,4 @@
import { useCallback, useMemo, useRef, useState } from 'react'
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import {
ChevronRight,
Copy,
@ -9,6 +9,8 @@ import {
FolderOpen,
FolderPlus,
Home,
Loader2,
MessageSquare,
Pencil,
Plus,
Trash2,
@ -27,6 +29,7 @@ import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from '@/components/ui/dropdown-menu'
import {
@ -55,10 +58,26 @@ type WorkspaceActions = {
copyPath: (path: string) => void
revealInFileManager: (path: string, isDir: boolean) => void
createNote: (parentPath?: string) => void
addGoogleDoc: (parentPath?: string) => void
createFolder: (parentPath?: string) => Promise<string>
onOpenInNewTab?: (path: string) => void
}
function GoogleDriveIcon({ className }: { className?: string }) {
return (
<svg
viewBox="0 0 24 24"
className={className}
aria-hidden="true"
focusable="false"
>
<path fill="#1FA463" d="M8.52 3.5h6.96l6.95 12.04h-6.96L8.52 3.5Z" />
<path fill="#FFD04B" d="M1.57 15.54 8.52 3.5l3.48 6.02-3.48 6.02H1.57Z" />
<path fill="#4688F1" d="M8.52 15.54h13.91L18.95 21H5.05l3.47-5.46Z" />
</svg>
)
}
type WorkspaceViewProps = {
tree: TreeNode[]
initialPath?: string | null
@ -68,6 +87,24 @@ type WorkspaceViewProps = {
onNavigate: (path: string) => void
onOpenNote: (path: string) => void
onCreateWorkspace: (name: string) => Promise<void>
// Opens a previous chat (run) whose work directory is set to this workspace.
onOpenRun: (runId: string) => void
}
type WorkspaceChat = {
id: string
title?: string
createdAt: string
modifiedAt: string
}
function formatChatAt(iso: string): string {
const d = new Date(iso)
if (Number.isNaN(d.getTime())) return ''
const now = new Date()
const sameDay = d.toDateString() === now.toDateString()
if (sameDay) return d.toLocaleTimeString([], { hour: 'numeric', minute: '2-digit' })
return d.toLocaleDateString([], { month: 'short', day: 'numeric' })
}
function getFileManagerName(): string {
@ -126,9 +163,12 @@ function readFileAsBase64(file: File): Promise<string> {
})
}
export function WorkspaceView({ tree, initialPath, actions, onNavigate, onOpenNote, onCreateWorkspace }: WorkspaceViewProps) {
export function WorkspaceView({ tree, initialPath, actions, onNavigate, onOpenNote, onCreateWorkspace, onOpenRun }: WorkspaceViewProps) {
const currentPath = initialPath || WORKSPACE_ROOT
const [addOpen, setAddOpen] = useState(false)
const [chatsOpen, setChatsOpen] = useState(false)
const [chats, setChats] = useState<WorkspaceChat[]>([])
const [chatsLoading, setChatsLoading] = useState(false)
const [newName, setNewName] = useState('')
const [creating, setCreating] = useState(false)
const [error, setError] = useState<string | null>(null)
@ -165,6 +205,32 @@ export function WorkspaceView({ tree, initialPath, actions, onNavigate, onOpenNo
})
}, [currentPath, isRoot])
// Load the chats whose work directory is this workspace folder (or nested
// inside it). The work directory is stored as an absolute path per run, so
// resolve this folder against the workspace root before querying.
const loadChats = useCallback(async () => {
if (isRoot) {
setChats([])
return
}
setChatsLoading(true)
try {
const { root } = await window.ipc.invoke('workspace:getRoot', null)
const abs = `${root.replace(/\/$/, '')}/${currentPath}`
const { runs } = await window.ipc.invoke('runs:listByWorkDir', { dir: abs })
setChats(runs.map((r) => ({ id: r.id, title: r.title, createdAt: r.createdAt, modifiedAt: r.modifiedAt })))
} catch (err) {
console.error('Failed to load workspace chats:', err)
setChats([])
} finally {
setChatsLoading(false)
}
}, [currentPath, isRoot])
useEffect(() => {
void loadChats()
}, [loadChats])
const handleItemClick = useCallback(
(item: TreeNode) => {
if (renameTarget) return
@ -335,30 +401,48 @@ export function WorkspaceView({ tree, initialPath, actions, onNavigate, onOpenNo
)
})}
</div>
<div className="grid shrink-0 grid-cols-2 items-center gap-2">
<div className="flex shrink-0 items-center gap-2">
{!isRoot && (
<Button
size="sm"
variant={chatsOpen ? 'secondary' : 'outline'}
onClick={() => setChatsOpen((v) => !v)}
>
<MessageSquare className="size-4" />
Chats{chats.length ? ` (${chats.length})` : ''}
</Button>
)}
<Button
size="sm"
variant="outline"
className="w-full"
onClick={() => actions.revealInFileManager(currentPath, true)}
>
<FolderOpen className="size-4" />
Open in {fileManagerName}
</Button>
{isRoot ? (
<Button size="sm" className="w-full" onClick={() => setAddOpen(true)}>
<Button size="sm" onClick={() => setAddOpen(true)}>
<Plus className="size-4" />
Add workspace
</Button>
) : (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button size="sm" className="w-full">
<Button size="sm">
<Plus className="size-4" />
Add
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuItem onClick={() => actions.createNote(currentPath)}>
<FilePlus className="mr-2 size-4" />
New note
</DropdownMenuItem>
<DropdownMenuItem onClick={() => actions.addGoogleDoc(currentPath)}>
<GoogleDriveIcon className="mr-2 size-4" />
Add Google Doc
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem onClick={() => filesInputRef.current?.click()}>
<FilePlus className="mr-2 size-4" />
Add files…
@ -396,6 +480,7 @@ export function WorkspaceView({ tree, initialPath, actions, onNavigate, onOpenNo
}}
/>
<div className="flex flex-1 overflow-hidden">
<div
className="relative flex-1 overflow-y-auto px-6 py-6"
onDragEnter={handleDragEnter}
@ -523,6 +608,45 @@ export function WorkspaceView({ tree, initialPath, actions, onNavigate, onOpenNo
)}
</div>
{!isRoot && chatsOpen && (
<aside className="flex w-72 shrink-0 flex-col overflow-hidden border-l border-border bg-background">
<div className="flex shrink-0 items-center justify-between gap-2 border-b border-border px-4 py-3">
<span className="text-sm font-medium text-foreground">Chats</span>
{chatsLoading && <Loader2 className="size-3.5 animate-spin text-muted-foreground" />}
</div>
<div className="flex-1 overflow-y-auto">
{!chatsLoading && chats.length === 0 ? (
<div className="px-4 py-8 text-center text-xs text-muted-foreground">
No chats use this workspace yet. Set a chat's work directory to this folder and it will appear here.
</div>
) : (
<div className="divide-y divide-border">
{chats.map((chat) => (
<button
key={chat.id}
type="button"
onClick={() => onOpenRun(chat.id)}
className="flex w-full items-center gap-3 px-4 py-2.5 text-left transition-colors hover:bg-accent/40"
>
<MessageSquare className="size-4 shrink-0 text-muted-foreground" />
<div className="flex min-w-0 flex-1 flex-col gap-0.5">
<span className="truncate text-[13px] text-foreground">
{chat.title || '(Untitled chat)'}
</span>
<span className="text-[11px] text-muted-foreground">
{formatChatAt(chat.createdAt)}
</span>
</div>
<ChevronRight className="size-3 shrink-0 text-muted-foreground" />
</button>
))}
</div>
)}
</div>
</aside>
)}
</div>
<Dialog
open={addOpen}
onOpenChange={(open) => {

45
apps/x/apps/renderer/src/hooks/useMeetingTranscription.ts
View file

@ -1,6 +1,7 @@
import { useCallback, useRef, useState } from 'react';
import { toast } from 'sonner';
import { buildDeepgramListenUrl } from '@/lib/deepgram-listen-url';
import { finalizeDeepgramStream } from '@/lib/deepgram-finalize';
import { useRowboatAccount } from '@/hooks/useRowboatAccount';
export type MeetingTranscriptionState = 'idle' | 'connecting' | 'recording' | 'stopping';
@ -196,6 +197,25 @@ export function useMeetingTranscription(onAutoStop?: () => void) {
}, 1000);
}, [writeTranscriptToFile]);
const stopInputCapture = useCallback(() => {
if (processorRef.current) {
processorRef.current.disconnect();
processorRef.current = null;
}
if (audioCtxRef.current) {
audioCtxRef.current.close();
audioCtxRef.current = null;
}
if (micStreamRef.current) {
micStreamRef.current.getTracks().forEach(t => t.stop());
micStreamRef.current = null;
}
if (systemStreamRef.current) {
systemStreamRef.current.getTracks().forEach(t => t.stop());
systemStreamRef.current = null;
}
}, []);
const cleanup = useCallback(() => {
if (writeTimerRef.current) {
clearTimeout(writeTimerRef.current);
@ -213,28 +233,13 @@ export function useMeetingTranscription(onAutoStop?: () => void) {
clearInterval(trackPollingRef.current);
trackPollingRef.current = null;
}
if (processorRef.current) {
processorRef.current.disconnect();
processorRef.current = null;
}
if (audioCtxRef.current) {
audioCtxRef.current.close();
audioCtxRef.current = null;
}
if (micStreamRef.current) {
micStreamRef.current.getTracks().forEach(t => t.stop());
micStreamRef.current = null;
}
if (systemStreamRef.current) {
systemStreamRef.current.getTracks().forEach(t => t.stop());
systemStreamRef.current = null;
}
stopInputCapture();
if (wsRef.current) {
wsRef.current.onclose = null;
wsRef.current.close();
wsRef.current = null;
}
}, []);
}, [stopInputCapture]);
const start = useCallback(async (calendarEvent?: CalendarEventMeta): Promise<string | null> => {
if (state !== 'idle') return null;
@ -551,12 +556,14 @@ export function useMeetingTranscription(onAutoStop?: () => void) {
if (state !== 'recording') return;
setState('stopping');
stopInputCapture();
await finalizeDeepgramStream(wsRef.current, 2200);
cleanup();
interimRef.current = new Map();
await writeTranscriptToFile();
interimRef.current = new Map();
setState('idle');
}, [state, cleanup, writeTranscriptToFile]);
}, [state, cleanup, stopInputCapture, writeTranscriptToFile]);
return { state, start, stop };
}

123
apps/x/apps/renderer/src/hooks/useSessionChat.test.tsx Normal file
View file

@ -0,0 +1,123 @@
import { StrictMode } from 'react'
import { act, renderHook, waitFor } from '@testing-library/react'
import { describe, expect, it } from 'vitest'
import type { SessionBusEvent } from '@x/shared/src/sessions.js'
import type { SessionsClient } from '@/lib/session-chat/client'
import type { SessionFeedListener } from '@/lib/session-chat/feed'
import {
assistantText,
completed,
completedTurnLog,
created,
requested,
sessionState,
turnCompleted,
user,
} from '@/lib/session-chat/test-fixtures'
import { isChatMessage } from '@/lib/chat-conversation'
import { useSessionChat } from './useSessionChat'
const S1 = 'sess-1'
function makeDeps() {
const calls: Array<{ method: string; args: unknown[] }> = []
let emit: SessionFeedListener = () => undefined
let unsubscribed = 0
const sessions = new Map([[S1, sessionState(S1, ['turn-1'])]])
const turns = new Map([['turn-1', completedTurnLog('turn-1', S1, 'q1', 'a1')]])
const client: SessionsClient = {
create: async () => ({ sessionId: 'x' }),
list: async () => ({ sessions: [] }),
get: async (sessionId) => {
const state = sessions.get(sessionId)
if (!state) throw new Error('session not found')
return state
},
getTurn: async (turnId) => ({ turnId, events: turns.get(turnId) ?? [] }),
sendMessage: async (...args) => {
calls.push({ method: 'sendMessage', args })
return { turnId: 'turn-2' }
},
respondToPermission: async (...args) => {
calls.push({ method: 'respondToPermission', args })
},
respondToAskHuman: async (...args) => {
calls.push({ method: 'respondToAskHuman', args })
},
stopTurn: async (...args) => {
calls.push({ method: 'stopTurn', args })
},
resumeTurn: async () => undefined,
setTitle: async () => undefined,
delete: async () => undefined,
}
return {
deps: {
client,
subscribeFeed: (listener: SessionFeedListener) => {
emit = listener
return () => {
unsubscribed += 1
}
},
},
calls,
emit: (event: SessionBusEvent) => emit(event),
getUnsubscribed: () => unsubscribed,
}
}
describe('useSessionChat', () => {
it('seeds from the session, follows live events, and routes actions', async () => {
const { deps, calls, emit } = makeDeps()
const { result } = renderHook(() => useSessionChat(S1, deps), { wrapper: StrictMode })
await waitFor(() => {
expect(result.current.latestTurnId).toBe('turn-1')
})
expect(
result.current.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['q1', 'a1'])
// A new turn streams in over the feed.
act(() => {
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: created('turn-2', S1, user('q2')) })
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: requested('turn-2', 0) })
emit({
kind: 'turn-event',
sessionId: S1,
turnId: 'turn-2',
event: { type: 'text_delta', turnId: 'turn-2', modelCallIndex: 0, delta: 'a2…' },
})
})
expect(result.current.latestTurnId).toBe('turn-2')
expect(result.current.chatState?.currentAssistantMessage).toBe('a2…')
expect(result.current.chatState?.isProcessing).toBe(true)
act(() => {
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: completed('turn-2', 0, assistantText('a2')) })
emit({ kind: 'turn-event', sessionId: S1, turnId: 'turn-2', event: turnCompleted('turn-2', 'a2') })
})
expect(result.current.chatState?.isProcessing).toBe(false)
await act(async () => {
await result.current.respondToPermission('tc1', 'deny')
await result.current.stop()
})
expect(calls).toEqual([
{ method: 'respondToPermission', args: ['turn-2', 'tc1', 'deny', undefined] },
{ method: 'stopTurn', args: ['turn-2'] },
])
})
it('unsubscribes from the feed on unmount (StrictMode double-mounts included)', async () => {
const { deps, getUnsubscribed } = makeDeps()
const { unmount } = renderHook(() => useSessionChat(S1, deps), {
wrapper: StrictMode,
})
unmount()
// StrictMode's simulated cleanup plus the real unmount: every subscribe
// was matched by an unsubscribe.
expect(getUnsubscribed()).toBe(2)
})
})

36
apps/x/apps/renderer/src/hooks/useSessionChat.ts Normal file
View file

@ -0,0 +1,36 @@
import { useEffect, useMemo, useState, useSyncExternalStore } from 'react'
import { ipcSessionsClient } from '@/lib/session-chat/client'
import { subscribeSessionFeed } from '@/lib/session-chat/feed'
import { SessionChatStore, type SessionChatStoreDeps } from '@/lib/session-chat/store'
const defaultDeps: SessionChatStoreDeps = {
client: ipcSessionsClient,
subscribeFeed: subscribeSessionFeed,
}
// Thin subscription over SessionChatStore — all logic (seeding, feed events,
// reducer, overlay, action routing) lives in the store, which is unit-tested
// without React. `deps` is injectable for tests.
export function useSessionChat(
sessionId: string | null,
deps: SessionChatStoreDeps = defaultDeps,
) {
const [store] = useState(() => new SessionChatStore(deps))
useEffect(() => store.connect(), [store])
useEffect(() => {
void store.setSession(sessionId)
}, [store, sessionId])
const snapshot = useSyncExternalStore(store.subscribe, store.getSnapshot)
return useMemo(
() => ({
...snapshot,
sendMessage: store.sendMessage,
respondToPermission: store.respondToPermission,
answerAskHuman: store.answerAskHuman,
stop: store.stop,
}),
[snapshot, store],
)
}
export type SessionChat = ReturnType<typeof useSessionChat>

33
apps/x/apps/renderer/src/hooks/useSessions.ts Normal file
View file

@ -0,0 +1,33 @@
import { useEffect, useMemo, useState, useSyncExternalStore } from 'react'
import { ipcSessionsClient } from '@/lib/session-chat/client'
import { subscribeSessionFeed } from '@/lib/session-chat/feed'
import { SessionListStore, type SessionChatStoreDeps } from '@/lib/session-chat/store'
const defaultDeps: SessionChatStoreDeps = {
client: ipcSessionsClient,
subscribeFeed: subscribeSessionFeed,
}
// The session list (chat history sidebar): seeded from sessions:list, kept
// current by index-changed feed events. Logic lives in SessionListStore.
export function useSessions(deps: SessionChatStoreDeps = defaultDeps) {
const [store] = useState(() => new SessionListStore(deps))
useEffect(() => {
const disconnect = store.connect()
void store.load()
return disconnect
}, [store])
const snapshot = useSyncExternalStore(store.subscribe, store.getSnapshot)
const client = deps.client
return useMemo(
() => ({
...snapshot,
createSession: (input: { title?: string } = {}) => client.create(input),
deleteSession: (sessionId: string) => client.delete(sessionId),
setTitle: (sessionId: string, title: string) => client.setTitle(sessionId, title),
}),
[snapshot, client],
)
}
export type Sessions = ReturnType<typeof useSessions>

326
apps/x/apps/renderer/src/hooks/useVideoMode.ts Normal file
View file

@ -0,0 +1,326 @@
import { useCallback, useEffect, useRef, useState } from 'react';
export type VideoModeState = 'idle' | 'starting' | 'live';
export type ScreenShareState = 'idle' | 'starting' | 'live';
export interface CapturedVideoFrame {
/** base64-encoded JPEG bytes (no data: prefix) — shape of the UserImagePart wire format */
data: string;
mediaType: string;
capturedAt: string; // ISO timestamp
/** data: URL of the same frame, for direct display in the transcript */
dataUrl: string;
source: 'camera' | 'screen';
}
// Frames are grabbed once per second — dense enough to catch expression and
// posture changes while the user talks. Per message we attach at most
// MAX_*_FRAMES_PER_MESSAGE frames, evenly sampled across the window since the
// last send, so long monologues don't balloon the request.
const CAPTURE_INTERVAL_MS = 1000;
const MAX_CAMERA_FRAMES_PER_MESSAGE = 12;
// Screen frames are ~4x the resolution (and tokens) of camera frames, and the
// latest view matters far more than the trajectory — keep the cap small.
const MAX_SCREEN_FRAMES_PER_MESSAGE = 4;
// Rolling buffer bound (~2 minutes). The buffer only needs to cover the gap
// between two sends; anything older is stale context anyway.
const MAX_BUFFERED_FRAMES = 120;
// Downscale targets. 512px wide JPEG keeps a webcam frame around 20-40KB —
// cheap enough to inline a dozen per message as multimodal image parts.
// Screen captures keep 1280px so on-screen text stays legible to the model.
const CAMERA_FRAME_WIDTH = 512;
const SCREEN_FRAME_WIDTH = 1280;
const CAMERA_JPEG_QUALITY = 0.65;
const SCREEN_JPEG_QUALITY = 0.7;
interface BufferedFrame {
dataUrl: string;
capturedAt: string;
ts: number;
}
// One capture pipeline: stream → offscreen <video> → canvas JPEG → ring buffer.
interface CapturePipe {
stream: MediaStream | null;
videoEl: HTMLVideoElement | null;
canvas: HTMLCanvasElement | null;
interval: ReturnType<typeof setInterval> | null;
frames: BufferedFrame[];
lastCollectTs: number;
}
const emptyPipe = (): CapturePipe => ({
stream: null,
videoEl: null,
canvas: null,
interval: null,
frames: [],
lastCollectTs: 0,
});
function capturePipeFrame(pipe: CapturePipe, width: number, quality: number) {
const videoEl = pipe.videoEl;
if (!videoEl || videoEl.readyState < 2 || videoEl.videoWidth === 0) return;
if (!pipe.canvas) {
pipe.canvas = document.createElement('canvas');
}
const canvas = pipe.canvas;
const scale = Math.min(1, width / videoEl.videoWidth);
canvas.width = Math.round(videoEl.videoWidth * scale);
canvas.height = Math.round(videoEl.videoHeight * scale);
const ctx = canvas.getContext('2d');
if (!ctx) return;
ctx.drawImage(videoEl, 0, 0, canvas.width, canvas.height);
const dataUrl = canvas.toDataURL('image/jpeg', quality);
// A near-empty data URL means the frame was blank (source still warming up)
if (dataUrl.length < 100) return;
pipe.frames.push({ dataUrl, capturedAt: new Date().toISOString(), ts: Date.now() });
if (pipe.frames.length > MAX_BUFFERED_FRAMES) {
pipe.frames.splice(0, pipe.frames.length - MAX_BUFFERED_FRAMES);
}
}
function attachPipeSource(pipe: CapturePipe, stream: MediaStream, grab: () => void) {
pipe.stream = stream;
// Offscreen <video> that feeds the capture canvas; any visible preview
// attaches to the same MediaStream separately.
const videoEl = document.createElement('video');
videoEl.muted = true;
videoEl.playsInline = true;
videoEl.srcObject = stream;
pipe.videoEl = videoEl;
videoEl.play().catch(() => {});
// First frame as soon as the source delivers data, then steady-state cadence.
videoEl.addEventListener('loadeddata', () => grab(), { once: true });
pipe.interval = setInterval(grab, CAPTURE_INTERVAL_MS);
}
function teardownPipe(pipe: CapturePipe) {
if (pipe.interval) {
clearInterval(pipe.interval);
pipe.interval = null;
}
if (pipe.videoEl) {
pipe.videoEl.srcObject = null;
pipe.videoEl = null;
}
if (pipe.stream) {
pipe.stream.getTracks().forEach((t) => t.stop());
pipe.stream = null;
}
pipe.frames = [];
pipe.lastCollectTs = 0;
}
/**
* Drain frames captured since the previous collection, evenly sampled down to
* `max` (always keeping the newest). Falls back to the single most recent
* frame when nothing new accumulated (rapid-fire messages), so every message
* carries at least one frame once the source has warmed up.
*/
function drainPipe(pipe: CapturePipe, max: number, source: CapturedVideoFrame['source']): CapturedVideoFrame[] {
const all = pipe.frames;
if (all.length === 0) return [];
let window_ = all.filter((f) => f.ts > pipe.lastCollectTs);
if (window_.length === 0) {
window_ = [all[all.length - 1]];
}
pipe.lastCollectTs = window_[window_.length - 1].ts;
let sampled: BufferedFrame[];
if (window_.length <= max) {
sampled = window_;
} else {
sampled = [];
const step = (window_.length - 1) / (max - 1);
for (let i = 0; i < max; i++) {
sampled.push(window_[Math.round(i * step)]);
}
}
return sampled.map((f) => ({
data: f.dataUrl.slice(f.dataUrl.indexOf(',') + 1),
mediaType: 'image/jpeg',
capturedAt: f.capturedAt,
dataUrl: f.dataUrl,
source,
}));
}
export function useVideoMode() {
const [state, setState] = useState<VideoModeState>('idle');
const [screenState, setScreenState] = useState<ScreenShareState>('idle');
// Camera can be turned off mid-session (Meet-style) while the mode — and
// any screen share — keeps running. Resets to on for the next session.
const [cameraOn, setCameraOn] = useState(true);
// In-call mute pauses capture entirely: nothing lands in the ring buffers
// and collectFrames() returns nothing, so a muted stretch can never ride
// along with a later message. Streams stay open for instant resume.
const capturePausedRef = useRef(false);
const cameraPipeRef = useRef<CapturePipe>(emptyPipe());
const screenPipeRef = useRef<CapturePipe>(emptyPipe());
// Stable stream refs for preview components (<video srcObject>).
const streamRef = useRef<MediaStream | null>(null);
const screenStreamRef = useRef<MediaStream | null>(null);
const stateRef = useRef<VideoModeState>('idle');
stateRef.current = state;
const screenStateRef = useRef<ScreenShareState>('idle');
screenStateRef.current = screenState;
const captureCameraFrame = useCallback(() => {
if (capturePausedRef.current) return;
capturePipeFrame(cameraPipeRef.current, CAMERA_FRAME_WIDTH, CAMERA_JPEG_QUALITY);
}, []);
const captureScreenFrame = useCallback(() => {
if (capturePausedRef.current) return;
capturePipeFrame(screenPipeRef.current, SCREEN_FRAME_WIDTH, SCREEN_JPEG_QUALITY);
}, []);
const setCapturePaused = useCallback((paused: boolean) => {
capturePausedRef.current = paused;
}, []);
const stopScreenShare = useCallback(() => {
teardownPipe(screenPipeRef.current);
screenStreamRef.current = null;
setScreenState('idle');
}, []);
const stop = useCallback(() => {
teardownPipe(cameraPipeRef.current);
streamRef.current = null;
setState('idle');
setCameraOn(true);
capturePausedRef.current = false;
stopScreenShare();
}, [stopScreenShare]);
// Acquire the webcam and start its capture pipeline. Shared by start()
// and by re-enabling the camera mid-session.
const acquireCamera = useCallback(async (): Promise<boolean> => {
// Settle the macOS TCC camera permission before getUserMedia, same as
// voice mode does for the mic — otherwise the first click silently
// fails while the native prompt is still up.
const access = await window.ipc
.invoke('voice:ensureCameraAccess', null)
.catch(() => ({ granted: true }));
if (!access.granted) {
console.error('[video] Camera access denied');
return false;
}
let stream: MediaStream | null = null;
try {
stream = await navigator.mediaDevices.getUserMedia({
video: { width: { ideal: 1280 }, height: { ideal: 720 }, facingMode: 'user' },
audio: false,
});
} catch (err) {
console.error('[video] Camera access failed:', err);
return false;
}
streamRef.current = stream;
attachPipeSource(cameraPipeRef.current, stream, captureCameraFrame);
return true;
}, [captureCameraFrame]);
/**
* Turn the camera off/on without leaving video mode (Meet-style). While
* off, no webcam frames are captured or attached; screen-share frames
* (if presenting) keep flowing.
*/
const setCameraEnabled = useCallback(async (enabled: boolean): Promise<boolean> => {
if (stateRef.current !== 'live') return false;
if (enabled) {
const ok = await acquireCamera();
if (ok) setCameraOn(true);
return ok;
}
teardownPipe(cameraPipeRef.current);
streamRef.current = null;
setCameraOn(false);
return true;
}, [acquireCamera]);
/**
* Start video mode. `camera: false` starts a camera-less session (voice
* call / screen-share-only) — the mode is live so frames can flow from
* other sources, and the camera can be enabled later via setCameraEnabled.
*/
const start = useCallback(async ({ camera = true }: { camera?: boolean } = {}): Promise<boolean> => {
if (stateRef.current !== 'idle') return true;
setState('starting');
if (camera) {
const ok = await acquireCamera();
if (!ok) {
setState('idle');
return false;
}
}
setCameraOn(camera);
setState('live');
return true;
}, [acquireCamera]);
/**
* Share the screen. The main process auto-approves getDisplayMedia with
* the primary screen (see setDisplayMediaRequestHandler in main.ts), so
* no source picker appears. Returns false if capture couldn't start
* (usually the macOS Screen Recording permission).
*/
const startScreenShare = useCallback(async (): Promise<boolean> => {
if (screenStateRef.current !== 'idle') return true;
setScreenState('starting');
// Surfaces the macOS Screen Recording permission state and, on first
// use, registers the app in System Settings (same flow meetings use).
await window.ipc.invoke('meeting:checkScreenPermission', null).catch(() => null);
let stream: MediaStream | null = null;
try {
stream = await navigator.mediaDevices.getDisplayMedia({
video: { frameRate: { ideal: 5 } },
audio: false,
});
} catch (err) {
console.error('[video] Screen share failed:', err);
setScreenState('idle');
return false;
}
screenStreamRef.current = stream;
// The capture can end outside our UI (display unplugged, OS revokes) —
// tear down cleanly so the UI doesn't show a dead share.
stream.getVideoTracks()[0]?.addEventListener('ended', () => stopScreenShare(), { once: true });
attachPipeSource(screenPipeRef.current, stream, captureScreenFrame);
setScreenState('live');
return true;
}, [captureScreenFrame, stopScreenShare]);
/**
* Drain webcam + screen-share frames buffered since the last send, tagged
* by source. Webcam frames come first, then screen frames.
*/
const collectFrames = useCallback((): CapturedVideoFrame[] => {
if (stateRef.current !== 'live') return [];
// Muted: no frames at all — not even pre-mute buffered ones — so a
// typed message during a mute carries nothing captured around it.
if (capturePausedRef.current) return [];
// Grab a frame right now so the message always includes the moment of send.
captureCameraFrame();
const frames = drainPipe(cameraPipeRef.current, MAX_CAMERA_FRAMES_PER_MESSAGE, 'camera');
if (screenStateRef.current === 'live') {
captureScreenFrame();
frames.push(...drainPipe(screenPipeRef.current, MAX_SCREEN_FRAMES_PER_MESSAGE, 'screen'));
}
return frames;
}, [captureCameraFrame, captureScreenFrame]);
// Release the camera/screen if the component unmounts with video mode on.
useEffect(() => stop, [stop]);
return { state, screenState, cameraOn, streamRef, screenStreamRef, start, stop, startScreenShare, stopScreenShare, setCameraEnabled, setCapturePaused, collectFrames };
}

256
apps/x/apps/renderer/src/hooks/useVoiceMode.ts
View file

@ -1,10 +1,11 @@
import { useCallback, useRef, useState } from 'react';
import { buildDeepgramListenUrl } from '@/lib/deepgram-listen-url';
import { finalizeDeepgramStream } from '@/lib/deepgram-finalize';
import { useRowboatAccount } from '@/hooks/useRowboatAccount';
import posthog from 'posthog-js';
import * as analytics from '@/lib/analytics';
export type VoiceState = 'idle' | 'connecting' | 'listening';
export type VoiceState = 'idle' | 'connecting' | 'listening' | 'submitting';
const DEEPGRAM_PARAMS = new URLSearchParams({
model: 'nova-3',
@ -18,7 +19,35 @@ const DEEPGRAM_PARAMS = new URLSearchParams({
endpointing: '100',
no_delay: 'true',
});
const DEEPGRAM_LISTEN_URL = `wss://api.deepgram.com/v1/listen?${DEEPGRAM_PARAMS.toString()}`;
// Hands-free (continuous) mode: Deepgram's endpoint fires FAST (600ms of
// silence) and we apply smart hold logic on our side — if the transcript
// already reads as a complete thought (terminal punctuation) the utterance
// fires immediately, otherwise we hold INCOMPLETE_HOLD_MS longer in case the
// user was mid-thought. Net effect: complete sentences turn around ~1.2s
// faster than the old fixed 1800ms endpoint, while thinking pauses still get
// the same total grace (~1.8s).
const CONTINUOUS_ENDPOINTING_MS = 600;
const INCOMPLETE_HOLD_MS = 1200;
// While the mic is paused (assistant speaking), keep the idle Deepgram socket
// alive — it closes after ~10s without audio otherwise.
const KEEPALIVE_INTERVAL_MS = 5000;
// Deepgram punctuates finals (punctuate=true) — a transcript ending in
// terminal punctuation (optionally inside a closing quote/paren) is treated
// as a complete thought.
const COMPLETE_THOUGHT_RE = /[.!?…]["')\]]*\s*$/;
function deepgramParams(continuous: boolean): URLSearchParams {
if (!continuous) return DEEPGRAM_PARAMS;
const params = new URLSearchParams(DEEPGRAM_PARAMS);
params.set('endpointing', String(CONTINUOUS_ENDPOINTING_MS));
// Second end-of-speech signal: speech_final can be missed (it often rides
// on a result with an empty transcript, or never fires when background
// noise keeps the endpointer engaged). UtteranceEnd is word-timing based
// and arrives as its own message type, so we listen for both.
params.set('utterance_end_ms', '1000');
return params;
}
// Cap on retained per-frame amplitude samples (~64ms/frame ⇒ ~5 min of history).
// The waveform only ever displays the most recent window, so older samples are dropped.
@ -52,6 +81,13 @@ export function useVoiceMode() {
const audioLevelsRef = useRef<number[]>([]);
// Running peak amplitude for the waveform auto-gain (see PEAK_DECAY/MIN_PEAK).
const audioPeakRef = useRef(0);
// Hands-free mode: invoked with each completed utterance (speech_final).
const continuousCbRef = useRef<((text: string) => void) | null>(null);
// While true (assistant is speaking), mic audio is dropped instead of streamed.
const pausedRef = useRef(false);
const keepAliveTimerRef = useRef<ReturnType<typeof setInterval> | null>(null);
// Pending mid-thought hold (smart endpointing) — see maybeEndUtterance.
const holdTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// Refresh cached auth details (called on warmup, not on mic click)
const refreshAuth = useCallback(async () => {
@ -70,9 +106,44 @@ export function useVoiceMode() {
}
}, [refreshRowboatAccount]);
// Hands-free mode: flush the accumulated utterance to the callback.
// Both end-of-speech signals may fire for the same utterance — the second
// finds an empty buffer and is a no-op.
const fireContinuousUtterance = useCallback(() => {
if (holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
if (!continuousCbRef.current || pausedRef.current) return;
const utterance = transcriptBufferRef.current.trim();
transcriptBufferRef.current = '';
interimRef.current = '';
setInterimText('');
if (utterance) continuousCbRef.current(utterance);
}, []);
// Smart endpoint: Deepgram's endpoint fires fast (600ms). If the
// transcript reads as a complete thought, hand it off immediately; if it
// trails off mid-sentence ("so what I want is…"), hold a little longer —
// resumed speech cancels the hold and the utterance keeps growing.
const maybeEndUtterance = useCallback(() => {
if (!continuousCbRef.current || pausedRef.current) return;
const buffered = transcriptBufferRef.current.trim();
if (!buffered) return;
if (COMPLETE_THOUGHT_RE.test(buffered)) {
fireContinuousUtterance();
return;
}
if (holdTimerRef.current) clearTimeout(holdTimerRef.current);
holdTimerRef.current = setTimeout(() => {
holdTimerRef.current = null;
fireContinuousUtterance();
}, INCOMPLETE_HOLD_MS);
}, [fireContinuousUtterance]);
// Create and connect a Deepgram WebSocket using cached auth.
// Starts the connection and returns immediately (does not wait for open).
const connectWs = useCallback(async () => {
const connectWs = useCallback(async (continuous = false) => {
if (wsRef.current && (wsRef.current.readyState === WebSocket.OPEN || wsRef.current.readyState === WebSocket.CONNECTING)) return;
// Refresh auth if we don't have it cached yet
@ -81,12 +152,13 @@ export function useVoiceMode() {
}
if (!cachedAuth) return;
const params = deepgramParams(continuous);
let ws: WebSocket;
if (cachedAuth.type === 'rowboat') {
const listenUrl = buildDeepgramListenUrl(cachedAuth.url, DEEPGRAM_PARAMS);
const listenUrl = buildDeepgramListenUrl(cachedAuth.url, params);
ws = new WebSocket(listenUrl, ['bearer', cachedAuth.token]);
} else {
ws = new WebSocket(DEEPGRAM_LISTEN_URL, ['token', cachedAuth.apiKey]);
ws = new WebSocket(`wss://api.deepgram.com/v1/listen?${params.toString()}`, ['token', cachedAuth.apiKey]);
}
wsRef.current = ws;
@ -102,16 +174,43 @@ export function useVoiceMode() {
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
if (!data.channel?.alternatives?.[0]) return;
// Hands-free mode: word-timing based end-of-speech marker.
if (data.type === 'UtteranceEnd') {
maybeEndUtterance();
return;
}
if (!data.channel?.alternatives?.[0]) return;
const transcript = data.channel.alternatives[0].transcript;
if (!transcript) return;
// The user resumed speaking — cancel any pending mid-thought hold
// so the utterance keeps growing instead of firing under them.
if (transcript && holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
if (data.is_final) {
transcriptBufferRef.current += (transcriptBufferRef.current ? ' ' : '') + transcript;
interimRef.current = '';
setInterimText(transcriptBufferRef.current);
} else {
// NOTE: the endpoint marker (speech_final) usually arrives on a
// result whose transcript is EMPTY — the silence after the user
// stops talking. Empty finals must still reach the speech_final
// check below or hands-free utterances never complete.
if (transcript) {
transcriptBufferRef.current += (transcriptBufferRef.current ? ' ' : '') + transcript;
interimRef.current = '';
}
// Hands-free mode: an endpoint may complete the utterance —
// immediately for complete thoughts, after a short hold for
// mid-sentence trails.
if (continuousCbRef.current && data.speech_final) {
maybeEndUtterance();
return;
}
if (transcript) {
setInterimText(transcriptBufferRef.current);
}
} else if (transcript) {
interimRef.current = transcript;
setInterimText(transcriptBufferRef.current + (transcriptBufferRef.current ? ' ' : '') + transcript);
}
@ -126,11 +225,57 @@ export function useVoiceMode() {
ws.onclose = () => {
console.log('[voice] WebSocket closed');
wsRef.current = null;
// A hands-free call is long-lived — if the socket drops while the
// call is still on, reconnect instead of silently going deaf.
if (continuousCbRef.current) {
setTimeout(() => {
if (continuousCbRef.current && !wsRef.current) {
void connectWs(true);
}
}, 1000);
}
};
}, [refreshAuth]);
}, [refreshAuth, maybeEndUtterance]);
// Stop audio capture and close WS
const stopAudioCapture = useCallback(() => {
const waitForWsOpen = useCallback(async (timeoutMs = 1500): Promise<boolean> => {
const ws = wsRef.current;
if (!ws) return false;
if (ws.readyState === WebSocket.OPEN) return true;
if (ws.readyState !== WebSocket.CONNECTING) return false;
return new Promise<boolean>((resolve) => {
let done = false;
let timeout: ReturnType<typeof setTimeout>;
const finish = (ok: boolean) => {
if (done) return;
done = true;
clearTimeout(timeout);
ws.removeEventListener('open', onOpen);
ws.removeEventListener('error', onError);
ws.removeEventListener('close', onClose);
resolve(ok);
};
const onOpen = () => finish(true);
const onError = () => finish(false);
const onClose = () => finish(false);
timeout = setTimeout(() => finish(false), timeoutMs);
ws.addEventListener('open', onOpen);
ws.addEventListener('error', onError);
ws.addEventListener('close', onClose);
});
}, []);
const flushBufferedAudio = useCallback(() => {
const ws = wsRef.current;
if (!ws || ws.readyState !== WebSocket.OPEN) return;
const buffered = audioBufferRef.current;
audioBufferRef.current = [];
for (const chunk of buffered) {
ws.send(chunk);
}
}, []);
const stopInputCapture = useCallback(() => {
if (processorRef.current) {
processorRef.current.disconnect();
processorRef.current = null;
@ -143,11 +288,26 @@ export function useVoiceMode() {
mediaStreamRef.current.getTracks().forEach(t => t.stop());
mediaStreamRef.current = null;
}
}, []);
// Stop audio capture and close WS
const stopAudioCapture = useCallback(() => {
stopInputCapture();
if (wsRef.current) {
wsRef.current.onclose = null;
wsRef.current.close();
wsRef.current = null;
}
continuousCbRef.current = null;
pausedRef.current = false;
if (holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
if (keepAliveTimerRef.current) {
clearInterval(keepAliveTimerRef.current);
keepAliveTimerRef.current = null;
}
audioBufferRef.current = [];
audioLevelsRef.current = [];
audioPeakRef.current = 0;
@ -155,9 +315,9 @@ export function useVoiceMode() {
transcriptBufferRef.current = '';
interimRef.current = '';
setState('idle');
}, []);
}, [stopInputCapture]);
const start = useCallback(async () => {
const start = useCallback(async (continuous = false) => {
if (state !== 'idle') return;
transcriptBufferRef.current = '';
@ -192,7 +352,7 @@ export function useVoiceMode() {
console.error('Microphone access denied:', err);
return null;
}),
connectWs(),
connectWs(continuous),
]);
if (!stream) {
@ -216,6 +376,9 @@ export function useVoiceMode() {
processorRef.current = processor;
processor.onaudioprocess = (e) => {
// Paused (assistant is speaking in a call): drop mic audio so the
// assistant's own TTS never gets transcribed back at it.
if (pausedRef.current) return;
const float32 = e.inputBuffer.getChannelData(0);
const int16 = new Int16Array(float32.length);
let sumSquares = 0;
@ -240,8 +403,13 @@ export function useVoiceMode() {
if (wsRef.current?.readyState === WebSocket.OPEN) {
wsRef.current.send(buffer);
} else {
// WebSocket still connecting — buffer the audio
// WebSocket still connecting (or reconnecting mid-call) —
// buffer the audio, bounded so an unreachable server during a
// long call can't grow it without limit (~30s at 64ms/chunk).
audioBufferRef.current.push(buffer);
if (audioBufferRef.current.length > 500) {
audioBufferRef.current.shift();
}
}
};
@ -250,25 +418,71 @@ export function useVoiceMode() {
}, [state, connectWs, stopAudioCapture]);
/** Stop recording and return the full transcript (finalized + any current interim) */
const submit = useCallback((): string => {
const submit = useCallback(async (): Promise<string> => {
setState('submitting');
stopInputCapture();
if (wsRef.current?.readyState === WebSocket.CONNECTING) {
await waitForWsOpen();
}
flushBufferedAudio();
await finalizeDeepgramStream(wsRef.current);
let text = transcriptBufferRef.current;
if (interimRef.current) {
text += (text ? ' ' : '') + interimRef.current;
}
text = text.trim();
stopAudioCapture();
return text;
}, [stopAudioCapture]);
}, [flushBufferedAudio, stopAudioCapture, stopInputCapture, waitForWsOpen]);
/** Cancel recording without returning transcript */
const cancel = useCallback(() => {
stopAudioCapture();
}, [stopAudioCapture]);
/**
* Hands-free (call) mode: listen continuously and invoke `onUtterance`
* with each completed utterance. Runs until cancel()/stop.
*/
const startContinuous = useCallback(async (onUtterance: (text: string) => void) => {
continuousCbRef.current = onUtterance;
await start(true);
}, [start]);
/**
* Mute/unmute the continuous stream (used while the assistant is
* thinking/speaking). Keeps the Deepgram socket alive with KeepAlives and
* discards any half-heard utterance from before the pause.
*/
const setPaused = useCallback((paused: boolean) => {
if (pausedRef.current === paused) return;
pausedRef.current = paused;
if (paused) {
if (holdTimerRef.current) {
clearTimeout(holdTimerRef.current);
holdTimerRef.current = null;
}
transcriptBufferRef.current = '';
interimRef.current = '';
setInterimText('');
keepAliveTimerRef.current = setInterval(() => {
if (wsRef.current?.readyState === WebSocket.OPEN) {
wsRef.current.send(JSON.stringify({ type: 'KeepAlive' }));
}
}, KEEPALIVE_INTERVAL_MS);
} else if (keepAliveTimerRef.current) {
clearInterval(keepAliveTimerRef.current);
keepAliveTimerRef.current = null;
}
}, []);
/** Pre-cache auth details so mic click skips IPC round-trips */
const warmup = useCallback(() => {
refreshAuth().catch(() => {});
}, [refreshAuth]);
return { state, interimText, audioLevelsRef, start, submit, cancel, warmup };
return { state, interimText, audioLevelsRef, start, submit, cancel, warmup, startContinuous, setPaused };
}

298
apps/x/apps/renderer/src/hooks/useVoiceTTS.ts
View file

@ -1,4 +1,4 @@
import { useCallback, useRef, useState } from 'react';
import { useCallback, useEffect, useRef, useState } from 'react';
import { dispatchCreditReplenished } from '@/lib/credit-status';
export type TTSState = 'idle' | 'synthesizing' | 'speaking';
@ -18,14 +18,23 @@ function synthesize(text: string): Promise<SynthesizedAudio> {
);
}
function playAudio(dataUrl: string, audioRef: React.MutableRefObject<HTMLAudioElement | null>): Promise<void> {
function playAudio(
dataUrl: string,
audioRef: React.MutableRefObject<HTMLAudioElement | null>,
onAudioElement?: (audio: HTMLAudioElement) => void
): Promise<void> {
return new Promise<void>((resolve, reject) => {
const audio = new Audio(dataUrl);
audioRef.current = audio;
onAudioElement?.(audio);
audio.onended = () => {
console.log('[tts] audio ended');
resolve();
};
// pause() (from cancel) must settle this promise too, or the queue
// loop stays parked on it forever. Natural end also fires 'pause'
// just before 'ended'; double-resolve is harmless.
audio.onpause = () => resolve();
audio.onerror = (e) => {
console.error('[tts] audio error:', e);
reject(new Error('Audio playback failed'));
@ -39,47 +48,282 @@ function playAudio(dataUrl: string, audioRef: React.MutableRefObject<HTMLAudioEl
});
}
/** A queue entry: text to synthesize, or a ready-to-play audio URL (e.g. a bundled clip). */
type QueueItem = { text: string } | { url: string };
type TtsChunkMsg = { requestId: string; chunkBase64?: string; done: boolean; error?: string };
export function useVoiceTTS() {
const [state, setState] = useState<TTSState>('idle');
const audioRef = useRef<HTMLAudioElement | null>(null);
const queueRef = useRef<string[]>([]);
const queueRef = useRef<QueueItem[]>([]);
const processingRef = useRef(false);
// Pre-fetched audio ready to play immediately
const prefetchedRef = useRef<Promise<SynthesizedAudio> | null>(null);
// Streaming synthesis: per-request chunk handlers + the in-flight request
// id (so cancel() can abort the main-process fetch).
const streamHandlersRef = useRef<Map<string, (msg: TtsChunkMsg) => void>>(new Map());
const activeStreamIdRef = useRef<string | null>(null);
// Bumped by cancel(). A queue loop that awaited across a cancel sees a
// stale generation and exits instead of playing audio that was cancelled
// while still synthesizing (which would overlap the next utterance).
const generationRef = useRef(0);
// Web Audio analyser tap for lip-sync (talking head)
const audioCtxRef = useRef<AudioContext | null>(null);
const analyserRef = useRef<AnalyserNode | null>(null);
const levelBufferRef = useRef<Uint8Array<ArrayBuffer> | null>(null);
// Route playback through an AnalyserNode so consumers can read the live
// output level. If Web Audio wiring fails, the element still plays directly.
const connectAnalyser = useCallback((audio: HTMLAudioElement) => {
try {
let ctx = audioCtxRef.current;
if (!ctx) {
ctx = new AudioContext();
audioCtxRef.current = ctx;
const analyser = ctx.createAnalyser();
analyser.fftSize = 512;
analyser.smoothingTimeConstant = 0.5;
analyser.connect(ctx.destination);
analyserRef.current = analyser;
}
if (ctx.state === 'suspended') {
void ctx.resume();
}
const source = ctx.createMediaElementSource(audio);
source.connect(analyserRef.current!);
// Detach once this chunk is done (ended, cancelled via pause, or
// failed) so source nodes don't accumulate over a long session.
const disconnect = () => {
try {
source.disconnect();
} catch {
// already disconnected
}
};
audio.addEventListener('ended', disconnect, { once: true });
audio.addEventListener('pause', disconnect, { once: true });
audio.addEventListener('error', disconnect, { once: true });
} catch (err) {
console.error('[tts] analyser hookup failed:', err);
}
}, []);
// Current output level, 0..1. Safe to call every animation frame.
// Release the audio graph when the owning component unmounts
useEffect(() => () => {
audioCtxRef.current?.close().catch(() => {});
audioCtxRef.current = null;
analyserRef.current = null;
}, []);
// Route streaming TTS chunks to whichever request is waiting for them.
useEffect(() => {
return window.ipc.on('voice:tts-chunk', (msg) => {
streamHandlersRef.current.get(msg.requestId)?.(msg);
});
}, []);
/**
* Streaming synthesis + playback via MediaSource: audio starts on the
* first chunk instead of after the full body. Rejects (for caller
* fallback to non-streaming synth) if the stream fails before any audio
* arrived; resolves when playback finishes.
*/
const streamSynthesizeAndPlay = useCallback((text: string, onStarted: () => void): Promise<void> => {
return new Promise<void>((resolve, reject) => {
if (typeof MediaSource === 'undefined' || !MediaSource.isTypeSupported('audio/mpeg')) {
reject(new Error('MSE audio/mpeg unsupported'));
return;
}
const requestId = `tts-${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
const mediaSource = new MediaSource();
const audio = new Audio();
audio.src = URL.createObjectURL(mediaSource);
audioRef.current = audio;
connectAnalyser(audio);
activeStreamIdRef.current = requestId;
let sourceBuffer: SourceBuffer | null = null;
const pending: Uint8Array[] = [];
let streamDone = false;
let gotAudio = false;
let settled = false;
const cleanup = () => {
streamHandlersRef.current.delete(requestId);
if (activeStreamIdRef.current === requestId) activeStreamIdRef.current = null;
URL.revokeObjectURL(audio.src);
};
const finish = (err?: Error) => {
if (settled) return;
settled = true;
cleanup();
if (err) reject(err);
else resolve();
};
// Drain pending chunks into the SourceBuffer one at a time
// (appendBuffer is async; only one append may be in flight).
const pump = () => {
if (!sourceBuffer || sourceBuffer.updating || settled) return;
const chunk = pending.shift();
if (chunk) {
try {
sourceBuffer.appendBuffer(chunk as BufferSource);
} catch (e) {
finish(e as Error);
}
return;
}
if (streamDone && mediaSource.readyState === 'open') {
try {
mediaSource.endOfStream();
} catch { /* already ended */ }
}
};
mediaSource.addEventListener('sourceopen', () => {
try {
sourceBuffer = mediaSource.addSourceBuffer('audio/mpeg');
} catch (e) {
finish(e as Error);
return;
}
sourceBuffer.addEventListener('updateend', pump);
pump();
}, { once: true });
streamHandlersRef.current.set(requestId, (msg) => {
if (msg.error && !gotAudio) {
streamDone = true;
finish(new Error(msg.error));
return;
}
if (msg.chunkBase64) {
gotAudio = true;
const bin = atob(msg.chunkBase64);
const bytes = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
pending.push(bytes);
pump();
}
if (msg.done) {
streamDone = true;
pump();
}
});
audio.addEventListener('playing', () => onStarted(), { once: true });
audio.onended = () => finish();
// pause() (from cancel) must settle this promise too; natural end
// also fires 'pause' just before 'ended'; double-settle is a no-op.
audio.onpause = () => finish();
audio.onerror = () => finish(new Error('stream playback failed'));
window.ipc
.invoke('voice:synthesizeStreamStart', { requestId, text })
.then((res) => {
if (!res.ok) finish(new Error(res.error || 'stream start failed'));
})
.catch((e) => finish(e as Error));
// Starts as soon as the first appended data is decodable.
audio.play().catch(() => { /* surfaced via onerror / chunk error */ });
// Nothing arrived at all — bail so the caller can fall back.
setTimeout(() => {
if (!gotAudio && !settled) finish(new Error('stream timeout'));
}, 10_000);
});
}, [connectAnalyser]);
const getLevel = useCallback((): number => {
const analyser = analyserRef.current;
if (!analyser) return 0;
let buffer = levelBufferRef.current;
if (!buffer || buffer.length !== analyser.fftSize) {
buffer = new Uint8Array(analyser.fftSize);
levelBufferRef.current = buffer;
}
analyser.getByteTimeDomainData(buffer);
let sum = 0;
for (let i = 0; i < buffer.length; i++) {
const d = (buffer[i] - 128) / 128;
sum += d * d;
}
const rms = Math.sqrt(sum / buffer.length);
return Math.min(1, rms * 4);
}, []);
const processQueue = useCallback(async () => {
if (processingRef.current) return;
processingRef.current = true;
const gen = generationRef.current;
// Kick off full-body pre-fetch for the next queued text while the
// current one plays — keeps sentence-to-sentence playback gapless.
const prefetchNext = () => {
const next = queueRef.current[0];
if (next && 'text' in next && next.text.trim() && !prefetchedRef.current) {
console.log('[tts] pre-fetching next:', next.text.substring(0, 80));
prefetchedRef.current = synthesize(next.text);
}
};
while (queueRef.current.length > 0) {
const text = queueRef.current.shift()!;
if (!text.trim()) continue;
const item = queueRef.current.shift()!;
if ('text' in item && !item.text.trim()) continue;
// Cold start (nothing playing, nothing pre-fetched): stream the
// synthesis so audio begins on the first chunk instead of after
// the full body — this is where first-response latency lives.
if ('text' in item && !prefetchedRef.current) {
setState('synthesizing');
console.log('[tts] stream-synthesizing:', item.text.substring(0, 80));
try {
await streamSynthesizeAndPlay(item.text, () => {
if (generationRef.current !== gen) return;
setState('speaking');
prefetchNext();
});
if (generationRef.current !== gen) return;
continue;
} catch (err) {
if (generationRef.current !== gen) return;
console.error('[tts] stream failed, falling back to full synth:', err);
// fall through to the non-streaming path below
}
}
try {
// Use pre-fetched result if available, otherwise synthesize now
// Pre-recorded URL plays as-is; text uses the pre-fetched
// result if available, otherwise synthesizes now.
let audioPromise: Promise<SynthesizedAudio>;
if (prefetchedRef.current) {
if ('url' in item) {
audioPromise = Promise.resolve({ dataUrl: item.url });
} else if (prefetchedRef.current) {
console.log('[tts] using pre-fetched audio');
audioPromise = prefetchedRef.current;
prefetchedRef.current = null;
} else {
setState('synthesizing');
console.log('[tts] synthesizing:', text.substring(0, 80));
audioPromise = synthesize(text);
console.log('[tts] synthesizing:', item.text.substring(0, 80));
audioPromise = synthesize(item.text);
}
const audio = await audioPromise;
// Cancelled while synthesizing — cancel() already reset all
// state (and a new loop may be running), so just bail.
if (generationRef.current !== gen) return;
setState('speaking');
// Kick off pre-fetch for next chunk while this one plays
const nextText = queueRef.current[0];
if (nextText?.trim()) {
console.log('[tts] pre-fetching next:', nextText.substring(0, 80));
prefetchedRef.current = synthesize(nextText);
}
prefetchNext();
await playAudio(audio.dataUrl, audioRef);
await playAudio(audio.dataUrl, audioRef, connectAnalyser);
if (generationRef.current !== gen) return;
} catch (err) {
if (generationRef.current !== gen) return;
console.error('[tts] error:', err);
prefetchedRef.current = null;
}
@ -89,17 +333,33 @@ export function useVoiceTTS() {
prefetchedRef.current = null;
processingRef.current = false;
setState('idle');
}, []);
}, [connectAnalyser, streamSynthesizeAndPlay]);
const speak = useCallback((text: string) => {
console.log('[tts] speak() called:', text.substring(0, 80));
queueRef.current.push(text);
queueRef.current.push({ text });
processQueue();
}, [processQueue]);
// Play a pre-recorded clip (e.g. bundled tour narration) through the same
// queue, so lip-sync levels, state, and cancel() all work unchanged.
const speakUrl = useCallback((url: string) => {
console.log('[tts] speakUrl() called:', url.substring(0, 120));
queueRef.current.push({ url });
processQueue();
}, [processQueue]);
const cancel = useCallback(() => {
generationRef.current++;
queueRef.current = [];
prefetchedRef.current = null;
// Abort any in-flight streaming synthesis in the main process.
if (activeStreamIdRef.current) {
void window.ipc
.invoke('voice:synthesizeStreamCancel', { requestId: activeStreamIdRef.current })
.catch(() => {});
activeStreamIdRef.current = null;
}
if (audioRef.current) {
audioRef.current.pause();
audioRef.current = null;
@ -108,5 +368,5 @@ export function useVoiceTTS() {
setState('idle');
}, []);
return { state, speak, cancel };
return { state, speak, speakUrl, cancel, getLevel };
}

28
apps/x/apps/renderer/src/lib/agent-transcript.test.ts Normal file
View file

@ -0,0 +1,28 @@
import { describe, expect, it } from 'vitest'
import { turnToTranscript } from './agent-transcript'
import { completedTurnLog, created, requested, user } from './session-chat/test-fixtures'
describe('turnToTranscript', () => {
it('maps a completed turn to id/createdAt/summary/items', () => {
const events = completedTurnLog('turn-1', 'sess-1', 'do the thing', 'all done')
const t = turnToTranscript('turn-1', events)
expect(t.id).toBe('turn-1')
expect(t.createdAt).toBeDefined()
expect(t.summary).toBe('all done')
expect(t.error).toBeUndefined()
expect(t.trigger).toBeUndefined()
expect(t.items.length).toBeGreaterThan(0)
})
it('surfaces turn_failed as error with no summary', () => {
const events = [
created('turn-2', 'sess-1', user('go')),
requested('turn-2', 0),
{ type: 'model_call_failed' as const, turnId: 'turn-2', ts: '2026-07-02T10:00:00Z', modelCallIndex: 0, error: 'boom' },
{ type: 'turn_failed' as const, turnId: 'turn-2', ts: '2026-07-02T10:00:00Z', error: 'boom', usage: {} },
]
const t = turnToTranscript('turn-2', events)
expect(t.error).toBe('boom')
expect(t.summary).toBeUndefined()
})
})

89
apps/x/apps/renderer/src/lib/agent-transcript.ts Normal file
View file

@ -0,0 +1,89 @@
import type { z } from 'zod'
import type { Run } from '@x/shared/src/runs.js'
import { reduceTurn, type TurnEvent } from '@x/shared/src/turns.js'
import type { ConversationItem } from '@/lib/chat-conversation'
import { runLogToConversation } from '@/lib/run-to-conversation'
import { buildTurnConversation } from '@/lib/session-chat/turn-view'
// Unified read model for a headless agent run's transcript, whether the id
// is a turn (new runtime) or a legacy run (pre-turn-runtime files under
// $WorkDir/runs/). Used by the background-tasks and live-note history views.
export interface AgentRunTranscript {
id: string
createdAt?: string
// Legacy runs only (subUseCase); turns carry the trigger inside the
// message text instead.
trigger?: string
summary?: string
error?: string
items: ConversationItem[]
}
export function turnToTranscript(
turnId: string,
events: Array<z.infer<typeof TurnEvent>>,
): AgentRunTranscript {
const state = reduceTurn(events)
const out: AgentRunTranscript = {
id: turnId,
createdAt: state.definition.ts,
items: buildTurnConversation(state),
}
if (state.terminal?.type === 'turn_failed') {
out.error = state.terminal.error
}
for (let i = state.modelCalls.length - 1; i >= 0; i--) {
const response = state.modelCalls[i].response
if (!response) continue
const content = response.content
const text =
typeof content === 'string'
? content
: content.map((p) => (p.type === 'text' ? p.text : '')).join('')
if (text) {
out.summary = text
break
}
}
return out
}
export function runToTranscript(run: z.infer<typeof Run>): AgentRunTranscript {
const out: AgentRunTranscript = {
id: run.id,
createdAt: run.createdAt,
trigger: run.subUseCase,
items: runLogToConversation(run.log),
}
for (const event of run.log) {
if (event.type === 'error' && typeof event.error === 'string') {
out.error = event.error
} else if (event.type === 'message' && event.message?.role === 'assistant') {
const content = event.message.content
if (typeof content === 'string') {
out.summary = content
} else if (Array.isArray(content)) {
const text = content
.filter((p) => p.type === 'text')
.map((p) => ('text' in p ? p.text : ''))
.join('')
if (text) out.summary = text
}
}
}
return out
}
// Turn-first with a legacy-run fallback so histories recorded before the
// turn-runtime migration stay readable. The fallback goes away with the
// runs runtime (stage 7).
export async function fetchAgentRunTranscript(id: string): Promise<AgentRunTranscript> {
try {
const turn = await window.ipc.invoke('sessions:getTurn', { turnId: id })
return turnToTranscript(id, turn.events)
} catch {
const run = await window.ipc.invoke('runs:fetch', { runId: id })
return runToTranscript(run)
}
}

20
apps/x/apps/renderer/src/lib/analytics.ts
View file

@ -65,6 +65,26 @@ export function voiceInputStarted() {
posthog.capture('voice_input_started')
}
export function callStarted(preset: 'voice' | 'video' | 'share' | 'practice') {
posthog.capture('call_started', { preset })
}
// Voice-to-voice latency breakdown for one call turn (all milliseconds):
// utterance accepted → message submitted → first TTS speak() → audio playing.
export function callTurnLatency(props: {
endpointToSubmitMs: number
submitToSpeakMs: number
speakToAudioMs: number
totalMs: number
}) {
posthog.capture('call_turn_latency', {
endpoint_to_submit_ms: Math.round(props.endpointToSubmitMs),
submit_to_speak_ms: Math.round(props.submitToSpeakMs),
speak_to_audio_ms: Math.round(props.speakToAudioMs),
total_ms: Math.round(props.totalMs),
})
}
export function searchExecuted(types: string[]) {
posthog.capture('search_executed', { types })
}

30
apps/x/apps/renderer/src/lib/call-sounds.ts Normal file
View file

@ -0,0 +1,30 @@
// Tiny synthesized UI sounds for calls — no audio assets, one lazy context.
let ctx: AudioContext | null = null
/**
* Soft rising blip played the instant an utterance is accepted — sub-second
* acknowledgment makes the (still ongoing) model turn feel responsive
* instead of dead air.
*/
export function playAckCue() {
try {
if (!ctx) ctx = new AudioContext()
if (ctx.state === 'suspended') void ctx.resume()
const t = ctx.currentTime
const osc = ctx.createOscillator()
const gain = ctx.createGain()
osc.type = 'sine'
osc.frequency.setValueAtTime(880, t)
osc.frequency.exponentialRampToValueAtTime(1320, t + 0.08)
gain.gain.setValueAtTime(0.0001, t)
gain.gain.exponentialRampToValueAtTime(0.08, t + 0.015)
gain.gain.exponentialRampToValueAtTime(0.0001, t + 0.12)
osc.connect(gain)
gain.connect(ctx.destination)
osc.start(t)
osc.stop(t + 0.13)
} catch {
// cosmetic — never let a sound failure affect the call
}
}

62
apps/x/apps/renderer/src/lib/chat-conversation.ts
View file

@ -10,6 +10,9 @@ export interface MessageAttachment {
mimeType: string
size?: number
thumbnailUrl?: string
/** Live webcam frame from video chat mode — rendered as a compact filmstrip.
* Carries no path; thumbnailUrl holds the frame as a data: URL. */
isVideoFrame?: boolean
}
export interface ChatMessage {
@ -118,6 +121,19 @@ export const normalizeToolOutput = (
return output
}
export const getToolErrorText = (tool: ToolCall): string | undefined => {
if (tool.status !== 'error') return undefined
if (typeof tool.result === 'string' && tool.result.trim()) return tool.result
if (tool.result !== undefined) {
try {
return JSON.stringify(tool.result, null, 2)
} catch {
// Fall through to the generic label for non-serializable legacy values.
}
}
return 'Tool error'
}
export type WebSearchCardResult = { title: string; url: string; description: string }
export type WebSearchCardData = {
@ -198,6 +214,22 @@ const summarizeFilterUpdates = (updates: Record<string, unknown>): string => {
return parts.length > 0 ? parts.join(', ') : 'Updated view'
}
const APP_VIEW_LABELS: Record<string, string> = {
home: 'home',
email: 'email',
meetings: 'meetings',
'live-notes': 'live notes',
'bg-tasks': 'background agents',
'chat-history': 'chat history',
knowledge: 'knowledge',
workspace: 'workspace',
code: 'code',
bases: 'bases',
graph: 'graph',
}
const appViewLabel = (view: unknown): string => APP_VIEW_LABELS[view as string] ?? String(view ?? 'view')
export const getAppActionCardData = (tool: ToolCall): AppActionCardData | null => {
if (tool.name !== 'app-navigation') return null
const result = tool.result as Record<string, unknown> | undefined
@ -209,7 +241,9 @@ export const getAppActionCardData = (tool: ToolCall): AppActionCardData | null =
const action = input.action as string
switch (action) {
case 'open-note': return { action, label: `Opening ${(input.path as string || '').split('/').pop()?.replace(/\.md$/, '') || 'note'}...` }
case 'open-view': return { action, label: `Opening ${input.view} view...` }
case 'open-view': return { action, label: `Opening ${appViewLabel(input.view)}...` }
case 'read-view': return { action, label: `Reading ${appViewLabel(input.view)}...` }
case 'open-item': return { action, label: 'Opening...' }
case 'update-base-view': return { action, label: 'Updating view...' }
case 'create-base': return { action, label: `Creating "${input.name}"...` }
case 'get-base-state': return null // renders as normal tool block
@ -224,7 +258,31 @@ export const getAppActionCardData = (tool: ToolCall): AppActionCardData | null =
return { action: 'open-note', label: `Opened ${name}` }
}
case 'open-view':
return { action: 'open-view', label: `Opened ${result.view} view` }
return { action: 'open-view', label: `Opened ${appViewLabel(result.view)}` }
case 'read-view': {
const counted =
(result.threads as unknown[] | undefined)?.length ??
(result.agents as unknown[] | undefined)?.length ??
(result.sessions as unknown[] | undefined)?.length
return {
action: 'read-view',
label: counted !== undefined
? `Read ${appViewLabel(result.view)} (${counted} item${counted === 1 ? '' : 's'})`
: `Read ${appViewLabel(result.view)}`,
}
}
case 'open-item': {
switch (result.kind) {
case 'email-thread': return { action: 'open-item', label: 'Opened email thread' }
case 'note': {
const name = (result.path as string || '').split('/').pop()?.replace(/\.md$/, '') || 'note'
return { action: 'open-item', label: `Opened ${name}` }
}
case 'bg-task': return { action: 'open-item', label: `Opened agent "${result.taskName}"` }
case 'session': return { action: 'open-item', label: 'Opened chat' }
default: return { action: 'open-item', label: 'Opened item' }
}
}
case 'update-base-view':
return {
action: 'update-base-view',

63
apps/x/apps/renderer/src/lib/code-mode-provisioning.ts Normal file
View file

@ -0,0 +1,63 @@
import { useEffect, useState } from "react"
export type AgentStatus = { installed: boolean; signedIn: boolean }
export type CodeModeAgentStatus = { claude: AgentStatus; codex: AgentStatus }
// Engine provisioning runs in the main process and keeps going even if the UI that
// started it (the Settings dialog OR the onboarding step) unmounts. Track its state at
// MODULE level — shared across both — so whichever view is mounted shows the live %
// instead of restarting. This is what lets a download kicked off from onboarding show
// its progress later in Settings → Code Mode. A single persistent listener on the
// progress channel feeds this store.
export type ProvState = { pct: number | null; error?: string }
const provStore: Record<string, ProvState | undefined> = {}
// Agents we provisioned this session — used to show "Ready" immediately on success
// without waiting for the async status refresh to round-trip (which caused the row to
// briefly flash the Enable button again).
export const enabledOptimistic = new Set<string>()
const provListeners = new Set<() => void>()
let provChannelHooked = false
function notifyProv() { provListeners.forEach((l) => l()) }
export function startProvisioning(agent: 'claude' | 'codex', onDone: () => void | Promise<void>): void {
if (provStore[agent] && !provStore[agent]!.error) return // already in flight
provStore[agent] = { pct: null }
notifyProv()
if (!provChannelHooked) {
provChannelHooked = true
window.ipc.on('codeMode:engineProgress', (p) => {
const cur = provStore[p.agent]
if (!cur) return
const pct = p.totalBytes ? Math.floor(((p.receivedBytes ?? 0) / p.totalBytes) * 100) : cur.pct
provStore[p.agent] = { pct }
notifyProv()
})
}
window.ipc.invoke('codeMode:provisionEngine', { agent })
.then((res) => {
if (res.success) {
// Mark installed optimistically so the row shows "Ready" the instant the flag
// clears — don't depend on the async status refresh (which re-renders the parent
// separately and left a window showing the Enable button). loadStatus still runs
// in the background to sync the real status.
enabledOptimistic.add(agent)
provStore[agent] = undefined
void onDone()
} else {
provStore[agent] = { pct: null, error: res.error ?? 'Failed to enable' }
}
})
.catch((e) => { provStore[agent] = { pct: null, error: e instanceof Error ? e.message : 'Failed to enable' } })
.finally(notifyProv)
}
export function useProvisioning(agent: string): ProvState | undefined {
const [, force] = useState(0)
useEffect(() => {
const l = () => force((n) => n + 1)
provListeners.add(l)
return () => { provListeners.delete(l) }
}, [])
return provStore[agent]
}

33
apps/x/apps/renderer/src/lib/deepgram-finalize.ts Normal file
View file

@ -0,0 +1,33 @@
export async function finalizeDeepgramStream(ws: WebSocket | null, timeoutMs = 1800): Promise<void> {
if (!ws || ws.readyState !== WebSocket.OPEN) return;
await new Promise<void>((resolve) => {
let done = false;
let timeout: ReturnType<typeof setTimeout>;
const finish = () => {
if (done) return;
done = true;
clearTimeout(timeout);
ws.removeEventListener('message', onMessage);
resolve();
};
timeout = setTimeout(finish, timeoutMs);
const onMessage = (event: MessageEvent) => {
try {
const data = JSON.parse(event.data);
if (data?.is_final || data?.speech_final || data?.type === 'UtteranceEnd') {
finish();
}
} catch {
// Ignore non-JSON control frames.
}
};
ws.addEventListener('message', onMessage);
try {
ws.send(JSON.stringify({ type: 'Finalize' }));
} catch {
finish();
}
});
}

28
apps/x/apps/renderer/src/lib/run-to-conversation.ts
View file

@ -41,6 +41,9 @@ export function runLogToConversation(log: RunLog): ConversationItem[] {
filename?: string
mimeType?: string
size?: number
data?: string
mediaType?: string
source?: string
toolCallId?: string
toolName?: string
arguments?: unknown
@ -52,13 +55,24 @@ export function runLogToConversation(log: RunLog): ConversationItem[] {
.join('')
const attachmentParts = parts.filter((p) => p.type === 'attachment' && p.path)
if (attachmentParts.length > 0) {
msgAttachments = attachmentParts.map((p) => ({
path: p.path!,
filename: p.filename || p.path!.split('/').pop() || p.path!,
mimeType: p.mimeType || 'application/octet-stream',
size: p.size,
}))
// Video-mode webcam frames — inline base64 image parts, shown as a filmstrip
const imageParts = parts.filter((p) => p.type === 'image' && p.data)
if (attachmentParts.length > 0 || imageParts.length > 0) {
msgAttachments = [
...attachmentParts.map((p) => ({
path: p.path!,
filename: p.filename || p.path!.split('/').pop() || p.path!,
mimeType: p.mimeType || 'application/octet-stream',
size: p.size,
})),
...imageParts.map((p, index) => ({
path: '',
filename: `${p.source === 'screen' ? 'screen' : 'camera'}-frame-${index + 1}.jpg`,
mimeType: p.mediaType || 'image/jpeg',
thumbnailUrl: `data:${p.mediaType || 'image/jpeg'};base64,${p.data}`,
isVideoFrame: true,
})),
]
}
if (msg.role === 'assistant') {

70
apps/x/apps/renderer/src/lib/session-chat/client.ts Normal file
View file

@ -0,0 +1,70 @@
import type { z } from 'zod'
import type { UserMessage } from '@x/shared/src/message.js'
import type { SessionIndexEntry, SessionState } from '@x/shared/src/sessions.js'
import type { JsonValue, RequestedAgent, TurnEvent } from '@x/shared/src/turns.js'
// Narrow, injectable surface over the sessions IPC channels so stores are
// testable with a plain fake instead of a window.ipc stub.
export interface SendMessageConfig {
agent: z.infer<typeof RequestedAgent>
autoPermission?: boolean
maxModelCalls?: number
}
export interface SessionsClient {
create(input: { title?: string }): Promise<{ sessionId: string }>
list(): Promise<{ sessions: SessionIndexEntry[] }>
get(sessionId: string): Promise<SessionState>
getTurn(turnId: string): Promise<{ turnId: string; events: Array<z.infer<typeof TurnEvent>> }>
sendMessage(
sessionId: string,
input: z.infer<typeof UserMessage>,
config: SendMessageConfig,
): Promise<{ turnId: string }>
respondToPermission(
turnId: string,
toolCallId: string,
decision: 'allow' | 'deny',
metadata?: JsonValue,
): Promise<void>
respondToAskHuman(turnId: string, toolCallId: string, answer: string): Promise<void>
stopTurn(turnId: string, reason?: string): Promise<void>
resumeTurn(sessionId: string): Promise<void>
setTitle(sessionId: string, title: string): Promise<void>
delete(sessionId: string): Promise<void>
}
export const ipcSessionsClient: SessionsClient = {
create: (input) => window.ipc.invoke('sessions:create', input),
list: () => window.ipc.invoke('sessions:list', {}),
get: (sessionId) => window.ipc.invoke('sessions:get', { sessionId }),
getTurn: (turnId) => window.ipc.invoke('sessions:getTurn', { turnId }),
sendMessage: (sessionId, input, config) =>
window.ipc.invoke('sessions:sendMessage', { sessionId, input, config }),
respondToPermission: async (turnId, toolCallId, decision, metadata) => {
await window.ipc.invoke('sessions:respondToPermission', {
turnId,
toolCallId,
decision,
...(metadata === undefined ? {} : { metadata }),
})
},
respondToAskHuman: async (turnId, toolCallId, answer) => {
await window.ipc.invoke('sessions:respondToAskHuman', { turnId, toolCallId, answer })
},
stopTurn: async (turnId, reason) => {
await window.ipc.invoke('sessions:stopTurn', {
turnId,
...(reason === undefined ? {} : { reason }),
})
},
resumeTurn: async (sessionId) => {
await window.ipc.invoke('sessions:resumeTurn', { sessionId })
},
setTitle: async (sessionId, title) => {
await window.ipc.invoke('sessions:setTitle', { sessionId, title })
},
delete: async (sessionId) => {
await window.ipc.invoke('sessions:delete', { sessionId })
},
}

52
apps/x/apps/renderer/src/lib/session-chat/feed.test.ts Normal file
View file

@ -0,0 +1,52 @@
import { describe, expect, it } from 'vitest'
import type { SessionBusEvent } from '@x/shared/src/sessions.js'
import { createSessionFeed, type SessionFeedListener } from './feed'
const event: SessionBusEvent = {
kind: 'index-changed',
sessionId: 's1',
entry: null,
}
describe('createSessionFeed', () => {
it('starts the source lazily on first subscribe and fans events out', () => {
let sourceStarted = 0
let push: SessionFeedListener = () => undefined
const feed = createSessionFeed((listener) => {
sourceStarted += 1
push = listener
return () => undefined
})
expect(sourceStarted).toBe(0)
const seenA: SessionBusEvent[] = []
const seenB: SessionBusEvent[] = []
feed.subscribe((e) => seenA.push(e))
feed.subscribe((e) => seenB.push(e))
expect(sourceStarted).toBe(1) // one shared IPC listener
push(event)
expect(seenA).toEqual([event])
expect(seenB).toEqual([event])
})
it('unsubscribes cleanly and isolates a throwing listener', () => {
let push: SessionFeedListener = () => undefined
const feed = createSessionFeed((listener) => {
push = listener
return () => undefined
})
const seen: SessionBusEvent[] = []
feed.subscribe(() => {
throw new Error('bad subscriber')
})
const unsubscribe = feed.subscribe((e) => seen.push(e))
push(event)
expect(seen).toEqual([event])
unsubscribe()
push(event)
expect(seen).toEqual([event])
})
})

42
apps/x/apps/renderer/src/lib/session-chat/feed.ts Normal file
View file

@ -0,0 +1,42 @@
import type { SessionBusEvent } from '@x/shared/src/sessions.js'
export type SessionFeedListener = (event: SessionBusEvent) => void
export type SessionFeedSource = (listener: SessionFeedListener) => () => void
// One shared consumer of the sessions:events push channel; stores tap this
// fan-out instead of each opening their own IPC listener. Factory so tests
// can drive a fake source.
export function createSessionFeed(source: SessionFeedSource) {
const listeners = new Set<SessionFeedListener>()
let detach: (() => void) | null = null
const ensureStarted = () => {
if (detach) return
detach = source((event) => {
// Copy so (un)subscribing during dispatch is safe.
for (const listener of [...listeners]) {
try {
listener(event)
} catch {
// A misbehaving subscriber must never break the feed.
}
}
})
}
return {
subscribe(listener: SessionFeedListener): () => void {
ensureStarted()
listeners.add(listener)
return () => {
listeners.delete(listener)
}
},
}
}
const appFeed = createSessionFeed((listener) => window.ipc.on('sessions:events', listener))
export function subscribeSessionFeed(listener: SessionFeedListener): () => void {
return appFeed.subscribe(listener)
}

314
apps/x/apps/renderer/src/lib/session-chat/store.test.ts Normal file
View file

@ -0,0 +1,314 @@
import { describe, expect, it } from 'vitest'
import type { SessionBusEvent, SessionState } from '@x/shared/src/sessions.js'
import { isChatMessage } from '@/lib/chat-conversation'
import type { SessionsClient } from './client'
import type { SessionFeedListener } from './feed'
import { SessionChatStore, SessionListStore } from './store'
import {
assistantText,
completed,
completedTurnLog,
created,
indexEntry,
requested,
sessionState,
turnCompleted,
user,
type TEvent,
} from './test-fixtures'
const S1 = 'sess-1'
class FakeClient implements SessionsClient {
sessions = new Map<string, SessionState>()
turns = new Map<string, TEvent[]>()
calls: Array<{ method: string; args: unknown[] }> = []
// When set, get() defers until the returned resolver is invoked.
deferredGet: (() => void) | null = null
private record(method: string, ...args: unknown[]) {
this.calls.push({ method, args })
}
async create(input: { title?: string }) {
this.record('create', input)
return { sessionId: 'new-session' }
}
async list() {
this.record('list')
return { sessions: [...this.sessions.keys()].map((id) => indexEntry(id)) }
}
async get(sessionId: string) {
this.record('get', sessionId)
if (this.deferredGet === null) {
const state = this.sessions.get(sessionId)
if (!state) throw new Error(`session not found: ${sessionId}`)
return state
}
return new Promise<SessionState>((resolve, reject) => {
this.deferredGet = () => {
const state = this.sessions.get(sessionId)
if (state) resolve(state)
else reject(new Error(`session not found: ${sessionId}`))
}
})
}
async getTurn(turnId: string) {
this.record('getTurn', turnId)
const events = this.turns.get(turnId)
if (!events) throw new Error(`turn not found: ${turnId}`)
return { turnId, events }
}
async sendMessage(sessionId: string, input: unknown, config: unknown) {
this.record('sendMessage', sessionId, input, config)
return { turnId: 'turn-next' }
}
async respondToPermission(...args: unknown[]) {
this.record('respondToPermission', ...args)
}
async respondToAskHuman(...args: unknown[]) {
this.record('respondToAskHuman', ...args)
}
async stopTurn(...args: unknown[]) {
this.record('stopTurn', ...args)
}
async resumeTurn(...args: unknown[]) {
this.record('resumeTurn', ...args)
}
async setTitle(...args: unknown[]) {
this.record('setTitle', ...args)
}
async delete(...args: unknown[]) {
this.record('delete', ...args)
}
}
function makeStore() {
const client = new FakeClient()
let emit: SessionFeedListener = () => undefined
let subscribed = 0
let unsubscribed = 0
const store = new SessionChatStore({
client,
subscribeFeed: (listener) => {
subscribed += 1
emit = listener
return () => {
unsubscribed += 1
emit = () => undefined
}
},
})
const disconnect = store.connect()
return {
client,
store,
disconnect,
emit: (event: SessionBusEvent) => emit(event),
getSubscribed: () => subscribed,
getUnsubscribed: () => unsubscribed,
}
}
function turnEvent(
sessionId: string,
turnId: string,
event:
| TEvent
| { type: 'text_delta' | 'reasoning_delta'; turnId: string; modelCallIndex: number; delta: string },
): SessionBusEvent {
return { kind: 'turn-event', sessionId, turnId, event }
}
describe('SessionChatStore', () => {
it('seeds a session: prior turns frozen, latest live, conversation composed', async () => {
const { client, store } = makeStore()
client.sessions.set(S1, sessionState(S1, ['turn-1', 'turn-2']))
client.turns.set('turn-1', completedTurnLog('turn-1', S1, 'first?', 'first answer'))
client.turns.set('turn-2', completedTurnLog('turn-2', S1, 'second?', 'second answer'))
await store.setSession(S1)
const snapshot = store.getSnapshot()
expect(snapshot.loading).toBe(false)
expect(snapshot.latestTurnId).toBe('turn-2')
expect(
snapshot.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['first?', 'first answer', 'second?', 'second answer'])
expect(snapshot.chatState?.isProcessing).toBe(false)
})
it('applies live durable events through the shared reducer', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1, user('go'))))
emit(turnEvent(S1, 'turn-1', requested('turn-1', 0)))
expect(store.getSnapshot().chatState?.isThinking).toBe(true)
emit(turnEvent(S1, 'turn-1', completed('turn-1', 0, assistantText('done'))))
emit(turnEvent(S1, 'turn-1', turnCompleted('turn-1')))
const snapshot = store.getSnapshot()
expect(snapshot.latestTurnId).toBe('turn-1')
expect(snapshot.chatState?.isProcessing).toBe(false)
expect(
snapshot.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['go', 'done'])
})
it('accumulates text deltas and clears them on the canonical response', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1, user('go'))))
emit(turnEvent(S1, 'turn-1', requested('turn-1', 0)))
emit(turnEvent(S1, 'turn-1', { type: 'text_delta', turnId: 'turn-1', modelCallIndex: 0, delta: 'he' }))
emit(turnEvent(S1, 'turn-1', { type: 'text_delta', turnId: 'turn-1', modelCallIndex: 0, delta: 'y' }))
expect(store.getSnapshot().chatState?.currentAssistantMessage).toBe('hey')
emit(turnEvent(S1, 'turn-1', completed('turn-1', 0, assistantText('hey'))))
expect(store.getSnapshot().chatState?.currentAssistantMessage).toBe('')
})
it('freezes the previous latest turn when a new turn starts', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, ['turn-1']))
client.turns.set('turn-1', completedTurnLog('turn-1', S1, 'q1', 'a1'))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-2', created('turn-2', S1, user('q2'))))
const snapshot = store.getSnapshot()
expect(snapshot.latestTurnId).toBe('turn-2')
expect(
snapshot.chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['q1', 'a1', 'q2'])
})
it('ignores events for other sessions', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent('other-session', 'turn-x', created('turn-x', 'other-session')))
expect(store.getSnapshot().chatState?.conversation).toEqual([])
})
it('reconciles an unknown mid-turn event by refetching the turn', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
// The feed attached mid-turn: we never saw turn_created for turn-9.
client.turns.set('turn-9', [
created('turn-9', S1, user('missed')),
requested('turn-9', 0),
])
emit(turnEvent(S1, 'turn-9', completed('turn-9', 0, assistantText('caught up'))))
await Promise.resolve()
await Promise.resolve()
expect(store.getSnapshot().latestTurnId).toBe('turn-9')
})
it('routes actions against the latest turn', async () => {
const { client, store, emit } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1)))
await store.sendMessage(user('next'), { agent: { agentId: 'copilot' } })
await store.respondToPermission('tc1', 'allow')
await store.answerAskHuman('ah1', '42')
await store.stop()
expect(client.calls.filter((c) => c.method !== 'get' && c.method !== 'getTurn')).toEqual([
{ method: 'sendMessage', args: [S1, user('next'), { agent: { agentId: 'copilot' } }] },
{ method: 'respondToPermission', args: ['turn-1', 'tc1', 'allow', undefined] },
{ method: 'respondToAskHuman', args: ['turn-1', 'ah1', '42'] },
{ method: 'stopTurn', args: ['turn-1'] },
])
})
it('rejects sendMessage without an active session', async () => {
const { store } = makeStore()
await expect(
store.sendMessage(user('x'), { agent: { agentId: 'copilot' } }),
).rejects.toThrowError('No active session')
})
it('drops stale loads after a session switch', async () => {
const { client, store } = makeStore()
client.sessions.set(S1, sessionState(S1, ['turn-1']))
client.turns.set('turn-1', completedTurnLog('turn-1', S1, 'old', 'old answer'))
client.sessions.set('sess-2', sessionState('sess-2', []))
client.deferredGet = () => undefined // arm deferral
const first = store.setSession(S1)
const release = client.deferredGet
client.deferredGet = null
const second = store.setSession('sess-2')
release?.() // S1's get resolves after the switch
await Promise.all([first, second])
expect(store.getSnapshot().sessionId).toBe('sess-2')
expect(store.getSnapshot().chatState?.conversation).toEqual([])
})
it('surfaces load errors and disconnects its feed subscription', async () => {
const { store, disconnect, getUnsubscribed } = makeStore()
await store.setSession('missing-session')
expect(store.getSnapshot().error).toMatch(/session not found/)
disconnect()
expect(getUnsubscribed()).toBe(1)
void store
})
it('survives a StrictMode-style connect -> cleanup -> connect cycle', async () => {
// React StrictMode runs every effect's mount/cleanup/mount cycle in dev;
// the feed must re-attach or the store goes permanently deaf (the bug
// this test pins: a constructor-made subscription torn down by the first
// cleanup was never restored, leaving the chat stuck at "Thinking…").
const { client, store, disconnect, emit, getSubscribed } = makeStore()
client.sessions.set(S1, sessionState(S1, []))
disconnect()
const disconnect2 = store.connect()
expect(getSubscribed()).toBe(2)
await store.setSession(S1)
emit(turnEvent(S1, 'turn-1', created('turn-1', S1, user('go'))))
emit(turnEvent(S1, 'turn-1', requested('turn-1', 0)))
emit(turnEvent(S1, 'turn-1', completed('turn-1', 0, assistantText('done'))))
emit(turnEvent(S1, 'turn-1', turnCompleted('turn-1')))
expect(store.getSnapshot().chatState?.isProcessing).toBe(false)
expect(
store.getSnapshot().chatState?.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['go', 'done'])
disconnect2()
})
})
describe('SessionListStore', () => {
it('loads, applies index updates and deletions, and sorts by updatedAt', async () => {
const client = new FakeClient()
client.sessions.set('a', sessionState('a', []))
client.sessions.set('b', sessionState('b', []))
let emit: SessionFeedListener = () => undefined
const store = new SessionListStore({
client,
subscribeFeed: (listener) => {
emit = listener
return () => undefined
},
})
store.connect()
await store.load()
expect(store.getSnapshot().sessions.map((s) => s.sessionId).sort()).toEqual(['a', 'b'])
emit({
kind: 'index-changed',
sessionId: 'c',
entry: indexEntry('c', { updatedAt: '2026-07-03T00:00:00Z' }),
})
expect(store.getSnapshot().sessions[0].sessionId).toBe('c')
emit({ kind: 'index-changed', sessionId: 'a', entry: null })
expect(store.getSnapshot().sessions.map((s) => s.sessionId)).toEqual(['c', 'b'])
})
})

314
apps/x/apps/renderer/src/lib/session-chat/store.ts Normal file
View file

@ -0,0 +1,314 @@
import type { z } from 'zod'
import type { UserMessage } from '@x/shared/src/message.js'
import type { SessionBusEvent, SessionIndexEntry } from '@x/shared/src/sessions.js'
import {
reduceTurn,
type JsonValue,
type TurnEvent,
type TurnState,
type TurnStreamEvent,
} from '@x/shared/src/turns.js'
import type { SendMessageConfig, SessionsClient } from './client'
import type { SessionFeedListener } from './feed'
import {
applyOverlay,
buildSessionChatState,
emptyOverlay,
type LiveOverlay,
type SessionChatState,
} from './turn-view'
type TEvent = z.infer<typeof TurnEvent>
export interface SessionChatSnapshot {
sessionId: string | null
chatState: SessionChatState | null
latestTurnId: string | null
loading: boolean
error: string | null
}
export interface SessionChatStoreDeps {
client: SessionsClient
subscribeFeed: (listener: SessionFeedListener) => () => void
}
// Framework-agnostic controller for one active session's chat. Owns all the
// logic (seeding via getSession/getTurn, applying live feed events with the
// shared reducer, the ephemeral overlay, action routing); the useSessionChat
// hook is a thin useSyncExternalStore subscription over it.
export class SessionChatStore {
private readonly client: SessionsClient
private readonly subscribeFeed: (listener: SessionFeedListener) => () => void
private feedDisconnect: (() => void) | null = null
private readonly listeners = new Set<() => void>()
private sessionId: string | null = null
// Settled earlier turns, reduced once and frozen.
private priorTurns: TurnState[] = []
// The latest turn's raw event log; re-reduced on each durable event.
private latestEvents: TEvent[] | null = null
private overlay: LiveOverlay = emptyOverlay()
private loading = false
private error: string | null = null
// Guards stale async loads after a session switch.
private generation = 0
private snapshot: SessionChatSnapshot = {
sessionId: null,
chatState: null,
latestTurnId: null,
loading: false,
error: null,
}
constructor(deps: SessionChatStoreDeps) {
this.client = deps.client
this.subscribeFeed = deps.subscribeFeed
}
// Feed attachment is effect-managed and idempotent so React StrictMode's
// mount -> cleanup -> mount cycle re-attaches cleanly (a constructor-made
// subscription would be torn down by the first cleanup and never restored).
connect(): () => void {
if (!this.feedDisconnect) {
this.feedDisconnect = this.subscribeFeed(this.onFeedEvent)
}
return () => {
this.feedDisconnect?.()
this.feedDisconnect = null
}
}
subscribe = (onChange: () => void): (() => void) => {
this.listeners.add(onChange)
return () => {
this.listeners.delete(onChange)
}
}
getSnapshot = (): SessionChatSnapshot => this.snapshot
async setSession(sessionId: string | null): Promise<void> {
if (sessionId === this.sessionId) return
this.generation += 1
const generation = this.generation
this.sessionId = sessionId
this.priorTurns = []
this.latestEvents = null
this.overlay = emptyOverlay()
this.error = null
this.loading = sessionId !== null
this.emit()
if (sessionId === null) return
try {
const state = await this.client.get(sessionId)
const turns = await Promise.all(
state.turns.map((ref) => this.client.getTurn(ref.turnId)),
)
if (generation !== this.generation) return
const reduced = turns.map((turn) => reduceTurn(turn.events))
this.priorTurns = reduced.slice(0, -1)
this.latestEvents = turns.length > 0 ? turns[turns.length - 1].events : null
this.loading = false
this.emit()
} catch (error) {
if (generation !== this.generation) return
console.error('[session-chat] failed to load session', sessionId, error)
this.loading = false
this.error = error instanceof Error ? error.message : String(error)
this.emit()
}
}
private onFeedEvent: SessionFeedListener = (event: SessionBusEvent) => {
if (event.kind !== 'turn-event' || event.sessionId !== this.sessionId) return
const turnEvent = event.event
if (isDurable(turnEvent)) {
if (turnEvent.type === 'turn_created') {
// A new turn started for this session: freeze the previous latest.
this.freezeLatest()
this.latestEvents = [turnEvent]
this.overlay = emptyOverlay()
} else if (this.latestEvents && this.latestEvents[0].turnId === turnEvent.turnId) {
this.latestEvents.push(turnEvent)
} else {
// An event for a turn we haven't seen (missed turn_created, e.g. the
// feed attached mid-turn): reconcile by refetching that turn.
void this.reloadTurn(event.turnId)
return
}
}
this.overlay = applyOverlay(this.overlay, turnEvent)
this.emit()
}
private freezeLatest(): void {
if (!this.latestEvents) return
try {
this.priorTurns = [...this.priorTurns, reduceTurn(this.latestEvents)]
} catch {
// A turn we can't reduce is dropped from history rather than wedging
// the whole conversation.
}
this.latestEvents = null
}
private async reloadTurn(turnId: string): Promise<void> {
const generation = this.generation
try {
const turn = await this.client.getTurn(turnId)
if (generation !== this.generation) return
if (this.latestEvents && this.latestEvents[0].turnId !== turnId) {
this.freezeLatest()
}
this.latestEvents = turn.events
this.emit()
} catch (error) {
// The next snapshot-worthy event will retry.
console.error('[session-chat] failed to reload turn', turnId, error)
}
}
// ── Actions ─────────────────────────────────────────────────────────────
sendMessage = async (
input: z.infer<typeof UserMessage>,
config: SendMessageConfig,
): Promise<{ turnId: string }> => {
if (!this.sessionId) throw new Error('No active session')
return this.client.sendMessage(this.sessionId, input, config)
}
respondToPermission = async (
toolCallId: string,
decision: 'allow' | 'deny',
metadata?: JsonValue,
): Promise<void> => {
const turnId = this.snapshot.latestTurnId
if (!turnId) return
await this.client.respondToPermission(turnId, toolCallId, decision, metadata)
}
answerAskHuman = async (toolCallId: string, answer: string): Promise<void> => {
const turnId = this.snapshot.latestTurnId
if (!turnId) return
await this.client.respondToAskHuman(turnId, toolCallId, answer)
}
stop = async (): Promise<void> => {
const turnId = this.snapshot.latestTurnId
if (!turnId) return
await this.client.stopTurn(turnId)
}
// ── Derivation ──────────────────────────────────────────────────────────
private emit(): void {
this.snapshot = this.derive()
for (const listener of [...this.listeners]) {
listener()
}
}
private derive(): SessionChatSnapshot {
let turns = this.priorTurns
let error = this.error
if (this.latestEvents) {
try {
turns = [...this.priorTurns, reduceTurn(this.latestEvents)]
} catch (reduceError) {
error =
reduceError instanceof Error ? reduceError.message : String(reduceError)
}
}
const latest = turns[turns.length - 1]
return {
sessionId: this.sessionId,
chatState:
this.sessionId !== null && !this.loading
? buildSessionChatState(turns, this.overlay)
: null,
latestTurnId: latest?.definition.turnId ?? null,
loading: this.loading,
error,
}
}
}
function isDurable(event: TurnStreamEvent): event is TEvent {
return event.type !== 'text_delta' && event.type !== 'reasoning_delta'
}
// ---------------------------------------------------------------------------
// Session list store
// ---------------------------------------------------------------------------
export interface SessionListSnapshot {
sessions: SessionIndexEntry[]
loading: boolean
}
export class SessionListStore {
private readonly client: SessionsClient
private readonly subscribeFeed: (listener: SessionFeedListener) => () => void
private feedDisconnect: (() => void) | null = null
private readonly listeners = new Set<() => void>()
private entries = new Map<string, SessionIndexEntry>()
private loading = true
private snapshot: SessionListSnapshot = { sessions: [], loading: true }
constructor(deps: SessionChatStoreDeps) {
this.client = deps.client
this.subscribeFeed = deps.subscribeFeed
}
connect(): () => void {
if (!this.feedDisconnect) {
this.feedDisconnect = this.subscribeFeed(this.onFeedEvent)
}
return () => {
this.feedDisconnect?.()
this.feedDisconnect = null
}
}
subscribe = (onChange: () => void): (() => void) => {
this.listeners.add(onChange)
return () => {
this.listeners.delete(onChange)
}
}
getSnapshot = (): SessionListSnapshot => this.snapshot
async load(): Promise<void> {
const { sessions } = await this.client.list()
this.entries = new Map(sessions.map((entry) => [entry.sessionId, entry]))
this.loading = false
this.emit()
}
private onFeedEvent: SessionFeedListener = (event: SessionBusEvent) => {
if (event.kind !== 'index-changed') return
if (event.entry === null) {
this.entries.delete(event.sessionId)
} else {
this.entries.set(event.sessionId, event.entry)
}
this.emit()
}
private emit(): void {
this.snapshot = {
sessions: [...this.entries.values()].sort((a, b) =>
b.updatedAt.localeCompare(a.updatedAt),
),
loading: this.loading,
}
for (const listener of [...this.listeners]) {
listener()
}
}
}

171
apps/x/apps/renderer/src/lib/session-chat/test-fixtures.ts Normal file
View file

@ -0,0 +1,171 @@
import type { z } from 'zod'
import type { SessionIndexEntry, SessionState } from '@x/shared/src/sessions.js'
import type { ResolvedAgent, TurnEvent } from '@x/shared/src/turns.js'
// Compact turn-event builders for renderer tests. Sequences must satisfy the
// shared reducer's invariants (reduceTurn is what the store runs on them).
export type TEvent = z.infer<typeof TurnEvent>
export const TS = '2026-07-02T10:00:00Z'
export const FIXTURE_AGENT: z.infer<typeof ResolvedAgent> = {
agentId: 'copilot',
systemPrompt: 'SYS',
model: { provider: 'openai', model: 'gpt-fixture' },
tools: [],
}
export function user(text: string) {
return { role: 'user' as const, content: text }
}
export function assistantText(text: string) {
return { role: 'assistant' as const, content: text }
}
export function toolCallPart(id: string, name: string, args: unknown = {}) {
return { type: 'tool-call' as const, toolCallId: id, toolName: name, arguments: args }
}
export function created(
turnId: string,
sessionId: string,
input: ReturnType<typeof user> = user('hello'),
): TEvent {
return {
type: 'turn_created',
schemaVersion: 1,
turnId,
ts: TS,
sessionId,
agent: { requested: { agentId: 'copilot' }, resolved: FIXTURE_AGENT },
context: [],
input,
config: { autoPermission: false, humanAvailable: true, maxModelCalls: 20 },
}
}
export function requested(
turnId: string,
index: number,
refs: string[] = ['input'],
): TEvent {
return {
type: 'model_call_requested',
turnId,
ts: TS,
modelCallIndex: index,
request: {
messages: refs,
parameters: {},
},
}
}
export function completed(
turnId: string,
index: number,
message: { role: 'assistant'; content: unknown },
): TEvent {
return {
type: 'model_call_completed',
turnId,
ts: TS,
modelCallIndex: index,
message: message as never,
finishReason: 'stop',
usage: {},
}
}
export function invocation(turnId: string, toolCallId: string, name: string): TEvent {
return {
type: 'tool_invocation_requested',
turnId,
ts: TS,
toolCallId,
toolId: `builtin:${name}`,
toolName: name,
execution: 'sync',
input: {},
}
}
export function toolResult(
turnId: string,
toolCallId: string,
name: string,
output: unknown = 'ok',
): TEvent {
return {
type: 'tool_result',
turnId,
ts: TS,
toolCallId,
toolName: name,
source: 'sync',
result: { output: output as never, isError: false },
}
}
export function turnCompleted(turnId: string, text = 'done'): TEvent {
return {
type: 'turn_completed',
turnId,
ts: TS,
output: assistantText(text),
finishReason: 'stop',
usage: {},
}
}
// A settled single-response turn.
export function completedTurnLog(
turnId: string,
sessionId: string,
question: string,
answer: string,
): TEvent[] {
return [
created(turnId, sessionId, user(question)),
requested(turnId, 0),
completed(turnId, 0, assistantText(answer)),
turnCompleted(turnId, answer),
]
}
export function indexEntry(
sessionId: string,
overrides: Partial<SessionIndexEntry> = {},
): SessionIndexEntry {
return {
sessionId,
title: `Session ${sessionId}`,
createdAt: TS,
updatedAt: TS,
turnCount: 1,
latestTurnStatus: 'completed',
...overrides,
}
}
export function sessionState(
sessionId: string,
turnIds: string[],
): SessionState {
return {
definition: { type: 'session_created', schemaVersion: 1, sessionId, ts: TS },
title: `Session ${sessionId}`,
turns: turnIds.map((turnId, i) => ({
turnId,
sessionSeq: i + 1,
agentId: 'copilot',
model: FIXTURE_AGENT.model,
ts: TS,
})),
latestTurnId: turnIds[turnIds.length - 1],
createdAt: TS,
updatedAt: TS,
}
}

362
apps/x/apps/renderer/src/lib/session-chat/turn-view.test.ts Normal file
View file

@ -0,0 +1,362 @@
import { describe, expect, it } from 'vitest'
import { reduceTurn } from '@x/shared/src/turns.js'
import { isChatMessage, isErrorMessage, isToolCall } from '@/lib/chat-conversation'
import {
applyOverlay,
buildSessionChatState,
buildTurnConversation,
emptyOverlay,
} from './turn-view'
import {
TS,
assistantText,
completed,
completedTurnLog,
created,
invocation,
requested,
toolCallPart,
toolResult,
turnCompleted,
user,
type TEvent,
} from './test-fixtures'
const T1 = 'turn-1'
const S1 = 'sess-1'
function assistantCalls(...parts: Array<ReturnType<typeof toolCallPart>>) {
return { role: 'assistant' as const, content: parts }
}
describe('applyOverlay', () => {
it('accumulates text and reasoning deltas', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, { type: 'text_delta', turnId: T1, modelCallIndex: 0, delta: 'he' })
overlay = applyOverlay(overlay, { type: 'text_delta', turnId: T1, modelCallIndex: 0, delta: 'y' })
overlay = applyOverlay(overlay, {
type: 'reasoning_delta',
turnId: T1,
modelCallIndex: 0,
delta: 'hmm',
})
expect(overlay.text).toBe('hey')
expect(overlay.reasoning).toBe('hmm')
})
it('clears text/reasoning when the canonical model response arrives', () => {
let overlay = { ...emptyOverlay(), text: 'streaming', reasoning: 'thinking' }
overlay = applyOverlay(overlay, completed(T1, 0, assistantText('final')))
expect(overlay.text).toBe('')
expect(overlay.reasoning).toBe('')
})
it('accumulates tool-output progress chunks and drops them on the terminal result', () => {
let overlay = emptyOverlay()
const progress = (chunk: string): TEvent => ({
type: 'tool_progress',
turnId: T1,
ts: TS,
toolCallId: 'tc1',
source: 'sync',
progress: { kind: 'tool-output', chunk },
})
overlay = applyOverlay(overlay, progress('$ ls\n'))
overlay = applyOverlay(overlay, progress('README.md\n'))
expect(overlay.toolOutput.tc1).toBe('$ ls\nREADME.md\n')
overlay = applyOverlay(overlay, toolResult(T1, 'tc1', 'executeCommand'))
expect(overlay.toolOutput.tc1).toBeUndefined()
})
it('ignores non-output progress payloads', () => {
const overlay = applyOverlay(emptyOverlay(), {
type: 'tool_progress',
turnId: T1,
ts: TS,
toolCallId: 'tc1',
source: 'sync',
progress: { pct: 50 },
})
expect(overlay.toolOutput).toEqual({})
})
})
describe('voice output', () => {
const delta = (d: string): Parameters<typeof applyOverlay>[1] =>
({ type: 'text_delta', turnId: T1, modelCallIndex: 0, delta: d })
it('extracts completed <voice> blocks across split deltas', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('Sure! <voi'))
expect(overlay.voiceSegments).toEqual([])
overlay = applyOverlay(overlay, delta('ce>hello there</voice> and <voice>bye'))
expect(overlay.voiceSegments).toEqual(['hello there'])
overlay = applyOverlay(overlay, delta('</voice> done'))
expect(overlay.voiceSegments).toEqual(['hello there', 'bye'])
})
it('emits an early clause from a long open block, then the remainder on close', () => {
let overlay = emptyOverlay()
const longClause = 'Okay so the first thing I would look at here is the error message,'
overlay = applyOverlay(overlay, delta(`<voice>${longClause} because`))
// Open block crossed the early-speech threshold at a clause boundary.
expect(overlay.voiceSegments).toEqual([longClause])
overlay = applyOverlay(overlay, delta(' it tells you the root cause.</voice>'))
// Remainder only — the early clause is not repeated.
expect(overlay.voiceSegments).toEqual([longClause, 'because it tells you the root cause.'])
})
it('does not emit early clauses from short open blocks', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('<voice>Sure, one sec'))
expect(overlay.voiceSegments).toEqual([])
overlay = applyOverlay(overlay, delta('.</voice>'))
expect(overlay.voiceSegments).toEqual(['Sure, one sec.'])
})
it('keeps segments but resets the scan on model_call_completed', () => {
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('<voice>one</voice>'))
overlay = applyOverlay(overlay, completed(T1, 0, assistantText('<voice>one</voice>')))
expect(overlay.voiceSegments).toEqual(['one'])
expect(overlay.text).toBe('')
overlay = applyOverlay(overlay, delta('<voice>two</voice>'))
expect(overlay.voiceSegments).toEqual(['one', 'two'])
})
it('strips voice tags from the streaming message and exposes segments on state', () => {
const turn = reduceTurn([created(T1, S1), requested(T1, 0)])
let overlay = emptyOverlay()
overlay = applyOverlay(overlay, delta('Plan: <voice>speak this</voice> rest'))
const state = buildSessionChatState([turn], overlay)
expect(state.currentAssistantMessage).toBe('Plan: speak this rest')
expect(state.voiceSegments).toEqual(['speak this'])
})
it('strips voice tags from persisted assistant messages', () => {
const state = reduceTurn(
completedTurnLog(T1, S1, 'q', 'Sure. <voice>Here you go.</voice> Done.'),
)
const items = buildTurnConversation(state)
const assistant = items.find((i) => isChatMessage(i) && i.role === 'assistant')
expect(isChatMessage(assistant!) && assistant.content).toBe('Sure. Here you go. Done.')
})
})
describe('buildTurnConversation', () => {
it('maps user input, assistant text, and settled tool calls', () => {
const call = assistantCalls(toolCallPart('tc1', 'echo', { x: 1 }))
const state = reduceTurn([
created(T1, S1, user('run it')),
requested(T1, 0),
completed(T1, 0, call),
invocation(T1, 'tc1', 'echo'),
toolResult(T1, 'tc1', 'echo', { echoed: true }),
requested(T1, 1, [
'assistant:0',
'toolResult:tc1',
]),
completed(T1, 1, assistantText('all done')),
turnCompleted(T1, 'all done'),
])
const items = buildTurnConversation(state)
expect(items.map((i) => (isToolCall(i) ? `tool:${i.name}:${i.status}` : isChatMessage(i) ? `${i.role}` : 'x'))).toEqual([
'user',
'tool:echo:completed',
'assistant',
])
const tool = items.find(isToolCall)
expect(tool?.result).toEqual({ echoed: true })
expect(tool?.input).toEqual({ x: 1 })
})
it('marks permission-pending tools pending and running tools running', () => {
const state = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('p1', 'executeCommand'), toolCallPart('r1', 'echo'))),
{
type: 'tool_permission_required',
turnId: T1,
ts: TS,
toolCallId: 'p1',
toolName: 'executeCommand',
request: { kind: 'command', commandNames: ['rm'] },
},
invocation(T1, 'r1', 'echo'),
])
const items = buildTurnConversation(state).filter(isToolCall)
expect(items.map((i) => i.status)).toEqual(['pending', 'running'])
})
it('uses the tool result envelope to determine error status', () => {
const state = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(
T1,
0,
assistantCalls(
toolCallPart('flagged', 'echo'),
toolCallPart('normal', 'echo'),
),
),
invocation(T1, 'flagged', 'echo'),
{
type: 'tool_result',
turnId: T1,
ts: TS,
toolCallId: 'flagged',
toolName: 'echo',
source: 'sync',
result: { output: 'boom', isError: true },
},
invocation(T1, 'normal', 'echo'),
toolResult(T1, 'normal', 'echo', { success: false, message: 'payload data' }),
])
const items = buildTurnConversation(state).filter(isToolCall)
expect(items.map((i) => i.status)).toEqual(['error', 'completed'])
})
it('renders user attachments and a failed turn as an error item', () => {
const input = {
role: 'user' as const,
content: [
{ type: 'text' as const, text: 'see file' },
{ type: 'attachment' as const, path: '/tmp/a.png', filename: 'a.png', mimeType: 'image/png' },
],
}
const state = reduceTurn([
created(T1, S1, input as never),
requested(T1, 0),
{ type: 'model_call_failed', turnId: T1, ts: TS, modelCallIndex: 0, error: 'boom' },
{ type: 'turn_failed', turnId: T1, ts: TS, error: 'boom', usage: {} },
])
const items = buildTurnConversation(state)
expect(isChatMessage(items[0]) && items[0].attachments?.[0].filename).toBe('a.png')
expect(isErrorMessage(items[1]) && items[1].message).toBe('boom')
})
})
describe('buildSessionChatState', () => {
it('composes the conversation across turns and derives processing flags', () => {
const prior = reduceTurn(completedTurnLog('turn-1', S1, 'first?', 'first answer'))
const latest = reduceTurn([
created('turn-2', S1, user('second?')),
requested('turn-2', 0),
])
const state = buildSessionChatState([prior, latest], emptyOverlay())
expect(
state.conversation.filter(isChatMessage).map((m) => m.content),
).toEqual(['first?', 'first answer', 'second?'])
expect(state.isProcessing).toBe(true) // latest turn idle = actively working
expect(state.isThinking).toBe(true)
})
it('is settled (not processing) when the latest turn is terminal', () => {
const turn = reduceTurn(completedTurnLog(T1, S1, 'q', 'a'))
const state = buildSessionChatState([turn], emptyOverlay())
expect(state.isProcessing).toBe(false)
expect(state.isThinking).toBe(false)
})
it('exposes pending permissions as request events; waiting is not thinking', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('p1', 'executeCommand', { command: 'rm -rf /' }))),
{
type: 'tool_permission_required',
turnId: T1,
ts: TS,
toolCallId: 'p1',
toolName: 'executeCommand',
request: { kind: 'command', commandNames: ['rm'] },
},
{
type: 'turn_suspended',
turnId: T1,
ts: TS,
pendingPermissions: [
{ toolCallId: 'p1', toolName: 'executeCommand', request: { kind: 'command', commandNames: ['rm'] } },
],
pendingAsyncTools: [],
usage: {},
},
])
const state = buildSessionChatState([turn], emptyOverlay())
const request = state.allPermissionRequests.get('p1')
expect(request).toMatchObject({
type: 'tool-permission-request',
toolCall: { toolCallId: 'p1', toolName: 'executeCommand' },
permission: { kind: 'command', commandNames: ['rm'] },
})
expect(state.isProcessing).toBe(true)
expect(state.isThinking).toBe(false)
})
it('maps human decisions to responses and classifier decisions to auto-decisions', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('h1', 'echo'), toolCallPart('c1', 'echo'))),
{ type: 'tool_permission_required', turnId: T1, ts: TS, toolCallId: 'h1', toolName: 'echo', request: { kind: 'command', commandNames: ['x'] } },
{ type: 'tool_permission_required', turnId: T1, ts: TS, toolCallId: 'c1', toolName: 'echo', request: { kind: 'command', commandNames: ['y'] } },
{ type: 'tool_permission_resolved', turnId: T1, ts: TS, toolCallId: 'h1', decision: 'allow', source: 'human' },
{ type: 'tool_permission_resolved', turnId: T1, ts: TS, toolCallId: 'c1', decision: 'deny', source: 'classifier', reason: 'risky' },
{ type: 'tool_result', turnId: T1, ts: TS, toolCallId: 'c1', toolName: 'echo', source: 'runtime', result: { output: 'denied', isError: true } },
invocation(T1, 'h1', 'echo'),
toolResult(T1, 'h1', 'echo'),
])
const state = buildSessionChatState([turn], emptyOverlay())
expect(state.permissionResponses.get('h1')).toBe('approve')
expect(state.autoPermissionDecisions.get('c1')).toMatchObject({
decision: 'deny',
reason: 'risky',
})
})
it('exposes pending ask-human calls with question and options', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('ah1', 'ask-human', { question: 'Deploy?', options: ['Yes', 'No'] }))),
{
type: 'tool_invocation_requested',
turnId: T1,
ts: TS,
toolCallId: 'ah1',
toolId: 'builtin:ask-human',
toolName: 'ask-human',
execution: 'async',
input: { question: 'Deploy?', options: ['Yes', 'No'] },
},
])
const state = buildSessionChatState([turn], emptyOverlay())
expect(state.pendingAskHumanRequests.get('ah1')).toMatchObject({
query: 'Deploy?',
options: ['Yes', 'No'],
})
expect(state.isThinking).toBe(false) // waiting on the human, no shimmer
})
it('stitches live tool output onto the matching tool item', () => {
const turn = reduceTurn([
created(T1, S1),
requested(T1, 0),
completed(T1, 0, assistantCalls(toolCallPart('tc1', 'executeCommand'))),
invocation(T1, 'tc1', 'executeCommand'),
])
const overlay = { ...emptyOverlay(), toolOutput: { tc1: 'partial output' } }
const state = buildSessionChatState([turn], overlay)
const tool = state.conversation.filter(isToolCall)[0]
expect(tool.streamingOutput).toBe('partial output')
})
it('surfaces streaming text as currentAssistantMessage', () => {
const turn = reduceTurn([created(T1, S1), requested(T1, 0)])
const state = buildSessionChatState([turn], { ...emptyOverlay(), text: 'typing…' })
expect(state.currentAssistantMessage).toBe('typing…')
})
})

395
apps/x/apps/renderer/src/lib/session-chat/turn-view.ts Normal file
View file

@ -0,0 +1,395 @@
import type { z } from 'zod'
import type {
AskHumanRequestEvent,
ToolPermissionAutoDecisionEvent,
ToolPermissionMetadata,
ToolPermissionRequestEvent,
} from '@x/shared/src/runs.js'
import {
deriveTurnStatus,
outstandingAsyncTools,
outstandingPermissions,
type ToolCallState,
type TurnState,
type TurnStreamEvent,
} from '@x/shared/src/turns.js'
import type {
ChatMessage,
ConversationItem,
ErrorMessage,
MessageAttachment,
PermissionResponse,
ToolCall,
} from '@/lib/chat-conversation'
// Pure derivations from reduced turn state (+ the ephemeral live overlay) to
// the view shapes the existing chat components already consume. No IPC, no
// React — everything here is unit-testable with plain state values.
// ---------------------------------------------------------------------------
// Live overlay: ephemeral streaming buffers (turn-runtime-design.md §14.4).
// ---------------------------------------------------------------------------
export type LiveOverlay = {
text: string
reasoning: string
toolOutput: Record<string, string>
// Speakable segments seen while streaming, in order, monotonically growing
// for the lifetime of the overlay (i.e. one active turn). Usually the
// contents of completed <voice>…</voice> blocks, but a long still-open
// block may emit an early clause (see EARLY_SPEECH_MIN_CHARS) so speech
// can start before the sentence finishes generating. Consumers speak
// segments beyond what they've already spoken; the overlay reset on turn
// switch starts a fresh list.
voiceSegments: string[]
// Scan cursor into `text` — everything before it has been checked for
// complete voice blocks.
voiceScanIndex: number
// Chars of the currently-open voice block's content already emitted as an
// early clause — the block's remainder (on close) excludes them.
voicePartialConsumed: number
}
export const emptyOverlay = (): LiveOverlay => ({
text: '',
reasoning: '',
toolOutput: {},
voiceSegments: [],
voiceScanIndex: 0,
voicePartialConsumed: 0,
})
// The model emits <voice>…</voice> around speakable text when voice output
// is enabled; tags are never shown to the user.
export function stripVoiceTags(text: string): string {
return text.replace(/<\/?voice>/g, '')
}
const VOICE_BLOCK = /<voice>([\s\S]*?)<\/voice>/g
const VOICE_OPEN_TAG = '<voice>'
// Early speech: once an open block has this many unconsumed chars, its last
// complete clause is emitted immediately instead of waiting for </voice> —
// TTS starts on the first clause while the rest of the sentence generates.
const EARLY_SPEECH_MIN_CHARS = 60
// ...but never emit a fragment shorter than this (prosody suffers).
const EARLY_SPEECH_MIN_EMIT = 30
// Clause boundaries (punctuation, optionally inside closing quote/paren,
// followed by whitespace or end-of-buffer).
const CLAUSE_BOUNDARY = /[,;:.!?…—]["')\]]*(?=\s|$)/g
// Accumulates deltas; canonical durable events supersede the buffers (the
// committed transcript now contains what was streaming).
export function applyOverlay(overlay: LiveOverlay, event: TurnStreamEvent): LiveOverlay {
switch (event.type) {
case 'text_delta': {
const text = overlay.text + event.delta
// Extract complete voice blocks past the scan cursor. Incomplete
// blocks (opening tag seen, closing not yet) stay unconsumed until a
// later delta completes them. The first complete block may have had an
// early clause emitted while it was open — skip those chars.
const segments: string[] = []
let scanIndex = overlay.voiceScanIndex
let partialConsumed = overlay.voicePartialConsumed
VOICE_BLOCK.lastIndex = scanIndex
for (let m = VOICE_BLOCK.exec(text); m; m = VOICE_BLOCK.exec(text)) {
const content = m[1].slice(partialConsumed).trim()
partialConsumed = 0
if (content) segments.push(content)
scanIndex = m.index + m[0].length
}
// Early speech: if a voice block is still open and has accumulated a
// long unconsumed run, emit its last complete clause now — speech can
// start while the rest of the sentence is still generating.
const openIdx = text.indexOf(VOICE_OPEN_TAG, scanIndex)
if (openIdx !== -1) {
const unconsumed = text.slice(openIdx + VOICE_OPEN_TAG.length + partialConsumed)
if (unconsumed.length >= EARLY_SPEECH_MIN_CHARS) {
let lastBoundaryEnd = -1
CLAUSE_BOUNDARY.lastIndex = 0
for (let b = CLAUSE_BOUNDARY.exec(unconsumed); b; b = CLAUSE_BOUNDARY.exec(unconsumed)) {
lastBoundaryEnd = b.index + b[0].length
}
if (lastBoundaryEnd >= EARLY_SPEECH_MIN_EMIT) {
const clause = unconsumed.slice(0, lastBoundaryEnd).trim()
if (clause) segments.push(clause)
partialConsumed += lastBoundaryEnd
}
}
} else {
// No open block — any partial bookkeeping belongs to a block that
// has since closed.
partialConsumed = 0
}
return {
...overlay,
text,
...(segments.length > 0
? { voiceSegments: [...overlay.voiceSegments, ...segments] }
: {}),
voiceScanIndex: scanIndex,
voicePartialConsumed: partialConsumed,
}
}
case 'reasoning_delta':
return { ...overlay, reasoning: overlay.reasoning + event.delta }
case 'model_call_completed':
return { ...overlay, text: '', reasoning: '', voiceScanIndex: 0, voicePartialConsumed: 0 }
case 'tool_progress': {
const progress = event.progress
if (
progress &&
typeof progress === 'object' &&
!Array.isArray(progress) &&
(progress as { kind?: unknown }).kind === 'tool-output' &&
typeof (progress as { chunk?: unknown }).chunk === 'string'
) {
const chunk = (progress as { chunk: string }).chunk
return {
...overlay,
toolOutput: {
...overlay.toolOutput,
[event.toolCallId]: (overlay.toolOutput[event.toolCallId] ?? '') + chunk,
},
}
}
return overlay
}
case 'tool_result': {
if (!(event.toolCallId in overlay.toolOutput)) return overlay
const toolOutput = { ...overlay.toolOutput }
delete toolOutput[event.toolCallId]
return { ...overlay, toolOutput }
}
default:
return overlay
}
}
// ---------------------------------------------------------------------------
// Conversation items
// ---------------------------------------------------------------------------
type UserContent = TurnState['definition']['input']['content']
function extractText(content: UserContent): string {
if (typeof content === 'string') return content
return content
.map((part) => (part.type === 'text' ? part.text : ''))
.filter(Boolean)
.join('\n')
}
function extractAttachments(content: UserContent): MessageAttachment[] | undefined {
if (typeof content === 'string') return undefined
const attachments = content.flatMap((part) =>
part.type === 'attachment'
? [
{
path: part.path,
filename: part.filename,
mimeType: part.mimeType,
...(part.size === undefined ? {} : { size: part.size }),
},
]
: [],
)
return attachments.length > 0 ? attachments : undefined
}
function toolStatus(tc: ToolCallState): ToolCall['status'] {
if (tc.result) return tc.result.result.isError ? 'error' : 'completed'
if (tc.permission && !tc.permission.resolved) return 'pending'
return 'running'
}
// One turn's contribution to the conversation: the user input, then per
// completed model call its text and tool calls (with live status/results).
export function buildTurnConversation(state: TurnState): ConversationItem[] {
const items: ConversationItem[] = []
const turnId = state.definition.turnId
let seq = 0
const ts = () => Date.parse(state.definition.ts) + seq++
const userText = extractText(state.definition.input.content)
const attachments = extractAttachments(state.definition.input.content)
if (userText || attachments) {
items.push({
id: `${turnId}:user`,
role: 'user',
content: userText,
timestamp: ts(),
...(attachments ? { attachments } : {}),
} satisfies ChatMessage)
}
const toolCallsById = new Map(state.toolCalls.map((tc) => [tc.toolCallId, tc]))
for (const call of state.modelCalls) {
if (call.response === undefined) continue
const content = call.response.content
// Voice tags are model-facing markup, never shown (parity with the
// legacy path's display-time strip).
const text = stripVoiceTags(
typeof content === 'string'
? content
: content
.map((part) => (part.type === 'text' ? part.text : ''))
.filter(Boolean)
.join('\n'),
)
if (text) {
items.push({
id: `${turnId}:a${call.index}`,
role: 'assistant',
content: text,
timestamp: ts(),
} satisfies ChatMessage)
}
if (Array.isArray(content)) {
for (const part of content) {
if (part.type !== 'tool-call') continue
const tc = toolCallsById.get(part.toolCallId)
items.push({
id: part.toolCallId,
name: part.toolName,
input: part.arguments as ToolCall['input'],
...(tc?.result ? { result: tc.result.result.output as ToolCall['result'] } : {}),
status: tc ? toolStatus(tc) : 'running',
timestamp: ts(),
} satisfies ToolCall)
}
}
}
if (state.terminal?.type === 'turn_failed') {
items.push({
id: `${turnId}:error`,
kind: 'error',
message: state.terminal.error,
timestamp: ts(),
} satisfies ErrorMessage)
}
return items
}
// ---------------------------------------------------------------------------
// Session chat state (superset of the ChatTabViewState fields)
// ---------------------------------------------------------------------------
type PermMeta = z.infer<typeof ToolPermissionMetadata>
export type SessionChatState = {
conversation: ConversationItem[]
currentAssistantMessage: string
// See LiveOverlay.voiceSegments.
voiceSegments: string[]
pendingAskHumanRequests: Map<string, z.infer<typeof AskHumanRequestEvent>>
allPermissionRequests: Map<string, z.infer<typeof ToolPermissionRequestEvent>>
permissionResponses: Map<string, PermissionResponse>
autoPermissionDecisions: Map<string, z.infer<typeof ToolPermissionAutoDecisionEvent>>
// Composer blocked / Stop shown until the latest turn settles (waiting on a
// permission or ask-human still counts as processing).
isProcessing: boolean
// Actively working (non-terminal, nothing waiting on the user) — drives the
// "Thinking…" shimmer; deliberately false under permission/ask-human cards.
isThinking: boolean
}
function toolCallPartOf(tc: ToolCallState) {
return {
type: 'tool-call' as const,
toolCallId: tc.toolCallId,
toolName: tc.toolName,
arguments: tc.input,
}
}
// Compose the whole session's conversation: prior (settled) turns plus the
// latest turn, with the live overlay stitched onto the latest turn's items.
export function buildSessionChatState(
turns: TurnState[],
overlay: LiveOverlay,
): SessionChatState {
const conversation: ConversationItem[] = []
for (const turn of turns) {
conversation.push(...buildTurnConversation(turn))
}
for (let i = 0; i < conversation.length; i++) {
const item = conversation[i]
if ('name' in item && overlay.toolOutput[item.id]) {
conversation[i] = { ...item, streamingOutput: overlay.toolOutput[item.id] }
}
}
const latest = turns[turns.length - 1]
const status = latest ? deriveTurnStatus(latest) : undefined
const latestTurnId = latest?.definition.turnId ?? ''
const allPermissionRequests = new Map<string, z.infer<typeof ToolPermissionRequestEvent>>()
const permissionResponses = new Map<string, PermissionResponse>()
const autoPermissionDecisions = new Map<
string,
z.infer<typeof ToolPermissionAutoDecisionEvent>
>()
const pendingAskHumanRequests = new Map<string, z.infer<typeof AskHumanRequestEvent>>()
if (latest) {
for (const tc of outstandingPermissions(latest)) {
allPermissionRequests.set(tc.toolCallId, {
runId: latestTurnId,
type: 'tool-permission-request',
subflow: [],
toolCall: toolCallPartOf(tc),
permission: tc.permission?.required.request as PermMeta,
})
}
for (const tc of latest.toolCalls) {
const resolved = tc.permission?.resolved
if (!resolved) continue
if (resolved.source === 'human') {
permissionResponses.set(tc.toolCallId, resolved.decision === 'allow' ? 'approve' : 'deny')
} else if (resolved.source === 'classifier') {
autoPermissionDecisions.set(tc.toolCallId, {
runId: latestTurnId,
type: 'tool-permission-auto-decision',
subflow: [],
toolCallId: tc.toolCallId,
toolCall: toolCallPartOf(tc),
permission: tc.permission?.required.request as PermMeta,
decision: resolved.decision,
reason: resolved.reason ?? '',
})
}
}
for (const tc of outstandingAsyncTools(latest)) {
if (tc.toolName !== 'ask-human') continue
const input = (tc.input ?? {}) as { question?: unknown; options?: unknown }
pendingAskHumanRequests.set(tc.toolCallId, {
runId: latestTurnId,
type: 'ask-human-request',
toolCallId: tc.toolCallId,
subflow: [],
query: typeof input.question === 'string' ? input.question : '',
...(Array.isArray(input.options) && input.options.every((o) => typeof o === 'string')
? { options: input.options }
: {}),
})
}
}
const settled = status === 'completed' || status === 'failed' || status === 'cancelled'
return {
conversation,
currentAssistantMessage: stripVoiceTags(overlay.text),
voiceSegments: overlay.voiceSegments,
pendingAskHumanRequests,
allPermissionRequests,
permissionResponses,
autoPermissionDecisions,
isProcessing: latest !== undefined && !settled,
isThinking: status === 'idle',
}
}

109
apps/x/apps/renderer/src/lib/tour-sounds.ts Normal file
View file

@ -0,0 +1,109 @@
/**
* Tiny synthesized sound effects for the product tour — oar splashes, dock
* bumps, and an arrival fanfare. Everything is generated with Web Audio
* oscillators/noise so no audio assets are needed. All methods fail silently
* if audio is unavailable.
*/
export class TourSounds {
private ctx: AudioContext | null = null
private master: GainNode | null = null
private ensure(): AudioContext | null {
try {
if (!this.ctx) {
this.ctx = new AudioContext()
this.master = this.ctx.createGain()
this.master.gain.value = 0.5
this.master.connect(this.ctx.destination)
}
if (this.ctx.state === 'suspended') void this.ctx.resume()
return this.ctx
} catch {
return null
}
}
/** Short filtered-noise burst with a falling pitch — an oar dipping. */
splash() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const dur = 0.22
const noise = ctx.createBufferSource()
const buffer = ctx.createBuffer(1, Math.ceil(ctx.sampleRate * dur), ctx.sampleRate)
const data = buffer.getChannelData(0)
for (let i = 0; i < data.length; i++) data[i] = Math.random() * 2 - 1
noise.buffer = buffer
const filter = ctx.createBiquadFilter()
filter.type = 'bandpass'
filter.Q.value = 1.2
filter.frequency.setValueAtTime(1600, ctx.currentTime)
filter.frequency.exponentialRampToValueAtTime(350, ctx.currentTime + dur)
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.14, ctx.currentTime)
gain.gain.exponentialRampToValueAtTime(0.001, ctx.currentTime + dur)
noise.connect(filter).connect(gain).connect(this.master)
noise.start()
noise.stop(ctx.currentTime + dur)
}
/** Soft low thump — the boat nudging a dock. */
bump() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const osc = ctx.createOscillator()
osc.type = 'sine'
osc.frequency.setValueAtTime(150, ctx.currentTime)
osc.frequency.exponentialRampToValueAtTime(70, ctx.currentTime + 0.18)
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.2, ctx.currentTime)
gain.gain.exponentialRampToValueAtTime(0.001, ctx.currentTime + 0.25)
osc.connect(gain).connect(this.master)
osc.start()
osc.stop(ctx.currentTime + 0.3)
}
/** Gentle high blip — an email landing in the boat. */
ding() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const osc = ctx.createOscillator()
osc.type = 'sine'
osc.frequency.setValueAtTime(880, ctx.currentTime)
osc.frequency.exponentialRampToValueAtTime(1320, ctx.currentTime + 0.05)
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.08, ctx.currentTime)
gain.gain.exponentialRampToValueAtTime(0.001, ctx.currentTime + 0.3)
osc.connect(gain).connect(this.master)
osc.start()
osc.stop(ctx.currentTime + 0.35)
}
/** Little four-note arpeggio for the tour finale. */
fanfare() {
const ctx = this.ensure()
if (!ctx || !this.master) return
const notes = [523.25, 659.25, 783.99, 1046.5] // C5 E5 G5 C6
notes.forEach((freq, i) => {
const start = ctx.currentTime + i * 0.13
const osc = ctx.createOscillator()
osc.type = 'triangle'
osc.frequency.value = freq
const gain = ctx.createGain()
gain.gain.setValueAtTime(0.0001, start)
gain.gain.exponentialRampToValueAtTime(0.16, start + 0.02)
gain.gain.exponentialRampToValueAtTime(0.001, start + (i === notes.length - 1 ? 0.7 : 0.3))
osc.connect(gain).connect(this.master!)
osc.start(start)
osc.stop(start + 0.8)
})
}
dispose() {
this.ctx?.close().catch(() => {})
this.ctx = null
this.master = null
}
}

13
apps/x/apps/renderer/src/main.tsx
View file

@ -6,6 +6,7 @@ import { PostHogProvider } from 'posthog-js/react'
import type { CaptureResult } from 'posthog-js'
import { ThemeProvider } from '@/contexts/theme-context'
import { configureAnalyticsContext } from './lib/analytics'
import { VideoPopout } from '@/components/video-popout'
// Fetch the stable installation ID from main so renderer + main share one
// PostHog distinct_id. Falls back to PostHog's auto-generated anonymous ID
@ -57,4 +58,14 @@ async function bootstrap() {
// The loaded callback applies api_url/app_version once PostHog has initialized.
}
bootstrap()
// The video-mode popout window loads the same bundle with a hash route and
// renders only the mini-call UI — no analytics or app bootstrap needed.
if (window.location.hash === '#video-popout') {
createRoot(document.getElementById('root')!).render(
<StrictMode>
<VideoPopout />
</StrictMode>,
)
} else {
bootstrap()
}

48
apps/x/apps/renderer/src/styles/editor.css
View file

@ -1551,9 +1551,9 @@
/* Compose / draft box */
.tiptap-editor .ProseMirror .email-gmail-compose {
margin-top: 4px;
border: 1px solid color-mix(in srgb, var(--foreground) 15%, transparent);
border: 1px solid var(--border);
border-radius: 8px;
box-shadow: 0 1px 3px 0 rgba(0, 0, 0, 0.08);
background: var(--background);
overflow: hidden;
}
@ -1562,13 +1562,13 @@
align-items: center;
gap: 6px;
padding: 8px 12px 6px;
border-bottom: 1px solid color-mix(in srgb, var(--foreground) 8%, transparent);
border-bottom: 1px solid var(--border);
}
.tiptap-editor .ProseMirror .email-gmail-compose-to-label {
font-size: 12px;
font-weight: 500;
color: color-mix(in srgb, var(--foreground) 45%, transparent);
color: var(--muted-foreground);
}
.tiptap-editor .ProseMirror .email-gmail-compose-to-addr {
@ -1592,7 +1592,7 @@
}
.tiptap-editor .ProseMirror .email-gmail-compose-body::placeholder {
color: color-mix(in srgb, var(--foreground) 35%, transparent);
color: var(--muted-foreground);
}
.tiptap-editor .ProseMirror .email-gmail-compose-footer {
@ -1600,7 +1600,7 @@
align-items: center;
gap: 8px;
padding: 8px 12px;
border-top: 1px solid color-mix(in srgb, var(--foreground) 8%, transparent);
border-top: 1px solid var(--border);
}
/* Action buttons */
@ -1615,35 +1615,33 @@
display: inline-flex;
align-items: center;
gap: 6px;
padding: 7px 16px;
font-size: 13px;
padding: 7px 12px;
font-size: 0.8125rem;
font-weight: 500;
color: color-mix(in srgb, var(--foreground) 60%, transparent);
color: var(--foreground);
background: transparent;
border: 1px solid color-mix(in srgb, var(--foreground) 20%, transparent);
border-radius: 18px;
border: 1px solid var(--border);
border-radius: 6px;
cursor: pointer;
transition: background-color 0.15s ease, box-shadow 0.15s ease;
transition: background-color 0.15s ease, border-color 0.15s ease, color 0.15s ease;
width: fit-content;
letter-spacing: 0.01em;
}
.tiptap-editor .ProseMirror .email-gmail-btn:hover {
background: color-mix(in srgb, var(--foreground) 6%, transparent);
box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.1);
color: var(--foreground);
background: var(--accent);
color: var(--accent-foreground);
}
.tiptap-editor .ProseMirror .email-gmail-btn-primary {
color: #fff;
background: #1a73e8;
border-color: #1a73e8;
color: var(--primary-foreground);
background: var(--primary);
border-color: var(--primary);
}
.tiptap-editor .ProseMirror .email-gmail-btn-primary:hover {
background: #1765cc;
box-shadow: 0 1px 2px 0 rgba(26, 115, 232, 0.45), 0 1px 3px 1px rgba(26, 115, 232, 0.3);
color: #fff;
background: color-mix(in oklab, var(--primary) 90%, transparent);
border-color: color-mix(in oklab, var(--primary) 90%, transparent);
color: var(--primary-foreground);
}
.tiptap-editor .ProseMirror .email-block-error,
@ -1690,9 +1688,8 @@
/* ---- Emails inbox block (language-emails) ---- */
.tiptap-editor .ProseMirror .email-inbox-card {
font-family: 'Google Sans', Roboto, RobotoDraft, Helvetica, Arial, sans-serif;
}
/* No font-family override — inherit the editor's app font (ui-sans-serif /
system-ui) so the inbox block matches the rest of the Rowboat UI. */
.tiptap-editor .ProseMirror .email-inbox-title {
font-size: 14px;
@ -1768,7 +1765,6 @@
.tiptap-editor .ProseMirror .email-inbox-sender {
font-size: 14px;
font-weight: 500;
font-family: 'Google Sans', Roboto, RobotoDraft, Helvetica, Arial, sans-serif;
color: var(--foreground);
white-space: nowrap;
overflow: hidden;

1
apps/x/apps/renderer/src/test/setup.ts Normal file
View file

@ -0,0 +1 @@
import '@testing-library/jest-dom/vitest'

8
apps/x/apps/renderer/vite.config.ts
View file

@ -1,5 +1,5 @@
import path from "path"
import { defineConfig } from 'vite'
import { defineConfig } from 'vitest/config'
import react from '@vitejs/plugin-react'
import tailwindcss from '@tailwindcss/vite'
@ -18,4 +18,10 @@ export default defineConfig({
build: {
outDir: 'dist',
},
test: {
environment: 'jsdom',
setupFiles: ['./src/test/setup.ts'],
include: ['src/**/*.test.{ts,tsx}'],
css: false,
},
})

8
apps/x/package.json
View file

@ -12,7 +12,11 @@
"deps": "npm run shared && npm run core && npm run preload",
"main": "wait-on http://localhost:5173 && cd apps/main && npm run build && npm run start",
"lint": "eslint .",
"lint:fix": "eslint . --fix"
"lint:fix": "eslint . --fix",
"test": "npm run shared && npm run test:shared && npm run test:core && npm run test:renderer",
"test:shared": "cd packages/shared && npm test",
"test:core": "cd packages/core && npm test",
"test:renderer": "cd apps/renderer && npm test"
},
"devDependencies": {
"@eslint/js": "^9.39.2",
@ -26,4 +30,4 @@
"typescript-eslint": "^8.50.1",
"wait-on": "^9.0.3"
}
}
}

561
apps/x/packages/core/docs/session-design.md Normal file
View file

@ -0,0 +1,561 @@
# Session Layer Technical Specification
Status: design complete for the session layer v1. No implementation exists
yet.
This document specifies the session layer that sits above the turn runtime
defined in `turn-runtime-design.md`. That document is assumed context; this
one does not restate turn semantics.
## 1. Goals
The session layer must:
1. Own conversations composed of ordered turns.
2. Persist each session as one append-only JSONL file with the same
validation discipline as turn files.
3. Enforce one active turn per session.
4. Assemble each turn's context as a reference to the previous turn.
5. Maintain an in-memory index for listing, sorting, and filtering sessions,
updated write-through and rebuilt by scanning at startup.
6. Route external inputs — permission decisions, ask-human answers, async
tool results — to the correct turn through dedicated APIs.
7. Forward live turn events to the renderer over IPC.
8. Provide headless standalone turns outside any session.
## 2. Non-goals (v1)
- Queued user messages. The committed future shape is in section 12.1.
- Steering / mid-turn message injection (section 12.4).
- Session-scoped permission grants ("always allow for this chat"); every
applicable tool call prompts in v1 (section 12.2).
- Context compaction; a viable mechanism sketch is recorded in section 12.3.
V1 behavior on context overflow is the turn-level model failure.
- LLM auto-titling (section 12.6).
- A persisted index cache; startup always scans (section 12.5).
- Cross-process coordination. A single main process is enforced.
- Data migration from the current runs system. Old conversations are not
converted; the old code path remains readable until it is deleted.
- Session list pagination. The index is in-memory and shipped whole.
## 3. Terminology
A **session** is a durable, ordered chain of turns plus presentation
metadata (title). Conversation content lives exclusively in turn files; the
session file stores turn references with denormalized metadata.
The **index** is an in-memory projection over all session files, used for
the session list UI. It is never a source of truth.
A **standalone turn** is a turn with `sessionId: null`, created outside any
session by headless callers. Standalone turns do not appear in the index.
## 4. Storage design
### 4.1 File location
Session files live under:
```text
WorkDir/storage/sessions/YYYY/MM/DD/<sessionId>.jsonl
```
Session IDs come from the existing
`IMonotonicallyIncreasingIdGenerator`, and the repository derives the
date-partitioned path from the ID exactly as the turn repository does,
including format validation and path-traversal rejection.
### 4.2 File rules
Identical discipline to turn files:
- The first line is always `session_created` with `schemaVersion: 1`.
- Every event contains `sessionId` and an ISO timestamp `ts`.
- Physical line order is authoritative.
- Reads validate every line strictly; any malformed line makes the session
corrupt; no truncation, repair, or skipping.
- Unknown schema versions and unknown event types fail loudly. Future
additive event types (queueing, grants, compaction) arrive as a schema
version bump; the reducer will accept old and new versions and write the
newest.
- Appends are awaited but not explicitly `fsync`ed.
### 4.3 Repository contract
```ts
interface ISessionRepo {
create(event: SessionCreated): Promise<void>;
read(sessionId: string): Promise<SessionEvent[]>;
append(sessionId: string, events: SessionEvent[]): Promise<void>;
withLock<T>(sessionId: string, fn: () => Promise<T>): Promise<T>;
listSessionIds(): Promise<string[]>;
delete(sessionId: string): Promise<void>;
}
```
- `create` fails if the file exists.
- `listSessionIds` enumerates the partition directories for the startup
scan.
- `delete` removes the session file only. Turn files referenced by the
session are left in place as harmless orphans (see section 9, deletion).
- `withLock` is in-process per-session exclusion, mirroring the turn repo.
## 5. Event schemas
All session event schemas live in `@x/shared` alongside the turn schemas.
```ts
interface BaseSessionEvent {
sessionId: string;
ts: string;
}
interface SessionCreated extends BaseSessionEvent {
type: "session_created";
schemaVersion: 1;
title?: string;
}
interface SessionTurnAppended extends BaseSessionEvent {
type: "turn_appended";
turnId: string;
sessionSeq: number; // 1-based position of the turn in the session
agentId: string;
model: ModelDescriptor; // resolved provider/model for the turn
}
interface SessionTitleChanged extends BaseSessionEvent {
type: "title_changed";
title: string;
}
type SessionEvent =
| SessionCreated
| SessionTurnAppended
| SessionTitleChanged;
```
`turn_appended` deliberately denormalizes `agentId` and `model` from the
turn so the index can fold from session files without opening turn files.
The turn file remains authoritative for the turn's actual configuration.
The session file never mirrors turn outcomes. Turn lifecycle facts live only
in turn files; deriving "is this session busy/suspended/failed" reads the
latest turn (section 8).
## 6. Session reducer
`@x/shared` owns one pure reducer shared by core and renderer:
```ts
function reduceSession(events: SessionEvent[]): SessionState;
interface SessionState {
definition: SessionCreated;
title?: string;
turns: Array<{
turnId: string;
sessionSeq: number;
agentId: string;
model: ModelDescriptor;
ts: string;
}>;
latestTurnId?: string;
createdAt: string; // definition.ts
updatedAt: string; // ts of the last event
}
```
Invariants (violations throw, as with the turn reducer):
- `session_created` is present, first, and unique.
- All event `sessionId` values match.
- `sessionSeq` is strictly increasing starting at 1, with no gaps.
- `turnId` values are unique.
- Unsupported schema versions and unknown event types fail loudly.
## 7. Write ordering and consistency
Per user message, the session layer performs, in order:
1. `turnRuntime.createTurn(...)` — the turn file is created.
2. `sessionRepo.append(turn_appended)` — the session references the turn.
3. `advanceTurn(...)` — execution begins.
Rules:
- A crash between steps 1 and 2 leaves an orphan turn file: unreferenced,
never advanced, and benign. Turns are only ever found by reference, so an
orphan is invisible. V1 does not garbage-collect orphans.
- The reverse order is forbidden: a `turn_appended` referencing a turn file
that was never created would be a dangling reference, which is corruption.
- Step 2 precedes step 3 so that a turn that is executing is always already
referenced by its session.
- Session-file appends happen under the session lock; turn-file appends are
the turn runtime's concern.
## 8. In-memory index
### 8.1 Shape
```ts
interface SessionIndexEntry {
sessionId: string;
title?: string;
createdAt: string;
updatedAt: string;
turnCount: number;
lastAgentId?: string;
lastModel?: ModelDescriptor;
latestTurnId?: string;
latestTurnStatus:
| "none" // session has no turns yet
| "completed"
| "failed"
| "cancelled"
| "suspended" // durable suspension: pending permissions/async tools
| "idle"; // non-terminal, not suspended: interrupted by a crash
}
```
`latestTurnStatus` is derived from the latest turn's reduced state, using
the same derivation everywhere: terminal event kind if present, else
`suspended` if a suspension with outstanding work is the resting state, else
`idle`. Whether a turn is *actively processing right now* is not in the
index; it is ephemeral bus state (`turn-processing-start/end`), per the turn
specification.
### 8.2 Startup scan
1. `listSessionIds()`.
2. For each session: read and reduce the session file, producing the entry's
session-derived fields.
3. For each session with turns: read and reduce the latest turn file only,
producing `latestTurnStatus`.
4. Publish the completed index to the renderer.
A corrupt session file or corrupt latest-turn file does not abort startup:
the entry is surfaced in an errored state (identifiable in the UI, excluded
from normal interaction) and the scan continues.
### 8.3 Maintenance
- Every session mutation updates the entry in the same code path that
appends to the session file (write-through), then publishes a
`session-index-changed` event on the application bus.
- When an `advanceTurn` outcome settles, the session layer updates
`latestTurnStatus` and publishes `session-index-changed`.
- There is no filesystem watcher. Out-of-band edits to session or turn files
while the app runs are unsupported; offline changes are reconciled by the
next startup scan.
- The main process enforces single-instance via
`app.requestSingleInstanceLock()`; all locking in both layers is
in-process.
## 9. Sessions API
```ts
interface SendMessageConfig {
agent: RequestedAgent; // agent id + optional model override
autoPermission?: boolean; // default false
maxModelCalls?: number; // default per turn spec
}
interface ISessions {
createSession(input?: { title?: string }): Promise<string>;
listSessions(): SessionIndexEntry[];
getSession(sessionId: string): Promise<SessionState>;
getTurn(turnId: string): Promise<Turn>; // passthrough to turn runtime
sendMessage(
sessionId: string,
input: UserMessage,
config: SendMessageConfig,
): Promise<{ turnId: string }>;
respondToPermission(
turnId: string,
toolCallId: string,
decision: "allow" | "deny",
metadata?: JsonValue,
): Promise<void>;
respondToAskHuman(
turnId: string,
toolCallId: string,
answer: string,
): Promise<void>;
deliverAsyncToolResult(
turnId: string,
toolCallId: string,
result: ToolResultData,
): Promise<void>;
stopTurn(turnId: string, reason?: string): Promise<void>;
resumeTurn(sessionId: string): Promise<void>;
setTitle(sessionId: string, title: string): Promise<void>;
deleteSession(sessionId: string): Promise<void>;
}
```
### 9.1 sendMessage
Under the per-session lock:
1. Read and reduce the session.
2. If the session has turns, read and reduce the latest turn. If it is
non-terminal — running, suspended, or idle — reject with a typed
`TurnNotSettledError`. There is no implicit queueing, steering, or
cancel-and-replace, and no implicit routing to a pending ask-human.
3. Build context: `[]` (inline, empty) for the first turn, else
`{ previousTurnId: latestTurnId }`.
4. Create the turn: config from `SendMessageConfig` lands on the turn
(`humanAvailable: true` always, for session turns). Sessions store no
configuration; every turn is self-describing.
5. Append `turn_appended` with the next `sessionSeq` and denormalized
agent/model.
6. If the session has no title, append `title_changed` derived from the
truncated first user message.
7. Start `advanceTurn` in the background; consume its events (section 10).
8. Return `{ turnId }` immediately. The renderer follows progress through
events, not the return value.
Continuation after a failed or exhausted (`code: "model-call-limit"`) turn
is just `sendMessage`: failed turns are terminal, and the new turn's context
reference includes the failed turn's structurally complete transcript.
### 9.2 External inputs
`respondToPermission`, `respondToAskHuman`, and `deliverAsyncToolResult`
each translate to one `advanceTurn(turnId, input)` call with the
corresponding `TurnExternalInput`. `respondToAskHuman` is the dedicated
endpoint for the `ask-human` tool — a thin wrapper over
`async_tool_result` — and is deliberately separate from `sendMessage`.
Validation (unknown call, already-resolved call, terminal turn) is the turn
runtime's job; the session layer passes its errors through.
### 9.3 stopTurn and resumeTurn
- `stopTurn` cancels via the turn runtime: aborting the active invocation's
signal if one is running, else advancing a suspended turn with a `cancel`
input.
- `resumeTurn` re-enters the latest turn with no input — the turn spec's
recovery entry point — for turns left `idle` by a crash. There is no
automatic resume sweep at startup: recovery re-issues interrupted model
calls, so resumption must be an explicit user action. Suspended turns need
no resumption; they advance when their inputs arrive.
### 9.4 Deletion
`deleteSession` removes the session file and the index entry, and publishes
`session-index-changed`. Referenced turn files are retained as orphans:
turns are only discoverable by reference, so orphaned files are inert.
Deleting an entity's file is not a violation of append-only discipline,
which governs mutation of live logs, not their removal.
## 10. Event forwarding and live UI
1. For every `advanceTurn` it initiates, the session layer consumes
`TurnExecution.events` and forwards each `TurnStreamEvent` — tagged with
`sessionId` — through the application bus to renderer windows over one
IPC channel (`sessions:events`).
2. When `outcome` settles, the session layer updates the index entry and
publishes `session-index-changed`.
3. The renderer follows the turn spec's historical/live pattern: fetch
turns via `getTurn`, run the shared `reduceTurn` per turn, compose the
session timeline turn-by-turn (each turn renders its input and its own
activity; the referenced prefix is never re-rendered from context),
append live durable events and re-reduce, and keep text/reasoning deltas
in an ephemeral overlay cleared by canonical responses.
4. Pending approvals and ask-human prompts render from the suspended turn's
reduced state, so they survive restarts without any session-layer
bookkeeping.
## 11. Headless standalone turns
A helper covers the non-session callers (background tasks, live notes,
knowledge pipelines, scheduled agents):
```ts
function runHeadlessTurn(input: {
agent: RequestedAgent;
context?: ConversationMessage[]; // inline; defaults to []
input: UserMessage;
maxModelCalls?: number;
signal?: AbortSignal;
}): Promise<TurnOutcome>;
```
- `sessionId: null`, `autoPermission: true`, `humanAvailable: false`.
- Creates the turn, advances to the first settled outcome, and returns it.
- Standalone turns never appear in the index; callers keep their own turn
IDs if they need history.
## 12. Deferred designs with committed shapes
These are not implemented in v1. Their shapes are recorded so v1 decisions
stay compatible; each arrives as a session schema version bump.
### 12.1 Queued messages
```ts
{ type: "message_queued", queueId, message, ts }
{ type: "queued_message_replaced", queueId, message, ts } // edit
{ type: "queued_message_removed", queueId, ts } // cancel
// promotion: turn_appended gains consumedQueueIds?: string[]
```
The reducer derives the pending queue by supersession. Promotion rules
(collapse-into-one-turn vs FIFO, behavior after failed turns) are decided
together with steering.
### 12.2 Session permission grants
```ts
{ type: "permission_grant_added", grantId, toolId, ts }
{ type: "permission_grant_removed", grantId, ts }
```
The injected `IPermissionChecker` consults a session-keyed grants view
before answering `required: true`. V1 grants would be blanket per-toolId;
argument-pattern matchers are a separate, security-sensitive project.
### 12.3 Compaction (mechanism sketch)
Compaction requires zero turn-schema change. A session-level compaction
event records `{ compactionId, summary, firstKeptTurnId }`; the next turn
after a compaction uses **inline** context (summary message + kept
transcript), which restarts the reference chain and bounds resolution depth
by construction. Trigger policy and summarizer design are unspecified.
### 12.4 Steering
Rides the queue events with a different promotion rule, and additionally
requires a turn-level `steer` external input (a turn schema bump) with an
injection boundary after tool-batch completion. Recorded here so the session
design does not foreclose it.
### 12.5 Persisted index cache
If the startup scan ever gets slow, a single cache file keyed by file
mtimes can be added. It is a rebuildable cache, never a source of truth:
missing, stale, or invalid means rebuild from session files.
### 12.6 Auto-titling
An LLM-generated title replacing the truncated-first-message default,
appended as an ordinary `title_changed` event.
## 13. Required test scenarios
All tests use the in-memory/mocked turn runtime and repo fakes.
### 13.1 Reducer
- Valid event sequences reduce to expected state.
- Every invariant violation throws: missing/duplicate `session_created`,
mismatched sessionId, non-monotonic or gapped `sessionSeq`, duplicate
turnIds, unknown type/version.
- Title folding: default, explicit changes, last-wins.
### 13.2 Repository
- Date-partitioned paths, ID validation, create-if-absent.
- Strict line validation on read; corrupt files rejected whole.
- `listSessionIds` enumeration across partitions.
- Deletion removes only the session file.
### 13.3 sendMessage
- First turn: inline `[]` context, `sessionSeq: 1`, default title appended.
- Subsequent turns: context references the latest turn; seq increments.
- Rejection with `TurnNotSettledError` when the latest turn is running,
suspended, or idle — and success after it settles.
- Continuation after failed and model-call-limit turns.
- Concurrent sendMessage calls serialize under the session lock; exactly
one wins, the other rejects.
- Config lands on the turn; the session file stores only denormalized
agent/model on `turn_appended`.
### 13.4 Ordering and crash simulation
- Simulated crash between `createTurn` and `turn_appended`: orphan turn
file, session unchanged, retry produces a fresh turn.
- `turn_appended` is present before the first advance begins.
### 13.5 External inputs
- Permission decision, ask-human answer, and async result each advance the
correct turn with the correct input type.
- Turn-runtime rejections (unknown call, terminal turn) pass through.
- `sendMessage` never routes to ask-human.
### 13.6 Index
- Startup fold matches write-through state for the same history.
- Latest-turn status derivation for every status value.
- Corrupt session file yields an errored entry without aborting the scan.
- Mutations publish `session-index-changed`; deletion removes the entry.
### 13.7 Event forwarding
- Forwarded events are tagged with sessionId and arrive in order.
- Outcome settlement updates `latestTurnStatus`.
### 13.8 Headless
- Standalone turns: `sessionId: null`, auto permission, human unavailable,
absent from the index.
## 14. Suggested module layout
```text
apps/x/packages/shared/src/sessions.ts # event schemas, reducer, index types
apps/x/packages/core/src/sessions/
sessions.ts # ISessions implementation
repo.ts # ISessionRepo contract
fs-repo.ts # filesystem implementation
index.ts # in-memory index + startup scan
headless.ts # runHeadlessTurn
```
Awilix registration mirrors the turn runtime: singleton scope, PROXY
constructor injection, no container resolution from inside the classes.
## 15. Integration sequence
The rollout is staged as commits on one branch (squash-merge acceptable);
old and new stacks coexist briefly but never share state, and no data is
migrated:
1. `@x/shared`: turn + session schemas and reducers.
2. Turn runtime + fs turn repo (unit tests only, wired to nothing).
3. Session layer + index (unit tests only).
4. Bridges: agent resolver, context resolver, tool runner, permission
checker/classifier — adapted from the `new-runtime` reference
implementation where applicable.
5. IPC (`sessions:*`) + renderer swap: Copilot chat UI moves to the
sessions API.
6. Headless callers move to `runHeadlessTurn`.
7. Delete the old runs runtime.
## 16. Implementation acceptance criteria
The session layer is implementation-complete only when:
1. Session event schemas and `reduceSession` live in `@x/shared` and are
consumed unchanged by core and renderer.
2. Session files follow the partitioned append-only JSONL layout with
strict validation.
3. Turn-file-first write ordering is enforced; orphan turns are benign.
4. `sendMessage` rejects non-terminal latest turns with a typed error; no
implicit queueing, steering, or ask-human routing exists.
5. Ask-human answers flow only through the dedicated endpoint.
6. The index is write-through with a startup scan, no watcher, and no
persisted cache; single-instance is enforced.
7. Deletion removes only the session file.
8. Headless callers run standalone turns and appear nowhere in the index.
9. All required test scenarios pass with mocked dependencies.

1887
apps/x/packages/core/docs/turn-runtime-design.md Normal file
View file

File diff suppressed because it is too large Load diff

7
apps/x/packages/core/package.json
View file

@ -8,7 +8,9 @@
"build": "rm -rf dist && tsc -p tsconfig.build.json",
"dev": "tsc -w -p tsconfig.build.json",
"test": "vitest run",
"test:watch": "vitest"
"test:watch": "vitest",
"inspect-turn": "node dist/turns/inspect-cli.js",
"inspect": "node dist/turns/inspect-cli.js"
},
"dependencies": {
"@agentclientprotocol/claude-agent-acp": "^0.39.0",
@ -28,6 +30,7 @@
"@x/shared": "workspace:*",
"ai": "^5.0.133",
"awilix": "^12.0.5",
"baileys": "7.0.0-rc13",
"chokidar": "^4.0.3",
"cors": "^2.8.6",
"cron-parser": "^5.5.0",
@ -43,6 +46,7 @@
"papaparse": "^5.5.3",
"pdf-parse": "^2.4.5",
"posthog-node": "^4.18.0",
"qrcode": "^1.5.4",
"react": "^19.2.3",
"xlsx": "^0.18.5",
"yaml": "^2.8.2",
@ -54,6 +58,7 @@
"@types/node": "^25.0.3",
"@types/papaparse": "^5.5.2",
"@types/pdf-parse": "^1.1.5",
"@types/qrcode": "^1.5.6",
"vitest": "catalog:"
}
}

58
apps/x/packages/core/src/agent-schedule/runner.ts
View file

@ -2,13 +2,10 @@ import { CronExpressionParser } from "cron-parser";
import container from "../di/container.js";
import { IAgentScheduleRepo } from "./repo.js";
import { IAgentScheduleStateRepo } from "./state-repo.js";
import { IRunsRepo } from "../runs/repo.js";
import { IAgentRuntime } from "../agents/runtime.js";
import { IMonotonicallyIncreasingIdGenerator } from "../application/lib/id-gen.js";
import { AgentScheduleConfig, AgentScheduleEntry } from "@x/shared/dist/agent-schedule.js";
import { AgentScheduleState, AgentScheduleStateEntry } from "@x/shared/dist/agent-schedule-state.js";
import { MessageEvent } from "@x/shared/dist/runs.js";
import { createRun } from "../runs/runs.js";
import { startHeadlessAgent } from "../agents/headless-app.js";
import { withUseCase } from "../analytics/use_case.js";
import z from "zod";
const DEFAULT_STARTING_MESSAGE = "go";
@ -147,10 +144,7 @@ function shouldRunNow(
async function runAgent(
agentName: string,
entry: z.infer<typeof AgentScheduleEntry>,
stateRepo: IAgentScheduleStateRepo,
runsRepo: IRunsRepo,
agentRuntime: IAgentRuntime,
idGenerator: IMonotonicallyIncreasingIdGenerator
stateRepo: IAgentScheduleStateRepo
): Promise<void> {
console.log(`[AgentRunner] Starting agent: ${agentName}`);
@ -163,31 +157,24 @@ async function runAgent(
});
try {
// Create a new run via core (resolves agent + default model+provider).
const run = await createRun({
agentId: agentName,
useCase: 'copilot_chat',
subUseCase: 'scheduled',
});
console.log(`[AgentRunner] Created run ${run.id} for agent ${agentName}`);
// Add the starting message as a user message
// Start a standalone turn (resolves agent + default model/provider).
// Fire-and-forget like the old trigger(): the schedule state machine
// below runs on wall-clock; completion is observed only for logging.
// The signal caps a hung turn at the same TIMEOUT_MS the state-based
// watchdog uses.
const startingMessage = entry.startingMessage ?? DEFAULT_STARTING_MESSAGE;
const messageEvent: z.infer<typeof MessageEvent> = {
runId: run.id,
type: "message",
messageId: await idGenerator.next(),
message: {
role: "user",
content: startingMessage,
},
subflow: [],
};
await runsRepo.appendEvents(run.id, [messageEvent]);
console.log(`[AgentRunner] Sent starting message to agent ${agentName}: "${startingMessage}"`);
// Trigger the run
await agentRuntime.trigger(run.id);
const handle = await withUseCase(
{ useCase: 'copilot_chat', subUseCase: 'scheduled' },
() => startHeadlessAgent({
agentId: agentName,
message: startingMessage,
signal: AbortSignal.timeout(TIMEOUT_MS),
}),
);
console.log(`[AgentRunner] Started turn ${handle.turnId} for agent ${agentName}: "${startingMessage}"`);
void handle.done
.then((result) => console.log(`[AgentRunner] Turn ${handle.turnId} (${agentName}) settled: ${result.outcome.status}`))
.catch((error) => console.error(`[AgentRunner] Turn ${handle.turnId} (${agentName}) errored:`, error instanceof Error ? error.message : error));
// Calculate next run time
const nextRunAt = calculateNextRunAt(entry.schedule);
@ -264,9 +251,6 @@ async function checkForTimeouts(
async function pollAndRun(): Promise<void> {
const scheduleRepo = container.resolve<IAgentScheduleRepo>("agentScheduleRepo");
const stateRepo = container.resolve<IAgentScheduleStateRepo>("agentScheduleStateRepo");
const runsRepo = container.resolve<IRunsRepo>("runsRepo");
const agentRuntime = container.resolve<IAgentRuntime>("agentRuntime");
const idGenerator = container.resolve<IMonotonicallyIncreasingIdGenerator>("idGenerator");
// Load config and state
let config: z.infer<typeof AgentScheduleConfig>;
@ -314,7 +298,7 @@ async function pollAndRun(): Promise<void> {
if (shouldRunNow(entry, agentState)) {
// Run agent (don't await - let it run in background)
runAgent(agentName, entry, stateRepo, runsRepo, agentRuntime, idGenerator).catch((error) => {
runAgent(agentName, entry, stateRepo).catch((error) => {
console.error(`[AgentRunner] Unhandled error in runAgent for ${agentName}:`, error);
});
}

25
apps/x/packages/core/src/agents/headless-app.ts Normal file
View file

@ -0,0 +1,25 @@
import container from "../di/container.js";
import {
type HeadlessAgentHandle,
type HeadlessAgentOptions,
type HeadlessAgentResult,
type IHeadlessAgentRunner,
} from "./headless.js";
function runner(): IHeadlessAgentRunner {
return container.resolve<IHeadlessAgentRunner>("headlessAgentRunner");
}
export function startHeadlessAgent(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentHandle> {
return runner().start(options);
}
export function runHeadlessAgent(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentResult & { turnId: string }> {
return runner().run(options);
}
export { toolInputPaths } from "./headless.js";

300
apps/x/packages/core/src/agents/headless.test.ts Normal file
View file

@ -0,0 +1,300 @@
import { describe, expect, it, vi } from "vitest";
import type { z } from "zod";
import { reduceTurn, type TurnEvent } from "@x/shared/dist/turns.js";
import type {
CreateTurnInput,
ITurnRuntime,
TurnExecution,
TurnOutcome,
} from "../turns/api.js";
import {
HeadlessRunError,
HeadlessAgentRunner,
assistantText,
lastAssistantText,
toolInputPaths,
} from "./headless.js";
import type { IDefaultModelResolver } from "../models/default-model-resolver.js";
type TEvent = z.infer<typeof TurnEvent>;
const TURN_ID = "2026-07-02T10-00-00Z-0000001-000";
const TS = "2026-07-02T10:00:00Z";
const echoTool = {
toolId: "builtin:file-editText",
name: "file-editText",
description: "Edit",
inputSchema: {},
execution: "sync" as const,
requiresHuman: false,
};
function turnLog(opts: {
responseText?: string;
toolCalls?: Array<{ id: string; name: string; input: unknown; invoked: boolean }>;
failed?: string;
}): TEvent[] {
const events: TEvent[] = [
{
type: "turn_created",
schemaVersion: 1,
turnId: TURN_ID,
ts: TS,
sessionId: null,
agent: {
requested: { agentId: "worker" },
resolved: {
agentId: "worker",
systemPrompt: "SYS",
model: { provider: "fake", model: "m" },
tools: [echoTool],
},
},
context: [],
input: { role: "user", content: "go" },
config: { autoPermission: true, humanAvailable: false, maxModelCalls: 20 },
},
{
type: "model_call_requested",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
request: { messages: ["input"], parameters: {} },
},
];
const toolCalls = opts.toolCalls ?? [];
if (toolCalls.length > 0) {
events.push({
type: "model_call_completed",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
message: {
role: "assistant",
content: toolCalls.map((tc) => ({
type: "tool-call" as const,
toolCallId: tc.id,
toolName: tc.name,
arguments: tc.input as never,
})),
},
finishReason: "tool-calls",
usage: {},
});
for (const tc of toolCalls) {
if (!tc.invoked) continue;
events.push({
type: "tool_invocation_requested",
turnId: TURN_ID,
ts: TS,
toolCallId: tc.id,
toolId: "builtin:" + tc.name,
toolName: tc.name,
execution: "sync",
input: tc.input as never,
});
events.push({
type: "tool_result",
turnId: TURN_ID,
ts: TS,
toolCallId: tc.id,
toolName: tc.name,
source: "sync",
result: { output: "ok", isError: false },
});
}
}
if (opts.failed) {
events.push({
type: "model_call_failed",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
error: opts.failed,
});
events.push({ type: "turn_failed", turnId: TURN_ID, ts: TS, error: opts.failed, usage: {} });
} else if (opts.responseText !== undefined && toolCalls.length === 0) {
events.push({
type: "model_call_completed",
turnId: TURN_ID,
ts: TS,
modelCallIndex: 0,
message: { role: "assistant", content: opts.responseText },
finishReason: "stop",
usage: {},
});
events.push({
type: "turn_completed",
turnId: TURN_ID,
ts: TS,
output: { role: "assistant", content: opts.responseText },
finishReason: "stop",
usage: {},
});
}
return events;
}
class FakeRuntime implements ITurnRuntime {
createInputs: CreateTurnInput[] = [];
constructor(
private readonly log: TEvent[],
private readonly outcome: TurnOutcome,
) {}
async createTurn(input: CreateTurnInput): Promise<string> {
this.createInputs.push(input);
return TURN_ID;
}
advanceTurn(): TurnExecution {
return {
events: (async function* () {})(),
outcome: Promise.resolve(this.outcome),
};
}
async getTurn() {
return { turnId: TURN_ID, events: this.log };
}
}
const completedOutcome: TurnOutcome = {
status: "completed",
output: { role: "assistant", content: "done" },
finishReason: "stop",
usage: {},
};
function createRunner(
runtime: ITurnRuntime,
resolve = vi.fn(async () => ({
provider: "default-provider",
model: "default-model",
})),
): { runner: HeadlessAgentRunner; resolve: IDefaultModelResolver["resolve"] } {
return {
runner: new HeadlessAgentRunner({
turnRuntime: runtime,
defaultModelResolver: { resolve },
}),
resolve,
};
}
describe("headless agent helpers", () => {
it("assistantText handles string and parts content", () => {
expect(assistantText({ role: "assistant", content: "hi" })).toBe("hi");
expect(
assistantText({
role: "assistant",
content: [
{ type: "reasoning", text: "hmm" },
{ type: "text", text: "a" },
{ type: "text", text: "b" },
],
}),
).toBe("ab");
expect(assistantText({ role: "assistant", content: "" })).toBeNull();
});
it("lastAssistantText returns the final assistant text in the transcript", () => {
const state = reduceTurn(turnLog({ responseText: "the summary" }));
expect(lastAssistantText(state)).toBe("the summary");
expect(lastAssistantText(reduceTurn(turnLog({ failed: "boom" })))).toBeNull();
});
it("toolInputPaths collects paths of invoked calls only", () => {
const state = reduceTurn(
turnLog({
toolCalls: [
{ id: "a", name: "file-editText", input: { path: "notes/x.md" }, invoked: true },
{ id: "b", name: "file-editText", input: { path: "notes/y.md" }, invoked: true },
{ id: "c", name: "file-writeText", input: { path: "notes/z.md" }, invoked: true },
{ id: "d", name: "file-editText", input: { path: "never-ran.md" }, invoked: false },
],
}),
);
expect(toolInputPaths(state, ["file-editText"])).toEqual(
new Set(["notes/x.md", "notes/y.md"]),
);
expect(toolInputPaths(state, ["file-writeText"])).toEqual(new Set(["notes/z.md"]));
});
});
describe("HeadlessAgentRunner", () => {
it("creates a standalone auto-permission turn and returns the id before settling", async () => {
const runtime = new FakeRuntime(turnLog({ responseText: "the summary" }), completedOutcome);
const { runner, resolve } = createRunner(runtime);
const handle = await runner.start({
agentId: "worker",
message: "go",
model: "m",
provider: "fake",
});
expect(handle.turnId).toBe(TURN_ID);
expect(resolve).not.toHaveBeenCalled();
expect(runtime.createInputs[0]).toMatchObject({
agent: { agentId: "worker", overrides: { model: { provider: "fake", model: "m" } } },
sessionId: null,
context: [],
input: { role: "user", content: "go" },
config: { autoPermission: true, humanAvailable: false },
});
const result = await handle.done;
expect(result.outcome.status).toBe("completed");
expect(result.summary).toBe("the summary");
});
it("omits the model override when neither model nor provider is set", async () => {
const runtime = new FakeRuntime(turnLog({ responseText: "ok" }), completedOutcome);
const { runner, resolve } = createRunner(runtime);
await runner.start({ agentId: "worker", message: "go" });
expect(resolve).not.toHaveBeenCalled();
expect(runtime.createInputs[0].agent.overrides).toBeUndefined();
});
it("uses an injected default only for a partial model override", async () => {
const runtime = new FakeRuntime(turnLog({ responseText: "ok" }), completedOutcome);
const { runner, resolve } = createRunner(runtime);
await runner.start({
agentId: "worker",
message: "go",
model: "custom-model",
});
expect(resolve).toHaveBeenCalledOnce();
expect(runtime.createInputs[0].agent.overrides).toEqual({
model: { provider: "default-provider", model: "custom-model" },
});
});
it("throwOnError rejects done with HeadlessRunError on failed outcomes", async () => {
const runtime = new FakeRuntime(turnLog({ failed: "provider exploded" }), {
status: "failed",
error: "provider exploded",
usage: {},
});
const { runner } = createRunner(runtime);
const handle = await runner.start({
agentId: "worker",
message: "go",
throwOnError: true,
});
await expect(handle.done).rejects.toThrowError(HeadlessRunError);
await expect(handle.done).rejects.toThrowError("provider exploded");
});
it("without throwOnError a failed outcome resolves (old wait semantics)", async () => {
const runtime = new FakeRuntime(turnLog({ failed: "boom" }), {
status: "failed",
error: "boom",
usage: {},
});
const { runner } = createRunner(runtime);
const handle = await runner.start({ agentId: "worker", message: "go" });
const result = await handle.done;
expect(result.outcome.status).toBe("failed");
expect(result.summary).toBeNull();
});
});

182
apps/x/packages/core/src/agents/headless.ts Normal file
View file

@ -0,0 +1,182 @@
import type { z } from "zod";
import type { AssistantMessage } from "@x/shared/dist/message.js";
import {
reduceTurn,
type TurnState,
} from "@x/shared/dist/turns.js";
import type { ITurnRuntime, TurnOutcome } from "../turns/api.js";
import type { IDefaultModelResolver } from "../models/default-model-resolver.js";
// Drop-in replacement for the old headless runs pattern
// (createRun → createMessage → waitForRunCompletion → extractAgentResponse):
// one standalone turn per invocation (sessionId null, automatic permissions,
// no human). HeadlessAgentRunner.start returns the turn id immediately (callers
// record it in pointer files / bus events before completion); `done` settles
// with the outcome, the reduced turn state, and the final assistant text.
export class HeadlessRunError extends Error {
constructor(
message: string,
readonly turnId: string,
readonly outcome: TurnOutcome,
) {
super(message);
this.name = "HeadlessRunError";
}
}
export interface HeadlessAgentOptions {
agentId: string;
message: string;
// Model id; when set without provider, the app-default provider applies.
model?: string;
provider?: string;
maxModelCalls?: number;
signal?: AbortSignal;
// Old waitForRunCompletion({ throwOnError: true }) semantics: `done`
// rejects with HeadlessRunError unless the turn completes.
throwOnError?: boolean;
}
export interface HeadlessAgentResult {
outcome: TurnOutcome;
state: TurnState;
// Last assistant text in the transcript (old extractAgentResponse).
summary: string | null;
}
export interface HeadlessAgentHandle {
turnId: string;
done: Promise<HeadlessAgentResult>;
}
export interface HeadlessAgentRunnerDependencies {
turnRuntime: ITurnRuntime;
defaultModelResolver: IDefaultModelResolver;
}
export interface IHeadlessAgentRunner {
start(options: HeadlessAgentOptions): Promise<HeadlessAgentHandle>;
run(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentResult & { turnId: string }>;
}
export function assistantText(
message: z.infer<typeof AssistantMessage>,
): string | null {
const content = message.content;
if (typeof content === "string") {
return content || null;
}
const text = content
.map((part) => (part.type === "text" ? part.text : ""))
.join("");
return text || null;
}
export function lastAssistantText(state: TurnState): string | null {
for (let i = state.modelCalls.length - 1; i >= 0; i--) {
const response = state.modelCalls[i].response;
if (response) {
const text = assistantText(response);
if (text) {
return text;
}
}
}
return null;
}
// Paths passed to the given file tools across the turn — replaces the old
// pattern of subscribing to the run bus for tool-invocation events. Only
// actually-invoked calls count (denied/unknown calls never ran).
export function toolInputPaths(
state: TurnState,
toolNames: string[],
): Set<string> {
const paths = new Set<string>();
for (const toolCall of state.toolCalls) {
if (!toolNames.includes(toolCall.toolName) || !toolCall.invocation) {
continue;
}
const input = toolCall.input as { path?: unknown } | null | undefined;
if (input && typeof input === "object" && typeof input.path === "string") {
paths.add(input.path);
}
}
return paths;
}
export class HeadlessAgentRunner implements IHeadlessAgentRunner {
private readonly turnRuntime: ITurnRuntime;
private readonly defaultModelResolver: IDefaultModelResolver;
constructor({
turnRuntime,
defaultModelResolver,
}: HeadlessAgentRunnerDependencies) {
this.turnRuntime = turnRuntime;
this.defaultModelResolver = defaultModelResolver;
}
async start(options: HeadlessAgentOptions): Promise<HeadlessAgentHandle> {
let modelOverride: { provider: string; model: string } | undefined;
if (options.model && options.provider) {
modelOverride = { provider: options.provider, model: options.model };
} else if (options.model || options.provider) {
const defaults = await this.defaultModelResolver.resolve();
modelOverride = {
provider: options.provider ?? defaults.provider,
model: options.model ?? defaults.model,
};
}
const turnId = await this.turnRuntime.createTurn({
agent: {
agentId: options.agentId,
...(modelOverride ? { overrides: { model: modelOverride } } : {}),
},
sessionId: null,
context: [],
input: { role: "user", content: options.message },
config: {
autoPermission: true,
humanAvailable: false,
...(options.maxModelCalls === undefined
? {}
: { maxModelCalls: options.maxModelCalls }),
},
});
const execution = this.turnRuntime.advanceTurn(turnId, undefined, {
signal: options.signal,
});
const done = execution.outcome.then(async (outcome) => {
const state = reduceTurn(
(await this.turnRuntime.getTurn(turnId)).events,
);
if (options.throwOnError && outcome.status !== "completed") {
throw new HeadlessRunError(
outcome.status === "failed"
? outcome.error
: `turn ${outcome.status}`,
turnId,
outcome,
);
}
return { outcome, state, summary: lastAssistantText(state) };
});
// The handle may be used fire-and-forget; rejections surface when awaited.
done.catch(() => undefined);
return { turnId, done };
}
async run(
options: HeadlessAgentOptions,
): Promise<HeadlessAgentResult & { turnId: string }> {
const handle = await this.start(options);
const result = await handle.done;
return { turnId: handle.turnId, ...result };
}
}

274
apps/x/packages/core/src/agents/runtime.ts
View file

@ -33,6 +33,7 @@ import { parse } from "yaml";
import { captureLlmUsage } from "../analytics/usage.js";
import { enterUseCase, withUseCase, type UseCase } from "../analytics/use_case.js";
import { getRaw as getNoteCreationRaw } from "../knowledge/note_creation.js";
import { getRaw as getNoteCurationRaw } from "../knowledge/note_curation.js";
import { getRaw as getLabelingAgentRaw } from "../knowledge/labeling_agent.js";
import { getRaw as getNoteTaggingAgentRaw } from "../knowledge/note_tagging_agent.js";
import { getRaw as getInlineTaskAgentRaw } from "../knowledge/inline_task_agent.js";
@ -114,7 +115,7 @@ function filePermissionTargets(toolName: string, args: Record<string, unknown>):
}
}
async function getToolPermissionMetadata(
export async function getToolPermissionMetadata(
toolCall: z.infer<typeof ToolCallPart>,
underlyingTool: z.infer<typeof ToolAttachment>,
sessionAllowedCommands: Set<string>,
@ -172,7 +173,7 @@ async function getToolPermissionMetadata(
};
}
function loadUserWorkDir(runId: string): string | null {
export function loadUserWorkDir(runId: string): string | null {
try {
const file = workDirConfigFile(runId);
if (!fs.existsSync(file)) return null;
@ -185,7 +186,7 @@ function loadUserWorkDir(runId: string): string | null {
}
}
function loadAgentNotesContext(): string | null {
export function loadAgentNotesContext(): string | null {
const sections: string[] = [];
const userFile = path.join(AGENT_NOTES_DIR, 'user.md');
@ -327,6 +328,129 @@ If Middle pane state is note, the supplied path and content are available so you
If Middle pane state is browser, only the URL and page title are supplied; the page content itself is NOT included. If you need the page content to answer, use the browser tools available to you to read the page. The user may or may not be talking about this page. Only reference or act on this page when the user's message clearly relates to it, such as "this page", "this article", "what I'm looking at", "this site", or "summarize this". For unrelated questions, ignore this page entirely and answer normally. Do not mention that you can see the browser unless it is relevant to the answer.`;
export interface ComposeSystemInstructionsInput {
instructions: string;
agentNotesContext: string | null;
userWorkDir: string | null;
voiceInput: boolean;
voiceOutput: 'summary' | 'full' | null;
searchEnabled: boolean;
codeMode: 'claude' | 'codex' | null;
codeCwd: string | null;
// Optional so legacy callers (old streamAgent path) are unaffected.
videoMode?: boolean;
coachMode?: boolean;
}
// System-prompt assembly, extracted verbatim from streamAgent so the new turn
// runtime's agent resolver composes byte-identical prompts. Pure: callers
// load agent notes / work dir themselves.
export function composeSystemInstructions({
instructions,
agentNotesContext,
userWorkDir,
voiceInput,
voiceOutput,
searchEnabled,
codeMode,
codeCwd,
videoMode,
coachMode,
}: ComposeSystemInstructionsInput): string {
let instructionsWithDateTime = `${instructions}\n\n${USER_CONTEXT_SYSTEM_INSTRUCTIONS}`;
if (agentNotesContext) {
instructionsWithDateTime += `\n\n${agentNotesContext}`;
}
if (userWorkDir) {
instructionsWithDateTime += `\n\n# User Work Directory
The user has chosen the following directory as their current **work directory**:
\`${userWorkDir}\`
Treat this as the **default location** for file operations whenever the user refers to files generically:
- "list the files", "show me what's in here", "what's the latest report" — list or look in the work directory.
- "save this", "export it", "write that to a file" — write the output into the work directory unless the user names another location.
- "open the file I was just working on", "the doc from earlier" — assume the work directory first.
Use absolute paths rooted at this directory with the \`file-*\` tools. For example, list with \`file-list({ path: "${userWorkDir}" })\`, read text with \`file-readText\`, and write text with \`file-writeText\`. For PDFs, Office docs, images, scanned docs, and other non-text files, use \`parseFile\` or \`LLMParse\` with the absolute path; you do NOT need to copy the file into the workspace first.
**Exceptions — these ALWAYS take precedence over the work directory default:**
1. **Knowledge base questions.** If the user asks about anything in the knowledge graph (notes, people, organizations, projects, topics) or paths starting with \`knowledge/\`, use file tools against \`knowledge/\` as documented above. Do NOT redirect those into the work directory.
2. **Explicit paths.** If the user names a different directory or gives an absolute/relative path (e.g. "in ~/Downloads", "from /tmp/foo", "the Desktop"), honor that path exactly and ignore the work-directory default for that request.
3. **Workspace-specific operations.** Anything that obviously belongs in the Rowboat workspace (config files, MCP servers, agent schedules, etc.) stays in the workspace, not the work directory.
Do not announce the work directory unless it's relevant. Just use it.`;
}
if (voiceInput) {
instructionsWithDateTime += `\n\n# Voice Input\nThe user's message was transcribed from speech. Be aware that:\n- There may be transcription errors. Silently correct obvious ones (e.g. homophones, misheard words). If an error is genuinely ambiguous, briefly mention your interpretation (e.g. "I'm assuming you meant X").\n- Spoken messages are often long-winded. The user may ramble, repeat themselves, or correct something they said earlier in the same message. Focus on their final intent, not every word verbatim.`;
}
if (videoMode) {
instructionsWithDateTime += `\n\n# Video Mode (Live Camera)
The user has turned on video mode: their webcam is on, and their messages arrive with a series of live webcam frames (ordered oldest to newest) captured while they were speaking or typing. The frames are a live view of the user themselves — not a document or file to analyze.
How to use the frames:
- Use them for what visual awareness genuinely adds: the user's expressions, body language, posture, gestures, eye contact with the camera, visible energy, anything they hold up to the camera, and their surroundings when relevant.
- Compare across frames to notice change over time within a message (e.g. increasingly slouched, started smiling, looked away for most of the message).
- When the user practices something performative (a pitch, presentation, interview, talk) or asks for delivery feedback, give specific, actionable coaching grounded in what you actually see — cite concrete observations ("in the last few frames you looked down and away from the camera") rather than generic advice. Cover posture, eye contact, facial expressiveness, gesturing, and visible energy.
- If they show something to the camera (an object, document, whiteboard), read or describe it and respond accordingly.
Driving the app:
- You can control the Rowboat app the user is looking at via the app-navigation tool (load the app-navigation skill first): open views, READ a view's contents as data (emails, background agents, chat history), and open specific items on their screen.
- When the user asks about anything that lives inside Rowboat ("what emails do I have?", "what agents are running?", "open the one from Arjun"), prefer driving over describing: read-view shows the view on their screen while returning its data, then answer out loud briefly; open-item when they pick one. Narrate as you act ("pulling up your inbox…"). Reading a view's data beats squinting at screen-share frames — it's exact.
Screen sharing:
- The user may also share their screen. Screen-share frames arrive in a separately labeled group after the webcam frames; they show the user's screen, not the user.
- When screen frames are present, treat them as the primary subject: the user is usually asking about, or working on, what's visible there. Read the screen carefully — code, documents, error messages, UI state — and help with it concretely.
- The LAST screen frame is the most current view of their screen; earlier ones show how it changed while they spoke.
- Frames are downscaled captures: small text may be hard to read. If something crucial is illegible, say what you need ("zoom into the error message" / "make that panel bigger") rather than guessing.
Etiquette:
- Do not narrate or list what you see in every response. Bring up visual observations only when relevant to the user's request or clearly worth mentioning.
- Never comment on the user's physical appearance, attractiveness, or personal attributes — visual feedback is strictly about delivery, expression, and body language.
- Frames are periodic snapshots, not continuous video; moments between frames are missing. Don't claim certainty about motion you couldn't have seen.`;
}
if (coachMode) {
instructionsWithDateTime += `\n\n# Practice Session (Coach Mode)
The user started a practice session: they are rehearsing something performative — a pitch, presentation, interview answer, or talk — and want live coaching. You are their coach for this session.
How to coach:
- Watch and listen for delivery: pacing, filler words, rambling, structure, and (from the webcam frames) posture, eye contact with the camera, facial expressiveness, gesturing, and visible energy.
- After each take or answer, give brief, specific, actionable feedback: 2-3 concrete observations, quoting what you actually saw or heard ("you looked down during the ask", "the opening 'um, so, basically' undercuts your hook"). Then let them go again.
- If they are clearly mid-flow, keep any interjection to one short sentence — or stay silent and save it for the break.
- Ask what they're practicing for at the start if it isn't obvious (investor pitch? interview? conference talk?) and tailor feedback to that audience.
- When they say they're done or wrap up, give a structured debrief: what's working, the top 3 things to improve, and one concrete drill or reframe to try next time.
- Be encouraging but honest — vague praise wastes their rehearsal time. Never comment on physical appearance; delivery, expression, and body language only.`;
}
if (voiceOutput === 'summary') {
instructionsWithDateTime += `\n\n# Voice Output (MANDATORY — READ THIS FIRST)\nThe user has voice output enabled. THIS IS YOUR #1 PRIORITY: you MUST start your response with <voice></voice> tags. If your response does not begin with <voice> tags, the user will hear nothing — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. Do not start with markdown, headings, or any other text. The literal first characters of your response must be "<voice>".\n2. Place ALL <voice> tags at the BEGINNING of your response, before any detailed content. Do NOT intersperse <voice> tags throughout the response.\n3. Wrap EACH spoken sentence in its own separate <voice> tag so it can be spoken incrementally. Do NOT wrap everything in a single <voice> block.\n4. Use voice as a TL;DR and navigation aid — do NOT read the entire response aloud.\n5. After all <voice> tags, you may include detailed written content (markdown, tables, code, etc.) that will be shown visually but not spoken.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things: the Q2 roadmap timeline, hiring for the backend role, and the client demo next week.</voice>\n<voice>I've pulled out the key details and action items below — the demo prep notes are at the end.</voice>\n\n## Meeting with Alex — March 11\n### Roadmap\n- Agreed to push Q2 launch to April 15...\n(detailed written content continues)\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You have five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you requested and Taylor flagged a contract issue.</voice>\n<voice>There's also a warm intro from a VC partner connecting you with someone at a prospective customer.</voice>\n<voice>I've drafted responses for three of them. The details and drafts are below.</voice>\n\n(email blocks, tables, and detailed content follow)\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a pretty packed day — seven meetings starting with standup at 9.</voice>\n<voice>The big ones are your investor call at 11, lunch with a partner from your lead VC at 12:30, and a customer call at 4.</voice>\n<voice>Your only free block for deep work is 2:30 to 4.</voice>\n\n(calendar block with full event details follows)\n\nExample 4 — User asks: "draft an email to Sam with our metrics"\n\n<voice>Done — I've drafted the email to Sam with your latest WAU and churn numbers.</voice>\n<voice>Take a look at the draft below and send it when you're ready.</voice>\n\n(email block with draft follows)\n\nREMEMBER: If you do not start with <voice> tags, the user hears silence. Always speak first, then write.`;
} else if (voiceOutput === 'full') {
instructionsWithDateTime += `\n\n# Voice Output — Full Read-Aloud (MANDATORY — READ THIS FIRST)\nThe user wants your ENTIRE response spoken aloud. THIS IS YOUR #1 PRIORITY: every single sentence must be wrapped in <voice></voice> tags. If you write anything outside <voice> tags, the user will not hear it — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. The literal first characters of your response must be "<voice>".\n2. Wrap EACH sentence in its own separate <voice> tag so it can be spoken incrementally.\n3. Write your response in a natural, conversational style suitable for listening — no markdown headings, bullet points, or formatting symbols. Use plain spoken language.\n4. Structure the content as if you are speaking to the user directly. Use transitions like "first", "also", "one more thing" instead of visual formatting.\n5. EVERY sentence MUST be inside a <voice> tag. Do not leave ANY content outside <voice> tags. If it's not in a <voice> tag, the user cannot hear it.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things.</voice>\n<voice>First, you discussed the Q2 roadmap timeline and agreed to push the launch to April.</voice>\n<voice>Second, you talked about hiring for the backend role — Alex will send over two candidates by Friday.</voice>\n<voice>And lastly, the client demo is next week on Thursday at 2pm, and you're handling the intro slides.</voice>\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You've got five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you asked for, and Taylor flagged a contract issue that needs your sign-off.</voice>\n<voice>There's a warm intro from a VC partner connecting you with an engineering lead at a potential customer.</voice>\n<voice>And someone from a prospective client wants to confirm your API tier before your call this afternoon.</voice>\n<voice>I've drafted replies for three of them — the metrics update, the intro, and the API question.</voice>\n<voice>The only one I left for you is Taylor's contract redline, since that needs your judgment on the liability cap.</voice>\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a packed day — seven meetings starting with standup at 9.</voice>\n<voice>The highlights are your investor call at 11, lunch with a VC partner at 12:30, and a customer call at 4.</voice>\n<voice>Your only open block for deep work is 2:30 to 4, so plan accordingly.</voice>\n<voice>Oh, and your 1-on-1 with your co-founder is at 5:30 — that's a walking meeting.</voice>\n\nExample 4 — User asks: "how are our metrics looking?"\n\n<voice>Metrics are looking strong this week.</voice>\n<voice>You hit 2,573 weekly active users, which is up 12% week over week.</voice>\n<voice>That means you've crossed the 2,500 milestone — worth calling out in your next investor update.</voice>\n<voice>Churn is down to 4.1%, improving month over month.</voice>\n<voice>The trailing 8-week compound growth rate is about 10%.</voice>\n\nREMEMBER: Start with <voice> immediately. No preamble, no markdown before it. Speak first.`;
}
if (searchEnabled) {
instructionsWithDateTime += `\n\n# Search\nThe user has requested a search. Use the web-search tool to answer their query.`;
}
if (codeMode) {
const agentDisplay = codeMode === 'claude' ? 'Claude Code' : 'Codex';
instructionsWithDateTime += `\n\n# Code Mode (Active) — Agent: ${agentDisplay}
The user has turned on **code mode** and the composer chip is set to **${agentDisplay}** (\`${codeMode}\`). For EVERY coding task this turn, use **${agentDisplay}**, and narrate that agent ("Using ${agentDisplay} to …").
The chip is the single source of truth for which agent runs:
- Do NOT carry over a different agent from earlier in this thread — even if a previous run used the other agent, use **${agentDisplay}** now.
- Do NOT switch agents based on an in-chat text request ("use codex", "switch to claude"). The agent only changes when the user toggles the chip; if they ask in chat, tell them to toggle the chip.
**How to run coding work — call the \`code_agent_run\` tool** with:
- \`agent\`: \`${codeMode}\` (always — match the chip).
- \`cwd\`: ${codeCwd ? `\`${codeCwd}\` (always — this coding session is pinned to that directory; never use another path)` : `the absolute project/working directory (resolve it per the code-with-agents skill — a path the user named, the "# User Work Directory" block, or ask once)`}.
- \`prompt\`: a clear, self-contained coding instruction.
The tool runs the agent on-device and streams its tool calls, file diffs, and plan into the chat; any action needing approval surfaces as an inline permission card, so you do NOT pre-confirm with an in-chat "reply yes". This chat keeps ONE persistent agent session, so follow-up coding requests automatically resume with full context — just call \`code_agent_run\` again. Do NOT shell out to \`acpx\` or \`executeCommand\` for coding, and do NOT fall back to your own file tools.
If the user's message is clearly NOT a coding request (small talk, an unrelated question), answer directly without invoking the coding agent. Code mode signals readiness, not that every message must route through the agent.`;
}
return instructionsWithDateTime;
}
export interface IAgentRuntime {
trigger(runId: string): Promise<void>;
}
@ -448,6 +572,20 @@ export class AgentRuntime implements IAgentRuntime {
finalState.ingest(event);
}
if (finalState.getPendingPermissions().length === 0) {
// This generic completion ping is only for real user
// chats (copilot_chat). Skip it for:
// - knowledge_sync: an internal, auto-running agent
// (knowledge-graph generation) that never notifies at
// all and has no user-facing chat to "Open".
// - background_task_agent: a user-configured agent that
// DOES notify, but exclusively through its own
// notify-user path; firing this ping too would
// duplicate that notification.
// (The finally block still runs on this early return.)
if (
finalState.runUseCase === "knowledge_sync" ||
finalState.runUseCase === "background_task_agent"
) return;
void notifyIfEnabled("chat_completion", {
title: "Response ready",
message: "Your agent finished responding.",
@ -659,8 +797,8 @@ export async function loadAgent(id: string): Promise<z.infer<typeof Agent>> {
return buildBackgroundTaskAgent();
}
if (id === 'note_creation') {
const raw = getNoteCreationRaw();
if (id === 'note_creation' || id === 'note_curation') {
const raw = id === 'note_curation' ? getNoteCurationRaw() : getNoteCreationRaw();
let agent: z.infer<typeof Agent> = {
name: id,
instructions: raw,
@ -847,15 +985,27 @@ export function convertFromMessages(messages: z.infer<typeof Message>[]): ModelM
providerOptions,
});
} else {
// New content parts array — collapse to text for LLM
// New content parts array — collapse text/attachments to text
// for the LLM; inline image parts (video-mode webcam and
// screen-share frames) are passed through as real multimodal
// image parts, grouped under labeled text headers so the
// model knows which images show the user vs their screen.
const textSegments: string[] = userMessageContextPrefix ? [userMessageContextPrefix] : [];
const attachmentLines: string[] = [];
type EncodedImagePart = { type: "image"; image: string; mediaType: string };
const cameraParts: EncodedImagePart[] = [];
const screenParts: EncodedImagePart[] = [];
const frameTimes: string[] = [];
for (const part of msg.content) {
if (part.type === "attachment") {
const sizeStr = part.size ? `, ${formatBytes(part.size)}` : '';
const lineStr = part.lineNumber ? ` (line ${part.lineNumber})` : '';
attachmentLines.push(`- ${part.filename} (${part.mimeType}${sizeStr}) at ${part.path}${lineStr}`);
} else if (part.type === "image") {
const target = part.source === "screen" ? screenParts : cameraParts;
target.push({ type: "image", image: part.data, mediaType: part.mediaType });
if (part.capturedAt) frameTimes.push(part.capturedAt);
} else {
textSegments.push(part.text);
}
@ -869,11 +1019,38 @@ export function convertFromMessages(messages: z.infer<typeof Message>[]): ModelM
}
}
result.push({
role: "user",
content: textSegments.join("\n"),
providerOptions,
});
const imageCount = cameraParts.length + screenParts.length;
if (imageCount > 0) {
const span = frameTimes.length >= 2
? ` spanning ${frameTimes[0]} to ${frameTimes[frameTimes.length - 1]}`
: frameTimes.length === 1
? ` captured at ${frameTimes[0]}`
: '';
const kinds: string[] = [];
if (cameraParts.length > 0) kinds.push(`${cameraParts.length} live webcam frame${cameraParts.length === 1 ? '' : 's'} of the user`);
if (screenParts.length > 0) kinds.push(`${screenParts.length} frame${screenParts.length === 1 ? '' : 's'} of the user's shared screen`);
textSegments.push(`[Video mode: ${kinds.join(' and ')} attached below, each group oldest to newest,${span ? span + ',' : ''} recorded while they composed this message.]`);
const content: Array<{ type: "text"; text: string } | EncodedImagePart> = [
{ type: "text", text: textSegments.join("\n") },
];
if (cameraParts.length > 0) {
content.push({ type: "text", text: "Webcam frames (oldest to newest):" }, ...cameraParts);
}
if (screenParts.length > 0) {
content.push({ type: "text", text: "Screen-share frames (oldest to newest):" }, ...screenParts);
}
result.push({
role: "user",
content,
providerOptions,
});
} else {
result.push({
role: "user",
content: textSegments.join("\n"),
providerOptions,
});
}
}
break;
}
@ -1409,70 +1586,17 @@ export async function* streamAgent({
loopLogger.log('running llm turn');
// stream agent response and build message
const messageBuilder = new StreamStepMessageBuilder();
let instructionsWithDateTime = `${agent.instructions}\n\n${USER_CONTEXT_SYSTEM_INSTRUCTIONS}`;
// Inject Agent Notes context for copilot
if (state.agentName === 'copilot' || state.agentName === 'rowboatx') {
const agentNotesContext = loadAgentNotesContext();
if (agentNotesContext) {
instructionsWithDateTime += `\n\n${agentNotesContext}`;
}
const userWorkDir = loadUserWorkDir(runId);
if (userWorkDir) {
loopLogger.log('injecting user work directory', userWorkDir);
instructionsWithDateTime += `\n\n# User Work Directory
The user has chosen the following directory as their current **work directory**:
\`${userWorkDir}\`
Treat this as the **default location** for file operations whenever the user refers to files generically:
- "list the files", "show me what's in here", "what's the latest report" — list or look in the work directory.
- "save this", "export it", "write that to a file" — write the output into the work directory unless the user names another location.
- "open the file I was just working on", "the doc from earlier" — assume the work directory first.
Use absolute paths rooted at this directory with the \`file-*\` tools. For example, list with \`file-list({ path: "${userWorkDir}" })\`, read text with \`file-readText\`, and write text with \`file-writeText\`. For PDFs, Office docs, images, scanned docs, and other non-text files, use \`parseFile\` or \`LLMParse\` with the absolute path; you do NOT need to copy the file into the workspace first.
**Exceptions — these ALWAYS take precedence over the work directory default:**
1. **Knowledge base questions.** If the user asks about anything in the knowledge graph (notes, people, organizations, projects, topics) or paths starting with \`knowledge/\`, use file tools against \`knowledge/\` as documented above. Do NOT redirect those into the work directory.
2. **Explicit paths.** If the user names a different directory or gives an absolute/relative path (e.g. "in ~/Downloads", "from /tmp/foo", "the Desktop"), honor that path exactly and ignore the work-directory default for that request.
3. **Workspace-specific operations.** Anything that obviously belongs in the Rowboat workspace (config files, MCP servers, agent schedules, etc.) stays in the workspace, not the work directory.
Do not announce the work directory unless it's relevant. Just use it.`;
}
}
if (voiceInput) {
loopLogger.log('voice input enabled, injecting voice input prompt');
instructionsWithDateTime += `\n\n# Voice Input\nThe user's message was transcribed from speech. Be aware that:\n- There may be transcription errors. Silently correct obvious ones (e.g. homophones, misheard words). If an error is genuinely ambiguous, briefly mention your interpretation (e.g. "I'm assuming you meant X").\n- Spoken messages are often long-winded. The user may ramble, repeat themselves, or correct something they said earlier in the same message. Focus on their final intent, not every word verbatim.`;
}
if (voiceOutput === 'summary') {
loopLogger.log('voice output enabled (summary mode), injecting voice output prompt');
instructionsWithDateTime += `\n\n# Voice Output (MANDATORY — READ THIS FIRST)\nThe user has voice output enabled. THIS IS YOUR #1 PRIORITY: you MUST start your response with <voice></voice> tags. If your response does not begin with <voice> tags, the user will hear nothing — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. Do not start with markdown, headings, or any other text. The literal first characters of your response must be "<voice>".\n2. Place ALL <voice> tags at the BEGINNING of your response, before any detailed content. Do NOT intersperse <voice> tags throughout the response.\n3. Wrap EACH spoken sentence in its own separate <voice> tag so it can be spoken incrementally. Do NOT wrap everything in a single <voice> block.\n4. Use voice as a TL;DR and navigation aid — do NOT read the entire response aloud.\n5. After all <voice> tags, you may include detailed written content (markdown, tables, code, etc.) that will be shown visually but not spoken.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things: the Q2 roadmap timeline, hiring for the backend role, and the client demo next week.</voice>\n<voice>I've pulled out the key details and action items below — the demo prep notes are at the end.</voice>\n\n## Meeting with Alex — March 11\n### Roadmap\n- Agreed to push Q2 launch to April 15...\n(detailed written content continues)\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You have five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you requested and Taylor flagged a contract issue.</voice>\n<voice>There's also a warm intro from a VC partner connecting you with someone at a prospective customer.</voice>\n<voice>I've drafted responses for three of them. The details and drafts are below.</voice>\n\n(email blocks, tables, and detailed content follow)\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a pretty packed day — seven meetings starting with standup at 9.</voice>\n<voice>The big ones are your investor call at 11, lunch with a partner from your lead VC at 12:30, and a customer call at 4.</voice>\n<voice>Your only free block for deep work is 2:30 to 4.</voice>\n\n(calendar block with full event details follows)\n\nExample 4 — User asks: "draft an email to Sam with our metrics"\n\n<voice>Done — I've drafted the email to Sam with your latest WAU and churn numbers.</voice>\n<voice>Take a look at the draft below and send it when you're ready.</voice>\n\n(email block with draft follows)\n\nREMEMBER: If you do not start with <voice> tags, the user hears silence. Always speak first, then write.`;
} else if (voiceOutput === 'full') {
loopLogger.log('voice output enabled (full mode), injecting voice output prompt');
instructionsWithDateTime += `\n\n# Voice Output — Full Read-Aloud (MANDATORY — READ THIS FIRST)\nThe user wants your ENTIRE response spoken aloud. THIS IS YOUR #1 PRIORITY: every single sentence must be wrapped in <voice></voice> tags. If you write anything outside <voice> tags, the user will not hear it — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. The literal first characters of your response must be "<voice>".\n2. Wrap EACH sentence in its own separate <voice> tag so it can be spoken incrementally.\n3. Write your response in a natural, conversational style suitable for listening — no markdown headings, bullet points, or formatting symbols. Use plain spoken language.\n4. Structure the content as if you are speaking to the user directly. Use transitions like "first", "also", "one more thing" instead of visual formatting.\n5. EVERY sentence MUST be inside a <voice> tag. Do not leave ANY content outside <voice> tags. If it's not in a <voice> tag, the user cannot hear it.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things.</voice>\n<voice>First, you discussed the Q2 roadmap timeline and agreed to push the launch to April.</voice>\n<voice>Second, you talked about hiring for the backend role — Alex will send over two candidates by Friday.</voice>\n<voice>And lastly, the client demo is next week on Thursday at 2pm, and you're handling the intro slides.</voice>\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You've got five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you asked for, and Taylor flagged a contract issue that needs your sign-off.</voice>\n<voice>There's a warm intro from a VC partner connecting you with an engineering lead at a potential customer.</voice>\n<voice>And someone from a prospective client wants to confirm your API tier before your call this afternoon.</voice>\n<voice>I've drafted replies for three of them — the metrics update, the intro, and the API question.</voice>\n<voice>The only one I left for you is Taylor's contract redline, since that needs your judgment on the liability cap.</voice>\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a packed day — seven meetings starting with standup at 9.</voice>\n<voice>The highlights are your investor call at 11, lunch with a VC partner at 12:30, and a customer call at 4.</voice>\n<voice>Your only open block for deep work is 2:30 to 4, so plan accordingly.</voice>\n<voice>Oh, and your 1-on-1 with your co-founder is at 5:30 — that's a walking meeting.</voice>\n\nExample 4 — User asks: "how are our metrics looking?"\n\n<voice>Metrics are looking strong this week.</voice>\n<voice>You hit 2,573 weekly active users, which is up 12% week over week.</voice>\n<voice>That means you've crossed the 2,500 milestone — worth calling out in your next investor update.</voice>\n<voice>Churn is down to 4.1%, improving month over month.</voice>\n<voice>The trailing 8-week compound growth rate is about 10%.</voice>\n\nREMEMBER: Start with <voice> immediately. No preamble, no markdown before it. Speak first.`;
}
if (searchEnabled) {
loopLogger.log('search enabled, injecting search prompt');
instructionsWithDateTime += `\n\n# Search\nThe user has requested a search. Use the web-search tool to answer their query.`;
}
if (codeMode) {
loopLogger.log('code mode enabled, injecting coding-agent context', codeMode);
const agentDisplay = codeMode === 'claude' ? 'Claude Code' : 'Codex';
instructionsWithDateTime += `\n\n# Code Mode (Active) — Agent: ${agentDisplay}
The user has turned on **code mode** and the composer chip is set to **${agentDisplay}** (\`${codeMode}\`). For EVERY coding task this turn, use **${agentDisplay}**, and narrate that agent ("Using ${agentDisplay} to …").
The chip is the single source of truth for which agent runs:
- Do NOT carry over a different agent from earlier in this thread — even if a previous run used the other agent, use **${agentDisplay}** now.
- Do NOT switch agents based on an in-chat text request ("use codex", "switch to claude"). The agent only changes when the user toggles the chip; if they ask in chat, tell them to toggle the chip.
**How to run coding work — call the \`code_agent_run\` tool** with:
- \`agent\`: \`${codeMode}\` (always — match the chip).
- \`cwd\`: ${codeCwd ? `\`${codeCwd}\` (always — this coding session is pinned to that directory; never use another path)` : `the absolute project/working directory (resolve it per the code-with-agents skill — a path the user named, the "# User Work Directory" block, or ask once)`}.
- \`prompt\`: a clear, self-contained coding instruction.
The tool runs the agent on-device and streams its tool calls, file diffs, and plan into the chat; any action needing approval surfaces as an inline permission card, so you do NOT pre-confirm with an in-chat "reply yes". This chat keeps ONE persistent agent session, so follow-up coding requests automatically resume with full context — just call \`code_agent_run\` again. Do NOT shell out to \`acpx\` or \`executeCommand\` for coding, and do NOT fall back to your own file tools.
If the user's message is clearly NOT a coding request (small talk, an unrelated question), answer directly without invoking the coding agent. Code mode signals readiness, not that every message must route through the agent.`;
}
const composeCopilotContext = state.agentName === 'copilot' || state.agentName === 'rowboatx';
const instructionsWithDateTime = composeSystemInstructions({
instructions: agent.instructions,
agentNotesContext: composeCopilotContext ? loadAgentNotesContext() : null,
userWorkDir: composeCopilotContext ? loadUserWorkDir(runId) : null,
voiceInput,
voiceOutput,
searchEnabled,
codeMode,
codeCwd,
});
let streamError: string | null = null;
for await (const event of streamLlm(
model,

2
apps/x/packages/core/src/analytics/use_case.ts
View file

@ -1,6 +1,6 @@
import { AsyncLocalStorage } from 'node:async_hooks';
export type UseCase = 'copilot_chat' | 'live_note_agent' | 'background_task_agent' | 'meeting_note' | 'knowledge_sync' | 'code_session';
export type UseCase = 'copilot_chat' | 'live_note_agent' | 'background_task_agent' | 'meeting_note' | 'meeting_prep' | 'knowledge_sync' | 'code_session';
export interface UseCaseContext {
useCase: UseCase;

77
apps/x/packages/core/src/application/assistant/instructions.ts
View file

@ -6,6 +6,7 @@ import { CURATED_TOOLKITS } from "@x/shared/dist/composio.js";
import container from "../../di/container.js";
import type { ICodeModeConfigRepo } from "../../code-mode/repo.js";
import type { ISlackConfigRepo } from "../../slack/repo.js";
import type { IOAuthRepo } from "../../auth/repo.js";
import { knowledgeSourcesRepo } from "../../knowledge/sources/repo.js";
const runtimeContextPrompt = getRuntimeContextPrompt(getRuntimeContext());
@ -14,7 +15,7 @@ const runtimeContextPrompt = getRuntimeContextPrompt(getRuntimeContext());
* Generate dynamic instructions section for Composio integrations.
* Lists connected toolkits and explains the meta-tool discovery flow.
*/
async function getComposioToolsPrompt(slackConnected: boolean = false): Promise<string> {
async function getComposioToolsPrompt(slackConnected: boolean = false, googleConnected: boolean = false): Promise<string> {
if (!(await isComposioConfigured())) {
return '';
}
@ -29,30 +30,50 @@ async function getComposioToolsPrompt(slackConnected: boolean = false): Promise<
? ` Exception: **Slack is connected natively** — use the \`slack\` skill for Slack, not Composio.`
: '';
// Google is connected natively, so email reading must not route to Composio.
const googleException = googleConnected
? ` Exception: **Gmail is connected natively** — read/check/search email with the \`app-navigation\` tool (\`read-view\`, \`view: "email"\`), not Composio.`
: '';
return `
## Composio Integrations
${connectedSection}
Load the \`composio-integration\` skill when the user asks to interact with any third-party service. NEVER say "I can't access [service]" without loading the skill and trying Composio first.${slackException}
Load the \`composio-integration\` skill when the user asks to interact with any third-party service. NEVER say "I can't access [service]" without loading the skill and trying Composio first.${slackException}${googleException}
`;
}
function buildStaticInstructions(composioEnabled: boolean, catalog: string, codeModeEnabled: boolean = true, slackConnected: boolean = false, slackChannelsHint: string = ''): string {
// Conditionally include Composio-related instruction sections
const emailDraftSuffix = composioEnabled
? ` Do NOT load this skill for reading, fetching, or checking emails — use the \`composio-integration\` skill for that instead.`
: ` Do NOT load this skill for reading, fetching, or checking emails.`;
function buildStaticInstructions(composioEnabled: boolean, catalog: string, codeModeEnabled: boolean = true, slackConnected: boolean = false, slackChannelsHint: string = '', googleConnected: boolean = false): string {
// Conditionally include Composio-related instruction sections.
// When Google is connected natively, email reading routes to the native
// app-navigation email view — never to Composio.
const emailDraftSuffix = googleConnected
? ` Do NOT load this skill for reading, fetching, or checking emails — Gmail is connected natively; use the \`app-navigation\` tool (\`read-view\`, \`view: "email"\`) for that instead.`
: composioEnabled
? ` Do NOT load this skill for reading, fetching, or checking emails — use the \`composio-integration\` skill for that instead.`
: ` Do NOT load this skill for reading, fetching, or checking emails.`;
// When Slack is connected natively (desktop/cURL auth, not Composio), keep it
// out of the Composio routing examples so the Copilot doesn't try to connect
// it through Composio and wrongly report it as unavailable.
const composioServiceExamples = slackConnected
? 'Gmail, GitHub, LinkedIn, Notion, Google Sheets, Jira, etc.'
: 'Gmail, GitHub, Slack, LinkedIn, Notion, Google Sheets, Jira, etc.';
// When Slack or Google is connected natively (not via Composio), keep them
// out of the Composio routing examples so the Copilot doesn't route their
// requests through Composio or wrongly report them as unavailable.
const composioServiceExamples = ['Gmail', 'GitHub', 'Slack', 'LinkedIn', 'Notion', 'Google Sheets', 'Jira']
.filter(service => !(slackConnected && service === 'Slack') && !(googleConnected && service === 'Gmail'))
.join(', ') + ', etc.';
const thirdPartyExamples = googleConnected
? 'listing issues, sending messages, fetching profiles'
: 'reading emails, listing issues, sending messages, fetching profiles';
const thirdPartyBlock = composioEnabled
? `\n**Third-Party Services:** When users ask to interact with any external service (${composioServiceExamples}) — reading emails, listing issues, sending messages, fetching profiles — load the \`composio-integration\` skill first. Do NOT look in local \`gmail_sync/\` or \`calendar_sync/\` folders for live data.\n`
? `\n**Third-Party Services:** When users ask to interact with any external service (${composioServiceExamples}) — ${thirdPartyExamples} — load the \`composio-integration\` skill first. Do NOT look in local \`gmail_sync/\` or \`calendar_sync/\` folders for live data.\n`
: '';
// Google is connected directly in Rowboat (native OAuth + background sync),
// independent of Composio. Route email reading to the native app-navigation
// email view so the Copilot never sends it through Composio.
const gmailBlock = googleConnected
? `\n**Gmail (connected natively):** The user's Google account is connected directly in Rowboat, and their email is synced continuously. For ANY request to read, fetch, check, or search emails — "get my last few emails", "any new emails?", "find the email from X", "search my gmail for Y" — load the \`app-navigation\` skill and use the \`app-navigation\` tool's \`read-view\` action with \`view: "email"\`. Its \`query\` parameter runs a LIVE Gmail search over the entire mailbox via the Gmail API with full Gmail search operators (\`from:\`, \`subject:\`, \`before:\`, etc.) — it IS Gmail's real search, so use it even when the user explicitly asks to "search Gmail directly". NEVER route email reading through the \`composio-integration\` skill or Composio Gmail tools, and NEVER tell the user Gmail isn't connected. Email *drafting* still goes through the \`draft-emails\` skill.\n`
: '';
// Slack is connected directly in Rowboat (agent-slack CLI), independent of
@ -69,9 +90,15 @@ function buildStaticInstructions(composioEnabled: boolean, catalog: string, code
? ` For Slack specifically, load the \`slack\` skill and use the agent-slack CLI — Slack is connected natively, not via Composio.`
: '';
const googleToolPriority = googleConnected
? ` For reading email specifically, use the \`app-navigation\` tool (\`read-view\`, \`view: "email"\`) — Gmail is connected natively, not via Composio.`
: '';
const toolPriorityServiceExamples = googleConnected ? 'GitHub, Notion, etc.' : 'GitHub, Gmail, etc.';
const toolPriority = composioEnabled
? `For third-party services (GitHub, Gmail, etc.), load the \`composio-integration\` skill.${slackToolPriority} For capabilities Composio doesn't cover (web search, file scraping, audio), use MCP tools via the \`mcp-integration\` skill.`
: `For capabilities like web search, file scraping, and audio, use MCP tools via the \`mcp-integration\` skill.${slackToolPriority}`;
? `For third-party services (${toolPriorityServiceExamples}), load the \`composio-integration\` skill.${slackToolPriority}${googleToolPriority} For capabilities Composio doesn't cover (web search, file scraping, audio), use MCP tools via the \`mcp-integration\` skill.`
: `For capabilities like web search, file scraping, and audio, use MCP tools via the \`mcp-integration\` skill.${slackToolPriority}${googleToolPriority}`;
const slackToolsLine = composioEnabled
? `- \`slack-checkConnection\`, \`slack-listAvailableTools\`, \`slack-executeAction\` - Slack integration (requires Slack to be connected via Composio). Use \`slack-listAvailableTools\` first to discover available tool slugs, then \`slack-executeAction\` to execute them.\n`
@ -104,7 +131,7 @@ Rowboat is an agentic assistant for everyday work - emails, meetings, projects,
**Email Drafting:** When users ask you to **draft** or **compose** emails (e.g., "draft a follow-up to Monica", "write an email to John about the project"), load the \`draft-emails\` skill first.${emailDraftSuffix}
${thirdPartyBlock}${slackBlock}**Meeting Prep:** When users ask you to prepare for a meeting, prep for a call, or brief them on attendees, load the \`meeting-prep\` skill first. It provides structured guidance for gathering context about attendees from the knowledge base and creating useful meeting briefs.
${thirdPartyBlock}${gmailBlock}${slackBlock}**Meeting Prep:** When users ask you to prepare for a meeting, prep for a call, or brief them on attendees, load the \`meeting-prep\` skill first. It provides structured guidance for gathering context about attendees from the knowledge base and creating useful meeting briefs.
**Create Presentations:** When users ask you to create a presentation, slide deck, pitch deck, or PDF slides, load the \`create-presentations\` skill first. It provides structured guidance for generating PDF presentations using context from the knowledge base.
@ -114,7 +141,7 @@ ${codeModeEnabled
? `**Code with Agents:** When users ask you to write code, build a project, create a script, fix a bug, or do any software development task — **including simple things like "create a .c file" or "write a hello-world in Python"** — your FIRST action MUST be \`loadSkill('code-with-agents')\`. Do NOT reach for \`executeCommand\` (PowerShell / bash / shell) or any workspace file tool to do code work yourself before loading this skill. The skill decides whether to delegate to Claude Code / Codex (via acpx) or hand control back to you, and it presents the user a one-click choice when needed. Paths outside the Rowboat workspace root (e.g. \`G:/...\`, \`~/projects/...\`) are NORMAL for coding tasks — do NOT raise "outside workspace" concerns or fall back to your own tools.`
: `**Code with Agents (disabled):** Code mode is currently OFF in the user's settings. Do NOT load \`code-with-agents\` and do NOT call acpx. Handle coding requests yourself with your normal tools if you can. After answering, add a final line letting the user know they can delegate coding to Claude Code or Codex by enabling Code Mode in Settings → Code Mode.`}
**App Control:** When users ask you to open notes, show the bases or graph view, filter or search notes, or manage saved views, load the \`app-navigation\` skill first. It provides structured guidance for navigating the app UI and controlling the knowledge base view.
**App Control (drive the app):** You can drive the Rowboat UI the user is looking at — open any view (email, meetings, background agents, chat history, knowledge, workspace, code, bases, graph), READ what a view contains (\`read-view\` returns emails / background agents / past chats as data while showing the view on screen), and open specific items (an email thread, a note, an agent, a past chat). When users ask to open, show, find, or ask about anything that lives inside Rowboat, load the \`app-navigation\` skill first — it documents the show-while-telling pattern. This matters most on calls: navigate so the user sees what you see, then answer briefly.
**Background Tasks (Self-Running Work):** Rowboat can run *background tasks* — persistent instructions the agent fires on a schedule and/or in response to incoming emails / calendar events. A bg-task either maintains a snapshot in its \`index.md\` (digest, dashboard, rolling summary) or performs a recurring side-effect (send a Slack message, draft an email, post to a webhook, call an API). This is the flagship surface for *anything recurring*.
@ -293,7 +320,7 @@ ${runtimeContextPrompt}
- \`addMcpServer\`, \`listMcpServers\`, \`listMcpTools\`, \`executeMcpTool\` - MCP server management and execution
- \`loadSkill\` - Skill loading
${slackToolsLine}- \`web-search\` - Search the web. Returns rich results with full text, highlights, and metadata. The \`category\` parameter defaults to \`general\` (full web search) — only use a specific category like \`news\`, \`company\`, \`research paper\` etc. when the query is clearly about that type. For everyday queries (weather, restaurants, prices, how-to), use \`general\`.
- \`app-navigation\` - Control the app UI: open notes, switch views, filter/search the knowledge base, manage saved views. **Load the \`app-navigation\` skill before using this tool.**
- \`app-navigation\` - Drive the app UI: open any view, read a view's contents (emails / background agents / chat history), open specific items (email thread, note, agent, past chat), filter/search the knowledge base, manage saved views. **Load the \`app-navigation\` skill before using this tool.**
- \`browser-control\` - Control the embedded browser pane: open sites, inspect the live page, switch tabs, and interact with indexed page elements. **Load the \`browser-control\` skill before using this tool.**
- \`save-to-memory\` - Save observations about the user to the agent memory system. Use this proactively during conversations.
${composioToolsLine}
@ -371,6 +398,14 @@ export async function buildCopilotInstructions(): Promise<string> {
} catch {
// repo unavailable — default to not connected
}
let googleConnected = false;
try {
const oauthRepo = container.resolve<IOAuthRepo>('oauthRepo');
const googleConnection = await oauthRepo.read('google');
googleConnected = !!googleConnection.tokens;
} catch {
// repo unavailable — default to not connected
}
if (slackConnected) {
try {
// Surface the channels the user selected for sync so the Copilot
@ -395,8 +430,8 @@ export async function buildCopilotInstructions(): Promise<string> {
const catalog = excludeIds.length > 0
? buildSkillCatalog({ excludeIds })
: skillCatalog;
const baseInstructions = buildStaticInstructions(composioEnabled, catalog, codeModeEnabled, slackConnected, slackChannelsHint);
const composioPrompt = await getComposioToolsPrompt(slackConnected);
const baseInstructions = buildStaticInstructions(composioEnabled, catalog, codeModeEnabled, slackConnected, slackChannelsHint, googleConnected);
const composioPrompt = await getComposioToolsPrompt(slackConnected, googleConnected);
cachedInstructions = composioPrompt
? baseInstructions + '\n' + composioPrompt
: baseInstructions;

Some files were not shown because too many files have changed in this diff Show more