apunkt/rowboat
1
0
Fork 0
mirror of https://github.com/rowboatlabs/rowboat.git synced 2026-07-15 21:11:08 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

popout and full screen work as expected

Browse source
This commit is contained in:
Arjun 2026-07-04 12:06:32 +05:30
parent d2501c4f4d
commit e9ee8fd975
15 changed files with 563 additions and 696 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CLAUDE.md
View file

@ -109,7 +109,7 @@ Long-form docs for specific features. Read the relevant file before making chang
| Feature | Doc | | Feature | Doc |
|---------|-----| |---------|-----|
| Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` | | Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` |
| Video mode — webcam + screen-share frames to the model (video+chat / hands-free call / full-screen Meet-style call), continuous voice loop, camera mute, popout window, prompts catalog | `apps/x/VIDEO_MODE.md` | | Calls (video mode) — one hands-free call engine with four presets (voice / video / share screen / practice coaching), device-derived surfaces (full-screen ⇄ floating popout), frame pipeline, prompts catalog | `apps/x/VIDEO_MODE.md` |
| Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` | | Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` |
| Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` | | Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` |

1
apps/x/ANALYTICS.md
View file

@ -92,6 +92,7 @@ All in `apps/renderer/src/lib/analytics.ts`:
- `chat_message_sent` — `{ voice_input, voice_output, search_enabled }` - `chat_message_sent` — `{ voice_input, voice_output, search_enabled }`
- `oauth_connected` / `oauth_disconnected` — `{ provider }` - `oauth_connected` / `oauth_disconnected` — `{ provider }`
- `voice_input_started` — no properties - `voice_input_started` — no properties
- `call_started` — `{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`)
- `search_executed` — `{ types: string[] }` - `search_executed` — `{ types: string[] }`
- `note_exported` — `{ format }` - `note_exported` — `{ format }`

140
apps/x/VIDEO_MODE.md
View file

@ -1,46 +1,70 @@
# Video Mode — Deep Dive # Calls (Video Mode) — Deep Dive
Video mode lets the assistant *see* the user (webcam) and their screen (screen Calls let the user talk to the assistant hands-free while it *sees* them
share), in three presentations: frames attached to normal chat, a hands-free (webcam) and their screen (screen share). There is ONE call engine —
spoken call, and a full-screen Meet-style call. This doc covers the product continuous listening, auto-submitted utterances, forced read-aloud TTS, frame
flow, the technical pipeline, and the LLM prompt surface with exact pointers. capture — entered through four presets that differ only in starting devices.
This doc covers the product flow, the technical pipeline, and the LLM prompt
surface with exact pointers.
## Product flow ## Product flow
The composer's video button (`chat-input-with-mentions.tsx`) toggles video The composer has a **call split-button** (`chat-input-with-mentions.tsx`).
mode; a chevron dropdown picks one of three modes (`VideoChatMode`): The main click is the "work together" default — preset `share`: screen
sharing ON, camera OFF, floating pill, so the user keeps working while the
assistant watches along (the button tooltip discloses the screen share). The
chevron menu holds the deviations. While a call is live the button turns red
and ends it.
| Mode | What it does | | Preset | Starting devices | First surface |
|------|--------------| |--------|------------------|---------------|
| `chat` — "Video + chat" | Camera on. Webcam (and screen-share) frames ride along with every typed or dictated message. Small PiP preview floats above the composer. | | `share` — main click | screen on, camera off | floating pill |
| `call` — "Video call (hands-free)" | Everything in `chat`, plus: continuous listening (each utterance auto-submits as a voice message) and forced full read-aloud TTS. No typing needed; composer still works. | | `voice` — "Voice call" | camera off, screen off | floating mascot pill |
| `meeting` — "Video call (full screen)" | Same pipeline as `call`, presented as a full-screen Meet-style layout: user tile + animated mascot tile, captions, control bar. | | `video` — "Video call" | camera on | full-screen call |
| `practice` — "Practice session" | camera on, + coaching persona | full-screen call |
On top of any mode: **One surface rule** (`callSurface` in `App.tsx`): full screen and screen
sharing are mutually exclusive in both directions — a full-screen call covers
the screen, so sharing it would show the call itself.
- **Screen share** (`MonitorUp` buttons on the PiP overlay and the meeting - sharing → floating popout, always (pill = working)
control bar): captures the primary screen; frames go to the model as a - not sharing → full screen unless `callMinimized` (full screen = facing
separately labeled group. In the meeting view the screen becomes the big each other)
tile with user + mascot in a side rail. - expanding the pill auto-STOPS any share; minimizing the full-screen call
- **Camera off** (Meet-style mute): video mode and screen share keep running, auto-STARTS one (the pill exists to work together) — presenting from full
no webcam frames are captured; tiles show a silhouette avatar. screen likewise collapses to the pill
- **Mascot dismissal** (meeting view): swaps the animated mascot for a - the camera toggle never changes the surface: turning it on from the pill
Meet-style letter avatar ("R"). puts your video IN the pill; expanding is its own explicit action
- **Popout**: starting a screen share immediately pops the mini-call out into
a small always-on-top frameless window (user + mascot tiles, live caption,
control bar) that floats over every app — including Rowboat itself, since
the shared screen may be Rowboat. It stays for the whole share (surviving
app switches) and disappears when sharing stops or video mode ends. While
sharing it is THE call surface: the in-app PiP hides, and presenting from
the full-screen meeting collapses the meeting into it (restored when the
share ends). Control-bar actions (camera toggle / stop share / end call)
round-trip `video:popoutAction` → main → `video:popout-action` → app
window, which owns the mic/camera/capture. The expand button focuses the
main window (`video:focusMain`).
`call`/`meeting` options are disabled unless both voice input (Deepgram) and **Screen-share consent** is three-layered: a toast the moment any share
voice output (TTS) are configured. Entering a call saves the user's TTS starts ("Your screen is being shared… [Stop sharing]"), a persistent
settings and forces `full` read-aloud; leaving restores them. "Sharing screen" badge on the pill, and macOS's purple recording indicator.
If the auto-share fails (Screen Recording permission not granted) the call
starts anyway as a voice call, with a toast linking to System Settings.
Practice/coaching is always an explicit choice — expanding to full screen
never turns the coach on.
In-call controls (identical bar on both surfaces): camera toggle (silhouette
avatar while off, no webcam frames captured), screen share toggle, mascot ⇄
"R" letter avatar, end call. Captions of the in-progress utterance and the
assistant's spoken line run along the bottom. Typing in the composer still
works mid-call; frames ride along with typed messages too.
Outside calls the composer keeps exactly two voice affordances: the **mic
button** (push-to-talk dictation, untouched) and the **read-aloud toggle**
(headphones; summary-style for typed chat — calls force full read-aloud and
restore the prior setting on hang-up). The old video dropdown, talking-head
toggle, and summary/full TTS dropdown are retired.
The call button is disabled unless both voice input (Deepgram) and voice
output (TTS) are configured. `call_started` (with `preset`) is captured in
PostHog — the adoption metric for this feature.
**Popout mechanics**: a small always-on-top frameless window (camera tile
when on + mascot tile, live caption, control bar) floating over every app —
including Rowboat. Control-bar actions round-trip `video:popoutAction` →
main → `video:popout-action` → app window, which owns the mic/camera/capture;
`expand` also refocuses the app window (handled in main).
## Frame pipeline ## Frame pipeline
@ -56,11 +80,13 @@ settings and forces `full` read-aloud; leaving restores them.
at the moment of send. Falls back to the single latest frame for at the moment of send. Falls back to the single latest frame for
rapid-fire messages. rapid-fire messages.
`App.tsx` `handlePromptSubmit` (~line 2767) attaches the drained frames to `App.tsx` `handlePromptSubmit` attaches the drained frames (whenever a call
the outgoing message as `UserImagePart`s and sets is live) to the outgoing message as `UserImagePart`s and sets
`composition.videoMode: true`. Frames also become `isVideoFrame` display `composition.videoMode` when the camera or screen is active, plus
attachments (filmstrip in the transcript — `chat-message-attachments.tsx`; `composition.coachMode` during a practice session. Frames also become
history hydration in `lib/run-to-conversation.ts`). `isVideoFrame` display attachments (filmstrip in the transcript —
`chat-message-attachments.tsx`; history hydration in
`lib/run-to-conversation.ts`).
## Message schema & model encoding ## Message schema & model encoding
@ -95,27 +121,27 @@ history hydration in `lib/run-to-conversation.ts`).
- Mid-call socket drops reconnect after 1s; the offline audio backlog is - Mid-call socket drops reconnect after 1s; the offline audio backlog is
capped (~30s). capped (~30s).
Mode transitions live in `App.tsx` `handleVideoModeChange` (~line 1161): Call lifecycle lives in `App.tsx` `startCall(preset)` / `endCall()`:
call ↔ meeting switches are presentation-only (mic/TTS untouched); entering a call saves/forces TTS settings, cancels any push-to-talk
entering/leaving hands-free saves/restores TTS settings. Push-to-talk is recording, and starts the continuous loop; ending restores everything.
disabled while a call owns the mic. Push-to-talk is disabled while a call owns the mic.
## Popout window ## Popout window
- Shown iff `videoChatMode !== 'off' && screenState === 'live'` (effect in - Shown iff the derived `callSurface === 'popout'` (effect in `App.tsx`).
`App.tsx`). Renderer asks `video:setPopout {show}` (main handler: Renderer asks `video:setPopout {show}`; main creates a frameless,
`apps/main/src/ipc.ts:1742`); main creates a frameless, `alwaysOnTop` `alwaysOnTop` ('floating'), all-workspaces BrowserWindow at the top-right
('floating'), all-workspaces BrowserWindow at the top-right of the primary of the primary display, loading the renderer bundle with `#video-popout`
display, loading the renderer bundle with `#video-popout`
(`apps/renderer/src/main.tsx` branches on the hash → (`apps/renderer/src/main.tsx` branches on the hash →
`components/video-popout.tsx`). `components/video-popout.tsx`).
- Call state streams over the `video:popout-state` push channel; main caches - Call state streams over the `video:popout-state` push channel; main caches
the last payload and replays it on popout load. Shown with the last payload and replays it on popout load. Shown with
`showInactive()` so it never steals focus (that would re-hide it). `showInactive()` so it never steals focus.
- The popout captures its **own** camera preview (MediaStreams can't cross - The popout captures its **own** camera preview (MediaStreams can't cross
windows) and synthesizes the mascot mouth level (no audio in that window). windows) and synthesizes the mascot mouth level (no audio in that window).
- `video:focusMain` matches only real app windows by URL — `getAllWindows()` - `video:popoutAction` relays control-bar actions to the app window, matched
also contains hidden utility windows (PDF export) that must not be shown. only by real app-window URLs — `getAllWindows()` also contains hidden
utility windows (PDF export) that must not be shown or messaged.
## Permissions ## Permissions
@ -132,8 +158,9 @@ disabled while a call owns the mic.
| Prompt | Where | | Prompt | Where |
|--------|-------| |--------|-------|
| `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) | | `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) |
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts:~1013` (`convertFromMessages`) | | `# Practice Session (Coach Mode)` system section — coaching persona: specific/actionable feedback after each take, one-sentence interjections mid-flow, structured debrief on wrap-up | `composeSystemInstructions`, gated on `coachMode` (directly after the video section) |
| `videoMode` composition override (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts:57,125`; set from `App.tsx` `sendConfig` | | Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts` (`convertFromMessages`) |
| `videoMode` / `coachMode` composition overrides (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts` (`CompositionOverrides`); set from `App.tsx` `sendConfig` |
Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are
reused untouched — calls set `voiceInput` per utterance and force reused untouched — calls set `voiceInput` per utterance and force
@ -156,4 +183,7 @@ screen frame per message unless the screen changed.
"energy" coaching leans on visual cues. "energy" coaching leans on visual cues.
- Screen share always captures the primary display (no window/display - Screen share always captures the primary display (no window/display
picker yet). picker yet).
- The meeting view covers the chat; there's no in-call transcript drawer. - The full-screen call covers the chat; there's no in-call transcript drawer.
- The "attach camera frames to typed chat without a call" combination (the
old video+chat mode) was cut in the call-model simplification; if analytics
show demand, it should return as an attachment chip, not a mode.

18
apps/x/apps/main/src/ipc.ts
View file

@ -390,6 +390,7 @@ let lastVideoPopoutState: {
ttsState: 'idle' | 'synthesizing' | 'speaking'; ttsState: 'idle' | 'synthesizing' | 'speaking';
status: 'listening' | 'thinking' | 'speaking' | null; status: 'listening' | 'thinking' | 'speaking' | null;
cameraOn: boolean; cameraOn: boolean;
screenSharing: boolean;
interimText: string | null; interimText: string | null;
} | null = null; } | null = null;
@ -1816,19 +1817,20 @@ export function setupIpcHandlers() {
} }
return {}; return {};
}, },
'video:focusMain': async () => { 'video:getPopoutState': async () => {
return { state: lastVideoPopoutState };
},
'video:popoutAction': async (_event, args) => {
// Relay a popout control-bar action to the app window, which owns the
// call (mic, camera, screen capture) and executes it there. 'expand'
// additionally brings the app window back to the foreground.
const main = findMainAppWindow(); const main = findMainAppWindow();
if (main) { if (args.action === 'expand' && main) {
if (main.isMinimized()) main.restore(); if (main.isMinimized()) main.restore();
main.show(); main.show();
main.focus(); main.focus();
} }
return {}; main?.webContents.send('video:popout-action', args);
},
'video:popoutAction': async (_event, args) => {
// Relay a popout control-bar action to the app window, which owns the
// call (mic, camera, screen capture) and executes it there.
findMainAppWindow()?.webContents.send('video:popout-action', args);
return {}; return {};
}, },
// Live-note handlers // Live-note handlers

367
apps/x/apps/renderer/src/App.tsx
View file

@ -12,7 +12,7 @@ import { ChatSidebar } from './components/chat-sidebar';
import { useSessionChat } from '@/hooks/useSessionChat'; import { useSessionChat } from '@/hooks/useSessionChat';
import { ChatHeader } from './components/chat-header'; import { ChatHeader } from './components/chat-header';
import { ChatEmptyState } from './components/chat-empty-state'; import { ChatEmptyState } from './components/chat-empty-state';
import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type VideoChatMode } from './components/chat-input-with-mentions'; import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment } from './components/chat-input-with-mentions';
import { ChatMessageAttachments } from '@/components/chat-message-attachments' import { ChatMessageAttachments } from '@/components/chat-message-attachments'
import { GraphView, type GraphEdge, type GraphNode } from '@/components/graph-view'; import { GraphView, type GraphEdge, type GraphNode } from '@/components/graph-view';
import { BasesView, type BaseConfig, DEFAULT_BASE_CONFIG } from '@/components/bases-view'; import { BasesView, type BaseConfig, DEFAULT_BASE_CONFIG } from '@/components/bases-view';
@ -120,8 +120,6 @@ import { toast } from "sonner"
import { useVoiceMode } from '@/hooks/useVoiceMode' import { useVoiceMode } from '@/hooks/useVoiceMode'
import { useVideoMode } from '@/hooks/useVideoMode' import { useVideoMode } from '@/hooks/useVideoMode'
import { useVoiceTTS } from '@/hooks/useVoiceTTS' import { useVoiceTTS } from '@/hooks/useVoiceTTS'
import { TalkingHeadOverlay } from '@/components/talking-head'
import { VideoPreviewOverlay } from '@/components/video-preview-overlay'
import { VideoCallView } from '@/components/video-call-view' import { VideoCallView } from '@/components/video-call-view'
import { ProductTour, type TourNavTarget } from '@/components/product-tour' import { ProductTour, type TourNavTarget } from '@/components/product-tour'
import { useMeetingTranscription, type CalendarEventMeta } from '@/hooks/useMeetingTranscription' import { useMeetingTranscription, type CalendarEventMeta } from '@/hooks/useMeetingTranscription'
@ -974,9 +972,9 @@ function App() {
const [ttsAvailable, setTtsAvailable] = useState(false) const [ttsAvailable, setTtsAvailable] = useState(false)
const [ttsEnabled, setTtsEnabled] = useState(false) const [ttsEnabled, setTtsEnabled] = useState(false)
const ttsEnabledRef = useRef(false) const ttsEnabledRef = useRef(false)
const [ttsMode, setTtsMode] = useState<'summary' | 'full'>('summary') // Read-aloud style: 'summary' for typed chat, forced to 'full' during a
// call and restored after. Context decides — the user never picks it.
const ttsModeRef = useRef<'summary' | 'full'>('summary') const ttsModeRef = useRef<'summary' | 'full'>('summary')
const [ttsAvatarEnabled, setTtsAvatarEnabled] = useState(false)
const [tourActive, setTourActive] = useState(false) const [tourActive, setTourActive] = useState(false)
const [isRecording, setIsRecording] = useState(false) const [isRecording, setIsRecording] = useState(false)
const voiceTextBufferRef = useRef('') const voiceTextBufferRef = useRef('')
@ -1021,16 +1019,20 @@ function App() {
const voiceRef = useRef(voice) const voiceRef = useRef(voice)
voiceRef.current = voice voiceRef.current = voice
// Video chat mode: while on, the webcam runs and frames are attached to // Calls: one engine (hands-free voice loop + forced read-aloud TTS + frame
// every outgoing message so the assistant can see the user. 'chat' keeps // capture), started via presets that only differ in device defaults. The
// the normal composer; 'call' is fully hands-free — continuous listening, // presentation is DERIVED from devices, never picked: screen sharing →
// auto-submitted utterances, spoken responses (handler defined below, after // floating popout; camera on → full-screen call; camera off → popout
// the voice/submit plumbing it drives). // (mascot pill). Handlers live below the voice/submit plumbing they drive.
const video = useVideoMode() const video = useVideoMode()
const [videoChatMode, setVideoChatMode] = useState<VideoChatMode>('off') const [inCall, setInCall] = useState(false)
const videoChatModeRef = useRef<VideoChatMode>('off') const inCallRef = useRef(false)
// TTS settings to restore when a hands-free call ends (a call forces // User explicitly shrank the full-screen call to the floating pill.
// full-read-aloud TTS for its duration). const [callMinimized, setCallMinimized] = useState(false)
// Practice preset: adds the coaching persona to the system prompt.
const [practiceMode, setPracticeMode] = useState(false)
const practiceModeRef = useRef(false)
// TTS settings to restore when a call ends (a call forces full read-aloud).
const preCallTtsRef = useRef<{ enabled: boolean; mode: 'summary' | 'full' } | null>(null) const preCallTtsRef = useRef<{ enabled: boolean; mode: 'summary' | 'full' } | null>(null)
const handleToggleMeetingRef = useRef<(() => void) | undefined>(undefined) const handleToggleMeetingRef = useRef<(() => void) | undefined>(undefined)
@ -1091,8 +1093,8 @@ function App() {
}, []) }, [])
const handleStartRecording = useCallback(() => { const handleStartRecording = useCallback(() => {
// Hands-free call mode owns the mic — ignore push-to-talk while it's on. // A live call owns the mic — ignore push-to-talk while one is running.
if (videoChatModeRef.current === 'call' || videoChatModeRef.current === 'meeting') return if (inCallRef.current) return
setIsRecording(true) setIsRecording(true)
isRecordingRef.current = true isRecordingRef.current = true
voice.start() voice.start()
@ -1123,143 +1125,128 @@ function App() {
ttsEnabledRef.current = next ttsEnabledRef.current = next
if (!next) { if (!next) {
ttsRef.current.cancel() ttsRef.current.cancel()
setTtsAvatarEnabled(false)
} }
return next return next
}) })
}, []) }, [])
// Talking-head mode implies voice output: enabling it turns TTS on,
// disabling it turns both off.
const handleToggleTtsAvatar = useCallback(() => {
setTtsAvatarEnabled(prev => {
const next = !prev
setTtsEnabled(next)
ttsEnabledRef.current = next
if (!next) {
ttsRef.current.cancel()
}
return next
})
}, [])
const handleTtsModeChange = useCallback((mode: 'summary' | 'full') => {
setTtsMode(mode)
ttsModeRef.current = mode
}, [])
const handleCancelRecording = useCallback(() => { const handleCancelRecording = useCallback(() => {
voice.cancel() voice.cancel()
setIsRecording(false) setIsRecording(false)
isRecordingRef.current = false isRecordingRef.current = false
}, [voice]) }, [voice])
// Video chat mode transitions. 'chat' just runs the camera; 'call' and // Start a call. Presets only differ in device defaults — the engine
// 'meeting' also listen continuously (auto-submitting each utterance as a // (continuous listening, auto-submitted utterances, forced read-aloud TTS,
// voice message) and force full read-aloud TTS so the assistant answers out // frame capture) is identical for all of them. The default entry ('share',
// loud. 'meeting' is the same pipeline presented as a full-screen call. // the call button's main click) is "work together": screen shared, camera
const handleVideoModeChange = useCallback(async (mode: VideoChatMode) => { // off, floating pill — the user keeps working while the assistant watches
const prev = videoChatModeRef.current // along. 'video'/'practice' open face-to-face full screen instead.
if (mode === prev) return const startCall = useCallback(async (preset: CallPreset) => {
const wasHandsFree = prev === 'call' || prev === 'meeting' if (inCallRef.current) return
const isHandsFree = mode === 'call' || mode === 'meeting' const camera = preset === 'video' || preset === 'practice'
const ok = await video.start({ camera })
if (!ok) return // camera denied/unavailable — stay out of the call
if (preset === 'share') {
// If screen capture fails (usually the macOS Screen Recording
// permission), continue as a voice call — sharing is one tap away on
// the pill once permission is granted.
const shared = await video.startScreenShare()
if (!shared) {
toast("Couldn't share your screen", {
description: 'Grant Rowboat Screen Recording access, then tap the share button on the call.',
action: {
label: 'Open Settings',
onClick: () => void window.ipc.invoke('meeting:openScreenRecordingSettings', null).catch(() => {}),
},
})
}
}
// Leaving hands-free: release the mic and restore the pre-call TTS settings. // A manual push-to-talk recording can't coexist with the call's mic.
if (wasHandsFree && !isHandsFree) { if (isRecordingRef.current) {
voiceRef.current.cancel() voiceRef.current.cancel()
const saved = preCallTtsRef.current setIsRecording(false)
preCallTtsRef.current = null isRecordingRef.current = false
const restoreEnabled = saved?.enabled ?? false
setTtsEnabled(restoreEnabled)
ttsEnabledRef.current = restoreEnabled
if (saved) {
setTtsMode(saved.mode)
ttsModeRef.current = saved.mode
}
if (!restoreEnabled) ttsRef.current.cancel()
} }
preCallTtsRef.current = { enabled: ttsEnabledRef.current, mode: ttsModeRef.current }
setTtsEnabled(true)
ttsEnabledRef.current = true
ttsModeRef.current = 'full'
void voiceRef.current.startContinuous((text) => {
pendingVoiceInputRef.current = true
handlePromptSubmitRef.current?.({ text, files: [] })
})
if (mode === 'off') { setPracticeMode(preset === 'practice')
video.stop() practiceModeRef.current = preset === 'practice'
videoChatModeRef.current = 'off' // Pill-first presets start minimized; face-to-face presets start expanded.
setVideoChatMode('off') setCallMinimized(preset === 'voice' || preset === 'share')
return inCallRef.current = true
setInCall(true)
analytics.callStarted(preset)
}, [video])
const endCall = useCallback(() => {
if (!inCallRef.current) return
voiceRef.current.cancel()
const saved = preCallTtsRef.current
preCallTtsRef.current = null
const restoreEnabled = saved?.enabled ?? false
setTtsEnabled(restoreEnabled)
ttsEnabledRef.current = restoreEnabled
if (saved) {
ttsModeRef.current = saved.mode
} }
if (!restoreEnabled) ttsRef.current.cancel()
if (prev === 'off') { video.stop()
const ok = await video.start() setPracticeMode(false)
if (!ok) return // camera denied/unavailable — stay off practiceModeRef.current = false
} setCallMinimized(false)
inCallRef.current = false
// Entering hands-free (call ↔ meeting switches keep the mic/TTS as-is). setInCall(false)
if (isHandsFree && !wasHandsFree) {
// A manual push-to-talk recording can't coexist with the call's mic.
if (isRecordingRef.current) {
voiceRef.current.cancel()
setIsRecording(false)
isRecordingRef.current = false
}
preCallTtsRef.current = { enabled: ttsEnabledRef.current, mode: ttsModeRef.current }
setTtsEnabled(true)
ttsEnabledRef.current = true
setTtsMode('full')
ttsModeRef.current = 'full'
void voiceRef.current.startContinuous((text) => {
pendingVoiceInputRef.current = true
handlePromptSubmitRef.current?.({ text, files: [] })
})
}
videoChatModeRef.current = mode
setVideoChatMode(mode)
}, [video]) }, [video])
// During a call, mute the mic while the assistant is thinking or speaking // During a call, mute the mic while the assistant is thinking or speaking
// so its own TTS (or a half-turn) never gets transcribed back at it. // so its own TTS (or a half-turn) never gets transcribed back at it.
useEffect(() => { useEffect(() => {
if (videoChatMode !== 'call' && videoChatMode !== 'meeting') return if (!inCall) return
voiceRef.current.setPaused(activeIsProcessing || tts.state !== 'idle') voiceRef.current.setPaused(activeIsProcessing || tts.state !== 'idle')
}, [videoChatMode, activeIsProcessing, tts.state]) }, [inCall, activeIsProcessing, tts.state])
// Presenting from the full-screen call collapses it into the floating // Screen sharing: frames of the shared screen ride along with each message
// popout (the user needs their screen — possibly Rowboat itself — free to // next to the webcam frames. The surface change (full screen → pill) falls
// navigate while sharing). When the share ends, return to full screen. // out of the derivation below.
const returnToMeetingAfterShareRef = useRef(false)
// Screen sharing (any video mode): frames of the shared screen ride along
// with each message next to the webcam frames.
const handleToggleScreenShare = useCallback(async () => { const handleToggleScreenShare = useCallback(async () => {
if (video.screenState === 'live') { if (video.screenState === 'live') {
video.stopScreenShare() video.stopScreenShare()
} else { } else {
const ok = await video.startScreenShare() await video.startScreenShare()
if (ok && videoChatModeRef.current === 'meeting') {
returnToMeetingAfterShareRef.current = true
void handleVideoModeChange('call')
}
} }
}, [video, handleVideoModeChange]) }, [video])
// Share ended (from the popout, the PiP, or the OS): restore the // Meet-style camera mute: the call (and any screen share) stays on, but no
// full-screen call if that's where the share was started from. // webcam frames are captured while the camera is off. Deliberately does NOT
useEffect(() => { // change the surface — turning your camera on from the pill puts your video
if (video.screenState === 'live') return // IN the pill; expanding to full screen is its own explicit action.
if (!returnToMeetingAfterShareRef.current) return
returnToMeetingAfterShareRef.current = false
if (videoChatModeRef.current === 'call') {
void handleVideoModeChange('meeting')
}
}, [video.screenState, handleVideoModeChange])
// Meet-style camera mute: video mode (and any screen share) stays on, but
// no webcam frames are captured while the camera is off.
const handleToggleCamera = useCallback(() => { const handleToggleCamera = useCallback(() => {
void video.setCameraEnabled(!video.cameraOn) void video.setCameraEnabled(!video.cameraOn)
}, [video]) }, [video])
// Current phase of a hands-free call (null outside call/meeting modes). // Minimizing the full-screen call drops you back to working — and the pill
// exists to work *together*, so sharing starts automatically (the symmetric
// twin of expand, which stops it). If capture fails (permission), the call
// still minimizes as a plain pill. `callMinimized` is also set so stopping
// the share from the pill keeps you in the pill rather than snapping back
// to full screen.
const handleMinimizeCall = useCallback(async () => {
setCallMinimized(true)
await video.startScreenShare()
}, [video])
// Current phase of the call (null when not in one).
const videoCallStatus: 'listening' | 'thinking' | 'speaking' | null = const videoCallStatus: 'listening' | 'thinking' | 'speaking' | null =
videoChatMode === 'call' || videoChatMode === 'meeting' inCall
? tts.state === 'speaking' ? tts.state === 'speaking'
? 'speaking' ? 'speaking'
: tts.state === 'synthesizing' || activeIsProcessing : tts.state === 'synthesizing' || activeIsProcessing
@ -1267,39 +1254,70 @@ function App() {
: 'listening' : 'listening'
: null : null
// Meet-style popout: the floating always-on-top mini-call appears the // The call's surface follows one rule: full screen and screen sharing are
// moment screen sharing starts and stays for the whole share — including // mutually exclusive (a full-screen call covers the screen — sharing it
// over Rowboat itself (the shared screen may BE Rowboat), and across app // would show the call itself). Sharing → floating pill, always. Not
// switches. It disappears when sharing stops or video mode ends. // sharing → full screen unless the user shrank it (`callMinimized`).
useEffect(() => { // Expanding the pill auto-stops any share; presenting from full screen
const shouldShow = videoChatMode !== 'off' && video.screenState === 'live' // auto-collapses to the pill.
void window.ipc.invoke('video:setPopout', { show: shouldShow }).catch(() => {}) const callSurface: 'fullscreen' | 'popout' | null = !inCall
}, [videoChatMode, video.screenState]) ? null
: video.screenState === 'live' || callMinimized
? 'popout'
: 'fullscreen'
// Keep the popout's mascot/status/camera/caption mirror of the call fresh. useEffect(() => {
void window.ipc.invoke('video:setPopout', { show: callSurface === 'popout' }).catch(() => {})
}, [callSurface])
// Consent surface for screen sharing: an unmissable toast the moment any
// share starts (auto-started calls included), with one-tap stop. The pill
// also carries a persistent "Sharing screen" badge, and macOS shows its
// purple recording indicator.
const prevScreenStateRef = useRef(video.screenState)
useEffect(() => {
const prev = prevScreenStateRef.current
prevScreenStateRef.current = video.screenState
if (video.screenState === 'live' && prev !== 'live') {
toast('Your screen is being shared', {
description: 'The assistant sees snapshots of it along with what you say.',
action: { label: 'Stop sharing', onClick: () => video.stopScreenShare() },
duration: 6000,
})
}
}, [video.screenState, video])
// Keep the popout's mascot/status/devices/caption mirror of the call fresh.
// The main process caches the latest state and replays it when the popout // The main process caches the latest state and replays it when the popout
// loads. // loads.
useEffect(() => { useEffect(() => {
if (videoChatMode === 'off') return if (!inCall) return
void window.ipc void window.ipc
.invoke('video:popoutState', { .invoke('video:popoutState', {
ttsState: tts.state, ttsState: tts.state,
status: videoCallStatus, status: videoCallStatus,
cameraOn: video.cameraOn, cameraOn: video.cameraOn,
interimText: videoCallStatus ? voice.interimText || null : null, screenSharing: video.screenState === 'live',
interimText: voice.interimText || null,
}) })
.catch(() => {}) .catch(() => {})
}, [videoChatMode, tts.state, videoCallStatus, video.cameraOn, voice.interimText]) }, [inCall, tts.state, videoCallStatus, video.cameraOn, video.screenState, voice.interimText])
// Execute popout control-bar actions (the popout window has no access to // Execute popout control-bar actions (the popout window has no access to
// the call's mic/camera/capture — they live here). // the call's mic/camera/capture — they live here). 'expand' goes full
// screen, which by the exclusivity rule stops any running share; the main
// process already refocused the app window.
useEffect(() => { useEffect(() => {
return window.ipc.on('video:popout-action', ({ action }) => { return window.ipc.on('video:popout-action', ({ action }) => {
if (action === 'toggle-camera') handleToggleCamera() if (action === 'toggle-camera') handleToggleCamera()
else if (action === 'stop-share') video.stopScreenShare() else if (action === 'toggle-share') void handleToggleScreenShare()
else if (action === 'end-call') void handleVideoModeChange('off') else if (action === 'end-call') endCall()
else if (action === 'expand') {
if (video.screenState === 'live') video.stopScreenShare()
setCallMinimized(false)
}
}) })
}, [handleToggleCamera, video, handleVideoModeChange]) }, [handleToggleCamera, handleToggleScreenShare, endCall, video])
// Enter to submit voice input, Escape to cancel // Enter to submit voice input, Escape to cancel
useEffect(() => { useEffect(() => {
@ -2783,7 +2801,7 @@ function App() {
// Video chat mode: drain the webcam frames buffered since the last send // Video chat mode: drain the webcam frames buffered since the last send
// so they ride along with this message as inline image parts. // so they ride along with this message as inline image parts.
const videoFrames = videoChatModeRef.current !== 'off' ? video.collectFrames() : [] const videoFrames = inCallRef.current ? video.collectFrames() : []
const userMessageId = `user-${Date.now()}` const userMessageId = `user-${Date.now()}`
const displayAttachments: ChatMessage['attachments'] = hasAttachments || videoFrames.length > 0 const displayAttachments: ChatMessage['attachments'] = hasAttachments || videoFrames.length > 0
@ -2854,7 +2872,8 @@ function App() {
...(ttsEnabledRef.current ? { voiceOutput: ttsModeRef.current } : {}), ...(ttsEnabledRef.current ? { voiceOutput: ttsModeRef.current } : {}),
...(searchEnabled ? { searchEnabled: true } : {}), ...(searchEnabled ? { searchEnabled: true } : {}),
...(codeMode ? { codeMode } : {}), ...(codeMode ? { codeMode } : {}),
...(videoChatModeRef.current !== 'off' ? { videoMode: true } : {}), ...(inCallRef.current && (video.cameraOn || video.screenState === 'live') ? { videoMode: true } : {}),
...(practiceModeRef.current ? { coachMode: true } : {}),
}, },
}, },
}, },
@ -6536,14 +6555,11 @@ function App() {
voiceAvailable={isActive && voiceAvailable} voiceAvailable={isActive && voiceAvailable}
ttsAvailable={isActive && ttsAvailable} ttsAvailable={isActive && ttsAvailable}
ttsEnabled={ttsEnabled} ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={isActive ? handleToggleTts : undefined} onToggleTts={isActive ? handleToggleTts : undefined}
onTtsModeChange={isActive ? handleTtsModeChange : undefined} inCall={inCall}
ttsAvatarEnabled={ttsAvatarEnabled} onStartCall={isActive ? startCall : undefined}
onToggleTtsAvatar={isActive ? handleToggleTtsAvatar : undefined} onEndCall={isActive ? endCall : undefined}
videoChatMode={videoChatMode} callAvailable={voiceAvailable && ttsAvailable}
onVideoModeChange={isActive ? handleVideoModeChange : undefined}
videoCallAvailable={voiceAvailable && ttsAvailable}
/> />
</div> </div>
) )
@ -6652,69 +6668,32 @@ function App() {
voiceAvailable={voiceAvailable} voiceAvailable={voiceAvailable}
ttsAvailable={ttsAvailable} ttsAvailable={ttsAvailable}
ttsEnabled={ttsEnabled} ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={handleToggleTts} onToggleTts={handleToggleTts}
onTtsModeChange={handleTtsModeChange} inCall={inCall}
ttsAvatarEnabled={ttsAvatarEnabled} onStartCall={startCall}
onToggleTtsAvatar={handleToggleTtsAvatar} onEndCall={endCall}
videoChatMode={videoChatMode} callAvailable={voiceAvailable && ttsAvailable}
onVideoModeChange={handleVideoModeChange}
videoCallAvailable={voiceAvailable && ttsAvailable}
onComposioConnected={handleComposioConnected} onComposioConnected={handleComposioConnected}
/> />
)} )}
{/* Webcam PiP preview while video chat mode is on. Hidden while {/* Full-screen call: user tile + animated mascot tile. Shown only
screen sharing (the floating popout is the call surface then) when the derived surface says so (camera on, no screen share,
and in the full-screen meeting view (which replaces it). */} not minimized) — otherwise the call lives in the floating
{(videoChatMode === 'chat' || videoChatMode === 'call') && video.screenState !== 'live' && ( popout window. */}
<VideoPreviewOverlay {callSurface === 'fullscreen' && (
streamRef={video.streamRef}
onTurnOff={() => handleVideoModeChange('off')}
callStatus={
videoChatMode === 'call'
? tts.state !== 'idle'
? 'speaking'
: activeIsProcessing
? 'thinking'
: 'listening'
: undefined
}
interimText={videoChatMode === 'call' ? voice.interimText : undefined}
onToggleScreenShare={handleToggleScreenShare}
cameraOn={video.cameraOn}
onToggleCamera={handleToggleCamera}
/>
)}
{/* Full-screen Meet-style call: user tile + animated mascot tile */}
{videoChatMode === 'meeting' && (
<VideoCallView <VideoCallView
streamRef={video.streamRef} streamRef={video.streamRef}
screenStreamRef={video.screenStreamRef}
isScreenSharing={video.screenState === 'live'}
onToggleScreenShare={handleToggleScreenShare} onToggleScreenShare={handleToggleScreenShare}
cameraOn={video.cameraOn} cameraOn={video.cameraOn}
onToggleCamera={handleToggleCamera} onToggleCamera={handleToggleCamera}
practiceMode={practiceMode}
onMinimize={() => void handleMinimizeCall()}
ttsState={tts.state} ttsState={tts.state}
getTtsLevel={tts.getLevel} getTtsLevel={tts.getLevel}
status={ status={videoCallStatus ?? 'listening'}
tts.state === 'speaking'
? 'speaking'
: tts.state === 'synthesizing' || activeIsProcessing
? 'thinking'
: 'listening'
}
interimText={voice.interimText} interimText={voice.interimText}
assistantCaption={assistantCaption} assistantCaption={assistantCaption}
onLeave={() => handleVideoModeChange('off')} onLeave={endCall}
/>
)}
{/* Talking head hovers over the active view while avatar voice mode is
on (hidden during the tour, which shows its own mascot) */}
{ttsAvatarEnabled && !tourActive && (
<TalkingHeadOverlay
ttsState={tts.state}
getLevel={tts.getLevel}
onDismiss={handleToggleTtsAvatar}
/> />
)} )}
{/* Mascot-guided product tour */} {/* Mascot-guided product tour */}

207
apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx
View file

@ -21,7 +21,10 @@ import {
Lock, Lock,
Mic, Mic,
MoreHorizontal, MoreHorizontal,
Phone,
PhoneOff,
Plus, Plus,
Presentation,
ShieldCheck, ShieldCheck,
Square, Square,
Terminal, Terminal,
@ -51,7 +54,6 @@ import {
} from '@/lib/attachment-presentation' } from '@/lib/attachment-presentation'
import { getExtension, getFileDisplayName, getMimeFromExtension, isImageMime } from '@/lib/file-utils' import { getExtension, getFileDisplayName, getMimeFromExtension, isImageMime } from '@/lib/file-utils'
import { cn } from '@/lib/utils' import { cn } from '@/lib/utils'
import { MascotFaceIcon } from '@/components/talking-head'
import { import {
type FileMention, type FileMention,
type PromptInputMessage, type PromptInputMessage,
@ -212,7 +214,17 @@ function compactWorkDirPath(path: string) {
return path.replace(/^\/Users\/[^/]+/, '~') return path.replace(/^\/Users\/[^/]+/, '~')
} }
export type VideoChatMode = 'off' | 'chat' | 'call' | 'meeting' // Call presets: front doors into the same call engine, differing only in
// starting devices. 'share' is the call button's main click — the "work
// together" default (screen shared, camera off, floating pill). The chevron
// menu holds the deviations.
export type CallPreset = 'voice' | 'video' | 'share' | 'practice'
const CALL_PRESET_MENU: Array<{ preset: CallPreset; label: string; description: string; Icon: typeof Phone }> = [
{ preset: 'voice', label: 'Voice call', description: 'Just talk — nothing is shared, the mascot hovers while you work', Icon: AudioLines },
{ preset: 'video', label: 'Video call', description: 'Camera on, face to face — it sees your expressions', Icon: Video },
{ preset: 'practice', label: 'Practice session', description: 'Rehearse a pitch or interview with live coaching', Icon: Presentation },
]
interface ChatInputInnerProps { interface ChatInputInnerProps {
onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void
@ -236,18 +248,14 @@ interface ChatInputInnerProps {
voiceAvailable?: boolean voiceAvailable?: boolean
ttsAvailable?: boolean ttsAvailable?: boolean
ttsEnabled?: boolean ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void /** A call is live (hands-free voice loop + spoken responses). */
ttsAvatarEnabled?: boolean inCall?: boolean
onToggleTtsAvatar?: () => void /** Start a call with the given preset's device defaults. */
/** Video chat mode: 'chat' attaches webcam frames to messages; 'call' and onStartCall?: (preset: CallPreset) => void
* 'meeting' are fully hands-free — continuous listening, spoken responses onEndCall?: () => void
* ('meeting' additionally takes over the whole screen, Meet-style). */ /** Calls need both voice input (STT) and voice output (TTS) configured. */
videoChatMode?: VideoChatMode callAvailable?: boolean
onVideoModeChange?: (mode: VideoChatMode) => void
/** Hands-free call needs both voice input (STT) and voice output (TTS). */
videoCallAvailable?: boolean
/** Fired when the user picks a different model in the dropdown (only when no run exists yet). */ /** Fired when the user picks a different model in the dropdown (only when no run exists yet). */
onSelectedModelChange?: (model: SelectedModel | null) => void onSelectedModelChange?: (model: SelectedModel | null) => void
/** Work directory for this chat (per-chat). Null when none is set. */ /** Work directory for this chat (per-chat). Null when none is set. */
@ -283,14 +291,11 @@ function ChatInputInner({
voiceAvailable, voiceAvailable,
ttsAvailable, ttsAvailable,
ttsEnabled, ttsEnabled,
ttsMode,
onToggleTts, onToggleTts,
onTtsModeChange, inCall,
ttsAvatarEnabled, onStartCall,
onToggleTtsAvatar, onEndCall,
videoChatMode = 'off', callAvailable,
onVideoModeChange,
videoCallAvailable,
onSelectedModelChange, onSelectedModelChange,
workDir = null, workDir = null,
onWorkDirChange, onWorkDirChange,
@ -1290,68 +1295,21 @@ function ChatInputInner({
</DropdownMenu> </DropdownMenu>
) : null} ) : null}
{onToggleTts && ttsAvailable && ( {onToggleTts && ttsAvailable && (
<div className="flex shrink-0 items-center">
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
<TooltipTrigger asChild>
<button
type="button"
onClick={onToggleTts}
className={cn(
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
ttsEnabled
? 'text-foreground hover:bg-muted'
: 'text-muted-foreground hover:bg-muted hover:text-foreground'
)}
aria-label={ttsEnabled ? 'Disable voice output' : 'Enable voice output'}
>
<Headphones className="h-4 w-4" />
{!ttsEnabled && (
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
</span>
)}
</button>
</TooltipTrigger>
<TooltipContent side="top">
{ttsEnabled ? 'Voice output on' : 'Voice output off'}
</TooltipContent>
</Tooltip>
{ttsEnabled && onTtsModeChange && (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<button
type="button"
className="flex h-7 w-4 shrink-0 items-center justify-center text-muted-foreground transition-colors hover:text-foreground"
>
<ChevronDown className="h-3 w-3" />
</button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuRadioGroup value={ttsMode ?? 'summary'} onValueChange={(v) => onTtsModeChange(v as 'summary' | 'full')}>
<DropdownMenuRadioItem value="summary">Speak summary</DropdownMenuRadioItem>
<DropdownMenuRadioItem value="full">Speak full response</DropdownMenuRadioItem>
</DropdownMenuRadioGroup>
</DropdownMenuContent>
</DropdownMenu>
)}
</div>
)}
{onToggleTtsAvatar && ttsAvailable && (
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}> <Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
<TooltipTrigger asChild> <TooltipTrigger asChild>
<button <button
type="button" type="button"
onClick={onToggleTtsAvatar} onClick={onToggleTts}
className={cn( className={cn(
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors', 'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
ttsAvatarEnabled ttsEnabled
? 'text-foreground hover:bg-muted' ? 'text-foreground hover:bg-muted'
: 'text-muted-foreground hover:bg-muted hover:text-foreground' : 'text-muted-foreground hover:bg-muted hover:text-foreground'
)} )}
aria-label={ttsAvatarEnabled ? 'Disable talking head' : 'Enable talking head'} aria-label={ttsEnabled ? 'Disable read-aloud' : 'Enable read-aloud'}
> >
<MascotFaceIcon /> <Headphones className="h-4 w-4" />
{!ttsAvatarEnabled && ( {!ttsEnabled && (
<span className="absolute inset-0 flex items-center justify-center pointer-events-none"> <span className="absolute inset-0 flex items-center justify-center pointer-events-none">
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" /> <span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
</span> </span>
@ -1359,64 +1317,70 @@ function ChatInputInner({
</button> </button>
</TooltipTrigger> </TooltipTrigger>
<TooltipContent side="top"> <TooltipContent side="top">
{ttsAvatarEnabled ? 'Talking head on' : 'Talking head off'} {ttsEnabled ? 'Read responses aloud: on' : 'Read responses aloud: off'}
</TooltipContent> </TooltipContent>
</Tooltip> </Tooltip>
)} )}
{onVideoModeChange && ( {onStartCall && (
<div className="flex shrink-0 items-center"> <div className="flex shrink-0 items-center">
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}> <Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
<TooltipTrigger asChild> <TooltipTrigger asChild>
<button <button
type="button" type="button"
onClick={() => onVideoModeChange(videoChatMode === 'off' ? 'chat' : 'off')} onClick={() => {
if (inCall) {
onEndCall?.()
} else if (callAvailable) {
onStartCall('share')
}
}}
className={cn( className={cn(
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors', 'flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
videoChatMode !== 'off' inCall
? 'text-red-500 hover:bg-muted' ? 'bg-red-600 text-white hover:bg-red-500'
: 'text-muted-foreground hover:bg-muted hover:text-foreground' : callAvailable
? 'text-muted-foreground hover:bg-muted hover:text-foreground'
: 'cursor-default text-muted-foreground/40'
)} )}
aria-label={videoChatMode === 'off' ? 'Turn on video chat' : 'Turn off video chat'} aria-label={inCall ? 'End call' : 'Start a call'}
> >
<Video className="h-4 w-4" /> {inCall ? <PhoneOff className="h-4 w-4" /> : <Phone className="h-4 w-4" />}
{videoChatMode === 'off' && (
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
</span>
)}
</button> </button>
</TooltipTrigger> </TooltipTrigger>
<TooltipContent side="top"> <TooltipContent side="top">
{videoChatMode === 'off' {inCall
? 'Video chat off' ? 'End call'
: videoChatMode === 'chat' : callAvailable
? 'Video chat on — camera frames are shared with the assistant' ? 'Start a call — it sees your screen while you talk it through'
: 'Video call on — speak freely, the assistant answers out loud'} : 'Calls need voice input and output configured'}
</TooltipContent> </TooltipContent>
</Tooltip> </Tooltip>
{videoChatMode !== 'off' && ( {!inCall && (
<DropdownMenu> <DropdownMenu>
<DropdownMenuTrigger asChild> <DropdownMenuTrigger asChild>
<button <button
type="button" type="button"
className="flex h-7 w-4 shrink-0 items-center justify-center text-muted-foreground transition-colors hover:text-foreground" className="flex h-7 w-4 shrink-0 items-center justify-center text-muted-foreground transition-colors hover:text-foreground"
aria-label="Call options"
> >
<ChevronDown className="h-3 w-3" /> <ChevronDown className="h-3 w-3" />
</button> </button>
</DropdownMenuTrigger> </DropdownMenuTrigger>
<DropdownMenuContent align="end"> <DropdownMenuContent align="end" className="w-72">
<DropdownMenuRadioGroup {CALL_PRESET_MENU.map(({ preset, label, description, Icon }) => (
value={videoChatMode} <DropdownMenuItem
onValueChange={(v) => onVideoModeChange(v as VideoChatMode)} key={preset}
> disabled={!callAvailable}
<DropdownMenuRadioItem value="chat">Video + chat</DropdownMenuRadioItem> onSelect={() => onStartCall(preset)}
<DropdownMenuRadioItem value="call" disabled={!videoCallAvailable}> className="items-start gap-3 py-2"
Video call (hands-free) >
</DropdownMenuRadioItem> <Icon className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" />
<DropdownMenuRadioItem value="meeting" disabled={!videoCallAvailable}> <span className="min-w-0">
Video call (full screen) <span className="block text-sm font-medium leading-tight">{label}</span>
</DropdownMenuRadioItem> <span className="block pt-0.5 text-xs leading-tight text-muted-foreground">{description}</span>
</DropdownMenuRadioGroup> </span>
</DropdownMenuItem>
))}
</DropdownMenuContent> </DropdownMenuContent>
</DropdownMenu> </DropdownMenu>
)} )}
@ -1593,14 +1557,11 @@ export interface ChatInputWithMentionsProps {
voiceAvailable?: boolean voiceAvailable?: boolean
ttsAvailable?: boolean ttsAvailable?: boolean
ttsEnabled?: boolean ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void inCall?: boolean
ttsAvatarEnabled?: boolean onStartCall?: (preset: CallPreset) => void
onToggleTtsAvatar?: () => void onEndCall?: () => void
videoChatMode?: VideoChatMode callAvailable?: boolean
onVideoModeChange?: (mode: VideoChatMode) => void
videoCallAvailable?: boolean
onSelectedModelChange?: (model: SelectedModel | null) => void onSelectedModelChange?: (model: SelectedModel | null) => void
workDir?: string | null workDir?: string | null
onWorkDirChange?: (value: string | null) => void onWorkDirChange?: (value: string | null) => void
@ -1632,14 +1593,11 @@ export function ChatInputWithMentions({
voiceAvailable, voiceAvailable,
ttsAvailable, ttsAvailable,
ttsEnabled, ttsEnabled,
ttsMode,
onToggleTts, onToggleTts,
onTtsModeChange, inCall,
ttsAvatarEnabled, onStartCall,
onToggleTtsAvatar, onEndCall,
videoChatMode, callAvailable,
onVideoModeChange,
videoCallAvailable,
onSelectedModelChange, onSelectedModelChange,
workDir, workDir,
onWorkDirChange, onWorkDirChange,
@ -1668,14 +1626,11 @@ export function ChatInputWithMentions({
voiceAvailable={voiceAvailable} voiceAvailable={voiceAvailable}
ttsAvailable={ttsAvailable} ttsAvailable={ttsAvailable}
ttsEnabled={ttsEnabled} ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={onToggleTts} onToggleTts={onToggleTts}
onTtsModeChange={onTtsModeChange} inCall={inCall}
ttsAvatarEnabled={ttsAvatarEnabled} onStartCall={onStartCall}
onToggleTtsAvatar={onToggleTtsAvatar} onEndCall={onEndCall}
videoChatMode={videoChatMode} callAvailable={callAvailable}
onVideoModeChange={onVideoModeChange}
videoCallAvailable={videoCallAvailable}
onSelectedModelChange={onSelectedModelChange} onSelectedModelChange={onSelectedModelChange}
workDir={workDir} workDir={workDir}
onWorkDirChange={onWorkDirChange} onWorkDirChange={onWorkDirChange}

35
apps/x/apps/renderer/src/components/chat-sidebar.tsx
View file

@ -37,7 +37,7 @@ import { MarkdownPreOverride } from '@/components/ai-elements/markdown-code-over
import { defaultRemarkPlugins } from 'streamdown' import { defaultRemarkPlugins } from 'streamdown'
import remarkBreaks from 'remark-breaks' import remarkBreaks from 'remark-breaks'
import { type ChatTab } from '@/components/tab-bar' import { type ChatTab } from '@/components/tab-bar'
import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type SelectedModel, type VideoChatMode } from '@/components/chat-input-with-mentions' import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions'
import { ChatMessageAttachments } from '@/components/chat-message-attachments' import { ChatMessageAttachments } from '@/components/chat-message-attachments'
import { useSidebar } from '@/components/ui/sidebar' import { useSidebar } from '@/components/ui/sidebar'
import { wikiLabel } from '@/lib/wiki-links' import { wikiLabel } from '@/lib/wiki-links'
@ -189,14 +189,11 @@ interface ChatSidebarProps {
voiceAvailable?: boolean voiceAvailable?: boolean
ttsAvailable?: boolean ttsAvailable?: boolean
ttsEnabled?: boolean ttsEnabled?: boolean
ttsMode?: 'summary' | 'full'
onToggleTts?: () => void onToggleTts?: () => void
onTtsModeChange?: (mode: 'summary' | 'full') => void inCall?: boolean
ttsAvatarEnabled?: boolean onStartCall?: (preset: CallPreset) => void
onToggleTtsAvatar?: () => void onEndCall?: () => void
videoChatMode?: VideoChatMode callAvailable?: boolean
onVideoModeChange?: (mode: VideoChatMode) => void
videoCallAvailable?: boolean
onComposioConnected?: (toolkitSlug: string) => void onComposioConnected?: (toolkitSlug: string) => void
} }
@ -258,14 +255,11 @@ export function ChatSidebar({
voiceAvailable, voiceAvailable,
ttsAvailable, ttsAvailable,
ttsEnabled, ttsEnabled,
ttsMode,
onToggleTts, onToggleTts,
onTtsModeChange, inCall,
ttsAvatarEnabled, onStartCall,
onToggleTtsAvatar, onEndCall,
videoChatMode, callAvailable,
onVideoModeChange,
videoCallAvailable,
onComposioConnected, onComposioConnected,
}: ChatSidebarProps) { }: ChatSidebarProps) {
const { state: sidebarState } = useSidebar() const { state: sidebarState } = useSidebar()
@ -837,14 +831,11 @@ export function ChatSidebar({
voiceAvailable={isActive && voiceAvailable} voiceAvailable={isActive && voiceAvailable}
ttsAvailable={isActive && ttsAvailable} ttsAvailable={isActive && ttsAvailable}
ttsEnabled={ttsEnabled} ttsEnabled={ttsEnabled}
ttsMode={ttsMode}
onToggleTts={isActive ? onToggleTts : undefined} onToggleTts={isActive ? onToggleTts : undefined}
onTtsModeChange={isActive ? onTtsModeChange : undefined} inCall={inCall}
ttsAvatarEnabled={ttsAvatarEnabled} onStartCall={isActive ? onStartCall : undefined}
onToggleTtsAvatar={isActive ? onToggleTtsAvatar : undefined} onEndCall={isActive ? onEndCall : undefined}
videoChatMode={videoChatMode} callAvailable={callAvailable}
onVideoModeChange={isActive ? onVideoModeChange : undefined}
videoCallAvailable={videoCallAvailable}
/> />
</div> </div>
) )

240
apps/x/apps/renderer/src/components/video-call-view.tsx
View file

@ -1,5 +1,5 @@
import { useEffect, useRef, useState } from 'react' import { useEffect, useRef, useState } from 'react'
import { MonitorUp, PhoneOff, User, Video, VideoOff } from 'lucide-react' import { Minimize2, MonitorUp, PhoneOff, Presentation, User, Video, VideoOff } from 'lucide-react'
import { MascotFaceIcon, TalkingHead } from '@/components/talking-head' import { MascotFaceIcon, TalkingHead } from '@/components/talking-head'
import type { TTSState } from '@/hooks/useVoiceTTS' import type { TTSState } from '@/hooks/useVoiceTTS'
@ -10,12 +10,15 @@ export type VideoCallStatus = 'listening' | 'thinking' | 'speaking'
interface VideoCallViewProps { interface VideoCallViewProps {
/** Live camera stream from useVideoMode — attached to the user's tile. */ /** Live camera stream from useVideoMode — attached to the user's tile. */
streamRef: React.MutableRefObject<MediaStream | null> streamRef: React.MutableRefObject<MediaStream | null>
/** Live screen-share stream — shown as the presentation tile when sharing. */
screenStreamRef: React.MutableRefObject<MediaStream | null>
isScreenSharing: boolean
onToggleScreenShare: () => void
cameraOn: boolean cameraOn: boolean
onToggleCamera: () => void onToggleCamera: () => void
/** Starting a share collapses this view into the floating popout (the
* surface is derived from devices — see App.tsx). */
onToggleScreenShare: () => void
/** Practice preset: the assistant is coaching this session. */
practiceMode?: boolean
/** Shrink to the floating pill without touching any devices. */
onMinimize: () => void
ttsState: TTSState ttsState: TTSState
/** Live TTS output level — drives the mascot's mouth animation. */ /** Live TTS output level — drives the mascot's mouth animation. */
getTtsLevel: () => number getTtsLevel: () => number
@ -33,53 +36,22 @@ const STATUS_DISPLAY: Record<VideoCallStatus, { label: string; dotClass: string
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' }, speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
} }
/** Attach a MediaStream ref to a <video> element for the lifetime of the mount. */
function StreamVideo({
streamRef,
mirrored,
className,
}: {
streamRef: React.MutableRefObject<MediaStream | null>
mirrored?: boolean
className?: string
}) {
const videoRef = useRef<HTMLVideoElement | null>(null)
useEffect(() => {
const videoEl = videoRef.current
if (!videoEl) return
videoEl.srcObject = streamRef.current
videoEl.play().catch(() => {})
return () => {
videoEl.srcObject = null
}
}, [streamRef])
return (
<video
ref={videoRef}
muted
playsInline
className={className}
style={mirrored ? { transform: 'scaleX(-1)' } : undefined}
/>
)
}
/** /**
* Full-screen hands-free call: a Meet-style layout with the user's webcam on * Full-screen call surface: a Meet-style two-tile layout with the user's
* one side and the mascot as the other participant. While presenting, the * webcam on one side and the mascot as the other participant. Shown only
* shared screen becomes the big tile and the participants shrink into a side * while the camera is on with no screen share (the derived-surface rule in
* rail. The mascot animates with the assistant's speech; dismissing it swaps * App.tsx) — sharing or muting the camera moves the call into the floating
* in a Meet-style letter avatar ("R"). Live captions run along the bottom. * popout. The mascot animates with the assistant's speech; dismissing it
* swaps in a Meet-style letter avatar ("R"). Live captions run along the
* bottom.
*/ */
export function VideoCallView({ export function VideoCallView({
streamRef, streamRef,
screenStreamRef,
isScreenSharing,
onToggleScreenShare,
cameraOn, cameraOn,
onToggleCamera, onToggleCamera,
onToggleScreenShare,
practiceMode,
onMinimize,
ttsState, ttsState,
getTtsLevel, getTtsLevel,
status, status,
@ -87,8 +59,20 @@ export function VideoCallView({
assistantCaption, assistantCaption,
onLeave, onLeave,
}: VideoCallViewProps) { }: VideoCallViewProps) {
const videoRef = useRef<HTMLVideoElement | null>(null)
const [mascotVisible, setMascotVisible] = useState(true) const [mascotVisible, setMascotVisible] = useState(true)
useEffect(() => {
if (!cameraOn) return
const videoEl = videoRef.current
if (!videoEl) return
videoEl.srcObject = streamRef.current
videoEl.play().catch(() => {})
return () => {
videoEl.srcObject = null
}
}, [streamRef, cameraOn])
const userSpeaking = status === 'listening' && Boolean(interimText) const userSpeaking = status === 'listening' && Boolean(interimText)
const assistantSpeaking = ttsState === 'speaking' const assistantSpeaking = ttsState === 'speaking'
@ -98,89 +82,80 @@ export function VideoCallView({
? { who: 'You', text: interimText } ? { who: 'You', text: interimText }
: null : null
const userTile = (
<div
className={cn(
'relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
userSpeaking && 'ring-2 ring-green-500/80',
isScreenSharing && 'aspect-video w-full'
)}
>
{cameraOn ? (
<StreamVideo streamRef={streamRef} mirrored className="h-full w-full object-cover" />
) : (
<span
className={cn(
'flex items-center justify-center rounded-full bg-neutral-700 text-neutral-400',
isScreenSharing ? 'h-16 w-16' : 'h-40 w-40'
)}
aria-label="Camera off"
>
<User className={isScreenSharing ? 'h-8 w-8' : 'h-20 w-20'} />
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
You
</span>
</div>
)
const assistantTile = (
<div
className={cn(
'group relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
assistantSpeaking && 'ring-2 ring-sky-400/80',
isScreenSharing && 'aspect-video w-full'
)}
>
{mascotVisible ? (
<TalkingHead ttsState={ttsState} getLevel={getTtsLevel} size={isScreenSharing ? 96 : 220} />
) : (
<span
className={cn(
'flex items-center justify-center rounded-full bg-sky-600 font-medium text-white',
isScreenSharing ? 'h-16 w-16 text-3xl' : 'h-40 w-40 text-7xl'
)}
aria-label="Rowboat"
>
R
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
Rowboat
</span>
<button
type="button"
onClick={() => setMascotVisible((v) => !v)}
className="absolute right-3 top-3 rounded-md bg-black/50 px-2 py-1 text-xs text-white/80 opacity-0 transition-opacity hover:text-white group-hover:opacity-100"
>
{mascotVisible ? 'Hide mascot' : 'Show mascot'}
</button>
</div>
)
return ( return (
<div className="fixed inset-0 z-[100] flex flex-col bg-neutral-950"> <div className="fixed inset-0 z-[100] flex flex-col bg-neutral-950">
{/* Participant tiles — Meet-style presentation layout while sharing */} {practiceMode && (
{isScreenSharing ? ( <span className="absolute left-4 top-4 z-10 flex items-center gap-1.5 rounded-full bg-violet-600/90 px-3 py-1 text-xs font-medium text-white">
<div className="flex min-h-0 flex-1 gap-3 p-4 pb-2"> <Presentation className="h-3.5 w-3.5" />
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-2xl bg-neutral-900"> Practice session
<StreamVideo streamRef={screenStreamRef} className="h-full w-full object-contain" /> </span>
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
Your screen
</span>
</div>
<div className="flex w-52 shrink-0 flex-col gap-3">
{userTile}
{assistantTile}
</div>
</div>
) : (
<div className="grid min-h-0 flex-1 grid-cols-1 gap-3 p-4 pb-2 md:grid-cols-2">
{userTile}
{assistantTile}
</div>
)} )}
<button
type="button"
onClick={onMinimize}
className="absolute right-4 top-4 z-10 flex h-8 w-8 items-center justify-center rounded-full bg-neutral-800 text-white/80 transition-colors hover:bg-neutral-700 hover:text-white"
aria-label="Minimize call (shares your screen)"
title="Minimize — shares your screen so it can help you work"
>
<Minimize2 className="h-4 w-4" />
</button>
{/* Participant tiles */}
<div className="grid min-h-0 flex-1 grid-cols-1 gap-3 p-4 pb-2 md:grid-cols-2">
{/* User */}
<div
className={cn(
'relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
userSpeaking && 'ring-2 ring-green-500/80'
)}
>
{cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-full w-full object-cover"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<span className="flex h-40 w-40 items-center justify-center rounded-full bg-neutral-700 text-neutral-400" aria-label="Camera off">
<User className="h-20 w-20" />
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
You
</span>
</div>
{/* Assistant */}
<div
className={cn(
'group relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
assistantSpeaking && 'ring-2 ring-sky-400/80'
)}
>
{mascotVisible ? (
<TalkingHead ttsState={ttsState} getLevel={getTtsLevel} size={220} />
) : (
<span
className="flex h-40 w-40 items-center justify-center rounded-full bg-sky-600 text-7xl font-medium text-white"
aria-label="Rowboat"
>
R
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
Rowboat
</span>
<button
type="button"
onClick={() => setMascotVisible((v) => !v)}
className="absolute right-3 top-3 rounded-md bg-black/50 px-2 py-1 text-xs text-white/80 opacity-0 transition-opacity hover:text-white group-hover:opacity-100"
>
{mascotVisible ? 'Hide mascot' : 'Show mascot'}
</button>
</div>
</div>
{/* Captions */} {/* Captions */}
<div className="flex h-14 items-center justify-center px-6"> <div className="flex h-14 items-center justify-center px-6">
@ -215,14 +190,9 @@ export function VideoCallView({
<button <button
type="button" type="button"
onClick={onToggleScreenShare} onClick={onToggleScreenShare}
className={cn( className="flex h-10 w-10 items-center justify-center rounded-full bg-neutral-800 text-white/90 transition-colors hover:bg-neutral-700"
'flex h-10 w-10 items-center justify-center rounded-full transition-colors', aria-label="Present your screen"
isScreenSharing title="Present your screen"
? 'bg-sky-600 text-white hover:bg-sky-500'
: 'bg-neutral-800 text-white/90 hover:bg-neutral-700'
)}
aria-label={isScreenSharing ? 'Stop presenting' : 'Present your screen'}
title={isScreenSharing ? 'Stop presenting' : 'Present your screen'}
> >
<MonitorUp className="h-5 w-5" /> <MonitorUp className="h-5 w-5" />
</button> </button>
@ -243,7 +213,7 @@ export function VideoCallView({
type="button" type="button"
onClick={onLeave} onClick={onLeave}
className="flex h-10 w-14 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500" className="flex h-10 w-14 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
aria-label="Leave call" aria-label="End call"
> >
<PhoneOff className="h-5 w-5" /> <PhoneOff className="h-5 w-5" />
</button> </button>

51
apps/x/apps/renderer/src/components/video-popout.tsx
View file

@ -7,6 +7,7 @@ type PopoutState = {
ttsState: 'idle' | 'synthesizing' | 'speaking' ttsState: 'idle' | 'synthesizing' | 'speaking'
status: 'listening' | 'thinking' | 'speaking' | null status: 'listening' | 'thinking' | 'speaking' | null
cameraOn: boolean cameraOn: boolean
screenSharing: boolean
interimText: string | null interimText: string | null
} }
@ -23,18 +24,30 @@ const noDragRegion = { WebkitAppRegion: 'no-drag' } as React.CSSProperties
* Content of the always-on-top popout window shown for the whole duration of * Content of the always-on-top popout window shown for the whole duration of
* a screen share (Meet-style floating mini-call) — it floats over every app, * a screen share (Meet-style floating mini-call) — it floats over every app,
* including Rowboat itself, and is the call's control surface while sharing: * including Rowboat itself, and is the call's control surface while sharing:
* camera toggle, stop-share, end-call. Rendered in its own BrowserWindow * camera toggle, share toggle, end-call. Rendered in its own BrowserWindow
* (see `video:setPopout` in the main process); call state streams in over * (see `video:setPopout` in the main process); call state streams in over
* the `video:popout-state` push channel and control actions round-trip back * the `video:popout-state` push channel and control actions round-trip back
* through `video:popoutAction`. Captures its own webcam feed — MediaStreams * through `video:popoutAction`. Captures its own webcam feed — MediaStreams
* can't cross windows. * can't cross windows.
*/ */
export function VideoPopout() { export function VideoPopout() {
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: true, interimText: null }) // Camera defaults OFF: guessing "on" would flash the user's video for a
// beat before the real state arrives — which reads as a bug. The true
// state is fetched immediately below.
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: false, screenSharing: false, interimText: null })
const videoRef = useRef<HTMLVideoElement | null>(null) const videoRef = useRef<HTMLVideoElement | null>(null)
useEffect(() => { useEffect(() => {
return window.ipc.on('video:popout-state', (next) => setState(next)) const cleanup = window.ipc.on('video:popout-state', (next) => setState(next))
// The main process replays the cached state on did-finish-load, but that
// can race this listener's registration — fetch it explicitly too.
window.ipc
.invoke('video:getPopoutState', null)
.then(({ state: cached }) => {
if (cached) setState(cached)
})
.catch(() => {})
return cleanup
}, []) }, [])
// Own camera feed, following the main window's camera-on/off state. // Own camera feed, following the main window's camera-on/off state.
@ -67,7 +80,7 @@ export function VideoPopout() {
// so the mascot still animates while the assistant speaks in the main window. // so the mascot still animates while the assistant speaks in the main window.
const getLevel = useCallback(() => 0.45 + 0.35 * Math.sin(performance.now() / 90), []) const getLevel = useCallback(() => 0.45 + 0.35 * Math.sin(performance.now() / 90), [])
const sendAction = useCallback((action: 'toggle-camera' | 'stop-share' | 'end-call') => { const sendAction = useCallback((action: 'toggle-camera' | 'toggle-share' | 'end-call' | 'expand') => {
void window.ipc.invoke('video:popoutAction', { action }).catch(() => {}) void window.ipc.invoke('video:popoutAction', { action }).catch(() => {})
}, []) }, [])
@ -98,6 +111,14 @@ export function VideoPopout() {
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white"> <span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
You You
</span> </span>
{/* Persistent consent badge — the user must always be able to see
at a glance that their screen is going out. */}
{state.screenSharing && (
<span className="absolute left-1.5 top-1.5 flex items-center gap-1 rounded-full bg-sky-600/90 px-1.5 py-0.5 text-[10px] font-medium text-white">
<span className="block h-1.5 w-1.5 animate-pulse rounded-full bg-white" />
Sharing screen
</span>
)}
</div> </div>
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-lg bg-neutral-800"> <div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-lg bg-neutral-800">
<TalkingHead ttsState={state.ttsState} getLevel={getLevel} size={84} /> <TalkingHead ttsState={state.ttsState} getLevel={getLevel} size={84} />
@ -138,10 +159,14 @@ export function VideoPopout() {
</button> </button>
<button <button
type="button" type="button"
onClick={() => sendAction('stop-share')} onClick={() => sendAction('toggle-share')}
className="flex h-6 w-6 items-center justify-center rounded-full bg-sky-600 text-white transition-colors hover:bg-sky-500" className={`flex h-6 w-6 items-center justify-center rounded-full transition-colors ${
aria-label="Stop sharing screen" state.screenSharing
title="Stop sharing screen" ? 'bg-sky-600 text-white hover:bg-sky-500'
: 'bg-neutral-700 text-white/90 hover:bg-neutral-600'
}`}
aria-label={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
title={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
> >
<MonitorUp className="h-3.5 w-3.5" /> <MonitorUp className="h-3.5 w-3.5" />
</button> </button>
@ -149,17 +174,17 @@ export function VideoPopout() {
type="button" type="button"
onClick={() => sendAction('end-call')} onClick={() => sendAction('end-call')}
className="flex h-6 w-8 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500" className="flex h-6 w-8 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
aria-label="End video mode" aria-label="End call"
title="End video mode" title="End call"
> >
<PhoneOff className="h-3.5 w-3.5" /> <PhoneOff className="h-3.5 w-3.5" />
</button> </button>
<button <button
type="button" type="button"
onClick={() => void window.ipc.invoke('video:focusMain', null)} onClick={() => sendAction('expand')}
className="flex h-6 w-6 items-center justify-center rounded-full bg-neutral-700 text-white/90 transition-colors hover:bg-neutral-600" className="flex h-6 w-6 items-center justify-center rounded-full bg-neutral-700 text-white/90 transition-colors hover:bg-neutral-600"
aria-label="Back to Rowboat" aria-label="Expand to full screen (stops screen sharing)"
title="Back to Rowboat" title="Expand to full screen (stops sharing)"
> >
<Maximize2 className="h-3.5 w-3.5" /> <Maximize2 className="h-3.5 w-3.5" />
</button> </button>

129
apps/x/apps/renderer/src/components/video-preview-overlay.tsx
View file

@ -1,129 +0,0 @@
import { useEffect, useRef } from 'react'
import { MonitorUp, User, VideoOff, X } from 'lucide-react'
interface VideoPreviewOverlayProps {
/** Live camera stream from useVideoMode — attached to the preview element. */
streamRef: React.MutableRefObject<MediaStream | null>
onTurnOff: () => void
/** Hands-free call mode: current phase of the conversation loop. */
callStatus?: 'listening' | 'thinking' | 'speaking'
/** Hands-free call mode: live transcript of the in-progress utterance. */
interimText?: string
isScreenSharing?: boolean
onToggleScreenShare?: () => void
cameraOn?: boolean
onToggleCamera?: () => void
}
const CALL_STATUS_DISPLAY: Record<NonNullable<VideoPreviewOverlayProps['callStatus']>, { label: string; dotClass: string }> = {
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
}
/**
* Floating picture-in-picture webcam preview shown while video chat mode is
* on. Mirrored like a selfie camera so the user's movements feel natural.
* Sits above the composer dock, mirroring the talking-head overlay's corner.
*/
export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimText, isScreenSharing, onToggleScreenShare, cameraOn = true, onToggleCamera }: VideoPreviewOverlayProps) {
const videoRef = useRef<HTMLVideoElement | null>(null)
useEffect(() => {
if (!cameraOn) return
const videoEl = videoRef.current
if (!videoEl) return
videoEl.srcObject = streamRef.current
videoEl.play().catch(() => {})
return () => {
videoEl.srcObject = null
}
}, [streamRef, cameraOn])
return (
<div
className="group fixed bottom-28 left-8 z-50"
style={{ animation: 'video-preview-pop 0.35s cubic-bezier(0.34, 1.56, 0.64, 1)' }}
>
<style>{`
@keyframes video-preview-pop {
0% { opacity: 0; scale: 0.4; }
100% { opacity: 1; scale: 1; }
}
`}</style>
<button
type="button"
onClick={onTurnOff}
className="absolute -right-1.5 -top-1.5 z-10 flex h-5 w-5 items-center justify-center rounded-full border border-border bg-background text-muted-foreground opacity-0 shadow-sm transition-opacity hover:text-foreground group-hover:opacity-100"
aria-label="Turn off video chat"
>
<X className="h-3 w-3" />
</button>
{onToggleScreenShare && (
<button
type="button"
onClick={onToggleScreenShare}
className={`absolute -left-1.5 -top-1.5 z-10 flex h-5 w-5 items-center justify-center rounded-full border shadow-sm transition-opacity ${
isScreenSharing
? 'border-sky-500 bg-sky-600 text-white opacity-100'
: 'border-border bg-background text-muted-foreground opacity-0 hover:text-foreground group-hover:opacity-100'
}`}
aria-label={isScreenSharing ? 'Stop sharing screen' : 'Share your screen'}
title={isScreenSharing ? 'Stop sharing screen' : 'Share your screen'}
>
<MonitorUp className="h-3 w-3" />
</button>
)}
{cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-32 w-auto rounded-xl border border-border/70 bg-black shadow-lg"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<div className="flex h-32 w-48 items-center justify-center rounded-xl border border-border/70 bg-black shadow-lg">
<span className="flex h-14 w-14 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
<User className="h-7 w-7" />
</span>
</div>
)}
{onToggleCamera && (
<button
type="button"
onClick={onToggleCamera}
className={`absolute -left-1.5 top-4 z-10 flex h-5 w-5 items-center justify-center rounded-full border shadow-sm transition-opacity ${
cameraOn
? 'border-border bg-background text-muted-foreground opacity-0 hover:text-foreground group-hover:opacity-100'
: 'border-red-500 bg-red-600 text-white opacity-100'
}`}
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
>
<VideoOff className="h-3 w-3" />
</button>
)}
<span className="pointer-events-none absolute bottom-1.5 left-1.5 flex items-center gap-1 rounded-full bg-black/60 px-1.5 py-0.5 text-[10px] font-medium text-white">
{callStatus ? (
<>
<span className={`block h-1.5 w-1.5 rounded-full ${CALL_STATUS_DISPLAY[callStatus].dotClass}`} />
{CALL_STATUS_DISPLAY[callStatus].label}
</>
) : (
<>
<span className="block h-1.5 w-1.5 animate-pulse rounded-full bg-red-500" />
Video on
</>
)}
</span>
{callStatus && interimText && (
<div className="pointer-events-none absolute inset-x-0 -bottom-1 translate-y-full pt-1">
<div className="max-h-16 overflow-hidden rounded-lg bg-black/60 px-2 py-1 text-[11px] leading-snug text-white/90">
{interimText}
</div>
</div>
)}
</div>
)
}

19
apps/x/apps/renderer/src/hooks/useVideoMode.ts
View file

@ -234,15 +234,22 @@ export function useVideoMode() {
return true; return true;
}, [acquireCamera]); }, [acquireCamera]);
const start = useCallback(async (): Promise<boolean> => { /**
* Start video mode. `camera: false` starts a camera-less session (voice
* call / screen-share-only) — the mode is live so frames can flow from
* other sources, and the camera can be enabled later via setCameraEnabled.
*/
const start = useCallback(async ({ camera = true }: { camera?: boolean } = {}): Promise<boolean> => {
if (stateRef.current !== 'idle') return true; if (stateRef.current !== 'idle') return true;
setState('starting'); setState('starting');
const ok = await acquireCamera(); if (camera) {
if (!ok) { const ok = await acquireCamera();
setState('idle'); if (!ok) {
return false; setState('idle');
return false;
}
} }
setCameraOn(true); setCameraOn(camera);
setState('live'); setState('live');
return true; return true;
}, [acquireCamera]); }, [acquireCamera]);

4
apps/x/apps/renderer/src/lib/analytics.ts
View file

@ -65,6 +65,10 @@ export function voiceInputStarted() {
posthog.capture('voice_input_started') posthog.capture('voice_input_started')
} }
export function callStarted(preset: 'voice' | 'video' | 'share' | 'practice') {
posthog.capture('call_started', { preset })
}
export function searchExecuted(types: string[]) { export function searchExecuted(types: string[]) {
posthog.capture('search_executed', { types }) posthog.capture('search_executed', { types })
} }

14
apps/x/packages/core/src/agents/runtime.ts
View file

@ -339,6 +339,7 @@ export interface ComposeSystemInstructionsInput {
codeCwd: string | null; codeCwd: string | null;
// Optional so legacy callers (old streamAgent path) are unaffected. // Optional so legacy callers (old streamAgent path) are unaffected.
videoMode?: boolean; videoMode?: boolean;
coachMode?: boolean;
} }
// System-prompt assembly, extracted verbatim from streamAgent so the new turn // System-prompt assembly, extracted verbatim from streamAgent so the new turn
@ -354,6 +355,7 @@ export function composeSystemInstructions({
codeMode, codeMode,
codeCwd, codeCwd,
videoMode, videoMode,
coachMode,
}: ComposeSystemInstructionsInput): string { }: ComposeSystemInstructionsInput): string {
let instructionsWithDateTime = `${instructions}\n\n${USER_CONTEXT_SYSTEM_INSTRUCTIONS}`; let instructionsWithDateTime = `${instructions}\n\n${USER_CONTEXT_SYSTEM_INSTRUCTIONS}`;
if (agentNotesContext) { if (agentNotesContext) {
@ -402,6 +404,18 @@ Etiquette:
- Do not narrate or list what you see in every response. Bring up visual observations only when relevant to the user's request or clearly worth mentioning. - Do not narrate or list what you see in every response. Bring up visual observations only when relevant to the user's request or clearly worth mentioning.
- Never comment on the user's physical appearance, attractiveness, or personal attributes — visual feedback is strictly about delivery, expression, and body language. - Never comment on the user's physical appearance, attractiveness, or personal attributes — visual feedback is strictly about delivery, expression, and body language.
- Frames are periodic snapshots, not continuous video; moments between frames are missing. Don't claim certainty about motion you couldn't have seen.`; - Frames are periodic snapshots, not continuous video; moments between frames are missing. Don't claim certainty about motion you couldn't have seen.`;
}
if (coachMode) {
instructionsWithDateTime += `\n\n# Practice Session (Coach Mode)
The user started a practice session: they are rehearsing something performative — a pitch, presentation, interview answer, or talk — and want live coaching. You are their coach for this session.
How to coach:
- Watch and listen for delivery: pacing, filler words, rambling, structure, and (from the webcam frames) posture, eye contact with the camera, facial expressiveness, gesturing, and visible energy.
- After each take or answer, give brief, specific, actionable feedback: 2-3 concrete observations, quoting what you actually saw or heard ("you looked down during the ask", "the opening 'um, so, basically' undercuts your hook"). Then let them go again.
- If they are clearly mid-flow, keep any interjection to one short sentence — or stay silent and save it for the break.
- Ask what they're practicing for at the start if it isn't obvious (investor pitch? interview? conference talk?) and tailor feedback to that audience.
- When they say they're done or wrap up, give a structured debrief: what's working, the top 3 things to improve, and one concrete drill or reframe to try next time.
- Be encouraging but honest — vague praise wastes their rehearsal time. Never comment on physical appearance; delivery, expression, and body language only.`;
} }
if (voiceOutput === 'summary') { if (voiceOutput === 'summary') {
instructionsWithDateTime += `\n\n# Voice Output (MANDATORY — READ THIS FIRST)\nThe user has voice output enabled. THIS IS YOUR #1 PRIORITY: you MUST start your response with <voice></voice> tags. If your response does not begin with <voice> tags, the user will hear nothing — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. Do not start with markdown, headings, or any other text. The literal first characters of your response must be "<voice>".\n2. Place ALL <voice> tags at the BEGINNING of your response, before any detailed content. Do NOT intersperse <voice> tags throughout the response.\n3. Wrap EACH spoken sentence in its own separate <voice> tag so it can be spoken incrementally. Do NOT wrap everything in a single <voice> block.\n4. Use voice as a TL;DR and navigation aid — do NOT read the entire response aloud.\n5. After all <voice> tags, you may include detailed written content (markdown, tables, code, etc.) that will be shown visually but not spoken.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things: the Q2 roadmap timeline, hiring for the backend role, and the client demo next week.</voice>\n<voice>I've pulled out the key details and action items below — the demo prep notes are at the end.</voice>\n\n## Meeting with Alex — March 11\n### Roadmap\n- Agreed to push Q2 launch to April 15...\n(detailed written content continues)\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You have five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you requested and Taylor flagged a contract issue.</voice>\n<voice>There's also a warm intro from a VC partner connecting you with someone at a prospective customer.</voice>\n<voice>I've drafted responses for three of them. The details and drafts are below.</voice>\n\n(email blocks, tables, and detailed content follow)\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a pretty packed day — seven meetings starting with standup at 9.</voice>\n<voice>The big ones are your investor call at 11, lunch with a partner from your lead VC at 12:30, and a customer call at 4.</voice>\n<voice>Your only free block for deep work is 2:30 to 4.</voice>\n\n(calendar block with full event details follows)\n\nExample 4 — User asks: "draft an email to Sam with our metrics"\n\n<voice>Done — I've drafted the email to Sam with your latest WAU and churn numbers.</voice>\n<voice>Take a look at the draft below and send it when you're ready.</voice>\n\n(email block with draft follows)\n\nREMEMBER: If you do not start with <voice> tags, the user hears silence. Always speak first, then write.`; instructionsWithDateTime += `\n\n# Voice Output (MANDATORY — READ THIS FIRST)\nThe user has voice output enabled. THIS IS YOUR #1 PRIORITY: you MUST start your response with <voice></voice> tags. If your response does not begin with <voice> tags, the user will hear nothing — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. Do not start with markdown, headings, or any other text. The literal first characters of your response must be "<voice>".\n2. Place ALL <voice> tags at the BEGINNING of your response, before any detailed content. Do NOT intersperse <voice> tags throughout the response.\n3. Wrap EACH spoken sentence in its own separate <voice> tag so it can be spoken incrementally. Do NOT wrap everything in a single <voice> block.\n4. Use voice as a TL;DR and navigation aid — do NOT read the entire response aloud.\n5. After all <voice> tags, you may include detailed written content (markdown, tables, code, etc.) that will be shown visually but not spoken.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things: the Q2 roadmap timeline, hiring for the backend role, and the client demo next week.</voice>\n<voice>I've pulled out the key details and action items below — the demo prep notes are at the end.</voice>\n\n## Meeting with Alex — March 11\n### Roadmap\n- Agreed to push Q2 launch to April 15...\n(detailed written content continues)\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You have five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you requested and Taylor flagged a contract issue.</voice>\n<voice>There's also a warm intro from a VC partner connecting you with someone at a prospective customer.</voice>\n<voice>I've drafted responses for three of them. The details and drafts are below.</voice>\n\n(email blocks, tables, and detailed content follow)\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a pretty packed day — seven meetings starting with standup at 9.</voice>\n<voice>The big ones are your investor call at 11, lunch with a partner from your lead VC at 12:30, and a customer call at 4.</voice>\n<voice>Your only free block for deep work is 2:30 to 4.</voice>\n\n(calendar block with full event details follows)\n\nExample 4 — User asks: "draft an email to Sam with our metrics"\n\n<voice>Done — I've drafted the email to Sam with your latest WAU and churn numbers.</voice>\n<voice>Take a look at the draft below and send it when you're ready.</voice>\n\n(email block with draft follows)\n\nREMEMBER: If you do not start with <voice> tags, the user hears silence. Always speak first, then write.`;

2
apps/x/packages/core/src/turns/bridges/real-agent-resolver.ts
View file

@ -55,6 +55,7 @@ const CompositionOverrides = z.object({
codeMode: z.enum(["claude", "codex"]).nullable().optional(), codeMode: z.enum(["claude", "codex"]).nullable().optional(),
codeCwd: z.string().nullable().optional(), codeCwd: z.string().nullable().optional(),
videoMode: z.boolean().optional(), videoMode: z.boolean().optional(),
coachMode: z.boolean().optional(),
}); });
export interface RealAgentResolverDeps { export interface RealAgentResolverDeps {
@ -123,6 +124,7 @@ export class RealAgentResolver implements IAgentResolver {
codeMode: composition.codeMode ?? null, codeMode: composition.codeMode ?? null,
codeCwd: composition.codeCwd ?? null, codeCwd: composition.codeCwd ?? null,
videoMode: composition.videoMode ?? false, videoMode: composition.videoMode ?? false,
coachMode: composition.coachMode ?? false,
}); });
const tools = await this.resolveTools(agent); const tools = await this.resolveTools(agent);

30
apps/x/packages/shared/src/ipc.ts
View file

@ -1394,21 +1394,36 @@ const ipcSchemas = {
ttsState: z.enum(['idle', 'synthesizing', 'speaking']), ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
status: z.enum(['listening', 'thinking', 'speaking']).nullable(), status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
cameraOn: z.boolean(), cameraOn: z.boolean(),
// Live transcript of the in-progress utterance (hands-free call only). screenSharing: z.boolean(),
// Live transcript of the in-progress utterance.
interimText: z.string().nullable(), interimText: z.string().nullable(),
}), }),
res: z.object({}), res: z.object({}),
}, },
// Popout window → bring the main app window back to the foreground. // Popout window → fetch the latest cached call state on mount. The
'video:focusMain': { // did-finish-load replay can race the React listener registration, and the
// popout must never guess (a wrong camera-on default flashes the user's
// video before the first state push corrects it).
'video:getPopoutState': {
req: z.null(), req: z.null(),
res: z.object({}), res: z.object({
state: z
.object({
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
cameraOn: z.boolean(),
screenSharing: z.boolean(),
interimText: z.string().nullable(),
})
.nullable(),
}),
}, },
// Popout control bar → main process → relayed to the app window, which // Popout control bar → main process → relayed to the app window, which
// executes the action on the live call. // executes the action on the live call. 'expand' additionally focuses the
// main app window (handled in the main process).
'video:popoutAction': { 'video:popoutAction': {
req: z.object({ req: z.object({
action: z.enum(['toggle-camera', 'stop-share', 'end-call']), action: z.enum(['toggle-camera', 'toggle-share', 'end-call', 'expand']),
}), }),
res: z.object({}), res: z.object({}),
}, },
@ -1418,6 +1433,7 @@ const ipcSchemas = {
ttsState: z.enum(['idle', 'synthesizing', 'speaking']), ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
status: z.enum(['listening', 'thinking', 'speaking']).nullable(), status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
cameraOn: z.boolean(), cameraOn: z.boolean(),
screenSharing: z.boolean(),
interimText: z.string().nullable(), interimText: z.string().nullable(),
}), }),
res: z.null(), res: z.null(),
@ -1425,7 +1441,7 @@ const ipcSchemas = {
// Push channel: main → app window with a popout control-bar action. // Push channel: main → app window with a popout control-bar action.
'video:popout-action': { 'video:popout-action': {
req: z.object({ req: z.object({
action: z.enum(['toggle-camera', 'stop-share', 'end-call']), action: z.enum(['toggle-camera', 'toggle-share', 'end-call', 'expand']),
}), }),
res: z.null(), res: z.null(),
}, },