diff --git a/CLAUDE.md b/CLAUDE.md index c80edc23..5e762a33 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,7 +109,7 @@ Long-form docs for specific features. Read the relevant file before making chang | Feature | Doc | |---------|-----| | Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` | -| Video mode — webcam + screen-share frames to the model (video+chat / hands-free call / full-screen Meet-style call), continuous voice loop, camera mute, popout window, prompts catalog | `apps/x/VIDEO_MODE.md` | +| Calls (video mode) — one hands-free call engine with four presets (voice / video / share screen / practice coaching), device-derived surfaces (full-screen ⇄ floating popout), frame pipeline, prompts catalog | `apps/x/VIDEO_MODE.md` | | Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` | | Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` | diff --git a/apps/x/ANALYTICS.md b/apps/x/ANALYTICS.md index 5ddfcf6e..becf99e2 100644 --- a/apps/x/ANALYTICS.md +++ b/apps/x/ANALYTICS.md @@ -92,6 +92,7 @@ All in `apps/renderer/src/lib/analytics.ts`: - `chat_message_sent` — `{ voice_input, voice_output, search_enabled }` - `oauth_connected` / `oauth_disconnected` — `{ provider }` - `voice_input_started` — no properties +- `call_started` — `{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`) - `search_executed` — `{ types: string[] }` - `note_exported` — `{ format }` diff --git a/apps/x/VIDEO_MODE.md b/apps/x/VIDEO_MODE.md index 5b04b770..de8c1dea 100644 --- a/apps/x/VIDEO_MODE.md +++ b/apps/x/VIDEO_MODE.md @@ -1,46 +1,70 @@ -# Video Mode — Deep Dive +# Calls (Video Mode) — Deep Dive -Video mode lets the assistant *see* the user (webcam) and their screen (screen -share), in three presentations: frames attached to normal chat, a hands-free -spoken call, and a full-screen Meet-style call. This doc covers the product -flow, the technical pipeline, and the LLM prompt surface with exact pointers. +Calls let the user talk to the assistant hands-free while it *sees* them +(webcam) and their screen (screen share). There is ONE call engine — +continuous listening, auto-submitted utterances, forced read-aloud TTS, frame +capture — entered through four presets that differ only in starting devices. +This doc covers the product flow, the technical pipeline, and the LLM prompt +surface with exact pointers. ## Product flow -The composer's video button (`chat-input-with-mentions.tsx`) toggles video -mode; a chevron dropdown picks one of three modes (`VideoChatMode`): +The composer has a **call split-button** (`chat-input-with-mentions.tsx`). +The main click is the "work together" default — preset `share`: screen +sharing ON, camera OFF, floating pill, so the user keeps working while the +assistant watches along (the button tooltip discloses the screen share). The +chevron menu holds the deviations. While a call is live the button turns red +and ends it. -| Mode | What it does | -|------|--------------| -| `chat` — "Video + chat" | Camera on. Webcam (and screen-share) frames ride along with every typed or dictated message. Small PiP preview floats above the composer. | -| `call` — "Video call (hands-free)" | Everything in `chat`, plus: continuous listening (each utterance auto-submits as a voice message) and forced full read-aloud TTS. No typing needed; composer still works. | -| `meeting` — "Video call (full screen)" | Same pipeline as `call`, presented as a full-screen Meet-style layout: user tile + animated mascot tile, captions, control bar. | +| Preset | Starting devices | First surface | +|--------|------------------|---------------| +| `share` — main click | screen on, camera off | floating pill | +| `voice` — "Voice call" | camera off, screen off | floating mascot pill | +| `video` — "Video call" | camera on | full-screen call | +| `practice` — "Practice session" | camera on, + coaching persona | full-screen call | -On top of any mode: +**One surface rule** (`callSurface` in `App.tsx`): full screen and screen +sharing are mutually exclusive in both directions — a full-screen call covers +the screen, so sharing it would show the call itself. -- **Screen share** (`MonitorUp` buttons on the PiP overlay and the meeting - control bar): captures the primary screen; frames go to the model as a - separately labeled group. In the meeting view the screen becomes the big - tile with user + mascot in a side rail. -- **Camera off** (Meet-style mute): video mode and screen share keep running, - no webcam frames are captured; tiles show a silhouette avatar. -- **Mascot dismissal** (meeting view): swaps the animated mascot for a - Meet-style letter avatar ("R"). -- **Popout**: starting a screen share immediately pops the mini-call out into - a small always-on-top frameless window (user + mascot tiles, live caption, - control bar) that floats over every app — including Rowboat itself, since - the shared screen may be Rowboat. It stays for the whole share (surviving - app switches) and disappears when sharing stops or video mode ends. While - sharing it is THE call surface: the in-app PiP hides, and presenting from - the full-screen meeting collapses the meeting into it (restored when the - share ends). Control-bar actions (camera toggle / stop share / end call) - round-trip `video:popoutAction` → main → `video:popout-action` → app - window, which owns the mic/camera/capture. The expand button focuses the - main window (`video:focusMain`). +- sharing → floating popout, always (pill = working) +- not sharing → full screen unless `callMinimized` (full screen = facing + each other) +- expanding the pill auto-STOPS any share; minimizing the full-screen call + auto-STARTS one (the pill exists to work together) — presenting from full + screen likewise collapses to the pill +- the camera toggle never changes the surface: turning it on from the pill + puts your video IN the pill; expanding is its own explicit action -`call`/`meeting` options are disabled unless both voice input (Deepgram) and -voice output (TTS) are configured. Entering a call saves the user's TTS -settings and forces `full` read-aloud; leaving restores them. +**Screen-share consent** is three-layered: a toast the moment any share +starts ("Your screen is being shared… [Stop sharing]"), a persistent +"Sharing screen" badge on the pill, and macOS's purple recording indicator. +If the auto-share fails (Screen Recording permission not granted) the call +starts anyway as a voice call, with a toast linking to System Settings. +Practice/coaching is always an explicit choice — expanding to full screen +never turns the coach on. + +In-call controls (identical bar on both surfaces): camera toggle (silhouette +avatar while off, no webcam frames captured), screen share toggle, mascot ⇄ +"R" letter avatar, end call. Captions of the in-progress utterance and the +assistant's spoken line run along the bottom. Typing in the composer still +works mid-call; frames ride along with typed messages too. + +Outside calls the composer keeps exactly two voice affordances: the **mic +button** (push-to-talk dictation, untouched) and the **read-aloud toggle** +(headphones; summary-style for typed chat — calls force full read-aloud and +restore the prior setting on hang-up). The old video dropdown, talking-head +toggle, and summary/full TTS dropdown are retired. + +The call button is disabled unless both voice input (Deepgram) and voice +output (TTS) are configured. `call_started` (with `preset`) is captured in +PostHog — the adoption metric for this feature. + +**Popout mechanics**: a small always-on-top frameless window (camera tile +when on + mascot tile, live caption, control bar) floating over every app — +including Rowboat. Control-bar actions round-trip `video:popoutAction` → +main → `video:popout-action` → app window, which owns the mic/camera/capture; +`expand` also refocuses the app window (handled in main). ## Frame pipeline @@ -56,11 +80,13 @@ settings and forces `full` read-aloud; leaving restores them. at the moment of send. Falls back to the single latest frame for rapid-fire messages. -`App.tsx` `handlePromptSubmit` (~line 2767) attaches the drained frames to -the outgoing message as `UserImagePart`s and sets -`composition.videoMode: true`. Frames also become `isVideoFrame` display -attachments (filmstrip in the transcript — `chat-message-attachments.tsx`; -history hydration in `lib/run-to-conversation.ts`). +`App.tsx` `handlePromptSubmit` attaches the drained frames (whenever a call +is live) to the outgoing message as `UserImagePart`s and sets +`composition.videoMode` when the camera or screen is active, plus +`composition.coachMode` during a practice session. Frames also become +`isVideoFrame` display attachments (filmstrip in the transcript — +`chat-message-attachments.tsx`; history hydration in +`lib/run-to-conversation.ts`). ## Message schema & model encoding @@ -95,27 +121,27 @@ history hydration in `lib/run-to-conversation.ts`). - Mid-call socket drops reconnect after 1s; the offline audio backlog is capped (~30s). -Mode transitions live in `App.tsx` `handleVideoModeChange` (~line 1161): -call ↔ meeting switches are presentation-only (mic/TTS untouched); -entering/leaving hands-free saves/restores TTS settings. Push-to-talk is -disabled while a call owns the mic. +Call lifecycle lives in `App.tsx` `startCall(preset)` / `endCall()`: +entering a call saves/forces TTS settings, cancels any push-to-talk +recording, and starts the continuous loop; ending restores everything. +Push-to-talk is disabled while a call owns the mic. ## Popout window -- Shown iff `videoChatMode !== 'off' && screenState === 'live'` (effect in - `App.tsx`). Renderer asks `video:setPopout {show}` (main handler: - `apps/main/src/ipc.ts:1742`); main creates a frameless, `alwaysOnTop` - ('floating'), all-workspaces BrowserWindow at the top-right of the primary - display, loading the renderer bundle with `#video-popout` +- Shown iff the derived `callSurface === 'popout'` (effect in `App.tsx`). + Renderer asks `video:setPopout {show}`; main creates a frameless, + `alwaysOnTop` ('floating'), all-workspaces BrowserWindow at the top-right + of the primary display, loading the renderer bundle with `#video-popout` (`apps/renderer/src/main.tsx` branches on the hash → `components/video-popout.tsx`). - Call state streams over the `video:popout-state` push channel; main caches the last payload and replays it on popout load. Shown with - `showInactive()` so it never steals focus (that would re-hide it). + `showInactive()` so it never steals focus. - The popout captures its **own** camera preview (MediaStreams can't cross windows) and synthesizes the mascot mouth level (no audio in that window). -- `video:focusMain` matches only real app windows by URL — `getAllWindows()` - also contains hidden utility windows (PDF export) that must not be shown. +- `video:popoutAction` relays control-bar actions to the app window, matched + only by real app-window URLs — `getAllWindows()` also contains hidden + utility windows (PDF export) that must not be shown or messaged. ## Permissions @@ -132,8 +158,9 @@ disabled while a call owns the mic. | Prompt | Where | |--------|-------| | `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) | -| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts:~1013` (`convertFromMessages`) | -| `videoMode` composition override (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts:57,125`; set from `App.tsx` `sendConfig` | +| `# Practice Session (Coach Mode)` system section — coaching persona: specific/actionable feedback after each take, one-sentence interjections mid-flow, structured debrief on wrap-up | `composeSystemInstructions`, gated on `coachMode` (directly after the video section) | +| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts` (`convertFromMessages`) | +| `videoMode` / `coachMode` composition overrides (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts` (`CompositionOverrides`); set from `App.tsx` `sendConfig` | Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are reused untouched — calls set `voiceInput` per utterance and force @@ -156,4 +183,7 @@ screen frame per message unless the screen changed. "energy" coaching leans on visual cues. - Screen share always captures the primary display (no window/display picker yet). -- The meeting view covers the chat; there's no in-call transcript drawer. +- The full-screen call covers the chat; there's no in-call transcript drawer. +- The "attach camera frames to typed chat without a call" combination (the + old video+chat mode) was cut in the call-model simplification; if analytics + show demand, it should return as an attachment chip, not a mode. diff --git a/apps/x/apps/main/src/ipc.ts b/apps/x/apps/main/src/ipc.ts index 846fa7fa..9df109e7 100644 --- a/apps/x/apps/main/src/ipc.ts +++ b/apps/x/apps/main/src/ipc.ts @@ -390,6 +390,7 @@ let lastVideoPopoutState: { ttsState: 'idle' | 'synthesizing' | 'speaking'; status: 'listening' | 'thinking' | 'speaking' | null; cameraOn: boolean; + screenSharing: boolean; interimText: string | null; } | null = null; @@ -1816,19 +1817,20 @@ export function setupIpcHandlers() { } return {}; }, - 'video:focusMain': async () => { + 'video:getPopoutState': async () => { + return { state: lastVideoPopoutState }; + }, + 'video:popoutAction': async (_event, args) => { + // Relay a popout control-bar action to the app window, which owns the + // call (mic, camera, screen capture) and executes it there. 'expand' + // additionally brings the app window back to the foreground. const main = findMainAppWindow(); - if (main) { + if (args.action === 'expand' && main) { if (main.isMinimized()) main.restore(); main.show(); main.focus(); } - return {}; - }, - 'video:popoutAction': async (_event, args) => { - // Relay a popout control-bar action to the app window, which owns the - // call (mic, camera, screen capture) and executes it there. - findMainAppWindow()?.webContents.send('video:popout-action', args); + main?.webContents.send('video:popout-action', args); return {}; }, // Live-note handlers diff --git a/apps/x/apps/renderer/src/App.tsx b/apps/x/apps/renderer/src/App.tsx index cfd82a1d..7c0dbafc 100644 --- a/apps/x/apps/renderer/src/App.tsx +++ b/apps/x/apps/renderer/src/App.tsx @@ -12,7 +12,7 @@ import { ChatSidebar } from './components/chat-sidebar'; import { useSessionChat } from '@/hooks/useSessionChat'; import { ChatHeader } from './components/chat-header'; import { ChatEmptyState } from './components/chat-empty-state'; -import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type VideoChatMode } from './components/chat-input-with-mentions'; +import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment } from './components/chat-input-with-mentions'; import { ChatMessageAttachments } from '@/components/chat-message-attachments' import { GraphView, type GraphEdge, type GraphNode } from '@/components/graph-view'; import { BasesView, type BaseConfig, DEFAULT_BASE_CONFIG } from '@/components/bases-view'; @@ -120,8 +120,6 @@ import { toast } from "sonner" import { useVoiceMode } from '@/hooks/useVoiceMode' import { useVideoMode } from '@/hooks/useVideoMode' import { useVoiceTTS } from '@/hooks/useVoiceTTS' -import { TalkingHeadOverlay } from '@/components/talking-head' -import { VideoPreviewOverlay } from '@/components/video-preview-overlay' import { VideoCallView } from '@/components/video-call-view' import { ProductTour, type TourNavTarget } from '@/components/product-tour' import { useMeetingTranscription, type CalendarEventMeta } from '@/hooks/useMeetingTranscription' @@ -974,9 +972,9 @@ function App() { const [ttsAvailable, setTtsAvailable] = useState(false) const [ttsEnabled, setTtsEnabled] = useState(false) const ttsEnabledRef = useRef(false) - const [ttsMode, setTtsMode] = useState<'summary' | 'full'>('summary') + // Read-aloud style: 'summary' for typed chat, forced to 'full' during a + // call and restored after. Context decides — the user never picks it. const ttsModeRef = useRef<'summary' | 'full'>('summary') - const [ttsAvatarEnabled, setTtsAvatarEnabled] = useState(false) const [tourActive, setTourActive] = useState(false) const [isRecording, setIsRecording] = useState(false) const voiceTextBufferRef = useRef('') @@ -1021,16 +1019,20 @@ function App() { const voiceRef = useRef(voice) voiceRef.current = voice - // Video chat mode: while on, the webcam runs and frames are attached to - // every outgoing message so the assistant can see the user. 'chat' keeps - // the normal composer; 'call' is fully hands-free — continuous listening, - // auto-submitted utterances, spoken responses (handler defined below, after - // the voice/submit plumbing it drives). + // Calls: one engine (hands-free voice loop + forced read-aloud TTS + frame + // capture), started via presets that only differ in device defaults. The + // presentation is DERIVED from devices, never picked: screen sharing → + // floating popout; camera on → full-screen call; camera off → popout + // (mascot pill). Handlers live below the voice/submit plumbing they drive. const video = useVideoMode() - const [videoChatMode, setVideoChatMode] = useState('off') - const videoChatModeRef = useRef('off') - // TTS settings to restore when a hands-free call ends (a call forces - // full-read-aloud TTS for its duration). + const [inCall, setInCall] = useState(false) + const inCallRef = useRef(false) + // User explicitly shrank the full-screen call to the floating pill. + const [callMinimized, setCallMinimized] = useState(false) + // Practice preset: adds the coaching persona to the system prompt. + const [practiceMode, setPracticeMode] = useState(false) + const practiceModeRef = useRef(false) + // TTS settings to restore when a call ends (a call forces full read-aloud). const preCallTtsRef = useRef<{ enabled: boolean; mode: 'summary' | 'full' } | null>(null) const handleToggleMeetingRef = useRef<(() => void) | undefined>(undefined) @@ -1091,8 +1093,8 @@ function App() { }, []) const handleStartRecording = useCallback(() => { - // Hands-free call mode owns the mic — ignore push-to-talk while it's on. - if (videoChatModeRef.current === 'call' || videoChatModeRef.current === 'meeting') return + // A live call owns the mic — ignore push-to-talk while one is running. + if (inCallRef.current) return setIsRecording(true) isRecordingRef.current = true voice.start() @@ -1123,143 +1125,128 @@ function App() { ttsEnabledRef.current = next if (!next) { ttsRef.current.cancel() - setTtsAvatarEnabled(false) } return next }) }, []) - // Talking-head mode implies voice output: enabling it turns TTS on, - // disabling it turns both off. - const handleToggleTtsAvatar = useCallback(() => { - setTtsAvatarEnabled(prev => { - const next = !prev - setTtsEnabled(next) - ttsEnabledRef.current = next - if (!next) { - ttsRef.current.cancel() - } - return next - }) - }, []) - - const handleTtsModeChange = useCallback((mode: 'summary' | 'full') => { - setTtsMode(mode) - ttsModeRef.current = mode - }, []) - const handleCancelRecording = useCallback(() => { voice.cancel() setIsRecording(false) isRecordingRef.current = false }, [voice]) - // Video chat mode transitions. 'chat' just runs the camera; 'call' and - // 'meeting' also listen continuously (auto-submitting each utterance as a - // voice message) and force full read-aloud TTS so the assistant answers out - // loud. 'meeting' is the same pipeline presented as a full-screen call. - const handleVideoModeChange = useCallback(async (mode: VideoChatMode) => { - const prev = videoChatModeRef.current - if (mode === prev) return - const wasHandsFree = prev === 'call' || prev === 'meeting' - const isHandsFree = mode === 'call' || mode === 'meeting' + // Start a call. Presets only differ in device defaults — the engine + // (continuous listening, auto-submitted utterances, forced read-aloud TTS, + // frame capture) is identical for all of them. The default entry ('share', + // the call button's main click) is "work together": screen shared, camera + // off, floating pill — the user keeps working while the assistant watches + // along. 'video'/'practice' open face-to-face full screen instead. + const startCall = useCallback(async (preset: CallPreset) => { + if (inCallRef.current) return + const camera = preset === 'video' || preset === 'practice' + const ok = await video.start({ camera }) + if (!ok) return // camera denied/unavailable — stay out of the call + if (preset === 'share') { + // If screen capture fails (usually the macOS Screen Recording + // permission), continue as a voice call — sharing is one tap away on + // the pill once permission is granted. + const shared = await video.startScreenShare() + if (!shared) { + toast("Couldn't share your screen", { + description: 'Grant Rowboat Screen Recording access, then tap the share button on the call.', + action: { + label: 'Open Settings', + onClick: () => void window.ipc.invoke('meeting:openScreenRecordingSettings', null).catch(() => {}), + }, + }) + } + } - // Leaving hands-free: release the mic and restore the pre-call TTS settings. - if (wasHandsFree && !isHandsFree) { + // A manual push-to-talk recording can't coexist with the call's mic. + if (isRecordingRef.current) { voiceRef.current.cancel() - const saved = preCallTtsRef.current - preCallTtsRef.current = null - const restoreEnabled = saved?.enabled ?? false - setTtsEnabled(restoreEnabled) - ttsEnabledRef.current = restoreEnabled - if (saved) { - setTtsMode(saved.mode) - ttsModeRef.current = saved.mode - } - if (!restoreEnabled) ttsRef.current.cancel() + setIsRecording(false) + isRecordingRef.current = false } + preCallTtsRef.current = { enabled: ttsEnabledRef.current, mode: ttsModeRef.current } + setTtsEnabled(true) + ttsEnabledRef.current = true + ttsModeRef.current = 'full' + void voiceRef.current.startContinuous((text) => { + pendingVoiceInputRef.current = true + handlePromptSubmitRef.current?.({ text, files: [] }) + }) - if (mode === 'off') { - video.stop() - videoChatModeRef.current = 'off' - setVideoChatMode('off') - return + setPracticeMode(preset === 'practice') + practiceModeRef.current = preset === 'practice' + // Pill-first presets start minimized; face-to-face presets start expanded. + setCallMinimized(preset === 'voice' || preset === 'share') + inCallRef.current = true + setInCall(true) + analytics.callStarted(preset) + }, [video]) + + const endCall = useCallback(() => { + if (!inCallRef.current) return + voiceRef.current.cancel() + const saved = preCallTtsRef.current + preCallTtsRef.current = null + const restoreEnabled = saved?.enabled ?? false + setTtsEnabled(restoreEnabled) + ttsEnabledRef.current = restoreEnabled + if (saved) { + ttsModeRef.current = saved.mode } - - if (prev === 'off') { - const ok = await video.start() - if (!ok) return // camera denied/unavailable — stay off - } - - // Entering hands-free (call ↔ meeting switches keep the mic/TTS as-is). - if (isHandsFree && !wasHandsFree) { - // A manual push-to-talk recording can't coexist with the call's mic. - if (isRecordingRef.current) { - voiceRef.current.cancel() - setIsRecording(false) - isRecordingRef.current = false - } - preCallTtsRef.current = { enabled: ttsEnabledRef.current, mode: ttsModeRef.current } - setTtsEnabled(true) - ttsEnabledRef.current = true - setTtsMode('full') - ttsModeRef.current = 'full' - void voiceRef.current.startContinuous((text) => { - pendingVoiceInputRef.current = true - handlePromptSubmitRef.current?.({ text, files: [] }) - }) - } - - videoChatModeRef.current = mode - setVideoChatMode(mode) + if (!restoreEnabled) ttsRef.current.cancel() + video.stop() + setPracticeMode(false) + practiceModeRef.current = false + setCallMinimized(false) + inCallRef.current = false + setInCall(false) }, [video]) // During a call, mute the mic while the assistant is thinking or speaking // so its own TTS (or a half-turn) never gets transcribed back at it. useEffect(() => { - if (videoChatMode !== 'call' && videoChatMode !== 'meeting') return + if (!inCall) return voiceRef.current.setPaused(activeIsProcessing || tts.state !== 'idle') - }, [videoChatMode, activeIsProcessing, tts.state]) + }, [inCall, activeIsProcessing, tts.state]) - // Presenting from the full-screen call collapses it into the floating - // popout (the user needs their screen — possibly Rowboat itself — free to - // navigate while sharing). When the share ends, return to full screen. - const returnToMeetingAfterShareRef = useRef(false) - - // Screen sharing (any video mode): frames of the shared screen ride along - // with each message next to the webcam frames. + // Screen sharing: frames of the shared screen ride along with each message + // next to the webcam frames. The surface change (full screen → pill) falls + // out of the derivation below. const handleToggleScreenShare = useCallback(async () => { if (video.screenState === 'live') { video.stopScreenShare() } else { - const ok = await video.startScreenShare() - if (ok && videoChatModeRef.current === 'meeting') { - returnToMeetingAfterShareRef.current = true - void handleVideoModeChange('call') - } + await video.startScreenShare() } - }, [video, handleVideoModeChange]) + }, [video]) - // Share ended (from the popout, the PiP, or the OS): restore the - // full-screen call if that's where the share was started from. - useEffect(() => { - if (video.screenState === 'live') return - if (!returnToMeetingAfterShareRef.current) return - returnToMeetingAfterShareRef.current = false - if (videoChatModeRef.current === 'call') { - void handleVideoModeChange('meeting') - } - }, [video.screenState, handleVideoModeChange]) - - // Meet-style camera mute: video mode (and any screen share) stays on, but - // no webcam frames are captured while the camera is off. + // Meet-style camera mute: the call (and any screen share) stays on, but no + // webcam frames are captured while the camera is off. Deliberately does NOT + // change the surface — turning your camera on from the pill puts your video + // IN the pill; expanding to full screen is its own explicit action. const handleToggleCamera = useCallback(() => { void video.setCameraEnabled(!video.cameraOn) }, [video]) - // Current phase of a hands-free call (null outside call/meeting modes). + // Minimizing the full-screen call drops you back to working — and the pill + // exists to work *together*, so sharing starts automatically (the symmetric + // twin of expand, which stops it). If capture fails (permission), the call + // still minimizes as a plain pill. `callMinimized` is also set so stopping + // the share from the pill keeps you in the pill rather than snapping back + // to full screen. + const handleMinimizeCall = useCallback(async () => { + setCallMinimized(true) + await video.startScreenShare() + }, [video]) + + // Current phase of the call (null when not in one). const videoCallStatus: 'listening' | 'thinking' | 'speaking' | null = - videoChatMode === 'call' || videoChatMode === 'meeting' + inCall ? tts.state === 'speaking' ? 'speaking' : tts.state === 'synthesizing' || activeIsProcessing @@ -1267,39 +1254,70 @@ function App() { : 'listening' : null - // Meet-style popout: the floating always-on-top mini-call appears the - // moment screen sharing starts and stays for the whole share — including - // over Rowboat itself (the shared screen may BE Rowboat), and across app - // switches. It disappears when sharing stops or video mode ends. - useEffect(() => { - const shouldShow = videoChatMode !== 'off' && video.screenState === 'live' - void window.ipc.invoke('video:setPopout', { show: shouldShow }).catch(() => {}) - }, [videoChatMode, video.screenState]) + // The call's surface follows one rule: full screen and screen sharing are + // mutually exclusive (a full-screen call covers the screen — sharing it + // would show the call itself). Sharing → floating pill, always. Not + // sharing → full screen unless the user shrank it (`callMinimized`). + // Expanding the pill auto-stops any share; presenting from full screen + // auto-collapses to the pill. + const callSurface: 'fullscreen' | 'popout' | null = !inCall + ? null + : video.screenState === 'live' || callMinimized + ? 'popout' + : 'fullscreen' - // Keep the popout's mascot/status/camera/caption mirror of the call fresh. + useEffect(() => { + void window.ipc.invoke('video:setPopout', { show: callSurface === 'popout' }).catch(() => {}) + }, [callSurface]) + + // Consent surface for screen sharing: an unmissable toast the moment any + // share starts (auto-started calls included), with one-tap stop. The pill + // also carries a persistent "Sharing screen" badge, and macOS shows its + // purple recording indicator. + const prevScreenStateRef = useRef(video.screenState) + useEffect(() => { + const prev = prevScreenStateRef.current + prevScreenStateRef.current = video.screenState + if (video.screenState === 'live' && prev !== 'live') { + toast('Your screen is being shared', { + description: 'The assistant sees snapshots of it along with what you say.', + action: { label: 'Stop sharing', onClick: () => video.stopScreenShare() }, + duration: 6000, + }) + } + }, [video.screenState, video]) + + // Keep the popout's mascot/status/devices/caption mirror of the call fresh. // The main process caches the latest state and replays it when the popout // loads. useEffect(() => { - if (videoChatMode === 'off') return + if (!inCall) return void window.ipc .invoke('video:popoutState', { ttsState: tts.state, status: videoCallStatus, cameraOn: video.cameraOn, - interimText: videoCallStatus ? voice.interimText || null : null, + screenSharing: video.screenState === 'live', + interimText: voice.interimText || null, }) .catch(() => {}) - }, [videoChatMode, tts.state, videoCallStatus, video.cameraOn, voice.interimText]) + }, [inCall, tts.state, videoCallStatus, video.cameraOn, video.screenState, voice.interimText]) // Execute popout control-bar actions (the popout window has no access to - // the call's mic/camera/capture — they live here). + // the call's mic/camera/capture — they live here). 'expand' goes full + // screen, which by the exclusivity rule stops any running share; the main + // process already refocused the app window. useEffect(() => { return window.ipc.on('video:popout-action', ({ action }) => { if (action === 'toggle-camera') handleToggleCamera() - else if (action === 'stop-share') video.stopScreenShare() - else if (action === 'end-call') void handleVideoModeChange('off') + else if (action === 'toggle-share') void handleToggleScreenShare() + else if (action === 'end-call') endCall() + else if (action === 'expand') { + if (video.screenState === 'live') video.stopScreenShare() + setCallMinimized(false) + } }) - }, [handleToggleCamera, video, handleVideoModeChange]) + }, [handleToggleCamera, handleToggleScreenShare, endCall, video]) // Enter to submit voice input, Escape to cancel useEffect(() => { @@ -2783,7 +2801,7 @@ function App() { // Video chat mode: drain the webcam frames buffered since the last send // so they ride along with this message as inline image parts. - const videoFrames = videoChatModeRef.current !== 'off' ? video.collectFrames() : [] + const videoFrames = inCallRef.current ? video.collectFrames() : [] const userMessageId = `user-${Date.now()}` const displayAttachments: ChatMessage['attachments'] = hasAttachments || videoFrames.length > 0 @@ -2854,7 +2872,8 @@ function App() { ...(ttsEnabledRef.current ? { voiceOutput: ttsModeRef.current } : {}), ...(searchEnabled ? { searchEnabled: true } : {}), ...(codeMode ? { codeMode } : {}), - ...(videoChatModeRef.current !== 'off' ? { videoMode: true } : {}), + ...(inCallRef.current && (video.cameraOn || video.screenState === 'live') ? { videoMode: true } : {}), + ...(practiceModeRef.current ? { coachMode: true } : {}), }, }, }, @@ -6536,14 +6555,11 @@ function App() { voiceAvailable={isActive && voiceAvailable} ttsAvailable={isActive && ttsAvailable} ttsEnabled={ttsEnabled} - ttsMode={ttsMode} onToggleTts={isActive ? handleToggleTts : undefined} - onTtsModeChange={isActive ? handleTtsModeChange : undefined} - ttsAvatarEnabled={ttsAvatarEnabled} - onToggleTtsAvatar={isActive ? handleToggleTtsAvatar : undefined} - videoChatMode={videoChatMode} - onVideoModeChange={isActive ? handleVideoModeChange : undefined} - videoCallAvailable={voiceAvailable && ttsAvailable} + inCall={inCall} + onStartCall={isActive ? startCall : undefined} + onEndCall={isActive ? endCall : undefined} + callAvailable={voiceAvailable && ttsAvailable} /> ) @@ -6652,69 +6668,32 @@ function App() { voiceAvailable={voiceAvailable} ttsAvailable={ttsAvailable} ttsEnabled={ttsEnabled} - ttsMode={ttsMode} onToggleTts={handleToggleTts} - onTtsModeChange={handleTtsModeChange} - ttsAvatarEnabled={ttsAvatarEnabled} - onToggleTtsAvatar={handleToggleTtsAvatar} - videoChatMode={videoChatMode} - onVideoModeChange={handleVideoModeChange} - videoCallAvailable={voiceAvailable && ttsAvailable} + inCall={inCall} + onStartCall={startCall} + onEndCall={endCall} + callAvailable={voiceAvailable && ttsAvailable} onComposioConnected={handleComposioConnected} /> )} - {/* Webcam PiP preview while video chat mode is on. Hidden while - screen sharing (the floating popout is the call surface then) - and in the full-screen meeting view (which replaces it). */} - {(videoChatMode === 'chat' || videoChatMode === 'call') && video.screenState !== 'live' && ( - handleVideoModeChange('off')} - callStatus={ - videoChatMode === 'call' - ? tts.state !== 'idle' - ? 'speaking' - : activeIsProcessing - ? 'thinking' - : 'listening' - : undefined - } - interimText={videoChatMode === 'call' ? voice.interimText : undefined} - onToggleScreenShare={handleToggleScreenShare} - cameraOn={video.cameraOn} - onToggleCamera={handleToggleCamera} - /> - )} - {/* Full-screen Meet-style call: user tile + animated mascot tile */} - {videoChatMode === 'meeting' && ( + {/* Full-screen call: user tile + animated mascot tile. Shown only + when the derived surface says so (camera on, no screen share, + not minimized) — otherwise the call lives in the floating + popout window. */} + {callSurface === 'fullscreen' && ( void handleMinimizeCall()} ttsState={tts.state} getTtsLevel={tts.getLevel} - status={ - tts.state === 'speaking' - ? 'speaking' - : tts.state === 'synthesizing' || activeIsProcessing - ? 'thinking' - : 'listening' - } + status={videoCallStatus ?? 'listening'} interimText={voice.interimText} assistantCaption={assistantCaption} - onLeave={() => handleVideoModeChange('off')} - /> - )} - {/* Talking head hovers over the active view while avatar voice mode is - on (hidden during the tour, which shows its own mascot) */} - {ttsAvatarEnabled && !tourActive && ( - )} {/* Mascot-guided product tour */} diff --git a/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx b/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx index a97365d5..1aeaa3b7 100644 --- a/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx +++ b/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx @@ -21,7 +21,10 @@ import { Lock, Mic, MoreHorizontal, + Phone, + PhoneOff, Plus, + Presentation, ShieldCheck, Square, Terminal, @@ -51,7 +54,6 @@ import { } from '@/lib/attachment-presentation' import { getExtension, getFileDisplayName, getMimeFromExtension, isImageMime } from '@/lib/file-utils' import { cn } from '@/lib/utils' -import { MascotFaceIcon } from '@/components/talking-head' import { type FileMention, type PromptInputMessage, @@ -212,7 +214,17 @@ function compactWorkDirPath(path: string) { return path.replace(/^\/Users\/[^/]+/, '~') } -export type VideoChatMode = 'off' | 'chat' | 'call' | 'meeting' +// Call presets: front doors into the same call engine, differing only in +// starting devices. 'share' is the call button's main click — the "work +// together" default (screen shared, camera off, floating pill). The chevron +// menu holds the deviations. +export type CallPreset = 'voice' | 'video' | 'share' | 'practice' + +const CALL_PRESET_MENU: Array<{ preset: CallPreset; label: string; description: string; Icon: typeof Phone }> = [ + { preset: 'voice', label: 'Voice call', description: 'Just talk — nothing is shared, the mascot hovers while you work', Icon: AudioLines }, + { preset: 'video', label: 'Video call', description: 'Camera on, face to face — it sees your expressions', Icon: Video }, + { preset: 'practice', label: 'Practice session', description: 'Rehearse a pitch or interview with live coaching', Icon: Presentation }, +] interface ChatInputInnerProps { onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void @@ -236,18 +248,14 @@ interface ChatInputInnerProps { voiceAvailable?: boolean ttsAvailable?: boolean ttsEnabled?: boolean - ttsMode?: 'summary' | 'full' onToggleTts?: () => void - onTtsModeChange?: (mode: 'summary' | 'full') => void - ttsAvatarEnabled?: boolean - onToggleTtsAvatar?: () => void - /** Video chat mode: 'chat' attaches webcam frames to messages; 'call' and - * 'meeting' are fully hands-free — continuous listening, spoken responses - * ('meeting' additionally takes over the whole screen, Meet-style). */ - videoChatMode?: VideoChatMode - onVideoModeChange?: (mode: VideoChatMode) => void - /** Hands-free call needs both voice input (STT) and voice output (TTS). */ - videoCallAvailable?: boolean + /** A call is live (hands-free voice loop + spoken responses). */ + inCall?: boolean + /** Start a call with the given preset's device defaults. */ + onStartCall?: (preset: CallPreset) => void + onEndCall?: () => void + /** Calls need both voice input (STT) and voice output (TTS) configured. */ + callAvailable?: boolean /** Fired when the user picks a different model in the dropdown (only when no run exists yet). */ onSelectedModelChange?: (model: SelectedModel | null) => void /** Work directory for this chat (per-chat). Null when none is set. */ @@ -283,14 +291,11 @@ function ChatInputInner({ voiceAvailable, ttsAvailable, ttsEnabled, - ttsMode, onToggleTts, - onTtsModeChange, - ttsAvatarEnabled, - onToggleTtsAvatar, - videoChatMode = 'off', - onVideoModeChange, - videoCallAvailable, + inCall, + onStartCall, + onEndCall, + callAvailable, onSelectedModelChange, workDir = null, onWorkDirChange, @@ -1290,68 +1295,21 @@ function ChatInputInner({ ) : null} {onToggleTts && ttsAvailable && ( -
- - - - - - {ttsEnabled ? 'Voice output on' : 'Voice output off'} - - - {ttsEnabled && onTtsModeChange && ( - - - - - - onTtsModeChange(v as 'summary' | 'full')}> - Speak summary - Speak full response - - - - )} -
- )} - {onToggleTtsAvatar && ttsAvailable && ( - {ttsAvatarEnabled ? 'Talking head on' : 'Talking head off'} + {ttsEnabled ? 'Read responses aloud: on' : 'Read responses aloud: off'} )} - {onVideoModeChange && ( + {onStartCall && (
- {videoChatMode === 'off' - ? 'Video chat off' - : videoChatMode === 'chat' - ? 'Video chat on — camera frames are shared with the assistant' - : 'Video call on — speak freely, the assistant answers out loud'} + {inCall + ? 'End call' + : callAvailable + ? 'Start a call — it sees your screen while you talk it through' + : 'Calls need voice input and output configured'} - {videoChatMode !== 'off' && ( + {!inCall && ( - - onVideoModeChange(v as VideoChatMode)} - > - Video + chat - - Video call (hands-free) - - - Video call (full screen) - - + + {CALL_PRESET_MENU.map(({ preset, label, description, Icon }) => ( + onStartCall(preset)} + className="items-start gap-3 py-2" + > + + + {label} + {description} + + + ))} )} @@ -1593,14 +1557,11 @@ export interface ChatInputWithMentionsProps { voiceAvailable?: boolean ttsAvailable?: boolean ttsEnabled?: boolean - ttsMode?: 'summary' | 'full' onToggleTts?: () => void - onTtsModeChange?: (mode: 'summary' | 'full') => void - ttsAvatarEnabled?: boolean - onToggleTtsAvatar?: () => void - videoChatMode?: VideoChatMode - onVideoModeChange?: (mode: VideoChatMode) => void - videoCallAvailable?: boolean + inCall?: boolean + onStartCall?: (preset: CallPreset) => void + onEndCall?: () => void + callAvailable?: boolean onSelectedModelChange?: (model: SelectedModel | null) => void workDir?: string | null onWorkDirChange?: (value: string | null) => void @@ -1632,14 +1593,11 @@ export function ChatInputWithMentions({ voiceAvailable, ttsAvailable, ttsEnabled, - ttsMode, onToggleTts, - onTtsModeChange, - ttsAvatarEnabled, - onToggleTtsAvatar, - videoChatMode, - onVideoModeChange, - videoCallAvailable, + inCall, + onStartCall, + onEndCall, + callAvailable, onSelectedModelChange, workDir, onWorkDirChange, @@ -1668,14 +1626,11 @@ export function ChatInputWithMentions({ voiceAvailable={voiceAvailable} ttsAvailable={ttsAvailable} ttsEnabled={ttsEnabled} - ttsMode={ttsMode} onToggleTts={onToggleTts} - onTtsModeChange={onTtsModeChange} - ttsAvatarEnabled={ttsAvatarEnabled} - onToggleTtsAvatar={onToggleTtsAvatar} - videoChatMode={videoChatMode} - onVideoModeChange={onVideoModeChange} - videoCallAvailable={videoCallAvailable} + inCall={inCall} + onStartCall={onStartCall} + onEndCall={onEndCall} + callAvailable={callAvailable} onSelectedModelChange={onSelectedModelChange} workDir={workDir} onWorkDirChange={onWorkDirChange} diff --git a/apps/x/apps/renderer/src/components/chat-sidebar.tsx b/apps/x/apps/renderer/src/components/chat-sidebar.tsx index c32bac18..26ef078a 100644 --- a/apps/x/apps/renderer/src/components/chat-sidebar.tsx +++ b/apps/x/apps/renderer/src/components/chat-sidebar.tsx @@ -37,7 +37,7 @@ import { MarkdownPreOverride } from '@/components/ai-elements/markdown-code-over import { defaultRemarkPlugins } from 'streamdown' import remarkBreaks from 'remark-breaks' import { type ChatTab } from '@/components/tab-bar' -import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type SelectedModel, type VideoChatMode } from '@/components/chat-input-with-mentions' +import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions' import { ChatMessageAttachments } from '@/components/chat-message-attachments' import { useSidebar } from '@/components/ui/sidebar' import { wikiLabel } from '@/lib/wiki-links' @@ -189,14 +189,11 @@ interface ChatSidebarProps { voiceAvailable?: boolean ttsAvailable?: boolean ttsEnabled?: boolean - ttsMode?: 'summary' | 'full' onToggleTts?: () => void - onTtsModeChange?: (mode: 'summary' | 'full') => void - ttsAvatarEnabled?: boolean - onToggleTtsAvatar?: () => void - videoChatMode?: VideoChatMode - onVideoModeChange?: (mode: VideoChatMode) => void - videoCallAvailable?: boolean + inCall?: boolean + onStartCall?: (preset: CallPreset) => void + onEndCall?: () => void + callAvailable?: boolean onComposioConnected?: (toolkitSlug: string) => void } @@ -258,14 +255,11 @@ export function ChatSidebar({ voiceAvailable, ttsAvailable, ttsEnabled, - ttsMode, onToggleTts, - onTtsModeChange, - ttsAvatarEnabled, - onToggleTtsAvatar, - videoChatMode, - onVideoModeChange, - videoCallAvailable, + inCall, + onStartCall, + onEndCall, + callAvailable, onComposioConnected, }: ChatSidebarProps) { const { state: sidebarState } = useSidebar() @@ -837,14 +831,11 @@ export function ChatSidebar({ voiceAvailable={isActive && voiceAvailable} ttsAvailable={isActive && ttsAvailable} ttsEnabled={ttsEnabled} - ttsMode={ttsMode} onToggleTts={isActive ? onToggleTts : undefined} - onTtsModeChange={isActive ? onTtsModeChange : undefined} - ttsAvatarEnabled={ttsAvatarEnabled} - onToggleTtsAvatar={isActive ? onToggleTtsAvatar : undefined} - videoChatMode={videoChatMode} - onVideoModeChange={isActive ? onVideoModeChange : undefined} - videoCallAvailable={videoCallAvailable} + inCall={inCall} + onStartCall={isActive ? onStartCall : undefined} + onEndCall={isActive ? onEndCall : undefined} + callAvailable={callAvailable} />
) diff --git a/apps/x/apps/renderer/src/components/video-call-view.tsx b/apps/x/apps/renderer/src/components/video-call-view.tsx index 91a12c4a..8f95ce8c 100644 --- a/apps/x/apps/renderer/src/components/video-call-view.tsx +++ b/apps/x/apps/renderer/src/components/video-call-view.tsx @@ -1,5 +1,5 @@ import { useEffect, useRef, useState } from 'react' -import { MonitorUp, PhoneOff, User, Video, VideoOff } from 'lucide-react' +import { Minimize2, MonitorUp, PhoneOff, Presentation, User, Video, VideoOff } from 'lucide-react' import { MascotFaceIcon, TalkingHead } from '@/components/talking-head' import type { TTSState } from '@/hooks/useVoiceTTS' @@ -10,12 +10,15 @@ export type VideoCallStatus = 'listening' | 'thinking' | 'speaking' interface VideoCallViewProps { /** Live camera stream from useVideoMode — attached to the user's tile. */ streamRef: React.MutableRefObject - /** Live screen-share stream — shown as the presentation tile when sharing. */ - screenStreamRef: React.MutableRefObject - isScreenSharing: boolean - onToggleScreenShare: () => void cameraOn: boolean onToggleCamera: () => void + /** Starting a share collapses this view into the floating popout (the + * surface is derived from devices — see App.tsx). */ + onToggleScreenShare: () => void + /** Practice preset: the assistant is coaching this session. */ + practiceMode?: boolean + /** Shrink to the floating pill without touching any devices. */ + onMinimize: () => void ttsState: TTSState /** Live TTS output level — drives the mascot's mouth animation. */ getTtsLevel: () => number @@ -33,53 +36,22 @@ const STATUS_DISPLAY: Record element for the lifetime of the mount. */ -function StreamVideo({ - streamRef, - mirrored, - className, -}: { - streamRef: React.MutableRefObject - mirrored?: boolean - className?: string -}) { - const videoRef = useRef(null) - - useEffect(() => { - const videoEl = videoRef.current - if (!videoEl) return - videoEl.srcObject = streamRef.current - videoEl.play().catch(() => {}) - return () => { - videoEl.srcObject = null - } - }, [streamRef]) - - return ( -