rowboat/apps/x/VIDEO_MODE.md

Video Mode — Deep Dive

Video mode lets the assistant see the user (webcam) and their screen (screen share), in three presentations: frames attached to normal chat, a hands-free spoken call, and a full-screen Meet-style call. This doc covers the product flow, the technical pipeline, and the LLM prompt surface with exact pointers.

Product flow

The composer's video button (chat-input-with-mentions.tsx) toggles video mode; a chevron dropdown picks one of three modes (VideoChatMode):

Mode	What it does
chat — "Video + chat"	Camera on. Webcam (and screen-share) frames ride along with every typed or dictated message. Small PiP preview floats above the composer.
call — "Video call (hands-free)"	Everything in chat, plus: continuous listening (each utterance auto-submits as a voice message) and forced full read-aloud TTS. No typing needed; composer still works.
meeting — "Video call (full screen)"	Same pipeline as call, presented as a full-screen Meet-style layout: user tile + animated mascot tile, captions, control bar.

On top of any mode:

• Screen share (MonitorUp buttons on the PiP overlay and the meeting control bar): captures the primary screen; frames go to the model as a separately labeled group. In the meeting view the screen becomes the big tile with user + mascot in a side rail.
• Camera off (Meet-style mute): video mode and screen share keep running, no webcam frames are captured; tiles show a silhouette avatar.
• Mascot dismissal (meeting view): swaps the animated mascot for a Meet-style letter avatar ("R").
• Popout: starting a screen share immediately pops the mini-call out into a small always-on-top frameless window (user + mascot tiles, live caption, control bar) that floats over every app — including Rowboat itself, since the shared screen may be Rowboat. It stays for the whole share (surviving app switches) and disappears when sharing stops or video mode ends. While sharing it is THE call surface: the in-app PiP hides, and presenting from the full-screen meeting collapses the meeting into it (restored when the share ends). Control-bar actions (camera toggle / stop share / end call) round-trip video:popoutAction → main → video:popout-action → app window, which owns the mic/camera/capture. The expand button focuses the main window (video:focusMain).

call/meeting options are disabled unless both voice input (Deepgram) and voice output (TTS) are configured. Entering a call saves the user's TTS settings and forces full read-aloud; leaving restores them.

Frame pipeline

apps/renderer/src/hooks/useVideoMode.ts runs one capture pipe per source (stream → offscreen <video> → canvas JPEG → ring buffer):

• Cadence: 1 fps (CAPTURE_INTERVAL_MS, line 20); ring buffer ~2 min.
• Webcam: 512px wide, JPEG q0.65, max 12 frames/message (lines 21, 31).
• Screen: 1280px wide (text legibility), JPEG q0.7, max 4 frames/message (lines 24, 32).
• collectFrames() drains frames buffered since the last send, evenly sampled down to the caps, always keeping the newest; grabs one final frame at the moment of send. Falls back to the single latest frame for rapid-fire messages.

App.tsx handlePromptSubmit (~line 2767) attaches the drained frames to the outgoing message as UserImageParts and sets composition.videoMode: true. Frames also become isVideoFrame display attachments (filmstrip in the transcript — chat-message-attachments.tsx; history hydration in lib/run-to-conversation.ts).

Message schema & model encoding

• packages/shared/src/message.ts:51 — UserImagePart: inline base64 (data, mediaType), source: 'camera' | 'screen', capturedAt. Unlike file attachments (path references read via the LLMParse tool), image parts go to the model as real multimodal image parts.
• packages/core/src/agents/runtime.ts convertFromMessages (~line 1013): emits a context line (frame counts + time span), then labeled groups — a "Webcam frames (oldest to newest):" text part before camera images and a "Screen-share frames (oldest to newest):" text part before screen images — so the model never confuses the user with their screen.
• Frames stay inline in history (no pruning) deliberately: pruning would bust provider prefix caching every turn and cost more than it saves.
• The auto-permission classifier stringifies + truncates content to ~3KB per message, so inline base64 can't blow up its prompt.

Hands-free voice loop

apps/renderer/src/hooks/useVoiceMode.ts:

• startContinuous(onUtterance) (line 404): push-to-talk params but with endpointing=1800 (line 25) so thinking pauses don't cut the user off, plus utterance_end_ms=2000 (line 38) as a second end-of-speech signal. Gotcha: Deepgram's speech_final usually arrives on a result with an EMPTY transcript — empty finals must reach the endpoint check or utterances never complete (see the NOTE in ws.onmessage).
• setPaused(true) (line 414) while the assistant thinks/speaks: drops mic audio (so TTS is never transcribed back), discards half-heard buffer, sends Deepgram KeepAlives every 5s. App.tsx drives this from activeIsProcessing || tts.state !== 'idle'.
• Mid-call socket drops reconnect after 1s; the offline audio backlog is capped (~30s).

Mode transitions live in App.tsx handleVideoModeChange (~line 1161): call ↔ meeting switches are presentation-only (mic/TTS untouched); entering/leaving hands-free saves/restores TTS settings. Push-to-talk is disabled while a call owns the mic.

Popout window

• Shown iff videoChatMode !== 'off' && screenState === 'live' (effect in App.tsx). Renderer asks video:setPopout {show} (main handler: apps/main/src/ipc.ts:1742); main creates a frameless, alwaysOnTop ('floating'), all-workspaces BrowserWindow at the top-right of the primary display, loading the renderer bundle with #video-popout (apps/renderer/src/main.tsx branches on the hash → components/video-popout.tsx).
• Call state streams over the video:popout-state push channel; main caches the last payload and replays it on popout load. Shown with showInactive() so it never steals focus (that would re-hide it).
• The popout captures its own camera preview (MediaStreams can't cross windows) and synthesizes the mascot mouth level (no audio in that window).
• video:focusMain matches only real app windows by URL — getAllWindows() also contains hidden utility windows (PDF export) that must not be shown.

Permissions

• Camera: voice:ensureCameraAccess settles the macOS TCC prompt before getUserMedia (same pattern as the mic). NSCameraUsageDescription is in forge.config.cjs extendInfo.
• Screen: getDisplayMedia is auto-approved with the primary screen by setDisplayMediaRequestHandler in main.ts (no picker); meeting:checkScreenPermission registers the app in macOS Screen Recording settings on first use.

LLM prompts catalog

Prompt	Where
# Video Mode (Live Camera) system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance)	packages/core/src/agents/runtime.ts:386 (composeSystemInstructions, gated on videoMode)
Per-message frame context line [Video mode: N live webcam frames … and M frames of the user's shared screen …] + group labels	packages/core/src/agents/runtime.ts:~1013 (convertFromMessages)
videoMode composition override (session-sticky; flips bust prefix cache)	packages/core/src/turns/bridges/real-agent-resolver.ts:57,125; set from App.tsx sendConfig

Voice input/output prompt sections (# Voice Input, # Voice Output) are reused untouched — calls set voiceInput per utterance and force voiceOutput: 'full'.

Cost notes

Webcam frames ≈ 250–350 tokens each (≤12/message ≈ 3–4k); screen frames ≈ 1.5–2k tokens each (≤4/message ≈ 6–8k). History keeps frames inline, so long sessions grow but stay prefix-cached. First lever if cost bites: drop to one screen frame per message unless the screen changed.

Known limitations

• Turn-taking is strict — no barge-in (would need echo cancellation against TTS output).
• Frame sampling, not video: motion between frames is invisible (the prompt tells the model not to claim otherwise).
• Vocal-delivery feedback is limited: Deepgram reduces speech to text, so "energy" coaching leans on visual cues.
• Screen share always captures the primary display (no window/display picker yet).
• The meeting view covers the chat; there's no in-call transcript drawer.