2026-07-04 12:06:32 +05:30
# Calls (Video Mode) — Deep Dive
2026-07-04 00:09:46 +05:30
2026-07-04 12:06:32 +05:30
Calls let the user talk to the assistant hands-free while it *sees* them
(webcam) and their screen (screen share). There is ONE call engine —
continuous listening, auto-submitted utterances, forced read-aloud TTS, frame
capture — entered through four presets that differ only in starting devices.
This doc covers the product flow, the technical pipeline, and the LLM prompt
surface with exact pointers.
2026-07-04 00:09:46 +05:30
## Product flow
2026-07-04 12:06:32 +05:30
The composer has a **call split-button** (`chat-input-with-mentions.tsx` ).
The main click is the "work together" default — preset `share` : screen
sharing ON, camera OFF, floating pill, so the user keeps working while the
assistant watches along (the button tooltip discloses the screen share). The
chevron menu holds the deviations. While a call is live the button turns red
and ends it.
| Preset | Starting devices | First surface |
|--------|------------------|---------------|
| `share` — main click | screen on, camera off | floating pill |
| `voice` — "Voice call" | camera off, screen off | floating mascot pill |
| `video` — "Video call" | camera on | full-screen call |
| `practice` — "Practice session" | camera on, + coaching persona | full-screen call |
**One surface rule** (`callSurface` in `App.tsx` ): full screen and screen
sharing are mutually exclusive in both directions — a full-screen call covers
the screen, so sharing it would show the call itself.
- sharing → floating popout, always (pill = working)
- not sharing → full screen unless `callMinimized` (full screen = facing
each other)
- expanding the pill auto-STOPS any share; minimizing the full-screen call
auto-STARTS one (the pill exists to work together) — presenting from full
screen likewise collapses to the pill
- the camera toggle never changes the surface: turning it on from the pill
puts your video IN the pill; expanding is its own explicit action
**Screen-share consent** is three-layered: a toast the moment any share
starts ("Your screen is being shared… [Stop sharing]"), a persistent
"Sharing screen" badge on the pill, and macOS's purple recording indicator.
If the auto-share fails (Screen Recording permission not granted) the call
starts anyway as a voice call, with a toast linking to System Settings.
Practice/coaching is always an explicit choice — expanding to full screen
never turns the coach on.
In-call controls (identical bar on both surfaces): camera toggle (silhouette
avatar while off, no webcam frames captured), screen share toggle, mascot ⇄
"R" letter avatar, end call. Captions of the in-progress utterance and the
assistant's spoken line run along the bottom. Typing in the composer still
works mid-call; frames ride along with typed messages too.
2026-07-04 12:21:15 +05:30
Outside calls the composer keeps exactly one voice affordance: the **mic
button** (push-to-talk dictation, untouched). Spoken responses exist only
inside calls (forced full read-aloud, off on hang-up). The old video
dropdown, talking-head toggle, read-aloud headphones toggle, and summary/full
TTS dropdown are all retired — a per-message "read aloud" action on assistant
messages is the planned replacement for text-in/voice-out.
2026-07-04 12:06:32 +05:30
The call button is disabled unless both voice input (Deepgram) and voice
output (TTS) are configured. `call_started` (with `preset` ) is captured in
PostHog — the adoption metric for this feature.
**Popout mechanics**: a small always-on-top frameless window (camera tile
when on + mascot tile, live caption, control bar) floating over every app —
including Rowboat. Control-bar actions round-trip `video:popoutAction` →
main → `video:popout-action` → app window, which owns the mic/camera/capture;
`expand` also refocuses the app window (handled in main).
2026-07-04 00:09:46 +05:30
## Frame pipeline
`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source
(stream → offscreen `<video>` → canvas JPEG → ring buffer):
- Cadence: 1 fps (`CAPTURE_INTERVAL_MS` , line 20); ring buffer ~2 min.
- Webcam: 512px wide, JPEG q0.65, max **12 frames/message** (lines 21, 31).
- Screen: 1280px wide (text legibility), JPEG q0.7, max **4 frames/message**
(lines 24, 32).
- `collectFrames()` drains frames buffered since the last send, evenly
sampled down to the caps, always keeping the newest; grabs one final frame
at the moment of send. Falls back to the single latest frame for
rapid-fire messages.
2026-07-04 12:06:32 +05:30
`App.tsx` `handlePromptSubmit` attaches the drained frames (whenever a call
is live) to the outgoing message as `UserImagePart` s and sets
`composition.videoMode` when the camera or screen is active, plus
`composition.coachMode` during a practice session. Frames also become
`isVideoFrame` display attachments (filmstrip in the transcript —
`chat-message-attachments.tsx` ; history hydration in
`lib/run-to-conversation.ts` ).
2026-07-04 00:09:46 +05:30
## Message schema & model encoding
- `packages/shared/src/message.ts:51` — `UserImagePart` : inline base64
(`data` , `mediaType` ), `source: 'camera' | 'screen'` , `capturedAt` . Unlike
file attachments (path references read via the `LLMParse` tool), image
parts go to the model as real multimodal image parts.
- `packages/core/src/agents/runtime.ts` `convertFromMessages` (~line 1013):
emits a context line (frame counts + time span), then labeled groups —
a `"Webcam frames (oldest to newest):"` text part before camera images and
a `"Screen-share frames (oldest to newest):"` text part before screen
images — so the model never confuses the user with their screen.
- Frames stay inline in history (no pruning) deliberately: pruning would
bust provider prefix caching every turn and cost more than it saves.
- The auto-permission classifier stringifies + truncates content to ~3KB per
message, so inline base64 can't blow up its prompt.
## Hands-free voice loop
`apps/renderer/src/hooks/useVoiceMode.ts` :
- `startContinuous(onUtterance)` (line 404): push-to-talk params but with
`endpointing=1800` (line 25) so thinking pauses don't cut the user off,
plus `utterance_end_ms=2000` (line 38) as a second end-of-speech signal.
**Gotcha:** Deepgram's `speech_final` usually arrives on a result with an
EMPTY transcript — empty finals must reach the endpoint check or
utterances never complete (see the NOTE in `ws.onmessage` ).
- `setPaused(true)` (line 414) while the assistant thinks/speaks: drops mic
audio (so TTS is never transcribed back), discards half-heard buffer,
sends Deepgram KeepAlives every 5s. `App.tsx` drives this from
`activeIsProcessing || tts.state !== 'idle'` .
- Mid-call socket drops reconnect after 1s; the offline audio backlog is
capped (~30s).
2026-07-04 12:06:32 +05:30
Call lifecycle lives in `App.tsx` `startCall(preset)` / `endCall()` :
entering a call saves/forces TTS settings, cancels any push-to-talk
recording, and starts the continuous loop; ending restores everything.
Push-to-talk is disabled while a call owns the mic.
2026-07-04 00:09:46 +05:30
## Popout window
2026-07-04 12:06:32 +05:30
- Shown iff the derived `callSurface === 'popout'` (effect in `App.tsx` ).
Renderer asks `video:setPopout {show}` ; main creates a frameless,
`alwaysOnTop` ('floating'), all-workspaces BrowserWindow at the top-right
of the primary display, loading the renderer bundle with `#video-popout`
2026-07-04 00:09:46 +05:30
(`apps/renderer/src/main.tsx` branches on the hash →
`components/video-popout.tsx` ).
- Call state streams over the `video:popout-state` push channel; main caches
the last payload and replays it on popout load. Shown with
2026-07-04 12:06:32 +05:30
`showInactive()` so it never steals focus.
2026-07-04 00:09:46 +05:30
- The popout captures its **own** camera preview (MediaStreams can't cross
windows) and synthesizes the mascot mouth level (no audio in that window).
2026-07-04 12:06:32 +05:30
- `video:popoutAction` relays control-bar actions to the app window, matched
only by real app-window URLs — `getAllWindows()` also contains hidden
utility windows (PDF export) that must not be shown or messaged.
2026-07-04 00:09:46 +05:30
## Permissions
- Camera: `voice:ensureCameraAccess` settles the macOS TCC prompt before
`getUserMedia` (same pattern as the mic). `NSCameraUsageDescription` is in
`forge.config.cjs` `extendInfo` .
- Screen: `getDisplayMedia` is auto-approved with the primary screen by
`setDisplayMediaRequestHandler` in `main.ts` (no picker);
`meeting:checkScreenPermission` registers the app in macOS Screen
Recording settings on first use.
## LLM prompts catalog
| Prompt | Where |
|--------|-------|
| `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions` , gated on `videoMode` ) |
2026-07-04 12:06:32 +05:30
| `# Practice Session (Coach Mode)` system section — coaching persona: specific/actionable feedback after each take, one-sentence interjections mid-flow, structured debrief on wrap-up | `composeSystemInstructions` , gated on `coachMode` (directly after the video section) |
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts` (`convertFromMessages` ) |
| `videoMode` / `coachMode` composition overrides (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts` (`CompositionOverrides` ); set from `App.tsx` `sendConfig` |
2026-07-04 00:09:46 +05:30
Voice input/output prompt sections (`# Voice Input` , `# Voice Output` ) are
reused untouched — calls set `voiceInput` per utterance and force
`voiceOutput: 'full'` .
2026-07-04 12:51:06 +05:30
## Latency
Voice-to-voice latency (user stops talking → assistant audio) is engineered
at four points; the `call_turn_latency` PostHog event measures the real
distribution (utterance → submit → first speak → audio playing):
- **Smart endpointing** (`useVoiceMode.ts` ): Deepgram endpoints at 600ms and
the client decides — a transcript ending in terminal punctuation fires
immediately (~600ms after last word); a mid-thought trail holds another
1.2s (resumed speech cancels the hold). Complete sentences turn around
~1.2s faster than the old fixed 1800ms endpoint.
- **Streaming TTS** (`voice:synthesizeStreamStart` → `voice:tts-chunk` →
MediaSource playback in `useVoiceTTS.ts` ): the first segment of an idle
queue plays from the first MP3 chunk instead of after the full body
(ElevenLabs `/stream` , flash model). Follow-up segments keep the gapless
full-body prefetch path. Falls back to non-streaming on any failure.
- **Early clause speech** (`turn-view.ts` `applyOverlay` ): a still-open
`<voice>` block ≥60 chars emits its last complete clause immediately, so
speech starts while the rest of the sentence generates.
- **Acknowledgment cue** (`lib/call-sounds.ts` ): a soft blip the instant an
utterance is accepted — perceived latency matters as much as measured.
2026-07-04 00:09:46 +05:30
## Cost notes
Webcam frames ≈ 250– –
1.5– –
sessions grow but stay prefix-cached. First lever if cost bites: drop to one
screen frame per message unless the screen changed.
## Known limitations
- Turn-taking is strict — no barge-in (would need echo cancellation against
TTS output).
- Frame sampling, not video: motion between frames is invisible (the prompt
tells the model not to claim otherwise).
- Vocal-delivery feedback is limited: Deepgram reduces speech to text, so
"energy" coaching leans on visual cues.
- Screen share always captures the primary display (no window/display
picker yet).
2026-07-04 12:06:32 +05:30
- The full-screen call covers the chat; there's no in-call transcript drawer.
- The "attach camera frames to typed chat without a call" combination (the
old video+chat mode) was cut in the call-model simplification; if analytics
show demand, it should return as an attachment chip, not a mode.
