screen share pop out when I switch to a different app

This commit is contained in:
Arjun 2026-07-04 00:09:46 +05:30
parent 48a3ff6a36
commit 1edc1b17f1
10 changed files with 567 additions and 23 deletions

View file

@ -109,6 +109,7 @@ Long-form docs for specific features. Read the relevant file before making chang
| Feature | Doc |
|---------|-----|
| Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` |
| Video mode — webcam + screen-share frames to the model (video+chat / hands-free call / full-screen Meet-style call), continuous voice loop, camera mute, popout window, prompts catalog | `apps/x/VIDEO_MODE.md` |
| Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` |
| Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` |

151
apps/x/VIDEO_MODE.md Normal file
View file

@ -0,0 +1,151 @@
# Video Mode — Deep Dive
Video mode lets the assistant *see* the user (webcam) and their screen (screen
share), in three presentations: frames attached to normal chat, a hands-free
spoken call, and a full-screen Meet-style call. This doc covers the product
flow, the technical pipeline, and the LLM prompt surface with exact pointers.
## Product flow
The composer's video button (`chat-input-with-mentions.tsx`) toggles video
mode; a chevron dropdown picks one of three modes (`VideoChatMode`):
| Mode | What it does |
|------|--------------|
| `chat` — "Video + chat" | Camera on. Webcam (and screen-share) frames ride along with every typed or dictated message. Small PiP preview floats above the composer. |
| `call` — "Video call (hands-free)" | Everything in `chat`, plus: continuous listening (each utterance auto-submits as a voice message) and forced full read-aloud TTS. No typing needed; composer still works. |
| `meeting` — "Video call (full screen)" | Same pipeline as `call`, presented as a full-screen Meet-style layout: user tile + animated mascot tile, captions, control bar. |
On top of any mode:
- **Screen share** (`MonitorUp` buttons on the PiP overlay and the meeting
control bar): captures the primary screen; frames go to the model as a
separately labeled group. In the meeting view the screen becomes the big
tile with user + mascot in a side rail.
- **Camera off** (Meet-style mute): video mode and screen share keep running,
no webcam frames are captured; tiles show a silhouette avatar.
- **Mascot dismissal** (meeting view): swaps the animated mascot for a
Meet-style letter avatar ("R").
- **Popout**: while screen sharing, if the app window loses focus (the user
switched to the app they're sharing), a small always-on-top frameless
window pops out with the user + mascot mini-tiles; refocusing dismisses it.
Its expand button focuses the main window (`video:focusMain`).
`call`/`meeting` options are disabled unless both voice input (Deepgram) and
voice output (TTS) are configured. Entering a call saves the user's TTS
settings and forces `full` read-aloud; leaving restores them.
## Frame pipeline
`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source
(stream → offscreen `<video>` → canvas JPEG → ring buffer):
- Cadence: 1 fps (`CAPTURE_INTERVAL_MS`, line 20); ring buffer ~2 min.
- Webcam: 512px wide, JPEG q0.65, max **12 frames/message** (lines 21, 31).
- Screen: 1280px wide (text legibility), JPEG q0.7, max **4 frames/message**
(lines 24, 32).
- `collectFrames()` drains frames buffered since the last send, evenly
sampled down to the caps, always keeping the newest; grabs one final frame
at the moment of send. Falls back to the single latest frame for
rapid-fire messages.
`App.tsx` `handlePromptSubmit` (~line 2767) attaches the drained frames to
the outgoing message as `UserImagePart`s and sets
`composition.videoMode: true`. Frames also become `isVideoFrame` display
attachments (filmstrip in the transcript — `chat-message-attachments.tsx`;
history hydration in `lib/run-to-conversation.ts`).
## Message schema & model encoding
- `packages/shared/src/message.ts:51``UserImagePart`: inline base64
(`data`, `mediaType`), `source: 'camera' | 'screen'`, `capturedAt`. Unlike
file attachments (path references read via the `LLMParse` tool), image
parts go to the model as real multimodal image parts.
- `packages/core/src/agents/runtime.ts` `convertFromMessages` (~line 1013):
emits a context line (frame counts + time span), then labeled groups —
a `"Webcam frames (oldest to newest):"` text part before camera images and
a `"Screen-share frames (oldest to newest):"` text part before screen
images — so the model never confuses the user with their screen.
- Frames stay inline in history (no pruning) deliberately: pruning would
bust provider prefix caching every turn and cost more than it saves.
- The auto-permission classifier stringifies + truncates content to ~3KB per
message, so inline base64 can't blow up its prompt.
## Hands-free voice loop
`apps/renderer/src/hooks/useVoiceMode.ts`:
- `startContinuous(onUtterance)` (line 404): push-to-talk params but with
`endpointing=1800` (line 25) so thinking pauses don't cut the user off,
plus `utterance_end_ms=2000` (line 38) as a second end-of-speech signal.
**Gotcha:** Deepgram's `speech_final` usually arrives on a result with an
EMPTY transcript — empty finals must reach the endpoint check or
utterances never complete (see the NOTE in `ws.onmessage`).
- `setPaused(true)` (line 414) while the assistant thinks/speaks: drops mic
audio (so TTS is never transcribed back), discards half-heard buffer,
sends Deepgram KeepAlives every 5s. `App.tsx` drives this from
`activeIsProcessing || tts.state !== 'idle'`.
- Mid-call socket drops reconnect after 1s; the offline audio backlog is
capped (~30s).
Mode transitions live in `App.tsx` `handleVideoModeChange` (~line 1161):
call ↔ meeting switches are presentation-only (mic/TTS untouched);
entering/leaving hands-free saves/restores TTS settings. Push-to-talk is
disabled while a call owns the mic.
## Popout window
- Renderer asks `video:setPopout {show}` (main handler:
`apps/main/src/ipc.ts:1742`); main creates a frameless, `alwaysOnTop`
('floating'), all-workspaces BrowserWindow at the top-right of the primary
display, loading the renderer bundle with `#video-popout`
(`apps/renderer/src/main.tsx` branches on the hash →
`components/video-popout.tsx`).
- Call state streams over the `video:popout-state` push channel; main caches
the last payload and replays it on popout load. Shown with
`showInactive()` so it never steals focus (that would re-hide it).
- The popout captures its **own** camera preview (MediaStreams can't cross
windows) and synthesizes the mascot mouth level (no audio in that window).
- `video:focusMain` matches only real app windows by URL — `getAllWindows()`
also contains hidden utility windows (PDF export) that must not be shown.
## Permissions
- Camera: `voice:ensureCameraAccess` settles the macOS TCC prompt before
`getUserMedia` (same pattern as the mic). `NSCameraUsageDescription` is in
`forge.config.cjs` `extendInfo`.
- Screen: `getDisplayMedia` is auto-approved with the primary screen by
`setDisplayMediaRequestHandler` in `main.ts` (no picker);
`meeting:checkScreenPermission` registers the app in macOS Screen
Recording settings on first use.
## LLM prompts catalog
| Prompt | Where |
|--------|-------|
| `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) |
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts:~1013` (`convertFromMessages`) |
| `videoMode` composition override (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts:57,125`; set from `App.tsx` `sendConfig` |
Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are
reused untouched — calls set `voiceInput` per utterance and force
`voiceOutput: 'full'`.
## Cost notes
Webcam frames ≈ 250350 tokens each (≤12/message ≈ 34k); screen frames ≈
1.52k tokens each (≤4/message ≈ 68k). History keeps frames inline, so long
sessions grow but stay prefix-cached. First lever if cost bites: drop to one
screen frame per message unless the screen changed.
## Known limitations
- Turn-taking is strict — no barge-in (would need echo cancellation against
TTS output).
- Frame sampling, not video: motion between frames is invisible (the prompt
tells the model not to claim otherwise).
- Vocal-delivery feedback is limited: Deepgram reduces speech to text, so
"energy" coaching leans on visual cues.
- Screen share always captures the primary display (no window/display
picker yet).
- The meeting view covers the chat; there's no in-call transcript drawer.

View file

@ -1,7 +1,8 @@
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app } from 'electron';
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app, screen } from 'electron';
import { ipc } from '@x/shared';
import path from 'node:path';
import os from 'node:os';
import { fileURLToPath } from 'node:url';
import {
connectProvider,
disconnectProvider,
@ -381,9 +382,19 @@ type InvokeHandlers = {
[K in InvokeChannels]: InvokeHandler<K>;
};
// Video-mode popout window (shown while screen sharing when the app loses
// focus) and the last call state pushed by the main window — replayed to the
// popout when it finishes loading.
let videoPopoutWin: BrowserWindow | null = null;
let lastVideoPopoutState: {
ttsState: 'idle' | 'synthesizing' | 'speaking';
status: 'listening' | 'thinking' | 'speaking' | null;
cameraOn: boolean;
} | null = null;
/**
* Register all IPC handlers with type safety and runtime validation
*
*
* This function ensures:
* 1. All invoke channels have handlers (exhaustiveness checking)
* 2. Handler signatures match channel definitions
@ -1728,6 +1739,87 @@ export function setupIpcHandlers() {
return { granted: false };
}
},
'video:setPopout': async (_event, args) => {
if (!args.show) {
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) videoPopoutWin.destroy();
videoPopoutWin = null;
return {};
}
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) return {};
const workArea = screen.getPrimaryDisplay().workArea;
const width = 340;
const height = 148;
const ipcDir = path.dirname(fileURLToPath(import.meta.url));
const preloadPath = app.isPackaged
? path.join(ipcDir, '../preload/dist/preload.js')
: path.join(ipcDir, '../../../preload/dist/preload.js');
const win = new BrowserWindow({
width,
height,
x: workArea.x + workArea.width - width - 24,
y: workArea.y + 24,
frame: false,
resizable: false,
alwaysOnTop: true,
skipTaskbar: true,
show: false,
hasShadow: true,
backgroundColor: '#171717',
webPreferences: {
nodeIntegration: false,
contextIsolation: true,
sandbox: true,
preload: preloadPath,
},
});
// Float above other apps (and fullscreen spaces on macOS) — the whole
// point is being visible while the user works elsewhere.
win.setAlwaysOnTop(true, 'floating');
win.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true });
win.webContents.once('did-finish-load', () => {
if (lastVideoPopoutState) {
win.webContents.send('video:popout-state', lastVideoPopoutState);
}
// showInactive: appearing must not steal focus from the app the user
// switched to — that would immediately re-hide the popout.
if (!win.isDestroyed()) win.showInactive();
});
win.on('closed', () => {
if (videoPopoutWin === win) videoPopoutWin = null;
});
videoPopoutWin = win;
if (app.isPackaged) {
win.loadURL('app://-/index.html#video-popout');
} else {
win.loadURL('http://localhost:5173/#video-popout');
}
return {};
},
'video:popoutState': async (_event, args) => {
lastVideoPopoutState = args;
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) {
videoPopoutWin.webContents.send('video:popout-state', args);
}
return {};
},
'video:focusMain': async () => {
// Match only real app windows — getAllWindows() can also contain the
// popout itself and hidden utility windows (e.g. PDF-export renderers),
// which must not be shown or focused.
const main = BrowserWindow.getAllWindows().find((w) => {
if (w === videoPopoutWin || w.isDestroyed()) return false;
const url = w.webContents.getURL();
const isAppWindow = url.startsWith('app://') || url.startsWith('http://localhost');
return isAppWindow && !url.includes('#video-popout');
});
if (main) {
if (main.isMinimized()) main.restore();
main.show();
main.focus();
}
return {};
},
// Live-note handlers
'live-note:run': async (_event, args) => {
const result = await runLiveNoteAgent(args.filePath, 'manual', args.context);

View file

@ -1231,6 +1231,57 @@ function App() {
}
}, [video])
// Meet-style camera mute: video mode (and any screen share) stays on, but
// no webcam frames are captured while the camera is off.
const handleToggleCamera = useCallback(() => {
void video.setCameraEnabled(!video.cameraOn)
}, [video])
// Current phase of a hands-free call (null outside call/meeting modes).
const videoCallStatus: 'listening' | 'thinking' | 'speaking' | null =
videoChatMode === 'call' || videoChatMode === 'meeting'
? tts.state === 'speaking'
? 'speaking'
: tts.state === 'synthesizing' || activeIsProcessing
? 'thinking'
: 'listening'
: null
// Meet-style popout: while screen sharing, losing app focus (the user
// switched to the app they're sharing) pops the mini-call out into an
// always-on-top window; refocusing Rowboat dismisses it. The short show
// delay keeps a quick cmd-tab pass-through from flashing a window.
const [appFocused, setAppFocused] = useState(true)
useEffect(() => {
const onFocus = () => setAppFocused(true)
const onBlur = () => setAppFocused(false)
window.addEventListener('focus', onFocus)
window.addEventListener('blur', onBlur)
return () => {
window.removeEventListener('focus', onFocus)
window.removeEventListener('blur', onBlur)
}
}, [])
useEffect(() => {
const shouldShow = videoChatMode !== 'off' && video.screenState === 'live' && !appFocused
if (shouldShow) {
const timer = setTimeout(() => {
void window.ipc.invoke('video:setPopout', { show: true }).catch(() => {})
}, 300)
return () => clearTimeout(timer)
}
void window.ipc.invoke('video:setPopout', { show: false }).catch(() => {})
}, [videoChatMode, video.screenState, appFocused])
// Keep the popout's mascot/status/camera mirror of the call fresh. The main
// process caches the latest state and replays it when the popout loads.
useEffect(() => {
if (videoChatMode === 'off') return
void window.ipc
.invoke('video:popoutState', { ttsState: tts.state, status: videoCallStatus, cameraOn: video.cameraOn })
.catch(() => {})
}, [videoChatMode, tts.state, videoCallStatus, video.cameraOn])
// Enter to submit voice input, Escape to cancel
useEffect(() => {
const handleKeyDown = (e: KeyboardEvent) => {
@ -6611,6 +6662,8 @@ function App() {
interimText={videoChatMode === 'call' ? voice.interimText : undefined}
isScreenSharing={video.screenState === 'live'}
onToggleScreenShare={handleToggleScreenShare}
cameraOn={video.cameraOn}
onToggleCamera={handleToggleCamera}
/>
)}
{/* Full-screen Meet-style call: user tile + animated mascot tile */}
@ -6620,6 +6673,8 @@ function App() {
screenStreamRef={video.screenStreamRef}
isScreenSharing={video.screenState === 'live'}
onToggleScreenShare={handleToggleScreenShare}
cameraOn={video.cameraOn}
onToggleCamera={handleToggleCamera}
ttsState={tts.state}
getTtsLevel={tts.getLevel}
status={

View file

@ -1,5 +1,5 @@
import { useEffect, useRef, useState } from 'react'
import { MonitorUp, PhoneOff } from 'lucide-react'
import { MonitorUp, PhoneOff, User, Video, VideoOff } from 'lucide-react'
import { MascotFaceIcon, TalkingHead } from '@/components/talking-head'
import type { TTSState } from '@/hooks/useVoiceTTS'
@ -14,6 +14,8 @@ interface VideoCallViewProps {
screenStreamRef: React.MutableRefObject<MediaStream | null>
isScreenSharing: boolean
onToggleScreenShare: () => void
cameraOn: boolean
onToggleCamera: () => void
ttsState: TTSState
/** Live TTS output level — drives the mascot's mouth animation. */
getTtsLevel: () => number
@ -76,6 +78,8 @@ export function VideoCallView({
screenStreamRef,
isScreenSharing,
onToggleScreenShare,
cameraOn,
onToggleCamera,
ttsState,
getTtsLevel,
status,
@ -102,7 +106,19 @@ export function VideoCallView({
isScreenSharing && 'aspect-video w-full'
)}
>
<StreamVideo streamRef={streamRef} mirrored className="h-full w-full object-cover" />
{cameraOn ? (
<StreamVideo streamRef={streamRef} mirrored className="h-full w-full object-cover" />
) : (
<span
className={cn(
'flex items-center justify-center rounded-full bg-neutral-700 text-neutral-400',
isScreenSharing ? 'h-16 w-16' : 'h-40 w-40'
)}
aria-label="Camera off"
>
<User className={isScreenSharing ? 'h-8 w-8' : 'h-20 w-20'} />
</span>
)}
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
You
</span>
@ -182,6 +198,20 @@ export function VideoCallView({
<span className={cn('block h-2 w-2 rounded-full', STATUS_DISPLAY[status].dotClass)} />
{STATUS_DISPLAY[status].label}
</span>
<button
type="button"
onClick={onToggleCamera}
className={cn(
'flex h-10 w-10 items-center justify-center rounded-full transition-colors',
cameraOn
? 'bg-neutral-800 text-white/90 hover:bg-neutral-700'
: 'bg-red-600 text-white hover:bg-red-500'
)}
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
>
{cameraOn ? <Video className="h-5 w-5" /> : <VideoOff className="h-5 w-5" />}
</button>
<button
type="button"
onClick={onToggleScreenShare}

View file

@ -0,0 +1,117 @@
import { useCallback, useEffect, useRef, useState } from 'react'
import { Maximize2, User } from 'lucide-react'
import { TalkingHead } from '@/components/talking-head'
type PopoutState = {
ttsState: 'idle' | 'synthesizing' | 'speaking'
status: 'listening' | 'thinking' | 'speaking' | null
cameraOn: boolean
}
const STATUS_DISPLAY: Record<NonNullable<PopoutState['status']>, { label: string; dotClass: string }> = {
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
}
const dragRegion = { WebkitAppRegion: 'drag' } as React.CSSProperties
const noDragRegion = { WebkitAppRegion: 'no-drag' } as React.CSSProperties
/**
* Content of the always-on-top popout window shown while screen sharing when
* the main app window loses focus (Meet-style floating mini-call). Rendered
* in its own BrowserWindow (see `video:setPopout` in the main process); call
* state streams in over the `video:popout-state` push channel. Captures its
* own webcam feed MediaStreams can't cross windows.
*/
export function VideoPopout() {
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: true })
const videoRef = useRef<HTMLVideoElement | null>(null)
useEffect(() => {
return window.ipc.on('video:popout-state', (next) => setState(next))
}, [])
// Own camera feed, following the main window's camera-on/off state.
useEffect(() => {
if (!state.cameraOn) return
let stream: MediaStream | null = null
let cancelled = false
navigator.mediaDevices
.getUserMedia({ video: { width: { ideal: 640 }, facingMode: 'user' }, audio: false })
.then((s) => {
if (cancelled) {
s.getTracks().forEach((t) => t.stop())
return
}
stream = s
if (videoRef.current) {
videoRef.current.srcObject = s
videoRef.current.play().catch(() => {})
}
})
.catch((err) => console.error('[popout] camera failed:', err))
return () => {
cancelled = true
stream?.getTracks().forEach((t) => t.stop())
if (videoRef.current) videoRef.current.srcObject = null
}
}, [state.cameraOn])
// The popout has no TTS audio pipeline — synthesize a plausible mouth level
// so the mascot still animates while the assistant speaks in the main window.
const getLevel = useCallback(() => 0.45 + 0.35 * Math.sin(performance.now() / 90), [])
const statusDisplay = state.status ? STATUS_DISPLAY[state.status] : null
return (
<div
className="flex h-screen w-screen select-none gap-1.5 bg-neutral-900 p-1.5"
style={dragRegion}
>
<div className="relative flex-1 overflow-hidden rounded-lg bg-neutral-800">
{state.cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-full w-full object-cover"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<div className="flex h-full w-full items-center justify-center">
<span className="flex h-14 w-14 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
<User className="h-7 w-7" />
</span>
</div>
)}
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
You
</span>
</div>
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-lg bg-neutral-800">
<TalkingHead ttsState={state.ttsState} getLevel={getLevel} size={92} />
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
Rowboat
</span>
{statusDisplay && (
<span className="absolute right-1.5 top-1.5 flex items-center gap-1 rounded-full bg-black/50 px-1.5 py-0.5 text-[10px] font-medium text-white">
<span className={`block h-1.5 w-1.5 rounded-full ${statusDisplay.dotClass}`} />
{statusDisplay.label}
</span>
)}
</div>
<button
type="button"
onClick={() => void window.ipc.invoke('video:focusMain', null)}
className="absolute left-1.5 top-1.5 flex h-5 w-5 items-center justify-center rounded bg-black/50 text-white/80 hover:text-white"
style={noDragRegion}
aria-label="Back to Rowboat"
title="Back to Rowboat"
>
<Maximize2 className="h-3 w-3" />
</button>
</div>
)
}

View file

@ -1,5 +1,5 @@
import { useEffect, useRef } from 'react'
import { MonitorUp, X } from 'lucide-react'
import { MonitorUp, User, VideoOff, X } from 'lucide-react'
interface VideoPreviewOverlayProps {
/** Live camera stream from useVideoMode — attached to the preview element. */
@ -11,6 +11,8 @@ interface VideoPreviewOverlayProps {
interimText?: string
isScreenSharing?: boolean
onToggleScreenShare?: () => void
cameraOn?: boolean
onToggleCamera?: () => void
}
const CALL_STATUS_DISPLAY: Record<NonNullable<VideoPreviewOverlayProps['callStatus']>, { label: string; dotClass: string }> = {
@ -24,10 +26,11 @@ const CALL_STATUS_DISPLAY: Record<NonNullable<VideoPreviewOverlayProps['callStat
* on. Mirrored like a selfie camera so the user's movements feel natural.
* Sits above the composer dock, mirroring the talking-head overlay's corner.
*/
export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimText, isScreenSharing, onToggleScreenShare }: VideoPreviewOverlayProps) {
export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimText, isScreenSharing, onToggleScreenShare, cameraOn = true, onToggleCamera }: VideoPreviewOverlayProps) {
const videoRef = useRef<HTMLVideoElement | null>(null)
useEffect(() => {
if (!cameraOn) return
const videoEl = videoRef.current
if (!videoEl) return
videoEl.srcObject = streamRef.current
@ -35,7 +38,7 @@ export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimT
return () => {
videoEl.srcObject = null
}
}, [streamRef])
}, [streamRef, cameraOn])
return (
<div
@ -71,13 +74,36 @@ export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimT
<MonitorUp className="h-3 w-3" />
</button>
)}
<video
ref={videoRef}
muted
playsInline
className="h-32 w-auto rounded-xl border border-border/70 bg-black shadow-lg"
style={{ transform: 'scaleX(-1)' }}
/>
{cameraOn ? (
<video
ref={videoRef}
muted
playsInline
className="h-32 w-auto rounded-xl border border-border/70 bg-black shadow-lg"
style={{ transform: 'scaleX(-1)' }}
/>
) : (
<div className="flex h-32 w-48 items-center justify-center rounded-xl border border-border/70 bg-black shadow-lg">
<span className="flex h-14 w-14 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
<User className="h-7 w-7" />
</span>
</div>
)}
{onToggleCamera && (
<button
type="button"
onClick={onToggleCamera}
className={`absolute -left-1.5 top-4 z-10 flex h-5 w-5 items-center justify-center rounded-full border shadow-sm transition-opacity ${
cameraOn
? 'border-border bg-background text-muted-foreground opacity-0 hover:text-foreground group-hover:opacity-100'
: 'border-red-500 bg-red-600 text-white opacity-100'
}`}
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
>
<VideoOff className="h-3 w-3" />
</button>
)}
<span className="pointer-events-none absolute bottom-1.5 left-1.5 flex items-center gap-1 rounded-full bg-black/60 px-1.5 py-0.5 text-[10px] font-medium text-white">
{callStatus ? (
<>

View file

@ -151,6 +151,9 @@ function drainPipe(pipe: CapturePipe, max: number, source: CapturedVideoFrame['s
export function useVideoMode() {
const [state, setState] = useState<VideoModeState>('idle');
const [screenState, setScreenState] = useState<ScreenShareState>('idle');
// Camera can be turned off mid-session (Meet-style) while the mode — and
// any screen share — keeps running. Resets to on for the next session.
const [cameraOn, setCameraOn] = useState(true);
const cameraPipeRef = useRef<CapturePipe>(emptyPipe());
const screenPipeRef = useRef<CapturePipe>(emptyPipe());
// Stable stream refs for preview components (<video srcObject>).
@ -179,13 +182,13 @@ export function useVideoMode() {
teardownPipe(cameraPipeRef.current);
streamRef.current = null;
setState('idle');
setCameraOn(true);
stopScreenShare();
}, [stopScreenShare]);
const start = useCallback(async (): Promise<boolean> => {
if (stateRef.current !== 'idle') return true;
setState('starting');
// Acquire the webcam and start its capture pipeline. Shared by start()
// and by re-enabling the camera mid-session.
const acquireCamera = useCallback(async (): Promise<boolean> => {
// Settle the macOS TCC camera permission before getUserMedia, same as
// voice mode does for the mic — otherwise the first click silently
// fails while the native prompt is still up.
@ -194,7 +197,6 @@ export function useVideoMode() {
.catch(() => ({ granted: true }));
if (!access.granted) {
console.error('[video] Camera access denied');
setState('idle');
return false;
}
@ -206,16 +208,45 @@ export function useVideoMode() {
});
} catch (err) {
console.error('[video] Camera access failed:', err);
setState('idle');
return false;
}
streamRef.current = stream;
attachPipeSource(cameraPipeRef.current, stream, captureCameraFrame);
setState('live');
return true;
}, [captureCameraFrame]);
/**
* Turn the camera off/on without leaving video mode (Meet-style). While
* off, no webcam frames are captured or attached; screen-share frames
* (if presenting) keep flowing.
*/
const setCameraEnabled = useCallback(async (enabled: boolean): Promise<boolean> => {
if (stateRef.current !== 'live') return false;
if (enabled) {
const ok = await acquireCamera();
if (ok) setCameraOn(true);
return ok;
}
teardownPipe(cameraPipeRef.current);
streamRef.current = null;
setCameraOn(false);
return true;
}, [acquireCamera]);
const start = useCallback(async (): Promise<boolean> => {
if (stateRef.current !== 'idle') return true;
setState('starting');
const ok = await acquireCamera();
if (!ok) {
setState('idle');
return false;
}
setCameraOn(true);
setState('live');
return true;
}, [acquireCamera]);
/**
* Share the screen. The main process auto-approves getDisplayMedia with
* the primary screen (see setDisplayMediaRequestHandler in main.ts), so
@ -270,5 +301,5 @@ export function useVideoMode() {
// Release the camera/screen if the component unmounts with video mode on.
useEffect(() => stop, [stop]);
return { state, screenState, streamRef, screenStreamRef, start, stop, startScreenShare, stopScreenShare, collectFrames };
return { state, screenState, cameraOn, streamRef, screenStreamRef, start, stop, startScreenShare, stopScreenShare, setCameraEnabled, collectFrames };
}

View file

@ -6,6 +6,7 @@ import { PostHogProvider } from 'posthog-js/react'
import type { CaptureResult } from 'posthog-js'
import { ThemeProvider } from '@/contexts/theme-context'
import { configureAnalyticsContext } from './lib/analytics'
import { VideoPopout } from '@/components/video-popout'
// Fetch the stable installation ID from main so renderer + main share one
// PostHog distinct_id. Falls back to PostHog's auto-generated anonymous ID
@ -57,4 +58,14 @@ async function bootstrap() {
// The loaded callback applies api_url/app_version once PostHog has initialized.
}
bootstrap()
// The video-mode popout window loads the same bundle with a hash route and
// renders only the mini-call UI — no analytics or app bootstrap needed.
if (window.location.hash === '#video-popout') {
createRoot(document.getElementById('root')!).render(
<StrictMode>
<VideoPopout />
</StrictMode>,
)
} else {
bootstrap()
}

View file

@ -1380,6 +1380,36 @@ const ipcSchemas = {
granted: z.boolean(),
}),
},
// Video-mode popout: show/hide a small always-on-top window (user + mascot
// tiles) while screen sharing when the main window loses focus, Meet-style.
'video:setPopout': {
req: z.object({ show: z.boolean() }),
res: z.object({}),
},
// Main-window renderer pushes the current call state; the main process
// caches it and relays to the popout window (replayed on popout load).
'video:popoutState': {
req: z.object({
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
cameraOn: z.boolean(),
}),
res: z.object({}),
},
// Popout window → bring the main app window back to the foreground.
'video:focusMain': {
req: z.null(),
res: z.object({}),
},
// Push channel: main → popout window with the latest call state.
'video:popout-state': {
req: z.object({
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
cameraOn: z.boolean(),
}),
res: z.null(),
},
'meeting:checkScreenPermission': {
req: z.null(),
res: z.object({