mirror of
https://github.com/rowboatlabs/rowboat.git
synced 2026-07-12 21:02:17 +02:00
screen share pop out when I switch to a different app
This commit is contained in:
parent
48a3ff6a36
commit
1edc1b17f1
10 changed files with 567 additions and 23 deletions
151
apps/x/VIDEO_MODE.md
Normal file
151
apps/x/VIDEO_MODE.md
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# Video Mode — Deep Dive
|
||||
|
||||
Video mode lets the assistant *see* the user (webcam) and their screen (screen
|
||||
share), in three presentations: frames attached to normal chat, a hands-free
|
||||
spoken call, and a full-screen Meet-style call. This doc covers the product
|
||||
flow, the technical pipeline, and the LLM prompt surface with exact pointers.
|
||||
|
||||
## Product flow
|
||||
|
||||
The composer's video button (`chat-input-with-mentions.tsx`) toggles video
|
||||
mode; a chevron dropdown picks one of three modes (`VideoChatMode`):
|
||||
|
||||
| Mode | What it does |
|
||||
|------|--------------|
|
||||
| `chat` — "Video + chat" | Camera on. Webcam (and screen-share) frames ride along with every typed or dictated message. Small PiP preview floats above the composer. |
|
||||
| `call` — "Video call (hands-free)" | Everything in `chat`, plus: continuous listening (each utterance auto-submits as a voice message) and forced full read-aloud TTS. No typing needed; composer still works. |
|
||||
| `meeting` — "Video call (full screen)" | Same pipeline as `call`, presented as a full-screen Meet-style layout: user tile + animated mascot tile, captions, control bar. |
|
||||
|
||||
On top of any mode:
|
||||
|
||||
- **Screen share** (`MonitorUp` buttons on the PiP overlay and the meeting
|
||||
control bar): captures the primary screen; frames go to the model as a
|
||||
separately labeled group. In the meeting view the screen becomes the big
|
||||
tile with user + mascot in a side rail.
|
||||
- **Camera off** (Meet-style mute): video mode and screen share keep running,
|
||||
no webcam frames are captured; tiles show a silhouette avatar.
|
||||
- **Mascot dismissal** (meeting view): swaps the animated mascot for a
|
||||
Meet-style letter avatar ("R").
|
||||
- **Popout**: while screen sharing, if the app window loses focus (the user
|
||||
switched to the app they're sharing), a small always-on-top frameless
|
||||
window pops out with the user + mascot mini-tiles; refocusing dismisses it.
|
||||
Its expand button focuses the main window (`video:focusMain`).
|
||||
|
||||
`call`/`meeting` options are disabled unless both voice input (Deepgram) and
|
||||
voice output (TTS) are configured. Entering a call saves the user's TTS
|
||||
settings and forces `full` read-aloud; leaving restores them.
|
||||
|
||||
## Frame pipeline
|
||||
|
||||
`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source
|
||||
(stream → offscreen `<video>` → canvas JPEG → ring buffer):
|
||||
|
||||
- Cadence: 1 fps (`CAPTURE_INTERVAL_MS`, line 20); ring buffer ~2 min.
|
||||
- Webcam: 512px wide, JPEG q0.65, max **12 frames/message** (lines 21, 31).
|
||||
- Screen: 1280px wide (text legibility), JPEG q0.7, max **4 frames/message**
|
||||
(lines 24, 32).
|
||||
- `collectFrames()` drains frames buffered since the last send, evenly
|
||||
sampled down to the caps, always keeping the newest; grabs one final frame
|
||||
at the moment of send. Falls back to the single latest frame for
|
||||
rapid-fire messages.
|
||||
|
||||
`App.tsx` `handlePromptSubmit` (~line 2767) attaches the drained frames to
|
||||
the outgoing message as `UserImagePart`s and sets
|
||||
`composition.videoMode: true`. Frames also become `isVideoFrame` display
|
||||
attachments (filmstrip in the transcript — `chat-message-attachments.tsx`;
|
||||
history hydration in `lib/run-to-conversation.ts`).
|
||||
|
||||
## Message schema & model encoding
|
||||
|
||||
- `packages/shared/src/message.ts:51` — `UserImagePart`: inline base64
|
||||
(`data`, `mediaType`), `source: 'camera' | 'screen'`, `capturedAt`. Unlike
|
||||
file attachments (path references read via the `LLMParse` tool), image
|
||||
parts go to the model as real multimodal image parts.
|
||||
- `packages/core/src/agents/runtime.ts` `convertFromMessages` (~line 1013):
|
||||
emits a context line (frame counts + time span), then labeled groups —
|
||||
a `"Webcam frames (oldest to newest):"` text part before camera images and
|
||||
a `"Screen-share frames (oldest to newest):"` text part before screen
|
||||
images — so the model never confuses the user with their screen.
|
||||
- Frames stay inline in history (no pruning) deliberately: pruning would
|
||||
bust provider prefix caching every turn and cost more than it saves.
|
||||
- The auto-permission classifier stringifies + truncates content to ~3KB per
|
||||
message, so inline base64 can't blow up its prompt.
|
||||
|
||||
## Hands-free voice loop
|
||||
|
||||
`apps/renderer/src/hooks/useVoiceMode.ts`:
|
||||
|
||||
- `startContinuous(onUtterance)` (line 404): push-to-talk params but with
|
||||
`endpointing=1800` (line 25) so thinking pauses don't cut the user off,
|
||||
plus `utterance_end_ms=2000` (line 38) as a second end-of-speech signal.
|
||||
**Gotcha:** Deepgram's `speech_final` usually arrives on a result with an
|
||||
EMPTY transcript — empty finals must reach the endpoint check or
|
||||
utterances never complete (see the NOTE in `ws.onmessage`).
|
||||
- `setPaused(true)` (line 414) while the assistant thinks/speaks: drops mic
|
||||
audio (so TTS is never transcribed back), discards half-heard buffer,
|
||||
sends Deepgram KeepAlives every 5s. `App.tsx` drives this from
|
||||
`activeIsProcessing || tts.state !== 'idle'`.
|
||||
- Mid-call socket drops reconnect after 1s; the offline audio backlog is
|
||||
capped (~30s).
|
||||
|
||||
Mode transitions live in `App.tsx` `handleVideoModeChange` (~line 1161):
|
||||
call ↔ meeting switches are presentation-only (mic/TTS untouched);
|
||||
entering/leaving hands-free saves/restores TTS settings. Push-to-talk is
|
||||
disabled while a call owns the mic.
|
||||
|
||||
## Popout window
|
||||
|
||||
- Renderer asks `video:setPopout {show}` (main handler:
|
||||
`apps/main/src/ipc.ts:1742`); main creates a frameless, `alwaysOnTop`
|
||||
('floating'), all-workspaces BrowserWindow at the top-right of the primary
|
||||
display, loading the renderer bundle with `#video-popout`
|
||||
(`apps/renderer/src/main.tsx` branches on the hash →
|
||||
`components/video-popout.tsx`).
|
||||
- Call state streams over the `video:popout-state` push channel; main caches
|
||||
the last payload and replays it on popout load. Shown with
|
||||
`showInactive()` so it never steals focus (that would re-hide it).
|
||||
- The popout captures its **own** camera preview (MediaStreams can't cross
|
||||
windows) and synthesizes the mascot mouth level (no audio in that window).
|
||||
- `video:focusMain` matches only real app windows by URL — `getAllWindows()`
|
||||
also contains hidden utility windows (PDF export) that must not be shown.
|
||||
|
||||
## Permissions
|
||||
|
||||
- Camera: `voice:ensureCameraAccess` settles the macOS TCC prompt before
|
||||
`getUserMedia` (same pattern as the mic). `NSCameraUsageDescription` is in
|
||||
`forge.config.cjs` `extendInfo`.
|
||||
- Screen: `getDisplayMedia` is auto-approved with the primary screen by
|
||||
`setDisplayMediaRequestHandler` in `main.ts` (no picker);
|
||||
`meeting:checkScreenPermission` registers the app in macOS Screen
|
||||
Recording settings on first use.
|
||||
|
||||
## LLM prompts catalog
|
||||
|
||||
| Prompt | Where |
|
||||
|--------|-------|
|
||||
| `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) |
|
||||
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts:~1013` (`convertFromMessages`) |
|
||||
| `videoMode` composition override (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts:57,125`; set from `App.tsx` `sendConfig` |
|
||||
|
||||
Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are
|
||||
reused untouched — calls set `voiceInput` per utterance and force
|
||||
`voiceOutput: 'full'`.
|
||||
|
||||
## Cost notes
|
||||
|
||||
Webcam frames ≈ 250–350 tokens each (≤12/message ≈ 3–4k); screen frames ≈
|
||||
1.5–2k tokens each (≤4/message ≈ 6–8k). History keeps frames inline, so long
|
||||
sessions grow but stay prefix-cached. First lever if cost bites: drop to one
|
||||
screen frame per message unless the screen changed.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- Turn-taking is strict — no barge-in (would need echo cancellation against
|
||||
TTS output).
|
||||
- Frame sampling, not video: motion between frames is invisible (the prompt
|
||||
tells the model not to claim otherwise).
|
||||
- Vocal-delivery feedback is limited: Deepgram reduces speech to text, so
|
||||
"energy" coaching leans on visual cues.
|
||||
- Screen share always captures the primary display (no window/display
|
||||
picker yet).
|
||||
- The meeting view covers the chat; there's no in-call transcript drawer.
|
||||
|
|
@ -1,7 +1,8 @@
|
|||
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app } from 'electron';
|
||||
import { ipcMain, BrowserWindow, shell, dialog, systemPreferences, desktopCapturer, app, screen } from 'electron';
|
||||
import { ipc } from '@x/shared';
|
||||
import path from 'node:path';
|
||||
import os from 'node:os';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import {
|
||||
connectProvider,
|
||||
disconnectProvider,
|
||||
|
|
@ -381,9 +382,19 @@ type InvokeHandlers = {
|
|||
[K in InvokeChannels]: InvokeHandler<K>;
|
||||
};
|
||||
|
||||
// Video-mode popout window (shown while screen sharing when the app loses
|
||||
// focus) and the last call state pushed by the main window — replayed to the
|
||||
// popout when it finishes loading.
|
||||
let videoPopoutWin: BrowserWindow | null = null;
|
||||
let lastVideoPopoutState: {
|
||||
ttsState: 'idle' | 'synthesizing' | 'speaking';
|
||||
status: 'listening' | 'thinking' | 'speaking' | null;
|
||||
cameraOn: boolean;
|
||||
} | null = null;
|
||||
|
||||
/**
|
||||
* Register all IPC handlers with type safety and runtime validation
|
||||
*
|
||||
*
|
||||
* This function ensures:
|
||||
* 1. All invoke channels have handlers (exhaustiveness checking)
|
||||
* 2. Handler signatures match channel definitions
|
||||
|
|
@ -1728,6 +1739,87 @@ export function setupIpcHandlers() {
|
|||
return { granted: false };
|
||||
}
|
||||
},
|
||||
'video:setPopout': async (_event, args) => {
|
||||
if (!args.show) {
|
||||
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) videoPopoutWin.destroy();
|
||||
videoPopoutWin = null;
|
||||
return {};
|
||||
}
|
||||
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) return {};
|
||||
|
||||
const workArea = screen.getPrimaryDisplay().workArea;
|
||||
const width = 340;
|
||||
const height = 148;
|
||||
const ipcDir = path.dirname(fileURLToPath(import.meta.url));
|
||||
const preloadPath = app.isPackaged
|
||||
? path.join(ipcDir, '../preload/dist/preload.js')
|
||||
: path.join(ipcDir, '../../../preload/dist/preload.js');
|
||||
const win = new BrowserWindow({
|
||||
width,
|
||||
height,
|
||||
x: workArea.x + workArea.width - width - 24,
|
||||
y: workArea.y + 24,
|
||||
frame: false,
|
||||
resizable: false,
|
||||
alwaysOnTop: true,
|
||||
skipTaskbar: true,
|
||||
show: false,
|
||||
hasShadow: true,
|
||||
backgroundColor: '#171717',
|
||||
webPreferences: {
|
||||
nodeIntegration: false,
|
||||
contextIsolation: true,
|
||||
sandbox: true,
|
||||
preload: preloadPath,
|
||||
},
|
||||
});
|
||||
// Float above other apps (and fullscreen spaces on macOS) — the whole
|
||||
// point is being visible while the user works elsewhere.
|
||||
win.setAlwaysOnTop(true, 'floating');
|
||||
win.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true });
|
||||
win.webContents.once('did-finish-load', () => {
|
||||
if (lastVideoPopoutState) {
|
||||
win.webContents.send('video:popout-state', lastVideoPopoutState);
|
||||
}
|
||||
// showInactive: appearing must not steal focus from the app the user
|
||||
// switched to — that would immediately re-hide the popout.
|
||||
if (!win.isDestroyed()) win.showInactive();
|
||||
});
|
||||
win.on('closed', () => {
|
||||
if (videoPopoutWin === win) videoPopoutWin = null;
|
||||
});
|
||||
videoPopoutWin = win;
|
||||
if (app.isPackaged) {
|
||||
win.loadURL('app://-/index.html#video-popout');
|
||||
} else {
|
||||
win.loadURL('http://localhost:5173/#video-popout');
|
||||
}
|
||||
return {};
|
||||
},
|
||||
'video:popoutState': async (_event, args) => {
|
||||
lastVideoPopoutState = args;
|
||||
if (videoPopoutWin && !videoPopoutWin.isDestroyed()) {
|
||||
videoPopoutWin.webContents.send('video:popout-state', args);
|
||||
}
|
||||
return {};
|
||||
},
|
||||
'video:focusMain': async () => {
|
||||
// Match only real app windows — getAllWindows() can also contain the
|
||||
// popout itself and hidden utility windows (e.g. PDF-export renderers),
|
||||
// which must not be shown or focused.
|
||||
const main = BrowserWindow.getAllWindows().find((w) => {
|
||||
if (w === videoPopoutWin || w.isDestroyed()) return false;
|
||||
const url = w.webContents.getURL();
|
||||
const isAppWindow = url.startsWith('app://') || url.startsWith('http://localhost');
|
||||
return isAppWindow && !url.includes('#video-popout');
|
||||
});
|
||||
if (main) {
|
||||
if (main.isMinimized()) main.restore();
|
||||
main.show();
|
||||
main.focus();
|
||||
}
|
||||
return {};
|
||||
},
|
||||
// Live-note handlers
|
||||
'live-note:run': async (_event, args) => {
|
||||
const result = await runLiveNoteAgent(args.filePath, 'manual', args.context);
|
||||
|
|
|
|||
|
|
@ -1231,6 +1231,57 @@ function App() {
|
|||
}
|
||||
}, [video])
|
||||
|
||||
// Meet-style camera mute: video mode (and any screen share) stays on, but
|
||||
// no webcam frames are captured while the camera is off.
|
||||
const handleToggleCamera = useCallback(() => {
|
||||
void video.setCameraEnabled(!video.cameraOn)
|
||||
}, [video])
|
||||
|
||||
// Current phase of a hands-free call (null outside call/meeting modes).
|
||||
const videoCallStatus: 'listening' | 'thinking' | 'speaking' | null =
|
||||
videoChatMode === 'call' || videoChatMode === 'meeting'
|
||||
? tts.state === 'speaking'
|
||||
? 'speaking'
|
||||
: tts.state === 'synthesizing' || activeIsProcessing
|
||||
? 'thinking'
|
||||
: 'listening'
|
||||
: null
|
||||
|
||||
// Meet-style popout: while screen sharing, losing app focus (the user
|
||||
// switched to the app they're sharing) pops the mini-call out into an
|
||||
// always-on-top window; refocusing Rowboat dismisses it. The short show
|
||||
// delay keeps a quick cmd-tab pass-through from flashing a window.
|
||||
const [appFocused, setAppFocused] = useState(true)
|
||||
useEffect(() => {
|
||||
const onFocus = () => setAppFocused(true)
|
||||
const onBlur = () => setAppFocused(false)
|
||||
window.addEventListener('focus', onFocus)
|
||||
window.addEventListener('blur', onBlur)
|
||||
return () => {
|
||||
window.removeEventListener('focus', onFocus)
|
||||
window.removeEventListener('blur', onBlur)
|
||||
}
|
||||
}, [])
|
||||
useEffect(() => {
|
||||
const shouldShow = videoChatMode !== 'off' && video.screenState === 'live' && !appFocused
|
||||
if (shouldShow) {
|
||||
const timer = setTimeout(() => {
|
||||
void window.ipc.invoke('video:setPopout', { show: true }).catch(() => {})
|
||||
}, 300)
|
||||
return () => clearTimeout(timer)
|
||||
}
|
||||
void window.ipc.invoke('video:setPopout', { show: false }).catch(() => {})
|
||||
}, [videoChatMode, video.screenState, appFocused])
|
||||
|
||||
// Keep the popout's mascot/status/camera mirror of the call fresh. The main
|
||||
// process caches the latest state and replays it when the popout loads.
|
||||
useEffect(() => {
|
||||
if (videoChatMode === 'off') return
|
||||
void window.ipc
|
||||
.invoke('video:popoutState', { ttsState: tts.state, status: videoCallStatus, cameraOn: video.cameraOn })
|
||||
.catch(() => {})
|
||||
}, [videoChatMode, tts.state, videoCallStatus, video.cameraOn])
|
||||
|
||||
// Enter to submit voice input, Escape to cancel
|
||||
useEffect(() => {
|
||||
const handleKeyDown = (e: KeyboardEvent) => {
|
||||
|
|
@ -6611,6 +6662,8 @@ function App() {
|
|||
interimText={videoChatMode === 'call' ? voice.interimText : undefined}
|
||||
isScreenSharing={video.screenState === 'live'}
|
||||
onToggleScreenShare={handleToggleScreenShare}
|
||||
cameraOn={video.cameraOn}
|
||||
onToggleCamera={handleToggleCamera}
|
||||
/>
|
||||
)}
|
||||
{/* Full-screen Meet-style call: user tile + animated mascot tile */}
|
||||
|
|
@ -6620,6 +6673,8 @@ function App() {
|
|||
screenStreamRef={video.screenStreamRef}
|
||||
isScreenSharing={video.screenState === 'live'}
|
||||
onToggleScreenShare={handleToggleScreenShare}
|
||||
cameraOn={video.cameraOn}
|
||||
onToggleCamera={handleToggleCamera}
|
||||
ttsState={tts.state}
|
||||
getTtsLevel={tts.getLevel}
|
||||
status={
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { useEffect, useRef, useState } from 'react'
|
||||
import { MonitorUp, PhoneOff } from 'lucide-react'
|
||||
import { MonitorUp, PhoneOff, User, Video, VideoOff } from 'lucide-react'
|
||||
|
||||
import { MascotFaceIcon, TalkingHead } from '@/components/talking-head'
|
||||
import type { TTSState } from '@/hooks/useVoiceTTS'
|
||||
|
|
@ -14,6 +14,8 @@ interface VideoCallViewProps {
|
|||
screenStreamRef: React.MutableRefObject<MediaStream | null>
|
||||
isScreenSharing: boolean
|
||||
onToggleScreenShare: () => void
|
||||
cameraOn: boolean
|
||||
onToggleCamera: () => void
|
||||
ttsState: TTSState
|
||||
/** Live TTS output level — drives the mascot's mouth animation. */
|
||||
getTtsLevel: () => number
|
||||
|
|
@ -76,6 +78,8 @@ export function VideoCallView({
|
|||
screenStreamRef,
|
||||
isScreenSharing,
|
||||
onToggleScreenShare,
|
||||
cameraOn,
|
||||
onToggleCamera,
|
||||
ttsState,
|
||||
getTtsLevel,
|
||||
status,
|
||||
|
|
@ -102,7 +106,19 @@ export function VideoCallView({
|
|||
isScreenSharing && 'aspect-video w-full'
|
||||
)}
|
||||
>
|
||||
<StreamVideo streamRef={streamRef} mirrored className="h-full w-full object-cover" />
|
||||
{cameraOn ? (
|
||||
<StreamVideo streamRef={streamRef} mirrored className="h-full w-full object-cover" />
|
||||
) : (
|
||||
<span
|
||||
className={cn(
|
||||
'flex items-center justify-center rounded-full bg-neutral-700 text-neutral-400',
|
||||
isScreenSharing ? 'h-16 w-16' : 'h-40 w-40'
|
||||
)}
|
||||
aria-label="Camera off"
|
||||
>
|
||||
<User className={isScreenSharing ? 'h-8 w-8' : 'h-20 w-20'} />
|
||||
</span>
|
||||
)}
|
||||
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
|
||||
You
|
||||
</span>
|
||||
|
|
@ -182,6 +198,20 @@ export function VideoCallView({
|
|||
<span className={cn('block h-2 w-2 rounded-full', STATUS_DISPLAY[status].dotClass)} />
|
||||
{STATUS_DISPLAY[status].label}
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onToggleCamera}
|
||||
className={cn(
|
||||
'flex h-10 w-10 items-center justify-center rounded-full transition-colors',
|
||||
cameraOn
|
||||
? 'bg-neutral-800 text-white/90 hover:bg-neutral-700'
|
||||
: 'bg-red-600 text-white hover:bg-red-500'
|
||||
)}
|
||||
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
|
||||
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
|
||||
>
|
||||
{cameraOn ? <Video className="h-5 w-5" /> : <VideoOff className="h-5 w-5" />}
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onToggleScreenShare}
|
||||
|
|
|
|||
117
apps/x/apps/renderer/src/components/video-popout.tsx
Normal file
117
apps/x/apps/renderer/src/components/video-popout.tsx
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||
import { Maximize2, User } from 'lucide-react'
|
||||
|
||||
import { TalkingHead } from '@/components/talking-head'
|
||||
|
||||
type PopoutState = {
|
||||
ttsState: 'idle' | 'synthesizing' | 'speaking'
|
||||
status: 'listening' | 'thinking' | 'speaking' | null
|
||||
cameraOn: boolean
|
||||
}
|
||||
|
||||
const STATUS_DISPLAY: Record<NonNullable<PopoutState['status']>, { label: string; dotClass: string }> = {
|
||||
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
|
||||
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
|
||||
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
|
||||
}
|
||||
|
||||
const dragRegion = { WebkitAppRegion: 'drag' } as React.CSSProperties
|
||||
const noDragRegion = { WebkitAppRegion: 'no-drag' } as React.CSSProperties
|
||||
|
||||
/**
|
||||
* Content of the always-on-top popout window shown while screen sharing when
|
||||
* the main app window loses focus (Meet-style floating mini-call). Rendered
|
||||
* in its own BrowserWindow (see `video:setPopout` in the main process); call
|
||||
* state streams in over the `video:popout-state` push channel. Captures its
|
||||
* own webcam feed — MediaStreams can't cross windows.
|
||||
*/
|
||||
export function VideoPopout() {
|
||||
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: true })
|
||||
const videoRef = useRef<HTMLVideoElement | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
return window.ipc.on('video:popout-state', (next) => setState(next))
|
||||
}, [])
|
||||
|
||||
// Own camera feed, following the main window's camera-on/off state.
|
||||
useEffect(() => {
|
||||
if (!state.cameraOn) return
|
||||
let stream: MediaStream | null = null
|
||||
let cancelled = false
|
||||
navigator.mediaDevices
|
||||
.getUserMedia({ video: { width: { ideal: 640 }, facingMode: 'user' }, audio: false })
|
||||
.then((s) => {
|
||||
if (cancelled) {
|
||||
s.getTracks().forEach((t) => t.stop())
|
||||
return
|
||||
}
|
||||
stream = s
|
||||
if (videoRef.current) {
|
||||
videoRef.current.srcObject = s
|
||||
videoRef.current.play().catch(() => {})
|
||||
}
|
||||
})
|
||||
.catch((err) => console.error('[popout] camera failed:', err))
|
||||
return () => {
|
||||
cancelled = true
|
||||
stream?.getTracks().forEach((t) => t.stop())
|
||||
if (videoRef.current) videoRef.current.srcObject = null
|
||||
}
|
||||
}, [state.cameraOn])
|
||||
|
||||
// The popout has no TTS audio pipeline — synthesize a plausible mouth level
|
||||
// so the mascot still animates while the assistant speaks in the main window.
|
||||
const getLevel = useCallback(() => 0.45 + 0.35 * Math.sin(performance.now() / 90), [])
|
||||
|
||||
const statusDisplay = state.status ? STATUS_DISPLAY[state.status] : null
|
||||
|
||||
return (
|
||||
<div
|
||||
className="flex h-screen w-screen select-none gap-1.5 bg-neutral-900 p-1.5"
|
||||
style={dragRegion}
|
||||
>
|
||||
<div className="relative flex-1 overflow-hidden rounded-lg bg-neutral-800">
|
||||
{state.cameraOn ? (
|
||||
<video
|
||||
ref={videoRef}
|
||||
muted
|
||||
playsInline
|
||||
className="h-full w-full object-cover"
|
||||
style={{ transform: 'scaleX(-1)' }}
|
||||
/>
|
||||
) : (
|
||||
<div className="flex h-full w-full items-center justify-center">
|
||||
<span className="flex h-14 w-14 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
|
||||
<User className="h-7 w-7" />
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
|
||||
You
|
||||
</span>
|
||||
</div>
|
||||
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-lg bg-neutral-800">
|
||||
<TalkingHead ttsState={state.ttsState} getLevel={getLevel} size={92} />
|
||||
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
|
||||
Rowboat
|
||||
</span>
|
||||
{statusDisplay && (
|
||||
<span className="absolute right-1.5 top-1.5 flex items-center gap-1 rounded-full bg-black/50 px-1.5 py-0.5 text-[10px] font-medium text-white">
|
||||
<span className={`block h-1.5 w-1.5 rounded-full ${statusDisplay.dotClass}`} />
|
||||
{statusDisplay.label}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => void window.ipc.invoke('video:focusMain', null)}
|
||||
className="absolute left-1.5 top-1.5 flex h-5 w-5 items-center justify-center rounded bg-black/50 text-white/80 hover:text-white"
|
||||
style={noDragRegion}
|
||||
aria-label="Back to Rowboat"
|
||||
title="Back to Rowboat"
|
||||
>
|
||||
<Maximize2 className="h-3 w-3" />
|
||||
</button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
import { useEffect, useRef } from 'react'
|
||||
import { MonitorUp, X } from 'lucide-react'
|
||||
import { MonitorUp, User, VideoOff, X } from 'lucide-react'
|
||||
|
||||
interface VideoPreviewOverlayProps {
|
||||
/** Live camera stream from useVideoMode — attached to the preview element. */
|
||||
|
|
@ -11,6 +11,8 @@ interface VideoPreviewOverlayProps {
|
|||
interimText?: string
|
||||
isScreenSharing?: boolean
|
||||
onToggleScreenShare?: () => void
|
||||
cameraOn?: boolean
|
||||
onToggleCamera?: () => void
|
||||
}
|
||||
|
||||
const CALL_STATUS_DISPLAY: Record<NonNullable<VideoPreviewOverlayProps['callStatus']>, { label: string; dotClass: string }> = {
|
||||
|
|
@ -24,10 +26,11 @@ const CALL_STATUS_DISPLAY: Record<NonNullable<VideoPreviewOverlayProps['callStat
|
|||
* on. Mirrored like a selfie camera so the user's movements feel natural.
|
||||
* Sits above the composer dock, mirroring the talking-head overlay's corner.
|
||||
*/
|
||||
export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimText, isScreenSharing, onToggleScreenShare }: VideoPreviewOverlayProps) {
|
||||
export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimText, isScreenSharing, onToggleScreenShare, cameraOn = true, onToggleCamera }: VideoPreviewOverlayProps) {
|
||||
const videoRef = useRef<HTMLVideoElement | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
if (!cameraOn) return
|
||||
const videoEl = videoRef.current
|
||||
if (!videoEl) return
|
||||
videoEl.srcObject = streamRef.current
|
||||
|
|
@ -35,7 +38,7 @@ export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimT
|
|||
return () => {
|
||||
videoEl.srcObject = null
|
||||
}
|
||||
}, [streamRef])
|
||||
}, [streamRef, cameraOn])
|
||||
|
||||
return (
|
||||
<div
|
||||
|
|
@ -71,13 +74,36 @@ export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimT
|
|||
<MonitorUp className="h-3 w-3" />
|
||||
</button>
|
||||
)}
|
||||
<video
|
||||
ref={videoRef}
|
||||
muted
|
||||
playsInline
|
||||
className="h-32 w-auto rounded-xl border border-border/70 bg-black shadow-lg"
|
||||
style={{ transform: 'scaleX(-1)' }}
|
||||
/>
|
||||
{cameraOn ? (
|
||||
<video
|
||||
ref={videoRef}
|
||||
muted
|
||||
playsInline
|
||||
className="h-32 w-auto rounded-xl border border-border/70 bg-black shadow-lg"
|
||||
style={{ transform: 'scaleX(-1)' }}
|
||||
/>
|
||||
) : (
|
||||
<div className="flex h-32 w-48 items-center justify-center rounded-xl border border-border/70 bg-black shadow-lg">
|
||||
<span className="flex h-14 w-14 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
|
||||
<User className="h-7 w-7" />
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
{onToggleCamera && (
|
||||
<button
|
||||
type="button"
|
||||
onClick={onToggleCamera}
|
||||
className={`absolute -left-1.5 top-4 z-10 flex h-5 w-5 items-center justify-center rounded-full border shadow-sm transition-opacity ${
|
||||
cameraOn
|
||||
? 'border-border bg-background text-muted-foreground opacity-0 hover:text-foreground group-hover:opacity-100'
|
||||
: 'border-red-500 bg-red-600 text-white opacity-100'
|
||||
}`}
|
||||
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
|
||||
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
|
||||
>
|
||||
<VideoOff className="h-3 w-3" />
|
||||
</button>
|
||||
)}
|
||||
<span className="pointer-events-none absolute bottom-1.5 left-1.5 flex items-center gap-1 rounded-full bg-black/60 px-1.5 py-0.5 text-[10px] font-medium text-white">
|
||||
{callStatus ? (
|
||||
<>
|
||||
|
|
|
|||
|
|
@ -151,6 +151,9 @@ function drainPipe(pipe: CapturePipe, max: number, source: CapturedVideoFrame['s
|
|||
export function useVideoMode() {
|
||||
const [state, setState] = useState<VideoModeState>('idle');
|
||||
const [screenState, setScreenState] = useState<ScreenShareState>('idle');
|
||||
// Camera can be turned off mid-session (Meet-style) while the mode — and
|
||||
// any screen share — keeps running. Resets to on for the next session.
|
||||
const [cameraOn, setCameraOn] = useState(true);
|
||||
const cameraPipeRef = useRef<CapturePipe>(emptyPipe());
|
||||
const screenPipeRef = useRef<CapturePipe>(emptyPipe());
|
||||
// Stable stream refs for preview components (<video srcObject>).
|
||||
|
|
@ -179,13 +182,13 @@ export function useVideoMode() {
|
|||
teardownPipe(cameraPipeRef.current);
|
||||
streamRef.current = null;
|
||||
setState('idle');
|
||||
setCameraOn(true);
|
||||
stopScreenShare();
|
||||
}, [stopScreenShare]);
|
||||
|
||||
const start = useCallback(async (): Promise<boolean> => {
|
||||
if (stateRef.current !== 'idle') return true;
|
||||
setState('starting');
|
||||
|
||||
// Acquire the webcam and start its capture pipeline. Shared by start()
|
||||
// and by re-enabling the camera mid-session.
|
||||
const acquireCamera = useCallback(async (): Promise<boolean> => {
|
||||
// Settle the macOS TCC camera permission before getUserMedia, same as
|
||||
// voice mode does for the mic — otherwise the first click silently
|
||||
// fails while the native prompt is still up.
|
||||
|
|
@ -194,7 +197,6 @@ export function useVideoMode() {
|
|||
.catch(() => ({ granted: true }));
|
||||
if (!access.granted) {
|
||||
console.error('[video] Camera access denied');
|
||||
setState('idle');
|
||||
return false;
|
||||
}
|
||||
|
||||
|
|
@ -206,16 +208,45 @@ export function useVideoMode() {
|
|||
});
|
||||
} catch (err) {
|
||||
console.error('[video] Camera access failed:', err);
|
||||
setState('idle');
|
||||
return false;
|
||||
}
|
||||
|
||||
streamRef.current = stream;
|
||||
attachPipeSource(cameraPipeRef.current, stream, captureCameraFrame);
|
||||
setState('live');
|
||||
return true;
|
||||
}, [captureCameraFrame]);
|
||||
|
||||
/**
|
||||
* Turn the camera off/on without leaving video mode (Meet-style). While
|
||||
* off, no webcam frames are captured or attached; screen-share frames
|
||||
* (if presenting) keep flowing.
|
||||
*/
|
||||
const setCameraEnabled = useCallback(async (enabled: boolean): Promise<boolean> => {
|
||||
if (stateRef.current !== 'live') return false;
|
||||
if (enabled) {
|
||||
const ok = await acquireCamera();
|
||||
if (ok) setCameraOn(true);
|
||||
return ok;
|
||||
}
|
||||
teardownPipe(cameraPipeRef.current);
|
||||
streamRef.current = null;
|
||||
setCameraOn(false);
|
||||
return true;
|
||||
}, [acquireCamera]);
|
||||
|
||||
const start = useCallback(async (): Promise<boolean> => {
|
||||
if (stateRef.current !== 'idle') return true;
|
||||
setState('starting');
|
||||
const ok = await acquireCamera();
|
||||
if (!ok) {
|
||||
setState('idle');
|
||||
return false;
|
||||
}
|
||||
setCameraOn(true);
|
||||
setState('live');
|
||||
return true;
|
||||
}, [acquireCamera]);
|
||||
|
||||
/**
|
||||
* Share the screen. The main process auto-approves getDisplayMedia with
|
||||
* the primary screen (see setDisplayMediaRequestHandler in main.ts), so
|
||||
|
|
@ -270,5 +301,5 @@ export function useVideoMode() {
|
|||
// Release the camera/screen if the component unmounts with video mode on.
|
||||
useEffect(() => stop, [stop]);
|
||||
|
||||
return { state, screenState, streamRef, screenStreamRef, start, stop, startScreenShare, stopScreenShare, collectFrames };
|
||||
return { state, screenState, cameraOn, streamRef, screenStreamRef, start, stop, startScreenShare, stopScreenShare, setCameraEnabled, collectFrames };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import { PostHogProvider } from 'posthog-js/react'
|
|||
import type { CaptureResult } from 'posthog-js'
|
||||
import { ThemeProvider } from '@/contexts/theme-context'
|
||||
import { configureAnalyticsContext } from './lib/analytics'
|
||||
import { VideoPopout } from '@/components/video-popout'
|
||||
|
||||
// Fetch the stable installation ID from main so renderer + main share one
|
||||
// PostHog distinct_id. Falls back to PostHog's auto-generated anonymous ID
|
||||
|
|
@ -57,4 +58,14 @@ async function bootstrap() {
|
|||
// The loaded callback applies api_url/app_version once PostHog has initialized.
|
||||
}
|
||||
|
||||
bootstrap()
|
||||
// The video-mode popout window loads the same bundle with a hash route and
|
||||
// renders only the mini-call UI — no analytics or app bootstrap needed.
|
||||
if (window.location.hash === '#video-popout') {
|
||||
createRoot(document.getElementById('root')!).render(
|
||||
<StrictMode>
|
||||
<VideoPopout />
|
||||
</StrictMode>,
|
||||
)
|
||||
} else {
|
||||
bootstrap()
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1380,6 +1380,36 @@ const ipcSchemas = {
|
|||
granted: z.boolean(),
|
||||
}),
|
||||
},
|
||||
// Video-mode popout: show/hide a small always-on-top window (user + mascot
|
||||
// tiles) while screen sharing when the main window loses focus, Meet-style.
|
||||
'video:setPopout': {
|
||||
req: z.object({ show: z.boolean() }),
|
||||
res: z.object({}),
|
||||
},
|
||||
// Main-window renderer pushes the current call state; the main process
|
||||
// caches it and relays to the popout window (replayed on popout load).
|
||||
'video:popoutState': {
|
||||
req: z.object({
|
||||
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
|
||||
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
|
||||
cameraOn: z.boolean(),
|
||||
}),
|
||||
res: z.object({}),
|
||||
},
|
||||
// Popout window → bring the main app window back to the foreground.
|
||||
'video:focusMain': {
|
||||
req: z.null(),
|
||||
res: z.object({}),
|
||||
},
|
||||
// Push channel: main → popout window with the latest call state.
|
||||
'video:popout-state': {
|
||||
req: z.object({
|
||||
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
|
||||
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
|
||||
cameraOn: z.boolean(),
|
||||
}),
|
||||
res: z.null(),
|
||||
},
|
||||
'meeting:checkScreenPermission': {
|
||||
req: z.null(),
|
||||
res: z.object({
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue