mirror of
https://github.com/rowboatlabs/rowboat.git
synced 2026-07-12 21:02:17 +02:00
popout and full screen work as expected
This commit is contained in:
Arjun
parent
d2501c4f4d
commit
e9ee8fd975
15 changed files with 563 additions and 696 deletions
|
|
@ -109,7 +109,7 @@ Long-form docs for specific features. Read the relevant file before making chang
|
|||
| Feature | Doc |
|
||||
|---------|-----|
|
||||
| Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` |
|
||||
| Video mode — webcam + screen-share frames to the model (video+chat / hands-free call / full-screen Meet-style call), continuous voice loop, camera mute, popout window, prompts catalog | `apps/x/VIDEO_MODE.md` |
|
||||
| Calls (video mode) — one hands-free call engine with four presets (voice / video / share screen / practice coaching), device-derived surfaces (full-screen ⇄ floating popout), frame pipeline, prompts catalog | `apps/x/VIDEO_MODE.md` |
|
||||
| Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` |
|
||||
| Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` |
|
||||
|
||||
|
|
|
|||
|
|
@ -92,6 +92,7 @@ All in `apps/renderer/src/lib/analytics.ts`:
|
|||
- `chat_message_sent` — `{ voice_input, voice_output, search_enabled }`
|
||||
- `oauth_connected` / `oauth_disconnected` — `{ provider }`
|
||||
- `voice_input_started` — no properties
|
||||
- `call_started` — `{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`)
|
||||
- `search_executed` — `{ types: string[] }`
|
||||
- `note_exported` — `{ format }`
|
||||
|
||||
|
|
|
|||
|
|
@ -1,46 +1,70 @@
|
|||
# Video Mode — Deep Dive
|
||||
# Calls (Video Mode) — Deep Dive
|
||||
|
||||
Video mode lets the assistant *see* the user (webcam) and their screen (screen
|
||||
share), in three presentations: frames attached to normal chat, a hands-free
|
||||
spoken call, and a full-screen Meet-style call. This doc covers the product
|
||||
flow, the technical pipeline, and the LLM prompt surface with exact pointers.
|
||||
Calls let the user talk to the assistant hands-free while it *sees* them
|
||||
(webcam) and their screen (screen share). There is ONE call engine —
|
||||
continuous listening, auto-submitted utterances, forced read-aloud TTS, frame
|
||||
capture — entered through four presets that differ only in starting devices.
|
||||
This doc covers the product flow, the technical pipeline, and the LLM prompt
|
||||
surface with exact pointers.
|
||||
|
||||
## Product flow
|
||||
|
||||
The composer's video button (`chat-input-with-mentions.tsx`) toggles video
|
||||
mode; a chevron dropdown picks one of three modes (`VideoChatMode`):
|
||||
The composer has a **call split-button** (`chat-input-with-mentions.tsx`).
|
||||
The main click is the "work together" default — preset `share`: screen
|
||||
sharing ON, camera OFF, floating pill, so the user keeps working while the
|
||||
assistant watches along (the button tooltip discloses the screen share). The
|
||||
chevron menu holds the deviations. While a call is live the button turns red
|
||||
and ends it.
|
||||
|
||||
| Mode | What it does |
|
||||
|------|--------------|
|
||||
| `chat` — "Video + chat" | Camera on. Webcam (and screen-share) frames ride along with every typed or dictated message. Small PiP preview floats above the composer. |
|
||||
| `call` — "Video call (hands-free)" | Everything in `chat`, plus: continuous listening (each utterance auto-submits as a voice message) and forced full read-aloud TTS. No typing needed; composer still works. |
|
||||
| `meeting` — "Video call (full screen)" | Same pipeline as `call`, presented as a full-screen Meet-style layout: user tile + animated mascot tile, captions, control bar. |
|
||||
| Preset | Starting devices | First surface |
|
||||
|--------|------------------|---------------|
|
||||
| `share` — main click | screen on, camera off | floating pill |
|
||||
| `voice` — "Voice call" | camera off, screen off | floating mascot pill |
|
||||
| `video` — "Video call" | camera on | full-screen call |
|
||||
| `practice` — "Practice session" | camera on, + coaching persona | full-screen call |
|
||||
|
||||
On top of any mode:
|
||||
**One surface rule** (`callSurface` in `App.tsx`): full screen and screen
|
||||
sharing are mutually exclusive in both directions — a full-screen call covers
|
||||
the screen, so sharing it would show the call itself.
|
||||
|
||||
- **Screen share** (`MonitorUp` buttons on the PiP overlay and the meeting
|
||||
control bar): captures the primary screen; frames go to the model as a
|
||||
separately labeled group. In the meeting view the screen becomes the big
|
||||
tile with user + mascot in a side rail.
|
||||
- **Camera off** (Meet-style mute): video mode and screen share keep running,
|
||||
no webcam frames are captured; tiles show a silhouette avatar.
|
||||
- **Mascot dismissal** (meeting view): swaps the animated mascot for a
|
||||
Meet-style letter avatar ("R").
|
||||
- **Popout**: starting a screen share immediately pops the mini-call out into
|
||||
a small always-on-top frameless window (user + mascot tiles, live caption,
|
||||
control bar) that floats over every app — including Rowboat itself, since
|
||||
the shared screen may be Rowboat. It stays for the whole share (surviving
|
||||
app switches) and disappears when sharing stops or video mode ends. While
|
||||
sharing it is THE call surface: the in-app PiP hides, and presenting from
|
||||
the full-screen meeting collapses the meeting into it (restored when the
|
||||
share ends). Control-bar actions (camera toggle / stop share / end call)
|
||||
round-trip `video:popoutAction` → main → `video:popout-action` → app
|
||||
window, which owns the mic/camera/capture. The expand button focuses the
|
||||
main window (`video:focusMain`).
|
||||
- sharing → floating popout, always (pill = working)
|
||||
- not sharing → full screen unless `callMinimized` (full screen = facing
|
||||
each other)
|
||||
- expanding the pill auto-STOPS any share; minimizing the full-screen call
|
||||
auto-STARTS one (the pill exists to work together) — presenting from full
|
||||
screen likewise collapses to the pill
|
||||
- the camera toggle never changes the surface: turning it on from the pill
|
||||
puts your video IN the pill; expanding is its own explicit action
|
||||
|
||||
`call`/`meeting` options are disabled unless both voice input (Deepgram) and
|
||||
voice output (TTS) are configured. Entering a call saves the user's TTS
|
||||
settings and forces `full` read-aloud; leaving restores them.
|
||||
**Screen-share consent** is three-layered: a toast the moment any share
|
||||
starts ("Your screen is being shared… [Stop sharing]"), a persistent
|
||||
"Sharing screen" badge on the pill, and macOS's purple recording indicator.
|
||||
If the auto-share fails (Screen Recording permission not granted) the call
|
||||
starts anyway as a voice call, with a toast linking to System Settings.
|
||||
Practice/coaching is always an explicit choice — expanding to full screen
|
||||
never turns the coach on.
|
||||
|
||||
In-call controls (identical bar on both surfaces): camera toggle (silhouette
|
||||
avatar while off, no webcam frames captured), screen share toggle, mascot ⇄
|
||||
"R" letter avatar, end call. Captions of the in-progress utterance and the
|
||||
assistant's spoken line run along the bottom. Typing in the composer still
|
||||
works mid-call; frames ride along with typed messages too.
|
||||
|
||||
Outside calls the composer keeps exactly two voice affordances: the **mic
|
||||
button** (push-to-talk dictation, untouched) and the **read-aloud toggle**
|
||||
(headphones; summary-style for typed chat — calls force full read-aloud and
|
||||
restore the prior setting on hang-up). The old video dropdown, talking-head
|
||||
toggle, and summary/full TTS dropdown are retired.
|
||||
|
||||
The call button is disabled unless both voice input (Deepgram) and voice
|
||||
output (TTS) are configured. `call_started` (with `preset`) is captured in
|
||||
PostHog — the adoption metric for this feature.
|
||||
|
||||
**Popout mechanics**: a small always-on-top frameless window (camera tile
|
||||
when on + mascot tile, live caption, control bar) floating over every app —
|
||||
including Rowboat. Control-bar actions round-trip `video:popoutAction` →
|
||||
main → `video:popout-action` → app window, which owns the mic/camera/capture;
|
||||
`expand` also refocuses the app window (handled in main).
|
||||
|
||||
## Frame pipeline
|
||||
|
||||
|
|
@ -56,11 +80,13 @@ settings and forces `full` read-aloud; leaving restores them.
|
|||
at the moment of send. Falls back to the single latest frame for
|
||||
rapid-fire messages.
|
||||
|
||||
`App.tsx` `handlePromptSubmit` (~line 2767) attaches the drained frames to
|
||||
the outgoing message as `UserImagePart`s and sets
|
||||
`composition.videoMode: true`. Frames also become `isVideoFrame` display
|
||||
attachments (filmstrip in the transcript — `chat-message-attachments.tsx`;
|
||||
history hydration in `lib/run-to-conversation.ts`).
|
||||
`App.tsx` `handlePromptSubmit` attaches the drained frames (whenever a call
|
||||
is live) to the outgoing message as `UserImagePart`s and sets
|
||||
`composition.videoMode` when the camera or screen is active, plus
|
||||
`composition.coachMode` during a practice session. Frames also become
|
||||
`isVideoFrame` display attachments (filmstrip in the transcript —
|
||||
`chat-message-attachments.tsx`; history hydration in
|
||||
`lib/run-to-conversation.ts`).
|
||||
|
||||
## Message schema & model encoding
|
||||
|
||||
|
|
@ -95,27 +121,27 @@ history hydration in `lib/run-to-conversation.ts`).
|
|||
- Mid-call socket drops reconnect after 1s; the offline audio backlog is
|
||||
capped (~30s).
|
||||
|
||||
Mode transitions live in `App.tsx` `handleVideoModeChange` (~line 1161):
|
||||
call ↔ meeting switches are presentation-only (mic/TTS untouched);
|
||||
entering/leaving hands-free saves/restores TTS settings. Push-to-talk is
|
||||
disabled while a call owns the mic.
|
||||
Call lifecycle lives in `App.tsx` `startCall(preset)` / `endCall()`:
|
||||
entering a call saves/forces TTS settings, cancels any push-to-talk
|
||||
recording, and starts the continuous loop; ending restores everything.
|
||||
Push-to-talk is disabled while a call owns the mic.
|
||||
|
||||
## Popout window
|
||||
|
||||
- Shown iff `videoChatMode !== 'off' && screenState === 'live'` (effect in
|
||||
`App.tsx`). Renderer asks `video:setPopout {show}` (main handler:
|
||||
`apps/main/src/ipc.ts:1742`); main creates a frameless, `alwaysOnTop`
|
||||
('floating'), all-workspaces BrowserWindow at the top-right of the primary
|
||||
display, loading the renderer bundle with `#video-popout`
|
||||
- Shown iff the derived `callSurface === 'popout'` (effect in `App.tsx`).
|
||||
Renderer asks `video:setPopout {show}`; main creates a frameless,
|
||||
`alwaysOnTop` ('floating'), all-workspaces BrowserWindow at the top-right
|
||||
of the primary display, loading the renderer bundle with `#video-popout`
|
||||
(`apps/renderer/src/main.tsx` branches on the hash →
|
||||
`components/video-popout.tsx`).
|
||||
- Call state streams over the `video:popout-state` push channel; main caches
|
||||
the last payload and replays it on popout load. Shown with
|
||||
`showInactive()` so it never steals focus (that would re-hide it).
|
||||
`showInactive()` so it never steals focus.
|
||||
- The popout captures its **own** camera preview (MediaStreams can't cross
|
||||
windows) and synthesizes the mascot mouth level (no audio in that window).
|
||||
- `video:focusMain` matches only real app windows by URL — `getAllWindows()`
|
||||
also contains hidden utility windows (PDF export) that must not be shown.
|
||||
- `video:popoutAction` relays control-bar actions to the app window, matched
|
||||
only by real app-window URLs — `getAllWindows()` also contains hidden
|
||||
utility windows (PDF export) that must not be shown or messaged.
|
||||
|
||||
## Permissions
|
||||
|
||||
|
|
@ -132,8 +158,9 @@ disabled while a call owns the mic.
|
|||
| Prompt | Where |
|
||||
|--------|-------|
|
||||
| `# Video Mode (Live Camera)` system section — how to use webcam frames, coaching guidance, screen-share rules ("treat the screen as the primary subject", "last screen frame is current"), etiquette (never comment on appearance) | `packages/core/src/agents/runtime.ts:386` (`composeSystemInstructions`, gated on `videoMode`) |
|
||||
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts:~1013` (`convertFromMessages`) |
|
||||
| `videoMode` composition override (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts:57,125`; set from `App.tsx` `sendConfig` |
|
||||
| `# Practice Session (Coach Mode)` system section — coaching persona: specific/actionable feedback after each take, one-sentence interjections mid-flow, structured debrief on wrap-up | `composeSystemInstructions`, gated on `coachMode` (directly after the video section) |
|
||||
| Per-message frame context line `[Video mode: N live webcam frames … and M frames of the user's shared screen …]` + group labels | `packages/core/src/agents/runtime.ts` (`convertFromMessages`) |
|
||||
| `videoMode` / `coachMode` composition overrides (session-sticky; flips bust prefix cache) | `packages/core/src/turns/bridges/real-agent-resolver.ts` (`CompositionOverrides`); set from `App.tsx` `sendConfig` |
|
||||
|
||||
Voice input/output prompt sections (`# Voice Input`, `# Voice Output`) are
|
||||
reused untouched — calls set `voiceInput` per utterance and force
|
||||
|
|
@ -156,4 +183,7 @@ screen frame per message unless the screen changed.
|
|||
"energy" coaching leans on visual cues.
|
||||
- Screen share always captures the primary display (no window/display
|
||||
picker yet).
|
||||
- The meeting view covers the chat; there's no in-call transcript drawer.
|
||||
- The full-screen call covers the chat; there's no in-call transcript drawer.
|
||||
- The "attach camera frames to typed chat without a call" combination (the
|
||||
old video+chat mode) was cut in the call-model simplification; if analytics
|
||||
show demand, it should return as an attachment chip, not a mode.
|
||||
|
|
|
|||
|
|
@ -390,6 +390,7 @@ let lastVideoPopoutState: {
|
|||
ttsState: 'idle' | 'synthesizing' | 'speaking';
|
||||
status: 'listening' | 'thinking' | 'speaking' | null;
|
||||
cameraOn: boolean;
|
||||
screenSharing: boolean;
|
||||
interimText: string | null;
|
||||
} | null = null;
|
||||
|
||||
|
|
@ -1816,19 +1817,20 @@ export function setupIpcHandlers() {
|
|||
}
|
||||
return {};
|
||||
},
|
||||
'video:focusMain': async () => {
|
||||
'video:getPopoutState': async () => {
|
||||
return { state: lastVideoPopoutState };
|
||||
},
|
||||
'video:popoutAction': async (_event, args) => {
|
||||
// Relay a popout control-bar action to the app window, which owns the
|
||||
// call (mic, camera, screen capture) and executes it there. 'expand'
|
||||
// additionally brings the app window back to the foreground.
|
||||
const main = findMainAppWindow();
|
||||
if (main) {
|
||||
if (args.action === 'expand' && main) {
|
||||
if (main.isMinimized()) main.restore();
|
||||
main.show();
|
||||
main.focus();
|
||||
}
|
||||
return {};
|
||||
},
|
||||
'video:popoutAction': async (_event, args) => {
|
||||
// Relay a popout control-bar action to the app window, which owns the
|
||||
// call (mic, camera, screen capture) and executes it there.
|
||||
findMainAppWindow()?.webContents.send('video:popout-action', args);
|
||||
main?.webContents.send('video:popout-action', args);
|
||||
return {};
|
||||
},
|
||||
// Live-note handlers
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ import { ChatSidebar } from './components/chat-sidebar';
|
|||
import { useSessionChat } from '@/hooks/useSessionChat';
|
||||
import { ChatHeader } from './components/chat-header';
|
||||
import { ChatEmptyState } from './components/chat-empty-state';
|
||||
import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type VideoChatMode } from './components/chat-input-with-mentions';
|
||||
import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment } from './components/chat-input-with-mentions';
|
||||
import { ChatMessageAttachments } from '@/components/chat-message-attachments'
|
||||
import { GraphView, type GraphEdge, type GraphNode } from '@/components/graph-view';
|
||||
import { BasesView, type BaseConfig, DEFAULT_BASE_CONFIG } from '@/components/bases-view';
|
||||
|
|
@ -120,8 +120,6 @@ import { toast } from "sonner"
|
|||
import { useVoiceMode } from '@/hooks/useVoiceMode'
|
||||
import { useVideoMode } from '@/hooks/useVideoMode'
|
||||
import { useVoiceTTS } from '@/hooks/useVoiceTTS'
|
||||
import { TalkingHeadOverlay } from '@/components/talking-head'
|
||||
import { VideoPreviewOverlay } from '@/components/video-preview-overlay'
|
||||
import { VideoCallView } from '@/components/video-call-view'
|
||||
import { ProductTour, type TourNavTarget } from '@/components/product-tour'
|
||||
import { useMeetingTranscription, type CalendarEventMeta } from '@/hooks/useMeetingTranscription'
|
||||
|
|
@ -974,9 +972,9 @@ function App() {
|
|||
const [ttsAvailable, setTtsAvailable] = useState(false)
|
||||
const [ttsEnabled, setTtsEnabled] = useState(false)
|
||||
const ttsEnabledRef = useRef(false)
|
||||
const [ttsMode, setTtsMode] = useState<'summary' | 'full'>('summary')
|
||||
// Read-aloud style: 'summary' for typed chat, forced to 'full' during a
|
||||
// call and restored after. Context decides — the user never picks it.
|
||||
const ttsModeRef = useRef<'summary' | 'full'>('summary')
|
||||
const [ttsAvatarEnabled, setTtsAvatarEnabled] = useState(false)
|
||||
const [tourActive, setTourActive] = useState(false)
|
||||
const [isRecording, setIsRecording] = useState(false)
|
||||
const voiceTextBufferRef = useRef('')
|
||||
|
|
@ -1021,16 +1019,20 @@ function App() {
|
|||
const voiceRef = useRef(voice)
|
||||
voiceRef.current = voice
|
||||
|
||||
// Video chat mode: while on, the webcam runs and frames are attached to
|
||||
// every outgoing message so the assistant can see the user. 'chat' keeps
|
||||
// the normal composer; 'call' is fully hands-free — continuous listening,
|
||||
// auto-submitted utterances, spoken responses (handler defined below, after
|
||||
// the voice/submit plumbing it drives).
|
||||
// Calls: one engine (hands-free voice loop + forced read-aloud TTS + frame
|
||||
// capture), started via presets that only differ in device defaults. The
|
||||
// presentation is DERIVED from devices, never picked: screen sharing →
|
||||
// floating popout; camera on → full-screen call; camera off → popout
|
||||
// (mascot pill). Handlers live below the voice/submit plumbing they drive.
|
||||
const video = useVideoMode()
|
||||
const [videoChatMode, setVideoChatMode] = useState<VideoChatMode>('off')
|
||||
const videoChatModeRef = useRef<VideoChatMode>('off')
|
||||
// TTS settings to restore when a hands-free call ends (a call forces
|
||||
// full-read-aloud TTS for its duration).
|
||||
const [inCall, setInCall] = useState(false)
|
||||
const inCallRef = useRef(false)
|
||||
// User explicitly shrank the full-screen call to the floating pill.
|
||||
const [callMinimized, setCallMinimized] = useState(false)
|
||||
// Practice preset: adds the coaching persona to the system prompt.
|
||||
const [practiceMode, setPracticeMode] = useState(false)
|
||||
const practiceModeRef = useRef(false)
|
||||
// TTS settings to restore when a call ends (a call forces full read-aloud).
|
||||
const preCallTtsRef = useRef<{ enabled: boolean; mode: 'summary' | 'full' } | null>(null)
|
||||
|
||||
const handleToggleMeetingRef = useRef<(() => void) | undefined>(undefined)
|
||||
|
|
@ -1091,8 +1093,8 @@ function App() {
|
|||
}, [])
|
||||
|
||||
const handleStartRecording = useCallback(() => {
|
||||
// Hands-free call mode owns the mic — ignore push-to-talk while it's on.
|
||||
if (videoChatModeRef.current === 'call' || videoChatModeRef.current === 'meeting') return
|
||||
// A live call owns the mic — ignore push-to-talk while one is running.
|
||||
if (inCallRef.current) return
|
||||
setIsRecording(true)
|
||||
isRecordingRef.current = true
|
||||
voice.start()
|
||||
|
|
@ -1123,143 +1125,128 @@ function App() {
|
|||
ttsEnabledRef.current = next
|
||||
if (!next) {
|
||||
ttsRef.current.cancel()
|
||||
setTtsAvatarEnabled(false)
|
||||
}
|
||||
return next
|
||||
})
|
||||
}, [])
|
||||
|
||||
// Talking-head mode implies voice output: enabling it turns TTS on,
|
||||
// disabling it turns both off.
|
||||
const handleToggleTtsAvatar = useCallback(() => {
|
||||
setTtsAvatarEnabled(prev => {
|
||||
const next = !prev
|
||||
setTtsEnabled(next)
|
||||
ttsEnabledRef.current = next
|
||||
if (!next) {
|
||||
ttsRef.current.cancel()
|
||||
}
|
||||
return next
|
||||
})
|
||||
}, [])
|
||||
|
||||
const handleTtsModeChange = useCallback((mode: 'summary' | 'full') => {
|
||||
setTtsMode(mode)
|
||||
ttsModeRef.current = mode
|
||||
}, [])
|
||||
|
||||
const handleCancelRecording = useCallback(() => {
|
||||
voice.cancel()
|
||||
setIsRecording(false)
|
||||
isRecordingRef.current = false
|
||||
}, [voice])
|
||||
|
||||
// Video chat mode transitions. 'chat' just runs the camera; 'call' and
|
||||
// 'meeting' also listen continuously (auto-submitting each utterance as a
|
||||
// voice message) and force full read-aloud TTS so the assistant answers out
|
||||
// loud. 'meeting' is the same pipeline presented as a full-screen call.
|
||||
const handleVideoModeChange = useCallback(async (mode: VideoChatMode) => {
|
||||
const prev = videoChatModeRef.current
|
||||
if (mode === prev) return
|
||||
const wasHandsFree = prev === 'call' || prev === 'meeting'
|
||||
const isHandsFree = mode === 'call' || mode === 'meeting'
|
||||
// Start a call. Presets only differ in device defaults — the engine
|
||||
// (continuous listening, auto-submitted utterances, forced read-aloud TTS,
|
||||
// frame capture) is identical for all of them. The default entry ('share',
|
||||
// the call button's main click) is "work together": screen shared, camera
|
||||
// off, floating pill — the user keeps working while the assistant watches
|
||||
// along. 'video'/'practice' open face-to-face full screen instead.
|
||||
const startCall = useCallback(async (preset: CallPreset) => {
|
||||
if (inCallRef.current) return
|
||||
const camera = preset === 'video' || preset === 'practice'
|
||||
const ok = await video.start({ camera })
|
||||
if (!ok) return // camera denied/unavailable — stay out of the call
|
||||
if (preset === 'share') {
|
||||
// If screen capture fails (usually the macOS Screen Recording
|
||||
// permission), continue as a voice call — sharing is one tap away on
|
||||
// the pill once permission is granted.
|
||||
const shared = await video.startScreenShare()
|
||||
if (!shared) {
|
||||
toast("Couldn't share your screen", {
|
||||
description: 'Grant Rowboat Screen Recording access, then tap the share button on the call.',
|
||||
action: {
|
||||
label: 'Open Settings',
|
||||
onClick: () => void window.ipc.invoke('meeting:openScreenRecordingSettings', null).catch(() => {}),
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Leaving hands-free: release the mic and restore the pre-call TTS settings.
|
||||
if (wasHandsFree && !isHandsFree) {
|
||||
// A manual push-to-talk recording can't coexist with the call's mic.
|
||||
if (isRecordingRef.current) {
|
||||
voiceRef.current.cancel()
|
||||
const saved = preCallTtsRef.current
|
||||
preCallTtsRef.current = null
|
||||
const restoreEnabled = saved?.enabled ?? false
|
||||
setTtsEnabled(restoreEnabled)
|
||||
ttsEnabledRef.current = restoreEnabled
|
||||
if (saved) {
|
||||
setTtsMode(saved.mode)
|
||||
ttsModeRef.current = saved.mode
|
||||
}
|
||||
if (!restoreEnabled) ttsRef.current.cancel()
|
||||
setIsRecording(false)
|
||||
isRecordingRef.current = false
|
||||
}
|
||||
preCallTtsRef.current = { enabled: ttsEnabledRef.current, mode: ttsModeRef.current }
|
||||
setTtsEnabled(true)
|
||||
ttsEnabledRef.current = true
|
||||
ttsModeRef.current = 'full'
|
||||
void voiceRef.current.startContinuous((text) => {
|
||||
pendingVoiceInputRef.current = true
|
||||
handlePromptSubmitRef.current?.({ text, files: [] })
|
||||
})
|
||||
|
||||
if (mode === 'off') {
|
||||
video.stop()
|
||||
videoChatModeRef.current = 'off'
|
||||
setVideoChatMode('off')
|
||||
return
|
||||
setPracticeMode(preset === 'practice')
|
||||
practiceModeRef.current = preset === 'practice'
|
||||
// Pill-first presets start minimized; face-to-face presets start expanded.
|
||||
setCallMinimized(preset === 'voice' || preset === 'share')
|
||||
inCallRef.current = true
|
||||
setInCall(true)
|
||||
analytics.callStarted(preset)
|
||||
}, [video])
|
||||
|
||||
const endCall = useCallback(() => {
|
||||
if (!inCallRef.current) return
|
||||
voiceRef.current.cancel()
|
||||
const saved = preCallTtsRef.current
|
||||
preCallTtsRef.current = null
|
||||
const restoreEnabled = saved?.enabled ?? false
|
||||
setTtsEnabled(restoreEnabled)
|
||||
ttsEnabledRef.current = restoreEnabled
|
||||
if (saved) {
|
||||
ttsModeRef.current = saved.mode
|
||||
}
|
||||
|
||||
if (prev === 'off') {
|
||||
const ok = await video.start()
|
||||
if (!ok) return // camera denied/unavailable — stay off
|
||||
}
|
||||
|
||||
// Entering hands-free (call ↔ meeting switches keep the mic/TTS as-is).
|
||||
if (isHandsFree && !wasHandsFree) {
|
||||
// A manual push-to-talk recording can't coexist with the call's mic.
|
||||
if (isRecordingRef.current) {
|
||||
voiceRef.current.cancel()
|
||||
setIsRecording(false)
|
||||
isRecordingRef.current = false
|
||||
}
|
||||
preCallTtsRef.current = { enabled: ttsEnabledRef.current, mode: ttsModeRef.current }
|
||||
setTtsEnabled(true)
|
||||
ttsEnabledRef.current = true
|
||||
setTtsMode('full')
|
||||
ttsModeRef.current = 'full'
|
||||
void voiceRef.current.startContinuous((text) => {
|
||||
pendingVoiceInputRef.current = true
|
||||
handlePromptSubmitRef.current?.({ text, files: [] })
|
||||
})
|
||||
}
|
||||
|
||||
videoChatModeRef.current = mode
|
||||
setVideoChatMode(mode)
|
||||
if (!restoreEnabled) ttsRef.current.cancel()
|
||||
video.stop()
|
||||
setPracticeMode(false)
|
||||
practiceModeRef.current = false
|
||||
setCallMinimized(false)
|
||||
inCallRef.current = false
|
||||
setInCall(false)
|
||||
}, [video])
|
||||
|
||||
// During a call, mute the mic while the assistant is thinking or speaking
|
||||
// so its own TTS (or a half-turn) never gets transcribed back at it.
|
||||
useEffect(() => {
|
||||
if (videoChatMode !== 'call' && videoChatMode !== 'meeting') return
|
||||
if (!inCall) return
|
||||
voiceRef.current.setPaused(activeIsProcessing || tts.state !== 'idle')
|
||||
}, [videoChatMode, activeIsProcessing, tts.state])
|
||||
}, [inCall, activeIsProcessing, tts.state])
|
||||
|
||||
// Presenting from the full-screen call collapses it into the floating
|
||||
// popout (the user needs their screen — possibly Rowboat itself — free to
|
||||
// navigate while sharing). When the share ends, return to full screen.
|
||||
const returnToMeetingAfterShareRef = useRef(false)
|
||||
|
||||
// Screen sharing (any video mode): frames of the shared screen ride along
|
||||
// with each message next to the webcam frames.
|
||||
// Screen sharing: frames of the shared screen ride along with each message
|
||||
// next to the webcam frames. The surface change (full screen → pill) falls
|
||||
// out of the derivation below.
|
||||
const handleToggleScreenShare = useCallback(async () => {
|
||||
if (video.screenState === 'live') {
|
||||
video.stopScreenShare()
|
||||
} else {
|
||||
const ok = await video.startScreenShare()
|
||||
if (ok && videoChatModeRef.current === 'meeting') {
|
||||
returnToMeetingAfterShareRef.current = true
|
||||
void handleVideoModeChange('call')
|
||||
}
|
||||
await video.startScreenShare()
|
||||
}
|
||||
}, [video, handleVideoModeChange])
|
||||
}, [video])
|
||||
|
||||
// Share ended (from the popout, the PiP, or the OS): restore the
|
||||
// full-screen call if that's where the share was started from.
|
||||
useEffect(() => {
|
||||
if (video.screenState === 'live') return
|
||||
if (!returnToMeetingAfterShareRef.current) return
|
||||
returnToMeetingAfterShareRef.current = false
|
||||
if (videoChatModeRef.current === 'call') {
|
||||
void handleVideoModeChange('meeting')
|
||||
}
|
||||
}, [video.screenState, handleVideoModeChange])
|
||||
|
||||
// Meet-style camera mute: video mode (and any screen share) stays on, but
|
||||
// no webcam frames are captured while the camera is off.
|
||||
// Meet-style camera mute: the call (and any screen share) stays on, but no
|
||||
// webcam frames are captured while the camera is off. Deliberately does NOT
|
||||
// change the surface — turning your camera on from the pill puts your video
|
||||
// IN the pill; expanding to full screen is its own explicit action.
|
||||
const handleToggleCamera = useCallback(() => {
|
||||
void video.setCameraEnabled(!video.cameraOn)
|
||||
}, [video])
|
||||
|
||||
// Current phase of a hands-free call (null outside call/meeting modes).
|
||||
// Minimizing the full-screen call drops you back to working — and the pill
|
||||
// exists to work *together*, so sharing starts automatically (the symmetric
|
||||
// twin of expand, which stops it). If capture fails (permission), the call
|
||||
// still minimizes as a plain pill. `callMinimized` is also set so stopping
|
||||
// the share from the pill keeps you in the pill rather than snapping back
|
||||
// to full screen.
|
||||
const handleMinimizeCall = useCallback(async () => {
|
||||
setCallMinimized(true)
|
||||
await video.startScreenShare()
|
||||
}, [video])
|
||||
|
||||
// Current phase of the call (null when not in one).
|
||||
const videoCallStatus: 'listening' | 'thinking' | 'speaking' | null =
|
||||
videoChatMode === 'call' || videoChatMode === 'meeting'
|
||||
inCall
|
||||
? tts.state === 'speaking'
|
||||
? 'speaking'
|
||||
: tts.state === 'synthesizing' || activeIsProcessing
|
||||
|
|
@ -1267,39 +1254,70 @@ function App() {
|
|||
: 'listening'
|
||||
: null
|
||||
|
||||
// Meet-style popout: the floating always-on-top mini-call appears the
|
||||
// moment screen sharing starts and stays for the whole share — including
|
||||
// over Rowboat itself (the shared screen may BE Rowboat), and across app
|
||||
// switches. It disappears when sharing stops or video mode ends.
|
||||
useEffect(() => {
|
||||
const shouldShow = videoChatMode !== 'off' && video.screenState === 'live'
|
||||
void window.ipc.invoke('video:setPopout', { show: shouldShow }).catch(() => {})
|
||||
}, [videoChatMode, video.screenState])
|
||||
// The call's surface follows one rule: full screen and screen sharing are
|
||||
// mutually exclusive (a full-screen call covers the screen — sharing it
|
||||
// would show the call itself). Sharing → floating pill, always. Not
|
||||
// sharing → full screen unless the user shrank it (`callMinimized`).
|
||||
// Expanding the pill auto-stops any share; presenting from full screen
|
||||
// auto-collapses to the pill.
|
||||
const callSurface: 'fullscreen' | 'popout' | null = !inCall
|
||||
? null
|
||||
: video.screenState === 'live' || callMinimized
|
||||
? 'popout'
|
||||
: 'fullscreen'
|
||||
|
||||
// Keep the popout's mascot/status/camera/caption mirror of the call fresh.
|
||||
useEffect(() => {
|
||||
void window.ipc.invoke('video:setPopout', { show: callSurface === 'popout' }).catch(() => {})
|
||||
}, [callSurface])
|
||||
|
||||
// Consent surface for screen sharing: an unmissable toast the moment any
|
||||
// share starts (auto-started calls included), with one-tap stop. The pill
|
||||
// also carries a persistent "Sharing screen" badge, and macOS shows its
|
||||
// purple recording indicator.
|
||||
const prevScreenStateRef = useRef(video.screenState)
|
||||
useEffect(() => {
|
||||
const prev = prevScreenStateRef.current
|
||||
prevScreenStateRef.current = video.screenState
|
||||
if (video.screenState === 'live' && prev !== 'live') {
|
||||
toast('Your screen is being shared', {
|
||||
description: 'The assistant sees snapshots of it along with what you say.',
|
||||
action: { label: 'Stop sharing', onClick: () => video.stopScreenShare() },
|
||||
duration: 6000,
|
||||
})
|
||||
}
|
||||
}, [video.screenState, video])
|
||||
|
||||
// Keep the popout's mascot/status/devices/caption mirror of the call fresh.
|
||||
// The main process caches the latest state and replays it when the popout
|
||||
// loads.
|
||||
useEffect(() => {
|
||||
if (videoChatMode === 'off') return
|
||||
if (!inCall) return
|
||||
void window.ipc
|
||||
.invoke('video:popoutState', {
|
||||
ttsState: tts.state,
|
||||
status: videoCallStatus,
|
||||
cameraOn: video.cameraOn,
|
||||
interimText: videoCallStatus ? voice.interimText || null : null,
|
||||
screenSharing: video.screenState === 'live',
|
||||
interimText: voice.interimText || null,
|
||||
})
|
||||
.catch(() => {})
|
||||
}, [videoChatMode, tts.state, videoCallStatus, video.cameraOn, voice.interimText])
|
||||
}, [inCall, tts.state, videoCallStatus, video.cameraOn, video.screenState, voice.interimText])
|
||||
|
||||
// Execute popout control-bar actions (the popout window has no access to
|
||||
// the call's mic/camera/capture — they live here).
|
||||
// the call's mic/camera/capture — they live here). 'expand' goes full
|
||||
// screen, which by the exclusivity rule stops any running share; the main
|
||||
// process already refocused the app window.
|
||||
useEffect(() => {
|
||||
return window.ipc.on('video:popout-action', ({ action }) => {
|
||||
if (action === 'toggle-camera') handleToggleCamera()
|
||||
else if (action === 'stop-share') video.stopScreenShare()
|
||||
else if (action === 'end-call') void handleVideoModeChange('off')
|
||||
else if (action === 'toggle-share') void handleToggleScreenShare()
|
||||
else if (action === 'end-call') endCall()
|
||||
else if (action === 'expand') {
|
||||
if (video.screenState === 'live') video.stopScreenShare()
|
||||
setCallMinimized(false)
|
||||
}
|
||||
})
|
||||
}, [handleToggleCamera, video, handleVideoModeChange])
|
||||
}, [handleToggleCamera, handleToggleScreenShare, endCall, video])
|
||||
|
||||
// Enter to submit voice input, Escape to cancel
|
||||
useEffect(() => {
|
||||
|
|
@ -2783,7 +2801,7 @@ function App() {
|
|||
|
||||
// Video chat mode: drain the webcam frames buffered since the last send
|
||||
// so they ride along with this message as inline image parts.
|
||||
const videoFrames = videoChatModeRef.current !== 'off' ? video.collectFrames() : []
|
||||
const videoFrames = inCallRef.current ? video.collectFrames() : []
|
||||
|
||||
const userMessageId = `user-${Date.now()}`
|
||||
const displayAttachments: ChatMessage['attachments'] = hasAttachments || videoFrames.length > 0
|
||||
|
|
@ -2854,7 +2872,8 @@ function App() {
|
|||
...(ttsEnabledRef.current ? { voiceOutput: ttsModeRef.current } : {}),
|
||||
...(searchEnabled ? { searchEnabled: true } : {}),
|
||||
...(codeMode ? { codeMode } : {}),
|
||||
...(videoChatModeRef.current !== 'off' ? { videoMode: true } : {}),
|
||||
...(inCallRef.current && (video.cameraOn || video.screenState === 'live') ? { videoMode: true } : {}),
|
||||
...(practiceModeRef.current ? { coachMode: true } : {}),
|
||||
},
|
||||
},
|
||||
},
|
||||
|
|
@ -6536,14 +6555,11 @@ function App() {
|
|||
voiceAvailable={isActive && voiceAvailable}
|
||||
ttsAvailable={isActive && ttsAvailable}
|
||||
ttsEnabled={ttsEnabled}
|
||||
ttsMode={ttsMode}
|
||||
onToggleTts={isActive ? handleToggleTts : undefined}
|
||||
onTtsModeChange={isActive ? handleTtsModeChange : undefined}
|
||||
ttsAvatarEnabled={ttsAvatarEnabled}
|
||||
onToggleTtsAvatar={isActive ? handleToggleTtsAvatar : undefined}
|
||||
videoChatMode={videoChatMode}
|
||||
onVideoModeChange={isActive ? handleVideoModeChange : undefined}
|
||||
videoCallAvailable={voiceAvailable && ttsAvailable}
|
||||
inCall={inCall}
|
||||
onStartCall={isActive ? startCall : undefined}
|
||||
onEndCall={isActive ? endCall : undefined}
|
||||
callAvailable={voiceAvailable && ttsAvailable}
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
|
|
@ -6652,69 +6668,32 @@ function App() {
|
|||
voiceAvailable={voiceAvailable}
|
||||
ttsAvailable={ttsAvailable}
|
||||
ttsEnabled={ttsEnabled}
|
||||
ttsMode={ttsMode}
|
||||
onToggleTts={handleToggleTts}
|
||||
onTtsModeChange={handleTtsModeChange}
|
||||
ttsAvatarEnabled={ttsAvatarEnabled}
|
||||
onToggleTtsAvatar={handleToggleTtsAvatar}
|
||||
videoChatMode={videoChatMode}
|
||||
onVideoModeChange={handleVideoModeChange}
|
||||
videoCallAvailable={voiceAvailable && ttsAvailable}
|
||||
inCall={inCall}
|
||||
onStartCall={startCall}
|
||||
onEndCall={endCall}
|
||||
callAvailable={voiceAvailable && ttsAvailable}
|
||||
onComposioConnected={handleComposioConnected}
|
||||
/>
|
||||
)}
|
||||
{/* Webcam PiP preview while video chat mode is on. Hidden while
|
||||
screen sharing (the floating popout is the call surface then)
|
||||
and in the full-screen meeting view (which replaces it). */}
|
||||
{(videoChatMode === 'chat' || videoChatMode === 'call') && video.screenState !== 'live' && (
|
||||
<VideoPreviewOverlay
|
||||
streamRef={video.streamRef}
|
||||
onTurnOff={() => handleVideoModeChange('off')}
|
||||
callStatus={
|
||||
videoChatMode === 'call'
|
||||
? tts.state !== 'idle'
|
||||
? 'speaking'
|
||||
: activeIsProcessing
|
||||
? 'thinking'
|
||||
: 'listening'
|
||||
: undefined
|
||||
}
|
||||
interimText={videoChatMode === 'call' ? voice.interimText : undefined}
|
||||
onToggleScreenShare={handleToggleScreenShare}
|
||||
cameraOn={video.cameraOn}
|
||||
onToggleCamera={handleToggleCamera}
|
||||
/>
|
||||
)}
|
||||
{/* Full-screen Meet-style call: user tile + animated mascot tile */}
|
||||
{videoChatMode === 'meeting' && (
|
||||
{/* Full-screen call: user tile + animated mascot tile. Shown only
|
||||
when the derived surface says so (camera on, no screen share,
|
||||
not minimized) — otherwise the call lives in the floating
|
||||
popout window. */}
|
||||
{callSurface === 'fullscreen' && (
|
||||
<VideoCallView
|
||||
streamRef={video.streamRef}
|
||||
screenStreamRef={video.screenStreamRef}
|
||||
isScreenSharing={video.screenState === 'live'}
|
||||
onToggleScreenShare={handleToggleScreenShare}
|
||||
cameraOn={video.cameraOn}
|
||||
onToggleCamera={handleToggleCamera}
|
||||
practiceMode={practiceMode}
|
||||
onMinimize={() => void handleMinimizeCall()}
|
||||
ttsState={tts.state}
|
||||
getTtsLevel={tts.getLevel}
|
||||
status={
|
||||
tts.state === 'speaking'
|
||||
? 'speaking'
|
||||
: tts.state === 'synthesizing' || activeIsProcessing
|
||||
? 'thinking'
|
||||
: 'listening'
|
||||
}
|
||||
status={videoCallStatus ?? 'listening'}
|
||||
interimText={voice.interimText}
|
||||
assistantCaption={assistantCaption}
|
||||
onLeave={() => handleVideoModeChange('off')}
|
||||
/>
|
||||
)}
|
||||
{/* Talking head hovers over the active view while avatar voice mode is
|
||||
on (hidden during the tour, which shows its own mascot) */}
|
||||
{ttsAvatarEnabled && !tourActive && (
|
||||
<TalkingHeadOverlay
|
||||
ttsState={tts.state}
|
||||
getLevel={tts.getLevel}
|
||||
onDismiss={handleToggleTtsAvatar}
|
||||
onLeave={endCall}
|
||||
/>
|
||||
)}
|
||||
{/* Mascot-guided product tour */}
|
||||
|
|
|
|||
|
|
@ -21,7 +21,10 @@ import {
|
|||
Lock,
|
||||
Mic,
|
||||
MoreHorizontal,
|
||||
Phone,
|
||||
PhoneOff,
|
||||
Plus,
|
||||
Presentation,
|
||||
ShieldCheck,
|
||||
Square,
|
||||
Terminal,
|
||||
|
|
@ -51,7 +54,6 @@ import {
|
|||
} from '@/lib/attachment-presentation'
|
||||
import { getExtension, getFileDisplayName, getMimeFromExtension, isImageMime } from '@/lib/file-utils'
|
||||
import { cn } from '@/lib/utils'
|
||||
import { MascotFaceIcon } from '@/components/talking-head'
|
||||
import {
|
||||
type FileMention,
|
||||
type PromptInputMessage,
|
||||
|
|
@ -212,7 +214,17 @@ function compactWorkDirPath(path: string) {
|
|||
return path.replace(/^\/Users\/[^/]+/, '~')
|
||||
}
|
||||
|
||||
export type VideoChatMode = 'off' | 'chat' | 'call' | 'meeting'
|
||||
// Call presets: front doors into the same call engine, differing only in
|
||||
// starting devices. 'share' is the call button's main click — the "work
|
||||
// together" default (screen shared, camera off, floating pill). The chevron
|
||||
// menu holds the deviations.
|
||||
export type CallPreset = 'voice' | 'video' | 'share' | 'practice'
|
||||
|
||||
const CALL_PRESET_MENU: Array<{ preset: CallPreset; label: string; description: string; Icon: typeof Phone }> = [
|
||||
{ preset: 'voice', label: 'Voice call', description: 'Just talk — nothing is shared, the mascot hovers while you work', Icon: AudioLines },
|
||||
{ preset: 'video', label: 'Video call', description: 'Camera on, face to face — it sees your expressions', Icon: Video },
|
||||
{ preset: 'practice', label: 'Practice session', description: 'Rehearse a pitch or interview with live coaching', Icon: Presentation },
|
||||
]
|
||||
|
||||
interface ChatInputInnerProps {
|
||||
onSubmit: (message: PromptInputMessage, mentions?: FileMention[], attachments?: StagedAttachment[], searchEnabled?: boolean, codeMode?: 'claude' | 'codex', permissionMode?: PermissionMode) => void
|
||||
|
|
@ -236,18 +248,14 @@ interface ChatInputInnerProps {
|
|||
voiceAvailable?: boolean
|
||||
ttsAvailable?: boolean
|
||||
ttsEnabled?: boolean
|
||||
ttsMode?: 'summary' | 'full'
|
||||
onToggleTts?: () => void
|
||||
onTtsModeChange?: (mode: 'summary' | 'full') => void
|
||||
ttsAvatarEnabled?: boolean
|
||||
onToggleTtsAvatar?: () => void
|
||||
/** Video chat mode: 'chat' attaches webcam frames to messages; 'call' and
|
||||
* 'meeting' are fully hands-free — continuous listening, spoken responses
|
||||
* ('meeting' additionally takes over the whole screen, Meet-style). */
|
||||
videoChatMode?: VideoChatMode
|
||||
onVideoModeChange?: (mode: VideoChatMode) => void
|
||||
/** Hands-free call needs both voice input (STT) and voice output (TTS). */
|
||||
videoCallAvailable?: boolean
|
||||
/** A call is live (hands-free voice loop + spoken responses). */
|
||||
inCall?: boolean
|
||||
/** Start a call with the given preset's device defaults. */
|
||||
onStartCall?: (preset: CallPreset) => void
|
||||
onEndCall?: () => void
|
||||
/** Calls need both voice input (STT) and voice output (TTS) configured. */
|
||||
callAvailable?: boolean
|
||||
/** Fired when the user picks a different model in the dropdown (only when no run exists yet). */
|
||||
onSelectedModelChange?: (model: SelectedModel | null) => void
|
||||
/** Work directory for this chat (per-chat). Null when none is set. */
|
||||
|
|
@ -283,14 +291,11 @@ function ChatInputInner({
|
|||
voiceAvailable,
|
||||
ttsAvailable,
|
||||
ttsEnabled,
|
||||
ttsMode,
|
||||
onToggleTts,
|
||||
onTtsModeChange,
|
||||
ttsAvatarEnabled,
|
||||
onToggleTtsAvatar,
|
||||
videoChatMode = 'off',
|
||||
onVideoModeChange,
|
||||
videoCallAvailable,
|
||||
inCall,
|
||||
onStartCall,
|
||||
onEndCall,
|
||||
callAvailable,
|
||||
onSelectedModelChange,
|
||||
workDir = null,
|
||||
onWorkDirChange,
|
||||
|
|
@ -1290,68 +1295,21 @@ function ChatInputInner({
|
|||
</DropdownMenu>
|
||||
) : null}
|
||||
{onToggleTts && ttsAvailable && (
|
||||
<div className="flex shrink-0 items-center">
|
||||
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
|
||||
<TooltipTrigger asChild>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onToggleTts}
|
||||
className={cn(
|
||||
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
|
||||
ttsEnabled
|
||||
? 'text-foreground hover:bg-muted'
|
||||
: 'text-muted-foreground hover:bg-muted hover:text-foreground'
|
||||
)}
|
||||
aria-label={ttsEnabled ? 'Disable voice output' : 'Enable voice output'}
|
||||
>
|
||||
<Headphones className="h-4 w-4" />
|
||||
{!ttsEnabled && (
|
||||
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
|
||||
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
|
||||
</span>
|
||||
)}
|
||||
</button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent side="top">
|
||||
{ttsEnabled ? 'Voice output on' : 'Voice output off'}
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
{ttsEnabled && onTtsModeChange && (
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<button
|
||||
type="button"
|
||||
className="flex h-7 w-4 shrink-0 items-center justify-center text-muted-foreground transition-colors hover:text-foreground"
|
||||
>
|
||||
<ChevronDown className="h-3 w-3" />
|
||||
</button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end">
|
||||
<DropdownMenuRadioGroup value={ttsMode ?? 'summary'} onValueChange={(v) => onTtsModeChange(v as 'summary' | 'full')}>
|
||||
<DropdownMenuRadioItem value="summary">Speak summary</DropdownMenuRadioItem>
|
||||
<DropdownMenuRadioItem value="full">Speak full response</DropdownMenuRadioItem>
|
||||
</DropdownMenuRadioGroup>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{onToggleTtsAvatar && ttsAvailable && (
|
||||
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
|
||||
<TooltipTrigger asChild>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onToggleTtsAvatar}
|
||||
onClick={onToggleTts}
|
||||
className={cn(
|
||||
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
|
||||
ttsAvatarEnabled
|
||||
ttsEnabled
|
||||
? 'text-foreground hover:bg-muted'
|
||||
: 'text-muted-foreground hover:bg-muted hover:text-foreground'
|
||||
)}
|
||||
aria-label={ttsAvatarEnabled ? 'Disable talking head' : 'Enable talking head'}
|
||||
aria-label={ttsEnabled ? 'Disable read-aloud' : 'Enable read-aloud'}
|
||||
>
|
||||
<MascotFaceIcon />
|
||||
{!ttsAvatarEnabled && (
|
||||
<Headphones className="h-4 w-4" />
|
||||
{!ttsEnabled && (
|
||||
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
|
||||
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
|
||||
</span>
|
||||
|
|
@ -1359,64 +1317,70 @@ function ChatInputInner({
|
|||
</button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent side="top">
|
||||
{ttsAvatarEnabled ? 'Talking head on' : 'Talking head off'}
|
||||
{ttsEnabled ? 'Read responses aloud: on' : 'Read responses aloud: off'}
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
)}
|
||||
{onVideoModeChange && (
|
||||
{onStartCall && (
|
||||
<div className="flex shrink-0 items-center">
|
||||
<Tooltip delayDuration={CHAT_INPUT_TOOLTIP_DELAY_MS}>
|
||||
<TooltipTrigger asChild>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => onVideoModeChange(videoChatMode === 'off' ? 'chat' : 'off')}
|
||||
onClick={() => {
|
||||
if (inCall) {
|
||||
onEndCall?.()
|
||||
} else if (callAvailable) {
|
||||
onStartCall('share')
|
||||
}
|
||||
}}
|
||||
className={cn(
|
||||
'relative flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
|
||||
videoChatMode !== 'off'
|
||||
? 'text-red-500 hover:bg-muted'
|
||||
: 'text-muted-foreground hover:bg-muted hover:text-foreground'
|
||||
'flex h-7 w-7 shrink-0 items-center justify-center rounded-full transition-colors',
|
||||
inCall
|
||||
? 'bg-red-600 text-white hover:bg-red-500'
|
||||
: callAvailable
|
||||
? 'text-muted-foreground hover:bg-muted hover:text-foreground'
|
||||
: 'cursor-default text-muted-foreground/40'
|
||||
)}
|
||||
aria-label={videoChatMode === 'off' ? 'Turn on video chat' : 'Turn off video chat'}
|
||||
aria-label={inCall ? 'End call' : 'Start a call'}
|
||||
>
|
||||
<Video className="h-4 w-4" />
|
||||
{videoChatMode === 'off' && (
|
||||
<span className="absolute inset-0 flex items-center justify-center pointer-events-none">
|
||||
<span className="block h-[1.5px] w-5 -rotate-45 rounded-full bg-muted-foreground" />
|
||||
</span>
|
||||
)}
|
||||
{inCall ? <PhoneOff className="h-4 w-4" /> : <Phone className="h-4 w-4" />}
|
||||
</button>
|
||||
</TooltipTrigger>
|
||||
<TooltipContent side="top">
|
||||
{videoChatMode === 'off'
|
||||
? 'Video chat off'
|
||||
: videoChatMode === 'chat'
|
||||
? 'Video chat on — camera frames are shared with the assistant'
|
||||
: 'Video call on — speak freely, the assistant answers out loud'}
|
||||
{inCall
|
||||
? 'End call'
|
||||
: callAvailable
|
||||
? 'Start a call — it sees your screen while you talk it through'
|
||||
: 'Calls need voice input and output configured'}
|
||||
</TooltipContent>
|
||||
</Tooltip>
|
||||
{videoChatMode !== 'off' && (
|
||||
{!inCall && (
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<button
|
||||
type="button"
|
||||
className="flex h-7 w-4 shrink-0 items-center justify-center text-muted-foreground transition-colors hover:text-foreground"
|
||||
aria-label="Call options"
|
||||
>
|
||||
<ChevronDown className="h-3 w-3" />
|
||||
</button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end">
|
||||
<DropdownMenuRadioGroup
|
||||
value={videoChatMode}
|
||||
onValueChange={(v) => onVideoModeChange(v as VideoChatMode)}
|
||||
>
|
||||
<DropdownMenuRadioItem value="chat">Video + chat</DropdownMenuRadioItem>
|
||||
<DropdownMenuRadioItem value="call" disabled={!videoCallAvailable}>
|
||||
Video call (hands-free)
|
||||
</DropdownMenuRadioItem>
|
||||
<DropdownMenuRadioItem value="meeting" disabled={!videoCallAvailable}>
|
||||
Video call (full screen)
|
||||
</DropdownMenuRadioItem>
|
||||
</DropdownMenuRadioGroup>
|
||||
<DropdownMenuContent align="end" className="w-72">
|
||||
{CALL_PRESET_MENU.map(({ preset, label, description, Icon }) => (
|
||||
<DropdownMenuItem
|
||||
key={preset}
|
||||
disabled={!callAvailable}
|
||||
onSelect={() => onStartCall(preset)}
|
||||
className="items-start gap-3 py-2"
|
||||
>
|
||||
<Icon className="mt-0.5 h-4 w-4 shrink-0 text-muted-foreground" />
|
||||
<span className="min-w-0">
|
||||
<span className="block text-sm font-medium leading-tight">{label}</span>
|
||||
<span className="block pt-0.5 text-xs leading-tight text-muted-foreground">{description}</span>
|
||||
</span>
|
||||
</DropdownMenuItem>
|
||||
))}
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
)}
|
||||
|
|
@ -1593,14 +1557,11 @@ export interface ChatInputWithMentionsProps {
|
|||
voiceAvailable?: boolean
|
||||
ttsAvailable?: boolean
|
||||
ttsEnabled?: boolean
|
||||
ttsMode?: 'summary' | 'full'
|
||||
onToggleTts?: () => void
|
||||
onTtsModeChange?: (mode: 'summary' | 'full') => void
|
||||
ttsAvatarEnabled?: boolean
|
||||
onToggleTtsAvatar?: () => void
|
||||
videoChatMode?: VideoChatMode
|
||||
onVideoModeChange?: (mode: VideoChatMode) => void
|
||||
videoCallAvailable?: boolean
|
||||
inCall?: boolean
|
||||
onStartCall?: (preset: CallPreset) => void
|
||||
onEndCall?: () => void
|
||||
callAvailable?: boolean
|
||||
onSelectedModelChange?: (model: SelectedModel | null) => void
|
||||
workDir?: string | null
|
||||
onWorkDirChange?: (value: string | null) => void
|
||||
|
|
@ -1632,14 +1593,11 @@ export function ChatInputWithMentions({
|
|||
voiceAvailable,
|
||||
ttsAvailable,
|
||||
ttsEnabled,
|
||||
ttsMode,
|
||||
onToggleTts,
|
||||
onTtsModeChange,
|
||||
ttsAvatarEnabled,
|
||||
onToggleTtsAvatar,
|
||||
videoChatMode,
|
||||
onVideoModeChange,
|
||||
videoCallAvailable,
|
||||
inCall,
|
||||
onStartCall,
|
||||
onEndCall,
|
||||
callAvailable,
|
||||
onSelectedModelChange,
|
||||
workDir,
|
||||
onWorkDirChange,
|
||||
|
|
@ -1668,14 +1626,11 @@ export function ChatInputWithMentions({
|
|||
voiceAvailable={voiceAvailable}
|
||||
ttsAvailable={ttsAvailable}
|
||||
ttsEnabled={ttsEnabled}
|
||||
ttsMode={ttsMode}
|
||||
onToggleTts={onToggleTts}
|
||||
onTtsModeChange={onTtsModeChange}
|
||||
ttsAvatarEnabled={ttsAvatarEnabled}
|
||||
onToggleTtsAvatar={onToggleTtsAvatar}
|
||||
videoChatMode={videoChatMode}
|
||||
onVideoModeChange={onVideoModeChange}
|
||||
videoCallAvailable={videoCallAvailable}
|
||||
inCall={inCall}
|
||||
onStartCall={onStartCall}
|
||||
onEndCall={onEndCall}
|
||||
callAvailable={callAvailable}
|
||||
onSelectedModelChange={onSelectedModelChange}
|
||||
workDir={workDir}
|
||||
onWorkDirChange={onWorkDirChange}
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ import { MarkdownPreOverride } from '@/components/ai-elements/markdown-code-over
|
|||
import { defaultRemarkPlugins } from 'streamdown'
|
||||
import remarkBreaks from 'remark-breaks'
|
||||
import { type ChatTab } from '@/components/tab-bar'
|
||||
import { ChatInputWithMentions, type PermissionMode, type StagedAttachment, type SelectedModel, type VideoChatMode } from '@/components/chat-input-with-mentions'
|
||||
import { ChatInputWithMentions, type CallPreset, type PermissionMode, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions'
|
||||
import { ChatMessageAttachments } from '@/components/chat-message-attachments'
|
||||
import { useSidebar } from '@/components/ui/sidebar'
|
||||
import { wikiLabel } from '@/lib/wiki-links'
|
||||
|
|
@ -189,14 +189,11 @@ interface ChatSidebarProps {
|
|||
voiceAvailable?: boolean
|
||||
ttsAvailable?: boolean
|
||||
ttsEnabled?: boolean
|
||||
ttsMode?: 'summary' | 'full'
|
||||
onToggleTts?: () => void
|
||||
onTtsModeChange?: (mode: 'summary' | 'full') => void
|
||||
ttsAvatarEnabled?: boolean
|
||||
onToggleTtsAvatar?: () => void
|
||||
videoChatMode?: VideoChatMode
|
||||
onVideoModeChange?: (mode: VideoChatMode) => void
|
||||
videoCallAvailable?: boolean
|
||||
inCall?: boolean
|
||||
onStartCall?: (preset: CallPreset) => void
|
||||
onEndCall?: () => void
|
||||
callAvailable?: boolean
|
||||
onComposioConnected?: (toolkitSlug: string) => void
|
||||
}
|
||||
|
||||
|
|
@ -258,14 +255,11 @@ export function ChatSidebar({
|
|||
voiceAvailable,
|
||||
ttsAvailable,
|
||||
ttsEnabled,
|
||||
ttsMode,
|
||||
onToggleTts,
|
||||
onTtsModeChange,
|
||||
ttsAvatarEnabled,
|
||||
onToggleTtsAvatar,
|
||||
videoChatMode,
|
||||
onVideoModeChange,
|
||||
videoCallAvailable,
|
||||
inCall,
|
||||
onStartCall,
|
||||
onEndCall,
|
||||
callAvailable,
|
||||
onComposioConnected,
|
||||
}: ChatSidebarProps) {
|
||||
const { state: sidebarState } = useSidebar()
|
||||
|
|
@ -837,14 +831,11 @@ export function ChatSidebar({
|
|||
voiceAvailable={isActive && voiceAvailable}
|
||||
ttsAvailable={isActive && ttsAvailable}
|
||||
ttsEnabled={ttsEnabled}
|
||||
ttsMode={ttsMode}
|
||||
onToggleTts={isActive ? onToggleTts : undefined}
|
||||
onTtsModeChange={isActive ? onTtsModeChange : undefined}
|
||||
ttsAvatarEnabled={ttsAvatarEnabled}
|
||||
onToggleTtsAvatar={isActive ? onToggleTtsAvatar : undefined}
|
||||
videoChatMode={videoChatMode}
|
||||
onVideoModeChange={isActive ? onVideoModeChange : undefined}
|
||||
videoCallAvailable={videoCallAvailable}
|
||||
inCall={inCall}
|
||||
onStartCall={isActive ? onStartCall : undefined}
|
||||
onEndCall={isActive ? onEndCall : undefined}
|
||||
callAvailable={callAvailable}
|
||||
/>
|
||||
</div>
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import { useEffect, useRef, useState } from 'react'
|
||||
import { MonitorUp, PhoneOff, User, Video, VideoOff } from 'lucide-react'
|
||||
import { Minimize2, MonitorUp, PhoneOff, Presentation, User, Video, VideoOff } from 'lucide-react'
|
||||
|
||||
import { MascotFaceIcon, TalkingHead } from '@/components/talking-head'
|
||||
import type { TTSState } from '@/hooks/useVoiceTTS'
|
||||
|
|
@ -10,12 +10,15 @@ export type VideoCallStatus = 'listening' | 'thinking' | 'speaking'
|
|||
interface VideoCallViewProps {
|
||||
/** Live camera stream from useVideoMode — attached to the user's tile. */
|
||||
streamRef: React.MutableRefObject<MediaStream | null>
|
||||
/** Live screen-share stream — shown as the presentation tile when sharing. */
|
||||
screenStreamRef: React.MutableRefObject<MediaStream | null>
|
||||
isScreenSharing: boolean
|
||||
onToggleScreenShare: () => void
|
||||
cameraOn: boolean
|
||||
onToggleCamera: () => void
|
||||
/** Starting a share collapses this view into the floating popout (the
|
||||
* surface is derived from devices — see App.tsx). */
|
||||
onToggleScreenShare: () => void
|
||||
/** Practice preset: the assistant is coaching this session. */
|
||||
practiceMode?: boolean
|
||||
/** Shrink to the floating pill without touching any devices. */
|
||||
onMinimize: () => void
|
||||
ttsState: TTSState
|
||||
/** Live TTS output level — drives the mascot's mouth animation. */
|
||||
getTtsLevel: () => number
|
||||
|
|
@ -33,53 +36,22 @@ const STATUS_DISPLAY: Record<VideoCallStatus, { label: string; dotClass: string
|
|||
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
|
||||
}
|
||||
|
||||
/** Attach a MediaStream ref to a <video> element for the lifetime of the mount. */
|
||||
function StreamVideo({
|
||||
streamRef,
|
||||
mirrored,
|
||||
className,
|
||||
}: {
|
||||
streamRef: React.MutableRefObject<MediaStream | null>
|
||||
mirrored?: boolean
|
||||
className?: string
|
||||
}) {
|
||||
const videoRef = useRef<HTMLVideoElement | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
const videoEl = videoRef.current
|
||||
if (!videoEl) return
|
||||
videoEl.srcObject = streamRef.current
|
||||
videoEl.play().catch(() => {})
|
||||
return () => {
|
||||
videoEl.srcObject = null
|
||||
}
|
||||
}, [streamRef])
|
||||
|
||||
return (
|
||||
<video
|
||||
ref={videoRef}
|
||||
muted
|
||||
playsInline
|
||||
className={className}
|
||||
style={mirrored ? { transform: 'scaleX(-1)' } : undefined}
|
||||
/>
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Full-screen hands-free call: a Meet-style layout with the user's webcam on
|
||||
* one side and the mascot as the other participant. While presenting, the
|
||||
* shared screen becomes the big tile and the participants shrink into a side
|
||||
* rail. The mascot animates with the assistant's speech; dismissing it swaps
|
||||
* in a Meet-style letter avatar ("R"). Live captions run along the bottom.
|
||||
* Full-screen call surface: a Meet-style two-tile layout with the user's
|
||||
* webcam on one side and the mascot as the other participant. Shown only
|
||||
* while the camera is on with no screen share (the derived-surface rule in
|
||||
* App.tsx) — sharing or muting the camera moves the call into the floating
|
||||
* popout. The mascot animates with the assistant's speech; dismissing it
|
||||
* swaps in a Meet-style letter avatar ("R"). Live captions run along the
|
||||
* bottom.
|
||||
*/
|
||||
export function VideoCallView({
|
||||
streamRef,
|
||||
screenStreamRef,
|
||||
isScreenSharing,
|
||||
onToggleScreenShare,
|
||||
cameraOn,
|
||||
onToggleCamera,
|
||||
onToggleScreenShare,
|
||||
practiceMode,
|
||||
onMinimize,
|
||||
ttsState,
|
||||
getTtsLevel,
|
||||
status,
|
||||
|
|
@ -87,8 +59,20 @@ export function VideoCallView({
|
|||
assistantCaption,
|
||||
onLeave,
|
||||
}: VideoCallViewProps) {
|
||||
const videoRef = useRef<HTMLVideoElement | null>(null)
|
||||
const [mascotVisible, setMascotVisible] = useState(true)
|
||||
|
||||
useEffect(() => {
|
||||
if (!cameraOn) return
|
||||
const videoEl = videoRef.current
|
||||
if (!videoEl) return
|
||||
videoEl.srcObject = streamRef.current
|
||||
videoEl.play().catch(() => {})
|
||||
return () => {
|
||||
videoEl.srcObject = null
|
||||
}
|
||||
}, [streamRef, cameraOn])
|
||||
|
||||
const userSpeaking = status === 'listening' && Boolean(interimText)
|
||||
const assistantSpeaking = ttsState === 'speaking'
|
||||
|
||||
|
|
@ -98,89 +82,80 @@ export function VideoCallView({
|
|||
? { who: 'You', text: interimText }
|
||||
: null
|
||||
|
||||
const userTile = (
|
||||
<div
|
||||
className={cn(
|
||||
'relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
|
||||
userSpeaking && 'ring-2 ring-green-500/80',
|
||||
isScreenSharing && 'aspect-video w-full'
|
||||
)}
|
||||
>
|
||||
{cameraOn ? (
|
||||
<StreamVideo streamRef={streamRef} mirrored className="h-full w-full object-cover" />
|
||||
) : (
|
||||
<span
|
||||
className={cn(
|
||||
'flex items-center justify-center rounded-full bg-neutral-700 text-neutral-400',
|
||||
isScreenSharing ? 'h-16 w-16' : 'h-40 w-40'
|
||||
)}
|
||||
aria-label="Camera off"
|
||||
>
|
||||
<User className={isScreenSharing ? 'h-8 w-8' : 'h-20 w-20'} />
|
||||
</span>
|
||||
)}
|
||||
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
|
||||
You
|
||||
</span>
|
||||
</div>
|
||||
)
|
||||
|
||||
const assistantTile = (
|
||||
<div
|
||||
className={cn(
|
||||
'group relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
|
||||
assistantSpeaking && 'ring-2 ring-sky-400/80',
|
||||
isScreenSharing && 'aspect-video w-full'
|
||||
)}
|
||||
>
|
||||
{mascotVisible ? (
|
||||
<TalkingHead ttsState={ttsState} getLevel={getTtsLevel} size={isScreenSharing ? 96 : 220} />
|
||||
) : (
|
||||
<span
|
||||
className={cn(
|
||||
'flex items-center justify-center rounded-full bg-sky-600 font-medium text-white',
|
||||
isScreenSharing ? 'h-16 w-16 text-3xl' : 'h-40 w-40 text-7xl'
|
||||
)}
|
||||
aria-label="Rowboat"
|
||||
>
|
||||
R
|
||||
</span>
|
||||
)}
|
||||
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
|
||||
Rowboat
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setMascotVisible((v) => !v)}
|
||||
className="absolute right-3 top-3 rounded-md bg-black/50 px-2 py-1 text-xs text-white/80 opacity-0 transition-opacity hover:text-white group-hover:opacity-100"
|
||||
>
|
||||
{mascotVisible ? 'Hide mascot' : 'Show mascot'}
|
||||
</button>
|
||||
</div>
|
||||
)
|
||||
|
||||
return (
|
||||
<div className="fixed inset-0 z-[100] flex flex-col bg-neutral-950">
|
||||
{/* Participant tiles — Meet-style presentation layout while sharing */}
|
||||
{isScreenSharing ? (
|
||||
<div className="flex min-h-0 flex-1 gap-3 p-4 pb-2">
|
||||
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-2xl bg-neutral-900">
|
||||
<StreamVideo streamRef={screenStreamRef} className="h-full w-full object-contain" />
|
||||
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
|
||||
Your screen
|
||||
</span>
|
||||
</div>
|
||||
<div className="flex w-52 shrink-0 flex-col gap-3">
|
||||
{userTile}
|
||||
{assistantTile}
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<div className="grid min-h-0 flex-1 grid-cols-1 gap-3 p-4 pb-2 md:grid-cols-2">
|
||||
{userTile}
|
||||
{assistantTile}
|
||||
</div>
|
||||
{practiceMode && (
|
||||
<span className="absolute left-4 top-4 z-10 flex items-center gap-1.5 rounded-full bg-violet-600/90 px-3 py-1 text-xs font-medium text-white">
|
||||
<Presentation className="h-3.5 w-3.5" />
|
||||
Practice session
|
||||
</span>
|
||||
)}
|
||||
<button
|
||||
type="button"
|
||||
onClick={onMinimize}
|
||||
className="absolute right-4 top-4 z-10 flex h-8 w-8 items-center justify-center rounded-full bg-neutral-800 text-white/80 transition-colors hover:bg-neutral-700 hover:text-white"
|
||||
aria-label="Minimize call (shares your screen)"
|
||||
title="Minimize — shares your screen so it can help you work"
|
||||
>
|
||||
<Minimize2 className="h-4 w-4" />
|
||||
</button>
|
||||
|
||||
{/* Participant tiles */}
|
||||
<div className="grid min-h-0 flex-1 grid-cols-1 gap-3 p-4 pb-2 md:grid-cols-2">
|
||||
{/* User */}
|
||||
<div
|
||||
className={cn(
|
||||
'relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
|
||||
userSpeaking && 'ring-2 ring-green-500/80'
|
||||
)}
|
||||
>
|
||||
{cameraOn ? (
|
||||
<video
|
||||
ref={videoRef}
|
||||
muted
|
||||
playsInline
|
||||
className="h-full w-full object-cover"
|
||||
style={{ transform: 'scaleX(-1)' }}
|
||||
/>
|
||||
) : (
|
||||
<span className="flex h-40 w-40 items-center justify-center rounded-full bg-neutral-700 text-neutral-400" aria-label="Camera off">
|
||||
<User className="h-20 w-20" />
|
||||
</span>
|
||||
)}
|
||||
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
|
||||
You
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Assistant */}
|
||||
<div
|
||||
className={cn(
|
||||
'group relative flex items-center justify-center overflow-hidden rounded-2xl bg-neutral-900 transition-shadow',
|
||||
assistantSpeaking && 'ring-2 ring-sky-400/80'
|
||||
)}
|
||||
>
|
||||
{mascotVisible ? (
|
||||
<TalkingHead ttsState={ttsState} getLevel={getTtsLevel} size={220} />
|
||||
) : (
|
||||
<span
|
||||
className="flex h-40 w-40 items-center justify-center rounded-full bg-sky-600 text-7xl font-medium text-white"
|
||||
aria-label="Rowboat"
|
||||
>
|
||||
R
|
||||
</span>
|
||||
)}
|
||||
<span className="absolute bottom-3 left-3 rounded-md bg-black/50 px-2 py-0.5 text-sm text-white">
|
||||
Rowboat
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setMascotVisible((v) => !v)}
|
||||
className="absolute right-3 top-3 rounded-md bg-black/50 px-2 py-1 text-xs text-white/80 opacity-0 transition-opacity hover:text-white group-hover:opacity-100"
|
||||
>
|
||||
{mascotVisible ? 'Hide mascot' : 'Show mascot'}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Captions */}
|
||||
<div className="flex h-14 items-center justify-center px-6">
|
||||
|
|
@ -215,14 +190,9 @@ export function VideoCallView({
|
|||
<button
|
||||
type="button"
|
||||
onClick={onToggleScreenShare}
|
||||
className={cn(
|
||||
'flex h-10 w-10 items-center justify-center rounded-full transition-colors',
|
||||
isScreenSharing
|
||||
? 'bg-sky-600 text-white hover:bg-sky-500'
|
||||
: 'bg-neutral-800 text-white/90 hover:bg-neutral-700'
|
||||
)}
|
||||
aria-label={isScreenSharing ? 'Stop presenting' : 'Present your screen'}
|
||||
title={isScreenSharing ? 'Stop presenting' : 'Present your screen'}
|
||||
className="flex h-10 w-10 items-center justify-center rounded-full bg-neutral-800 text-white/90 transition-colors hover:bg-neutral-700"
|
||||
aria-label="Present your screen"
|
||||
title="Present your screen"
|
||||
>
|
||||
<MonitorUp className="h-5 w-5" />
|
||||
</button>
|
||||
|
|
@ -243,7 +213,7 @@ export function VideoCallView({
|
|||
type="button"
|
||||
onClick={onLeave}
|
||||
className="flex h-10 w-14 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
|
||||
aria-label="Leave call"
|
||||
aria-label="End call"
|
||||
>
|
||||
<PhoneOff className="h-5 w-5" />
|
||||
</button>
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ type PopoutState = {
|
|||
ttsState: 'idle' | 'synthesizing' | 'speaking'
|
||||
status: 'listening' | 'thinking' | 'speaking' | null
|
||||
cameraOn: boolean
|
||||
screenSharing: boolean
|
||||
interimText: string | null
|
||||
}
|
||||
|
||||
|
|
@ -23,18 +24,30 @@ const noDragRegion = { WebkitAppRegion: 'no-drag' } as React.CSSProperties
|
|||
* Content of the always-on-top popout window shown for the whole duration of
|
||||
* a screen share (Meet-style floating mini-call) — it floats over every app,
|
||||
* including Rowboat itself, and is the call's control surface while sharing:
|
||||
* camera toggle, stop-share, end-call. Rendered in its own BrowserWindow
|
||||
* camera toggle, share toggle, end-call. Rendered in its own BrowserWindow
|
||||
* (see `video:setPopout` in the main process); call state streams in over
|
||||
* the `video:popout-state` push channel and control actions round-trip back
|
||||
* through `video:popoutAction`. Captures its own webcam feed — MediaStreams
|
||||
* can't cross windows.
|
||||
*/
|
||||
export function VideoPopout() {
|
||||
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: true, interimText: null })
|
||||
// Camera defaults OFF: guessing "on" would flash the user's video for a
|
||||
// beat before the real state arrives — which reads as a bug. The true
|
||||
// state is fetched immediately below.
|
||||
const [state, setState] = useState<PopoutState>({ ttsState: 'idle', status: null, cameraOn: false, screenSharing: false, interimText: null })
|
||||
const videoRef = useRef<HTMLVideoElement | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
return window.ipc.on('video:popout-state', (next) => setState(next))
|
||||
const cleanup = window.ipc.on('video:popout-state', (next) => setState(next))
|
||||
// The main process replays the cached state on did-finish-load, but that
|
||||
// can race this listener's registration — fetch it explicitly too.
|
||||
window.ipc
|
||||
.invoke('video:getPopoutState', null)
|
||||
.then(({ state: cached }) => {
|
||||
if (cached) setState(cached)
|
||||
})
|
||||
.catch(() => {})
|
||||
return cleanup
|
||||
}, [])
|
||||
|
||||
// Own camera feed, following the main window's camera-on/off state.
|
||||
|
|
@ -67,7 +80,7 @@ export function VideoPopout() {
|
|||
// so the mascot still animates while the assistant speaks in the main window.
|
||||
const getLevel = useCallback(() => 0.45 + 0.35 * Math.sin(performance.now() / 90), [])
|
||||
|
||||
const sendAction = useCallback((action: 'toggle-camera' | 'stop-share' | 'end-call') => {
|
||||
const sendAction = useCallback((action: 'toggle-camera' | 'toggle-share' | 'end-call' | 'expand') => {
|
||||
void window.ipc.invoke('video:popoutAction', { action }).catch(() => {})
|
||||
}, [])
|
||||
|
||||
|
|
@ -98,6 +111,14 @@ export function VideoPopout() {
|
|||
<span className="absolute bottom-1 left-1.5 rounded bg-black/50 px-1 py-px text-[10px] text-white">
|
||||
You
|
||||
</span>
|
||||
{/* Persistent consent badge — the user must always be able to see
|
||||
at a glance that their screen is going out. */}
|
||||
{state.screenSharing && (
|
||||
<span className="absolute left-1.5 top-1.5 flex items-center gap-1 rounded-full bg-sky-600/90 px-1.5 py-0.5 text-[10px] font-medium text-white">
|
||||
<span className="block h-1.5 w-1.5 animate-pulse rounded-full bg-white" />
|
||||
Sharing screen
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<div className="relative flex flex-1 items-center justify-center overflow-hidden rounded-lg bg-neutral-800">
|
||||
<TalkingHead ttsState={state.ttsState} getLevel={getLevel} size={84} />
|
||||
|
|
@ -138,10 +159,14 @@ export function VideoPopout() {
|
|||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => sendAction('stop-share')}
|
||||
className="flex h-6 w-6 items-center justify-center rounded-full bg-sky-600 text-white transition-colors hover:bg-sky-500"
|
||||
aria-label="Stop sharing screen"
|
||||
title="Stop sharing screen"
|
||||
onClick={() => sendAction('toggle-share')}
|
||||
className={`flex h-6 w-6 items-center justify-center rounded-full transition-colors ${
|
||||
state.screenSharing
|
||||
? 'bg-sky-600 text-white hover:bg-sky-500'
|
||||
: 'bg-neutral-700 text-white/90 hover:bg-neutral-600'
|
||||
}`}
|
||||
aria-label={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
|
||||
title={state.screenSharing ? 'Stop sharing screen' : 'Share your screen'}
|
||||
>
|
||||
<MonitorUp className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
|
|
@ -149,17 +174,17 @@ export function VideoPopout() {
|
|||
type="button"
|
||||
onClick={() => sendAction('end-call')}
|
||||
className="flex h-6 w-8 items-center justify-center rounded-full bg-red-600 text-white transition-colors hover:bg-red-500"
|
||||
aria-label="End video mode"
|
||||
title="End video mode"
|
||||
aria-label="End call"
|
||||
title="End call"
|
||||
>
|
||||
<PhoneOff className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => void window.ipc.invoke('video:focusMain', null)}
|
||||
onClick={() => sendAction('expand')}
|
||||
className="flex h-6 w-6 items-center justify-center rounded-full bg-neutral-700 text-white/90 transition-colors hover:bg-neutral-600"
|
||||
aria-label="Back to Rowboat"
|
||||
title="Back to Rowboat"
|
||||
aria-label="Expand to full screen (stops screen sharing)"
|
||||
title="Expand to full screen (stops sharing)"
|
||||
>
|
||||
<Maximize2 className="h-3.5 w-3.5" />
|
||||
</button>
|
||||
|
|
|
|||
|
|
@ -1,129 +0,0 @@
|
|||
import { useEffect, useRef } from 'react'
|
||||
import { MonitorUp, User, VideoOff, X } from 'lucide-react'
|
||||
|
||||
interface VideoPreviewOverlayProps {
|
||||
/** Live camera stream from useVideoMode — attached to the preview element. */
|
||||
streamRef: React.MutableRefObject<MediaStream | null>
|
||||
onTurnOff: () => void
|
||||
/** Hands-free call mode: current phase of the conversation loop. */
|
||||
callStatus?: 'listening' | 'thinking' | 'speaking'
|
||||
/** Hands-free call mode: live transcript of the in-progress utterance. */
|
||||
interimText?: string
|
||||
isScreenSharing?: boolean
|
||||
onToggleScreenShare?: () => void
|
||||
cameraOn?: boolean
|
||||
onToggleCamera?: () => void
|
||||
}
|
||||
|
||||
const CALL_STATUS_DISPLAY: Record<NonNullable<VideoPreviewOverlayProps['callStatus']>, { label: string; dotClass: string }> = {
|
||||
listening: { label: 'Listening', dotClass: 'bg-green-500 animate-pulse' },
|
||||
thinking: { label: 'Thinking…', dotClass: 'bg-amber-400' },
|
||||
speaking: { label: 'Speaking', dotClass: 'bg-sky-400 animate-pulse' },
|
||||
}
|
||||
|
||||
/**
|
||||
* Floating picture-in-picture webcam preview shown while video chat mode is
|
||||
* on. Mirrored like a selfie camera so the user's movements feel natural.
|
||||
* Sits above the composer dock, mirroring the talking-head overlay's corner.
|
||||
*/
|
||||
export function VideoPreviewOverlay({ streamRef, onTurnOff, callStatus, interimText, isScreenSharing, onToggleScreenShare, cameraOn = true, onToggleCamera }: VideoPreviewOverlayProps) {
|
||||
const videoRef = useRef<HTMLVideoElement | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
if (!cameraOn) return
|
||||
const videoEl = videoRef.current
|
||||
if (!videoEl) return
|
||||
videoEl.srcObject = streamRef.current
|
||||
videoEl.play().catch(() => {})
|
||||
return () => {
|
||||
videoEl.srcObject = null
|
||||
}
|
||||
}, [streamRef, cameraOn])
|
||||
|
||||
return (
|
||||
<div
|
||||
className="group fixed bottom-28 left-8 z-50"
|
||||
style={{ animation: 'video-preview-pop 0.35s cubic-bezier(0.34, 1.56, 0.64, 1)' }}
|
||||
>
|
||||
<style>{`
|
||||
@keyframes video-preview-pop {
|
||||
0% { opacity: 0; scale: 0.4; }
|
||||
100% { opacity: 1; scale: 1; }
|
||||
}
|
||||
`}</style>
|
||||
<button
|
||||
type="button"
|
||||
onClick={onTurnOff}
|
||||
className="absolute -right-1.5 -top-1.5 z-10 flex h-5 w-5 items-center justify-center rounded-full border border-border bg-background text-muted-foreground opacity-0 shadow-sm transition-opacity hover:text-foreground group-hover:opacity-100"
|
||||
aria-label="Turn off video chat"
|
||||
>
|
||||
<X className="h-3 w-3" />
|
||||
</button>
|
||||
{onToggleScreenShare && (
|
||||
<button
|
||||
type="button"
|
||||
onClick={onToggleScreenShare}
|
||||
className={`absolute -left-1.5 -top-1.5 z-10 flex h-5 w-5 items-center justify-center rounded-full border shadow-sm transition-opacity ${
|
||||
isScreenSharing
|
||||
? 'border-sky-500 bg-sky-600 text-white opacity-100'
|
||||
: 'border-border bg-background text-muted-foreground opacity-0 hover:text-foreground group-hover:opacity-100'
|
||||
}`}
|
||||
aria-label={isScreenSharing ? 'Stop sharing screen' : 'Share your screen'}
|
||||
title={isScreenSharing ? 'Stop sharing screen' : 'Share your screen'}
|
||||
>
|
||||
<MonitorUp className="h-3 w-3" />
|
||||
</button>
|
||||
)}
|
||||
{cameraOn ? (
|
||||
<video
|
||||
ref={videoRef}
|
||||
muted
|
||||
playsInline
|
||||
className="h-32 w-auto rounded-xl border border-border/70 bg-black shadow-lg"
|
||||
style={{ transform: 'scaleX(-1)' }}
|
||||
/>
|
||||
) : (
|
||||
<div className="flex h-32 w-48 items-center justify-center rounded-xl border border-border/70 bg-black shadow-lg">
|
||||
<span className="flex h-14 w-14 items-center justify-center rounded-full bg-neutral-700 text-neutral-400">
|
||||
<User className="h-7 w-7" />
|
||||
</span>
|
||||
</div>
|
||||
)}
|
||||
{onToggleCamera && (
|
||||
<button
|
||||
type="button"
|
||||
onClick={onToggleCamera}
|
||||
className={`absolute -left-1.5 top-4 z-10 flex h-5 w-5 items-center justify-center rounded-full border shadow-sm transition-opacity ${
|
||||
cameraOn
|
||||
? 'border-border bg-background text-muted-foreground opacity-0 hover:text-foreground group-hover:opacity-100'
|
||||
: 'border-red-500 bg-red-600 text-white opacity-100'
|
||||
}`}
|
||||
aria-label={cameraOn ? 'Turn off camera' : 'Turn on camera'}
|
||||
title={cameraOn ? 'Turn off camera' : 'Turn on camera'}
|
||||
>
|
||||
<VideoOff className="h-3 w-3" />
|
||||
</button>
|
||||
)}
|
||||
<span className="pointer-events-none absolute bottom-1.5 left-1.5 flex items-center gap-1 rounded-full bg-black/60 px-1.5 py-0.5 text-[10px] font-medium text-white">
|
||||
{callStatus ? (
|
||||
<>
|
||||
<span className={`block h-1.5 w-1.5 rounded-full ${CALL_STATUS_DISPLAY[callStatus].dotClass}`} />
|
||||
{CALL_STATUS_DISPLAY[callStatus].label}
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<span className="block h-1.5 w-1.5 animate-pulse rounded-full bg-red-500" />
|
||||
Video on
|
||||
</>
|
||||
)}
|
||||
</span>
|
||||
{callStatus && interimText && (
|
||||
<div className="pointer-events-none absolute inset-x-0 -bottom-1 translate-y-full pt-1">
|
||||
<div className="max-h-16 overflow-hidden rounded-lg bg-black/60 px-2 py-1 text-[11px] leading-snug text-white/90">
|
||||
{interimText}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
|
@ -234,15 +234,22 @@ export function useVideoMode() {
|
|||
return true;
|
||||
}, [acquireCamera]);
|
||||
|
||||
const start = useCallback(async (): Promise<boolean> => {
|
||||
/**
|
||||
* Start video mode. `camera: false` starts a camera-less session (voice
|
||||
* call / screen-share-only) — the mode is live so frames can flow from
|
||||
* other sources, and the camera can be enabled later via setCameraEnabled.
|
||||
*/
|
||||
const start = useCallback(async ({ camera = true }: { camera?: boolean } = {}): Promise<boolean> => {
|
||||
if (stateRef.current !== 'idle') return true;
|
||||
setState('starting');
|
||||
const ok = await acquireCamera();
|
||||
if (!ok) {
|
||||
setState('idle');
|
||||
return false;
|
||||
if (camera) {
|
||||
const ok = await acquireCamera();
|
||||
if (!ok) {
|
||||
setState('idle');
|
||||
return false;
|
||||
}
|
||||
}
|
||||
setCameraOn(true);
|
||||
setCameraOn(camera);
|
||||
setState('live');
|
||||
return true;
|
||||
}, [acquireCamera]);
|
||||
|
|
|
|||
|
|
@ -65,6 +65,10 @@ export function voiceInputStarted() {
|
|||
posthog.capture('voice_input_started')
|
||||
}
|
||||
|
||||
export function callStarted(preset: 'voice' | 'video' | 'share' | 'practice') {
|
||||
posthog.capture('call_started', { preset })
|
||||
}
|
||||
|
||||
export function searchExecuted(types: string[]) {
|
||||
posthog.capture('search_executed', { types })
|
||||
}
|
||||
|
|
|
|||
|
|
@ -339,6 +339,7 @@ export interface ComposeSystemInstructionsInput {
|
|||
codeCwd: string | null;
|
||||
// Optional so legacy callers (old streamAgent path) are unaffected.
|
||||
videoMode?: boolean;
|
||||
coachMode?: boolean;
|
||||
}
|
||||
|
||||
// System-prompt assembly, extracted verbatim from streamAgent so the new turn
|
||||
|
|
@ -354,6 +355,7 @@ export function composeSystemInstructions({
|
|||
codeMode,
|
||||
codeCwd,
|
||||
videoMode,
|
||||
coachMode,
|
||||
}: ComposeSystemInstructionsInput): string {
|
||||
let instructionsWithDateTime = `${instructions}\n\n${USER_CONTEXT_SYSTEM_INSTRUCTIONS}`;
|
||||
if (agentNotesContext) {
|
||||
|
|
@ -402,6 +404,18 @@ Etiquette:
|
|||
- Do not narrate or list what you see in every response. Bring up visual observations only when relevant to the user's request or clearly worth mentioning.
|
||||
- Never comment on the user's physical appearance, attractiveness, or personal attributes — visual feedback is strictly about delivery, expression, and body language.
|
||||
- Frames are periodic snapshots, not continuous video; moments between frames are missing. Don't claim certainty about motion you couldn't have seen.`;
|
||||
}
|
||||
if (coachMode) {
|
||||
instructionsWithDateTime += `\n\n# Practice Session (Coach Mode)
|
||||
The user started a practice session: they are rehearsing something performative — a pitch, presentation, interview answer, or talk — and want live coaching. You are their coach for this session.
|
||||
|
||||
How to coach:
|
||||
- Watch and listen for delivery: pacing, filler words, rambling, structure, and (from the webcam frames) posture, eye contact with the camera, facial expressiveness, gesturing, and visible energy.
|
||||
- After each take or answer, give brief, specific, actionable feedback: 2-3 concrete observations, quoting what you actually saw or heard ("you looked down during the ask", "the opening 'um, so, basically' undercuts your hook"). Then let them go again.
|
||||
- If they are clearly mid-flow, keep any interjection to one short sentence — or stay silent and save it for the break.
|
||||
- Ask what they're practicing for at the start if it isn't obvious (investor pitch? interview? conference talk?) and tailor feedback to that audience.
|
||||
- When they say they're done or wrap up, give a structured debrief: what's working, the top 3 things to improve, and one concrete drill or reframe to try next time.
|
||||
- Be encouraging but honest — vague praise wastes their rehearsal time. Never comment on physical appearance; delivery, expression, and body language only.`;
|
||||
}
|
||||
if (voiceOutput === 'summary') {
|
||||
instructionsWithDateTime += `\n\n# Voice Output (MANDATORY — READ THIS FIRST)\nThe user has voice output enabled. THIS IS YOUR #1 PRIORITY: you MUST start your response with <voice></voice> tags. If your response does not begin with <voice> tags, the user will hear nothing — which is a broken experience. NEVER skip this.\n\nRules:\n1. YOUR VERY FIRST OUTPUT MUST BE A <voice> TAG. No exceptions. Do not start with markdown, headings, or any other text. The literal first characters of your response must be "<voice>".\n2. Place ALL <voice> tags at the BEGINNING of your response, before any detailed content. Do NOT intersperse <voice> tags throughout the response.\n3. Wrap EACH spoken sentence in its own separate <voice> tag so it can be spoken incrementally. Do NOT wrap everything in a single <voice> block.\n4. Use voice as a TL;DR and navigation aid — do NOT read the entire response aloud.\n5. After all <voice> tags, you may include detailed written content (markdown, tables, code, etc.) that will be shown visually but not spoken.\n\n## Examples\n\nExample 1 — User asks: "what happened in my meeting with Alex yesterday?"\n\n<voice>Your meeting with Alex covered three main things: the Q2 roadmap timeline, hiring for the backend role, and the client demo next week.</voice>\n<voice>I've pulled out the key details and action items below — the demo prep notes are at the end.</voice>\n\n## Meeting with Alex — March 11\n### Roadmap\n- Agreed to push Q2 launch to April 15...\n(detailed written content continues)\n\nExample 2 — User asks: "summarize my emails"\n\n<voice>You have five new emails since this morning.</voice>\n<voice>Two are from your team — Jordan sent the RFC you requested and Taylor flagged a contract issue.</voice>\n<voice>There's also a warm intro from a VC partner connecting you with someone at a prospective customer.</voice>\n<voice>I've drafted responses for three of them. The details and drafts are below.</voice>\n\n(email blocks, tables, and detailed content follow)\n\nExample 3 — User asks: "what's on my calendar today?"\n\n<voice>You've got a pretty packed day — seven meetings starting with standup at 9.</voice>\n<voice>The big ones are your investor call at 11, lunch with a partner from your lead VC at 12:30, and a customer call at 4.</voice>\n<voice>Your only free block for deep work is 2:30 to 4.</voice>\n\n(calendar block with full event details follows)\n\nExample 4 — User asks: "draft an email to Sam with our metrics"\n\n<voice>Done — I've drafted the email to Sam with your latest WAU and churn numbers.</voice>\n<voice>Take a look at the draft below and send it when you're ready.</voice>\n\n(email block with draft follows)\n\nREMEMBER: If you do not start with <voice> tags, the user hears silence. Always speak first, then write.`;
|
||||
|
|
|
|||
|
|
@ -55,6 +55,7 @@ const CompositionOverrides = z.object({
|
|||
codeMode: z.enum(["claude", "codex"]).nullable().optional(),
|
||||
codeCwd: z.string().nullable().optional(),
|
||||
videoMode: z.boolean().optional(),
|
||||
coachMode: z.boolean().optional(),
|
||||
});
|
||||
|
||||
export interface RealAgentResolverDeps {
|
||||
|
|
@ -123,6 +124,7 @@ export class RealAgentResolver implements IAgentResolver {
|
|||
codeMode: composition.codeMode ?? null,
|
||||
codeCwd: composition.codeCwd ?? null,
|
||||
videoMode: composition.videoMode ?? false,
|
||||
coachMode: composition.coachMode ?? false,
|
||||
});
|
||||
|
||||
const tools = await this.resolveTools(agent);
|
||||
|
|
|
|||
|
|
@ -1394,21 +1394,36 @@ const ipcSchemas = {
|
|||
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
|
||||
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
|
||||
cameraOn: z.boolean(),
|
||||
// Live transcript of the in-progress utterance (hands-free call only).
|
||||
screenSharing: z.boolean(),
|
||||
// Live transcript of the in-progress utterance.
|
||||
interimText: z.string().nullable(),
|
||||
}),
|
||||
res: z.object({}),
|
||||
},
|
||||
// Popout window → bring the main app window back to the foreground.
|
||||
'video:focusMain': {
|
||||
// Popout window → fetch the latest cached call state on mount. The
|
||||
// did-finish-load replay can race the React listener registration, and the
|
||||
// popout must never guess (a wrong camera-on default flashes the user's
|
||||
// video before the first state push corrects it).
|
||||
'video:getPopoutState': {
|
||||
req: z.null(),
|
||||
res: z.object({}),
|
||||
res: z.object({
|
||||
state: z
|
||||
.object({
|
||||
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
|
||||
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
|
||||
cameraOn: z.boolean(),
|
||||
screenSharing: z.boolean(),
|
||||
interimText: z.string().nullable(),
|
||||
})
|
||||
.nullable(),
|
||||
}),
|
||||
},
|
||||
// Popout control bar → main process → relayed to the app window, which
|
||||
// executes the action on the live call.
|
||||
// executes the action on the live call. 'expand' additionally focuses the
|
||||
// main app window (handled in the main process).
|
||||
'video:popoutAction': {
|
||||
req: z.object({
|
||||
action: z.enum(['toggle-camera', 'stop-share', 'end-call']),
|
||||
action: z.enum(['toggle-camera', 'toggle-share', 'end-call', 'expand']),
|
||||
}),
|
||||
res: z.object({}),
|
||||
},
|
||||
|
|
@ -1418,6 +1433,7 @@ const ipcSchemas = {
|
|||
ttsState: z.enum(['idle', 'synthesizing', 'speaking']),
|
||||
status: z.enum(['listening', 'thinking', 'speaking']).nullable(),
|
||||
cameraOn: z.boolean(),
|
||||
screenSharing: z.boolean(),
|
||||
interimText: z.string().nullable(),
|
||||
}),
|
||||
res: z.null(),
|
||||
|
|
@ -1425,7 +1441,7 @@ const ipcSchemas = {
|
|||
// Push channel: main → app window with a popout control-bar action.
|
||||
'video:popout-action': {
|
||||
req: z.object({
|
||||
action: z.enum(['toggle-camera', 'stop-share', 'end-call']),
|
||||
action: z.enum(['toggle-camera', 'toggle-share', 'end-call', 'expand']),
|
||||
}),
|
||||
res: z.null(),
|
||||
},
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue