From 1edc1b17f1b04421c7d58206e1116e98652f33dd Mon Sep 17 00:00:00 2001 From: Arjun <6592213+arkml@users.noreply.github.com> Date: Sat, 4 Jul 2026 00:09:46 +0530 Subject: [PATCH] screen share pop out when I switch to a different app --- CLAUDE.md | 1 + apps/x/VIDEO_MODE.md | 151 ++++++++++++++++++ apps/x/apps/main/src/ipc.ts | 96 ++++++++++- apps/x/apps/renderer/src/App.tsx | 55 +++++++ .../src/components/video-call-view.tsx | 34 +++- .../renderer/src/components/video-popout.tsx | 117 ++++++++++++++ .../src/components/video-preview-overlay.tsx | 46 ++++-- .../x/apps/renderer/src/hooks/useVideoMode.ts | 47 +++++- apps/x/apps/renderer/src/main.tsx | 13 +- apps/x/packages/shared/src/ipc.ts | 30 ++++ 10 files changed, 567 insertions(+), 23 deletions(-) create mode 100644 apps/x/VIDEO_MODE.md create mode 100644 apps/x/apps/renderer/src/components/video-popout.tsx diff --git a/CLAUDE.md b/CLAUDE.md index bf34bd90..c80edc23 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,6 +109,7 @@ Long-form docs for specific features. Read the relevant file before making chang | Feature | Doc | |---------|-----| | Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` | +| Video mode — webcam + screen-share frames to the model (video+chat / hands-free call / full-screen Meet-style call), continuous voice loop, camera mute, popout window, prompts catalog | `apps/x/VIDEO_MODE.md` | | Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` | | Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` | diff --git a/apps/x/VIDEO_MODE.md b/apps/x/VIDEO_MODE.md new file mode 100644 index 00000000..fff8f6f4 --- /dev/null +++ b/apps/x/VIDEO_MODE.md @@ -0,0 +1,151 @@ +# Video Mode — Deep Dive + +Video mode lets the assistant *see* the user (webcam) and their screen (screen +share), in three presentations: frames attached to normal chat, a hands-free +spoken call, and a full-screen Meet-style call. This doc covers the product +flow, the technical pipeline, and the LLM prompt surface with exact pointers. + +## Product flow + +The composer's video button (`chat-input-with-mentions.tsx`) toggles video +mode; a chevron dropdown picks one of three modes (`VideoChatMode`): + +| Mode | What it does | +|------|--------------| +| `chat` — "Video + chat" | Camera on. Webcam (and screen-share) frames ride along with every typed or dictated message. Small PiP preview floats above the composer. | +| `call` — "Video call (hands-free)" | Everything in `chat`, plus: continuous listening (each utterance auto-submits as a voice message) and forced full read-aloud TTS. No typing needed; composer still works. | +| `meeting` — "Video call (full screen)" | Same pipeline as `call`, presented as a full-screen Meet-style layout: user tile + animated mascot tile, captions, control bar. | + +On top of any mode: + +- **Screen share** (`MonitorUp` buttons on the PiP overlay and the meeting + control bar): captures the primary screen; frames go to the model as a + separately labeled group. In the meeting view the screen becomes the big + tile with user + mascot in a side rail. +- **Camera off** (Meet-style mute): video mode and screen share keep running, + no webcam frames are captured; tiles show a silhouette avatar. +- **Mascot dismissal** (meeting view): swaps the animated mascot for a + Meet-style letter avatar ("R"). +- **Popout**: while screen sharing, if the app window loses focus (the user + switched to the app they're sharing), a small always-on-top frameless + window pops out with the user + mascot mini-tiles; refocusing dismisses it. + Its expand button focuses the main window (`video:focusMain`). + +`call`/`meeting` options are disabled unless both voice input (Deepgram) and +voice output (TTS) are configured. Entering a call saves the user's TTS +settings and forces `full` read-aloud; leaving restores them. + +## Frame pipeline + +`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source +(stream → offscreen `