diff --git a/.github/workflows/electron-build.yml b/.github/workflows/electron-build.yml index 787e28e6..e68ce4f4 100644 --- a/.github/workflows/electron-build.yml +++ b/.github/workflows/electron-build.yml @@ -163,12 +163,25 @@ jobs: run: pnpm install --frozen-lockfile working-directory: apps/x + - name: Build node-pty native binary for Linux + working-directory: apps/x + run: | + # node-pty ships prebuilt binaries only for darwin/win32; compile the + # linux-x64 binary so bundle.mjs can stage it into the package. Without + # this the Linux app crashes on launch (missing prebuilds/linux-x64/pty.node). + PTY="node_modules/.pnpm/node-pty@1.1.0/node_modules/node-pty" + cd "$PTY" + npx node-gyp rebuild + mkdir -p prebuilds/linux-x64 + cp build/Release/pty.node prebuilds/linux-x64/ + - name: Build electron app env: VITE_PUBLIC_POSTHOG_KEY: ${{ secrets.VITE_PUBLIC_POSTHOG_KEY }} VITE_PUBLIC_POSTHOG_HOST: ${{ secrets.VITE_PUBLIC_POSTHOG_HOST }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: npx electron-forge publish --arch=x64,arm64 --platform=linux + ROWBOAT_SKIP_PACMAN: '1' # Arch Linux package is built locally only, never in CI + run: npx electron-forge publish --arch=x64 --platform=linux working-directory: apps/x/apps/main - name: Upload workflow artifacts diff --git a/.github/workflows/x-tests.yml b/.github/workflows/x-tests.yml new file mode 100644 index 00000000..f657d478 --- /dev/null +++ b/.github/workflows/x-tests.yml @@ -0,0 +1,40 @@ +name: apps/x tests + +on: + pull_request: + paths: + - "apps/x/**" + - ".github/workflows/x-tests.yml" + push: + branches: [main] + paths: + - "apps/x/**" + - ".github/workflows/x-tests.yml" + workflow_dispatch: + +jobs: + test: + runs-on: ubuntu-latest + defaults: + run: + working-directory: apps/x + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: 10 + + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + cache-dependency-path: apps/x/pnpm-lock.yaml + + - name: Install dependencies + run: pnpm install --frozen-lockfile + + # Builds @x/shared (core/renderer tests import its dist), then runs + # the shared, core, and renderer vitest suites in order. + - name: Run tests + run: npm test diff --git a/.gitignore b/.gitignore index 086ea0b5..c9eee447 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,6 @@ data/ .venv/ .claude/ + +# Local-only meeting-prep planning doc +/PLAN.md diff --git a/CLAUDE.md b/CLAUDE.md index 75a0f6a5..5e762a33 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,7 +109,9 @@ Long-form docs for specific features. Read the relevant file before making chang | Feature | Doc | |---------|-----| | Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` | +| Calls (video mode) — one hands-free call engine with four presets (voice / video / share screen / practice coaching), device-derived surfaces (full-screen ⇄ floating popout), frame pipeline, prompts catalog | `apps/x/VIDEO_MODE.md` | | Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` | +| Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` | ## Common Tasks diff --git a/README.md b/README.md index 361b87a0..934ab3b4 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,8 @@ - - rowboat-github-2 - -
+

Rowboat

+

A desktop AI coworker with a memory of your work and built-in surfaces to act on it.

+

rowboatlabs/rowboat | Trendshift @@ -25,28 +24,105 @@

-# Rowboat -**Open-source AI coworker that turns work into a knowledge graph and acts on it** - -Rowboat connects to your email and meeting notes, builds a long-lived knowledge graph, and uses that context to help you get work done - privately, on your machine. +Rowboat indexes your work into a living knowledge graph and uses that to get work done on your machine. It includes work surfaces for collaborating with AI: email client, notes, browser, code mode, meeting note taker, and workspaces for different projects. -You can do things like: -- `Build me a deck about our next quarter roadmap` → generates a PDF using context from your knowledge graph -- `Prep me for my meeting with Alex` → pulls past decisions, open questions, and relevant threads into a crisp brief (or a voice note) -- Track a person, company or topic through live notes -- Visualize, edit, and update your knowledge graph anytime (it’s just Markdown) -- Record voice memos that automatically capture and update key takeaways in the graph Download latest for Mac/Windows/Linux: [Download](https://www.rowboatlabs.com/downloads) +

+ +Rowboat demo video + +

+ +

+ Demo - apps to code · Demo - knowledge graph +

+ + ⭐ If you find Rowboat useful, please star the repo. It helps more people find it. -## Demo -[![Demo](https://github.com/user-attachments/assets/8b9a859b-d4f1-47ca-9d1d-9d26d982e15d)](https://www.youtube.com/watch?v=7xTpciZCfpw) +--- +## Overview -[Watch the full video](https://www.youtube.com/watch?v=7xTpciZCfpw) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+

Brain

+Rowboat indexes email, meetings, slack and assistant conversations into a living Obsidian-style backlinked knowledge graph. +
+Screenshot 2026-06-24 at 11 22 52 PM +
+

Email

+The built-in email client sorts emails into important and everything else. Rowboat automatically drafts responses for important email using all the work context. +
+Email screenshot +
+

Background agents

+You can set up background agents that run on events like new email or on schedule like every day at 8am. They can connect to tools, search the web, use the browser and write code using Claude Code or Codex. +
+Background agents screenshot + +
+

Built-in Browser

+Rowboat includes a browser that lets you and assistant collaborate on web tasks. Because its isolated from your main browser, you can log in only to the accounts that want the assistant to access. +
+Browser screenshot +
+

Meeting Notes

+A local meeting note-taker that taps into mic & speaker, produces live transcript and summarizes the meeting in a markdown file and updates the knowledge graph. +
+Meeting notes screenshot +
+

Code Mode

+Code mode lets you spin up parallel coding agents with Claude Code or Codex, and have Rowboat drive them with all the work context where needed. +
+Code mode screenshot +
+

Apps

+You can bulild your own work surfaces inside Rowboat — they get acess to all the tools and integrations, and you can share them with other people. +
+Apps screenshot +
+

Integrations

+Includes one-click integrations to most popular products. +
+Integrations screenshot +
--- @@ -81,23 +157,6 @@ All API key files use the same format: } ``` -## What it does - -Rowboat is a **local-first AI coworker** that can: -- **Remember** the important context you don’t want to re-explain (people, projects, decisions, commitments) -- **Understand** what’s relevant right now (before a meeting, while replying to an email, when writing a doc) -- **Help you act** by drafting, summarizing, planning, and producing real artifacts (briefs, emails, docs, PDF slides) - -Under the hood, Rowboat maintains an **Obsidian-compatible vault** of plain Markdown notes with backlinks — a transparent “working memory” you can inspect and edit. - -## Integrations - -Rowboat builds memory from the work you already do, including: -- **Gmail** (email) -- **Google Calendar** -- **Rowboat meeting notes** or **Fireflies** - -It also contains a library of product integrations through Composio.dev ## How it’s different @@ -111,24 +170,6 @@ Rowboat maintains **long-lived knowledge** instead: The result is memory that compounds, rather than retrieval that starts cold every time. -## What you can do with it - -- **Meeting prep** from prior decisions, threads, and open questions -- **Email drafting** grounded in history and commitments -- **Docs & decks** generated from your ongoing context (including PDF slides) -- **Follow-ups**: capture decisions, action items, and owners so nothing gets dropped -- **On-your-machine help**: create files, summarize into notes, and run workflows using local tools (with explicit, reviewable actions) - -## Live notes - -Live notes are notes that stay updated automatically. You can create one by typing '@rowboat' on a note. - -- Track a competitor or market topic across X, Reddit, and the news -- Monitor a person, project, or deal across web or your communications -- Keep a running summary of any subject you care about - -Everything is written back into your local Markdown vault. You control what runs and when. - ## Bring your own model Rowboat works with the model setup you prefer: diff --git a/apps/x/.gitignore b/apps/x/.gitignore index db195fb4..2c5f63fd 100644 --- a/apps/x/.gitignore +++ b/apps/x/.gitignore @@ -1,2 +1,3 @@ node_modules/ test-fixtures/ +*.tsbuildinfo diff --git a/apps/x/.pnpm-store/v11/index.db b/apps/x/.pnpm-store/v11/index.db new file mode 100644 index 00000000..7f9770bd Binary files /dev/null and b/apps/x/.pnpm-store/v11/index.db differ diff --git a/apps/x/ANALYTICS.md b/apps/x/ANALYTICS.md index 2d9816d0..b58d5f25 100644 --- a/apps/x/ANALYTICS.md +++ b/apps/x/ANALYTICS.md @@ -24,7 +24,7 @@ Emitted whenever ai-sdk returns token usage (one event per LLM call, not per run | Property | Type | Notes | |---|---|---| -| `use_case` | enum | `copilot_chat` / `live_note_agent` / `meeting_note` / `knowledge_sync` | +| `use_case` | enum | `copilot_chat` / `live_note_agent` / `meeting_note` / `knowledge_sync` / `code_session` | | `sub_use_case` | string? | Refines `use_case` — see taxonomy table below | | `agent_name` | string? | Present when the call goes through an agent run (`createRun`); omitted for direct `generateText`/`generateObject` | | `model` | string | e.g. `claude-sonnet-4-6` | @@ -57,6 +57,7 @@ Every `llm_usage` emit point in the codebase: | `knowledge_sync` | `inline_task_run` | yes | Inline `@rowboat` task execution (two call sites) | `packages/core/src/knowledge/inline_tasks.ts:471, 552` (createRun) | | `knowledge_sync` | `inline_task_classify` | no | Inline task scheduling classifier (`generateText`) | `packages/core/src/knowledge/inline_tasks.ts:673` | | `knowledge_sync` | `pre_built` | yes | Pre-built scheduled agents | `packages/core/src/pre_built/runner.ts:43` (createRun) | +| `code_session` | (none) | yes | Code-section coding session in Rowboat mode (direct mode talks to the on-device coding agent and emits no `llm_usage`) | `packages/core/src/code-mode/sessions/service.ts` (createRun) | ##### `live_note_agent` sub-use-case shape @@ -91,6 +92,8 @@ All in `apps/renderer/src/lib/analytics.ts`: - `chat_message_sent` — `{ voice_input, voice_output, search_enabled }` - `oauth_connected` / `oauth_disconnected` — `{ provider }` - `voice_input_started` — no properties +- `call_started` — `{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`) +- `call_turn_latency` — `{ endpoint_to_submit_ms, submit_to_speak_ms, speak_to_audio_ms, total_ms }` — voice-to-voice latency breakdown for one call turn (utterance accepted → submitted → first TTS speak → audio playing) - `search_executed` — `{ types: string[] }` - `note_exported` — `{ format }` diff --git a/apps/x/CODE_MODE_ENGINES_PLAN.md b/apps/x/CODE_MODE_ENGINES_PLAN.md new file mode 100644 index 00000000..ce7a4958 --- /dev/null +++ b/apps/x/CODE_MODE_ENGINES_PLAN.md @@ -0,0 +1,271 @@ +# Code Mode — Managed Engine Provisioning Plan + +Branch: `feat/code-mode-managed-engines` (off `dev` @ `8ce24ebb`) + +## 1. Problem & Goal + +Code mode runs two coding agents — **Claude Code** and **Codex** — by spawning their +ACP adapters, which in turn spawn a heavy **native engine binary** (~205 MB claude, +~194 MB codex). We need code mode to work in **packaged releases with ~99% reliability +for both agents**, without shipping a ~400 MB installer. + +### Current state (HEAD = revert of #614) +- Packaged builds **do not stage** the ACP adapters, and `forge.config.cjs` + `ignore: /^\/node_modules\//` strips them. At runtime `agents.ts` resolves the + adapter via `require.resolve(...)` then spawns it — which **throws + `Cannot find module '@agentclientprotocol/...'`**. +- **Net: packaged code mode is broken in every release.** It only works in `dev` + because pnpm symlinks exist. There is no 400 MB bloat today — but no function either. + +### Why the two prior approaches are insufficient +- **Bundle engines (A):** +~400 MB per installer (one claude + one codex native binary + per OS/arch). Works offline, but installer is huge. +- **Drive from user's local install (B)** — what #614 settled on and was reverted: + requires the user to have **both** CLIs installed **and** logged in, correct version, + on the right PATH. Depends on the user's machine → **structurally cannot hit 99%** + (version skew, GUI-launch PATH stripping, missing installs). This is why #614 was + reverted. + +## 2. Chosen Architecture — Managed Engine Provisioning (validated against Conductor) + +Split the problem into **engine** vs **auth**, treat them differently — exactly what +Conductor (conductor.build) does: + +1. **Engine = owned by the app.** We provision **version-pinned** engine binaries into + **app-support** (`~/.rowboat/engines///`), download-on-demand on first + use, sha256-verified, symlink/path-pinned. Not the user's global npm, not their PATH. + → no version skew, no PATH quirks → this is what delivers the 99%. +2. **Auth = reused from the user.** The engines read existing credentials + (`~/.claude` API key / Pro / Max, `~/.codex` auth.json). No second login. `status.ts` + already inspects these. + +### Empirical proof from Conductor on this machine +- DMG is **123 MB** — far smaller than the ~400 MB of engines → engines are **not** in + the installer; they're downloaded after install. +- Layout observed at `~/Library/Application Support/com.conductor.app/`: + ``` + agent-binaries/claude/2.1.170/claude (222 MB, single Mach-O arm64) + agent-binaries/codex/0.138.0/codex (205 MB, single Mach-O arm64) + bin/claude -> agent-binaries/claude/2.1.170/claude (symlink to active version) + bin/codex -> agent-binaries/codex/0.138.0/codex + agent-binaries/.meta/claude-2.1.170.json = {sha256, size, downloaded_at_unix_ms, ...} + ``` +- So: versioned dirs + stable symlink + sha256 `.meta` ledger + download. We mirror this. + +## 3. Concrete facts that make this clean (verified) + +### Adapters honor an external engine via env var (no code change in adapters needed) +- **Claude** — `@agentclientprotocol/claude-agent-acp@0.39.0`, + `dist/acp-agent.js:39`: `if (process.env.CLAUDE_CODE_EXECUTABLE) return it;` + and line 1552 `pathToClaudeCodeExecutable: process.env.CLAUDE_CODE_EXECUTABLE ?? ...`. + If unset and no bundled native dep → it throws "set CLAUDE_CODE_EXECUTABLE". +- **Codex** — `@agentclientprotocol/codex-acp@0.0.44`, + `dist/index.js:20900`: `const codexPath = process.env["CODEX_PATH"] ?? "codex";` + → `spawn(codexPath, ["app-server"])`. +- **Implication:** provisioning + setting these two env vars is the *entire* engine story. + +### Our engine packages are already single self-contained binaries +- `@anthropic-ai/claude-agent-sdk-darwin-arm64@0.3.156` → contains one `claude` + executable (+ LICENSE/README). +- `@openai/codex@0.128.0-darwin-arm64` → `vendor//codex/codex` native binary + **plus a bundled `rg` (ripgrep) at `vendor//path/rg`** — see Risk R1. + +### Pinned versions + platform package names (read from the installed adapter trees) +- **Claude:** adapter `claude-agent-acp@0.39.0` → engine `@anthropic-ai/claude-agent-sdk@0.3.156`. + Platform optional deps `@anthropic-ai/claude-agent-sdk-@0.3.156`: + `darwin-arm64`, `darwin-x64`, `linux-x64`, `linux-arm64`, `linux-x64-musl`, + `linux-arm64-musl`, `win32-x64`, `win32-arm64`. +- **Codex:** adapter `codex-acp@0.0.44` → `@openai/codex@^0.128.0` (pnpm-**patched**, see R2). + Platform deps aliased `@openai/codex-` → `npm:@openai/codex@0.128.0-`: + `darwin-arm64`, `darwin-x64`, `linux-x64`, `linux-arm64`, `win32-x64`, `win32-arm64`. + +## 4. Distribution source — npm platform packages, adapter-pinned (DECIDED) + +We fetch the **per-platform engine packages from the npm registry at the exact versions +our ACP adapters depend on**, extract the native binary, and provision it into +`~/.rowboat/engines/...`. No self-host, no curl installer, no fallback. + +- **Tarball URL:** `https://registry.npmjs.org//-/-.tgz` + - claude: `@anthropic-ai/claude-agent-sdk-@0.3.156` → extract the `claude` binary. + - codex: `@openai/codex@0.128.0-` → extract `vendor//codex/codex` + **and keep `vendor//path/rg`** (see R1). +- **Why adapter-pinned npm (not curl installer, not release bucket, not self-host):** + - The binary is the **exact version our adapter was built/tested against** → ACP + handshake guaranteed → the key to ~99% reliability. + - npm registry is highly available, immutable per-version, and the packument provides + `dist.integrity` (sha512) + `dist.shasum` (sha1) → integrity verification with **zero + infra**. + - The official `curl | bash` installer is for end-users: it installs globally to + `~/.local/bin` and **auto-updates in the background** — exactly what we must avoid. We + want an isolated, pinned, app-managed copy that never moves under the adapter. + (Conductor likewise fetches the raw binary rather than running the installer.) +- **Versions are read from our lockfile at build time** and embedded in + `engine-manifest.json`, so the manifest can never drift from the adapters we ship. + +### Reference: how this relates to Conductor / official installers +- Conductor self-hosts the same kind of native binaries on `storage.conductor.build` + (raw/`.gz`/`.zst` + `{url,gzipUrl,zstdUrl,sha256}` manifest), pairing the adapters with + recent **standalone CLI** versions (claude 2.1.x, codex 0.138). That proves recent + versions work — but we deliberately pin to the **adapter's own** engine version for + determinism. +- Official Claude releases also publish a GPG-signed `manifest.json` (SHA256/platform) at + `downloads.claude.ai/claude-code-releases//`. We don't use it now (npm is simpler + and adapter-matched), but it's the upgrade path if we ever want the latest CLI line. +- If we later want control/availability/compression, we can mirror these npm tarballs to + our own bucket — **runtime stays identical, only manifest URLs change.** + +### Locked decisions +- **Source: npm platform packages, adapter-pinned versions (user).** +- **No self-host (user).** +- **Always provision; no fallback** to a user's pre-installed `claude`/`codex` (user). + Reverted path B is fully retired. +- **Codex pnpm patch — IRRELEVANT (user):** it targets the JS launcher; we point + `CODEX_PATH` at the native binary, so it does not apply. (Risk R2 removed.) + +## 5. Design + +### 5.1 Build-time: generate an engine manifest + stage adapter JS + +**(a) Engine manifest (`engine-manifest.json`, embedded in the app).** +A build script reads the installed adapter dependency trees and emits, per agent: +```jsonc +{ + "claude": { + "version": "0.3.156", + "platforms": { + "darwin-arm64": { "pkg": "@anthropic-ai/claude-agent-sdk-darwin-arm64", + "tarball": "https://registry.npmjs.org/.../-/...-0.3.156.tgz", + "integrity": "sha512-...", + "binRelPath": "claude" }, + "...": {} + } + }, + "codex": { + "version": "0.128.0", + "platforms": { + "darwin-arm64": { "pkg": "@openai/codex-darwin-arm64", + "tarball": "https://registry.npmjs.org/...", + "integrity": "sha512-...", + "binRelPath": "vendor/aarch64-apple-darwin/codex/codex", + "extraPaths": ["vendor/aarch64-apple-darwin/path/rg"] } + } + } +} +``` +- Versions + tarball URLs + integrity are pulled from the lockfile / npm packument so the + manifest is always in sync with the adapters we ship. (Generating it at build time + sidesteps hardcoding fragile package names.) +- `binRelPath` / `extraPaths` capture where the executable (and codex's `rg`) live inside + each tarball. + +**(b) Stage the ACP adapter JS into the package (the part of #614 we KEEP).** +The adapters themselves (tiny, ~15 MB total incl. their non-native JS deps) must exist on +disk in packaged builds so `agents.ts` can resolve + spawn them. In `forge.config.cjs` +`generateAssets`, stage the two adapters and their **non-native** production dependency +closure into `.package/acp/node_modules` (npm-style nested layout), and **exempt +`.package` from the `node_modules` ignore rule**. We **drop** #614's native-engine staging +entirely — engines come from provisioning, not the bundle. + +### 5.2 Runtime: engine provisioner (new module in `packages/core`) + +New file: `packages/core/src/code-mode/acp/engine-provisioner.ts`. + +``` +ensureEngine(agent): Promise<{ executablePath: string }> + 1. Read manifest entry for (agent, currentPlatform). Error clearly if unsupported. + 2. dir = ~/.rowboat/engines/// + 3. If dir exists AND .meta/-.json sha256 matches → return binPath. + 4. Else acquire a cross-process lock (avoid double download), then: + a. Download tarball to a temp file, streaming, with progress events. + b. Verify integrity (sha512/sha256 from manifest). + c. Extract into a temp dir, then atomic rename into / (tar gzip). + d. chmod +x the binary (and codex's rg) on unix. + e. Write .meta/-.json {sha256, size, downloaded_at_unix_ms}. + 5. Return absolute path to the engine executable (binRelPath joined to dir). +``` +- **Progress + cancellation** surfaced over IPC for the first-run "Downloading engine…" UI. +- **Offline / failure** → typed error with a clear, actionable message (not a hang). +- **Resumability / atomicity:** download to temp, verify, then rename; never leave a + half-extracted version dir that passes the existence check. + +### 5.3 Wire provisioning into launch + +In `agents.ts` `getAgentLaunchSpec()` (currently sets only `CLAUDE_CODE_EXECUTABLE`): +- Make it (or its caller in `client.ts` / `manager.ts`) `await ensureEngine(agent)` first, + then set: + - claude → `env.CLAUDE_CODE_EXECUTABLE = ` + - codex → `env.CODEX_PATH = ` (and ensure its `rg` sibling resolves; + keep the vendor dir layout intact so codex finds `../path/rg`). +- Keep `ELECTRON_RUN_AS_NODE=1` for the adapter spawn (unchanged). +- The provisioner replaces both the bundled-engine dependency *and* the reverted + "resolve user's local install" path. (Optionally: if a healthy provisioned engine is + absent but a compatible local install exists, we *may* fall back to it — but the default + and reliable path is the provisioned engine. Decide in review; default = provisioned only.) + +### 5.4 Status & UX (`status.ts` + renderer) +- `checkCodeModeAgentStatus()` becomes: engine = provisioned? (instead of "installed on + PATH?"); auth = existing credentials present? (unchanged logic). +- First-run flow: user picks agent → if engine missing, show "Downloading engine + (~200 MB), one time…" with progress; on completion, proceed. Subsequent uses: instant. +- Clear error states: download failed / offline / unsupported platform / auth missing. + +## 6. File-by-file change plan + +| File | Change | +|---|---| +| `apps/main/forge.config.cjs` | Stage adapter JS closure into `.package/acp/node_modules`; exempt `.package` from node_modules ignore; generate + copy `engine-manifest.json`. (No native engines staged.) | +| `apps/main/scripts/gen-engine-manifest.mjs` *(new)* | Build-time: read lockfile/packuments → emit `engine-manifest.json` (versions, tarball URLs, integrity, bin paths). | +| `packages/core/src/code-mode/acp/engine-provisioner.ts` *(new)* | `ensureEngine()` — download, verify, extract, lock, progress, `.meta` ledger. | +| `packages/core/src/code-mode/acp/agents.ts` | Resolve adapter from staged `.package/acp` first (fallback to node_modules in dev); `await ensureEngine`; set `CLAUDE_CODE_EXECUTABLE`/`CODEX_PATH` to provisioned binaries. | +| `packages/core/src/code-mode/acp/client.ts` / `manager.ts` | Await provisioning before spawn; surface provisioning progress/errors; keep startup deadline. | +| `packages/core/src/code-mode/status.ts` | Engine status = provisioned (not PATH); keep auth checks. | +| `apps/main/src/ipc.ts` + preload + renderer | IPC for provisioning progress + first-run download UI + error states. | +| CI (optional) | Smoke: packaged app boots each adapter, provisions a (small fake) engine, sets env var, answers ACP `initialize`; offline → clear error not a hang. | + +## 7. Edge cases & risks + +- **R1 — Codex needs `rg`.** The codex platform package bundles ripgrep at + `vendor//path/rg`. We must extract/keep the vendor layout so codex finds it + (don't extract the bare binary). Verify codex `app-server` boots from the provisioned dir. +- **R2 — Codex pnpm patch. RESOLVED (irrelevant).** The patch targets the JS launcher; we + point `CODEX_PATH` at the native binary, so it does not apply. +- **R3 — Platform/arch matrix.** Manifest must cover darwin x64/arm64, linux x64/arm64 + (+musl for claude), win32 x64/arm64. Windows engine is `claude.exe`/`codex.exe`. +- **R4 — Integrity & supply chain.** Always verify the manifest integrity hash before + chmod/exec. Treat a hash mismatch as a hard failure. +- **R5 — Disk + upgrades.** Versioned dirs accumulate. Add a cleanup that keeps only the + active pinned version per agent. +- **R6 — First-run network.** Required once per agent; cached forever after. Must be a + clear, cancellable UX, never a silent hang (reuse the #614 startup-deadline lesson). +- **R7 — code signing / Gatekeeper (macOS).** Downloaded native binaries aren't covered by + our app's signature. Verify they run under Gatekeeper (they're already + signed/notarized by Anthropic/OpenAI; quarantine attr may need clearing). Conductor runs + them fine from app-support — confirm we do too. +- **R8 — `engine-manifest` staleness.** Manifest must regenerate whenever the adapter/engine + versions change; tie generation into the build so it can't drift. + +## 8. Phasing + +1. **P1 — Packaging fix (unblocks dev→packaged parity):** stage adapter JS into `.package`, + resolver checks staged path first. Verify packaged code mode can at least *spawn* the + adapter. (Independent of provisioning; the part of #614 worth keeping.) +2. **P2 — Provisioner (core):** `ensureEngine()` + manifest + wire env vars. Verify both + engines provision + boot from `~/.rowboat/engines` on macOS arm64. +3. **P3 — UX + status:** first-run download UI, status panel, error states. +4. **P4 — Cross-platform + CI smoke:** matrix manifest, mac/linux/win smoke. +5. **P5 — Polish:** version cleanup, cancellation, offline messaging, optional local-install + fallback. + +## 9. Decisions & remaining questions + +### Resolved (locked) +1. **Source = npm platform packages, adapter-pinned versions** (not self-host, not curl + installer, not release bucket). +2. **Always provision; no local-install fallback.** +3. **Codex pnpm patch irrelevant.** + +### Still open (can decide during implementation) +- **Provision timing:** on first code-mode *use* (lazy) vs first app launch (eager, + background download). Plan assumes **lazy + cached**. +- **Version-dir cleanup:** keep only the active pinned version per agent (R5). +- **Verify R1** (codex `rg`) and **R7** (Gatekeeper on downloaded macOS binaries) during P2. diff --git a/apps/x/MINI_APPS_PLAN.md b/apps/x/MINI_APPS_PLAN.md new file mode 100644 index 00000000..72a82a29 --- /dev/null +++ b/apps/x/MINI_APPS_PLAN.md @@ -0,0 +1,307 @@ +# Mini Apps — Implementation Plan + +> Status: **Phase 1 (UI + rendering, hand-coded apps)** in progress. +> Branch: `feature/mini-apps`. Working dir: `apps/x`. + +## 1. What we're building (agreed design) + +A **Mini App** is a user-created, personal app that lives inside Rowboat under a +**"Mini Apps"** sidebar tab, shown as cards; click a card to open and use it like +a standalone app. + +The settled mental model: + +> A Mini App = **custom UI code** (a single self-contained HTML file: React + +> Tailwind, no build step, runs in a sandboxed iframe) + an **agent "backend"** +> that produces fresh **data** on a trigger + an optional **state store** + a +> **scoped bridge** that lets the UI call real **Composio** actions. + +Key decisions (the "why" behind the architecture): + +- **Opens to a finished, populated screen.** The agent runs *before* open (on a + trigger); the UI just renders the latest result. Maps onto the existing + background-agents engine. +- **Frontend/backend split.** Code = frontend, written once. Agent = backend, + produces fresh `data.json` each run. The UI is *not* regenerated per run. + Cheaper, faster, and the UI can't break on refresh. +- **A data contract** ties the three pieces together: at creation the builder + produces a matched triple — UI code + data schema + agent instructions. +- **Fully custom UI per app** (generated code), not a fixed block kit. +- **Real interactivity** via a **scoped bridge** to Composio (each app declares + the integrations it may touch; scope drives both enforcement and the auth + prompt). +- **Built by whatever engine is selected in the chat box** — Code Mode + (Claude Code / Codex) if the user has it, else the Copilot. +- **Living apps:** refined by chatting; breakage is first-class (app surfaces + errors → "fix with copilot"). +- **State is an optional provided primitive** (per-app persistent store the agent + and UI can read/write). Stateless apps ignore it. +- **Personal/local only** for now, but each app is a **self-contained folder** + so it can become portable later. + +### Framework for the apps + +Single self-contained `index.html` rendered in a sandboxed iframe. + +> **Learning (Phase 1):** remote CDNs do NOT work inside the sandboxed, +> opaque-origin `srcdoc` iframe (React/Tailwind/Babel from unpkg/cdn.tailwindcss +> failed → blank screen). Apps must be **fully self-contained, no network**. +> Phase 1 therefore ships **vanilla HTML + CSS + JS** (no build, no transpile). +> The React+Tailwind LLM-friendly target still stands for generation — but via a +> **locally-bundled** runtime (esbuild-at-save, libs injected into the HTML), +> never remote CDNs. The `window.rowboat` bridge is unchanged either way. + +The UI talks to the host through a `window.rowboat` **bridge** (the product +surface *and* the security boundary): + +``` +rowboat.getData(): Promise // latest agent output +rowboat.onData(cb): void // re-render on refresh +rowboat.getState() / setState(patch) // per-app persistent store +rowboat.callAction(scope, action, args) // scoped Composio call +rowboat.ready() // handshake: "send me data" +``` + +## 2. Phased roadmap + +1. **Surface + one real app (UI-first — THIS PHASE).** "Mini Apps" sidebar tab, + card grid, click to open. One **hardcoded** app (Twitter client) rendered in a + sandboxed iframe from a self-contained HTML file, reading static data through a + minimal bridge. Proves the *experience* and the *rendering path*. No agent, + no storage, no real Composio yet. +2. **The scoped bridge.** Real interactive buttons → real Composio actions, + enforced against a per-app manifest scope; auth prompt on demand. +3. **Generation.** Copilot/Code-Mode generates the UI+schema+agent triple from a + chat description. Rides the existing chat mode selector + `code-with-agents`. +4. **Living apps + breakage recovery.** Conversational edits; error surfacing. +5. **Scale & polish.** Context-overload windowing (e.g. 100 tweets); app + notifications (reuse `notify-user`). +6. **(V2)** Drag an app into the main workspace (macOS-style). + +## 2b. Phase 2.5 — On-disk apps + `app://miniapp` serving (CURRENT) + +Move built-in apps out of source into `~/.rowboat/apps//`, served as static +assets, with an optional background agent producing `data.json`. Sets up the +copilot builder path (it will write the same folder layout). Team-agreed. + +### On-disk layout (one folder per app) +``` +~/.rowboat/apps// + manifest.json # id, title, description, source, scope[], active, lastRun, + # entry (default "dist/index.html"), agent (optional bg-task slug) + dist/ # static assets served via app://miniapp//... + index.html # the app (self-contained; bridge shim inlined) + data.json # latest agent output (read by the host, pushed to the app) +``` + +### Serving — `app://miniapp//` (option A) +- Extend `registerAppProtocol` (`apps/main/src/main.ts`) with a new host + `miniapp`: map `app://miniapp//` → `~/.rowboat/apps//dist/` + (path-traversal guarded; default `index.html`). `app://` is already registered + privileged (standard/secure/fetch/cors) so apps get a **real origin** — + remote CDNs/images and `fetch` work, unlike the opaque `srcdoc` origin. +- The iframe loads via `src="app://miniapp//index.html"` (not `srcDoc`). + Sandbox becomes `allow-scripts allow-same-origin allow-popups allow-forms + allow-modals allow-downloads` — same-origin is its OWN `app://miniapp` origin, + still isolated from the renderer (different host). + +### Data path +- Built-in/static `data` is seeded to `data.json`. A future background agent + (reuse bg-tasks engine, linked via manifest `agent`) overwrites it on schedule. +- App keeps using `rowboat.onData`; the **host** now sources that data by reading + `~/.rowboat/apps//data.json` (via IPC) and posting it on `ready`. Composio + still flows through the bridge RPC. (GitHub app needs no `data.json` — it pulls + live via Composio.) + +### Steps +1. **Shared schema** — `packages/shared/src/mini-app.ts`: `MiniAppManifest` zod + + type. Export it. New IPC channels in `shared/src/ipc.ts`: + - `mini-apps:seed` (req: `{apps:[{manifest, html, data?}]}`) — idempotent. + - `mini-apps:list` (res: `{manifests: MiniAppManifest[]}`). + - `mini-apps:get-data` (req `{id}`, res `{data: unknown|null}`). +2. **Main** — `apps/main/src/mini-apps-handler.ts`: `seedApps` (write + manifest/dist/data to `~/.rowboat/apps//` only if absent), `listApps` + (read manifests), `getAppData` (read data.json). Register handlers in + `ipc.ts`. Extend `registerAppProtocol` for the `miniapp` host. +3. **Renderer** — keep `MiniApp` defs + `buildMiniAppHtml`; `registry.ts` adds + `toSeed(app)` (→ `{manifest, html, data}`). `MiniAppsView`: on mount → `seed` + → `list` → render cards from manifests. `MiniAppFrame`: take the manifest, + load `src=app://miniapp//`, on `ready` fetch `mini-apps:get-data` + and post it; RPC scope from `manifest.scope`. +4. **Verify**: GitHub app end-to-end from disk (`~/.rowboat/apps/github-radar/`), + then the others; confirm connect + live PRs still work. + +### Out of scope here (next: copilot builder, tomorrow) +Copilot writing a new app folder + an associated background agent; the agent +browsing via embedded browser for social feeds; copilot verifying Composio wiring +by actually calling tools before finalizing. + +## 2c. Remaining work (after on-disk move) + +Done so far: surface + runtime (Phase 1), real scoped Composio bridge (Phase 2), +apps on disk served via `app://miniapp` (Phase 2.5). + +**Next — Copilot builder (demo target).** +- Copilot creates an app folder in `~/.rowboat/apps//` via the `mini-apps:seed` + install primitive (writes `manifest.json` + `dist/index.html`). +- Copilot must **verify wiring by actually calling Composio tools** and inspecting + the returned data before finalizing — never speculate the shape. +- Give generated apps the bridge contract — prefer **serving a canonical shim** + from the protocol (e.g. `app://miniapp/__bridge__.js`) over inlining. + +**Background-agent data pipeline (deterministic).** +- Reuse the existing bg-tasks engine; link via `manifest.agent`. +- The agent does NOT write files. It **returns structured data validated against + the app's data schema**; the bg-task **runner (code) atomically writes the + app's `data.json`** (temp→rename, keep last-good on failure). Path / atomicity + / validation / fallback all live in code — only the *content* is LLM-driven. +- Social feeds (Twitter/LinkedIn/Reddit): the agent browses via the embedded + browser, curates, returns data → runner writes `data.json`. + +**Later (roadmap).** +- Living apps + breakage recovery (edit by chat; surface errors → fix-with-copilot). +- Scale/polish: context-overload windowing; app notifications; design-consistency + / component templates (team-deferred). +- V2: drag an app into the main workspace. + +## 2d. Mini App builder skill — spec + +A Copilot skill (`build-mini-app`, in `packages/core/src/application/assistant/ +skills/`) that turns a chat request into an installed app under `~/.rowboat/ +apps//`. Copilot orchestrates; the actual code-writing is delegated by the +chat's active engine (the chip), but the on-disk artifact is identical either way. + +**Trigger + intent gate** (like live-note/background-task signal tiers): +- **Strong (build directly):** "make/build/create an app · mini app · dashboard + for …", "turn this into an app". +- **Medium (CONFIRM FIRST):** requests that could be a one-off answer or a + recurring app — e.g. "show me my open PRs", "track competitor launches". Ask: + "Want this as a Mini App you can reopen, or just a one-time answer?" Build only + on yes. (Building installs a folder + maybe a bg agent + an OAuth prompt — too + heavy to trigger on a casual question.) +- **Anti (do NOT build):** clearly one-off lookups/questions → just answer. + +**Flow:** +1. **Scope the app** — derive `id` (slug), `title`, `description`, `source`, the + Composio `scope[]`, and whether it's **agent-backed** (scheduled data) or + **live** (calls Composio on use). +2. **Verify wiring BEFORE building** (mandatory, engine-agnostic) — ensure the + toolkits are connected (prompt OAuth if not), then actually call the needed + tools (`composio-search-tools` → `composio-execute-tool`), inspect the REAL + returned data, and derive the **data schema from actual responses** — never + guess the shape. +3. **Pick the writer (branch):** + - **Code Mode active** → create the folder + a manifest skeleton, then + `code_agent_run` with `cwd = ~/.rowboat/apps//` to author + `dist/index.html` against the verified schema + bridge contract; it can + iterate/test on-device. + - **No Code Mode** → Copilot writes `dist/index.html` itself (from the app + template + bridge shim) and installs via `mini-apps:seed`. +4. **Bridge contract** — generated app references the canonical shim + (`app://miniapp/__bridge__.js`) and uses `window.rowboat` + (`getData/onData`, `isConnected/connect`, `searchTools/callAction`). +5. **Data pipeline (if agent-backed)** — create a background task (existing + bg-tasks engine), set `manifest.agent` to its slug. The agent **returns + schema-validated data**; the **runner writes `data.json` deterministically** + (temp→rename, last-good on failure). App reads it via `onData`. +6. **Finalize** — write `manifest.json` (incl. `scope`, `agent?`), ensure + `dist/index.html`, install → app appears in the gallery (`mini-apps:list`). + Confirm end-to-end (open it; data loads). + +**Infra this needs (repo side):** +- Serve the canonical bridge shim from the protocol: `app://miniapp/__bridge__.js` + (move the shim out of per-app inlining into served infra). +- bg-tasks runner: when a task is app-linked, validate its structured output + against the app schema and write that app's `data.json` (deterministic write + mode) instead of `index.md`. +- A `build-mini-app` skill registered in the skills catalog. + +## 3. Phase 1 — detailed implementation + +UI-first. Everything hand-coded; no `~/.rowboat` storage, no IPC, no agent yet. +We *do* mirror the eventual shapes so later phases slot in. + +### 3.1 New files (renderer) + +| File | Purpose | +|------|---------| +| `apps/renderer/src/mini-apps/types.ts` | `MiniApp` type (id, name, description, icon, accent, scope, html, data). | +| `apps/renderer/src/mini-apps/registry.ts` | Hardcoded list of `MiniApp`s for Phase 1. | +| `apps/renderer/src/mini-apps/apps/twitter-client.ts` | The sample app: self-contained HTML (React+Tailwind+Babel CDN) + static `data`. | +| `apps/renderer/src/components/mini-apps-view.tsx` | Card grid; internal `selectedAppId` state; renders grid or open app. | +| `apps/renderer/src/components/mini-app-frame.tsx` | Sandboxed iframe (`srcdoc`) + `window.rowboat` postMessage bridge (data wired; actions stubbed → toast/log). | + +The bridge message protocol (host ↔ iframe), defined once and shared: +- iframe → host: `{ type: 'rowboat:mini-app:ready' }`, + `{ type: 'rowboat:mini-app:action', id, scope, action, args }`, + `{ type: 'rowboat:mini-app:setState', patch }` +- host → iframe: `{ type: 'rowboat:mini-app:data', data }`, + `{ type: 'rowboat:mini-app:state', state }`, + `{ type: 'rowboat:mini-app:action-result', id, ok, result?, error? }` + +### 3.2 Wiring into `App.tsx` (mirror the `bg-tasks` view exactly) + +Add a first-class `apps` view. Edit sites (all mirror an existing view): + +1. **Tab path const** (~L198): add `const APPS_TAB_PATH = '__rowboat_mini_apps__'`. +2. **Tab predicate** (~L374): `const isAppsTabPath = (path) => path === APPS_TAB_PATH`. +3. **ViewState union** (~L636): add `| { type: 'apps' }`. +4. **`parseDeepLink`** (~L713): add `case 'apps': return { type: 'apps' }`. +5. **State flag** (~L825): `const [isAppsOpen, setIsAppsOpen] = useState(false)`. +6. **Tab title** (~L1253): `if (isAppsTabPath(tab.path)) return 'Mini Apps'`. +7. **Tab activation → flag** (~L3269, ~L3443): set `isAppsOpen` from tab path. +8. **Closing-tab guard** (~L3376): add `&& !isAppsTabPath(closingTab.path)`. +9. **`currentViewState`** (~L3770): `if (isAppsOpen) return { type: 'apps' }` + (place alongside bg-tasks; add to deps array). +10. **`ensureAppsFileTab`** (~L3853): mirror `ensureBgTasksFileTab`. +11. **`openAppsView`** (~L3953): mirror `openBgTasksView`; reset all other flags, + `setIsAppsOpen(true)`, `ensureAppsFileTab()`. +12. **`applyViewState` switch** (~L4195): add `case 'apps':` mirroring `bg-tasks`. +13. **Add `setIsAppsOpen(false)`** to every *other* view's reset block (the + shared multi-`set...(false)` lines) so opening another view clears apps and + closing it doesn't fall back to apps. Use targeted edits per block. +14. **Derived booleans** that OR all view flags (isFullScreenChat, + rightPaneAvailable, inFileView, rightPaneContext, viewOpen, keyboard guards): + add `|| isAppsOpen` / `&& !isAppsOpen` consistently (search `isBgTasksOpen`). +15. **Tab-path mapping** (~L4668): add `: isAppsOpen ? APPS_TAB_PATH`. +16. **Render branch** (~L5894): add `) : isAppsOpen ? (`. + +### 3.3 Sidebar entry (`sidebar-content.tsx`) + +- Add `onOpenApps?: () => void` prop (~L163) and destructure (~L412). +- Add a `SidebarMenuButton` with `isActive={activeNav === 'apps'}` and a + `LayoutGrid` (lucide) icon, label "Mini Apps", in the lower nav group near + "Background agents" (~L830). +- Thread `activeNav === 'apps'` from the parent (App.tsx passes `activeNav` based + on `isAppsOpen`, mirroring how `'agents'` is derived ~L5651). +- Wire `onOpenApps={openAppsView}` at the `SidebarContent`/home call sites + (~L5657, ~L5832). + +### 3.4 The sample app (twitter-client) + +Self-contained HTML string. React + ReactDOM + Babel-standalone + Tailwind, all +via CDN. Renders three sections — **To read / To repost / To respond** (reply +pre-drafted) — from data delivered via `rowboat.onData`. Buttons call +`rowboat.callAction('twitter', ...)`; in Phase 1 the host just toasts/logs and +returns a fake ok. Static `data` lives next to the HTML. + +> Note: CDN scripts require network + `allow-scripts` in the sandbox. The frame +> uses `sandbox="allow-scripts allow-popups allow-forms allow-modals"` — **no** +> `allow-same-origin` (keeps the app from reaching host cookies/storage; the +> bridge is the only channel). Verify CDN loads under this sandbox during testing; +> if blocked, fall back to bundling the libs locally as a renderer asset. + +### 3.5 Verify + +- `cd apps/x && npm run deps && npm run lint` compiles clean. +- `npm run dev`: "Mini Apps" appears in the sidebar; opens the card grid; clicking + the Twitter card renders the app full-screen in the pane; sections populate from + data; clicking a button shows the stubbed action result; tabs/back/forward and + switching to other views behave like every other view. + +## 4. Out of scope for Phase 1 (later phases) + +`~/.rowboat/apps//` storage + IPC; the agent backend (reuse bg-tasks +engine); real Composio execution through the bridge; per-app scope enforcement & +auth prompting; generation via Copilot/Code-Mode; persistent state store; +conversational editing & error recovery; the V2 workspace drag. diff --git a/apps/x/VIDEO_MODE.md b/apps/x/VIDEO_MODE.md new file mode 100644 index 00000000..653521cb --- /dev/null +++ b/apps/x/VIDEO_MODE.md @@ -0,0 +1,253 @@ +# Calls (Video Mode) — Deep Dive + +Calls let the user talk to the assistant hands-free while it *sees* them +(webcam) and their screen (screen share). There is ONE call engine — +continuous listening, auto-submitted utterances, forced read-aloud TTS, frame +capture — entered through four presets that differ only in starting devices. +This doc covers the product flow, the technical pipeline, and the LLM prompt +surface with exact pointers. + +## Product flow + +The composer has a **call split-button** (`chat-input-with-mentions.tsx`). +The main click is the "work together" default — preset `share`: screen +sharing ON, camera OFF, floating pill, so the user keeps working while the +assistant watches along (the button tooltip discloses the screen share). The +chevron menu holds the deviations. While a call is live the button turns red +and ends it. + +| Preset | Starting devices | First surface | +|--------|------------------|---------------| +| `share` — main click | screen on, camera off | floating pill | +| `voice` — "Voice call" | camera off, screen off | floating mascot pill | +| `video` — "Video call" | camera on | full-screen call | +| `practice` — "Practice session" | camera on, + coaching persona | full-screen call | + +**One surface rule** (`callSurface` in `App.tsx`): full screen and screen +sharing are mutually exclusive in both directions — a full-screen call covers +the screen, so sharing it would show the call itself. + +- sharing → floating popout, always (pill = working) +- not sharing → full screen unless `callMinimized` (full screen = facing + each other) +- expanding the pill auto-STOPS any share; minimizing the full-screen call + auto-STARTS one (the pill exists to work together) — presenting from full + screen likewise collapses to the pill +- the camera toggle never changes the surface: turning it on from the pill + puts your video IN the pill; expanding is its own explicit action + +**Screen-share consent** is three-layered: a toast the moment any share +starts ("Your screen is being shared… [Stop sharing]"), a persistent +"Sharing screen" badge on the pill, and macOS's purple recording indicator. +If the auto-share fails (Screen Recording permission not granted) the call +starts anyway as a voice call, with a toast linking to System Settings. +Practice/coaching is always an explicit choice — expanding to full screen +never turns the coach on. + +In-call controls (identical bar on both surfaces): mic mute, camera toggle +(silhouette avatar while off, no webcam frames captured), screen share +toggle, mascot ⇄ "R" letter avatar, end call. **Mute is a full input +pause**, not just audio — mic audio stops reaching Deepgram +(`useVoiceMode.setPaused`, OR'd with the automatic thinking/speaking pause) +AND camera/screen frame capture stops (`useVideoMode.setCapturePaused`; +`collectFrames()` returns nothing while muted, so typed messages carry no +frames either), letting the user talk to someone in the room without the +assistant listening in. Devices stay acquired for instant unmute (camera +light and macOS share indicator stay on — the pill's share badge switches to +"Sharing paused"), the status chip shows "Muted" instead of "Listening", +and assistant output is unaffected (in-flight speech keeps playing; Stop +handles that). Mute resets to off at call start/end. While the assistant is thinking or speaking, a +red **Stop** button appears on the mascot tile — it silences TTS instantly, +skips queued voice segments, and aborts the run if it's still generating +(stopping a run from anywhere, including the composer, also silences TTS). Captions of the in-progress utterance and the +assistant's spoken line run along the bottom. Typing in the composer still +works mid-call; frames ride along with typed messages too. + +Outside calls the composer keeps exactly one voice affordance: the **mic +button** (push-to-talk dictation, untouched). Spoken responses exist only +inside calls (forced full read-aloud, off on hang-up). The old video +dropdown, talking-head toggle, read-aloud headphones toggle, and summary/full +TTS dropdown are all retired — a per-message "read aloud" action on assistant +messages is the planned replacement for text-in/voice-out. + +The call button is disabled unless both voice input (Deepgram) and voice +output (TTS) are configured. `call_started` (with `preset`) is captured in +PostHog — the adoption metric for this feature. + +**Popout mechanics**: a small always-on-top frameless window (camera tile +when on + mascot tile, live caption, control bar) floating over every app — +including Rowboat. Control-bar actions round-trip `video:popoutAction` → +main → `video:popout-action` → app window, which owns the mic/camera/capture; +`expand` also refocuses the app window (handled in main). + +## Frame pipeline + +`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source +(stream → offscreen `