diff --git a/CLAUDE.md b/CLAUDE.md index bf34bd90..5e762a33 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,6 +109,7 @@ Long-form docs for specific features. Read the relevant file before making chang | Feature | Doc | |---------|-----| | Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` | +| Calls (video mode) — one hands-free call engine with four presets (voice / video / share screen / practice coaching), device-derived surfaces (full-screen ⇄ floating popout), frame pipeline, prompts catalog | `apps/x/VIDEO_MODE.md` | | Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` | | Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` | diff --git a/apps/x/.pnpm-store/v11/index.db b/apps/x/.pnpm-store/v11/index.db new file mode 100644 index 00000000..7f9770bd Binary files /dev/null and b/apps/x/.pnpm-store/v11/index.db differ diff --git a/apps/x/ANALYTICS.md b/apps/x/ANALYTICS.md index 5ddfcf6e..b58d5f25 100644 --- a/apps/x/ANALYTICS.md +++ b/apps/x/ANALYTICS.md @@ -92,6 +92,8 @@ All in `apps/renderer/src/lib/analytics.ts`: - `chat_message_sent` — `{ voice_input, voice_output, search_enabled }` - `oauth_connected` / `oauth_disconnected` — `{ provider }` - `voice_input_started` — no properties +- `call_started` — `{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`) +- `call_turn_latency` — `{ endpoint_to_submit_ms, submit_to_speak_ms, speak_to_audio_ms, total_ms }` — voice-to-voice latency breakdown for one call turn (utterance accepted → submitted → first TTS speak → audio playing) - `search_executed` — `{ types: string[] }` - `note_exported` — `{ format }` diff --git a/apps/x/MINI_APPS_PLAN.md b/apps/x/MINI_APPS_PLAN.md new file mode 100644 index 00000000..72a82a29 --- /dev/null +++ b/apps/x/MINI_APPS_PLAN.md @@ -0,0 +1,307 @@ +# Mini Apps — Implementation Plan + +> Status: **Phase 1 (UI + rendering, hand-coded apps)** in progress. +> Branch: `feature/mini-apps`. Working dir: `apps/x`. + +## 1. What we're building (agreed design) + +A **Mini App** is a user-created, personal app that lives inside Rowboat under a +**"Mini Apps"** sidebar tab, shown as cards; click a card to open and use it like +a standalone app. + +The settled mental model: + +> A Mini App = **custom UI code** (a single self-contained HTML file: React + +> Tailwind, no build step, runs in a sandboxed iframe) + an **agent "backend"** +> that produces fresh **data** on a trigger + an optional **state store** + a +> **scoped bridge** that lets the UI call real **Composio** actions. + +Key decisions (the "why" behind the architecture): + +- **Opens to a finished, populated screen.** The agent runs *before* open (on a + trigger); the UI just renders the latest result. Maps onto the existing + background-agents engine. +- **Frontend/backend split.** Code = frontend, written once. Agent = backend, + produces fresh `data.json` each run. The UI is *not* regenerated per run. + Cheaper, faster, and the UI can't break on refresh. +- **A data contract** ties the three pieces together: at creation the builder + produces a matched triple — UI code + data schema + agent instructions. +- **Fully custom UI per app** (generated code), not a fixed block kit. +- **Real interactivity** via a **scoped bridge** to Composio (each app declares + the integrations it may touch; scope drives both enforcement and the auth + prompt). +- **Built by whatever engine is selected in the chat box** — Code Mode + (Claude Code / Codex) if the user has it, else the Copilot. +- **Living apps:** refined by chatting; breakage is first-class (app surfaces + errors → "fix with copilot"). +- **State is an optional provided primitive** (per-app persistent store the agent + and UI can read/write). Stateless apps ignore it. +- **Personal/local only** for now, but each app is a **self-contained folder** + so it can become portable later. + +### Framework for the apps + +Single self-contained `index.html` rendered in a sandboxed iframe. + +> **Learning (Phase 1):** remote CDNs do NOT work inside the sandboxed, +> opaque-origin `srcdoc` iframe (React/Tailwind/Babel from unpkg/cdn.tailwindcss +> failed → blank screen). Apps must be **fully self-contained, no network**. +> Phase 1 therefore ships **vanilla HTML + CSS + JS** (no build, no transpile). +> The React+Tailwind LLM-friendly target still stands for generation — but via a +> **locally-bundled** runtime (esbuild-at-save, libs injected into the HTML), +> never remote CDNs. The `window.rowboat` bridge is unchanged either way. + +The UI talks to the host through a `window.rowboat` **bridge** (the product +surface *and* the security boundary): + +``` +rowboat.getData(): Promise // latest agent output +rowboat.onData(cb): void // re-render on refresh +rowboat.getState() / setState(patch) // per-app persistent store +rowboat.callAction(scope, action, args) // scoped Composio call +rowboat.ready() // handshake: "send me data" +``` + +## 2. Phased roadmap + +1. **Surface + one real app (UI-first — THIS PHASE).** "Mini Apps" sidebar tab, + card grid, click to open. One **hardcoded** app (Twitter client) rendered in a + sandboxed iframe from a self-contained HTML file, reading static data through a + minimal bridge. Proves the *experience* and the *rendering path*. No agent, + no storage, no real Composio yet. +2. **The scoped bridge.** Real interactive buttons → real Composio actions, + enforced against a per-app manifest scope; auth prompt on demand. +3. **Generation.** Copilot/Code-Mode generates the UI+schema+agent triple from a + chat description. Rides the existing chat mode selector + `code-with-agents`. +4. **Living apps + breakage recovery.** Conversational edits; error surfacing. +5. **Scale & polish.** Context-overload windowing (e.g. 100 tweets); app + notifications (reuse `notify-user`). +6. **(V2)** Drag an app into the main workspace (macOS-style). + +## 2b. Phase 2.5 — On-disk apps + `app://miniapp` serving (CURRENT) + +Move built-in apps out of source into `~/.rowboat/apps//`, served as static +assets, with an optional background agent producing `data.json`. Sets up the +copilot builder path (it will write the same folder layout). Team-agreed. + +### On-disk layout (one folder per app) +``` +~/.rowboat/apps// + manifest.json # id, title, description, source, scope[], active, lastRun, + # entry (default "dist/index.html"), agent (optional bg-task slug) + dist/ # static assets served via app://miniapp//... + index.html # the app (self-contained; bridge shim inlined) + data.json # latest agent output (read by the host, pushed to the app) +``` + +### Serving — `app://miniapp//` (option A) +- Extend `registerAppProtocol` (`apps/main/src/main.ts`) with a new host + `miniapp`: map `app://miniapp//` → `~/.rowboat/apps//dist/` + (path-traversal guarded; default `index.html`). `app://` is already registered + privileged (standard/secure/fetch/cors) so apps get a **real origin** — + remote CDNs/images and `fetch` work, unlike the opaque `srcdoc` origin. +- The iframe loads via `src="app://miniapp//index.html"` (not `srcDoc`). + Sandbox becomes `allow-scripts allow-same-origin allow-popups allow-forms + allow-modals allow-downloads` — same-origin is its OWN `app://miniapp` origin, + still isolated from the renderer (different host). + +### Data path +- Built-in/static `data` is seeded to `data.json`. A future background agent + (reuse bg-tasks engine, linked via manifest `agent`) overwrites it on schedule. +- App keeps using `rowboat.onData`; the **host** now sources that data by reading + `~/.rowboat/apps//data.json` (via IPC) and posting it on `ready`. Composio + still flows through the bridge RPC. (GitHub app needs no `data.json` — it pulls + live via Composio.) + +### Steps +1. **Shared schema** — `packages/shared/src/mini-app.ts`: `MiniAppManifest` zod + + type. Export it. New IPC channels in `shared/src/ipc.ts`: + - `mini-apps:seed` (req: `{apps:[{manifest, html, data?}]}`) — idempotent. + - `mini-apps:list` (res: `{manifests: MiniAppManifest[]}`). + - `mini-apps:get-data` (req `{id}`, res `{data: unknown|null}`). +2. **Main** — `apps/main/src/mini-apps-handler.ts`: `seedApps` (write + manifest/dist/data to `~/.rowboat/apps//` only if absent), `listApps` + (read manifests), `getAppData` (read data.json). Register handlers in + `ipc.ts`. Extend `registerAppProtocol` for the `miniapp` host. +3. **Renderer** — keep `MiniApp` defs + `buildMiniAppHtml`; `registry.ts` adds + `toSeed(app)` (→ `{manifest, html, data}`). `MiniAppsView`: on mount → `seed` + → `list` → render cards from manifests. `MiniAppFrame`: take the manifest, + load `src=app://miniapp//`, on `ready` fetch `mini-apps:get-data` + and post it; RPC scope from `manifest.scope`. +4. **Verify**: GitHub app end-to-end from disk (`~/.rowboat/apps/github-radar/`), + then the others; confirm connect + live PRs still work. + +### Out of scope here (next: copilot builder, tomorrow) +Copilot writing a new app folder + an associated background agent; the agent +browsing via embedded browser for social feeds; copilot verifying Composio wiring +by actually calling tools before finalizing. + +## 2c. Remaining work (after on-disk move) + +Done so far: surface + runtime (Phase 1), real scoped Composio bridge (Phase 2), +apps on disk served via `app://miniapp` (Phase 2.5). + +**Next — Copilot builder (demo target).** +- Copilot creates an app folder in `~/.rowboat/apps//` via the `mini-apps:seed` + install primitive (writes `manifest.json` + `dist/index.html`). +- Copilot must **verify wiring by actually calling Composio tools** and inspecting + the returned data before finalizing — never speculate the shape. +- Give generated apps the bridge contract — prefer **serving a canonical shim** + from the protocol (e.g. `app://miniapp/__bridge__.js`) over inlining. + +**Background-agent data pipeline (deterministic).** +- Reuse the existing bg-tasks engine; link via `manifest.agent`. +- The agent does NOT write files. It **returns structured data validated against + the app's data schema**; the bg-task **runner (code) atomically writes the + app's `data.json`** (temp→rename, keep last-good on failure). Path / atomicity + / validation / fallback all live in code — only the *content* is LLM-driven. +- Social feeds (Twitter/LinkedIn/Reddit): the agent browses via the embedded + browser, curates, returns data → runner writes `data.json`. + +**Later (roadmap).** +- Living apps + breakage recovery (edit by chat; surface errors → fix-with-copilot). +- Scale/polish: context-overload windowing; app notifications; design-consistency + / component templates (team-deferred). +- V2: drag an app into the main workspace. + +## 2d. Mini App builder skill — spec + +A Copilot skill (`build-mini-app`, in `packages/core/src/application/assistant/ +skills/`) that turns a chat request into an installed app under `~/.rowboat/ +apps//`. Copilot orchestrates; the actual code-writing is delegated by the +chat's active engine (the chip), but the on-disk artifact is identical either way. + +**Trigger + intent gate** (like live-note/background-task signal tiers): +- **Strong (build directly):** "make/build/create an app · mini app · dashboard + for …", "turn this into an app". +- **Medium (CONFIRM FIRST):** requests that could be a one-off answer or a + recurring app — e.g. "show me my open PRs", "track competitor launches". Ask: + "Want this as a Mini App you can reopen, or just a one-time answer?" Build only + on yes. (Building installs a folder + maybe a bg agent + an OAuth prompt — too + heavy to trigger on a casual question.) +- **Anti (do NOT build):** clearly one-off lookups/questions → just answer. + +**Flow:** +1. **Scope the app** — derive `id` (slug), `title`, `description`, `source`, the + Composio `scope[]`, and whether it's **agent-backed** (scheduled data) or + **live** (calls Composio on use). +2. **Verify wiring BEFORE building** (mandatory, engine-agnostic) — ensure the + toolkits are connected (prompt OAuth if not), then actually call the needed + tools (`composio-search-tools` → `composio-execute-tool`), inspect the REAL + returned data, and derive the **data schema from actual responses** — never + guess the shape. +3. **Pick the writer (branch):** + - **Code Mode active** → create the folder + a manifest skeleton, then + `code_agent_run` with `cwd = ~/.rowboat/apps//` to author + `dist/index.html` against the verified schema + bridge contract; it can + iterate/test on-device. + - **No Code Mode** → Copilot writes `dist/index.html` itself (from the app + template + bridge shim) and installs via `mini-apps:seed`. +4. **Bridge contract** — generated app references the canonical shim + (`app://miniapp/__bridge__.js`) and uses `window.rowboat` + (`getData/onData`, `isConnected/connect`, `searchTools/callAction`). +5. **Data pipeline (if agent-backed)** — create a background task (existing + bg-tasks engine), set `manifest.agent` to its slug. The agent **returns + schema-validated data**; the **runner writes `data.json` deterministically** + (temp→rename, last-good on failure). App reads it via `onData`. +6. **Finalize** — write `manifest.json` (incl. `scope`, `agent?`), ensure + `dist/index.html`, install → app appears in the gallery (`mini-apps:list`). + Confirm end-to-end (open it; data loads). + +**Infra this needs (repo side):** +- Serve the canonical bridge shim from the protocol: `app://miniapp/__bridge__.js` + (move the shim out of per-app inlining into served infra). +- bg-tasks runner: when a task is app-linked, validate its structured output + against the app schema and write that app's `data.json` (deterministic write + mode) instead of `index.md`. +- A `build-mini-app` skill registered in the skills catalog. + +## 3. Phase 1 — detailed implementation + +UI-first. Everything hand-coded; no `~/.rowboat` storage, no IPC, no agent yet. +We *do* mirror the eventual shapes so later phases slot in. + +### 3.1 New files (renderer) + +| File | Purpose | +|------|---------| +| `apps/renderer/src/mini-apps/types.ts` | `MiniApp` type (id, name, description, icon, accent, scope, html, data). | +| `apps/renderer/src/mini-apps/registry.ts` | Hardcoded list of `MiniApp`s for Phase 1. | +| `apps/renderer/src/mini-apps/apps/twitter-client.ts` | The sample app: self-contained HTML (React+Tailwind+Babel CDN) + static `data`. | +| `apps/renderer/src/components/mini-apps-view.tsx` | Card grid; internal `selectedAppId` state; renders grid or open app. | +| `apps/renderer/src/components/mini-app-frame.tsx` | Sandboxed iframe (`srcdoc`) + `window.rowboat` postMessage bridge (data wired; actions stubbed → toast/log). | + +The bridge message protocol (host ↔ iframe), defined once and shared: +- iframe → host: `{ type: 'rowboat:mini-app:ready' }`, + `{ type: 'rowboat:mini-app:action', id, scope, action, args }`, + `{ type: 'rowboat:mini-app:setState', patch }` +- host → iframe: `{ type: 'rowboat:mini-app:data', data }`, + `{ type: 'rowboat:mini-app:state', state }`, + `{ type: 'rowboat:mini-app:action-result', id, ok, result?, error? }` + +### 3.2 Wiring into `App.tsx` (mirror the `bg-tasks` view exactly) + +Add a first-class `apps` view. Edit sites (all mirror an existing view): + +1. **Tab path const** (~L198): add `const APPS_TAB_PATH = '__rowboat_mini_apps__'`. +2. **Tab predicate** (~L374): `const isAppsTabPath = (path) => path === APPS_TAB_PATH`. +3. **ViewState union** (~L636): add `| { type: 'apps' }`. +4. **`parseDeepLink`** (~L713): add `case 'apps': return { type: 'apps' }`. +5. **State flag** (~L825): `const [isAppsOpen, setIsAppsOpen] = useState(false)`. +6. **Tab title** (~L1253): `if (isAppsTabPath(tab.path)) return 'Mini Apps'`. +7. **Tab activation → flag** (~L3269, ~L3443): set `isAppsOpen` from tab path. +8. **Closing-tab guard** (~L3376): add `&& !isAppsTabPath(closingTab.path)`. +9. **`currentViewState`** (~L3770): `if (isAppsOpen) return { type: 'apps' }` + (place alongside bg-tasks; add to deps array). +10. **`ensureAppsFileTab`** (~L3853): mirror `ensureBgTasksFileTab`. +11. **`openAppsView`** (~L3953): mirror `openBgTasksView`; reset all other flags, + `setIsAppsOpen(true)`, `ensureAppsFileTab()`. +12. **`applyViewState` switch** (~L4195): add `case 'apps':` mirroring `bg-tasks`. +13. **Add `setIsAppsOpen(false)`** to every *other* view's reset block (the + shared multi-`set...(false)` lines) so opening another view clears apps and + closing it doesn't fall back to apps. Use targeted edits per block. +14. **Derived booleans** that OR all view flags (isFullScreenChat, + rightPaneAvailable, inFileView, rightPaneContext, viewOpen, keyboard guards): + add `|| isAppsOpen` / `&& !isAppsOpen` consistently (search `isBgTasksOpen`). +15. **Tab-path mapping** (~L4668): add `: isAppsOpen ? APPS_TAB_PATH`. +16. **Render branch** (~L5894): add `) : isAppsOpen ? (`. + +### 3.3 Sidebar entry (`sidebar-content.tsx`) + +- Add `onOpenApps?: () => void` prop (~L163) and destructure (~L412). +- Add a `SidebarMenuButton` with `isActive={activeNav === 'apps'}` and a + `LayoutGrid` (lucide) icon, label "Mini Apps", in the lower nav group near + "Background agents" (~L830). +- Thread `activeNav === 'apps'` from the parent (App.tsx passes `activeNav` based + on `isAppsOpen`, mirroring how `'agents'` is derived ~L5651). +- Wire `onOpenApps={openAppsView}` at the `SidebarContent`/home call sites + (~L5657, ~L5832). + +### 3.4 The sample app (twitter-client) + +Self-contained HTML string. React + ReactDOM + Babel-standalone + Tailwind, all +via CDN. Renders three sections — **To read / To repost / To respond** (reply +pre-drafted) — from data delivered via `rowboat.onData`. Buttons call +`rowboat.callAction('twitter', ...)`; in Phase 1 the host just toasts/logs and +returns a fake ok. Static `data` lives next to the HTML. + +> Note: CDN scripts require network + `allow-scripts` in the sandbox. The frame +> uses `sandbox="allow-scripts allow-popups allow-forms allow-modals"` — **no** +> `allow-same-origin` (keeps the app from reaching host cookies/storage; the +> bridge is the only channel). Verify CDN loads under this sandbox during testing; +> if blocked, fall back to bundling the libs locally as a renderer asset. + +### 3.5 Verify + +- `cd apps/x && npm run deps && npm run lint` compiles clean. +- `npm run dev`: "Mini Apps" appears in the sidebar; opens the card grid; clicking + the Twitter card renders the app full-screen in the pane; sections populate from + data; clicking a button shows the stubbed action result; tabs/back/forward and + switching to other views behave like every other view. + +## 4. Out of scope for Phase 1 (later phases) + +`~/.rowboat/apps//` storage + IPC; the agent backend (reuse bg-tasks +engine); real Composio execution through the bridge; per-app scope enforcement & +auth prompting; generation via Copilot/Code-Mode; persistent state store; +conversational editing & error recovery; the V2 workspace drag. diff --git a/apps/x/VIDEO_MODE.md b/apps/x/VIDEO_MODE.md new file mode 100644 index 00000000..653521cb --- /dev/null +++ b/apps/x/VIDEO_MODE.md @@ -0,0 +1,253 @@ +# Calls (Video Mode) — Deep Dive + +Calls let the user talk to the assistant hands-free while it *sees* them +(webcam) and their screen (screen share). There is ONE call engine — +continuous listening, auto-submitted utterances, forced read-aloud TTS, frame +capture — entered through four presets that differ only in starting devices. +This doc covers the product flow, the technical pipeline, and the LLM prompt +surface with exact pointers. + +## Product flow + +The composer has a **call split-button** (`chat-input-with-mentions.tsx`). +The main click is the "work together" default — preset `share`: screen +sharing ON, camera OFF, floating pill, so the user keeps working while the +assistant watches along (the button tooltip discloses the screen share). The +chevron menu holds the deviations. While a call is live the button turns red +and ends it. + +| Preset | Starting devices | First surface | +|--------|------------------|---------------| +| `share` — main click | screen on, camera off | floating pill | +| `voice` — "Voice call" | camera off, screen off | floating mascot pill | +| `video` — "Video call" | camera on | full-screen call | +| `practice` — "Practice session" | camera on, + coaching persona | full-screen call | + +**One surface rule** (`callSurface` in `App.tsx`): full screen and screen +sharing are mutually exclusive in both directions — a full-screen call covers +the screen, so sharing it would show the call itself. + +- sharing → floating popout, always (pill = working) +- not sharing → full screen unless `callMinimized` (full screen = facing + each other) +- expanding the pill auto-STOPS any share; minimizing the full-screen call + auto-STARTS one (the pill exists to work together) — presenting from full + screen likewise collapses to the pill +- the camera toggle never changes the surface: turning it on from the pill + puts your video IN the pill; expanding is its own explicit action + +**Screen-share consent** is three-layered: a toast the moment any share +starts ("Your screen is being shared… [Stop sharing]"), a persistent +"Sharing screen" badge on the pill, and macOS's purple recording indicator. +If the auto-share fails (Screen Recording permission not granted) the call +starts anyway as a voice call, with a toast linking to System Settings. +Practice/coaching is always an explicit choice — expanding to full screen +never turns the coach on. + +In-call controls (identical bar on both surfaces): mic mute, camera toggle +(silhouette avatar while off, no webcam frames captured), screen share +toggle, mascot ⇄ "R" letter avatar, end call. **Mute is a full input +pause**, not just audio — mic audio stops reaching Deepgram +(`useVoiceMode.setPaused`, OR'd with the automatic thinking/speaking pause) +AND camera/screen frame capture stops (`useVideoMode.setCapturePaused`; +`collectFrames()` returns nothing while muted, so typed messages carry no +frames either), letting the user talk to someone in the room without the +assistant listening in. Devices stay acquired for instant unmute (camera +light and macOS share indicator stay on — the pill's share badge switches to +"Sharing paused"), the status chip shows "Muted" instead of "Listening", +and assistant output is unaffected (in-flight speech keeps playing; Stop +handles that). Mute resets to off at call start/end. While the assistant is thinking or speaking, a +red **Stop** button appears on the mascot tile — it silences TTS instantly, +skips queued voice segments, and aborts the run if it's still generating +(stopping a run from anywhere, including the composer, also silences TTS). Captions of the in-progress utterance and the +assistant's spoken line run along the bottom. Typing in the composer still +works mid-call; frames ride along with typed messages too. + +Outside calls the composer keeps exactly one voice affordance: the **mic +button** (push-to-talk dictation, untouched). Spoken responses exist only +inside calls (forced full read-aloud, off on hang-up). The old video +dropdown, talking-head toggle, read-aloud headphones toggle, and summary/full +TTS dropdown are all retired — a per-message "read aloud" action on assistant +messages is the planned replacement for text-in/voice-out. + +The call button is disabled unless both voice input (Deepgram) and voice +output (TTS) are configured. `call_started` (with `preset`) is captured in +PostHog — the adoption metric for this feature. + +**Popout mechanics**: a small always-on-top frameless window (camera tile +when on + mascot tile, live caption, control bar) floating over every app — +including Rowboat. Control-bar actions round-trip `video:popoutAction` → +main → `video:popout-action` → app window, which owns the mic/camera/capture; +`expand` also refocuses the app window (handled in main). + +## Frame pipeline + +`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source +(stream → offscreen `