diff --git a/.github/workflows/x-tests.yml b/.github/workflows/x-tests.yml new file mode 100644 index 00000000..f657d478 --- /dev/null +++ b/.github/workflows/x-tests.yml @@ -0,0 +1,40 @@ +name: apps/x tests + +on: + pull_request: + paths: + - "apps/x/**" + - ".github/workflows/x-tests.yml" + push: + branches: [main] + paths: + - "apps/x/**" + - ".github/workflows/x-tests.yml" + workflow_dispatch: + +jobs: + test: + runs-on: ubuntu-latest + defaults: + run: + working-directory: apps/x + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: 10 + + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + cache-dependency-path: apps/x/pnpm-lock.yaml + + - name: Install dependencies + run: pnpm install --frozen-lockfile + + # Builds @x/shared (core/renderer tests import its dist), then runs + # the shared, core, and renderer vitest suites in order. + - name: Run tests + run: npm test diff --git a/.gitignore b/.gitignore index 086ea0b5..c9eee447 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,6 @@ data/ .venv/ .claude/ + +# Local-only meeting-prep planning doc +/PLAN.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..0b40683a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,101 @@ +# AGENTS.md — Working on the Rowboat runtime + +Context for AI coding agents (and humans) working on this repo. General +codebase orientation lives in `CLAUDE.md`; this file covers the **new +turn/session runtime** in `apps/x` — its storage, its debugging tools, and +the invariants you must not break. + +## The runtime in one paragraph + +Chats are **sessions**; each user message starts a **turn**. Both are +append-only JSONL event logs under `~/.rowboat/storage/{turns,sessions}/YYYY/MM/DD/`. +All state is derived by pure reducers (`reduceTurn`, `reduceSession` in +`@x/shared/src/turns.ts` / `sessions.ts`) shared byte-for-byte between the +main process and the renderer. Design specs: +`apps/x/packages/core/docs/turn-runtime-design.md` and `session-design.md` — +read the relevant spec before changing runtime behavior. + +## Storage is reference-based — files store each fact once + +Three applications of the same mechanism: + +1. **Context**: a session turn's `context` is `{ previousTurnId }`; the + conversation prefix is materialized by walking the chain + (`TurnRepoContextResolver`). +2. **Model requests**: `model_call_requested.request.messages` is a list of + string refs into the turn's own events — `"context"`, `"input"`, + `"assistant:"`, `"toolResult:"` — recording only what + is NEW since the previous call. +3. **Agent snapshots**: when a turn's system prompt + tools are + byte-identical to its predecessor's, `turn_created.agent.resolved` is + `{ agentId, model, inheritedFrom }` instead of re-persisting ~70KB. The + model stays concrete (mid-session model switches still inherit). + +The exact provider payload is rebuilt by `composeModelRequest` +(`packages/core/src/turns/compose-model-request.ts`) — **the same code path +the loop transmits through**, so the file plus the composer reproduce the +wire bytes exactly (there is a property test asserting composed == sent). + +## Inspecting turns and sessions + +```bash +cd apps/x/packages/core + +# A whole session: title, per-turn status/size/input preview +npm run inspect -- + +# One turn: per model call, the EXACT provider payload — resolved system +# prompt, tool list, wire-form messages (user-message context woven in, +# tool-result envelopes), and the response/failure +npm run inspect -- [modelCallIndex] [--full] + +# Cascade full turn inspection across a session +npm run inspect -- --turns +``` + +`--full` prints untruncated system prompts and message contents. Turn vs +session ids are auto-detected. This is the intended way to see "what did the +model actually receive" — the raw JSONL deliberately stores structural facts +and references, never the derived wire form. + +## The legacy runs runtime is code-mode-only + +The old runs runtime (`packages/core/src/runs/`, `AgentRuntime`/`streamAgent` +in `packages/core/src/agents/runtime.ts`, the `runs:*` IPC channels, and the +renderer's legacy chat-tab state machine in `App.tsx`) remains in the tree +**solely for code-mode sessions** — a deliberate, documented carve-out: + +- A code session IS a run (`sessionId === runId`). Direct-mode prompts drive + an external ACP agent and hand-assemble `code-run-event`s into the run log + (`code-mode/sessions/service.ts`); rowboat-mode code tabs go through + `runs:createMessage` → the old `AgentRuntime` loop. +- Everything else — main chat, background tasks, live notes, knowledge + pipelines, scheduled agents — runs on turns/sessions. Do NOT add new + callers of `createRun`/`createMessage`/`streamAgent`; headless work uses + `packages/core/src/agents/headless.ts` (`startHeadlessAgent`/ + `runHeadlessAgent`). +- Temporary bridges that die when code-mode is unified: the renderer + transcript loader's `runs:fetch` fallback (`lib/agent-transcript.ts`), the + `runs:downloadLog` fallback in the chat sidebar, and the `notify-user` + gate's `fetchRun` fallback (`application/lib/builtin-tools.ts`). +- The remaining migration (designed but deferred): port code sessions onto + sessions/turns — rowboat-mode prompts become normal turns with + `codeMode`/`codeCwd` composition (already supported by the agent + resolver's composition overrides), direct-mode prompts become a delegated + turn kind carrying opaque code events. That project deletes `runs/` + entirely. + +## Invariants to respect + +- Turn/session files are **append-only**; reducers reject impossible + histories loudly (`TurnCorruptionError`). Never hand-edit files. +- Durable events are persisted **before** side effects (model calls, tool + invocations). Deltas (`text_delta`, …) are stream-only, never persisted. +- The reducers in `@x/shared` must stay pure (no I/O, no node imports) — + the renderer imports them directly. +- Every behavior change needs tests: reducers in `packages/shared`, + runtime/sessions in `packages/core`, renderer stores/views in + `apps/renderer` (all vitest; run `npm test` per package). +- Schema changes: the schema is pre-release (`schemaVersion: 1` throughout); + breaking changes are acceptable but require wiping `~/.rowboat/storage` + and a note in the commit message. diff --git a/CLAUDE.md b/CLAUDE.md index 75a0f6a5..5e762a33 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,7 +109,9 @@ Long-form docs for specific features. Read the relevant file before making chang | Feature | Doc | |---------|-----| | Live Notes — single `live:` frontmatter block (one objective + optional cron / windows / eventMatchCriteria) that turns a note into a self-updating artifact, panel UI, Copilot skill, prompts catalog | `apps/x/LIVE_NOTE.md` | +| Calls (video mode) — one hands-free call engine with four presets (voice / video / share screen / practice coaching), device-derived surfaces (full-screen ⇄ floating popout), frame pipeline, prompts catalog | `apps/x/VIDEO_MODE.md` | | Analytics — PostHog event catalog, person properties, use-case taxonomy, how to add a new event | `apps/x/ANALYTICS.md` | +| Turn/session runtime — event-sourced storage, reference model, the `npm run inspect` debugger | `AGENTS.md` (repo root), `apps/x/packages/core/docs/turn-runtime-design.md`, `session-design.md` | ## Common Tasks diff --git a/apps/x/.gitignore b/apps/x/.gitignore index db195fb4..2c5f63fd 100644 --- a/apps/x/.gitignore +++ b/apps/x/.gitignore @@ -1,2 +1,3 @@ node_modules/ test-fixtures/ +*.tsbuildinfo diff --git a/apps/x/.pnpm-store/v11/index.db b/apps/x/.pnpm-store/v11/index.db new file mode 100644 index 00000000..7f9770bd Binary files /dev/null and b/apps/x/.pnpm-store/v11/index.db differ diff --git a/apps/x/ANALYTICS.md b/apps/x/ANALYTICS.md index 5ddfcf6e..b58d5f25 100644 --- a/apps/x/ANALYTICS.md +++ b/apps/x/ANALYTICS.md @@ -92,6 +92,8 @@ All in `apps/renderer/src/lib/analytics.ts`: - `chat_message_sent` — `{ voice_input, voice_output, search_enabled }` - `oauth_connected` / `oauth_disconnected` — `{ provider }` - `voice_input_started` — no properties +- `call_started` — `{ preset: 'voice' | 'video' | 'share' | 'practice' }` — a hands-free call began (see `apps/x/VIDEO_MODE.md`) +- `call_turn_latency` — `{ endpoint_to_submit_ms, submit_to_speak_ms, speak_to_audio_ms, total_ms }` — voice-to-voice latency breakdown for one call turn (utterance accepted → submitted → first TTS speak → audio playing) - `search_executed` — `{ types: string[] }` - `note_exported` — `{ format }` diff --git a/apps/x/VIDEO_MODE.md b/apps/x/VIDEO_MODE.md new file mode 100644 index 00000000..63b9621e --- /dev/null +++ b/apps/x/VIDEO_MODE.md @@ -0,0 +1,243 @@ +# Calls (Video Mode) — Deep Dive + +Calls let the user talk to the assistant hands-free while it *sees* them +(webcam) and their screen (screen share). There is ONE call engine — +continuous listening, auto-submitted utterances, forced read-aloud TTS, frame +capture — entered through four presets that differ only in starting devices. +This doc covers the product flow, the technical pipeline, and the LLM prompt +surface with exact pointers. + +## Product flow + +The composer has a **call split-button** (`chat-input-with-mentions.tsx`). +The main click is the "work together" default — preset `share`: screen +sharing ON, camera OFF, floating pill, so the user keeps working while the +assistant watches along (the button tooltip discloses the screen share). The +chevron menu holds the deviations. While a call is live the button turns red +and ends it. + +| Preset | Starting devices | First surface | +|--------|------------------|---------------| +| `share` — main click | screen on, camera off | floating pill | +| `voice` — "Voice call" | camera off, screen off | floating mascot pill | +| `video` — "Video call" | camera on | full-screen call | +| `practice` — "Practice session" | camera on, + coaching persona | full-screen call | + +**One surface rule** (`callSurface` in `App.tsx`): full screen and screen +sharing are mutually exclusive in both directions — a full-screen call covers +the screen, so sharing it would show the call itself. + +- sharing → floating popout, always (pill = working) +- not sharing → full screen unless `callMinimized` (full screen = facing + each other) +- expanding the pill auto-STOPS any share; minimizing the full-screen call + auto-STARTS one (the pill exists to work together) — presenting from full + screen likewise collapses to the pill +- the camera toggle never changes the surface: turning it on from the pill + puts your video IN the pill; expanding is its own explicit action + +**Screen-share consent** is three-layered: a toast the moment any share +starts ("Your screen is being shared… [Stop sharing]"), a persistent +"Sharing screen" badge on the pill, and macOS's purple recording indicator. +If the auto-share fails (Screen Recording permission not granted) the call +starts anyway as a voice call, with a toast linking to System Settings. +Practice/coaching is always an explicit choice — expanding to full screen +never turns the coach on. + +In-call controls (identical bar on both surfaces): camera toggle (silhouette +avatar while off, no webcam frames captured), screen share toggle, mascot ⇄ +"R" letter avatar, end call. While the assistant is thinking or speaking, a +red **Stop** button appears on the mascot tile — it silences TTS instantly, +skips queued voice segments, and aborts the run if it's still generating +(stopping a run from anywhere, including the composer, also silences TTS). Captions of the in-progress utterance and the +assistant's spoken line run along the bottom. Typing in the composer still +works mid-call; frames ride along with typed messages too. + +Outside calls the composer keeps exactly one voice affordance: the **mic +button** (push-to-talk dictation, untouched). Spoken responses exist only +inside calls (forced full read-aloud, off on hang-up). The old video +dropdown, talking-head toggle, read-aloud headphones toggle, and summary/full +TTS dropdown are all retired — a per-message "read aloud" action on assistant +messages is the planned replacement for text-in/voice-out. + +The call button is disabled unless both voice input (Deepgram) and voice +output (TTS) are configured. `call_started` (with `preset`) is captured in +PostHog — the adoption metric for this feature. + +**Popout mechanics**: a small always-on-top frameless window (camera tile +when on + mascot tile, live caption, control bar) floating over every app — +including Rowboat. Control-bar actions round-trip `video:popoutAction` → +main → `video:popout-action` → app window, which owns the mic/camera/capture; +`expand` also refocuses the app window (handled in main). + +## Frame pipeline + +`apps/renderer/src/hooks/useVideoMode.ts` runs one capture pipe per source +(stream → offscreen `