diff --git a/.gitignore b/.gitignore index 2480e5e1..086ea0b5 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ .vscode/ data/ .venv/ +.claude/ diff --git a/CLAUDE.md b/CLAUDE.md index db51cb63..51a11e35 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -102,6 +102,14 @@ pnpm uses symlinks for workspace packages. Electron Forge's dependency walker ca | Workspace config | `apps/x/pnpm-workspace.yaml` | | Root scripts | `apps/x/package.json` | +## Feature Deep-Dives + +Long-form docs for specific features. Read the relevant file before making changes in that area — it has the full product flow, technical flows, and (where applicable) a catalog of the LLM prompts involved with exact file:line pointers. + +| Feature | Doc | +|---------|-----| +| Track Blocks — auto-updating note content (scheduled / event-driven / manual), Copilot skill, prompts catalog | `apps/x/TRACKS.md` | + ## Common Tasks ### LLM configuration (single provider) diff --git a/apps/x/.claude/launch.json b/apps/x/.claude/launch.json deleted file mode 100644 index 3ba43066..00000000 --- a/apps/x/.claude/launch.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "version": "0.0.1", - "configurations": [ - { - "name": "renderer-dev", - "runtimeExecutable": "/Users/tusharmagar/Rowboat/rowboat-V2/apps/x/apps/renderer/node_modules/.bin/vite", - "runtimeArgs": ["--port", "5173"], - "port": 5173 - } - ] -} diff --git a/apps/x/TRACKS.md b/apps/x/TRACKS.md new file mode 100644 index 00000000..3caf9e41 --- /dev/null +++ b/apps/x/TRACKS.md @@ -0,0 +1,343 @@ +# Track Blocks + +> Living blocks of content embedded in markdown notes that auto-refresh on a schedule, when relevant events arrive, or on demand. + +A track block is a small YAML code fence plus a sibling HTML-comment region in a note. The YAML defines what to produce and when; the comment region holds the generated output, rewritten on each run. A single note can hold many independent tracks — one for weather, one for an email digest, one for a running project summary. + +**Example** (a Chicago-time track refreshed hourly): + +~~~markdown +```track +trackId: chicago-time +instruction: Show the current time in Chicago, IL in 12-hour format. +active: true +schedule: + type: cron + expression: "0 * * * *" +``` + + +2:30 PM, Central Time + +~~~ + +## Table of Contents + +1. [Product Overview](#product-overview) +2. [Architecture at a Glance](#architecture-at-a-glance) +3. [Technical Flows](#technical-flows) +4. [Schema Reference](#schema-reference) +5. [Prompts Catalog](#prompts-catalog) +6. [File Map](#file-map) +7. [Known Follow-ups](#known-follow-ups) + +--- + +## Product Overview + +### Trigger types + +A track block has exactly one of four trigger configurations. Schedule and event triggers can co-exist on a single track. + +| Trigger | When it fires | How to express it | +|---|---|---| +| **Manual** | Only when the user (or Copilot) hits Run | Omit `schedule`, leave `eventMatchCriteria` unset | +| **Scheduled — cron** | At exact cron times | `schedule: { type: cron, expression: "0 * * * *" }` | +| **Scheduled — window** | At most once per cron occurrence, only within a time-of-day window | `schedule: { type: window, cron, startTime: "09:00", endTime: "17:00" }` | +| **Scheduled — once** | Once at a future time, then never | `schedule: { type: once, runAt: "2026-04-14T09:00:00" }` | +| **Event-driven** | When a matching event arrives (e.g. new Gmail thread) | `eventMatchCriteria: "Emails about Q3 planning"` | + +Decision shorthand: cron for exact times, window for "sometime in the morning", once for one-shots, event for signal-driven updates. Combine `schedule` + `eventMatchCriteria` for a track that refreshes on a cadence **and** reacts to incoming signals. + +### Creating a track + +Three paths, all produce identical on-disk YAML: + +1. **Hand-written** — type the YAML fence directly in a note. The renderer picks it up instantly via the Tiptap `track-block` extension. +2. **Cmd+K with cursor context** — press Cmd+K anywhere in a note, say "add a track that shows Chicago weather hourly". Copilot loads the `tracks` skill and splices the block at the cursor using `workspace-edit`. +3. **Sidebar chat** — mention a note (or have it attached) and ask for a track. Copilot appends the block at the end or under the heading you name. + +### Viewing and managing a track + +The editor shows track blocks as an inline **chip** — a pill with the track ID, a clipped instruction, a color rail matching the trigger type, and a spinner while running. + +Clicking the chip opens the **track modal**, where everything happens: + +- **Header** — track ID, schedule summary (e.g. "Hourly"), and a Switch to pause/resume (flips `active`). +- **Tabs** — *What to track* (renders `instruction` as markdown), *When to run* (schedule details, shown if `schedule` is present), *Event matching* (shown if `eventMatchCriteria` is present), *Details* (ID, file, run metadata). +- **Advanced** — expandable raw-YAML editor for power users. +- **Danger zone** (Details tab) — delete with two-step confirmation; also removes the target region. +- **Footer** — *Edit with Copilot* (opens sidebar chat with the note attached) and *Run now* (triggers immediately). + +Every mutation inside the modal goes through IPC to the backend — the renderer never writes the file itself. This is the fix that prevents the editor's save cycle from clobbering backend-written fields like `lastRunAt`. + +### What Copilot can do + +- **Create, edit, and delete** track blocks (via the `tracks` skill + `workspace-edit` / `workspace-readFile`). +- **Run** a track with the `run-track-block` builtin tool. An optional `context` parameter biases this single run — e.g. `"Backfill from gmail_sync/ for the last 90 days about this topic"`. Common use: seed a newly-created event-driven track so its target region isn't empty waiting for the next matching event. +- **Proactively suggest** tracks when the user signals interest ("track", "monitor", "watch", "every morning"), per the trigger paragraph in `instructions.ts`. +- **Re-enter edit mode** via the modal's *Edit with Copilot* button, which seeds a new chat with the note attached and invites Copilot to load the `tracks` skill. + +### After a run + +- The **target region** (between `` markers) is rewritten by the track-run agent using the `update-track-content` tool. +- `lastRunAt`, `lastRunId`, `lastRunSummary` are updated in the YAML. +- The chip pulses while running, then displays the latest `lastRunAt`. +- Bus events (`track_run_start` / `track_run_complete`) are forwarded to the renderer via the `tracks:events` IPC channel, consumed by the `useTrackStatus` hook. + +--- + +## Architecture at a Glance + +``` +Editor chip (display-only) ──click──► TrackModal (React) + │ + ├──► IPC: track:get / update / + │ replaceYaml / delete / run + │ +Backend (main process) + ├─ Scheduler loop (15 s) ──┐ + ├─ Event processor (5 s) ──┼──► triggerTrackUpdate() ──► track-run agent + └─ Copilot tool run-track-block ──┘ │ + ▼ + update-track-content tool + │ + ▼ + target region rewritten on disk +``` + +**Single-writer invariant:** the renderer is never a file writer. All on-disk changes go through backend helpers in `packages/core/src/knowledge/track/fileops.ts` (`updateTrackBlock`, `replaceTrackBlockYaml`, `deleteTrackBlock`). This avoids races with the scheduler/runner writing runtime fields. + +**Event contract:** `window.dispatchEvent(CustomEvent('rowboat:open-track-modal', { detail }))` is the sole entry point from editor chip → modal. Similarly, `rowboat:open-copilot-edit-track` opens the sidebar chat with context. + +--- + +## Technical Flows + +### 4.1 Scheduling (cron / window / once) + +- Module: `packages/core/src/knowledge/track/scheduler.ts`. Polls every **15 seconds** (`POLL_INTERVAL_MS`). +- Each tick: `workspace.readdir('knowledge', { recursive: true })`, filter `.md`, iterate all track blocks via `fetchAll(relPath)`. +- Per-track due check: `isTrackScheduleDue(schedule, lastRunAt)` in `schedule-utils.ts`. All three schedule types enforce a **2-minute grace window** — missed schedules (app offline at trigger time) are skipped, not replayed. +- When due, fires `triggerTrackUpdate(trackId, filePath, undefined, 'timed')` (fire-and-forget; the runner's concurrency guard prevents duplicates). +- Startup: `initTrackScheduler()` is called in `apps/main/src/main.ts` at app-ready, alongside `initTrackEventProcessor()`. + +### 4.2 Event pipeline + +**Producers** — any data source that should feed tracks emits events: + +- **Gmail** (`packages/core/src/knowledge/sync_gmail.ts`) — three call sites, each after a successful thread sync, call `createEvent({ source: 'gmail', type: 'email.synced', payload: })`. +- **Calendar** (`packages/core/src/knowledge/sync_calendar.ts`) — one bundled event per sync. The payload is a markdown digest of all new/updated/deleted events built by `summarizeCalendarSync()` (line 68). `publishCalendarSyncEvent()` (line ~126) wraps it with `source: 'calendar'`, `type: 'calendar.synced'`. + +**Storage** — `packages/core/src/knowledge/track/events.ts` writes each event as a JSON file under `~/.rowboat/events/pending/.json`. IDs come from the DI-resolved `IdGen` (ISO-based, lexicographically sortable) — so `readdirSync(...).sort()` is strict FIFO. + +**Consumer loop** — same file, `init()` polls every **5 seconds**, then `processPendingEvents()` walks sorted filenames. For each event: + +1. Parse via `KnowledgeEventSchema`; malformed files go to `done/` with `error` set (the loop stays alive). +2. `listAllTracks()` scans every `.md` under `knowledge/` and collects `ParsedTrack[]`. +3. `findCandidates(event, allTracks)` in `routing.ts` runs Pass 1 LLM routing (below). +4. For each candidate, `triggerTrackUpdate(trackId, filePath, event.payload, 'event')` **sequentially** — preserves total ordering within the event. +5. Enrich the event JSON with `processedAt`, `candidates`, `runIds`, `error?`, then `moveEventToDone()` — write to `events/done/.json`, unlink from `pending/`. + +**Pass 1 routing** (`routing.ts:73+ findCandidates`): + +- **Short-circuit**: if `event.targetTrackId` + `targetFilePath` are set (manual re-run events), skip the LLM and return that track directly. +- Filter to `active && instruction && eventMatchCriteria` tracks. +- Batches of `BATCH_SIZE = 20`. +- Per batch, `generateObject()` with `ROUTING_SYSTEM_PROMPT` + `buildRoutingPrompt()` and `Pass1OutputSchema` → `candidates: { trackId, filePath }[]`. The composite key `trackId::filePath` deduplicates across files since `trackId` is only unique per file. +- Model resolution: gateway if signed in, local provider otherwise; prefers `knowledgeGraphModel` from config. + +**Pass 2 decision** happens inside the track-run agent (see Run flow below) — the liberal Pass 1 produces candidates, the agent vetoes false positives before touching the target region. + +### 4.3 Run flow (`triggerTrackUpdate`) + +Module: `packages/core/src/knowledge/track/runner.ts`. + +1. **Concurrency guard** — static `runningTracks: Set` keyed by `${trackId}:${filePath}`. Duplicate calls return `{ action: 'no_update', error: 'Already running' }`. +2. **Fetch block** via `fetchAll(filePath)`, locate by `trackId`. +3. **Create agent run** — `createRun({ agentId: 'track-run' })`. +4. **Set `lastRunAt` + `lastRunId` immediately** (before the agent executes). Prevents retry storms: if the run fails, the scheduler's next tick won't re-trigger, and for `once` tracks the "done" flag is already set. +5. **Emit `track_run_start`** on the `trackBus` with the trigger type (`manual` / `timed` / `event`). +6. **Send agent message** built by `buildMessage(filePath, track, trigger, context?)` — three branches (see Prompts Catalog). For `'event'` trigger, the message includes the event payload and a Pass 2 decision directive. +7. **Wait for completion** — `waitForRunCompletion(runId)`, then `extractAgentResponse(runId)` for the summary. +8. **Compare content**: re-read the file, diff `contentBefore` vs `contentAfter`. If changed → `action: 'replace'`; else → `action: 'no_update'`. +9. **Store `lastRunSummary`** via `updateTrackBlock`. +10. **Emit `track_run_complete`** with `summary` or `error`. +11. **Cleanup**: `runningTracks.delete(key)` in a `finally` block. + +Returned to callers: `{ trackId, runId, action, contentBefore, contentAfter, summary, error? }`. + +### 4.4 IPC surface + +| Channel | Caller → handler | Purpose | +|---|---|---| +| `track:run` | Renderer (chip/modal Run button) | Fires `triggerTrackUpdate(..., 'manual')` | +| `track:get` | Modal on open | Returns fresh YAML from disk via `fetchYaml` | +| `track:update` | Modal toggle / partial edits | `updateTrackBlock` merges a partial into on-disk YAML | +| `track:replaceYaml` | Advanced raw-YAML save | `replaceTrackBlockYaml` validates + writes full YAML | +| `track:delete` | Danger-zone confirm | `deleteTrackBlock` removes YAML fence + target region | +| `tracks:events` | Server → renderer (`webContents.send`) | Forwards `trackBus` events to the `useTrackStatus` hook | + +Request/response schemas live in `packages/shared/src/ipc.ts`; handlers in `apps/main/src/ipc.ts`; backend helpers in `packages/core/src/knowledge/track/fileops.ts`. + +### 4.5 Renderer integration + +- **Chip** — `apps/renderer/src/extensions/track-block.tsx`. Tiptap atom node. Render-only: click dispatches `CustomEvent('rowboat:open-track-modal', { detail: { trackId, filePath, initialYaml, onDeleted } })`. The `onDeleted` callback is the Tiptap `deleteNode` — the modal calls it after a successful `track:delete` IPC so the editor also drops the node and doesn't resurrect it on next save. +- **Modal** — `apps/renderer/src/components/track-modal.tsx`. Mounted once in `App.tsx`, self-listens for the open event. On open: calls `track:get` to get the authoritative YAML, not the Tiptap cached attr. All mutations go through IPC; `updateAttributes` is never called. +- **Status hook** — `apps/renderer/src/hooks/use-track-status.ts`. Subscribes to `tracks:events` and maintains a `Map<"${trackId}:${filePath}", RunState>` so chips and the modal reflect live run state. +- **Edit with Copilot** — the modal's footer button dispatches `rowboat:open-copilot-edit-track`. An `useEffect` listener in `App.tsx` opens the chat sidebar via `submitFromPalette(text, mention)`, where `text` is a short opener that invites Copilot to load the `tracks` skill and `mention` attaches the note file. + +### 4.6 Copilot skill integration + +- **Skill content** — `packages/core/src/application/assistant/skills/tracks/skill.ts` (~318 lines). Exported as a `String.raw` template literal; injected into the Copilot system prompt when the `loadSkill('tracks')` tool is called. +- **Canonical schema** inside the skill is **auto-generated at module load**: `stringifyYaml(z.toJSONSchema(TrackBlockSchema))`. Editing `TrackBlockSchema` in `packages/shared/src/track-block.ts` propagates to the skill without manual sync. +- **Skill registration** — `packages/core/src/application/assistant/skills/index.ts` (import + entry in the `definitions` array). +- **Loading trigger** — `packages/core/src/application/assistant/instructions.ts:73` — one paragraph tells Copilot to load the skill on track / monitor / watch / "keep an eye on" / Cmd+K auto-refresh requests. +- **Builtin tools** — `packages/core/src/application/lib/builtin-tools.ts`: + - `update-track-content` — low-level: rewrite the target region between `` markers. Used mainly by the track-run agent. + - `run-track-block` — high-level: trigger a run with an optional `context`. Uses `await import("../../knowledge/track/runner.js")` inside the execute body to break a module-init cycle (`builtin-tools → track/runner → runs/runs → agents/runtime → builtin-tools`). + +### 4.7 Concurrency & FIFO guarantees + +- **Per-track serialization** — the `runningTracks` guard in `runner.ts`. A track is at most running once; overlapping triggers (manual + scheduled + event) return `error: 'Already running'` cleanly through IPC. +- **Backend is single writer** — renderer uses IPC for every edit; backend helpers in `fileops.ts` are the only code paths that touch the file. +- **Event FIFO** — monotonic `IdGen` IDs → lexicographic filenames → `sort()` in `processPendingEvents()` = strict FIFO across events. Candidates within one event are processed sequentially (`await` in loop) so ordering holds within an event too. +- **No retry storms** — `lastRunAt` is set at the *start* of a run, not the end. A crash mid-run leaves the track marked as ran; the scheduler's next tick computes the next occurrence from that point. + +--- + +## Schema Reference + +All canonical schemas live in `packages/shared/src/track-block.ts`: + +- `TrackBlockSchema` — the YAML that goes inside a ` ```track ` fence. User-authored fields: `trackId` (kebab-case, unique within the file), `instruction`, `active`, `schedule?`, `eventMatchCriteria?`. **Runtime-managed fields (never hand-write):** `lastRunAt`, `lastRunId`, `lastRunSummary`. +- `TrackScheduleSchema` — discriminated union over `{ type: 'cron' | 'window' | 'once' }`. +- `KnowledgeEventSchema` — the on-disk shape of each event JSON in `events/pending/` and `events/done/`. Enrichment fields (`processedAt`, `candidates`, `runIds`, `error`) are populated when moving to `done/`. +- `Pass1OutputSchema` — the structured response the routing classifier returns: `{ candidates: { trackId, filePath }[] }`. + +Since the skill's schema block is generated from `TrackBlockSchema`, schema changes should start here — the skill, the validator in `replaceTrackBlockYaml`, and modal display logic all follow from the Zod source of truth. + +--- + +## Prompts Catalog + +Every LLM-facing prompt in the feature, with file + line pointers so you can edit in place. After any edit: `cd apps/x && npm run deps` to rebuild the affected package, then restart the app (`npm run dev`). + +### 1. Routing system prompt (Pass 1 classifier) + +- **Purpose**: decide which tracks *might* be relevant to an incoming event. Liberal — prefers false positives; Pass 2 inside the agent catches them. +- **File**: `packages/core/src/knowledge/track/routing.ts:22–37` (`ROUTING_SYSTEM_PROMPT`). +- **Inputs**: none interpolated — constant system prompt. +- **Output**: structured `Pass1OutputSchema` — `{ candidates: { trackId, filePath }[] }`. +- **Invoked by**: `findCandidates()` at `routing.ts:73+`, per batch of 20 tracks, via `generateObject({ model, system, prompt, schema })`. + +### 2. Routing user prompt template + +- **Purpose**: formats the event and the current batch of tracks into the user message fed alongside the system prompt. +- **File**: `packages/core/src/knowledge/track/routing.ts:51–66` (`buildRoutingPrompt`). +- **Inputs**: `event` (source, type, createdAt, payload), `batch: ParsedTrack[]` (each: `trackId`, `filePath`, `eventMatchCriteria`). +- **Output**: plain text, two sections — `## Event` and `## Track Blocks`. +- **Note**: the event `payload` is what the producer wrote. For Gmail it's a thread markdown dump; for Calendar it's a digest (see #7 below). + +### 3. Track-run agent instructions + +- **Purpose**: system prompt for the background agent that actually rewrites target regions. Sets tone ("no user present — don't ask questions"), points at the knowledge graph, and prescribes the `update-track-content` tool as the write path. +- **File**: `packages/core/src/knowledge/track/run-agent.ts:6–50` (`TRACK_RUN_INSTRUCTIONS`). +- **Inputs**: `${WorkDir}` template literal (substituted at module load). +- **Output**: free-form — agent calls tools, ends with a 1-2 sentence summary used as `lastRunSummary`. +- **Invoked by**: `buildTrackRunAgent()` at line 52, called during agent runtime setup. Tool set = all `BuiltinTools` except `executeCommand`. + +### 4. Track-run agent message (`buildMessage`) + +- **Purpose**: the user message seeded into each track-run. Three shape variants based on `trigger`. +- **File**: `packages/core/src/knowledge/track/runner.ts:23–62`. +- **Inputs**: `filePath`, `track.trackId`, `track.instruction`, `track.content`, `track.eventMatchCriteria`, `trigger`, optional `context`, plus `localNow` / `tz`. +- **Output**: free-form — the agent decides whether to call `update-track-content`. + +Three branches: + +- **`manual`** — base message (instruction + current content + tool hint). If `context` is passed, it's appended as a `**Context:**` section. The `run-track-block` tool uses this path for both plain refreshes and context-biased backfills. +- **`timed`** — same as `manual`. Called by the scheduler with no `context`. +- **`event`** — adds a **Pass 2 decision block** (lines 45–56). Quoted verbatim: + + > **Trigger:** Event match (a Pass 1 routing classifier flagged this track as potentially relevant to the event below) + > + > **Event match criteria for this track:** … + > + > **Event payload:** … + > + > **Decision:** Determine whether this event genuinely warrants updating the track content. If the event is not meaningfully relevant on closer inspection, skip the update — do NOT call `update-track-content`. Only call the tool if the event provides new or changed information that should be reflected in the track. + +### 5. Tracks skill (Copilot-facing) + +- **Purpose**: teaches Copilot everything about track blocks — anatomy, schema, schedules, event matching, insertion workflow, when to proactively suggest, when to run with context. +- **File**: `packages/core/src/application/assistant/skills/tracks/skill.ts` (~318 lines). Exported `skill` constant. +- **Inputs**: at module load, `stringifyYaml(z.toJSONSchema(TrackBlockSchema))` is interpolated into the "Canonical Schema" section. Rebuilds propagate schema edits automatically. +- **Output**: markdown, injected into the Copilot system prompt when `loadSkill('tracks')` fires. +- **Invoked by**: Copilot's `loadSkill` builtin tool (see `packages/core/src/application/lib/builtin-tools.ts`). Registration in `skills/index.ts`. +- **Edit guidance**: for schema-shaped changes, start at `packages/shared/src/track-block.ts` — the skill picks it up on the next build. For prose/workflow tweaks, edit the `String.raw` template. + +### 6. Copilot trigger paragraph + +- **Purpose**: tells Copilot *when* to load the `tracks` skill. +- **File**: `packages/core/src/application/assistant/instructions.ts:73`. +- **Inputs**: none; static prose. +- **Output**: part of the baseline Copilot system prompt. +- **Trigger words**: *track*, *monitor*, *watch*, *keep an eye on*, "*every morning tell me X*", "*show the current Y in this note*", "*pin live updates of Z here*", plus any Cmd+K request that implies auto-refresh. + +### 7. `run-track-block` tool — `context` parameter description + +- **Purpose**: a mini-prompt (a Zod `.describe()`) that guides Copilot on when to pass extra context for a run. Copilot sees this text as part of the tool's schema. +- **File**: `packages/core/src/application/lib/builtin-tools.ts:1454+` (the `run-track-block` tool definition; the `context` field's `.describe()` block is the mini-prompt). +- **Inputs**: free-form string from Copilot. +- **Output**: flows into `triggerTrackUpdate(..., 'manual')` → `buildMessage` → appended as `**Context:**` in the agent message. +- **Key use case**: backfill a newly-created event-driven track so its target region isn't empty on day 1 — e.g. `"Backfill from gmail_sync/ for the last 90 days about this topic"`. + +### 8. Calendar sync digest (event payload template) + +- **Purpose**: shapes what the routing classifier sees for a calendar event. Not a system prompt, but a deterministic markdown template that becomes `event.payload`. +- **File**: `packages/core/src/knowledge/sync_calendar.ts:68+` (`summarizeCalendarSync`). Wrapped by `publishCalendarSyncEvent()` at line ~126. +- **Inputs**: `newEvents`, `updatedEvents`, `deletedEventIds` gathered during a sync. +- **Output**: markdown with counts header, a `## Changed events` section (per-event block: title, ID, time, organizer, location, attendees, truncated description), and a `## Deleted event IDs` section. Capped at ~50 events with a "…and N more" line; descriptions truncated to 500 chars. +- **Why care**: the quality of Pass 1 matching depends on how clear this payload is. If the classifier misses obvious matches, this is a place to look. + +--- + +## File Map + +| Purpose | File | +|---|---| +| Zod schemas (tracks, schedules, events, Pass1) | `packages/shared/src/track-block.ts` | +| IPC channel schemas | `packages/shared/src/ipc.ts` | +| IPC handlers (main process) | `apps/main/src/ipc.ts` | +| File operations (fetch / update / replace / delete) | `packages/core/src/knowledge/track/fileops.ts` | +| Scheduler (cron / window / once) | `packages/core/src/knowledge/track/scheduler.ts` | +| Schedule due-check helper | `packages/core/src/knowledge/track/schedule-utils.ts` | +| Event producer + consumer loop | `packages/core/src/knowledge/track/events.ts` | +| Pass 1 routing (LLM classifier) | `packages/core/src/knowledge/track/routing.ts` | +| Run orchestrator (`triggerTrackUpdate`, `buildMessage`) | `packages/core/src/knowledge/track/runner.ts` | +| Track-run agent definition | `packages/core/src/knowledge/track/run-agent.ts` | +| Track bus (pub-sub for lifecycle events) | `packages/core/src/knowledge/track/bus.ts` | +| Track state type | `packages/core/src/knowledge/track/types.ts` | +| Gmail event producer | `packages/core/src/knowledge/sync_gmail.ts` | +| Calendar event producer + digest | `packages/core/src/knowledge/sync_calendar.ts` | +| Copilot skill | `packages/core/src/application/assistant/skills/tracks/skill.ts` | +| Skill registration | `packages/core/src/application/assistant/skills/index.ts` | +| Copilot trigger paragraph | `packages/core/src/application/assistant/instructions.ts` | +| Builtin tools (`update-track-content`, `run-track-block`) | `packages/core/src/application/lib/builtin-tools.ts` | +| Tiptap chip (editor node view) | `apps/renderer/src/extensions/track-block.tsx` | +| React modal (all UI mutations) | `apps/renderer/src/components/track-modal.tsx` | +| Status hook (`useTrackStatus`) | `apps/renderer/src/hooks/use-track-status.ts` | +| App-level listeners (modal + Copilot edit) | `apps/renderer/src/App.tsx` | +| CSS | `apps/renderer/src/styles/editor.css`, `apps/renderer/src/styles/track-modal.css` | +| Main process startup (schedulers & processors) | `apps/main/src/main.ts` | + +--- + +## Known Follow-ups + +- **Tiptap save can still re-serialize a stale node attr.** The chip+modal refactor makes every *intentional* mutation go through IPC, so toggles, raw-YAML edits, and deletes are all safe. But if the backend writes runtime fields (`lastRunAt`, `lastRunSummary`) after the note was loaded, and the user then types anywhere in the note body, Tiptap's markdown serializer writes out the cached (stale) YAML for each track block, clobbering those fields. + - **Cleanest fix**: subscribe to `trackBus` events in the renderer and refresh the corresponding node attr via `updateAttributes` before Tiptap can save. + - **Alternative**: make the markdown serializer pull fresh YAML from disk per track block at write time (async-friendly refactor). + +- **Only Gmail + Calendar produce events today.** Granola, Fireflies, and other sync services are plumbed for the pattern but don't emit yet. Adding a producer is a one-liner `createEvent({ source, type, createdAt, payload })` at the right spot in the sync flow. diff --git a/apps/x/apps/main/src/browser/control-service.ts b/apps/x/apps/main/src/browser/control-service.ts new file mode 100644 index 00000000..b83ea7cb --- /dev/null +++ b/apps/x/apps/main/src/browser/control-service.ts @@ -0,0 +1,243 @@ +import type { IBrowserControlService } from '@x/core/dist/application/browser-control/service.js'; +import type { BrowserControlAction, BrowserControlInput, BrowserControlResult } from '@x/shared/dist/browser-control.js'; +import { browserViewManager } from './view.js'; +import { normalizeNavigationTarget } from './navigation.js'; + +function buildSuccessResult( + action: BrowserControlAction, + message: string, + page?: BrowserControlResult['page'], +): BrowserControlResult { + return { + success: true, + action, + message, + browser: browserViewManager.getState(), + ...(page ? { page } : {}), + }; +} + +function buildErrorResult(action: BrowserControlAction, error: string): BrowserControlResult { + return { + success: false, + action, + error, + browser: browserViewManager.getState(), + }; +} + +export class ElectronBrowserControlService implements IBrowserControlService { + async execute( + input: BrowserControlInput, + ctx?: { signal?: AbortSignal }, + ): Promise { + const signal = ctx?.signal; + + try { + switch (input.action) { + case 'open': { + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('open', 'Opened a browser session.', page); + } + + case 'get-state': + return buildSuccessResult('get-state', 'Read the current browser state.'); + + case 'new-tab': { + const target = input.target ? normalizeNavigationTarget(input.target) : undefined; + const result = await browserViewManager.newTab(target); + if (!result.ok) { + return buildErrorResult('new-tab', result.error ?? 'Failed to open a new tab.'); + } + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult( + 'new-tab', + target ? `Opened a new tab for ${target}.` : 'Opened a new tab.', + page, + ); + } + + case 'switch-tab': { + const tabId = input.tabId; + if (!tabId) { + return buildErrorResult('switch-tab', 'tabId is required for switch-tab.'); + } + const result = browserViewManager.switchTab(tabId); + if (!result.ok) { + return buildErrorResult('switch-tab', `No browser tab exists with id ${tabId}.`); + } + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('switch-tab', `Switched to tab ${tabId}.`, page); + } + + case 'close-tab': { + const tabId = input.tabId; + if (!tabId) { + return buildErrorResult('close-tab', 'tabId is required for close-tab.'); + } + const result = browserViewManager.closeTab(tabId); + if (!result.ok) { + return buildErrorResult('close-tab', `Could not close tab ${tabId}.`); + } + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('close-tab', `Closed tab ${tabId}.`, page); + } + + case 'navigate': { + const rawTarget = input.target; + if (!rawTarget) { + return buildErrorResult('navigate', 'target is required for navigate.'); + } + const target = normalizeNavigationTarget(rawTarget); + const result = await browserViewManager.navigate(target); + if (!result.ok) { + return buildErrorResult('navigate', result.error ?? `Failed to navigate to ${target}.`); + } + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('navigate', `Navigated to ${target}.`, page); + } + + case 'back': { + const result = browserViewManager.back(); + if (!result.ok) { + return buildErrorResult('back', 'The active tab cannot go back.'); + } + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('back', 'Went back in the active tab.', page); + } + + case 'forward': { + const result = browserViewManager.forward(); + if (!result.ok) { + return buildErrorResult('forward', 'The active tab cannot go forward.'); + } + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('forward', 'Went forward in the active tab.', page); + } + + case 'reload': { + browserViewManager.reload(); + await browserViewManager.ensureActiveTabReady(signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('reload', 'Reloaded the active tab.', page); + } + + case 'read-page': { + const result = await browserViewManager.readPage( + { + maxElements: input.maxElements, + maxTextLength: input.maxTextLength, + }, + signal, + ); + if (!result.ok || !result.page) { + return buildErrorResult('read-page', result.error ?? 'Failed to read the current page.'); + } + return buildSuccessResult('read-page', 'Read the current page.', result.page); + } + + case 'click': { + const result = await browserViewManager.click( + { + index: input.index, + selector: input.selector, + snapshotId: input.snapshotId, + }, + signal, + ); + if (!result.ok) { + return buildErrorResult('click', result.error ?? 'Failed to click the requested element.'); + } + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult( + 'click', + result.description ? `Clicked ${result.description}.` : 'Clicked the requested element.', + page, + ); + } + + case 'type': { + const text = input.text; + if (text === undefined) { + return buildErrorResult('type', 'text is required for type.'); + } + const result = await browserViewManager.type( + { + index: input.index, + selector: input.selector, + snapshotId: input.snapshotId, + }, + text, + signal, + ); + if (!result.ok) { + return buildErrorResult('type', result.error ?? 'Failed to type into the requested element.'); + } + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult( + 'type', + result.description ? `Typed into ${result.description}.` : 'Typed into the requested element.', + page, + ); + } + + case 'press': { + const key = input.key; + if (!key) { + return buildErrorResult('press', 'key is required for press.'); + } + const result = await browserViewManager.press( + key, + { + index: input.index, + selector: input.selector, + snapshotId: input.snapshotId, + }, + signal, + ); + if (!result.ok) { + return buildErrorResult('press', result.error ?? `Failed to press ${key}.`); + } + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult( + 'press', + result.description ? `Pressed ${result.description}.` : `Pressed ${key}.`, + page, + ); + } + + case 'scroll': { + const result = await browserViewManager.scroll( + input.direction ?? 'down', + input.amount ?? 700, + signal, + ); + if (!result.ok) { + return buildErrorResult('scroll', result.error ?? 'Failed to scroll the page.'); + } + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('scroll', `Scrolled ${input.direction ?? 'down'}.`, page); + } + + case 'wait': { + const duration = input.ms ?? 1000; + await browserViewManager.wait(duration, signal); + const page = await browserViewManager.readPageSummary(signal, { waitForReady: false }) ?? undefined; + return buildSuccessResult('wait', `Waited ${duration}ms for the page to settle.`, page); + } + } + } catch (error) { + return buildErrorResult( + input.action, + error instanceof Error ? error.message : 'Browser control failed unexpectedly.', + ); + } + } +} diff --git a/apps/x/apps/main/src/browser/ipc.ts b/apps/x/apps/main/src/browser/ipc.ts new file mode 100644 index 00000000..fa3b1ac1 --- /dev/null +++ b/apps/x/apps/main/src/browser/ipc.ts @@ -0,0 +1,81 @@ +import { BrowserWindow } from 'electron'; +import { ipc } from '@x/shared'; +import { browserViewManager, type BrowserState } from './view.js'; + +type IPCChannels = ipc.IPCChannels; + +type InvokeHandler = ( + event: Electron.IpcMainInvokeEvent, + args: IPCChannels[K]['req'], +) => IPCChannels[K]['res'] | Promise; + +type BrowserHandlers = { + 'browser:setBounds': InvokeHandler<'browser:setBounds'>; + 'browser:setVisible': InvokeHandler<'browser:setVisible'>; + 'browser:newTab': InvokeHandler<'browser:newTab'>; + 'browser:switchTab': InvokeHandler<'browser:switchTab'>; + 'browser:closeTab': InvokeHandler<'browser:closeTab'>; + 'browser:navigate': InvokeHandler<'browser:navigate'>; + 'browser:back': InvokeHandler<'browser:back'>; + 'browser:forward': InvokeHandler<'browser:forward'>; + 'browser:reload': InvokeHandler<'browser:reload'>; + 'browser:getState': InvokeHandler<'browser:getState'>; +}; + +/** + * Browser-specific IPC handlers, exported as a plain object so they can be + * spread into the main `registerIpcHandlers({...})` call in ipc.ts. This + * mirrors the convention of keeping feature handlers flat and namespaced by + * channel prefix (`browser:*`). + */ +export const browserIpcHandlers: BrowserHandlers = { + 'browser:setBounds': async (_event, args) => { + browserViewManager.setBounds(args); + return { ok: true }; + }, + 'browser:setVisible': async (_event, args) => { + browserViewManager.setVisible(args.visible); + return { ok: true }; + }, + 'browser:newTab': async (_event, args) => { + return browserViewManager.newTab(args.url); + }, + 'browser:switchTab': async (_event, args) => { + return browserViewManager.switchTab(args.tabId); + }, + 'browser:closeTab': async (_event, args) => { + return browserViewManager.closeTab(args.tabId); + }, + 'browser:navigate': async (_event, args) => { + return browserViewManager.navigate(args.url); + }, + 'browser:back': async () => { + return browserViewManager.back(); + }, + 'browser:forward': async () => { + return browserViewManager.forward(); + }, + 'browser:reload': async () => { + browserViewManager.reload(); + return { ok: true }; + }, + 'browser:getState': async () => { + return browserViewManager.getState(); + }, +}; + +/** + * Wire the BrowserViewManager's state-updated event to all renderer windows + * as a `browser:didUpdateState` push. Must be called once after the main + * window is created so the manager has a window to attach to. + */ +export function setupBrowserEventForwarding(): void { + browserViewManager.on('state-updated', (state: BrowserState) => { + const windows = BrowserWindow.getAllWindows(); + for (const win of windows) { + if (!win.isDestroyed() && win.webContents) { + win.webContents.send('browser:didUpdateState', state); + } + } + }); +} diff --git a/apps/x/apps/main/src/browser/navigation.ts b/apps/x/apps/main/src/browser/navigation.ts new file mode 100644 index 00000000..ac840956 --- /dev/null +++ b/apps/x/apps/main/src/browser/navigation.ts @@ -0,0 +1,41 @@ +const SEARCH_ENGINE_BASE_URL = 'https://www.google.com/search?q='; + +const HAS_SCHEME_RE = /^[a-z][a-z0-9+.-]*:/i; +const IPV4_HOST_RE = /^\d{1,3}(?:\.\d{1,3}){3}(?::\d+)?(?:\/.*)?$/; +const LOCALHOST_RE = /^localhost(?::\d+)?(?:\/.*)?$/i; +const DOMAIN_LIKE_RE = /^[\w.-]+\.[a-z]{2,}(?::\d+)?(?:\/.*)?$/i; + +export function normalizeNavigationTarget(target: string): string { + const trimmed = target.trim(); + if (!trimmed) { + throw new Error('Navigation target cannot be empty.'); + } + + const lower = trimmed.toLowerCase(); + if ( + lower.startsWith('javascript:') + || lower.startsWith('file://') + || lower.startsWith('chrome://') + || lower.startsWith('chrome-extension://') + ) { + throw new Error('That URL scheme is not allowed in the embedded browser.'); + } + + if (HAS_SCHEME_RE.test(trimmed)) { + return trimmed; + } + + const looksLikeHost = + LOCALHOST_RE.test(trimmed) + || DOMAIN_LIKE_RE.test(trimmed) + || IPV4_HOST_RE.test(trimmed); + + if (looksLikeHost && !/\s/.test(trimmed)) { + const scheme = LOCALHOST_RE.test(trimmed) || IPV4_HOST_RE.test(trimmed) + ? 'http://' + : 'https://'; + return `${scheme}${trimmed}`; + } + + return `${SEARCH_ENGINE_BASE_URL}${encodeURIComponent(trimmed)}`; +} diff --git a/apps/x/apps/main/src/browser/page-scripts.ts b/apps/x/apps/main/src/browser/page-scripts.ts new file mode 100644 index 00000000..fc079327 --- /dev/null +++ b/apps/x/apps/main/src/browser/page-scripts.ts @@ -0,0 +1,546 @@ +import type { BrowserPageElement } from '@x/shared/dist/browser-control.js'; + +const INTERACTABLE_SELECTORS = [ + 'a[href]', + 'button', + 'input', + 'textarea', + 'select', + 'summary', + '[role="button"]', + '[role="link"]', + '[role="tab"]', + '[role="menuitem"]', + '[role="option"]', + '[contenteditable="true"]', + '[tabindex]:not([tabindex="-1"])', +].join(', '); + +const CLICKABLE_TARGET_SELECTORS = [ + 'a[href]', + 'button', + 'summary', + 'label', + 'input', + 'textarea', + 'select', + '[role="button"]', + '[role="link"]', + '[role="tab"]', + '[role="menuitem"]', + '[role="option"]', + '[role="checkbox"]', + '[role="radio"]', + '[role="switch"]', + '[role="menuitemcheckbox"]', + '[role="menuitemradio"]', + '[aria-pressed]', + '[aria-expanded]', + '[aria-checked]', + '[contenteditable="true"]', + '[tabindex]:not([tabindex="-1"])', +].join(', '); + +const DOM_HELPERS_SOURCE = String.raw` +const truncateText = (value, max) => { + const normalized = String(value ?? '').replace(/\s+/g, ' ').trim(); + if (!normalized) return ''; + if (normalized.length <= max) return normalized; + const safeMax = Math.max(0, max - 3); + return normalized.slice(0, safeMax).trim() + '...'; +}; + +const cssEscapeValue = (value) => { + if (typeof CSS !== 'undefined' && typeof CSS.escape === 'function') { + return CSS.escape(value); + } + return String(value).replace(/[^a-zA-Z0-9_-]/g, (char) => '\\' + char); +}; + +const isVisibleElement = (element) => { + if (!(element instanceof Element)) return false; + const style = window.getComputedStyle(element); + if (style.display === 'none' || style.visibility === 'hidden' || style.opacity === '0') { + return false; + } + if (element.getAttribute('aria-hidden') === 'true') return false; + const rect = element.getBoundingClientRect(); + return rect.width > 0 && rect.height > 0; +}; + +const isDisabledElement = (element) => { + if (!(element instanceof Element)) return true; + if (element.getAttribute('aria-disabled') === 'true') return true; + return 'disabled' in element && Boolean(element.disabled); +}; + +const isUselessClickTarget = (element) => ( + element === document.body + || element === document.documentElement +); + +const getElementRole = (element) => { + const explicitRole = element.getAttribute('role'); + if (explicitRole) return explicitRole; + if (element instanceof HTMLAnchorElement) return 'link'; + if (element instanceof HTMLButtonElement) return 'button'; + if (element instanceof HTMLInputElement) return element.type === 'checkbox' ? 'checkbox' : 'input'; + if (element instanceof HTMLTextAreaElement) return 'textbox'; + if (element instanceof HTMLSelectElement) return 'combobox'; + if (element instanceof HTMLElement && element.isContentEditable) return 'textbox'; + return null; +}; + +const getElementType = (element) => { + if (element instanceof HTMLInputElement) return element.type || 'text'; + if (element instanceof HTMLTextAreaElement) return 'textarea'; + if (element instanceof HTMLSelectElement) return 'select'; + if (element instanceof HTMLButtonElement) return 'button'; + if (element instanceof HTMLElement && element.isContentEditable) return 'contenteditable'; + return null; +}; + +const getElementLabel = (element) => { + const ariaLabel = truncateText(element.getAttribute('aria-label') ?? '', 120); + if (ariaLabel) return ariaLabel; + + if ('labels' in element && element.labels && element.labels.length > 0) { + const labelText = truncateText( + Array.from(element.labels).map((label) => label.innerText || label.textContent || '').join(' '), + 120, + ); + if (labelText) return labelText; + } + + if (element.id) { + const label = document.querySelector('label[for="' + cssEscapeValue(element.id) + '"]'); + const labelText = truncateText(label?.textContent ?? '', 120); + if (labelText) return labelText; + } + + const placeholder = truncateText(element.getAttribute('placeholder') ?? '', 120); + if (placeholder) return placeholder; + + const text = truncateText( + element instanceof HTMLInputElement || element instanceof HTMLTextAreaElement + ? element.value + : element.textContent ?? '', + 120, + ); + return text || null; +}; + +const describeElement = (element) => { + const role = getElementRole(element) || element.tagName.toLowerCase(); + const label = getElementLabel(element); + return label ? role + ' "' + label + '"' : role; +}; + +const clampNumber = (value, min, max) => Math.min(Math.max(value, min), max); + +const getAssociatedControl = (element) => { + if (!(element instanceof Element)) return null; + if (element instanceof HTMLLabelElement) return element.control; + const parentLabel = element.closest('label'); + return parentLabel instanceof HTMLLabelElement ? parentLabel.control : null; +}; + +const resolveClickTarget = (element) => { + if (!(element instanceof Element)) return null; + + const clickableAncestor = element.closest(${JSON.stringify(CLICKABLE_TARGET_SELECTORS)}); + const labelAncestor = element.closest('label'); + const associatedControl = getAssociatedControl(element); + const candidates = [clickableAncestor, labelAncestor, associatedControl, element]; + + for (const candidate of candidates) { + if (!(candidate instanceof Element)) continue; + if (isUselessClickTarget(candidate)) continue; + if (!isVisibleElement(candidate)) continue; + if (isDisabledElement(candidate)) continue; + return candidate; + } + + for (const candidate of candidates) { + if (candidate instanceof Element) return candidate; + } + + return null; +}; + +const getVerificationTargetState = (element) => { + if (!(element instanceof Element)) return null; + + const text = truncateText(element.innerText || element.textContent || '', 200); + const activeElement = document.activeElement; + const isActive = + activeElement instanceof Element + ? activeElement === element || element.contains(activeElement) + : false; + + return { + selector: buildUniqueSelector(element), + descriptor: describeElement(element), + text: text || null, + checked: + element instanceof HTMLInputElement && (element.type === 'checkbox' || element.type === 'radio') + ? element.checked + : null, + value: + element instanceof HTMLInputElement || element instanceof HTMLTextAreaElement + ? truncateText(element.value ?? '', 200) + : element instanceof HTMLSelectElement + ? truncateText(element.value ?? '', 200) + : element instanceof HTMLElement && element.isContentEditable + ? truncateText(element.innerText || element.textContent || '', 200) + : null, + selectedIndex: element instanceof HTMLSelectElement ? element.selectedIndex : null, + open: + 'open' in element && typeof element.open === 'boolean' + ? element.open + : null, + disabled: isDisabledElement(element), + active: isActive, + ariaChecked: element.getAttribute('aria-checked'), + ariaPressed: element.getAttribute('aria-pressed'), + ariaExpanded: element.getAttribute('aria-expanded'), + }; +}; + +const getPageVerificationState = () => { + const activeElement = document.activeElement instanceof Element ? document.activeElement : null; + return { + url: window.location.href, + title: document.title || '', + textSample: truncateText(document.body?.innerText || document.body?.textContent || '', 2000), + activeSelector: activeElement ? buildUniqueSelector(activeElement) : null, + }; +}; + +const buildUniqueSelector = (element) => { + if (!(element instanceof Element)) return null; + + if (element.id) { + const idSelector = '#' + cssEscapeValue(element.id); + try { + if (document.querySelectorAll(idSelector).length === 1) return idSelector; + } catch {} + } + + const segments = []; + let current = element; + while (current && current instanceof Element && current !== document.documentElement) { + const tag = current.tagName.toLowerCase(); + if (!tag) break; + + let segment = tag; + const name = current.getAttribute('name'); + if (name) { + const nameSelector = tag + '[name="' + cssEscapeValue(name) + '"]'; + try { + if (document.querySelectorAll(nameSelector).length === 1) { + segments.unshift(nameSelector); + return segments.join(' > '); + } + } catch {} + } + + const parent = current.parentElement; + if (parent) { + const sameTagSiblings = Array.from(parent.children).filter((child) => child.tagName === current.tagName); + const position = sameTagSiblings.indexOf(current) + 1; + segment += ':nth-of-type(' + position + ')'; + } + + segments.unshift(segment); + const selector = segments.join(' > '); + try { + if (document.querySelectorAll(selector).length === 1) return selector; + } catch {} + + current = current.parentElement; + } + + return segments.length > 0 ? segments.join(' > ') : null; +}; +`; + +type RawBrowserPageElement = BrowserPageElement & { + selector: string; +}; + +export type RawBrowserPageSnapshot = { + url: string; + title: string; + loading: boolean; + text: string; + elements: RawBrowserPageElement[]; +}; + +export type ElementTarget = { + index?: number; + selector?: string; + snapshotId?: string; +}; + +export function buildReadPageScript(maxElements: number, maxTextLength: number): string { + return `(() => { + ${DOM_HELPERS_SOURCE} + const candidates = Array.from(document.querySelectorAll(${JSON.stringify(INTERACTABLE_SELECTORS)})); + const elements = []; + const seenSelectors = new Set(); + + for (const candidate of candidates) { + if (!(candidate instanceof Element)) continue; + if (!isVisibleElement(candidate)) continue; + + const selector = buildUniqueSelector(candidate); + if (!selector || seenSelectors.has(selector)) continue; + seenSelectors.add(selector); + + elements.push({ + index: elements.length + 1, + selector, + tagName: candidate.tagName.toLowerCase(), + role: getElementRole(candidate), + type: getElementType(candidate), + label: getElementLabel(candidate), + text: truncateText(candidate.innerText || candidate.textContent || '', 120) || null, + placeholder: truncateText(candidate.getAttribute('placeholder') ?? '', 120) || null, + href: candidate instanceof HTMLAnchorElement ? candidate.href : candidate.getAttribute('href'), + disabled: isDisabledElement(candidate), + }); + + if (elements.length >= ${JSON.stringify(maxElements)}) break; + } + + return { + url: window.location.href, + title: document.title || '', + loading: document.readyState !== 'complete', + text: truncateText(document.body?.innerText || document.body?.textContent || '', ${JSON.stringify(maxTextLength)}), + elements, + }; + })()`; +} + +export function buildClickScript(selector: string): string { + return `(() => { + ${DOM_HELPERS_SOURCE} + const requestedSelector = ${JSON.stringify(selector)}; + if (/^(body|html)$/i.test(requestedSelector.trim())) { + return { + ok: false, + error: 'Refusing to click the page body. Read the page again and target a specific element.', + }; + } + + const element = document.querySelector(requestedSelector); + if (!(element instanceof Element)) { + return { ok: false, error: 'Element not found.' }; + } + if (isUselessClickTarget(element)) { + return { + ok: false, + error: 'Refusing to click the page body. Read the page again and target a specific element.', + }; + } + + const target = resolveClickTarget(element); + if (!(target instanceof Element)) { + return { ok: false, error: 'Could not resolve a clickable target.' }; + } + if (isUselessClickTarget(target)) { + return { + ok: false, + error: 'Resolved click target was too generic. Read the page again and choose a specific control.', + }; + } + if (!isVisibleElement(target)) { + return { ok: false, error: 'Resolved click target is not visible.' }; + } + if (isDisabledElement(target)) { + return { ok: false, error: 'Resolved click target is disabled.' }; + } + + const before = { + page: getPageVerificationState(), + target: getVerificationTargetState(target), + }; + + if (target instanceof HTMLElement) { + target.scrollIntoView({ block: 'center', inline: 'center' }); + target.focus({ preventScroll: true }); + } + + const rect = target.getBoundingClientRect(); + const clientX = clampNumber(rect.left + (rect.width / 2), 1, Math.max(1, window.innerWidth - 1)); + const clientY = clampNumber(rect.top + (rect.height / 2), 1, Math.max(1, window.innerHeight - 1)); + const topElement = document.elementFromPoint(clientX, clientY); + const eventTarget = + topElement instanceof Element && (topElement === target || topElement.contains(target) || target.contains(topElement)) + ? topElement + : target; + + if (eventTarget instanceof HTMLElement) { + eventTarget.focus({ preventScroll: true }); + } + + return { + ok: true, + description: describeElement(target), + clickPoint: { + x: Math.round(clientX), + y: Math.round(clientY), + }, + verification: { + before, + targetSelector: buildUniqueSelector(target) || requestedSelector, + }, + }; + })()`; +} + +export function buildVerifyClickScript(targetSelector: string | null, before: unknown): string { + return `(() => { + ${DOM_HELPERS_SOURCE} + const beforeState = ${JSON.stringify(before)}; + const selector = ${JSON.stringify(targetSelector)}; + const afterPage = getPageVerificationState(); + const afterTarget = selector ? getVerificationTargetState(document.querySelector(selector)) : null; + const beforeTarget = beforeState?.target ?? null; + const reasons = []; + + if (beforeState?.page?.url !== afterPage.url) reasons.push('url changed'); + if (beforeState?.page?.title !== afterPage.title) reasons.push('title changed'); + if (beforeState?.page?.textSample !== afterPage.textSample) reasons.push('page text changed'); + if (beforeState?.page?.activeSelector !== afterPage.activeSelector) reasons.push('focus changed'); + + if (beforeTarget && !afterTarget) { + reasons.push('clicked element disappeared'); + } + + if (beforeTarget && afterTarget) { + if (beforeTarget.checked !== afterTarget.checked) reasons.push('checked state changed'); + if (beforeTarget.value !== afterTarget.value) reasons.push('value changed'); + if (beforeTarget.selectedIndex !== afterTarget.selectedIndex) reasons.push('selection changed'); + if (beforeTarget.open !== afterTarget.open) reasons.push('open state changed'); + if (beforeTarget.disabled !== afterTarget.disabled) reasons.push('disabled state changed'); + if (beforeTarget.active !== afterTarget.active) reasons.push('target focus changed'); + if (beforeTarget.ariaChecked !== afterTarget.ariaChecked) reasons.push('aria-checked changed'); + if (beforeTarget.ariaPressed !== afterTarget.ariaPressed) reasons.push('aria-pressed changed'); + if (beforeTarget.ariaExpanded !== afterTarget.ariaExpanded) reasons.push('aria-expanded changed'); + if (beforeTarget.text !== afterTarget.text) reasons.push('target text changed'); + } + + return { + changed: reasons.length > 0, + reasons, + }; + })()`; +} + +export function buildTypeScript(selector: string, text: string): string { + return `(() => { + ${DOM_HELPERS_SOURCE} + const element = document.querySelector(${JSON.stringify(selector)}); + if (!(element instanceof Element)) { + return { ok: false, error: 'Element not found.' }; + } + if (!isVisibleElement(element)) { + return { ok: false, error: 'Element is not visible.' }; + } + if (isDisabledElement(element)) { + return { ok: false, error: 'Element is disabled.' }; + } + + const nextValue = ${JSON.stringify(text)}; + + const setNativeValue = (target, value) => { + const prototype = Object.getPrototypeOf(target); + const descriptor = Object.getOwnPropertyDescriptor(prototype, 'value'); + if (descriptor && typeof descriptor.set === 'function') { + descriptor.set.call(target, value); + } else { + target.value = value; + } + }; + + if (element instanceof HTMLInputElement || element instanceof HTMLTextAreaElement) { + if (element.readOnly) { + return { ok: false, error: 'Element is read-only.' }; + } + element.scrollIntoView({ block: 'center', inline: 'center' }); + element.focus({ preventScroll: true }); + setNativeValue(element, nextValue); + element.dispatchEvent(new InputEvent('input', { bubbles: true, data: nextValue, inputType: 'insertText' })); + element.dispatchEvent(new Event('change', { bubbles: true })); + return { ok: true, description: describeElement(element) }; + } + + if (element instanceof HTMLElement && element.isContentEditable) { + element.scrollIntoView({ block: 'center', inline: 'center' }); + element.focus({ preventScroll: true }); + element.textContent = nextValue; + element.dispatchEvent(new InputEvent('input', { bubbles: true, data: nextValue, inputType: 'insertText' })); + return { ok: true, description: describeElement(element) }; + } + + return { ok: false, error: 'Element does not accept text input.' }; + })()`; +} + +export function buildFocusScript(selector: string): string { + return `(() => { + ${DOM_HELPERS_SOURCE} + const element = document.querySelector(${JSON.stringify(selector)}); + if (!(element instanceof Element)) { + return { ok: false, error: 'Element not found.' }; + } + if (!isVisibleElement(element)) { + return { ok: false, error: 'Element is not visible.' }; + } + if (element instanceof HTMLElement) { + element.scrollIntoView({ block: 'center', inline: 'center' }); + element.focus({ preventScroll: true }); + } + return { ok: true, description: describeElement(element) }; + })()`; +} + +export function buildScrollScript(offset: number): string { + return `(() => { + window.scrollBy({ top: ${JSON.stringify(offset)}, left: 0, behavior: 'auto' }); + return { ok: true }; + })()`; +} + +export function normalizeKeyCode(key: string): string { + const trimmed = key.trim(); + if (!trimmed) return 'Enter'; + + const aliases: Record = { + esc: 'Escape', + escape: 'Escape', + return: 'Enter', + enter: 'Enter', + tab: 'Tab', + space: 'Space', + ' ': 'Space', + left: 'ArrowLeft', + right: 'ArrowRight', + up: 'ArrowUp', + down: 'ArrowDown', + arrowleft: 'ArrowLeft', + arrowright: 'ArrowRight', + arrowup: 'ArrowUp', + arrowdown: 'ArrowDown', + backspace: 'Backspace', + delete: 'Delete', + }; + + const alias = aliases[trimmed.toLowerCase()]; + if (alias) return alias; + if (trimmed.length === 1) return trimmed.toUpperCase(); + return trimmed[0].toUpperCase() + trimmed.slice(1); +} diff --git a/apps/x/apps/main/src/browser/view.ts b/apps/x/apps/main/src/browser/view.ts new file mode 100644 index 00000000..d319c5fb --- /dev/null +++ b/apps/x/apps/main/src/browser/view.ts @@ -0,0 +1,797 @@ +import { randomUUID } from 'node:crypto'; +import { EventEmitter } from 'node:events'; +import { BrowserWindow, WebContentsView, session, shell, type Session } from 'electron'; +import type { + BrowserPageElement, + BrowserPageSnapshot, + BrowserState, + BrowserTabState, +} from '@x/shared/dist/browser-control.js'; +import { normalizeNavigationTarget } from './navigation.js'; +import { + buildClickScript, + buildFocusScript, + buildReadPageScript, + buildScrollScript, + buildTypeScript, + buildVerifyClickScript, + normalizeKeyCode, + type ElementTarget, + type RawBrowserPageSnapshot, +} from './page-scripts.js'; + +export type { BrowserPageSnapshot, BrowserState, BrowserTabState }; + +/** + * Embedded browser pane implementation. + * + * Each browser tab owns its own WebContentsView. Only the active tab's view is + * attached to the main window at a time, but inactive tabs keep their own page + * history and loaded state in memory so switching tabs feels immediate. + * + * All tabs share one persistent session partition so cookies/localStorage/ + * form-fill state survive app restarts, and the browser surface spoofs a + * standard Chrome UA so sites like Google (OAuth) don't reject it. + */ + +export const BROWSER_PARTITION = 'persist:rowboat-browser'; + +// Claims Chrome 130 on macOS — close enough to recent stable for OAuth servers +// that sniff the UA looking for "real browser" shapes. +const SPOOF_UA = + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36'; + +const HOME_URL = 'https://www.google.com'; +const NAVIGATION_TIMEOUT_MS = 10000; +const POST_ACTION_IDLE_MS = 400; +const POST_ACTION_MAX_ELEMENTS = 25; +const POST_ACTION_MAX_TEXT_LENGTH = 4000; +const DEFAULT_READ_MAX_ELEMENTS = 50; +const DEFAULT_READ_MAX_TEXT_LENGTH = 8000; + +export interface BrowserBounds { + x: number; + y: number; + width: number; + height: number; +} + +type BrowserTab = { + id: string; + view: WebContentsView; + domReadyAt: number | null; + loadError: string | null; +}; + +type CachedSnapshot = { + snapshotId: string; + elements: Array<{ index: number; selector: string }>; +}; + +const EMPTY_STATE: BrowserState = { + activeTabId: null, + tabs: [], +}; + +function abortIfNeeded(signal?: AbortSignal): void { + if (!signal?.aborted) return; + throw signal.reason instanceof Error ? signal.reason : new Error('Browser action aborted'); +} + +async function sleep(ms: number, signal?: AbortSignal): Promise { + if (ms <= 0) return; + abortIfNeeded(signal); + await new Promise((resolve, reject) => { + const abortSignal = signal; + const timer = setTimeout(() => { + abortSignal?.removeEventListener('abort', onAbort); + resolve(); + }, ms); + + const onAbort = () => { + clearTimeout(timer); + abortSignal?.removeEventListener('abort', onAbort); + reject(abortSignal?.reason instanceof Error ? abortSignal.reason : new Error('Browser action aborted')); + }; + + abortSignal?.addEventListener('abort', onAbort, { once: true }); + }); +} + + +export class BrowserViewManager extends EventEmitter { + private window: BrowserWindow | null = null; + private browserSession: Session | null = null; + private tabs = new Map(); + private tabOrder: string[] = []; + private activeTabId: string | null = null; + private attachedTabId: string | null = null; + private visible = false; + private bounds: BrowserBounds = { x: 0, y: 0, width: 0, height: 0 }; + private snapshotCache = new Map(); + + attach(window: BrowserWindow): void { + this.window = window; + window.on('closed', () => { + this.window = null; + this.browserSession = null; + this.tabs.clear(); + this.tabOrder = []; + this.activeTabId = null; + this.attachedTabId = null; + this.visible = false; + this.snapshotCache.clear(); + }); + } + + private getSession(): Session { + if (this.browserSession) return this.browserSession; + const browserSession = session.fromPartition(BROWSER_PARTITION); + browserSession.setUserAgent(SPOOF_UA); + this.browserSession = browserSession; + return browserSession; + } + + private emitState(): void { + this.emit('state-updated', this.snapshotState()); + } + + private getTab(tabId: string | null): BrowserTab | null { + if (!tabId) return null; + return this.tabs.get(tabId) ?? null; + } + + private getActiveTab(): BrowserTab | null { + return this.getTab(this.activeTabId); + } + + private invalidateSnapshot(tabId: string): void { + this.snapshotCache.delete(tabId); + } + + private isEmbeddedTabUrl(url: string): boolean { + return /^https?:\/\//i.test(url) || url === 'about:blank'; + } + + private createView(): WebContentsView { + const view = new WebContentsView({ + webPreferences: { + session: this.getSession(), + contextIsolation: true, + sandbox: true, + nodeIntegration: false, + }, + }); + + view.webContents.setUserAgent(SPOOF_UA); + return view; + } + + private wireEvents(tab: BrowserTab): void { + const { id: tabId, view } = tab; + const wc = view.webContents; + + const reapplyBounds = () => { + if ( + this.attachedTabId === tabId && + this.visible && + this.bounds.width > 0 && + this.bounds.height > 0 + ) { + view.setBounds(this.bounds); + } + }; + + const invalidateAndEmit = () => { + this.invalidateSnapshot(tabId); + this.emitState(); + }; + + wc.on('did-start-navigation', (_event, _url, _isInPlace, isMainFrame) => { + if (isMainFrame !== false) { + tab.domReadyAt = null; + tab.loadError = null; + } + this.invalidateSnapshot(tabId); + reapplyBounds(); + }); + wc.on('did-navigate', () => { reapplyBounds(); invalidateAndEmit(); }); + wc.on('did-navigate-in-page', () => { reapplyBounds(); invalidateAndEmit(); }); + wc.on('did-start-loading', () => { + tab.loadError = null; + this.invalidateSnapshot(tabId); + reapplyBounds(); + this.emitState(); + }); + wc.on('did-stop-loading', () => { reapplyBounds(); invalidateAndEmit(); }); + wc.on('did-finish-load', () => { reapplyBounds(); invalidateAndEmit(); }); + wc.on('dom-ready', () => { + tab.domReadyAt = Date.now(); + reapplyBounds(); + invalidateAndEmit(); + }); + wc.on('did-frame-finish-load', reapplyBounds); + wc.on('did-fail-load', (_event, errorCode, errorDescription, validatedURL, isMainFrame) => { + if (isMainFrame && errorCode !== -3) { + const target = validatedURL || wc.getURL() || 'page'; + tab.loadError = errorDescription + ? `Failed to load ${target}: ${errorDescription}.` + : `Failed to load ${target}.`; + } + reapplyBounds(); + invalidateAndEmit(); + }); + wc.on('page-title-updated', this.emitState.bind(this)); + + wc.setWindowOpenHandler(({ url }) => { + if (this.isEmbeddedTabUrl(url)) { + void this.newTab(url); + } else { + void shell.openExternal(url); + } + return { action: 'deny' }; + }); + } + + private snapshotTabState(tab: BrowserTab): BrowserTabState { + const wc = tab.view.webContents; + return { + id: tab.id, + url: wc.getURL(), + title: wc.getTitle(), + canGoBack: wc.navigationHistory.canGoBack(), + canGoForward: wc.navigationHistory.canGoForward(), + loading: wc.isLoading(), + }; + } + + private syncAttachedView(): void { + if (!this.window) return; + + const contentView = this.window.contentView; + const activeTab = this.getActiveTab(); + + if (!this.visible || !activeTab) { + const attachedTab = this.getTab(this.attachedTabId); + if (attachedTab) { + contentView.removeChildView(attachedTab.view); + } + this.attachedTabId = null; + return; + } + + if (this.attachedTabId && this.attachedTabId !== activeTab.id) { + const attachedTab = this.getTab(this.attachedTabId); + if (attachedTab) { + contentView.removeChildView(attachedTab.view); + } + this.attachedTabId = null; + } + + if (this.attachedTabId !== activeTab.id) { + contentView.addChildView(activeTab.view); + this.attachedTabId = activeTab.id; + } + + if (this.bounds.width > 0 && this.bounds.height > 0) { + activeTab.view.setBounds(this.bounds); + } + } + + private createTab(initialUrl: string): BrowserTab { + if (!this.window) { + throw new Error('BrowserViewManager: no window attached'); + } + + const tabId = randomUUID(); + const tab: BrowserTab = { + id: tabId, + view: this.createView(), + domReadyAt: null, + loadError: null, + }; + + this.wireEvents(tab); + this.tabs.set(tabId, tab); + this.tabOrder.push(tabId); + this.activeTabId = tabId; + this.invalidateSnapshot(tabId); + this.syncAttachedView(); + this.emitState(); + + const targetUrl = + initialUrl === 'about:blank' + ? HOME_URL + : normalizeNavigationTarget(initialUrl); + void tab.view.webContents.loadURL(targetUrl).catch((error) => { + tab.loadError = error instanceof Error + ? error.message + : `Failed to load ${targetUrl}.`; + this.emitState(); + }); + + return tab; + } + + private ensureInitialTab(): BrowserTab { + const activeTab = this.getActiveTab(); + if (activeTab) return activeTab; + return this.createTab(HOME_URL); + } + + private destroyTab(tab: BrowserTab): void { + this.invalidateSnapshot(tab.id); + tab.view.webContents.removeAllListeners(); + if (!tab.view.webContents.isDestroyed()) { + tab.view.webContents.close(); + } + } + + private async waitForWebContentsSettle( + tab: BrowserTab, + signal?: AbortSignal, + idleMs = POST_ACTION_IDLE_MS, + timeoutMs = NAVIGATION_TIMEOUT_MS, + ): Promise { + const wc = tab.view.webContents; + const startedAt = Date.now(); + let sawLoading = wc.isLoading(); + + while (Date.now() - startedAt < timeoutMs) { + abortIfNeeded(signal); + if (wc.isDestroyed()) return; + if (tab.loadError) { + throw new Error(tab.loadError); + } + + if (tab.domReadyAt != null) { + const domReadyForMs = Date.now() - tab.domReadyAt; + const requiredIdleMs = sawLoading ? idleMs : Math.min(idleMs, 200); + if (domReadyForMs >= requiredIdleMs) return; + await sleep(Math.min(100, requiredIdleMs - domReadyForMs), signal); + continue; + } + + if (wc.isLoading()) { + sawLoading = true; + await sleep(100, signal); + continue; + } + + await sleep(sawLoading ? idleMs : Math.min(idleMs, 200), signal); + if (tab.loadError) { + throw new Error(tab.loadError); + } + if (!wc.isLoading() || tab.domReadyAt != null) return; + sawLoading = true; + } + } + + private async executeOnActiveTab( + script: string, + signal?: AbortSignal, + options?: { waitForReady?: boolean }, + ): Promise { + abortIfNeeded(signal); + const activeTab = this.getActiveTab() ?? this.ensureInitialTab(); + if (options?.waitForReady !== false) { + await this.waitForWebContentsSettle(activeTab, signal); + } + abortIfNeeded(signal); + return activeTab.view.webContents.executeJavaScript(script, true) as Promise; + } + + private cacheSnapshot(tabId: string, rawSnapshot: RawBrowserPageSnapshot, loading: boolean): BrowserPageSnapshot { + const snapshotId = randomUUID(); + const elements: BrowserPageElement[] = rawSnapshot.elements.map((element, index) => { + const { selector, ...rest } = element; + void selector; + return { + ...rest, + index: index + 1, + }; + }); + + this.snapshotCache.set(tabId, { + snapshotId, + elements: rawSnapshot.elements.map((element, index) => ({ + index: index + 1, + selector: element.selector, + })), + }); + + return { + snapshotId, + url: rawSnapshot.url, + title: rawSnapshot.title, + loading, + text: rawSnapshot.text, + elements, + }; + } + + private resolveElementSelector(tabId: string, target: ElementTarget): { ok: true; selector: string } | { ok: false; error: string } { + if (target.selector?.trim()) { + return { ok: true, selector: target.selector.trim() }; + } + + if (target.index == null) { + return { ok: false, error: 'Provide an element index or selector.' }; + } + + const cachedSnapshot = this.snapshotCache.get(tabId); + if (!cachedSnapshot) { + return { ok: false, error: 'No page snapshot is available yet. Call read-page first.' }; + } + + if (target.snapshotId && cachedSnapshot.snapshotId !== target.snapshotId) { + return { ok: false, error: 'The page changed since the last read-page call. Call read-page again.' }; + } + + const entry = cachedSnapshot.elements.find((element) => element.index === target.index); + if (!entry) { + return { ok: false, error: `No element found for index ${target.index}.` }; + } + + return { ok: true, selector: entry.selector }; + } + + setVisible(visible: boolean): void { + this.visible = visible; + if (visible) { + this.ensureInitialTab(); + } + this.syncAttachedView(); + } + + setBounds(bounds: BrowserBounds): void { + this.bounds = bounds; + const activeTab = this.getActiveTab(); + if (activeTab && this.attachedTabId === activeTab.id && this.visible) { + activeTab.view.setBounds(bounds); + } + } + + async ensureActiveTabReady(signal?: AbortSignal): Promise { + const activeTab = this.getActiveTab() ?? this.ensureInitialTab(); + await this.waitForWebContentsSettle(activeTab, signal); + } + + async newTab(rawUrl?: string): Promise<{ ok: boolean; tabId?: string; error?: string }> { + try { + const tab = this.createTab(rawUrl?.trim() ? rawUrl : HOME_URL); + return { ok: true, tabId: tab.id }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : String(err) }; + } + } + + switchTab(tabId: string): { ok: boolean } { + if (!this.tabs.has(tabId)) return { ok: false }; + if (this.activeTabId === tabId) return { ok: true }; + this.activeTabId = tabId; + this.syncAttachedView(); + this.emitState(); + return { ok: true }; + } + + closeTab(tabId: string): { ok: boolean } { + const tab = this.tabs.get(tabId); + if (!tab) return { ok: false }; + if (this.tabOrder.length <= 1) return { ok: false }; + + const closingIndex = this.tabOrder.indexOf(tabId); + const nextActiveTabId = + this.activeTabId === tabId + ? this.tabOrder[closingIndex + 1] ?? this.tabOrder[closingIndex - 1] ?? null + : this.activeTabId; + + if (this.attachedTabId === tabId && this.window) { + this.window.contentView.removeChildView(tab.view); + this.attachedTabId = null; + } + + this.tabs.delete(tabId); + this.tabOrder = this.tabOrder.filter((id) => id !== tabId); + this.activeTabId = nextActiveTabId; + this.destroyTab(tab); + this.syncAttachedView(); + this.emitState(); + + return { ok: true }; + } + + async navigate(rawUrl: string): Promise<{ ok: boolean; error?: string }> { + try { + const activeTab = this.getActiveTab() ?? this.ensureInitialTab(); + this.invalidateSnapshot(activeTab.id); + await activeTab.view.webContents.loadURL(normalizeNavigationTarget(rawUrl)); + return { ok: true }; + } catch (err) { + return { ok: false, error: err instanceof Error ? err.message : String(err) }; + } + } + + back(): { ok: boolean } { + const activeTab = this.getActiveTab(); + if (!activeTab) return { ok: false }; + const history = activeTab.view.webContents.navigationHistory; + if (!history.canGoBack()) return { ok: false }; + this.invalidateSnapshot(activeTab.id); + history.goBack(); + return { ok: true }; + } + + forward(): { ok: boolean } { + const activeTab = this.getActiveTab(); + if (!activeTab) return { ok: false }; + const history = activeTab.view.webContents.navigationHistory; + if (!history.canGoForward()) return { ok: false }; + this.invalidateSnapshot(activeTab.id); + history.goForward(); + return { ok: true }; + } + + reload(): void { + const activeTab = this.getActiveTab(); + if (!activeTab) return; + this.invalidateSnapshot(activeTab.id); + activeTab.view.webContents.reload(); + } + + async readPage( + options?: { maxElements?: number; maxTextLength?: number; waitForReady?: boolean }, + signal?: AbortSignal, + ): Promise<{ ok: boolean; page?: BrowserPageSnapshot; error?: string }> { + try { + const activeTab = this.getActiveTab() ?? this.ensureInitialTab(); + const rawSnapshot = await this.executeOnActiveTab( + buildReadPageScript( + options?.maxElements ?? DEFAULT_READ_MAX_ELEMENTS, + options?.maxTextLength ?? DEFAULT_READ_MAX_TEXT_LENGTH, + ), + signal, + { waitForReady: options?.waitForReady }, + ); + return { + ok: true, + page: this.cacheSnapshot(activeTab.id, rawSnapshot, activeTab.view.webContents.isLoading()), + }; + } catch (error) { + return { + ok: false, + error: error instanceof Error ? error.message : 'Failed to read the current page.', + }; + } + } + + async readPageSummary( + signal?: AbortSignal, + options?: { waitForReady?: boolean }, + ): Promise { + const result = await this.readPage( + { + maxElements: POST_ACTION_MAX_ELEMENTS, + maxTextLength: POST_ACTION_MAX_TEXT_LENGTH, + waitForReady: options?.waitForReady, + }, + signal, + ); + return result.ok ? result.page ?? null : null; + } + + async click(target: ElementTarget, signal?: AbortSignal): Promise<{ ok: boolean; error?: string; description?: string }> { + const activeTab = this.getActiveTab(); + if (!activeTab) { + return { ok: false, error: 'No active browser tab is open.' }; + } + + const resolved = this.resolveElementSelector(activeTab.id, target); + if (!resolved.ok) return resolved; + + try { + const result = await this.executeOnActiveTab<{ + ok: boolean; + error?: string; + description?: string; + clickPoint?: { + x: number; + y: number; + }; + verification?: { + before: unknown; + targetSelector: string | null; + }; + }>( + buildClickScript(resolved.selector), + signal, + ); + if (!result.ok) return result; + if (!result.clickPoint) { + return { + ok: false, + error: 'Could not determine where to click on the page.', + }; + } + + this.window?.focus(); + activeTab.view.webContents.focus(); + activeTab.view.webContents.sendInputEvent({ + type: 'mouseMove', + x: result.clickPoint.x, + y: result.clickPoint.y, + movementX: 0, + movementY: 0, + }); + activeTab.view.webContents.sendInputEvent({ + type: 'mouseDown', + x: result.clickPoint.x, + y: result.clickPoint.y, + button: 'left', + clickCount: 1, + }); + activeTab.view.webContents.sendInputEvent({ + type: 'mouseUp', + x: result.clickPoint.x, + y: result.clickPoint.y, + button: 'left', + clickCount: 1, + }); + + this.invalidateSnapshot(activeTab.id); + await this.waitForWebContentsSettle(activeTab, signal); + + if (result.verification) { + const verification = await this.executeOnActiveTab<{ changed: boolean; reasons: string[] }>( + buildVerifyClickScript(result.verification.targetSelector, result.verification.before), + signal, + { waitForReady: false }, + ); + + if (!verification.changed) { + return { + ok: false, + error: 'Click did not change the page state. Target may not be the correct control.', + description: result.description, + }; + } + } + + return result; + } catch (error) { + return { + ok: false, + error: error instanceof Error ? error.message : 'Failed to click the element.', + }; + } + } + + async type(target: ElementTarget, text: string, signal?: AbortSignal): Promise<{ ok: boolean; error?: string; description?: string }> { + const activeTab = this.getActiveTab(); + if (!activeTab) { + return { ok: false, error: 'No active browser tab is open.' }; + } + + const resolved = this.resolveElementSelector(activeTab.id, target); + if (!resolved.ok) return resolved; + + try { + const result = await this.executeOnActiveTab<{ ok: boolean; error?: string; description?: string }>( + buildTypeScript(resolved.selector, text), + signal, + ); + if (!result.ok) return result; + this.invalidateSnapshot(activeTab.id); + await this.waitForWebContentsSettle(activeTab, signal); + return result; + } catch (error) { + return { + ok: false, + error: error instanceof Error ? error.message : 'Failed to type into the element.', + }; + } + } + + async press( + key: string, + target?: ElementTarget, + signal?: AbortSignal, + ): Promise<{ ok: boolean; error?: string; description?: string }> { + const activeTab = this.getActiveTab(); + if (!activeTab) { + return { ok: false, error: 'No active browser tab is open.' }; + } + + let description = 'active element'; + + if (target?.index != null || target?.selector?.trim()) { + const resolved = this.resolveElementSelector(activeTab.id, target); + if (!resolved.ok) return resolved; + + try { + const focusResult = await this.executeOnActiveTab<{ ok: boolean; error?: string; description?: string }>( + buildFocusScript(resolved.selector), + signal, + ); + if (!focusResult.ok) return focusResult; + description = focusResult.description ?? description; + } catch (error) { + return { + ok: false, + error: error instanceof Error ? error.message : 'Failed to focus the element before pressing a key.', + }; + } + } + + try { + const wc = activeTab.view.webContents; + const keyCode = normalizeKeyCode(key); + wc.sendInputEvent({ type: 'keyDown', keyCode }); + if (keyCode.length === 1) { + wc.sendInputEvent({ type: 'char', keyCode }); + } + wc.sendInputEvent({ type: 'keyUp', keyCode }); + + this.invalidateSnapshot(activeTab.id); + await this.waitForWebContentsSettle(activeTab, signal); + + return { + ok: true, + description: `${keyCode} on ${description}`, + }; + } catch (error) { + return { + ok: false, + error: error instanceof Error ? error.message : 'Failed to press the requested key.', + }; + } + } + + async scroll(direction: 'up' | 'down' = 'down', amount = 700, signal?: AbortSignal): Promise<{ ok: boolean; error?: string }> { + const activeTab = this.getActiveTab(); + if (!activeTab) { + return { ok: false, error: 'No active browser tab is open.' }; + } + + try { + const offset = Math.max(1, amount) * (direction === 'up' ? -1 : 1); + const result = await this.executeOnActiveTab<{ ok: boolean; error?: string }>( + buildScrollScript(offset), + signal, + ); + if (!result.ok) return result; + this.invalidateSnapshot(activeTab.id); + await sleep(250, signal); + return result; + } catch (error) { + return { + ok: false, + error: error instanceof Error ? error.message : 'Failed to scroll the page.', + }; + } + } + + async wait(ms = 1000, signal?: AbortSignal): Promise { + await sleep(ms, signal); + const activeTab = this.getActiveTab(); + if (!activeTab) return; + await this.waitForWebContentsSettle(activeTab, signal); + } + + getState(): BrowserState { + return this.snapshotState(); + } + + private snapshotState(): BrowserState { + if (this.tabOrder.length === 0) return { ...EMPTY_STATE }; + return { + activeTabId: this.activeTabId, + tabs: this.tabOrder + .map((tabId) => this.tabs.get(tabId)) + .filter((tab): tab is BrowserTab => tab != null) + .map((tab) => this.snapshotTabState(tab)), + }; + } +} + +export const browserViewManager = new BrowserViewManager(); diff --git a/apps/x/apps/main/src/composio-handler.ts b/apps/x/apps/main/src/composio-handler.ts index 111eb5a5..274cfb2a 100644 --- a/apps/x/apps/main/src/composio-handler.ts +++ b/apps/x/apps/main/src/composio-handler.ts @@ -44,6 +44,7 @@ export async function isConfigured(): Promise<{ configured: boolean }> { export function setApiKey(apiKey: string): { success: boolean; error?: string } { try { composioClient.setApiKey(apiKey); + invalidateCopilotInstructionsCache(); return { success: true }; } catch (error) { return { diff --git a/apps/x/apps/main/src/ipc.ts b/apps/x/apps/main/src/ipc.ts index a2230eda..a9de9572 100644 --- a/apps/x/apps/main/src/ipc.ts +++ b/apps/x/apps/main/src/ipc.ts @@ -44,6 +44,15 @@ import { getBillingInfo } from '@x/core/dist/billing/billing.js'; import { summarizeMeeting } from '@x/core/dist/knowledge/summarize_meeting.js'; import { getAccessToken } from '@x/core/dist/auth/tokens.js'; import { getRowboatConfig } from '@x/core/dist/config/rowboat.js'; +import { triggerTrackUpdate } from '@x/core/dist/knowledge/track/runner.js'; +import { trackBus } from '@x/core/dist/knowledge/track/bus.js'; +import { + fetchYaml, + updateTrackBlock, + replaceTrackBlockYaml, + deleteTrackBlock, +} from '@x/core/dist/knowledge/track/fileops.js'; +import { browserIpcHandlers } from './browser/ipc.js'; /** * Convert markdown to a styled HTML document for PDF/DOCX export. @@ -110,6 +119,18 @@ function markdownToHtml(markdown: string, title: string): string { ${html}` } +function resolveShellPath(filePath: string): string { + if (filePath.startsWith('~')) { + return path.join(os.homedir(), filePath.slice(1)); + } + + if (path.isAbsolute(filePath)) { + return filePath; + } + + return workspace.resolveWorkspacePath(filePath); +} + type InvokeChannels = ipc.InvokeChannels; type IPCChannels = ipc.IPCChannels; @@ -271,7 +292,7 @@ function handleWorkspaceChange(event: z.infer { }); } +let tracksWatcher: (() => void) | null = null; +export function startTracksWatcher(): void { + if (tracksWatcher) return; + tracksWatcher = trackBus.subscribe((event) => { + const windows = BrowserWindow.getAllWindows(); + for (const win of windows) { + if (!win.isDestroyed() && win.webContents) { + win.webContents.send('tracks:events', event); + } + } + }); +} + export function stopRunsWatcher(): void { if (runsWatcher) { runsWatcher(); @@ -421,7 +455,7 @@ export function setupIpcHandlers() { return runsCore.createRun(args); }, 'runs:createMessage': async (_event, args) => { - return { messageId: await runsCore.createMessage(args.runId, args.message, args.voiceInput, args.voiceOutput, args.searchEnabled) }; + return { messageId: await runsCore.createMessage(args.runId, args.message, args.voiceInput, args.voiceOutput, args.searchEnabled, args.middlePaneContext) }; }, 'runs:authorizePermission': async (_event, args) => { await runsCore.authorizePermission(args.runId, args.authorization); @@ -607,24 +641,12 @@ export function setupIpcHandlers() { }, // Shell integration handlers 'shell:openPath': async (_event, args) => { - let filePath = args.path; - if (filePath.startsWith('~')) { - filePath = path.join(os.homedir(), filePath.slice(1)); - } else if (!path.isAbsolute(filePath)) { - // Workspace-relative path — resolve against ~/.rowboat/ - filePath = path.join(os.homedir(), '.rowboat', filePath); - } + const filePath = resolveShellPath(args.path); const error = await shell.openPath(filePath); return { error: error || undefined }; }, 'shell:readFileBase64': async (_event, args) => { - let filePath = args.path; - if (filePath.startsWith('~')) { - filePath = path.join(os.homedir(), filePath.slice(1)); - } else if (!path.isAbsolute(filePath)) { - // Workspace-relative path — resolve against ~/.rowboat/ - filePath = path.join(os.homedir(), '.rowboat', filePath); - } + const filePath = resolveShellPath(args.path); const stat = await fs.stat(filePath); if (stat.size > 10 * 1024 * 1024) { throw new Error('File too large (>10MB)'); @@ -758,9 +780,53 @@ export function setupIpcHandlers() { 'voice:synthesize': async (_event, args) => { return voice.synthesizeSpeech(args.text); }, + // Track handlers + 'track:run': async (_event, args) => { + const result = await triggerTrackUpdate(args.trackId, args.filePath); + return { success: !result.error, summary: result.summary ?? undefined, error: result.error }; + }, + 'track:get': async (_event, args) => { + try { + const yaml = await fetchYaml(args.filePath, args.trackId); + if (yaml === null) return { success: false, error: 'Track not found' }; + return { success: true, yaml }; + } catch (err) { + return { success: false, error: err instanceof Error ? err.message : String(err) }; + } + }, + 'track:update': async (_event, args) => { + try { + await updateTrackBlock(args.filePath, args.trackId, args.updates as Record); + const yaml = await fetchYaml(args.filePath, args.trackId); + if (yaml === null) return { success: false, error: 'Track vanished after update' }; + return { success: true, yaml }; + } catch (err) { + return { success: false, error: err instanceof Error ? err.message : String(err) }; + } + }, + 'track:replaceYaml': async (_event, args) => { + try { + await replaceTrackBlockYaml(args.filePath, args.trackId, args.yaml); + const yaml = await fetchYaml(args.filePath, args.trackId); + if (yaml === null) return { success: false, error: 'Track vanished after replace' }; + return { success: true, yaml }; + } catch (err) { + return { success: false, error: err instanceof Error ? err.message : String(err) }; + } + }, + 'track:delete': async (_event, args) => { + try { + await deleteTrackBlock(args.filePath, args.trackId); + return { success: true }; + } catch (err) { + return { success: false, error: err instanceof Error ? err.message : String(err) }; + } + }, // Billing handler 'billing:getInfo': async () => { return await getBillingInfo(); }, + // Embedded browser handlers (WebContentsView + navigation) + ...browserIpcHandlers, }); } diff --git a/apps/x/apps/main/src/main.ts b/apps/x/apps/main/src/main.ts index 42c9f3fd..eea21481 100644 --- a/apps/x/apps/main/src/main.ts +++ b/apps/x/apps/main/src/main.ts @@ -1,9 +1,10 @@ -import { app, BrowserWindow, desktopCapturer, protocol, net, shell, session } from "electron"; +import { app, BrowserWindow, desktopCapturer, protocol, net, shell, session, type Session } from "electron"; import path from "node:path"; import { setupIpcHandlers, startRunsWatcher, startServicesWatcher, + startTracksWatcher, startWorkspaceWatcher, stopRunsWatcher, stopServicesWatcher, @@ -22,11 +23,19 @@ import { init as initNoteTagging } from "@x/core/dist/knowledge/tag_notes.js"; import { init as initInlineTasks } from "@x/core/dist/knowledge/inline_tasks.js"; import { init as initAgentRunner } from "@x/core/dist/agent-schedule/runner.js"; import { init as initAgentNotes } from "@x/core/dist/knowledge/agent_notes.js"; +import { init as initTrackScheduler } from "@x/core/dist/knowledge/track/scheduler.js"; +import { init as initTrackEventProcessor } from "@x/core/dist/knowledge/track/events.js"; +import { init as initLocalSites, shutdown as shutdownLocalSites } from "@x/core/dist/local-sites/server.js"; + import { initConfigs } from "@x/core/dist/config/initConfigs.js"; import started from "electron-squirrel-startup"; import { execSync, exec, execFileSync } from "node:child_process"; import { promisify } from "node:util"; import { init as initChromeSync } from "@x/core/dist/knowledge/chrome-extension/server/server.js"; +import { registerBrowserControlService } from "@x/core/dist/di/container.js"; +import { browserViewManager, BROWSER_PARTITION } from "./browser/view.js"; +import { setupBrowserEventForwarding } from "./browser/ipc.js"; +import { ElectronBrowserControlService } from "./browser/control-service.js"; const execAsync = promisify(exec); @@ -108,6 +117,30 @@ protocol.registerSchemesAsPrivileged([ }, ]); +const ALLOWED_SESSION_PERMISSIONS = new Set(["media", "display-capture", "clipboard-read", "clipboard-sanitized-write"]); + +function configureSessionPermissions(targetSession: Session): void { + targetSession.setPermissionCheckHandler((_webContents, permission) => { + return ALLOWED_SESSION_PERMISSIONS.has(permission); + }); + + targetSession.setPermissionRequestHandler((_webContents, permission, callback) => { + callback(ALLOWED_SESSION_PERMISSIONS.has(permission)); + }); + + // Auto-approve display media requests and route system audio as loopback. + // Electron requires a video source in the callback even if we only want audio. + // We pass the first available screen source; the renderer discards the video track. + targetSession.setDisplayMediaRequestHandler(async (_request, callback) => { + const sources = await desktopCapturer.getSources({ types: ['screen'] }); + if (sources.length === 0) { + callback({}); + return; + } + callback({ video: sources[0], audio: 'loopback' }); + }); +} + function createWindow() { const win = new BrowserWindow({ width: 1280, @@ -127,26 +160,8 @@ function createWindow() { }, }); - // Grant microphone and display-capture permissions - session.defaultSession.setPermissionRequestHandler((_webContents, permission, callback) => { - if (permission === 'media' || permission === 'display-capture') { - callback(true); - } else { - callback(false); - } - }); - - // Auto-approve display media requests and route system audio as loopback. - // Electron requires a video source in the callback even if we only want audio. - // We pass the first available screen source; the renderer discards the video track. - session.defaultSession.setDisplayMediaRequestHandler(async (_request, callback) => { - const sources = await desktopCapturer.getSources({ types: ['screen'] }); - if (sources.length === 0) { - callback({}); - return; - } - callback({ video: sources[0], audio: 'loopback' }); - }); + configureSessionPermissions(session.defaultSession); + configureSessionPermissions(session.fromPartition(BROWSER_PARTITION)); // Show window when content is ready to prevent blank screen win.once("ready-to-show", () => { @@ -171,6 +186,10 @@ function createWindow() { } }); + // Attach the embedded browser pane manager to this window. + // The WebContentsView is created lazily on first `browser:setVisible`. + browserViewManager.attach(win); + if (app.isPackaged) { win.loadURL("app://-/index.html"); } else { @@ -211,7 +230,10 @@ app.whenReady().then(async () => { // Initialize all config files before UI can access them await initConfigs(); + registerBrowserControlService(new ElectronBrowserControlService()); + setupIpcHandlers(); + setupBrowserEventForwarding(); createWindow(); @@ -228,6 +250,15 @@ app.whenReady().then(async () => { // start services watcher startServicesWatcher(); + // start tracks watcher + startTracksWatcher(); + + // start track scheduler (cron/window/once) + initTrackScheduler(); + + // start track event processor (consumes events/pending/, triggers matching tracks) + initTrackEventProcessor(); + // start gmail sync initGmailSync(); @@ -261,6 +292,11 @@ app.whenReady().then(async () => { // start chrome extension sync server initChromeSync(); + // start local sites server for iframe dashboards and other mini apps + initLocalSites().catch((error) => { + console.error('[LocalSites] Failed to start:', error); + }); + app.on("activate", () => { if (BrowserWindow.getAllWindows().length === 0) { createWindow(); @@ -279,4 +315,7 @@ app.on("before-quit", () => { stopWorkspaceWatcher(); stopRunsWatcher(); stopServicesWatcher(); + shutdownLocalSites().catch((error) => { + console.error('[LocalSites] Failed to shut down cleanly:', error); + }); }); diff --git a/apps/x/apps/main/src/oauth-handler.ts b/apps/x/apps/main/src/oauth-handler.ts index 483f25ee..3bb9063b 100644 --- a/apps/x/apps/main/src/oauth-handler.ts +++ b/apps/x/apps/main/src/oauth-handler.ts @@ -11,6 +11,7 @@ import { triggerSync as triggerGmailSync } from '@x/core/dist/knowledge/sync_gma import { triggerSync as triggerCalendarSync } from '@x/core/dist/knowledge/sync_calendar.js'; import { triggerSync as triggerFirefliesSync } from '@x/core/dist/knowledge/sync_fireflies.js'; import { emitOAuthEvent } from './ipc.js'; +import { getBillingInfo } from '@x/core/dist/billing/billing.js'; const REDIRECT_URI = 'http://localhost:8080/oauth/callback'; @@ -271,6 +272,17 @@ export async function connectProvider(provider: string, credentials?: { clientId triggerFirefliesSync(); } + // For Rowboat sign-in, ensure user + Stripe customer exist before + // notifying the renderer. Without this, parallel API calls from + // multiple renderer hooks race to create the user, causing duplicates. + if (provider === 'rowboat') { + try { + await getBillingInfo(); + } catch (meError) { + console.error('[OAuth] Failed to initialize user via /v1/me:', meError); + } + } + // Emit success event to renderer emitOAuthEvent({ provider, success: true }); } catch (error) { diff --git a/apps/x/apps/main/src/test-agent.ts b/apps/x/apps/main/src/test-agent.ts index 836deea7..738d861a 100644 --- a/apps/x/apps/main/src/test-agent.ts +++ b/apps/x/apps/main/src/test-agent.ts @@ -3,7 +3,7 @@ import { bus } from '@x/core/dist/runs/bus.js'; async function main() { const { id } = await runsCore.createRun({ - // this will expect an agent file to exist at ~/.rowboat/agents/test-agent.md + // this expects an agent file to exist at WorkDir/agents/test-agent.md agentId: 'test-agent', }); console.log(`created run: ${id}`); @@ -16,4 +16,4 @@ async function main() { console.log(`created message: ${msgId}`); } -main(); \ No newline at end of file +main(); diff --git a/apps/x/apps/preload/src/preload.ts b/apps/x/apps/preload/src/preload.ts index 7d7d53e4..bc69d4bb 100644 --- a/apps/x/apps/preload/src/preload.ts +++ b/apps/x/apps/preload/src/preload.ts @@ -1,4 +1,4 @@ -import { contextBridge, ipcRenderer, webUtils } from 'electron'; +import { contextBridge, ipcRenderer, webFrame, webUtils } from 'electron'; import { ipc as ipcShared } from '@x/shared'; type InvokeChannels = ipcShared.InvokeChannels; @@ -55,4 +55,5 @@ contextBridge.exposeInMainWorld('ipc', ipc); contextBridge.exposeInMainWorld('electronUtils', { getPathForFile: (file: File) => webUtils.getPathForFile(file), -}); \ No newline at end of file + getZoomFactor: () => webFrame.getZoomFactor(), +}); diff --git a/apps/x/apps/renderer/package.json b/apps/x/apps/renderer/package.json index ebf8a650..d9216de1 100644 --- a/apps/x/apps/renderer/package.json +++ b/apps/x/apps/renderer/package.json @@ -28,6 +28,7 @@ "@tiptap/extension-image": "^3.16.0", "@tiptap/extension-link": "^3.15.3", "@tiptap/extension-placeholder": "^3.15.3", + "@tiptap/extension-table": "^3.22.4", "@tiptap/extension-task-item": "^3.15.3", "@tiptap/extension-task-list": "^3.15.3", "@tiptap/pm": "^3.15.3", @@ -40,6 +41,7 @@ "clsx": "^2.1.1", "cmdk": "^1.1.1", "lucide-react": "^0.562.0", + "mermaid": "^11.14.0", "motion": "^12.23.26", "nanoid": "^5.1.6", "posthog-js": "^1.332.0", @@ -47,6 +49,7 @@ "react": "^19.2.0", "react-dom": "^19.2.0", "recharts": "^3.8.0", + "remark-breaks": "^4.0.0", "sonner": "^2.0.7", "streamdown": "^1.6.10", "tailwind-merge": "^3.4.0", @@ -54,6 +57,7 @@ "tiptap-markdown": "^0.9.0", "tokenlens": "^1.3.1", "use-stick-to-bottom": "^1.1.1", + "yaml": "^2.8.2", "zod": "^4.2.1" }, "devDependencies": { diff --git a/apps/x/apps/renderer/src/App.tsx b/apps/x/apps/renderer/src/App.tsx index 17e49f6e..67f3f06a 100644 --- a/apps/x/apps/renderer/src/App.tsx +++ b/apps/x/apps/renderer/src/App.tsx @@ -5,9 +5,9 @@ import { RunEvent, ListRunsResponse } from '@x/shared/src/runs.js'; import type { LanguageModelUsage, ToolUIPart } from 'ai'; import './App.css' import z from 'zod'; -import { CheckIcon, LoaderIcon, PanelLeftIcon, Maximize2, Minimize2, ChevronLeftIcon, ChevronRightIcon, SquarePen, SearchIcon, HistoryIcon, RadioIcon, SquareIcon } from 'lucide-react'; +import { CheckIcon, LoaderIcon, PanelLeftIcon, Maximize2, Minimize2, ChevronLeftIcon, ChevronRightIcon, SquarePen, HistoryIcon } from 'lucide-react'; import { cn } from '@/lib/utils'; -import { MarkdownEditor } from './components/markdown-editor'; +import { MarkdownEditor, type MarkdownEditorHandle } from './components/markdown-editor'; import { ChatSidebar } from './components/chat-sidebar'; import { ChatInputWithMentions, type StagedAttachment } from './components/chat-input-with-mentions'; import { ChatMessageAttachments } from '@/components/chat-message-attachments' @@ -15,6 +15,7 @@ import { GraphView, type GraphEdge, type GraphNode } from '@/components/graph-vi import { BasesView, type BaseConfig, DEFAULT_BASE_CONFIG } from '@/components/bases-view'; import { useDebounce } from './hooks/use-debounce'; import { SidebarContentPanel } from '@/components/sidebar-content'; +import { SuggestedTopicsView } from '@/components/suggested-topics-view'; import { SidebarSectionProvider } from '@/contexts/sidebar-context'; import { Conversation, @@ -54,11 +55,15 @@ import { Toaster } from "@/components/ui/sonner" import { stripKnowledgePrefix, toKnowledgePath, wikiLabel } from '@/lib/wiki-links' import { splitFrontmatter, joinFrontmatter } from '@/lib/frontmatter' import { OnboardingModal } from '@/components/onboarding' -import { SearchDialog } from '@/components/search-dialog' +import { CommandPalette, type CommandPaletteContext, type CommandPaletteMention } from '@/components/search-dialog' +import { TrackModal } from '@/components/track-modal' import { BackgroundTaskDetail } from '@/components/background-task-detail' +import { BrowserPane } from '@/components/browser-pane/BrowserPane' import { VersionHistoryPanel } from '@/components/version-history-panel' import { FileCardProvider } from '@/contexts/file-card-context' import { MarkdownPreOverride } from '@/components/ai-elements/markdown-code-override' +import { defaultRemarkPlugins } from 'streamdown' +import remarkBreaks from 'remark-breaks' import { TabBar, type ChatTab, type FileTab } from '@/components/tab-bar' import { type ChatMessage, @@ -86,7 +91,7 @@ import { AgentScheduleState } from '@x/shared/dist/agent-schedule-state.js' import { toast } from "sonner" import { useVoiceMode } from '@/hooks/useVoiceMode' import { useVoiceTTS } from '@/hooks/useVoiceTTS' -import { useMeetingTranscription, type MeetingTranscriptionState, type CalendarEventMeta } from '@/hooks/useMeetingTranscription' +import { useMeetingTranscription, type CalendarEventMeta } from '@/hooks/useMeetingTranscription' import { useAnalyticsIdentity } from '@/hooks/useAnalyticsIdentity' import * as analytics from '@/lib/analytics' @@ -101,6 +106,11 @@ interface TreeNode extends DirEntry { const streamdownComponents = { pre: MarkdownPreOverride } +// Render user messages with markdown so bullets, bold, links, etc. survive the +// round-trip from the input textarea. `remarkBreaks` turns single newlines +// into
so typed line breaks are preserved without requiring blank lines. +const userMessageRemarkPlugins = [...Object.values(defaultRemarkPlugins), remarkBreaks] + function SmoothStreamingMessage({ text, components }: { text: string; components: typeof streamdownComponents }) { const smoothText = useSmoothedText(text) return {smoothText} @@ -124,9 +134,10 @@ const TITLEBAR_BUTTON_PX = 32 const TITLEBAR_BUTTON_GAP_PX = 4 const TITLEBAR_HEADER_GAP_PX = 8 const TITLEBAR_TOGGLE_MARGIN_LEFT_PX = 12 -const TITLEBAR_BUTTONS_COLLAPSED = 4 -const TITLEBAR_BUTTON_GAPS_COLLAPSED = 3 +const TITLEBAR_BUTTONS_COLLAPSED = 1 +const TITLEBAR_BUTTON_GAPS_COLLAPSED = 0 const GRAPH_TAB_PATH = '__rowboat_graph_view__' +const SUGGESTED_TOPICS_TAB_PATH = '__rowboat_suggested_topics__' const BASES_DEFAULT_TAB_PATH = '__rowboat_bases_default__' const clampNumber = (value: number, min: number, max: number) => @@ -255,8 +266,63 @@ const getAncestorDirectoryPaths = (path: string): string[] => { } const isGraphTabPath = (path: string) => path === GRAPH_TAB_PATH +const isSuggestedTopicsTabPath = (path: string) => path === SUGGESTED_TOPICS_TAB_PATH const isBaseFilePath = (path: string) => path.endsWith('.base') || path === BASES_DEFAULT_TAB_PATH +const getSuggestedTopicTargetFolder = (category?: string) => { + const normalized = category?.trim().toLowerCase() + switch (normalized) { + case 'people': + case 'person': + return 'People' + case 'organizations': + case 'organization': + return 'Organizations' + case 'projects': + case 'project': + return 'Projects' + case 'meetings': + case 'meeting': + return 'Meetings' + case 'topics': + case 'topic': + default: + return 'Topics' + } +} + +const buildSuggestedTopicExplorePrompt = ({ + title, + description, + category, +}: { + title: string + description: string + category?: string +}) => { + const folder = getSuggestedTopicTargetFolder(category) + const categoryLabel = category?.trim() || 'Topics' + return [ + 'I am exploring a suggested topic card from the Suggested Topics panel.', + 'This card may represent a person, organization, topic, or project.', + '', + 'Card context:', + `- Title: ${title}`, + `- Category: ${categoryLabel}`, + `- Description: ${description}`, + `- Target folder if we set this up: knowledge/${folder}/`, + '', + `Please start by telling me that you can set up a tracking note for "${title}" under knowledge/${folder}/.`, + 'Then briefly explain what that tracking note would monitor or refresh and ask me if you should set it up.', + 'Do not create or modify anything yet.', + 'Treat a clear confirmation from me as explicit approval to proceed.', + `If I confirm later, load the \`tracks\` skill first, check whether a matching note already exists under knowledge/${folder}/, and update it instead of creating a duplicate.`, + `If no matching note exists, create a new note under knowledge/${folder}/ with an appropriate filename.`, + 'Use a track block in that note rather than only writing static content, and keep any surrounding note scaffolding short and useful.', + 'Do not ask me to choose a note path unless there is a real ambiguity you cannot resolve from the card.', + ].join('\n') +} + const normalizeUsage = (usage?: Partial | null): LanguageModelUsage | null => { if (!usage) return null const hasNumbers = Object.values(usage).some((value) => typeof value === 'number') @@ -437,6 +503,7 @@ type ViewState = | { type: 'file'; path: string } | { type: 'graph' } | { type: 'task'; name: string } + | { type: 'suggested-topics' } function viewStatesEqual(a: ViewState, b: ViewState): boolean { if (a.type !== b.type) return false @@ -446,22 +513,10 @@ function viewStatesEqual(a: ViewState, b: ViewState): boolean { return true // both graph } -/** Sidebar toggle + utility buttons (fixed position, top-left) */ +/** Sidebar toggle (fixed position, top-left) */ function FixedSidebarToggle({ - onNewChat, - onOpenSearch, - meetingState, - meetingSummarizing, - meetingAvailable, - onToggleMeeting, leftInsetPx, }: { - onNewChat: () => void - onOpenSearch: () => void - meetingState: MeetingTranscriptionState - meetingSummarizing: boolean - meetingAvailable: boolean - onToggleMeeting: () => void leftInsetPx: number }) { const { toggleSidebar } = useSidebar() @@ -478,55 +533,6 @@ function FixedSidebarToggle({ > - - - {meetingAvailable && ( - - - - - - {meetingSummarizing ? 'Generating meeting notes...' : meetingState === 'connecting' ? 'Starting transcription...' : meetingState === 'recording' ? 'Stop meeting notes' : 'Take new meeting notes'} - - - )} ) } @@ -605,7 +611,9 @@ function App() { const [expandedPaths, setExpandedPaths] = useState>(new Set()) const [recentWikiFiles, setRecentWikiFiles] = useState([]) const [isGraphOpen, setIsGraphOpen] = useState(false) - const [expandedFrom, setExpandedFrom] = useState<{ path: string | null; graph: boolean } | null>(null) + const [isBrowserOpen, setIsBrowserOpen] = useState(false) + const [isSuggestedTopicsOpen, setIsSuggestedTopicsOpen] = useState(false) + const [expandedFrom, setExpandedFrom] = useState<{ path: string | null; graph: boolean; suggestedTopics: boolean } | null>(null) const [baseConfigByPath, setBaseConfigByPath] = useState>({}) const [graphData, setGraphData] = useState<{ nodes: GraphNode[]; edges: GraphEdge[] }>({ nodes: [], @@ -739,6 +747,12 @@ function App() { const handlePromptSubmitRef = useRef<((message: PromptInputMessage, mentions?: FileMention[], stagedAttachments?: StagedAttachment[], searchEnabled?: boolean) => Promise) | null>(null) const pendingVoiceInputRef = useRef(false) + // Palette: per-tab editor handles for capturing cursor context on Cmd+K, and pending payload + // queued across the new-chat-tab state flush before submit fires. + const editorRefsByTabId = useRef>(new Map()) + const [paletteContext, setPaletteContext] = useState(null) + const [pendingPaletteSubmit, setPendingPaletteSubmit] = useState<{ text: string; mention: CommandPaletteMention | null } | null>(null) + const handleSubmitRecording = useCallback(() => { const text = voice.submit() setIsRecording(false) @@ -810,6 +824,7 @@ function App() { const chatTabIdCounterRef = useRef(0) const newChatTabId = () => `chat-tab-${++chatTabIdCounterRef.current}` const chatDraftsRef = useRef(new Map()) + const selectedModelByTabRef = useRef(new Map()) const chatScrollTopByTabRef = useRef(new Map()) const [toolOpenByTab, setToolOpenByTab] = useState>>({}) const [chatViewportAnchorByTab, setChatViewportAnchorByTab] = useState>({}) @@ -885,6 +900,8 @@ function App() { // File tab state const [fileTabs, setFileTabs] = useState([]) const [activeFileTabId, setActiveFileTabId] = useState(null) + const activeFileTabIdRef = useRef(activeFileTabId) + activeFileTabIdRef.current = activeFileTabId const [editorSessionByTabId, setEditorSessionByTabId] = useState>({}) const fileHistoryHandlersRef = useRef>(new Map()) const fileTabIdCounterRef = useRef(0) @@ -892,6 +909,7 @@ function App() { const getFileTabTitle = useCallback((tab: FileTab) => { if (isGraphTabPath(tab.path)) return 'Graph View' + if (isSuggestedTopicsTabPath(tab.path)) return 'Suggested Topics' if (tab.path === BASES_DEFAULT_TAB_PATH) return 'Bases' if (tab.path.endsWith('.base')) return tab.path.split('/').pop()?.replace(/\.base$/i, '') || 'Base' return tab.path.split('/').pop()?.replace(/\.md$/i, '') || tab.path @@ -2087,6 +2105,34 @@ function App() { return cleanup }, [handleRunEvent]) + type MiddlePaneContextPayload = + | { kind: 'note'; path: string; content: string } + | { kind: 'browser'; url: string; title: string } + const buildMiddlePaneContext = async (): Promise => { + // Nothing visible in the middle pane when the right pane is maximized. + if (isRightPaneMaximized) return undefined + + // Browser is an overlay on top of any note — when it's open, it's what the user is looking at. + if (isBrowserOpen) { + try { + const state = await window.ipc.invoke('browser:getState', null) + const activeTab = state.tabs.find((t) => t.id === state.activeTabId) + if (activeTab) { + return { kind: 'browser', url: activeTab.url, title: activeTab.title } + } + } catch { + // fall through to no-context if browser state is unavailable + } + return undefined + } + + // Note case: only markdown files are meaningfully readable as context. + const path = selectedPathRef.current + if (!path || !path.endsWith('.md')) return undefined + const content = editorContentRef.current ?? '' + return { kind: 'note', path, content } + } + const handlePromptSubmit = async ( message: PromptInputMessage, mentions?: FileMention[], @@ -2127,8 +2173,10 @@ function App() { let isNewRun = false let newRunCreatedAt: string | null = null if (!currentRunId) { + const selected = selectedModelByTabRef.current.get(submitTabId) const run = await window.ipc.invoke('runs:create', { agentId, + ...(selected ? { model: selected.model, provider: selected.provider } : {}), }) currentRunId = run.id newRunCreatedAt = run.createdAt @@ -2144,8 +2192,9 @@ function App() { } let titleSource = userMessage + const hasMentions = (mentions?.length ?? 0) > 0 - if (hasAttachments) { + if (hasAttachments || hasMentions) { type ContentPart = | { type: 'text'; text: string } | { @@ -2154,6 +2203,7 @@ function App() { filename: string mimeType: string size?: number + lineNumber?: number } const contentParts: ContentPart[] = [] @@ -2165,6 +2215,7 @@ function App() { path: mention.path, filename: mention.displayName || mention.path.split('/').pop() || mention.path, mimeType: 'text/markdown', + ...(mention.lineNumber !== undefined ? { lineNumber: mention.lineNumber } : {}), }) } } @@ -2182,17 +2233,19 @@ function App() { if (userMessage) { contentParts.push({ type: 'text', text: userMessage }) } else { - titleSource = stagedAttachments[0]?.filename ?? '' + titleSource = stagedAttachments[0]?.filename ?? mentions?.[0]?.displayName ?? mentions?.[0]?.path ?? '' } // Shared IPC payload types can lag until package rebuilds; runtime validation still enforces schema. const attachmentPayload = contentParts as unknown as string + const middlePaneContext = await buildMiddlePaneContext() await window.ipc.invoke('runs:createMessage', { runId: currentRunId, message: attachmentPayload, voiceInput: pendingVoiceInputRef.current || undefined, voiceOutput: ttsEnabledRef.current ? ttsModeRef.current : undefined, searchEnabled: searchEnabled || undefined, + middlePaneContext, }) analytics.chatMessageSent({ voiceInput: pendingVoiceInputRef.current || undefined, @@ -2200,43 +2253,20 @@ function App() { searchEnabled: searchEnabled || undefined, }) } else { - // Legacy path: plain string with optional XML-formatted @mentions. - let formattedMessage = userMessage - if (mentions && mentions.length > 0) { - const attachedFiles = await Promise.all( - mentions.map(async (mention) => { - try { - const result = await window.ipc.invoke('workspace:readFile', { path: mention.path }) - return { path: mention.path, content: result.data as string } - } catch (err) { - console.error('Failed to read mentioned file:', mention.path, err) - return { path: mention.path, content: `[Error reading file: ${mention.path}]` } - } - }) - ) - - if (attachedFiles.length > 0) { - const filesXml = attachedFiles - .map((file) => `\n${file.content}\n`) - .join('\n') - formattedMessage = `\n${filesXml}\n\n\n${userMessage}` - } - } - + const middlePaneContext = await buildMiddlePaneContext() await window.ipc.invoke('runs:createMessage', { runId: currentRunId, - message: formattedMessage, + message: userMessage, voiceInput: pendingVoiceInputRef.current || undefined, voiceOutput: ttsEnabledRef.current ? ttsModeRef.current : undefined, searchEnabled: searchEnabled || undefined, + middlePaneContext, }) analytics.chatMessageSent({ voiceInput: pendingVoiceInputRef.current || undefined, voiceOutput: ttsEnabledRef.current ? ttsModeRef.current : undefined, searchEnabled: searchEnabled || undefined, }) - - titleSource = formattedMessage } pendingVoiceInputRef.current = false @@ -2451,6 +2481,7 @@ function App() { return next }) chatDraftsRef.current.delete(tabId) + selectedModelByTabRef.current.delete(tabId) chatScrollTopByTabRef.current.delete(tabId) setToolOpenByTab((prev) => { if (!(tabId in prev)) return prev @@ -2577,9 +2608,17 @@ function App() { if (isGraphTabPath(tab.path)) { setSelectedPath(null) setIsGraphOpen(true) + setIsSuggestedTopicsOpen(false) + return + } + if (isSuggestedTopicsTabPath(tab.path)) { + setSelectedPath(null) + setIsGraphOpen(false) + setIsSuggestedTopicsOpen(true) return } setIsGraphOpen(false) + setIsSuggestedTopicsOpen(false) setSelectedPath(tab.path) }, [fileTabs, isRightPaneMaximized]) @@ -2607,6 +2646,7 @@ function App() { setActiveFileTabId(null) setSelectedPath(null) setIsGraphOpen(false) + setIsSuggestedTopicsOpen(false) return [] } const idx = prev.findIndex(t => t.id === tabId) @@ -2619,8 +2659,14 @@ function App() { if (isGraphTabPath(newActiveTab.path)) { setSelectedPath(null) setIsGraphOpen(true) + setIsSuggestedTopicsOpen(false) + } else if (isSuggestedTopicsTabPath(newActiveTab.path)) { + setSelectedPath(null) + setIsGraphOpen(false) + setIsSuggestedTopicsOpen(true) } else { setIsGraphOpen(false) + setIsSuggestedTopicsOpen(false) setSelectedPath(newActiveTab.path) } } @@ -2650,15 +2696,16 @@ function App() { } handleNewChat() // Left-pane "new chat" should always open full chat view. - if (selectedPath || isGraphOpen) { - setExpandedFrom({ path: selectedPath, graph: isGraphOpen }) + if (selectedPath || isGraphOpen || isSuggestedTopicsOpen) { + setExpandedFrom({ path: selectedPath, graph: isGraphOpen, suggestedTopics: isSuggestedTopicsOpen }) } else { setExpandedFrom(null) } setIsRightPaneMaximized(false) setSelectedPath(null) setIsGraphOpen(false) - }, [chatTabs, activeChatTabId, handleNewChat, selectedPath, isGraphOpen]) + setIsSuggestedTopicsOpen(false) + }, [chatTabs, activeChatTabId, handleNewChat, selectedPath, isGraphOpen, isSuggestedTopicsOpen]) // Sidebar variant: create/switch chat tab without leaving file/graph context. const handleNewChatTabInSidebar = useCallback(() => { @@ -2675,11 +2722,97 @@ function App() { handleNewChat() }, [chatTabs, activeChatTabId, handleNewChat]) + // Palette → sidebar submission. Opens the sidebar (if closed), forces a fresh chat tab, + // queues the message; the pending-submit effect (below) flushes it once state has settled + // so handlePromptSubmit sees the new tab's null runId. + const submitFromPalette = useCallback((text: string, mention: CommandPaletteMention | null) => { + if (!isChatSidebarOpen) setIsChatSidebarOpen(true) + handleNewChatTabInSidebar() + setPendingPaletteSubmit({ text, mention }) + }, [isChatSidebarOpen, handleNewChatTabInSidebar]) + + useEffect(() => { + if (!pendingPaletteSubmit) return + const fileMention: FileMention | undefined = pendingPaletteSubmit.mention + ? { + id: `palette-${Date.now()}`, + path: pendingPaletteSubmit.mention.path, + displayName: pendingPaletteSubmit.mention.displayName, + lineNumber: pendingPaletteSubmit.mention.lineNumber, + } + : undefined + void handlePromptSubmitRef.current?.( + { text: pendingPaletteSubmit.text, files: [] }, + fileMention ? [fileMention] : undefined, + ) + setPendingPaletteSubmit(null) + }, [pendingPaletteSubmit]) + + // Listener for track-block "Edit with Copilot" events + // (dispatched by apps/renderer/src/extensions/track-block.tsx) + useEffect(() => { + const handler = (e: Event) => { + const ev = e as CustomEvent<{ + trackId?: string + filePath?: string + }> + const trackId = ev.detail?.trackId + const filePath = ev.detail?.filePath + if (!trackId || !filePath) return + const displayName = filePath.split('/').pop() ?? filePath + submitFromPalette( + `Let's work on the \`${trackId}\` track in this note. Please load the \`tracks\` skill first, then ask me what I want to change.`, + { path: filePath, displayName }, + ) + } + window.addEventListener('rowboat:open-copilot-edit-track', handler as EventListener) + return () => window.removeEventListener('rowboat:open-copilot-edit-track', handler as EventListener) + }, [submitFromPalette]) + + // Listener for prompt-block "Run" events + // (dispatched by apps/renderer/src/extensions/prompt-block.tsx) + useEffect(() => { + const handler = (e: Event) => { + const ev = e as CustomEvent<{ + instruction?: string + filePath?: string + label?: string + }> + const instruction = ev.detail?.instruction + const filePath = ev.detail?.filePath + if (!instruction) return + const mention = filePath + ? { path: filePath, displayName: filePath.split('/').pop() ?? filePath } + : null + submitFromPalette(instruction, mention) + } + window.addEventListener('rowboat:open-copilot-prompt', handler as EventListener) + return () => window.removeEventListener('rowboat:open-copilot-prompt', handler as EventListener) + }, [submitFromPalette]) + const toggleKnowledgePane = useCallback(() => { setIsRightPaneMaximized(false) setIsChatSidebarOpen(prev => !prev) }, []) + // Browser is an overlay on the middle pane: opening it forces the chat + // sidebar to be visible on the right; closing it restores whatever the + // middle pane was showing previously (file/graph/task/chat). + const handleToggleBrowser = useCallback(() => { + setIsBrowserOpen(prev => { + const next = !prev + if (next) { + setIsChatSidebarOpen(true) + setIsRightPaneMaximized(false) + } + return next + }) + }, []) + + const handleCloseBrowser = useCallback(() => { + setIsBrowserOpen(false) + }, []) + const toggleRightPaneMaximize = useCallback(() => { setIsChatSidebarOpen(true) setIsRightPaneMaximized(prev => !prev) @@ -2687,19 +2820,26 @@ function App() { const handleOpenFullScreenChat = useCallback(() => { // Remember where we came from so the close button can return - if (selectedPath || isGraphOpen) { - setExpandedFrom({ path: selectedPath, graph: isGraphOpen }) + if (selectedPath || isGraphOpen || isSuggestedTopicsOpen) { + setExpandedFrom({ path: selectedPath, graph: isGraphOpen, suggestedTopics: isSuggestedTopicsOpen }) } setIsRightPaneMaximized(false) setSelectedPath(null) setIsGraphOpen(false) - }, [selectedPath, isGraphOpen]) + setIsSuggestedTopicsOpen(false) + }, [selectedPath, isGraphOpen, isSuggestedTopicsOpen]) const handleCloseFullScreenChat = useCallback(() => { if (expandedFrom) { if (expandedFrom.graph) { setIsGraphOpen(true) + setIsSuggestedTopicsOpen(false) + } else if (expandedFrom.suggestedTopics) { + setIsGraphOpen(false) + setIsSuggestedTopicsOpen(true) } else if (expandedFrom.path) { + setIsGraphOpen(false) + setIsSuggestedTopicsOpen(false) setSelectedPath(expandedFrom.path) } setExpandedFrom(null) @@ -2709,10 +2849,11 @@ function App() { const currentViewState = React.useMemo(() => { if (selectedBackgroundTask) return { type: 'task', name: selectedBackgroundTask } + if (isSuggestedTopicsOpen) return { type: 'suggested-topics' } if (selectedPath) return { type: 'file', path: selectedPath } if (isGraphOpen) return { type: 'graph' } return { type: 'chat', runId } - }, [selectedBackgroundTask, selectedPath, isGraphOpen, runId]) + }, [selectedBackgroundTask, isSuggestedTopicsOpen, selectedPath, isGraphOpen, runId]) const appendUnique = useCallback((stack: ViewState[], entry: ViewState) => { const last = stack[stack.length - 1] @@ -2758,11 +2899,26 @@ function App() { setActiveFileTabId(id) }, [fileTabs]) + const ensureSuggestedTopicsFileTab = useCallback(() => { + const existing = fileTabs.find((tab) => isSuggestedTopicsTabPath(tab.path)) + if (existing) { + setActiveFileTabId(existing.id) + return + } + const id = newFileTabId() + setFileTabs((prev) => [...prev, { id, path: SUGGESTED_TOPICS_TAB_PATH }]) + setActiveFileTabId(id) + }, [fileTabs]) + const applyViewState = useCallback(async (view: ViewState) => { switch (view.type) { case 'file': setSelectedBackgroundTask(null) setIsGraphOpen(false) + // Navigating to a file dismisses the browser overlay so the file is + // visible in the middle pane. + setIsBrowserOpen(false) + setIsSuggestedTopicsOpen(false) setExpandedFrom(null) // Preserve split vs knowledge-max mode when navigating knowledge files. // Only exit chat-only maximize, because that would hide the selected file. @@ -2775,6 +2931,8 @@ function App() { case 'graph': setSelectedBackgroundTask(null) setSelectedPath(null) + setIsBrowserOpen(false) + setIsSuggestedTopicsOpen(false) setExpandedFrom(null) setIsGraphOpen(true) ensureGraphFileTab() @@ -2785,16 +2943,31 @@ function App() { case 'task': setSelectedPath(null) setIsGraphOpen(false) + setIsBrowserOpen(false) + setIsSuggestedTopicsOpen(false) setExpandedFrom(null) setIsRightPaneMaximized(false) setSelectedBackgroundTask(view.name) return - case 'chat': + case 'suggested-topics': setSelectedPath(null) setIsGraphOpen(false) + setIsBrowserOpen(false) setExpandedFrom(null) setIsRightPaneMaximized(false) setSelectedBackgroundTask(null) + setIsSuggestedTopicsOpen(true) + ensureSuggestedTopicsFileTab() + return + case 'chat': + setSelectedPath(null) + setIsGraphOpen(false) + // Don't touch isBrowserOpen here — chat navigation should land in + // the right sidebar when the browser overlay is active. + setExpandedFrom(null) + setIsRightPaneMaximized(false) + setSelectedBackgroundTask(null) + setIsSuggestedTopicsOpen(false) if (view.runId) { await loadRun(view.runId) } else { @@ -2802,7 +2975,7 @@ function App() { } return } - }, [ensureFileTabForPath, ensureGraphFileTab, handleNewChat, isRightPaneMaximized, loadRun]) + }, [ensureFileTabForPath, ensureGraphFileTab, ensureSuggestedTopicsFileTab, handleNewChat, isRightPaneMaximized, loadRun]) const navigateToView = useCallback(async (nextView: ViewState) => { const current = currentViewState @@ -3067,7 +3240,7 @@ function App() { }, []) // Keyboard shortcut: Ctrl+L to toggle main chat view - const isFullScreenChat = !selectedPath && !isGraphOpen && !selectedBackgroundTask + const isFullScreenChat = !selectedPath && !isGraphOpen && !isSuggestedTopicsOpen && !selectedBackgroundTask && !isBrowserOpen useEffect(() => { const handleKeyDown = (e: KeyboardEvent) => { if ((e.ctrlKey || e.metaKey) && e.key === 'l') { @@ -3083,11 +3256,16 @@ function App() { return () => document.removeEventListener('keydown', handleKeyDown) }, [handleCloseFullScreenChat, isFullScreenChat, expandedFrom, navigateToFullScreenChat]) - // Keyboard shortcut: Cmd+K / Ctrl+K to open search + // Keyboard shortcut: Cmd+K / Ctrl+K opens the unified palette (defaults to Chat mode). + // If an editor tab is currently active, capture cursor context so Chat mode shows the + // note + line as a removable chip. useEffect(() => { const handleKeyDown = (e: KeyboardEvent) => { if ((e.ctrlKey || e.metaKey) && e.key === 'k') { e.preventDefault() + const activeId = activeFileTabIdRef.current + const handle = activeId ? editorRefsByTabId.current.get(activeId) : null + setPaletteContext(handle?.getCursorContext() ?? null) setIsSearchOpen(true) } } @@ -3140,12 +3318,16 @@ function App() { const handleTabKeyDown = (e: KeyboardEvent) => { const mod = e.metaKey || e.ctrlKey if (!mod) return - const rightPaneAvailable = Boolean((selectedPath || isGraphOpen) && isChatSidebarOpen) + const rightPaneAvailable = Boolean((selectedPath || isGraphOpen || isSuggestedTopicsOpen) && isChatSidebarOpen) const targetPane: ShortcutPane = rightPaneAvailable ? (isRightPaneMaximized ? 'right' : activeShortcutPane) : 'left' - const inFileView = targetPane === 'left' && Boolean(selectedPath || isGraphOpen) - const selectedKnowledgePath = isGraphOpen ? GRAPH_TAB_PATH : selectedPath + const inFileView = targetPane === 'left' && Boolean(selectedPath || isGraphOpen || isSuggestedTopicsOpen) + const selectedKnowledgePath = isGraphOpen + ? GRAPH_TAB_PATH + : isSuggestedTopicsOpen + ? SUGGESTED_TOPICS_TAB_PATH + : selectedPath const targetFileTabId = activeFileTabId ?? ( selectedKnowledgePath ? (fileTabs.find((tab) => tab.path === selectedKnowledgePath)?.id ?? null) @@ -3199,7 +3381,7 @@ function App() { } document.addEventListener('keydown', handleTabKeyDown) return () => document.removeEventListener('keydown', handleTabKeyDown) - }, [selectedPath, isGraphOpen, isChatSidebarOpen, isRightPaneMaximized, activeShortcutPane, chatTabs, fileTabs, activeChatTabId, activeFileTabId, closeChatTab, closeFileTab, switchChatTab, switchFileTab]) + }, [selectedPath, isGraphOpen, isSuggestedTopicsOpen, isChatSidebarOpen, isRightPaneMaximized, activeShortcutPane, chatTabs, fileTabs, activeChatTabId, activeFileTabId, closeChatTab, closeFileTab, switchChatTab, switchFileTab]) const toggleExpand = (path: string, kind: 'file' | 'dir') => { if (kind === 'file') { @@ -3207,9 +3389,9 @@ function App() { return } - // Top-level knowledge folders (except Notes) open as a bases view with folder filter + // Top-level knowledge folders open as a bases view with folder filter const parts = path.split('/') - if (parts.length === 2 && parts[0] === 'knowledge' && parts[1] !== 'Notes') { + if (parts.length === 2 && parts[0] === 'knowledge') { const folderName = parts[1] const folderCfg = FOLDER_BASE_CONFIGS[folderName] setBaseConfigByPath((prev) => ({ @@ -3224,7 +3406,7 @@ function App() { }), }, })) - if (!selectedPath && !isGraphOpen && !selectedBackgroundTask) { + if (!selectedPath && !isGraphOpen && !isSuggestedTopicsOpen && !selectedBackgroundTask) { setIsChatSidebarOpen(false) setIsRightPaneMaximized(false) } @@ -3346,14 +3528,14 @@ function App() { }, openGraph: () => { // From chat-only landing state, open graph directly in full knowledge view. - if (!selectedPath && !isGraphOpen && !selectedBackgroundTask) { + if (!selectedPath && !isGraphOpen && !isSuggestedTopicsOpen && !selectedBackgroundTask) { setIsChatSidebarOpen(false) setIsRightPaneMaximized(false) } void navigateToView({ type: 'graph' }) }, openBases: () => { - if (!selectedPath && !isGraphOpen && !selectedBackgroundTask) { + if (!selectedPath && !isGraphOpen && !isSuggestedTopicsOpen && !selectedBackgroundTask) { setIsChatSidebarOpen(false) setIsRightPaneMaximized(false) } @@ -3799,7 +3981,14 @@ function App() { {item.content && ( - {item.content} + + + {item.content} + + )} ) @@ -3820,7 +4009,12 @@ function App() { ))} )} - {message} + + {message} + ) @@ -3925,7 +4119,7 @@ function App() { const selectedTask = selectedBackgroundTask ? backgroundTasks.find(t => t.name === selectedBackgroundTask) : null - const isRightPaneContext = Boolean(selectedPath || isGraphOpen) + const isRightPaneContext = Boolean(selectedPath || isGraphOpen || isSuggestedTopicsOpen || isBrowserOpen) const isRightPaneOnlyMode = isRightPaneContext && isChatSidebarOpen && isRightPaneMaximized const shouldCollapseLeftPane = isRightPaneOnlyMode const openMarkdownTabs = React.useMemo(() => { @@ -3958,6 +4152,14 @@ function App() { selectedPath={selectedPath} expandedPaths={expandedPaths} onSelectFile={toggleExpand} + onToggleFolder={(path) => { + setExpandedPaths((prev) => { + const next = new Set(prev) + if (next.has(path)) next.delete(path) + else next.add(path) + return next + }) + }} knowledgeActions={knowledgeActions} onVoiceNoteCreated={handleVoiceNoteCreated} runs={runs} @@ -3967,7 +4169,7 @@ function App() { onNewChat: handleNewChatTab, onSelectRun: (runIdToLoad) => { cancelRecordingIfActive() - if (selectedPath || isGraphOpen) { + if (selectedPath || isGraphOpen || isSuggestedTopicsOpen || isBrowserOpen) { setIsChatSidebarOpen(true) } @@ -3977,8 +4179,8 @@ function App() { switchChatTab(existingTab.id) return } - // In two-pane mode, keep current knowledge/graph context and just swap chat context. - if (selectedPath || isGraphOpen) { + // In two-pane mode (file/graph/browser), keep the middle pane and just swap chat context in the right sidebar. + if (selectedPath || isGraphOpen || isSuggestedTopicsOpen || isBrowserOpen) { setChatTabs(prev => prev.map(t => t.id === activeChatTabId ? { ...t, runId: runIdToLoad } : t)) loadRun(runIdToLoad) return @@ -4002,14 +4204,14 @@ function App() { } else { // Only one tab, reset it to new chat setChatTabs([{ id: tabForRun.id, runId: null }]) - if (selectedPath || isGraphOpen) { + if (selectedPath || isGraphOpen || isSuggestedTopicsOpen || isBrowserOpen) { handleNewChat() } else { void navigateToView({ type: 'chat', runId: null }) } } } else if (runId === runIdToDelete) { - if (selectedPath || isGraphOpen) { + if (selectedPath || isGraphOpen || isSuggestedTopicsOpen || isBrowserOpen) { setChatTabs(prev => prev.map(t => t.id === activeChatTabId ? { ...t, runId: null } : t)) handleNewChat() } else { @@ -4027,6 +4229,16 @@ function App() { }} backgroundTasks={backgroundTasks} selectedBackgroundTask={selectedBackgroundTask} + onNewChat={handleNewChatTab} + onOpenSearch={() => setIsSearchOpen(true)} + meetingState={meetingTranscription.state} + meetingSummarizing={meetingSummarizing} + meetingAvailable={voiceAvailable} + onToggleMeeting={() => { void handleToggleMeeting() }} + isBrowserOpen={isBrowserOpen} + onToggleBrowser={handleToggleBrowser} + isSuggestedTopicsOpen={isSuggestedTopicsOpen} + onOpenSuggestedTopics={() => void navigateToView({ type: 'suggested-topics' })} /> - {(selectedPath || isGraphOpen) && fileTabs.length >= 1 ? ( + {(selectedPath || isGraphOpen || isSuggestedTopicsOpen) && fileTabs.length >= 1 ? ( t.id} onSwitchTab={switchFileTab} onCloseTab={closeFileTab} - allowSingleTabClose={fileTabs.length === 1 && (isGraphOpen || (selectedPath != null && isBaseFilePath(selectedPath)))} + allowSingleTabClose={fileTabs.length === 1 && (isGraphOpen || isSuggestedTopicsOpen || (selectedPath != null && isBaseFilePath(selectedPath)))} /> ) : ( Version history )} - {!selectedPath && !isGraphOpen && !selectedTask && ( + {!selectedPath && !isGraphOpen && !isSuggestedTopicsOpen && !selectedTask && !isBrowserOpen && ( + + +
+ + + +
+ setAddressValue(e.target.value)} + onFocus={(e) => { + addressFocusedRef.current = true + e.currentTarget.select() + }} + onBlur={() => { + addressFocusedRef.current = false + setAddressValue(activeTab?.url ?? '') + }} + placeholder="Enter URL or search..." + className={cn( + 'h-7 w-full rounded-md border border-transparent bg-background px-3 text-sm text-foreground', + 'placeholder:text-muted-foreground/60', + 'focus:border-border focus:outline-hidden', + )} + spellCheck={false} + autoCorrect="off" + autoCapitalize="off" + /> +
+ +
+ +
+
+ ) +} diff --git a/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx b/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx index 37d8d053..e1fb950f 100644 --- a/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx +++ b/apps/x/apps/renderer/src/components/chat-input-with-mentions.tsx @@ -69,13 +69,20 @@ const providerDisplayNames: Record = { rowboat: 'Rowboat', } +type ProviderName = "openai" | "anthropic" | "google" | "openrouter" | "aigateway" | "ollama" | "openai-compatible" | "rowboat" + interface ConfiguredModel { - flavor: "openai" | "anthropic" | "google" | "openrouter" | "aigateway" | "ollama" | "openai-compatible" | "rowboat" + provider: ProviderName model: string - apiKey?: string - baseURL?: string - headers?: Record - knowledgeGraphModel?: string +} + +export interface SelectedModel { + provider: string + model: string +} + +function getSelectedModelDisplayName(model: string) { + return model.split('/').pop() || model } function getAttachmentIcon(kind: AttachmentIconKind) { @@ -120,6 +127,8 @@ interface ChatInputInnerProps { ttsMode?: 'summary' | 'full' onToggleTts?: () => void onTtsModeChange?: (mode: 'summary' | 'full') => void + /** Fired when the user picks a different model in the dropdown (only when no run exists yet). */ + onSelectedModelChange?: (model: SelectedModel | null) => void } function ChatInputInner({ @@ -145,6 +154,7 @@ function ChatInputInner({ ttsMode, onToggleTts, onTtsModeChange, + onSelectedModelChange, }: ChatInputInnerProps) { const controller = usePromptInputController() const message = controller.textInput.value @@ -155,10 +165,27 @@ function ChatInputInner({ const [configuredModels, setConfiguredModels] = useState([]) const [activeModelKey, setActiveModelKey] = useState('') + const [lockedModel, setLockedModel] = useState(null) const [searchEnabled, setSearchEnabled] = useState(false) const [searchAvailable, setSearchAvailable] = useState(false) const [isRowboatConnected, setIsRowboatConnected] = useState(false) + // When a run exists, freeze the dropdown to the run's resolved model+provider. + useEffect(() => { + if (!runId) { + setLockedModel(null) + return + } + let cancelled = false + window.ipc.invoke('runs:fetch', { runId }).then((run) => { + if (cancelled) return + if (run.provider && run.model) { + setLockedModel({ provider: run.provider, model: run.model }) + } + }).catch(() => { /* legacy run or fetch failure — leave unlocked */ }) + return () => { cancelled = true } + }, [runId]) + // Check Rowboat sign-in state useEffect(() => { window.ipc.invoke('oauth:getState', null).then((result) => { @@ -176,42 +203,20 @@ function ChatInputInner({ return cleanup }, []) - // Load model config (gateway when signed in, local config when BYOK) + // Load the list of models the user can choose from. + // Signed-in: gateway model list. Signed-out: providers configured in models.json. const loadModelConfig = useCallback(async () => { try { if (isRowboatConnected) { - // Fetch gateway models const listResult = await window.ipc.invoke('models:list', null) const rowboatProvider = listResult.providers?.find( (p: { id: string }) => p.id === 'rowboat' ) const models: ConfiguredModel[] = (rowboatProvider?.models || []).map( - (m: { id: string }) => ({ flavor: 'rowboat', model: m.id }) + (m: { id: string }) => ({ provider: 'rowboat', model: m.id }) ) - - // Read current default from config - let defaultModel = '' - try { - const result = await window.ipc.invoke('workspace:readFile', { path: 'config/models.json' }) - const parsed = JSON.parse(result.data) - defaultModel = parsed?.model || '' - } catch { /* no config yet */ } - - if (defaultModel) { - models.sort((a, b) => { - if (a.model === defaultModel) return -1 - if (b.model === defaultModel) return 1 - return 0 - }) - } - setConfiguredModels(models) - const activeKey = defaultModel - ? `rowboat/${defaultModel}` - : models[0] ? `rowboat/${models[0].model}` : '' - if (activeKey) setActiveModelKey(activeKey) } else { - // BYOK: read from local models.json const result = await window.ipc.invoke('workspace:readFile', { path: 'config/models.json' }) const parsed = JSON.parse(result.data) const models: ConfiguredModel[] = [] @@ -223,32 +228,12 @@ function ChatInputInner({ const allModels = modelList.length > 0 ? modelList : singleModel ? [singleModel] : [] for (const model of allModels) { if (model) { - models.push({ - flavor: flavor as ConfiguredModel['flavor'], - model, - apiKey: (e.apiKey as string) || undefined, - baseURL: (e.baseURL as string) || undefined, - headers: (e.headers as Record) || undefined, - knowledgeGraphModel: (e.knowledgeGraphModel as string) || undefined, - }) + models.push({ provider: flavor as ProviderName, model }) } } } } - const defaultKey = parsed?.provider?.flavor && parsed?.model - ? `${parsed.provider.flavor}/${parsed.model}` - : '' - models.sort((a, b) => { - const aKey = `${a.flavor}/${a.model}` - const bKey = `${b.flavor}/${b.model}` - if (aKey === defaultKey) return -1 - if (bKey === defaultKey) return 1 - return 0 - }) setConfiguredModels(models) - if (defaultKey) { - setActiveModelKey(defaultKey) - } } } catch { // No config yet @@ -284,40 +269,15 @@ function ChatInputInner({ checkSearch() }, [isActive, isRowboatConnected]) - const handleModelChange = useCallback(async (key: string) => { - const entry = configuredModels.find((m) => `${m.flavor}/${m.model}` === key) + // Selecting a model affects only the *next* run created from this tab. + // Once a run exists, model is frozen on the run and the dropdown is read-only. + const handleModelChange = useCallback((key: string) => { + if (lockedModel) return + const entry = configuredModels.find((m) => `${m.provider}/${m.model}` === key) if (!entry) return setActiveModelKey(key) - - try { - if (entry.flavor === 'rowboat') { - // Gateway model — save with valid Zod flavor, no credentials - await window.ipc.invoke('models:saveConfig', { - provider: { flavor: 'openrouter' as const }, - model: entry.model, - knowledgeGraphModel: entry.knowledgeGraphModel, - }) - } else { - // BYOK — preserve full provider config - const providerModels = configuredModels - .filter((m) => m.flavor === entry.flavor) - .map((m) => m.model) - await window.ipc.invoke('models:saveConfig', { - provider: { - flavor: entry.flavor, - apiKey: entry.apiKey, - baseURL: entry.baseURL, - headers: entry.headers, - }, - model: entry.model, - models: providerModels, - knowledgeGraphModel: entry.knowledgeGraphModel, - }) - } - } catch { - toast.error('Failed to switch model') - } - }, [configuredModels]) + onSelectedModelChange?.({ provider: entry.provider, model: entry.model }) + }, [configuredModels, lockedModel, onSelectedModelChange]) // Restore the tab draft when this input mounts. useEffect(() => { @@ -555,7 +515,14 @@ function ChatInputInner({ ) )}
- {configuredModels.length > 0 && ( + {lockedModel ? ( + + {getSelectedModelDisplayName(lockedModel.model)} + + ) : configuredModels.length > 0 ? ( @@ -571,18 +538,18 @@ function ChatInputInner({ {configuredModels.map((m) => { - const key = `${m.flavor}/${m.model}` + const key = `${m.provider}/${m.model}` return ( {m.model} - {providerDisplayNames[m.flavor] || m.flavor} + {providerDisplayNames[m.provider] || m.provider} ) })} - )} + ) : null} {onToggleTts && ttsAvailable && (
@@ -729,6 +696,7 @@ export interface ChatInputWithMentionsProps { ttsMode?: 'summary' | 'full' onToggleTts?: () => void onTtsModeChange?: (mode: 'summary' | 'full') => void + onSelectedModelChange?: (model: SelectedModel | null) => void } export function ChatInputWithMentions({ @@ -757,6 +725,7 @@ export function ChatInputWithMentions({ ttsMode, onToggleTts, onTtsModeChange, + onSelectedModelChange, }: ChatInputWithMentionsProps) { return ( @@ -783,6 +752,7 @@ export function ChatInputWithMentions({ ttsMode={ttsMode} onToggleTts={onToggleTts} onTtsModeChange={onTtsModeChange} + onSelectedModelChange={onSelectedModelChange} /> ) diff --git a/apps/x/apps/renderer/src/components/chat-sidebar.tsx b/apps/x/apps/renderer/src/components/chat-sidebar.tsx index f94c94ba..0a407d5d 100644 --- a/apps/x/apps/renderer/src/components/chat-sidebar.tsx +++ b/apps/x/apps/renderer/src/components/chat-sidebar.tsx @@ -25,8 +25,10 @@ import { Suggestions } from '@/components/ai-elements/suggestions' import { type PromptInputMessage, type FileMention } from '@/components/ai-elements/prompt-input' import { FileCardProvider } from '@/contexts/file-card-context' import { MarkdownPreOverride } from '@/components/ai-elements/markdown-code-override' +import { defaultRemarkPlugins } from 'streamdown' +import remarkBreaks from 'remark-breaks' import { TabBar, type ChatTab } from '@/components/tab-bar' -import { ChatInputWithMentions, type StagedAttachment } from '@/components/chat-input-with-mentions' +import { ChatInputWithMentions, type StagedAttachment, type SelectedModel } from '@/components/chat-input-with-mentions' import { ChatMessageAttachments } from '@/components/chat-message-attachments' import { wikiLabel } from '@/lib/wiki-links' import { @@ -49,6 +51,11 @@ import { const streamdownComponents = { pre: MarkdownPreOverride } +// Render user messages with markdown so bullets, bold, links, etc. survive the +// round-trip from the input textarea. `remarkBreaks` turns single newlines +// into
so typed line breaks are preserved without requiring blank lines. +const userMessageRemarkPlugins = [...Object.values(defaultRemarkPlugins), remarkBreaks] + /* ─── Billing error helpers ─── */ const BILLING_ERROR_PATTERNS = [ @@ -76,12 +83,18 @@ function matchBillingError(message: string) { return BILLING_ERROR_PATTERNS.find(({ pattern }) => pattern.test(message)) ?? null } +interface BillingRowboatAccount { + config?: { + appUrl?: string | null + } | null +} + function BillingErrorCTA({ label }: { label: string }) { const [appUrl, setAppUrl] = useState(null) useEffect(() => { window.ipc.invoke('account:getRowboat', null) - .then((account: any) => setAppUrl(account.config?.appUrl ?? null)) + .then((account: BillingRowboatAccount) => setAppUrl(account.config?.appUrl ?? null)) .catch(() => {}) }, []) @@ -152,6 +165,7 @@ interface ChatSidebarProps { onPresetMessageConsumed?: () => void getInitialDraft?: (tabId: string) => string | undefined onDraftChangeForTab?: (tabId: string, text: string) => void + onSelectedModelChangeForTab?: (tabId: string, model: SelectedModel | null) => void pendingAskHumanRequests?: ChatTabViewState['pendingAskHumanRequests'] allPermissionRequests?: ChatTabViewState['allPermissionRequests'] permissionResponses?: ChatTabViewState['permissionResponses'] @@ -205,6 +219,7 @@ export function ChatSidebar({ onPresetMessageConsumed, getInitialDraft, onDraftChangeForTab, + onSelectedModelChangeForTab, pendingAskHumanRequests = new Map(), allPermissionRequests = new Map(), permissionResponses = new Map(), @@ -345,7 +360,14 @@ export function ChatSidebar({ {item.content && ( - {item.content} + + + {item.content} + + )} ) @@ -366,7 +388,12 @@ export function ChatSidebar({ ))}
)} - {message} + + {message} + ) @@ -467,6 +494,7 @@ export function ChatSidebar({ return (
onDraftChangeForTab(tab.id, text) : undefined} + onSelectedModelChange={onSelectedModelChangeForTab ? (m) => onSelectedModelChangeForTab(tab.id, m) : undefined} isRecording={isActive && isRecording} recordingText={isActive ? recordingText : undefined} recordingState={isActive ? recordingState : undefined} diff --git a/apps/x/apps/renderer/src/components/markdown-editor.tsx b/apps/x/apps/renderer/src/components/markdown-editor.tsx index d7920b8b..e97f7c6e 100644 --- a/apps/x/apps/renderer/src/components/markdown-editor.tsx +++ b/apps/x/apps/renderer/src/components/markdown-editor.tsx @@ -7,17 +7,24 @@ import Image from '@tiptap/extension-image' import Placeholder from '@tiptap/extension-placeholder' import TaskList from '@tiptap/extension-task-list' import TaskItem from '@tiptap/extension-task-item' +import { TableKit, renderTableToMarkdown } from '@tiptap/extension-table' +import type { JSONContent, MarkdownRendererHelpers } from '@tiptap/react' import { ImageUploadPlaceholderExtension, createImageUploadHandler } from '@/extensions/image-upload' import { TaskBlockExtension } from '@/extensions/task-block' +import { TrackBlockExtension } from '@/extensions/track-block' +import { PromptBlockExtension } from '@/extensions/prompt-block' +import { TrackTargetOpenExtension, TrackTargetCloseExtension } from '@/extensions/track-target' import { ImageBlockExtension } from '@/extensions/image-block' import { EmbedBlockExtension } from '@/extensions/embed-block' +import { IframeBlockExtension } from '@/extensions/iframe-block' import { ChartBlockExtension } from '@/extensions/chart-block' import { TableBlockExtension } from '@/extensions/table-block' import { CalendarBlockExtension } from '@/extensions/calendar-block' import { EmailBlockExtension } from '@/extensions/email-block' import { TranscriptBlockExtension } from '@/extensions/transcript-block' +import { MermaidBlockExtension } from '@/extensions/mermaid-block' import { Markdown } from 'tiptap-markdown' -import { useEffect, useCallback, useMemo, useRef, useState } from 'react' +import { useEffect, useCallback, useMemo, useRef, useState, forwardRef, useImperativeHandle } from 'react' import { Calendar, ChevronDown, ExternalLink } from 'lucide-react' // Zero-width space used as invisible marker for blank lines @@ -41,6 +48,36 @@ function preprocessMarkdown(markdown: string): string { }) } +// Convert track-target open/close HTML comment markers into placeholder divs +// that TrackTargetOpenExtension / TrackTargetCloseExtension pick up as atom +// nodes. Content *between* the markers is left untouched — tiptap-markdown +// parses it naturally as whatever it is (paragraphs, lists, custom-block +// fences, etc.), all rendered live by the existing extension set. +// +// CommonMark rule: a type-6 HTML block (div, etc.) runs from the opening tag +// line until a blank line terminates it, and markdown inline rules (bold, +// italics, links) don't apply inside the block. Without surrounding blank +// lines, the line right after our placeholder div gets absorbed as HTML and +// its markdown is not parsed. +// +// Consume ALL adjacent newlines (\n*, not \n?) so the emitted `\n\n…\n\n` +// is load/save stable. serializeBlocksToMarkdown emits `\n\n` between blocks +// on save; a `\n?` regex on reload would only consume one of those two +// newlines, so every cycle would add a net newline on each side of every +// marker — causing tracks running on an open note to steadily inflate the +// file with blank lines around target regions. +function preprocessTrackTargets(md: string): string { + return md + .replace( + /\n*\n*/g, + (_m, id: string) => `\n\n
\n\n`, + ) + .replace( + /\n*\n*/g, + (_m, id: string) => `\n\n
\n\n`, + ) +} + // Post-process to clean up any zero-width spaces in the output function postprocessMarkdown(markdown: string): string { // Remove lines that contain only the zero-width space marker @@ -53,158 +90,244 @@ function postprocessMarkdown(markdown: string): string { }).join('\n') } -// Custom function to get markdown that preserves empty paragraphs as blank lines -function getMarkdownWithBlankLines(editor: Editor): string { - const json = editor.getJSON() - if (!json.content) return '' +type JsonNode = { + type?: string + content?: JsonNode[] + text?: string + marks?: Array<{ type: string; attrs?: Record }> + attrs?: Record +} - const blocks: string[] = [] - - // Helper to convert a node to markdown text - const nodeToText = (node: { - type?: string - content?: Array<{ - type?: string - text?: string - marks?: Array<{ type: string; attrs?: Record }> - attrs?: Record - }> - attrs?: Record - }): string => { - if (!node.content) return '' - return node.content.map(child => { - if (child.type === 'text') { - let text = child.text || '' - // Apply marks (bold, italic, etc.) - if (child.marks) { - for (const mark of child.marks) { - if (mark.type === 'bold') text = `**${text}**` - else if (mark.type === 'italic') text = `*${text}*` - else if (mark.type === 'code') text = `\`${text}\`` - else if (mark.type === 'link' && mark.attrs?.href) text = `[${text}](${mark.attrs.href})` - } +// Convert a node's inline content (text + marks + wikiLinks + hardBreaks) to markdown text +function nodeToText(node: JsonNode): string { + if (!node.content) return '' + return node.content.map(child => { + if (child.type === 'text') { + let text = child.text || '' + if (child.marks) { + for (const mark of child.marks) { + if (mark.type === 'bold') text = `**${text}**` + else if (mark.type === 'italic') text = `*${text}*` + else if (mark.type === 'code') text = `\`${text}\`` + else if (mark.type === 'link' && mark.attrs?.href) text = `[${text}](${mark.attrs.href})` } - return text - } else if (child.type === 'wikiLink') { - const path = (child.attrs?.path as string) || '' - return path ? `[[${path}]]` : '' - } else if (child.type === 'hardBreak') { - return '\n' } - return '' - }).join('') - } + return text + } else if (child.type === 'wikiLink') { + const path = (child.attrs?.path as string) || '' + return path ? `[[${path}]]` : '' + } else if (child.type === 'hardBreak') { + return '\n' + } + return '' + }).join('') +} - for (const node of json.content) { - if (node.type === 'paragraph') { - const text = nodeToText(node) - // If the paragraph contains only the blank line marker or is empty, it's a blank line - if (!text || text === BLANK_LINE_MARKER || text.trim() === BLANK_LINE_MARKER) { - // Push empty string to represent blank line - will add extra newline when joining - blocks.push('') +// Recursively serialize a list node (one line per item; nested lists indented two spaces) +function serializeList(listNode: JsonNode, indent: number): string[] { + const lines: string[] = [] + const items = (listNode.content || []) as JsonNode[] + items.forEach((item, index) => { + const indentStr = ' '.repeat(indent) + let prefix: string + if (listNode.type === 'taskList') { + const checked = item.attrs?.checked ? 'x' : ' ' + prefix = `- [${checked}] ` + } else if (listNode.type === 'orderedList') { + prefix = `${index + 1}. ` + } else { + prefix = '- ' + } + const itemContent = (item.content || []) as JsonNode[] + let firstPara = true + itemContent.forEach(child => { + if (child.type === 'bulletList' || child.type === 'orderedList' || child.type === 'taskList') { + lines.push(...serializeList(child, indent + 1)) } else { - blocks.push(text) + const text = nodeToText(child) + if (firstPara) { + lines.push(indentStr + prefix + text) + firstPara = false + } else { + lines.push(indentStr + ' ' + text) + } } - } else if (node.type === 'heading') { - const level = (node.attrs?.level as number) || 1 + }) + }) + return lines +} + +// Adapter for tiptap's first-party renderTableToMarkdown. Only renderChildren is +// actually invoked — the other helpers are stubs to satisfy the type. +const tableRenderHelpers: MarkdownRendererHelpers = { + renderChildren: (nodes) => { + const arr = Array.isArray(nodes) ? nodes : [nodes] + return arr.map(n => n.type === 'paragraph' ? nodeToText(n as JsonNode) : '').join('') + }, + wrapInBlock: (prefix, content) => prefix + content, + indent: (content) => content, +} + +// Serialize a single top-level block to its markdown string. Empty paragraphs (or blank-marker +// paragraphs) return '' to signal "blank line slot" for the join logic in serializeBlocksToMarkdown. +function blockToMarkdown(node: JsonNode): string { + switch (node.type) { + case 'paragraph': { const text = nodeToText(node) - blocks.push('#'.repeat(level) + ' ' + text) - } else if (node.type === 'bulletList' || node.type === 'orderedList' || node.type === 'taskList') { - // Recursively serialize lists to handle nested bullets - const serializeList = ( - listNode: { type?: string; content?: Array>; attrs?: Record }, - indent: number - ): string[] => { - const lines: string[] = [] - const items = (listNode.content || []) as Array<{ content?: Array>; attrs?: Record }> - items.forEach((item, index) => { - const indentStr = ' '.repeat(indent) - let prefix: string - if (listNode.type === 'taskList') { - const checked = item.attrs?.checked ? 'x' : ' ' - prefix = `- [${checked}] ` - } else if (listNode.type === 'orderedList') { - prefix = `${index + 1}. ` - } else { - prefix = '- ' - } - const itemContent = (item.content || []) as Array<{ type?: string; content?: Array<{ type?: string; text?: string; marks?: Array<{ type: string; attrs?: Record }> }>; attrs?: Record }> - let firstPara = true - itemContent.forEach(child => { - if (child.type === 'bulletList' || child.type === 'orderedList' || child.type === 'taskList') { - lines.push(...serializeList(child, indent + 1)) - } else { - const text = nodeToText(child) - if (firstPara) { - lines.push(indentStr + prefix + text) - firstPara = false - } else { - lines.push(indentStr + ' ' + text) - } - } - }) - }) - return lines - } - blocks.push(serializeList(node, 0).join('\n')) - } else if (node.type === 'taskBlock') { - blocks.push('```task\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'imageBlock') { - blocks.push('```image\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'embedBlock') { - blocks.push('```embed\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'chartBlock') { - blocks.push('```chart\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'tableBlock') { - blocks.push('```table\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'calendarBlock') { - blocks.push('```calendar\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'emailBlock') { - blocks.push('```email\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'transcriptBlock') { - blocks.push('```transcript\n' + (node.attrs?.data as string || '{}') + '\n```') - } else if (node.type === 'codeBlock') { + if (!text || text === BLANK_LINE_MARKER || text.trim() === BLANK_LINE_MARKER) return '' + return text + } + case 'heading': { + const level = (node.attrs?.level as number) || 1 + return '#'.repeat(level) + ' ' + nodeToText(node) + } + case 'bulletList': + case 'orderedList': + case 'taskList': + return serializeList(node, 0).join('\n') + case 'taskBlock': + return '```task\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'promptBlock': + return '```prompt\n' + (node.attrs?.data as string || '') + '\n```' + case 'trackBlock': + return '```track\n' + (node.attrs?.data as string || '') + '\n```' + case 'trackTargetOpen': + return `` + case 'trackTargetClose': + return `` + case 'imageBlock': + return '```image\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'embedBlock': + return '```embed\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'iframeBlock': + return '```iframe\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'chartBlock': + return '```chart\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'tableBlock': + return '```table\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'calendarBlock': + return '```calendar\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'emailBlock': + return '```email\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'transcriptBlock': + return '```transcript\n' + (node.attrs?.data as string || '{}') + '\n```' + case 'mermaidBlock': + return '```mermaid\n' + (node.attrs?.data as string || '') + '\n```' + case 'table': + return renderTableToMarkdown(node as JSONContent, tableRenderHelpers).trim() + case 'codeBlock': { const lang = (node.attrs?.language as string) || '' - blocks.push('```' + lang + '\n' + nodeToText(node) + '\n```') - } else if (node.type === 'blockquote') { - const content = node.content || [] - const quoteLines = content.map(para => '> ' + nodeToText(para)) - blocks.push(quoteLines.join('\n')) - } else if (node.type === 'horizontalRule') { - blocks.push('---') - } else if (node.type === 'wikiLink') { + return '```' + lang + '\n' + nodeToText(node) + '\n```' + } + case 'blockquote': { + const content = (node.content || []) as JsonNode[] + return content.map(para => '> ' + nodeToText(para)).join('\n') + } + case 'horizontalRule': + return '---' + case 'wikiLink': { const path = (node.attrs?.path as string) || '' - blocks.push(`[[${path}]]`) - } else if (node.type === 'image') { + return `[[${path}]]` + } + case 'image': { const src = (node.attrs?.src as string) || '' const alt = (node.attrs?.alt as string) || '' - blocks.push(`![${alt}](${src})`) + return `![${alt}](${src})` } + default: + return '' } +} - // Custom join: content blocks get \n\n before them, empty blocks add \n each - // This produces: 1 empty paragraph = 3 newlines (1 blank line on disk) +// Pure helper: serialize a slice of top-level block nodes to markdown. +// Custom join: content blocks get \n\n before them, empty blocks add \n each. +// 1 empty paragraph = 3 newlines on disk (1 blank line). +function serializeBlocksToMarkdown(blocks: JsonNode[]): string { if (blocks.length === 0) return '' - let result = '' - for (let i = 0; i < blocks.length; i++) { - const block = blocks[i] + const block = blockToMarkdown(blocks[i]) const isContent = block !== '' - if (i === 0) { result = block } else if (isContent) { - // Content block: add \n\n before it (standard paragraph break) result += '\n\n' + block } else { - // Empty block: just add \n (one extra newline for blank line) result += '\n' } } - return result } + +// Custom function to get markdown that preserves empty paragraphs as blank lines +function getMarkdownWithBlankLines(editor: Editor): string { + const json = editor.getJSON() as JsonNode + if (!json.content) return '' + return serializeBlocksToMarkdown(json.content as JsonNode[]) +} + +// Compute the cursor's 1-indexed line number in the markdown that getMarkdownWithBlankLines +// would produce. Used to attach precise line-references when inserting editor-context mentions. +function getCursorContextLine(editor: Editor): number { + const $from = editor.state.selection.$from + const json = editor.getJSON() as JsonNode + const blocks = (json.content ?? []) as JsonNode[] + if (blocks.length === 0) return 1 + + const blockIndex = $from.index(0) + if (blockIndex < 0 || blockIndex >= blocks.length) return 1 + + // Line where the cursor's top-level block starts. + // Joining: prefix + '\n\n' + nextContentBlock → next block sits two lines below the prefix's last line. + let blockStartLine: number + if (blockIndex === 0) { + blockStartLine = 1 + } else { + const prefix = serializeBlocksToMarkdown(blocks.slice(0, blockIndex)) + const prefixLineCount = prefix === '' ? 0 : prefix.split('\n').length + blockStartLine = prefixLineCount + 2 + } + + return blockStartLine + computeWithinBlockOffset(blocks[blockIndex], $from) +} + +// Lines into the cursor's top-level block. 0 for the common single-line cases (paragraph/heading); +// for multi-line containers, computed against how the block serializes. +function computeWithinBlockOffset( + block: JsonNode, + $from: { parentOffset: number; depth: number; index: (depth: number) => number } +): number { + switch (block.type) { + case 'paragraph': + case 'heading': { + // Each hardBreak before the cursor moves us down one rendered line. + const offset = $from.parentOffset + let pos = 0 + let hbCount = 0 + for (const child of (block.content ?? [])) { + if (pos >= offset) break + const size = child.type === 'text' ? (child.text?.length ?? 0) : 1 + if (child.type === 'hardBreak' && pos < offset) hbCount++ + pos += size + } + return hbCount + } + case 'bulletList': + case 'orderedList': + case 'taskList': + case 'blockquote': + // Item index within the container = lines into the block (one item per line for shallow lists/quotes). + return $from.depth >= 1 ? $from.index(1) : 0 + case 'codeBlock': { + // +1 for the opening ``` fence line, plus newlines within the code text before the cursor. + const text = block.content?.[0]?.text ?? '' + const before = text.substring(0, $from.parentOffset) + return 1 + (before.match(/\n/g)?.length ?? 0) + } + default: + return 0 + } +} import { EditorToolbar } from './editor-toolbar' import { FrontmatterProperties } from './frontmatter-properties' import { WikiLink } from '@/extensions/wiki-link' @@ -436,7 +559,12 @@ const TabIndentExtension = Extension.create({ }, }) -export function MarkdownEditor({ +export interface MarkdownEditorHandle { + /** Returns {path, lineNumber} for the cursor's current position, or null if no notePath / no editor. */ + getCursorContext: () => { path: string; lineNumber: number } | null +} + +export const MarkdownEditor = forwardRef(function MarkdownEditor({ content, onChange, onPrimaryHeadingCommit, @@ -451,7 +579,7 @@ export function MarkdownEditor({ onFrontmatterChange, onExport, notePath, -}: MarkdownEditorProps) { +}, ref) { const isInternalUpdate = useRef(false) const wrapperRef = useRef(null) const [activeWikiLink, setActiveWikiLink] = useState(null) @@ -569,13 +697,19 @@ export function MarkdownEditor({ }), ImageUploadPlaceholderExtension, TaskBlockExtension, + TrackBlockExtension.configure({ notePath }), + PromptBlockExtension.configure({ notePath }), + TrackTargetOpenExtension, + TrackTargetCloseExtension, ImageBlockExtension, EmbedBlockExtension, + IframeBlockExtension, ChartBlockExtension, TableBlockExtension, CalendarBlockExtension, EmailBlockExtension, TranscriptBlockExtension, + MermaidBlockExtension, WikiLink.configure({ onCreate: wikiLinks?.onCreate ? (path) => { @@ -587,6 +721,9 @@ export function MarkdownEditor({ TaskItem.configure({ nested: true, }), + TableKit.configure({ + table: { resizable: false }, + }), Placeholder.configure({ placeholder, }), @@ -785,6 +922,17 @@ export function MarkdownEditor({ }) }, [editor, wikiLinks]) + useImperativeHandle(ref, () => ({ + getCursorContext: () => { + if (!notePath || !editor) return null + try { + return { path: notePath, lineNumber: getCursorContextLine(editor) } + } catch { + return null + } + }, + }), [notePath, editor]) + const updateRowboatMentionState = useCallback(() => { if (!editor) return const { selection } = editor.state @@ -951,8 +1099,9 @@ export function MarkdownEditor({ const normalizeForCompare = (s: string) => s.split('\n').map(line => line.trimEnd()).join('\n').trim() if (normalizeForCompare(currentContent) !== normalizeForCompare(content)) { isInternalUpdate.current = true - // Pre-process to preserve blank lines - const preprocessed = preprocessMarkdown(content) + // Pre-process to preserve blank lines, then wrap track-target comment + // regions into placeholder divs so TrackTargetExtension can pick them up. + const preprocessed = preprocessMarkdown(preprocessTrackTargets(content)) // Treat tab-open content as baseline: do not add hydration to undo history. editor.chain().setMeta('addToHistory', false).setContent(preprocessed).run() isInternalUpdate.current = false @@ -1448,4 +1597,4 @@ export function MarkdownEditor({
) -} +}) diff --git a/apps/x/apps/renderer/src/components/mermaid-renderer.tsx b/apps/x/apps/renderer/src/components/mermaid-renderer.tsx new file mode 100644 index 00000000..db42df2e --- /dev/null +++ b/apps/x/apps/renderer/src/components/mermaid-renderer.tsx @@ -0,0 +1,89 @@ +import { useEffect, useId, useRef, useState } from 'react' +import mermaid from 'mermaid' +import { useTheme } from '@/contexts/theme-context' + +let lastTheme: string | null = null + +function ensureInit(theme: 'default' | 'dark') { + if (lastTheme === theme) return + mermaid.initialize({ + startOnLoad: false, + theme, + securityLevel: 'strict', + }) + lastTheme = theme +} + +interface MermaidRendererProps { + source: string + className?: string +} + +export function MermaidRenderer({ source, className }: MermaidRendererProps) { + const { resolvedTheme } = useTheme() + const id = useId().replace(/:/g, '-') + const containerRef = useRef(null) + const [svg, setSvg] = useState(null) + const [error, setError] = useState(null) + + useEffect(() => { + if (!source.trim()) { + setSvg(null) + setError(null) + return + } + + let cancelled = false + const mermaidTheme = resolvedTheme === 'dark' ? 'dark' : 'default' + ensureInit(mermaidTheme) + + mermaid + .render(`mermaid-${id}`, source.trim()) + .then(({ svg: renderedSvg }) => { + if (!cancelled) { + setSvg(renderedSvg) + setError(null) + } + }) + .catch((err: unknown) => { + if (!cancelled) { + setSvg(null) + setError(err instanceof Error ? err.message : 'Failed to render diagram') + } + }) + + return () => { + cancelled = true + } + }, [source, resolvedTheme, id]) + + if (error) { + return ( +
+
+ Invalid mermaid syntax +
+ 
+          {source}
+
+
+ ) + } + + if (!svg) { + return ( +
+ Rendering diagram... +
+ ) + } + + return ( +
+ ) +} diff --git a/apps/x/apps/renderer/src/components/onboarding-modal.tsx b/apps/x/apps/renderer/src/components/onboarding-modal.tsx index c7f723ac..469ac35d 100644 --- a/apps/x/apps/renderer/src/components/onboarding-modal.tsx +++ b/apps/x/apps/renderer/src/components/onboarding-modal.tsx @@ -59,14 +59,14 @@ export function OnboardingModal({ open, onComplete }: OnboardingModalProps) { const [modelsCatalog, setModelsCatalog] = useState>({}) const [modelsLoading, setModelsLoading] = useState(false) const [modelsError, setModelsError] = useState(null) - const [providerConfigs, setProviderConfigs] = useState>({ - openai: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - anthropic: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - google: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - openrouter: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - aigateway: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - ollama: { apiKey: "", baseURL: "http://localhost:11434", model: "", knowledgeGraphModel: "" }, - "openai-compatible": { apiKey: "", baseURL: "http://localhost:1234/v1", model: "", knowledgeGraphModel: "" }, + const [providerConfigs, setProviderConfigs] = useState>({ + openai: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + anthropic: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + google: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + openrouter: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + aigateway: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + ollama: { apiKey: "", baseURL: "http://localhost:11434", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + "openai-compatible": { apiKey: "", baseURL: "http://localhost:1234/v1", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, }) const [testState, setTestState] = useState<{ status: "idle" | "testing" | "success" | "error"; error?: string }>({ status: "idle", @@ -109,7 +109,7 @@ export function OnboardingModal({ open, onComplete }: OnboardingModalProps) { const [googleCalendarConnecting, setGoogleCalendarConnecting] = useState(false) const updateProviderConfig = useCallback( - (provider: LlmProviderFlavor, updates: Partial<{ apiKey: string; baseURL: string; model: string; knowledgeGraphModel: string }>) => { + (provider: LlmProviderFlavor, updates: Partial<{ apiKey: string; baseURL: string; model: string; knowledgeGraphModel: string; meetingNotesModel: string; trackBlockModel: string }>) => { setProviderConfigs(prev => ({ ...prev, [provider]: { ...prev[provider], ...updates }, @@ -458,6 +458,8 @@ export function OnboardingModal({ open, onComplete }: OnboardingModalProps) { const baseURL = activeConfig.baseURL.trim() || undefined const model = activeConfig.model.trim() const knowledgeGraphModel = activeConfig.knowledgeGraphModel.trim() || undefined + const meetingNotesModel = activeConfig.meetingNotesModel.trim() || undefined + const trackBlockModel = activeConfig.trackBlockModel.trim() || undefined const providerConfig = { provider: { flavor: llmProvider, @@ -466,6 +468,8 @@ export function OnboardingModal({ open, onComplete }: OnboardingModalProps) { }, model, knowledgeGraphModel, + meetingNotesModel, + trackBlockModel, } const result = await window.ipc.invoke("models:test", providerConfig) if (result.success) { @@ -1157,6 +1161,72 @@ export function OnboardingModal({ open, onComplete }: OnboardingModalProps) { )}
+ +
+ Meeting notes model + {modelsLoading ? ( +
+ + Loading... +
+ ) : showModelInput ? ( + updateProviderConfig(llmProvider, { meetingNotesModel: e.target.value })} + placeholder={activeConfig.model || "Enter model"} + /> + ) : ( + + )} +
+ +
+ Track block model + {modelsLoading ? ( +
+ + Loading... +
+ ) : showModelInput ? ( + updateProviderConfig(llmProvider, { trackBlockModel: e.target.value })} + placeholder={activeConfig.model || "Enter model"} + /> + ) : ( + + )} +
{showApiKey && ( diff --git a/apps/x/apps/renderer/src/components/onboarding/steps/llm-setup-step.tsx b/apps/x/apps/renderer/src/components/onboarding/steps/llm-setup-step.tsx index a9956245..a11b0d5f 100644 --- a/apps/x/apps/renderer/src/components/onboarding/steps/llm-setup-step.tsx +++ b/apps/x/apps/renderer/src/components/onboarding/steps/llm-setup-step.tsx @@ -221,6 +221,76 @@ export function LlmSetupStep({ state }: LlmSetupStepProps) { )} + +
+ + {modelsLoading ? ( +
+ + Loading... +
+ ) : showModelInput ? ( + updateProviderConfig(llmProvider, { meetingNotesModel: e.target.value })} + placeholder={activeConfig.model || "Enter model"} + /> + ) : ( + + )} +
+ +
+ + {modelsLoading ? ( +
+ + Loading... +
+ ) : showModelInput ? ( + updateProviderConfig(llmProvider, { trackBlockModel: e.target.value })} + placeholder={activeConfig.model || "Enter model"} + /> + ) : ( + + )} +
{showApiKey && ( diff --git a/apps/x/apps/renderer/src/components/onboarding/use-onboarding-state.ts b/apps/x/apps/renderer/src/components/onboarding/use-onboarding-state.ts index a55b23fe..edb3616b 100644 --- a/apps/x/apps/renderer/src/components/onboarding/use-onboarding-state.ts +++ b/apps/x/apps/renderer/src/components/onboarding/use-onboarding-state.ts @@ -29,14 +29,14 @@ export function useOnboardingState(open: boolean, onComplete: () => void) { const [modelsCatalog, setModelsCatalog] = useState>({}) const [modelsLoading, setModelsLoading] = useState(false) const [modelsError, setModelsError] = useState(null) - const [providerConfigs, setProviderConfigs] = useState>({ - openai: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - anthropic: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - google: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - openrouter: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - aigateway: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "" }, - ollama: { apiKey: "", baseURL: "http://localhost:11434", model: "", knowledgeGraphModel: "" }, - "openai-compatible": { apiKey: "", baseURL: "http://localhost:1234/v1", model: "", knowledgeGraphModel: "" }, + const [providerConfigs, setProviderConfigs] = useState>({ + openai: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + anthropic: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + google: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + openrouter: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + aigateway: { apiKey: "", baseURL: "", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + ollama: { apiKey: "", baseURL: "http://localhost:11434", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + "openai-compatible": { apiKey: "", baseURL: "http://localhost:1234/v1", model: "", knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, }) const [testState, setTestState] = useState<{ status: "idle" | "testing" | "success" | "error"; error?: string }>({ status: "idle", @@ -81,7 +81,7 @@ export function useOnboardingState(open: boolean, onComplete: () => void) { const [googleCalendarConnecting, setGoogleCalendarConnecting] = useState(false) const updateProviderConfig = useCallback( - (provider: LlmProviderFlavor, updates: Partial<{ apiKey: string; baseURL: string; model: string; knowledgeGraphModel: string }>) => { + (provider: LlmProviderFlavor, updates: Partial<{ apiKey: string; baseURL: string; model: string; knowledgeGraphModel: string; meetingNotesModel: string; trackBlockModel: string }>) => { setProviderConfigs(prev => ({ ...prev, [provider]: { ...prev[provider], ...updates }, @@ -435,6 +435,8 @@ export function useOnboardingState(open: boolean, onComplete: () => void) { const baseURL = activeConfig.baseURL.trim() || undefined const model = activeConfig.model.trim() const knowledgeGraphModel = activeConfig.knowledgeGraphModel.trim() || undefined + const meetingNotesModel = activeConfig.meetingNotesModel.trim() || undefined + const trackBlockModel = activeConfig.trackBlockModel.trim() || undefined const providerConfig = { provider: { flavor: llmProvider, @@ -443,6 +445,8 @@ export function useOnboardingState(open: boolean, onComplete: () => void) { }, model, knowledgeGraphModel, + meetingNotesModel, + trackBlockModel, } const result = await window.ipc.invoke("models:test", providerConfig) if (result.success) { @@ -459,7 +463,7 @@ export function useOnboardingState(open: boolean, onComplete: () => void) { setTestState({ status: "error", error: "Connection test failed" }) toast.error("Connection test failed") } - }, [activeConfig.apiKey, activeConfig.baseURL, activeConfig.model, activeConfig.knowledgeGraphModel, canTest, llmProvider, handleNext]) + }, [activeConfig.apiKey, activeConfig.baseURL, activeConfig.model, activeConfig.knowledgeGraphModel, activeConfig.meetingNotesModel, activeConfig.trackBlockModel, canTest, llmProvider, handleNext]) // Check connection status for all providers const refreshAllStatuses = useCallback(async () => { diff --git a/apps/x/apps/renderer/src/components/search-dialog.tsx b/apps/x/apps/renderer/src/components/search-dialog.tsx index 32bca1b3..66a37802 100644 --- a/apps/x/apps/renderer/src/components/search-dialog.tsx +++ b/apps/x/apps/renderer/src/components/search-dialog.tsx @@ -1,7 +1,7 @@ -import { useState, useEffect, useCallback } from 'react' +import { useState, useEffect, useCallback, useRef } from 'react' import posthog from 'posthog-js' import * as analytics from '@/lib/analytics' -import { FileTextIcon, MessageSquareIcon } from 'lucide-react' +import { FileTextIcon, MessageSquareIcon, XIcon } from 'lucide-react' import { CommandDialog, CommandInput, @@ -22,21 +22,50 @@ interface SearchResult { } type SearchType = 'knowledge' | 'chat' +type Mode = 'chat' | 'search' function activeTabToTypes(section: ActiveSection): SearchType[] { if (section === 'knowledge') return ['knowledge'] return ['chat'] // "tasks" tab maps to chat } -interface SearchDialogProps { - open: boolean - onOpenChange: (open: boolean) => void - onSelectFile: (path: string) => void - onSelectRun: (runId: string) => void +export type CommandPaletteContext = { + path: string + lineNumber: number } -export function SearchDialog({ open, onOpenChange, onSelectFile, onSelectRun }: SearchDialogProps) { +export type CommandPaletteMention = { + path: string + displayName: string + lineNumber?: number +} + +interface CommandPaletteProps { + open: boolean + onOpenChange: (open: boolean) => void + // Search mode + onSelectFile: (path: string) => void + onSelectRun: (runId: string) => void + // Chat mode + initialContext?: CommandPaletteContext | null + onChatSubmit: (text: string, mention: CommandPaletteMention | null) => void +} + +export function CommandPalette({ + open, + onOpenChange, + onSelectFile, + onSelectRun, + initialContext, + onChatSubmit, +}: CommandPaletteProps) { const { activeSection } = useSidebarSection() + const [mode, setMode] = useState('chat') + const [chatInput, setChatInput] = useState('') + const [contextChip, setContextChip] = useState(null) + const chatInputRef = useRef(null) + const searchInputRef = useRef(null) + const [query, setQuery] = useState('') const [results, setResults] = useState([]) const [isSearching, setIsSearching] = useState(false) @@ -45,17 +74,45 @@ export function SearchDialog({ open, onOpenChange, onSelectFile, onSelectRun }: ) const debouncedQuery = useDebounce(query, 250) - // Sync filter preselection when dialog opens + // On open: always reset to Chat mode (per spec — no mode persistence), sync context chip + // and reset search filters. useEffect(() => { if (open) { + setMode('chat') + setChatInput('') + setContextChip(initialContext ?? null) setActiveTypes(new Set(activeTabToTypes(activeSection))) } - }, [open, activeSection]) + }, [open, activeSection, initialContext]) + + // Tab cycles modes. Captured at document level so cmdk's internal Tab handling doesn't + // swallow it. Only fires while the dialog is open. + useEffect(() => { + if (!open) return + const handler = (e: KeyboardEvent) => { + if (e.key !== 'Tab') return + if (e.metaKey || e.ctrlKey || e.altKey || e.shiftKey) return + e.preventDefault() + e.stopPropagation() + setMode(prev => (prev === 'chat' ? 'search' : 'chat')) + } + document.addEventListener('keydown', handler, true) + return () => document.removeEventListener('keydown', handler, true) + }, [open]) + + // Refocus the appropriate input on mode change so the user can start typing immediately. + useEffect(() => { + if (!open) return + const target = mode === 'chat' ? chatInputRef : searchInputRef + target.current?.focus() + }, [open, mode]) const toggleType = useCallback((type: SearchType) => { setActiveTypes(new Set([type])) }, []) + // Search query effect (only meaningful while in search mode, but the debounce keeps running + // harmlessly otherwise — empty query skips the IPC call below). useEffect(() => { if (!debouncedQuery.trim()) { setResults([]) @@ -89,11 +146,12 @@ export function SearchDialog({ open, onOpenChange, onSelectFile, onSelectRun }: return () => { cancelled = true } }, [debouncedQuery, activeTypes]) - // Reset state when dialog closes + // Reset transient state on close. useEffect(() => { if (!open) { setQuery('') setResults([]) + setChatInput('') } }, [open]) @@ -106,6 +164,20 @@ export function SearchDialog({ open, onOpenChange, onSelectFile, onSelectRun }: } }, [onOpenChange, onSelectFile, onSelectRun]) + const submitChat = useCallback(() => { + const text = chatInput.trim() + if (!text && !contextChip) return + const mention: CommandPaletteMention | null = contextChip + ? { + path: contextChip.path, + displayName: deriveDisplayName(contextChip.path), + lineNumber: contextChip.lineNumber, + } + : null + onChatSubmit(text, mention) + onOpenChange(false) + }, [chatInput, contextChip, onChatSubmit, onOpenChange]) + const knowledgeResults = results.filter(r => r.type === 'knowledge') const chatResults = results.filter(r => r.type === 'chat') @@ -113,76 +185,178 @@ export function SearchDialog({ open, onOpenChange, onSelectFile, onSelectRun }: - + {/* Mode strip */}
- toggleType('knowledge')} - icon={} - label="Knowledge" - /> - toggleType('chat')} + setMode('chat')} icon={} - label="Chats" + label="Chat" /> + setMode('search')} + icon={} + label="Search" + /> + Tab to switch
- - {!query.trim() && ( - Type to search... - )} - {query.trim() && !isSearching && results.length === 0 && ( - No results found. - )} - {knowledgeResults.length > 0 && ( - - {knowledgeResults.map((result) => ( - handleSelect(result)} - > - -
- {result.title} - {result.preview} -
- - ))} - - )} - {chatResults.length > 0 && ( - - {chatResults.map((result) => ( - handleSelect(result)} - > - -
- {result.title} - {result.preview} -
- - ))} - - )} - + + {mode === 'chat' ? ( +
+ setChatInput(e.target.value)} + onKeyDown={(e) => { + // cmdk's Command component intercepts Enter for item selection — stop it + // before bubbling so we control the chat submit ourselves. + if (e.key === 'Enter' && !e.shiftKey && !e.metaKey && !e.ctrlKey && !e.altKey) { + e.preventDefault() + e.stopPropagation() + submitChat() + } + }} + placeholder="Ask copilot anything…" + autoFocus + className="w-full bg-transparent px-4 py-3 text-sm outline-none placeholder:text-muted-foreground" + /> + {contextChip && ( +
+ + + {deriveDisplayName(contextChip.path)} + · Line {contextChip.lineNumber} + + + Enter to send +
+ )} + {!contextChip && ( +
+ Enter to send +
+ )} +
+ ) : ( + <> + +
+ toggleType('knowledge')} + icon={} + label="Knowledge" + /> + toggleType('chat')} + icon={} + label="Chats" + /> +
+ + {!query.trim() && ( + Type to search... + )} + {query.trim() && !isSearching && results.length === 0 && ( + No results found. + )} + {knowledgeResults.length > 0 && ( + + {knowledgeResults.map((result) => ( + handleSelect(result)} + > + +
+ {result.title} + {result.preview} +
+ + ))} + + )} + {chatResults.length > 0 && ( + + {chatResults.map((result) => ( + handleSelect(result)} + > + +
+ {result.title} + {result.preview} +
+ + ))} + + )} + + + )} ) } +// Back-compat export so existing import sites don't break in one go; thin alias to CommandPalette. +export const SearchDialog = CommandPalette + +function deriveDisplayName(path: string): string { + const base = path.split('/').pop() ?? path + return base.replace(/\.md$/, '') +} + +function ModeButton({ + active, + onClick, + icon, + label, +}: { + active: boolean + onClick: () => void + icon: React.ReactNode + label: string +}) { + return ( + + ) +} + function FilterToggle({ active, onClick, diff --git a/apps/x/apps/renderer/src/components/settings-dialog.tsx b/apps/x/apps/renderer/src/components/settings-dialog.tsx index 143c6292..ddc506c9 100644 --- a/apps/x/apps/renderer/src/components/settings-dialog.tsx +++ b/apps/x/apps/renderer/src/components/settings-dialog.tsx @@ -196,14 +196,14 @@ const defaultBaseURLs: Partial> = { function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { const [provider, setProvider] = useState("openai") const [defaultProvider, setDefaultProvider] = useState(null) - const [providerConfigs, setProviderConfigs] = useState>({ - openai: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "" }, - anthropic: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "" }, - google: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "" }, - openrouter: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "" }, - aigateway: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "" }, - ollama: { apiKey: "", baseURL: "http://localhost:11434", models: [""], knowledgeGraphModel: "" }, - "openai-compatible": { apiKey: "", baseURL: "http://localhost:1234/v1", models: [""], knowledgeGraphModel: "" }, + const [providerConfigs, setProviderConfigs] = useState>({ + openai: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + anthropic: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + google: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + openrouter: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + aigateway: { apiKey: "", baseURL: "", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + ollama: { apiKey: "", baseURL: "http://localhost:11434", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, + "openai-compatible": { apiKey: "", baseURL: "http://localhost:1234/v1", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, }) const [modelsCatalog, setModelsCatalog] = useState>({}) const [modelsLoading, setModelsLoading] = useState(false) @@ -229,7 +229,7 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { (!requiresBaseURL || activeConfig.baseURL.trim().length > 0) const updateConfig = useCallback( - (prov: LlmProviderFlavor, updates: Partial<{ apiKey: string; baseURL: string; models: string[]; knowledgeGraphModel: string }>) => { + (prov: LlmProviderFlavor, updates: Partial<{ apiKey: string; baseURL: string; models: string[]; knowledgeGraphModel: string; meetingNotesModel: string; trackBlockModel: string }>) => { setProviderConfigs(prev => ({ ...prev, [prov]: { ...prev[prov], ...updates }, @@ -302,6 +302,8 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { baseURL: e.baseURL || (defaultBaseURLs[key as LlmProviderFlavor] || ""), models: savedModels, knowledgeGraphModel: e.knowledgeGraphModel || "", + meetingNotesModel: e.meetingNotesModel || "", + trackBlockModel: e.trackBlockModel || "", }; } } @@ -318,6 +320,8 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { baseURL: parsed.provider.baseURL || (defaultBaseURLs[flavor] || ""), models: activeModels.length > 0 ? activeModels : [""], knowledgeGraphModel: parsed.knowledgeGraphModel || "", + meetingNotesModel: parsed.meetingNotesModel || "", + trackBlockModel: parsed.trackBlockModel || "", }; } return next; @@ -391,6 +395,8 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { model: allModels[0] || "", models: allModels, knowledgeGraphModel: activeConfig.knowledgeGraphModel.trim() || undefined, + meetingNotesModel: activeConfig.meetingNotesModel.trim() || undefined, + trackBlockModel: activeConfig.trackBlockModel.trim() || undefined, } const result = await window.ipc.invoke("models:test", providerConfig) if (result.success) { @@ -423,6 +429,8 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { model: allModels[0], models: allModels, knowledgeGraphModel: config.knowledgeGraphModel.trim() || undefined, + meetingNotesModel: config.meetingNotesModel.trim() || undefined, + trackBlockModel: config.trackBlockModel.trim() || undefined, }) setDefaultProvider(prov) window.dispatchEvent(new Event('models-config-changed')) @@ -452,6 +460,8 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { parsed.model = defModels[0] || "" parsed.models = defModels parsed.knowledgeGraphModel = defConfig.knowledgeGraphModel.trim() || undefined + parsed.meetingNotesModel = defConfig.meetingNotesModel.trim() || undefined + parsed.trackBlockModel = defConfig.trackBlockModel.trim() || undefined } await window.ipc.invoke("workspace:writeFile", { path: "config/models.json", @@ -459,7 +469,7 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { }) setProviderConfigs(prev => ({ ...prev, - [prov]: { apiKey: "", baseURL: defaultBaseURLs[prov] || "", models: [""], knowledgeGraphModel: "" }, + [prov]: { apiKey: "", baseURL: defaultBaseURLs[prov] || "", models: [""], knowledgeGraphModel: "", meetingNotesModel: "", trackBlockModel: "" }, })) setTestState({ status: "idle" }) window.dispatchEvent(new Event('models-config-changed')) @@ -649,6 +659,74 @@ function ModelSettings({ dialogOpen }: { dialogOpen: boolean }) { )} + + {/* Meeting notes model */} +
+ Meeting notes model + {modelsLoading ? ( +
+ + Loading... +
+ ) : showModelInput ? ( + updateConfig(provider, { meetingNotesModel: e.target.value })} + placeholder={primaryModel || "Enter model"} + /> + ) : ( + + )} +
+ + {/* Track block model */} +
+ Track block model + {modelsLoading ? ( +
+ + Loading... +
+ ) : showModelInput ? ( + updateConfig(provider, { trackBlockModel: e.target.value })} + placeholder={primaryModel || "Enter model"} + /> + ) : ( + + )} +
{/* API Key */} diff --git a/apps/x/apps/renderer/src/components/sidebar-content.tsx b/apps/x/apps/renderer/src/components/sidebar-content.tsx index 3fcb1acc..41d6b622 100644 --- a/apps/x/apps/renderer/src/components/sidebar-content.tsx +++ b/apps/x/apps/renderer/src/components/sidebar-content.tsx @@ -12,13 +12,18 @@ import { FilePlus, Folder, FolderPlus, + Globe, AlertTriangle, HelpCircle, Mic, Network, Pencil, + Radio, + SearchIcon, + SquarePen, Table2, Plug, + Lightbulb, LoaderIcon, Settings, Square, @@ -58,6 +63,7 @@ import { SidebarGroupContent, SidebarHeader, SidebarMenu, + SidebarMenuAction, SidebarMenuButton, SidebarMenuItem, SidebarMenuSub, @@ -90,6 +96,7 @@ import { SettingsDialog } from "@/components/settings-dialog" import { toast } from "@/lib/toast" import { useBilling } from "@/hooks/useBilling" import { ServiceEvent } from "@x/shared/src/service-events.js" +import type { MeetingTranscriptionState } from "@/hooks/useMeetingTranscription" import z from "zod" interface TreeNode { @@ -164,6 +171,7 @@ type SidebarContentPanelProps = { selectedPath: string | null expandedPaths: Set onSelectFile: (path: string, kind: "file" | "dir") => void + onToggleFolder?: (path: string) => void knowledgeActions: KnowledgeActions onVoiceNoteCreated?: (path: string) => void runs?: RunListItem[] @@ -172,6 +180,16 @@ type SidebarContentPanelProps = { tasksActions?: TasksActions backgroundTasks?: BackgroundTaskItem[] selectedBackgroundTask?: string | null + onNewChat?: () => void + onOpenSearch?: () => void + meetingState?: MeetingTranscriptionState + meetingSummarizing?: boolean + meetingAvailable?: boolean + onToggleMeeting?: () => void + isBrowserOpen?: boolean + onToggleBrowser?: () => void + isSuggestedTopicsOpen?: boolean + onOpenSuggestedTopics?: () => void } & React.ComponentProps const sectionTabs: { id: ActiveSection; label: string }[] = [ @@ -387,6 +405,7 @@ export function SidebarContentPanel({ selectedPath, expandedPaths, onSelectFile, + onToggleFolder, knowledgeActions, onVoiceNoteCreated, runs = [], @@ -395,6 +414,16 @@ export function SidebarContentPanel({ tasksActions, backgroundTasks = [], selectedBackgroundTask, + onNewChat, + onOpenSearch, + meetingState = 'idle', + meetingSummarizing = false, + meetingAvailable = false, + onToggleMeeting, + isBrowserOpen = false, + onToggleBrowser, + isSuggestedTopicsOpen = false, + onOpenSuggestedTopics, ...props }: SidebarContentPanelProps) { const { activeSection, setActiveSection } = useSidebarSection() @@ -488,6 +517,89 @@ export function SidebarContentPanel({ ))} + {/* Quick action buttons */} +
+ {onNewChat && ( + + )} + {onOpenSearch && ( + + )} + {meetingAvailable && onToggleMeeting && ( + + )} + {onToggleBrowser && ( + + )} + {onOpenSuggestedTopics && ( + + )} +
{activeSection === "knowledge" && ( @@ -496,6 +608,7 @@ export function SidebarContentPanel({ selectedPath={selectedPath} expandedPaths={expandedPaths} onSelectFile={onSelectFile} + onToggleFolder={onToggleFolder} actions={knowledgeActions} onVoiceNoteCreated={onVoiceNoteCreated} /> @@ -884,6 +997,7 @@ function KnowledgeSection({ selectedPath, expandedPaths, onSelectFile, + onToggleFolder, actions, onVoiceNoteCreated, }: { @@ -891,6 +1005,7 @@ function KnowledgeSection({ selectedPath: string | null expandedPaths: Set onSelectFile: (path: string, kind: "file" | "dir") => void + onToggleFolder?: (path: string) => void actions: KnowledgeActions onVoiceNoteCreated?: (path: string) => void }) { @@ -980,6 +1095,7 @@ function KnowledgeSection({ selectedPath={selectedPath} expandedPaths={expandedPaths} onSelect={onSelectFile} + onToggleFolder={onToggleFolder} actions={actions} /> ))} @@ -1008,9 +1124,7 @@ function countFiles(node: TreeNode): number { } /** Display name overrides for top-level knowledge folders */ -const FOLDER_DISPLAY_NAMES: Record = { - Notes: 'My Notes', -} +const FOLDER_DISPLAY_NAMES: Record = {} // Tree component for file browser function Tree({ @@ -1018,12 +1132,14 @@ function Tree({ selectedPath, expandedPaths, onSelect, + onToggleFolder, actions, }: { item: TreeNode selectedPath: string | null expandedPaths: Set onSelect: (path: string, kind: "file" | "dir") => void + onToggleFolder?: (path: string) => void actions: KnowledgeActions }) { const isDir = item.kind === 'dir' @@ -1160,15 +1276,15 @@ function Tree({ ) } - // Top-level knowledge folders (except Notes) open bases view — render as flat items + // Top-level knowledge folders open bases view — render as flat items const parts = item.path.split('/') - const isBasesFolder = isDir && parts.length === 2 && parts[0] === 'knowledge' && parts[1] !== 'Notes' + const isBasesFolder = isDir && parts.length === 2 && parts[0] === 'knowledge' if (isBasesFolder) { return ( - + onSelect(item.path, item.kind)}>
@@ -1176,6 +1292,38 @@ function Tree({ {countFiles(item)}
+ {onToggleFolder && (item.children?.length ?? 0) > 0 && ( + { + e.stopPropagation() + onToggleFolder(item.path) + }} + > + + + )} + {isExpanded && ( + + {(item.children ?? []).map((subItem, index) => ( + + ))} + + )} {contextMenuContent} @@ -1240,6 +1388,7 @@ function Tree({ selectedPath={selectedPath} expandedPaths={expandedPaths} onSelect={onSelect} + onToggleFolder={onToggleFolder} actions={actions} /> ))} diff --git a/apps/x/apps/renderer/src/components/suggested-topics-view.tsx b/apps/x/apps/renderer/src/components/suggested-topics-view.tsx new file mode 100644 index 00000000..4440aba9 --- /dev/null +++ b/apps/x/apps/renderer/src/components/suggested-topics-view.tsx @@ -0,0 +1,246 @@ +import { useCallback, useEffect, useState } from 'react' +import { ArrowRight, Lightbulb, Loader2 } from 'lucide-react' +import { SuggestedTopicBlockSchema, type SuggestedTopicBlock } from '@x/shared/dist/blocks.js' + +const SUGGESTED_TOPICS_PATH = 'suggested-topics.md' +const LEGACY_SUGGESTED_TOPICS_PATHS = [ + 'config/suggested-topics.md', + 'knowledge/Notes/Suggested Topics.md', +] + +/** Parse suggestedtopic code-fence blocks from the markdown file content. */ +function parseTopics(content: string): SuggestedTopicBlock[] { + const topics: SuggestedTopicBlock[] = [] + const regex = /```suggestedtopic\s*\n([\s\S]*?)```/g + let match: RegExpExecArray | null + while ((match = regex.exec(content)) !== null) { + try { + const parsed = JSON.parse(match[1].trim()) + const topic = SuggestedTopicBlockSchema.parse(parsed) + topics.push(topic) + } catch { + // Skip malformed blocks + } + } + + if (topics.length > 0) return topics + + const lines = content + .split('\n') + .map(line => line.trim()) + .filter(line => line && !line.startsWith('#')) + + for (const line of lines) { + try { + const parsed = JSON.parse(line) + const topic = SuggestedTopicBlockSchema.parse(parsed) + topics.push(topic) + } catch { + // Skip malformed lines + } + } + + return topics +} + +function serializeTopics(topics: SuggestedTopicBlock[]): string { + const blocks = topics.map((topic) => [ + '```suggestedtopic', + JSON.stringify(topic), + '```', + ].join('\n')) + + return ['# Suggested Topics', ...blocks].join('\n\n') + '\n' +} + +const CATEGORY_COLORS: Record = { + Meetings: 'bg-violet-500/10 text-violet-600 dark:text-violet-400', + Projects: 'bg-blue-500/10 text-blue-600 dark:text-blue-400', + People: 'bg-emerald-500/10 text-emerald-600 dark:text-emerald-400', + Organizations: 'bg-cyan-500/10 text-cyan-600 dark:text-cyan-400', + Topics: 'bg-amber-500/10 text-amber-600 dark:text-amber-400', +} + +function getCategoryColor(category?: string): string { + if (!category) return 'bg-muted text-muted-foreground' + return CATEGORY_COLORS[category] ?? 'bg-muted text-muted-foreground' +} + +interface TopicCardProps { + topic: SuggestedTopicBlock + onTrack: () => void + isRemoving: boolean +} + +function TopicCard({ topic, onTrack, isRemoving }: TopicCardProps) { + return ( +
+
+

+ {topic.title} +

+ {topic.category && ( + + {topic.category} + + )} +
+

+ {topic.description} +

+ +
+ ) +} + +interface SuggestedTopicsViewProps { + onExploreTopic: (topic: SuggestedTopicBlock) => void +} + +export function SuggestedTopicsView({ onExploreTopic }: SuggestedTopicsViewProps) { + const [topics, setTopics] = useState([]) + const [loading, setLoading] = useState(true) + const [error, setError] = useState(null) + const [removingIndex, setRemovingIndex] = useState(null) + + useEffect(() => { + let cancelled = false + async function load() { + try { + let result + try { + result = await window.ipc.invoke('workspace:readFile', { + path: SUGGESTED_TOPICS_PATH, + }) + } catch { + let legacyResult: { data?: string } | null = null + let legacyPath: string | null = null + for (const path of LEGACY_SUGGESTED_TOPICS_PATHS) { + try { + legacyResult = await window.ipc.invoke('workspace:readFile', { path }) + legacyPath = path + break + } catch { + // Try next legacy location. + } + } + if (!legacyResult || !legacyPath || legacyResult.data === undefined) { + throw new Error('Suggested topics file not found') + } + await window.ipc.invoke('workspace:writeFile', { + path: SUGGESTED_TOPICS_PATH, + data: legacyResult.data, + opts: { encoding: 'utf8' }, + }) + await window.ipc.invoke('workspace:remove', { + path: legacyPath, + opts: { trash: true }, + }) + result = legacyResult + } + if (cancelled) return + if (result.data) { + setTopics(parseTopics(result.data)) + } + } catch { + if (!cancelled) setError('No suggested topics yet. Check back once your knowledge graph has more data.') + } finally { + if (!cancelled) setLoading(false) + } + } + void load() + return () => { cancelled = true } + }, []) + + const handleTrack = useCallback( + async (topic: SuggestedTopicBlock, topicIndex: number) => { + if (removingIndex !== null) return + const nextTopics = topics.filter((_, idx) => idx !== topicIndex) + setRemovingIndex(topicIndex) + setError(null) + try { + await window.ipc.invoke('workspace:writeFile', { + path: SUGGESTED_TOPICS_PATH, + data: serializeTopics(nextTopics), + opts: { encoding: 'utf8' }, + }) + setTopics(nextTopics) + } catch (err) { + console.error('Failed to remove suggested topic:', err) + setError('Failed to update suggested topics. Please try again.') + return + } finally { + setRemovingIndex(null) + } + + onExploreTopic(topic) + }, + [onExploreTopic, removingIndex, topics], + ) + + if (loading) { + return ( +
+ +
+ ) + } + + if (error || topics.length === 0) { + return ( +
+
+ +
+

+ {error ?? 'No suggested topics yet. Check back once your knowledge graph has more data.'} +

+
+ ) + } + + return ( +
+
+
+ +

Suggested Topics

+
+

+ Suggested notes surfaced from your knowledge graph. Track one to start a tracking note. +

+
+
+
+ {topics.map((topic, i) => ( + { void handleTrack(topic, i) }} + isRemoving={removingIndex === i} + /> + ))} +
+
+
+ ) +} diff --git a/apps/x/apps/renderer/src/components/track-modal.tsx b/apps/x/apps/renderer/src/components/track-modal.tsx new file mode 100644 index 00000000..a4c0b512 --- /dev/null +++ b/apps/x/apps/renderer/src/components/track-modal.tsx @@ -0,0 +1,530 @@ +import { useCallback, useEffect, useMemo, useRef, useState } from 'react' +import { z } from 'zod' +import '@/styles/track-modal.css' +import { + Dialog, + DialogContent, + DialogHeader, + DialogTitle, + DialogDescription, + DialogFooter, +} from '@/components/ui/dialog' +import { Button } from '@/components/ui/button' +import { Switch } from '@/components/ui/switch' +import { Textarea } from '@/components/ui/textarea' +import { + Radio, Clock, Play, Loader2, Sparkles, Code2, CalendarClock, Zap, + Trash2, ChevronDown, ChevronUp, +} from 'lucide-react' +import { parse as parseYaml } from 'yaml' +import { Streamdown } from 'streamdown' +import { TrackBlockSchema, type TrackSchedule } from '@x/shared/dist/track-block.js' +import { useTrackStatus } from '@/hooks/use-track-status' +import type { OpenTrackModalDetail } from '@/extensions/track-block' + +function formatDateTime(iso: string): string { + const d = new Date(iso) + return d.toLocaleString('en-US', { + month: 'short', + day: 'numeric', + hour: 'numeric', + minute: '2-digit', + hour12: true, + }) +} + +// --------------------------------------------------------------------------- +// Schedule helpers +// --------------------------------------------------------------------------- + +const CRON_PHRASES: Record = { + '* * * * *': 'Every minute', + '*/5 * * * *': 'Every 5 minutes', + '*/15 * * * *': 'Every 15 minutes', + '*/30 * * * *': 'Every 30 minutes', + '0 * * * *': 'Hourly', + '0 */2 * * *': 'Every 2 hours', + '0 */6 * * *': 'Every 6 hours', + '0 */12 * * *': 'Every 12 hours', + '0 0 * * *': 'Daily at midnight', + '0 8 * * *': 'Daily at 8 AM', + '0 9 * * *': 'Daily at 9 AM', + '0 12 * * *': 'Daily at noon', + '0 18 * * *': 'Daily at 6 PM', + '0 9 * * 1-5': 'Weekdays at 9 AM', + '0 17 * * 1-5': 'Weekdays at 5 PM', + '0 0 * * 0': 'Sundays at midnight', + '0 0 * * 1': 'Mondays at midnight', + '0 0 1 * *': 'First of each month', +} + +function describeCron(expr: string): string { + return CRON_PHRASES[expr.trim()] ?? expr +} + +type ScheduleIconKind = 'timer' | 'calendar' | 'target' | 'bolt' +type ScheduleSummary = { icon: ScheduleIconKind; text: string } + +function summarizeSchedule(schedule?: TrackSchedule): ScheduleSummary { + if (!schedule) return { icon: 'bolt', text: 'Manual only' } + if (schedule.type === 'once') { + return { icon: 'target', text: `Once at ${formatDateTime(schedule.runAt)}` } + } + if (schedule.type === 'cron') { + return { icon: 'timer', text: describeCron(schedule.expression) } + } + if (schedule.type === 'window') { + return { icon: 'calendar', text: `${describeCron(schedule.cron)} · ${schedule.startTime}–${schedule.endTime}` } + } + return { icon: 'calendar', text: 'Scheduled' } +} + +function ScheduleIcon({ icon, size = 14 }: { icon: ScheduleIconKind; size?: number }) { + if (icon === 'timer') return + if (icon === 'calendar' || icon === 'target') return + return +} + +// --------------------------------------------------------------------------- +// Modal +// --------------------------------------------------------------------------- + +type Tab = 'what' | 'when' | 'event' | 'details' + +export function TrackModal() { + const [open, setOpen] = useState(false) + const [detail, setDetail] = useState(null) + const [yaml, setYaml] = useState('') + const [loading, setLoading] = useState(false) + const [activeTab, setActiveTab] = useState('what') + const [editingRaw, setEditingRaw] = useState(false) + const [rawDraft, setRawDraft] = useState('') + const [showAdvanced, setShowAdvanced] = useState(false) + const [confirmingDelete, setConfirmingDelete] = useState(false) + const [saving, setSaving] = useState(false) + const [error, setError] = useState(null) + const textareaRef = useRef(null) + + // Listen for the open event and seed modal state. + useEffect(() => { + const handler = (e: Event) => { + const ev = e as CustomEvent + const d = ev.detail + if (!d?.trackId || !d?.filePath) return + setDetail(d) + setYaml(d.initialYaml ?? '') + setActiveTab('what') + setEditingRaw(false) + setRawDraft('') + setShowAdvanced(false) + setConfirmingDelete(false) + setError(null) + setOpen(true) + void fetchFresh(d) + } + window.addEventListener('rowboat:open-track-modal', handler as EventListener) + return () => window.removeEventListener('rowboat:open-track-modal', handler as EventListener) + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []) + + const fetchFresh = useCallback(async (d: OpenTrackModalDetail) => { + try { + setLoading(true) + const res = await window.ipc.invoke('track:get', { trackId: d.trackId, filePath: stripKnowledgePrefix(d.filePath) }) + if (res?.success && res.yaml) { + setYaml(res.yaml) + } else if (res?.error) { + setError(res.error) + } + } catch (err) { + setError(err instanceof Error ? err.message : String(err)) + } finally { + setLoading(false) + } + }, []) + + const track = useMemo | null>(() => { + if (!yaml) return null + try { return TrackBlockSchema.parse(parseYaml(yaml)) } catch { return null } + }, [yaml]) + + const trackId = track?.trackId ?? detail?.trackId ?? '' + const instruction = track?.instruction ?? '' + const active = track?.active ?? true + const schedule = track?.schedule + const eventMatchCriteria = track?.eventMatchCriteria ?? '' + const lastRunAt = track?.lastRunAt ?? '' + const lastRunId = track?.lastRunId ?? '' + const lastRunSummary = track?.lastRunSummary ?? '' + const model = track?.model ?? '' + const provider = track?.provider ?? '' + const scheduleSummary = useMemo(() => summarizeSchedule(schedule), [schedule]) + const triggerType: 'scheduled' | 'event' | 'manual' = + schedule ? 'scheduled' : eventMatchCriteria ? 'event' : 'manual' + + const knowledgeRelPath = detail ? stripKnowledgePrefix(detail.filePath) : '' + + const allTrackStatus = useTrackStatus() + const runState = allTrackStatus.get(`${trackId}:${knowledgeRelPath}`) ?? { status: 'idle' as const } + const isRunning = runState.status === 'running' + + useEffect(() => { + if (editingRaw && textareaRef.current) { + textareaRef.current.focus() + textareaRef.current.setSelectionRange( + textareaRef.current.value.length, + textareaRef.current.value.length, + ) + } + }, [editingRaw]) + + const visibleTabs: { key: Tab; label: string; visible: boolean }[] = [ + { key: 'what', label: 'What to track', visible: true }, + { key: 'when', label: 'When to run', visible: !!schedule }, + { key: 'event', label: 'Event matching', visible: !!eventMatchCriteria }, + { key: 'details', label: 'Details', visible: true }, + ] + const shown = visibleTabs.filter(t => t.visible) + + useEffect(() => { + if (!shown.some(t => t.key === activeTab)) setActiveTab('what') + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [schedule, eventMatchCriteria]) + + // ------------------------------------------------------------------------- + // IPC-backed mutations + // ------------------------------------------------------------------------- + + const runUpdate = useCallback(async (updates: Record) => { + if (!detail) return + setSaving(true) + setError(null) + try { + const res = await window.ipc.invoke('track:update', { + trackId: detail.trackId, + filePath: stripKnowledgePrefix(detail.filePath), + updates, + }) + if (res?.success && res.yaml) { + setYaml(res.yaml) + } else if (res?.error) { + setError(res.error) + } + } catch (err) { + setError(err instanceof Error ? err.message : String(err)) + } finally { + setSaving(false) + } + }, [detail]) + + const handleToggleActive = useCallback(() => { + void runUpdate({ active: !active }) + }, [active, runUpdate]) + + const handleRun = useCallback(async () => { + if (!detail || isRunning) return + try { + await window.ipc.invoke('track:run', { + trackId: detail.trackId, + filePath: stripKnowledgePrefix(detail.filePath), + }) + } catch (err) { + setError(err instanceof Error ? err.message : String(err)) + } + }, [detail, isRunning]) + + const handleSaveRaw = useCallback(async () => { + if (!detail) return + setSaving(true) + setError(null) + try { + const res = await window.ipc.invoke('track:replaceYaml', { + trackId: detail.trackId, + filePath: stripKnowledgePrefix(detail.filePath), + yaml: rawDraft, + }) + if (res?.success && res.yaml) { + setYaml(res.yaml) + setEditingRaw(false) + } else if (res?.error) { + setError(res.error) + } + } catch (err) { + setError(err instanceof Error ? err.message : String(err)) + } finally { + setSaving(false) + } + }, [detail, rawDraft]) + + const handleDelete = useCallback(async () => { + if (!detail) return + setSaving(true) + setError(null) + try { + const res = await window.ipc.invoke('track:delete', { + trackId: detail.trackId, + filePath: stripKnowledgePrefix(detail.filePath), + }) + if (res?.success) { + // Tell the editor to remove the node so Tiptap's next save doesn't + // re-create the track block on disk. + try { detail.onDeleted() } catch { /* editor may have unmounted */ } + setOpen(false) + } else if (res?.error) { + setError(res.error) + } + } catch (err) { + setError(err instanceof Error ? err.message : String(err)) + } finally { + setSaving(false) + } + }, [detail]) + + const handleEditWithCopilot = useCallback(() => { + if (!detail) return + window.dispatchEvent(new CustomEvent('rowboat:open-copilot-edit-track', { + detail: { + trackId: detail.trackId, + filePath: detail.filePath, + }, + })) + setOpen(false) + }, [detail]) + + if (!detail) return null + + return ( + + +
+
+
+ +
+
+ + + {trackId || 'Track'} + + + + {scheduleSummary.text} + {eventMatchCriteria && triggerType === 'scheduled' && ( + · also event-driven + )} + + +
+
+
+ +
+
+ + {/* Tabs */} +
+ {shown.map(tab => ( + + ))} +
+ + {/* Body */} +
+ {loading &&
Loading latest…
} + + {activeTab === 'what' && ( +
+ {instruction + ? {instruction} + : No instruction set.} +
+ )} + + {activeTab === 'when' && schedule && ( +
+
+ + {scheduleSummary.text} +
+
+
Type
{schedule.type}
+ {schedule.type === 'cron' && ( + <> +
Expression
{schedule.expression}
+ + )} + {schedule.type === 'window' && ( + <> +
Expression
{schedule.cron}
+
Window
{schedule.startTime} – {schedule.endTime}
+ + )} + {schedule.type === 'once' && ( + <> +
Runs at
{formatDateTime(schedule.runAt)}
+ + )} +
+
+ )} + + {activeTab === 'event' && ( +
+ {eventMatchCriteria + ? {eventMatchCriteria} + : No event matching set.} +
+ )} + + {activeTab === 'details' && ( +
+
+
Track ID
{trackId}
+
File
{detail.filePath}
+
Status
{active ? 'Active' : 'Paused'}
+ {model && (<> +
Model
{model}
+ )} + {provider && (<> +
Provider
{provider}
+ )} + {lastRunAt && (<> +
Last run
{formatDateTime(lastRunAt)}
+ )} + {lastRunId && (<> +
Run ID
{lastRunId}
+ )} + {lastRunSummary && (<> +
Summary
{lastRunSummary}
+ )} +
+
+ )} + + {/* Advanced (raw YAML) — all tabs */} +
+ + {showAdvanced && ( +
+