feat(apps): copilot answers from installed apps and surfaces them

Generic app-copilot bridge (no app-specific wiring anywhere):

- app-read-data builtin: read (or list) an installed app's data/ files —
  the agent-maintained JSON the app renders. Same path confinement as
  app-set-data; 50k char cap with truncation flag; parsed JSON returned
  when possible. Attached via the app-navigation skill.
- read-view view=apps: installed apps with name/description/dataFiles/
  agentSlugs, navigating the user's screen to the Apps view (same
  show-while-telling contract as email/bg-tasks). Renderer routes the
  new view value.
- copilot instructions embed the installed-apps list (name +
  description per app) with routing guidance: prefer an app's fresh
  data over external calls, answer from it, open-app to surface it.
  The apps:list handler fingerprints the app set and invalidates the
  instructions cache when it changes (installs, deletes, copilot-
  created folders all flow through that poll).
- app-navigation skill: apps section + a worked example framing the
  pattern generically (match apps by their own descriptions, never a
  hardcoded topic map).

Verified against real data: app-read-data returns pr-dashboard's live
10-PR data.json, escape paths rejected, key-order golden test updated
and passing.
This commit is contained in:
Gagan 2026-07-10 14:23:03 +05:30
parent 214cae7190
commit 7d38dc84a2
7 changed files with 172 additions and 9 deletions

View file

@ -79,6 +79,9 @@ import * as appsPublisher from '@x/core/dist/apps/publisher.js';
// D18 install previews awaiting confirmation, keyed by app name.
const appInstallPreviews = new Map<string, Awaited<ReturnType<typeof appsInstaller.previewInstall>>>();
// Last-seen app-set fingerprint; a change invalidates the copilot
// instructions cache (they embed the installed-apps list).
let lastAppsFingerprint: string | null = null;
import { consumePendingDeepLink } from './deeplink.js';
import { qualifyAndDisconnectComposioGoogle } from '@x/core/dist/migrations/composio-google-migration.js';
import { IAgentScheduleRepo } from '@x/core/dist/agent-schedule/repo.js';
@ -1618,6 +1621,15 @@ export function setupIpcHandlers() {
for (const app of apps) {
if (app.agentSlugs.length) await appsAgents.syncAppAgents(app);
}
// The copilot instructions embed the installed-apps list. This handler
// is the one place that sees every change to the app set (installs,
// deletes, copilot-created folders — the renderer polls it), so refresh
// the instructions cache when the set actually changes.
const fingerprint = JSON.stringify(apps.map((a) => [a.folder, a.manifest?.name, a.manifest?.description, a.hasDist]));
if (fingerprint !== lastAppsFingerprint) {
lastAppsFingerprint = fingerprint;
invalidateCopilotInstructionsCache();
}
return {
serverRunning: status.running,
...(status.error ? { serverError: status.error } : {}),

View file

@ -4773,6 +4773,7 @@ function App() {
case 'knowledge': void navigateToView({ type: 'knowledge-view' }); break
case 'workspace': void navigateToView({ type: 'workspace' }); break
case 'code': void navigateToView({ type: 'code' }); break
case 'apps': openAppsView(); break
}
}

View file

@ -10,9 +10,43 @@ import { composioAccountsRepo } from "../../composio/repo.js";
import { isConfigured as isComposioConfigured } from "../../composio/client.js";
import { CURATED_TOOLKITS } from "@x/shared/dist/composio.js";
import { knowledgeSourcesRepo } from "../../knowledge/sources/repo.js";
import { listApps } from "../../apps/indexer.js";
const runtimeContextPrompt = getRuntimeContextPrompt(getRuntimeContext());
/**
* Dynamic section listing the user's Rowboat apps, so questions an app
* already tracks route to it AMBIENTLY (no discovery call). Deliberately
* generic apps are matched by their own name/description; nothing here is
* specific to any app. Cache staleness is handled by the apps:list handler
* invalidating the instructions cache when the app set changes.
*/
async function getInstalledAppsPrompt(): Promise<string> {
let apps;
try {
apps = await listApps();
} catch {
return '';
}
const usable = apps.filter((a) => a.status === 'ok' && a.hasDist);
if (usable.length === 0) return '';
const lines = usable.map((a) => {
const name = a.manifest?.name ?? a.folder;
const desc = (a.manifest?.description ?? '').slice(0, 160);
const agents = a.agentSlugs.length ? ` (self-updating via ${a.agentSlugs.length} background agent${a.agentSlugs.length > 1 ? 's' : ''})` : '';
return `- \`${a.folder}\` — **${name}**: ${desc}${agents}`;
});
return `
## Installed Rowboat Apps
The user has these Rowboat apps (mini web apps in the Apps view, each holding fresh data its background agent maintains):
${lines.join('\n')}
When a question matches what an app tracks, PREFER the app over external calls: load the \`app-navigation\` skill, read the app's data with \`app-read-data\` (fresh, instant), answer from it, and surface the app on screen with \`app-navigation\` \`open-app\` — show while telling. This applies to ANY app above; match by its name/description.
`;
}
/**
* Generate dynamic instructions section for Composio integrations.
* Lists connected toolkits and explains the meta-tool discovery flow.
@ -312,7 +346,7 @@ ${slackToolsLine}${composioToolsLine}
Every other builtin is skill-scoped load the owning skill from the catalog to attach it:
- Write-side file tools (\`file-writeText\`, \`file-editText\`, \`file-mkdir\`, \`file-rename\`, \`file-copy\`, \`file-remove\`, \`file-stat\`) via \`organize-files\`, \`doc-collab\`, and related skills
- MCP server management (\`addMcpServer\`, \`listMcpServers\`, \`listMcpTools\`, \`executeMcpTool\`) via \`mcp-integration\`
- \`app-navigation\` / \`app-set-data\` via the \`app-navigation\` skill; \`browser-control\` via the \`browser-control\` skill; notifications via \`notify-user\`; background tasks via \`background-task\`; everything else via the \`builtin-tools\` escape hatch
- \`app-navigation\` / \`app-read-data\` / \`app-set-data\` via the \`app-navigation\` skill; \`browser-control\` via the \`browser-control\` skill; notifications via \`notify-user\`; background tasks via \`background-task\`; everything else via the \`builtin-tools\` escape hatch
**Prefer these tools whenever possible.** For file operations anywhere on the machine, use file tools instead of \`executeCommand\`.
@ -404,8 +438,9 @@ export async function buildCopilotInstructions(): Promise<string> {
const catalog = await buildAvailableSkillCatalog();
const baseInstructions = buildStaticInstructions(composioEnabled, catalog, codeModeEnabled, slackConnected, slackChannelsHint, googleConnected);
const composioPrompt = await getComposioToolsPrompt(slackConnected, googleConnected);
cachedInstructions = composioPrompt
? baseInstructions + '\n' + composioPrompt
: baseInstructions;
const appsPrompt = await getInstalledAppsPrompt();
cachedInstructions = baseInstructions
+ (composioPrompt ? '\n' + composioPrompt : '')
+ (appsPrompt ? '\n' + appsPrompt : '');
return cachedInstructions;
}

View file

@ -43,6 +43,7 @@ that view so the user sees it.
any thread the search returned, including old threads outside the inbox.
- ` + "`view: \"bg-tasks\"`" + ` → background agents: ` + "`{ name, slug, active, triggers, lastRunAt, lastRunSummary, lastRunError }`" + `.
- ` + "`view: \"chat-history\"`" + ` → past chats: ` + "`{ sessionId, title, updatedAt, turnCount }`" + `.
- ` + "`view: \"apps\"`" + ` → installed Rowboat apps: ` + "`{ folder, name, description, kind, dataFiles, agentSlugs }`" + `.
- ` + "`limit`" + ` (optional, default 15).
For notes, meetings, and live notes use the ` + "`file-*`" + ` tools (they are
@ -55,9 +56,28 @@ markdown files in the workspace) and then open-note / open-item to show them.
- ` + "`kind: \"session\"`" + ` + ` + "`sessionId`" + ` (from read-view chat-history)
### open-view just switch the screen
` + "`view`" + `: ` + "`home | email | meetings | live-notes | bg-tasks | chat-history | knowledge | workspace | code | bases | graph`" + `
` + "`view`" + `: ` + "`home | email | meetings | live-notes | bg-tasks | chat-history | knowledge | workspace | code | bases | graph | apps`" + `
Use when the user asks to "go to"/"show" a view without a question to answer.
## Answering from Rowboat apps (any app match by description)
Installed Rowboat apps hold FRESH data their background agents maintain (see
the "Installed Rowboat Apps" section of your context, or ` + "`read-view view: \"apps\"`" + `).
When a question matches what an app tracks, the app IS the answer source
no external API call needed, and the user gets a visual:
1. Identify the app by its name/description (context list, or ` + "`read-view apps`" + `).
2. ` + "`app-read-data({ appFolder, file: \"data.json\" })`" + ` — omit ` + "`file`" + ` to list
what data files exist. Answer from this data, concisely.
3. ` + "`app-navigation({ action: \"open-app\", appId: appFolder })`" + ` — the app opens
on the user's screen while you answer. Show while telling.
This is GENERIC: never hardcode a mapping from topics to specific apps. A
PR-dashboard app answers "what PRs do I have?" today; a weather app answers
"what's the weekend look like?" the moment it's installed same three steps.
If the data looks stale and the app has an agent, offer to run it
(` + "`run-background-task-agent`" + ` via the ` + "`background-task`" + ` skill).
### open-note
Open a knowledge file in the editor. ` + "`path`" + `: full workspace-relative path
(e.g. ` + "`knowledge/People/John Smith.md`" + `). Use ` + "`file-grep`" + ` first if unsure
@ -90,6 +110,12 @@ Knowledge-base table control (unchanged):
1. ` + "`get-base-state`" + ` to see available categories, then
2. ` + "`update-base-view`" + ` with ` + "`filters.set: [{ category: \"relationship\", value: \"customer\" }]`" + `
**"What PRs do I have?"** (an installed app tracks open PRs)
1. The context lists an app whose description matches (e.g. ` + "`pr-dashboard`" + ` "Open PRs on …").
2. ` + "`app-read-data({ appFolder: \"pr-dashboard\", file: \"data.json\" })`" + `
3. ` + "`app-navigation({ action: \"open-app\", appId: \"pr-dashboard\" })`" + ` — the dashboard opens.
4. "Seven open — two need your review: #676 apps M3, and #675 the packaging fix."
## Notes
- read-view/open-view/open-item change what the user is looking at that is
the point, but don't bounce their screen around needlessly; navigate when

View file

@ -131,9 +131,9 @@ const definitions: SkillDefinition[] = [
{
id: "app-navigation",
title: "App Navigation",
summary: "Navigate the app UI - open notes, switch views, filter/search the knowledge base, and manage saved views.",
summary: "Navigate the app UI - open notes, switch views, answer from an installed Rowboat app's data and surface it, filter/search the knowledge base, and manage saved views.",
content: appNavigationSkill,
tools: ["app-navigation", "app-set-data"],
tools: ["app-navigation", "app-read-data", "app-set-data"],
},
{
id: "code-with-agents",

View file

@ -192,6 +192,7 @@ const HISTORICAL_KEY_ORDER = [
"composio-search-tools",
"composio-execute-tool",
"composio-connect-toolkit",
"app-read-data",
"app-set-data",
"list-models",
"fetch-url",

View file

@ -9,6 +9,7 @@ import container from "../../../di/container.js";
import * as files from "../../../filesystem/files.js";
import { WorkDir } from "../../../config/config.js";
import { RowboatAppManifestSchema } from "@x/shared/dist/rowboat-app.js";
import { listApps } from "../../../apps/indexer.js";
import { listImportantThreads, searchThreads } from "../../../knowledge/sync_gmail.js";
import { listTasks as listBackgroundTasks } from "../../../background-tasks/fileops.js";
import type { ISessions } from "../../../sessions/api.js";
@ -27,7 +28,7 @@ export const appNavigationTools: z.infer<typeof BuiltinToolsSchema> = {
// open-app
appId: z.string().optional().describe("App folder slug under ~/.rowboat/apps (for open-app) — opens the app in the middle pane."),
// open-view / read-view
view: z.enum(["home", "email", "meetings", "live-notes", "bg-tasks", "chat-history", "knowledge", "workspace", "code", "bases", "graph"]).optional().describe("Which view to open (open-view) or read (read-view; supported for read: email, bg-tasks, chat-history)"),
view: z.enum(["home", "email", "meetings", "live-notes", "bg-tasks", "chat-history", "knowledge", "workspace", "code", "bases", "graph", "apps"]).optional().describe("Which view to open (open-view) or read (read-view; supported for read: email, bg-tasks, chat-history, apps)"),
// read-view (email)
query: z.string().optional().describe("For read-view on email: runs a LIVE Gmail search over the user's ENTIRE mailbox (not just synced mail) via the Gmail API. Supports full Gmail search operators: from:, to:, subject:, before:/after:, has:attachment, quoted phrases, OR, etc. Omit to list the latest important inbox threads."),
limit: z.number().int().min(1).max(50).optional().describe("For read-view: max items to return (default 15)"),
@ -144,10 +145,33 @@ export const appNavigationTools: z.infer<typeof BuiltinToolsSchema> = {
}));
return { success: true, action: 'read-view', view, sessions };
}
case 'apps': {
// Installed/local Rowboat apps — the copilot uses this to
// route questions to an app's data (app-read-data) and to
// surface the app (open-app). Generic: apps are matched by
// their own name/description, nothing app-specific here.
const summaries = await listApps();
const apps = await Promise.all(summaries.slice(0, limit).map(async (a) => {
let dataFiles: string[] = [];
try {
const entries = await fs.readdir(path.join(WorkDir, 'apps', a.folder, 'data'), { withFileTypes: true });
dataFiles = entries.filter((e) => e.isFile()).map((e) => e.name).slice(0, 10);
} catch { /* no data dir */ }
return {
folder: a.folder,
name: a.manifest?.name ?? a.folder,
description: a.manifest?.description ?? '',
kind: a.kind,
dataFiles,
agentSlugs: a.agentSlugs,
};
}));
return { success: true, action: 'read-view', view, apps };
}
default:
return {
success: false,
error: `read-view supports: email, bg-tasks, chat-history. For notes/meetings/live-notes use the file-* tools (they are files under the workspace); for other views use open-view and describe what you need.`,
error: `read-view supports: email, bg-tasks, chat-history, apps. For notes/meetings/live-notes use the file-* tools (they are files under the workspace); for other views use open-view and describe what you need.`,
};
}
} catch (error) {
@ -287,7 +311,71 @@ export const appNavigationTools: z.infer<typeof BuiltinToolsSchema> = {
// ============================================================================,
};
// Cap what a data file read returns to the model — app data can be large
// (feeds, series); the copilot needs enough to answer, not the whole store.
const APP_READ_DATA_MAX_CHARS = 50_000;
export const appDataTools: z.infer<typeof BuiltinToolsSchema> = {
'app-read-data': {
description: "Read a Rowboat App's data file — the JSON its background agent maintains and its frontend renders. THE way to answer questions an installed app already tracks (fresh, no API calls): find the app via app-navigation read-view apps, read its data file, answer from it. Omit `file` to list the files under the app's data/.",
inputSchema: z.object({
appFolder: z.string().describe('The app folder slug under ~/.rowboat/apps.'),
file: z.string().optional().describe("Path relative to the app's data/ directory, e.g. \"data.json\". Omit to list available files."),
}),
execute: async ({ appFolder, file }: { appFolder: string; file?: string }) => {
try {
const dir = path.join(WorkDir, 'apps', appFolder);
try {
RowboatAppManifestSchema.parse(JSON.parse(await fs.readFile(path.join(dir, 'rowboat-app.json'), 'utf-8')));
} catch {
return { success: false, error: `No app "${appFolder}" (missing or invalid rowboat-app.json).` };
}
const dataRoot = path.join(dir, 'data');
if (!file) {
try {
const entries = await fs.readdir(dataRoot, { withFileTypes: true });
const files = await Promise.all(entries.filter((e) => e.isFile()).map(async (e) => {
const stat = await fs.stat(path.join(dataRoot, e.name)).catch(() => null);
return { file: e.name, size: stat?.size ?? 0, mtime: stat ? new Date(stat.mtimeMs).toISOString() : '' };
}));
return { success: true, appFolder, files };
} catch {
return { success: true, appFolder, files: [] };
}
}
// Same path confinement rules as app-set-data / the data API.
const relNorm = path.posix.normalize(file).replace(/^\/+/, '');
if (!relNorm || relNorm === '.' || relNorm.startsWith('..') || relNorm.includes('\0') || relNorm.includes('\\')) {
return { success: false, error: `invalid file path: ${file}` };
}
const abs = path.resolve(dataRoot, relNorm);
if (abs !== dataRoot && !abs.startsWith(dataRoot + path.sep)) {
return { success: false, error: `file path escapes data/: ${file}` };
}
let text: string;
try {
text = await fs.readFile(abs, 'utf-8');
} catch {
return { success: false, error: `no such data file: ${relNorm} (omit \`file\` to list what exists)` };
}
const truncated = text.length > APP_READ_DATA_MAX_CHARS;
if (truncated) text = text.slice(0, APP_READ_DATA_MAX_CHARS);
// Parsed JSON is easiest for the model to reason over; fall back
// to raw text for non-JSON (or truncated-mid-JSON) content.
if (!truncated) {
try {
return { success: true, appFolder, file: relNorm, data: JSON.parse(text) as unknown };
} catch { /* not JSON — return as text */ }
}
return { success: true, appFolder, file: relNorm, text, ...(truncated ? { truncated: true } : {}) };
} catch (e) {
return { success: false, error: e instanceof Error ? e.message : String(e) };
}
},
},
'app-set-data': {
description: "Write a Rowboat App's data file — JSON its frontend reads via GET /_rowboat/data/<file>. Deterministic: you supply the content, code handles the path, atomicity (temp→rename), and the app's dataContracts validation. This is how a background task refreshes an app's data — the agent RETURNS the data; never hand-write files under apps/.",
inputSchema: z.object({