diff --git a/README.md b/README.md index 45c20bd9..086499d0 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ The context layer for analytics agents +

by Kaelio

+

npm version Codecov @@ -32,7 +34,7 @@ SQLite. ## Quick Start ```bash -pnpm add --global @kaelio/ktx +npm install -g @kaelio/ktx ktx setup ktx status ``` @@ -40,6 +42,19 @@ ktx status `ktx setup` creates or resumes a local KTX project, configures providers and connections, builds context, and installs agent integration. +Example `ktx status` output after setup: + +```text +KTX project: /home/user/analytics +Project ready: yes +LLM ready: yes (claude-sonnet-4-6) +Embeddings ready: yes (text-embedding-3-small) +Databases configured: yes (postgres-warehouse) +Context sources configured: yes (dbt-main) +KTX context built: yes +Agent integration ready: yes (codex:project) +``` + ## Common Commands | Command | Purpose | @@ -92,6 +107,49 @@ ktx wiki search "refund policy" --json ktx sl query --connection-id warehouse --measure orders.revenue --format sql ``` +During agent setup, choose **MCP tools + analytics skill** for client agents. +Choose **MCP tools + analytics skill + admin CLI skill** only when a developer +or operator agent also needs pinned `ktx` admin commands. + +The analytics skill teaches client agents the MCP workflow: discover data, +prefer semantic-layer measures, inspect entity details before raw SQL, and +capture durable learnings. Admin CLI skills call `ktx` commands directly +through a skill file installed in your agent's config: + +```bash +ktx sl query --measure orders.revenue --dimension orders.status --format sql +ktx wiki search "revenue definition" +ktx sl validate orders +``` + +Supported client agents: Claude Code, Claude Desktop, Codex, Cursor, OpenCode, +and clients that can use the printed MCP endpoint or `.agents` admin skills. +Claude Desktop setup registers a local `ktx mcp stdio` server in Claude +Desktop's config and generates `.ktx/agents/claude/ktx-plugin.zip` with the +analytics skill. + +The release artifact manifest contains the public npm tarball and the bundled +`kaelio-ktx` runtime wheel. The `python/ktx-sl` and `python/ktx-daemon` +directories remain source packages for development, not public release +artifacts. + +## Workspace packages + +| Package | Purpose | +|---------|---------| +| `packages/cli` | CLI entry point | +| `packages/context` | Core context engine | +| `packages/llm` | LLM and embedding providers | +| `packages/connector-bigquery` | BigQuery scan connector | +| `packages/connector-clickhouse` | ClickHouse scan connector | +| `packages/connector-mysql` | MySQL scan connector | +| `packages/connector-postgres` | Postgres scan connector | +| `packages/connector-snowflake` | Snowflake scan connector | +| `packages/connector-sqlite` | SQLite scan connector | +| `packages/connector-sqlserver` | SQL Server scan connector | +| `python/ktx-sl` | Semantic-layer query planning | +| `python/ktx-daemon` | Portable compute service | + ## Development ```bash diff --git a/docs-site/content/docs/integrations/agent-clients.mdx b/docs-site/content/docs/integrations/agent-clients.mdx index 01cbbca5..1934c978 100644 --- a/docs-site/content/docs/integrations/agent-clients.mdx +++ b/docs-site/content/docs/integrations/agent-clients.mdx @@ -1,18 +1,26 @@ --- title: Agent Clients -description: Set up KTX with Claude Code, Cursor, Codex, and OpenCode. +description: Set up KTX with Claude Code, Claude Desktop, Cursor, Codex, and OpenCode. --- -KTX integrates with coding agents through CLI skills and command files. These -files teach agents to call public `ktx` commands directly from the terminal for -semantic-layer context and wiki knowledge. +KTX exposes context to end-user agents through MCP tools. The CLI remains the +admin surface for setup, ingest, status, daemon lifecycle, and debugging. -Run `ktx setup` and select your agent targets, or configure manually using the -snippets below. Setup pins generated skill files to the KTX CLI path that -created them, so agents do not need `ktx` on `PATH`. +Run `ktx setup` and select your client agent targets, or configure manually +using the snippets below. Choose **MCP tools + analytics skill** for client +agents. Choose **MCP tools + analytics skill + admin CLI skill** only when a +developer or operator agent also needs pinned `ktx` admin commands. ## Install with setup +Start the MCP server before connecting an end-user agent: + +```bash +ktx mcp start +``` + +Then install client integration: + ```bash ktx setup --agents ``` @@ -23,7 +31,8 @@ Use `--target` for one target: ktx setup --agents --target codex ``` -Use `--global` only with `claude-code` or `codex`: +Use `--global` only with `claude-code` or `codex`. Claude Desktop always +generates a project-local plugin ZIP: ```bash ktx setup --agents --target claude-code --global @@ -34,19 +43,54 @@ KTX records installed files in `.ktx/agents/install-manifest.json`. That manifest lets status checks report agent readiness and lets future cleanup remove only files KTX installed. +The interactive command asks two questions: + +```txt +◆ How should client agents connect to this KTX project? +│ ○ MCP tools + analytics skill +│ ○ MCP tools + analytics skill + admin CLI skill +└ + +◆ Which agent targets should KTX install? +│ ◻ Claude Code +│ ◻ Claude Desktop +│ ◻ Codex +│ ◻ Cursor +│ ◻ OpenCode +│ ◻ Universal .agents +└ +``` + +When every selected target supports both project and global setup, the command +also asks where to install supported agent config: + +```txt +◆ Where should KTX install supported agent config? +│ ○ Project +│ ○ Global +└ +``` + ## Generated files -| Target | Project-scoped files | Global files | -|--------|----------------------|--------------| -| Claude Code | `.claude/skills/ktx/SKILL.md`, `.claude/rules/ktx.md` | `~/.claude/skills/ktx/SKILL.md`, `~/.claude/rules/ktx.md` | -| Codex | `.agents/skills/ktx/SKILL.md`, `.codex/instructions/ktx.md` | `$CODEX_HOME/skills/ktx/SKILL.md`, `$CODEX_HOME/instructions/ktx.md` | -| Cursor | `.cursor/rules/ktx.mdc` | Not supported | -| OpenCode | `.opencode/commands/ktx.md` | Not supported | -| Universal `.agents` | `.agents/skills/ktx/SKILL.md` | Not supported | +KTX writes MCP client configuration and an analytics skill by default. It writes +admin CLI skills only when you choose **MCP tools + analytics skill + admin CLI +skill**. -Skill files list pinned `ktx` commands. Rule files tell the agent when KTX is -appropriate, such as data schemas, metrics, dimensions, database structure, and -SQL questions. +| Target | MCP tools + analytics skill | Adds with admin CLI skill | +|--------|------------------------------|---------------------------| +| Claude Code | `.mcp.json`, `.claude/skills/ktx-analytics/SKILL.md` | `.claude/skills/ktx/SKILL.md`, `.claude/rules/ktx.md` | +| Claude Desktop | `~/Library/Application Support/Claude/claude_desktop_config.json` stdio entry + `.ktx/agents/claude/ktx-plugin.zip` with analytics skill | Adds `skills/ktx/SKILL.md` inside the plugin ZIP | +| Codex | Printed snippet for `~/.codex/config.toml`, `.agents/skills/ktx-analytics/SKILL.md` | `.agents/skills/ktx/SKILL.md`, `.codex/instructions/ktx.md` | +| Cursor | `.cursor/mcp.json`, `.cursor/rules/ktx-analytics.mdc` | `.cursor/rules/ktx.mdc` | +| OpenCode | Printed snippet for `opencode.json`, `.opencode/commands/ktx-analytics.md` | `.opencode/commands/ktx.md` | +| Universal `.agents` | Printed MCP endpoint, `.agents/skills/ktx-analytics/SKILL.md` | `.agents/skills/ktx/SKILL.md` | + +MCP config gives agents access to KTX context tools such as discovery, +semantic-layer queries, wiki search, SQL execution, and memory ingest. The +analytics skill explains how to use those tools for semantic-layer-first +analysis. Optional admin skill and rule files list pinned CLI commands for +developer or operator agents. ## Claude Code @@ -56,13 +100,16 @@ During setup, select **Claude Code** from the agent targets. KTX writes: | Scope | Files | |-------|-------| -| Project | `.claude/skills/ktx/SKILL.md`, `.claude/rules/ktx.md` | -| Global | `~/.claude/skills/ktx/SKILL.md`, `~/.claude/rules/ktx.md` | +| Project | `.mcp.json`, `.claude/skills/ktx-analytics/SKILL.md`; optional `.claude/skills/ktx/SKILL.md`, `.claude/rules/ktx.md` | +| Global | `~/.claude.json`, `~/.claude/skills/ktx-analytics/SKILL.md`; optional `~/.claude/skills/ktx/SKILL.md`, `~/.claude/rules/ktx.md` | Both project-scoped and global installations are supported. ### Manual CLI skills configuration +Use manual CLI skills only for developer or operator agents that need admin +commands. End-user data agents use MCP. + Create `.claude/skills/ktx/SKILL.md`: ```markdown title=".claude/skills/ktx/SKILL.md" @@ -82,6 +129,7 @@ Available commands: ### Workflow tips - Claude Code discovers skills automatically from `.claude/skills/`. +- Claude Code reads MCP config from `.mcp.json` for project-scoped MCP tools. - Claude rules in `.claude/rules/` tell Claude when KTX should be used. - Global installation makes KTX available in all projects without per-project setup. - Keep generated skills committed only when your team wants project-local agent instructions in git. @@ -96,13 +144,19 @@ During setup, select **Cursor** from the agent targets. KTX writes: | Mode | File | |------|------| -| CLI rules | `.cursor/rules/ktx.mdc` | +| MCP tools + analytics skill | `.cursor/mcp.json`, `.cursor/rules/ktx-analytics.mdc` | +| Admin CLI rules | `.cursor/rules/ktx.mdc` | Cursor supports project-scoped installation only. ### Manual CLI rules configuration -Create `.cursor/rules/ktx.mdc` with the same content structure as the Claude Code `SKILL.md` file. Cursor rules use the `.mdc` extension but support the same markdown command definitions. +Use manual CLI rules only for developer or operator agents that need admin +commands. End-user data agents use MCP. + +Create `.cursor/rules/ktx.mdc` with the same content structure as the Claude +Code `SKILL.md` file. Cursor rules use the `.mdc` extension but support the +same markdown command definitions. ### Workflow tips @@ -111,6 +165,37 @@ Create `.cursor/rules/ktx.mdc` with the same content structure as the Claude Cod --- +## Claude Desktop + +During setup, select **Claude Desktop** from the agent targets. KTX writes the +MCP server entry directly into Claude Desktop's config and generates a separate +plugin ZIP for the analytics skill: + +- `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or + `%AppData%/Claude/claude_desktop_config.json` (Windows) gets an + `mcpServers.ktx` entry that runs the KTX MCP server over stdio via a local + launcher shim at `.ktx/agents/claude/ktx-plugin-runner.sh`. The shim locates + a usable Node.js (Volta, NVM, Homebrew, system) so Claude Desktop can spawn + the server without needing `node` in PATH. +- `.ktx/agents/claude/ktx-plugin.zip` contains the `ktx-analytics` skill (and + the admin `ktx` skill if you choose **MCP tools + analytics skill + admin + CLI skill**). Install the ZIP from Claude Desktop's plugin UI to load the + skill. + +After `ktx setup`, restart Claude Desktop so it picks up the new MCP server +entry, then install the plugin ZIP. No daemon needs to be running — Claude +Desktop spawns the MCP server itself per session. + +Claude Desktop does not introspect local stdio MCP servers, so the per-tool +"Connector"-style UI is not rendered for KTX. The tools are still callable +from any Claude Desktop chat. + +If you move the KTX checkout or project directory, rerun `ktx setup --agents` +to refresh the absolute paths in `claude_desktop_config.json` and the launcher +shim, then reinstall the regenerated plugin ZIP. + +--- + ## Codex ### Install via `ktx setup` @@ -119,15 +204,19 @@ During setup, select **Codex** from the agent targets. KTX writes: | Scope | Files | |-------|-------| -| Project | `.agents/skills/ktx/SKILL.md`, `.codex/instructions/ktx.md` | -| Global | `$CODEX_HOME/skills/ktx/SKILL.md`, `$CODEX_HOME/instructions/ktx.md` | +| Project | MCP snippet, `.agents/skills/ktx-analytics/SKILL.md`; optional `.agents/skills/ktx/SKILL.md`, `.codex/instructions/ktx.md` | +| Global | MCP snippet, `$CODEX_HOME/skills/ktx-analytics/SKILL.md`; optional `$CODEX_HOME/skills/ktx/SKILL.md`, `$CODEX_HOME/instructions/ktx.md` | Both project-scoped and global installations are supported. `CODEX_HOME` defaults to `~/.codex`. ### Manual CLI skills configuration -Create `.agents/skills/ktx/SKILL.md` with the same content structure as Claude Code's `SKILL.md`. +Use manual CLI skills only for developer or operator agents that need admin +commands. End-user data agents use MCP. + +Create `.agents/skills/ktx/SKILL.md` with the same content structure as Claude +Code's `SKILL.md`. ### Workflow tips @@ -146,13 +235,18 @@ During setup, select **OpenCode** from the agent targets. KTX writes: | Mode | File | |------|------| -| CLI commands | `.opencode/commands/ktx.md` | +| MCP tools + analytics skill | Snippet for `opencode.json`, `.opencode/commands/ktx-analytics.md` | +| Admin CLI commands | `.opencode/commands/ktx.md` | OpenCode supports project-scoped installation only. ### Manual CLI commands configuration -Create `.opencode/commands/ktx.md` with the same command definitions as Claude Code's `SKILL.md`. +Use manual CLI commands only for developer or operator agents that need admin +commands. End-user data agents use MCP. + +Create `.opencode/commands/ktx.md` with the same command definitions as Claude +Code's `SKILL.md`. ### Workflow tips @@ -163,7 +257,7 @@ Create `.opencode/commands/ktx.md` with the same command definitions as Claude C ## Command reference -All supported agent clients call the same KTX CLI commands: +Admin CLI skills call the same KTX CLI commands: | Command | Description | |---------|-------------| @@ -183,9 +277,11 @@ All supported agent clients call the same KTX CLI commands: ## Comparison -| | Claude Code | Cursor | Codex | OpenCode | -|---|---|---|---|---| -| CLI skills | Yes | Yes (.mdc) | Yes | Yes | -| Global install | Yes | No | Yes | No | -| Rule or instruction file | `.claude/rules/ktx.md` | `.cursor/rules/ktx.mdc` | `.codex/instructions/ktx.md` | `.opencode/commands/ktx.md` | -| Skill file | `.claude/skills/ktx/SKILL.md` | Not separate | `.agents/skills/ktx/SKILL.md` | Not separate | +| | Claude Code | Claude Desktop | Cursor | Codex | OpenCode | +|---|---|---|---|---|---| +| MCP tools | Yes | Local stdio via `claude_desktop_config.json` | Yes | Snippet | Snippet | +| Analytics skill | `.claude/skills/ktx-analytics/SKILL.md` | Included in plugin ZIP | `.cursor/rules/ktx-analytics.mdc` | `.agents/skills/ktx-analytics/SKILL.md` | `.opencode/commands/ktx-analytics.md` | +| Admin CLI skills | Optional | Optional in plugin ZIP | Optional (.mdc) | Optional | Optional | +| Global install | Yes | Project-local ZIP | No | Yes | No | +| Rule or instruction file | `.claude/rules/ktx.md` | Plugin `SETUP.md` | `.cursor/rules/ktx.mdc` | `.codex/instructions/ktx.md` | `.opencode/commands/ktx.md` | +| Skill file | `.claude/skills/ktx/SKILL.md` | `skills/ktx/SKILL.md` in plugin ZIP | Not separate | `.agents/skills/ktx/SKILL.md` | Not separate | diff --git a/docs-site/next-env.d.ts b/docs-site/next-env.d.ts index c4b7818f..9edff1c7 100644 --- a/docs-site/next-env.d.ts +++ b/docs-site/next-env.d.ts @@ -1,6 +1,6 @@ /// /// -import "./.next/dev/types/routes.d.ts"; +import "./.next/types/routes.d.ts"; // NOTE: This file should not be edited // see https://nextjs.org/docs/app/api-reference/config/typescript for more information. diff --git a/docs/superpowers/plans/2026-05-14-research-agent-mcp-discover-data.md b/docs/superpowers/plans/2026-05-14-research-agent-mcp-discover-data.md index a917eb72..351aed67 100644 --- a/docs/superpowers/plans/2026-05-14-research-agent-mcp-discover-data.md +++ b/docs/superpowers/plans/2026-05-14-research-agent-mcp-discover-data.md @@ -903,14 +903,16 @@ Add this test after the `dictionary_search` registration test: limit: 5, }), ).resolves.toMatchObject({ - structuredContent: [ - { - kind: 'table', - id: 'public.orders', - connectionId: 'warehouse', - tableRef: { catalog: null, db: 'public', name: 'orders' }, - }, - ], + structuredContent: { + refs: [ + { + kind: 'table', + id: 'public.orders', + connectionId: 'warehouse', + tableRef: { catalog: null, db: 'public', name: 'orders' }, + }, + ], + }, }); expect(discover.search).toHaveBeenCalledWith({ query: 'orders', diff --git a/docs/superpowers/plans/2026-05-16-mcp-tool-polish-v1-metadata-progress.md b/docs/superpowers/plans/2026-05-16-mcp-tool-polish-v1-metadata-progress.md new file mode 100644 index 00000000..28221089 --- /dev/null +++ b/docs/superpowers/plans/2026-05-16-mcp-tool-polish-v1-metadata-progress.md @@ -0,0 +1,1459 @@ +# MCP Tool Polish V1 Metadata and Progress Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Finish the remaining v1-blocking MCP polish work after the surface +change: tool metadata, schemas, in-band errors, normalization, resolved-source +invariants, and progress notifications. + +**Architecture:** Keep the 11-tool research surface already implemented. Add +metadata and output schemas through the shared `registerParsedTool` path, keep +runtime handlers small, and plumb progress as optional callbacks through the +MCP ports that execute work. + +**Tech Stack:** TypeScript, Zod v4, MCP SDK 1.29, Vitest, pnpm workspace +commands. + +--- + +## Audit summary + +The original spec is +`docs/superpowers/specs/2026-05-16-mcp-tool-polish-design.md`. + +Already implemented by +`docs/superpowers/plans/2026-05-16-mcp-tool-polish-v1-surface-change.md`: + +- The MCP surface is reduced to 11 registered tools in + `packages/context/src/mcp/context-tools.ts`. +- `memory_capture` and `memory_capture_status` are replaced by + `memory_ingest` and `memory_ingest_status`. +- Memory ingest runs through `registerKtxContextTools`, so it shares the same + registration path as the other retained tools. +- `packages/cli/src/skills/analytics/SKILL.md` uses `memory_ingest` and + documents the multi-connection rule. +- `docs-site/content/docs/integrations/agent-clients.mdx` says memory ingest. + +Remaining v1-blocking gaps covered by this plan: + +- Add MCP tool annotations and `outputSchema` for all 11 retained tools. +- Add `.describe()` to every input field and rewrite tool descriptions with + concrete argument examples. +- Move in-band runtime error wrapping into `registerParsedTool` and remove the + local `sql_execution` catch. +- Normalize `sl_query.dimensions` Cube-style `{ dimension, granularity }`. +- Normalize `entity_details.entities[].table` SQL-style + `{ schema, table }` into `{ catalog: null, db: schema, name: table }`. +- Type-narrow `jsonToolResult` so bare arrays do not type-check. +- Add the `toResolvedWire` invariant comment and narrow compute-port source + types to resolved sources. +- Emit progress notifications for `sql_execution` and `sl_query` when the MCP + request includes `_meta.progressToken`. + +Non-blocking gaps left outside this plan: + +- Delete admin tool implementation code after a future `ktx-admin` skill lands. +- MCP resources, MCP prompts, elicitation, sampling, tool icons, code execution, + multi-tenancy, telemetry, and rate limiting. +- More exhaustive multi-client manual smoke beyond the automated in-memory MCP + SDK coverage in this plan. + +## File structure + +- `packages/context/src/mcp/types.ts`: expand the local MCP server facade with + output schemas, annotations, handler context, and progress callback types. +- `packages/context/src/mcp/context-tools.ts`: add output schemas, annotations, + input descriptions, tool descriptions, centralized error wrapping, + normalization, type-narrowed `jsonToolResult`, and progress callback wiring. +- `packages/context/src/mcp/server.test.ts`: add schema, annotation, + normalization, in-band error, progress, and type-narrowing coverage. +- `packages/context/src/daemon/semantic-layer-compute.ts`: document and type + the resolved-source invariant for daemon-backed semantic-layer calls. +- `packages/context/src/sl/local-query.ts`: accept an optional progress + callback and emit semantic-layer query stages. +- `packages/context/src/mcp/local-project-ports.ts`: pass progress callbacks + into `compileLocalSlQuery` and emit SQL execution stages. +- `packages/context/src/mcp/local-project-ports.test.ts`: verify local port + progress stages. +- `packages/context/src/sl/local-query.test.ts`: verify compile and execution + progress stages. + +### Task 1: Add failing MCP metadata, schema, normalization, error, and progress tests + +**Files:** + +- Modify: `packages/context/src/mcp/server.test.ts` + +- [ ] **Step 1: Update imports and fake server types** + +In `packages/context/src/mcp/server.test.ts`, replace the import from +`./server.js` and the MCP type import with: + +```typescript +import { Client } from '@modelcontextprotocol/sdk/client/index.js'; +import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js'; +import { z } from 'zod'; +import { createDefaultKtxMcpServer, createKtxMcpServer } from './server.js'; +import { jsonToolResult } from './context-tools.js'; +import type { + KtxDiscoverDataMcpPort, + KtxDictionarySearchMcpPort, + KtxEntityDetailsMcpPort, + KtxKnowledgeMcpPort, + KtxMcpContextPorts, + KtxMcpToolHandlerContext, + KtxSemanticLayerMcpPort, + KtxSqlExecutionMcpPort, + KtxSqlExecutionResponse, + MemoryIngestPort, +} from './types.js'; +``` + +Replace the `RegisteredTool` type with: + +```typescript +type RegisteredTool = { + name: string; + config: { + title?: string; + description?: string; + inputSchema: unknown; + outputSchema?: unknown; + annotations?: Record; + }; + handler: (input: Record, context?: KtxMcpToolHandlerContext) => Promise; +}; +``` + +- [ ] **Step 2: Add shared test helpers** + +After `getTool`, add: + +```typescript +const retainedToolNames = [ + 'connection_list', + 'dictionary_search', + 'discover_data', + 'entity_details', + 'memory_ingest', + 'memory_ingest_status', + 'sl_query', + 'sl_read_source', + 'sql_execution', + 'wiki_read', + 'wiki_search', +] as const; + +function makeAllContextTools(): KtxMcpContextPorts { + return { + connections: { + list: vi.fn().mockResolvedValue([{ id: 'warehouse', name: 'Warehouse', connectionType: 'POSTGRES' }]), + }, + knowledge: { + search: vi.fn().mockResolvedValue({ results: [], totalFound: 0 }), + read: vi.fn().mockResolvedValue({ + key: 'revenue', + summary: 'Paid order value', + content: '# Revenue', + scope: 'GLOBAL', + tags: ['finance'], + refs: [], + slRefs: ['orders'], + }), + }, + semanticLayer: { + readSource: vi.fn().mockResolvedValue({ + sourceName: 'orders', + yaml: 'name: orders\n', + }), + query: vi.fn().mockResolvedValue({ + sql: 'select 1', + headers: ['count'], + rows: [[1]], + totalRows: 1, + plan: { sources: ['orders'] }, + }), + }, + entityDetails: { + read: vi.fn().mockResolvedValue({ results: [] }), + }, + dictionarySearch: { + search: vi.fn().mockResolvedValue({ searched: [], results: [] }), + }, + discover: { + search: vi.fn().mockResolvedValue([]), + }, + sqlExecution: { + execute: vi.fn().mockResolvedValue({ + headers: ['count'], + headerTypes: ['integer'], + rows: [[1]], + rowCount: 1, + }), + }, + memoryIngest: { + ingest: vi.fn().mockResolvedValue({ runId: 'run-1' }), + status: vi.fn().mockResolvedValue({ + runId: 'run-1', + status: 'done', + stage: 'done', + done: true, + captured: { wiki: [], sl: [], xrefs: [] }, + error: null, + commitHash: null, + skillsLoaded: [], + signalDetected: false, + }), + }, + }; +} + +async function listToolsThroughSdk(contextTools: KtxMcpContextPorts) { + const server = createDefaultKtxMcpServer({ + name: 'ktx-test', + version: '0.0.0-test', + userContext: { userId: 'mcp-user' }, + contextTools, + }); + const client = new Client({ name: 'ktx-test-client', version: '0.0.0-test' }); + const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); + + await Promise.all([server.connect(serverTransport), client.connect(clientTransport)]); + try { + return await client.listTools(); + } finally { + await client.close(); + await server.close(); + } +} +``` + +- [ ] **Step 3: Add annotations and output schema assertions** + +Inside `describe('createKtxMcpServer', () => {`, add: + +```typescript + it('registers annotations and output schemas for every retained tool', async () => { + const fake = makeFakeServer(); + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'mcp-user' }, + contextTools: makeAllContextTools(), + }); + + expect(fake.tools.map((tool) => tool.name).sort()).toEqual([...retainedToolNames].sort()); + + const expectedAnnotations: Record> = { + connection_list: { title: 'Connection List', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + discover_data: { title: 'Discover Data', readOnlyHint: true, openWorldHint: false }, + wiki_search: { title: 'Wiki Search', readOnlyHint: true, openWorldHint: false }, + wiki_read: { title: 'Wiki Read', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + entity_details: { title: 'Entity Details', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + dictionary_search: { title: 'Dictionary Search', readOnlyHint: true, openWorldHint: false }, + sl_read_source: { + title: 'Semantic Layer Read Source', + readOnlyHint: true, + idempotentHint: true, + openWorldHint: false, + }, + sl_query: { title: 'Semantic Layer Query', readOnlyHint: true, openWorldHint: false }, + sql_execution: { title: 'SQL Execution', readOnlyHint: true, openWorldHint: false }, + memory_ingest: { title: 'Memory Ingest', destructiveHint: true, openWorldHint: false }, + memory_ingest_status: { title: 'Memory Ingest Status', readOnlyHint: true, openWorldHint: false }, + }; + + for (const toolName of retainedToolNames) { + const tool = getTool(fake.tools, toolName); + expect(tool.config.title).toBe(expectedAnnotations[toolName]?.title); + expect(tool.config.annotations).toEqual(expectedAnnotations[toolName]); + expect(tool.config.outputSchema).toBeDefined(); + const inputShape = tool.config.inputSchema as Record; + for (const inputSchema of Object.values(inputShape)) { + expect(inputSchema.description).toEqual(expect.any(String)); + } + } + }); +``` + +- [ ] **Step 4: Add the SDK tools/list schema snapshot test** + +Add: + +```typescript + it('exposes annotations and output schemas through the SDK tools/list response', async () => { + const result = await listToolsThroughSdk(makeAllContextTools()); + const toolNames = result.tools.map((tool) => tool.name).sort(); + expect(toolNames).toEqual([...retainedToolNames].sort()); + + await expect(result.tools).toMatchFileSnapshot('__snapshots__/mcp-tools-list.json'); + }); +``` + +- [ ] **Step 5: Add normalization tests for the two remaining drift shapes** + +Add: + +```typescript + it('sl_query normalizes cube-style dimensions to field dimensions', async () => { + const fake = makeFakeServer(); + const semanticLayer = makeAllContextTools().semanticLayer!; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { semanticLayer }, + }); + + await getTool(fake.tools, 'sl_query').handler({ + connectionId: 'warehouse', + measures: ['orders.count'], + dimensions: [{ dimension: 'orders.created_at', granularity: 'month' }, 'orders.status'], + }); + + expect(semanticLayer.query).toHaveBeenCalledWith( + { + connectionId: 'warehouse', + query: expect.objectContaining({ + dimensions: [{ field: 'orders.created_at', granularity: 'month' }, { field: 'orders.status' }], + }), + }, + undefined, + ); + }); + + it('entity_details normalizes sql-style schema table refs', async () => { + const fake = makeFakeServer(); + const entityDetails = makeAllContextTools().entityDetails!; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { entityDetails }, + }); + + await getTool(fake.tools, 'entity_details').handler({ + connectionId: 'warehouse', + entities: [{ table: { schema: 'public', table: 'orders' }, columns: ['id'] }], + }); + + expect(entityDetails.read).toHaveBeenCalledWith({ + connectionId: 'warehouse', + entities: [{ table: { catalog: null, db: 'public', name: 'orders' }, columns: ['id'] }], + }); + }); +``` + +- [ ] **Step 6: Add centralized runtime error wrapping tests** + +Add: + +```typescript + it('wraps handler exceptions in-band for non-sql tools', async () => { + const fake = makeFakeServer(); + const knowledge: KtxKnowledgeMcpPort = { + search: vi.fn().mockRejectedValue(new Error('wiki index unavailable')), + read: vi.fn(), + }; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { knowledge }, + }); + + await expect(getTool(fake.tools, 'wiki_search').handler({ query: 'revenue' })).resolves.toEqual({ + content: [{ type: 'text', text: 'wiki index unavailable' }], + isError: true, + }); + }); +``` + +- [ ] **Step 7: Add MCP progress notification tests** + +Add: + +```typescript + it('wires sql_execution progress to MCP notifications when a progress token is present', async () => { + const fake = makeFakeServer(); + const notifications: unknown[] = []; + const sqlExecution: KtxSqlExecutionMcpPort = { + execute: vi.fn().mockImplementation(async (_input, options) => { + await options?.onProgress?.({ progress: 0, message: 'Validating SQL' }); + await options?.onProgress?.({ progress: 0.3, message: 'Executing' }); + await options?.onProgress?.({ progress: 1, message: 'Fetched 1 rows' }); + return { headers: ['count'], rows: [[1]], rowCount: 1 }; + }), + }; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { sqlExecution }, + }); + + await getTool(fake.tools, 'sql_execution').handler( + { connectionId: 'warehouse', sql: 'select 1' }, + { + _meta: { progressToken: 'progress-1' }, + sendNotification: async (notification) => { + notifications.push(notification); + }, + }, + ); + + expect(notifications).toEqual([ + { + method: 'notifications/progress', + params: { progressToken: 'progress-1', progress: 0, message: 'Validating SQL' }, + }, + { + method: 'notifications/progress', + params: { progressToken: 'progress-1', progress: 0.3, message: 'Executing' }, + }, + { + method: 'notifications/progress', + params: { progressToken: 'progress-1', progress: 1, message: 'Fetched 1 rows' }, + }, + ]); + }); +``` + +- [ ] **Step 8: Add the compile-time array rejection assertion** + +Add this test near the bottom of the describe block: + +```typescript + it('keeps jsonToolResult typed to non-array objects', () => { + expect(jsonToolResult({ ok: true }).structuredContent).toEqual({ ok: true }); + + if (false) { + // @ts-expect-error bare arrays are not valid MCP structuredContent objects in KTX + jsonToolResult([]); + } + }); +``` + +- [ ] **Step 9: Run MCP tests and confirm they fail** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts +``` + +Expected: FAIL with missing annotations, missing output schemas, missing +normalization, missing centralized error wrapping, missing progress callback +wiring, and a missing snapshot. + +### Task 2: Implement MCP annotations, output schemas, descriptions, normalization, and in-band error wrapping + +**Files:** + +- Modify: `packages/context/src/mcp/types.ts` +- Modify: `packages/context/src/mcp/context-tools.ts` +- Modify: `packages/context/src/mcp/server.test.ts` + +- [ ] **Step 1: Extend MCP facade types** + +In `packages/context/src/mcp/types.ts`, replace `KtxMcpToolResult`, +`KtxMcpServerLike`, `KtxSemanticLayerMcpPort`, and +`KtxSqlExecutionMcpPort` with: + +```typescript +export type NonArrayObject = object & { length?: never }; + +export interface KtxMcpTextContent { + type: 'text'; + text: string; +} + +export interface KtxMcpToolResult { + content: KtxMcpTextContent[]; + structuredContent?: T; + isError?: true; +} + +export interface KtxMcpProgressEvent { + progress: number; + total?: number; + message: string; +} + +export type KtxMcpProgressCallback = (event: KtxMcpProgressEvent) => void | Promise; + +export interface KtxMcpToolHandlerContext { + _meta?: { progressToken?: string | number; [key: string]: unknown }; + sendNotification?: (notification: { + method: 'notifications/progress'; + params: { + progressToken: string | number; + progress: number; + total?: number; + message?: string; + }; + }) => Promise; +} + +export interface KtxMcpServerLike { + registerTool( + name: string, + config: { + title?: string; + description?: string; + inputSchema: unknown; + outputSchema?: unknown; + annotations?: Record; + }, + handler: (input: Record, context?: KtxMcpToolHandlerContext) => Promise, + ): void; +} + +export interface KtxSemanticLayerMcpPort { + readSource(input: { connectionId: string; sourceName: string }): Promise; + query( + input: { connectionId?: string; query: SemanticLayerQueryInput }, + options?: { onProgress?: KtxMcpProgressCallback }, + ): Promise; +} + +export interface KtxSqlExecutionMcpPort { + execute( + input: { connectionId: string; sql: string; maxRows: number }, + options?: { onProgress?: KtxMcpProgressCallback }, + ): Promise; +} +``` + +- [ ] **Step 2: Add output schemas and annotations** + +In `packages/context/src/mcp/context-tools.ts`, add this import: + +```typescript +import type { ToolAnnotations } from '@modelcontextprotocol/sdk/types.js'; +``` + +Replace the MCP type import with: + +```typescript +import type { + KtxMcpContextPorts, + KtxMcpProgressCallback, + KtxMcpServerLike, + KtxMcpToolHandlerContext, + KtxMcpToolResult, + KtxMcpUserContext, + NonArrayObject, +} from './types.js'; +``` + +After `const connectionIdSchema = z.string().min(1);`, add: + +```typescript +const unknownRecordSchema = z.record(z.string(), z.unknown()); +const tableRefSchema = z.object({ + catalog: z.string().nullable(), + db: z.string().nullable(), + name: z.string(), +}); + +const toolAnnotations = { + connection_list: { title: 'Connection List', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + discover_data: { title: 'Discover Data', readOnlyHint: true, openWorldHint: false }, + wiki_search: { title: 'Wiki Search', readOnlyHint: true, openWorldHint: false }, + wiki_read: { title: 'Wiki Read', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + entity_details: { title: 'Entity Details', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + dictionary_search: { title: 'Dictionary Search', readOnlyHint: true, openWorldHint: false }, + sl_read_source: { title: 'Semantic Layer Read Source', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + sl_query: { title: 'Semantic Layer Query', readOnlyHint: true, openWorldHint: false }, + sql_execution: { title: 'SQL Execution', readOnlyHint: true, openWorldHint: false }, + memory_ingest: { title: 'Memory Ingest', destructiveHint: true, openWorldHint: false }, + memory_ingest_status: { title: 'Memory Ingest Status', readOnlyHint: true, openWorldHint: false }, +} satisfies Record; + +const toolDescriptions = { + connection_list: + 'List configured read-only data connections available to this KTX project. Use this before connection-scoped tools when the project may have multiple warehouses.', + discover_data: + 'Search across KTX wiki pages, semantic-layer sources, measures, dimensions, raw tables, and columns. Example: discover_data({ query: "monthly orders by customer", connectionId: "warehouse", kinds: ["sl_source", "table"] }).', + wiki_search: + 'Search KTX wiki pages for reusable business context. Example: wiki_search({ query: "revenue recognition", limit: 5 }).', + wiki_read: + 'Read a KTX wiki page by key returned from wiki_search. Example: wiki_read({ key: "global/revenue" }).', + entity_details: + 'Read table and column metadata from the latest live-database scan snapshot. Example: entity_details({ connectionId: "warehouse", entities: [{ table: { schema: "public", table: "orders" }, columns: ["id"] }] }).', + dictionary_search: + 'Search profile-sampled warehouse values to locate likely source columns for business values. Example: dictionary_search({ values: ["Acme Corp"], connectionId: "warehouse" }).', + sl_read_source: + 'Read a semantic-layer YAML source by connection id and source name. Example: sl_read_source({ connectionId: "warehouse", sourceName: "orders" }).', + sl_query: + 'Execute a semantic-layer query and return rows, headers, generated SQL, and plan details. Example: sl_query({ connectionId: "warehouse", measures: ["orders.order_count"], dimensions: [{ dimension: "orders.created_at", granularity: "month" }] }).', + sql_execution: + 'Execute one parser-validated read-only SQL query against a configured KTX connection. Example: sql_execution({ connectionId: "warehouse", sql: "select count(*) from public.orders", maxRows: 100 }).', + memory_ingest: + 'Ingest free-form markdown knowledge into durable KTX memory. Use this for business rules, metric definitions, schema gotchas, recurring findings, or explicit user requests to remember something. Example: memory_ingest({ connectionId: "warehouse", content: "ARR is reported in cents in this warehouse." }).', + memory_ingest_status: + 'Read the current or final status for a memory ingest run. Example: memory_ingest_status({ runId: "memory-run-1" }).', +} satisfies Record; +``` + +After `memoryIngestStatusSchema`, add: + +```typescript +const connectionListOutputSchema = z.object({ + connections: z.array( + z.object({ + id: z.string(), + name: z.string(), + connectionType: z.string(), + }), + ), +}); + +const wikiSearchOutputSchema = z.object({ + results: z.array( + z.object({ + key: z.string(), + path: z.string(), + scope: z.enum(['GLOBAL', 'USER']), + summary: z.string(), + score: z.number(), + matchReasons: z.array(z.string()).optional(), + lanes: z + .array( + z.object({ + lane: z.string(), + status: z.string(), + requestedCandidatePoolLimit: z.number(), + effectiveCandidatePoolLimit: z.number(), + returnedCandidateCount: z.number(), + weight: z.number(), + reason: z.string().optional(), + }), + ) + .optional(), + }), + ), + totalFound: z.number(), +}); + +const wikiReadOutputSchema = z.object({ + key: z.string(), + summary: z.string(), + content: z.string(), + scope: z.enum(['GLOBAL', 'USER']), + tags: z.array(z.string()).optional(), + refs: z.array(z.string()).optional(), + slRefs: z.array(z.string()).optional(), +}); + +const slReadSourceOutputSchema = z.object({ + sourceName: z.string(), + yaml: z.string(), +}); + +const slQueryOutputSchema = z.object({ + connectionId: z.string().optional(), + dialect: z.string().optional(), + sql: z.string(), + headers: z.array(z.string()), + rows: z.array(z.array(z.unknown())), + totalRows: z.number(), + plan: unknownRecordSchema.optional(), +}); + +const entityDetailsSnapshotOutputSchema = z.object({ + syncId: z.string(), + extractedAt: z.string(), + scanRunId: z.string().nullable(), +}); + +const entityDetailsColumnOutputSchema = z.object({ + name: z.string(), + nativeType: z.string(), + normalizedType: z.string(), + dimensionType: z.enum(['time', 'string', 'number', 'boolean']), + nullable: z.boolean(), + primaryKey: z.boolean(), + comment: z.string().nullable(), +}); + +const entityDetailsForeignKeyOutputSchema = z.object({ + fromColumn: z.string(), + toCatalog: z.string().nullable(), + toDb: z.string().nullable(), + toTable: z.string(), + toColumn: z.string(), + constraintName: z.string().nullable(), +}); + +const entityDetailsOutputSchema = z.object({ + results: z.array( + z.union([ + z.object({ + ok: z.literal(true), + connectionId: z.string(), + tableRef: tableRefSchema, + display: z.string(), + kind: z.enum(['table', 'view', 'external', 'event_stream']), + comment: z.string().nullable(), + estimatedRows: z.number().nullable(), + columns: z.array(entityDetailsColumnOutputSchema), + foreignKeys: z.array(entityDetailsForeignKeyOutputSchema), + snapshot: entityDetailsSnapshotOutputSchema, + }), + z.object({ + ok: z.literal(false), + connectionId: z.string(), + table: z.union([z.string(), tableRefSchema]), + snapshot: entityDetailsSnapshotOutputSchema.optional(), + error: z.object({ + code: z.enum(['scan_missing', 'table_not_found', 'ambiguous_table', 'column_not_found']), + message: z.string(), + candidates: z.union([z.array(z.object({ tableRef: tableRefSchema, display: z.string() })), z.array(z.string())]).optional(), + }), + }), + ]), + ), +}); + +const dictionarySearchOutputSchema = z.object({ + searched: z.array( + z.object({ + connectionId: z.string(), + coverage: z.object({ + sampledRows: z.number().nullable(), + valuesPerColumn: z.number().nullable(), + profiledColumns: z.number(), + syncId: z.string().nullable(), + profiledAt: z.string().nullable(), + }), + status: z.enum(['ready', 'no_profile_artifact', 'no_candidate_columns']), + }), + ), + results: z.array( + z.object({ + value: z.string(), + matches: z.array( + z.object({ + connectionId: z.string(), + sourceName: z.string(), + columnName: z.string(), + matchedValue: z.string(), + cardinality: z.number().nullable(), + }), + ), + misses: z.array( + z.object({ + connectionId: z.string(), + reason: z.enum(['no_profile_artifact', 'no_candidate_columns', 'value_not_in_sample']), + }), + ), + }), + ), +}); + +const discoverDataOutputSchema = z.object({ + refs: z.array( + z.object({ + kind: discoverDataKindSchema, + id: z.string(), + score: z.number(), + summary: z.string().nullable(), + snippet: z.string().nullable(), + matchedOn: z.enum(['name', 'display', 'description', 'comment', 'expr', 'sample_value', 'body']), + connectionId: z.string().optional(), + tableRef: tableRefSchema.optional(), + columnName: z.string().optional(), + }), + ), +}); + +const sqlExecutionOutputSchema = z.object({ + headers: z.array(z.string()), + headerTypes: z.array(z.string()).optional(), + rows: z.array(z.array(z.unknown())), + rowCount: z.number(), +}); + +const memoryIngestOutputSchema = z.object({ + runId: z.string(), +}); + +const memoryIngestStatusOutputSchema = z.object({ + runId: z.string(), + status: z.enum(['running', 'done', 'error']), + stage: z.string(), + done: z.boolean(), + captured: z.object({ + wiki: z.array(z.string()), + sl: z.array(z.string()), + xrefs: z.array(z.string()), + }), + error: z.string().nullable(), + commitHash: z.string().nullable(), + skillsLoaded: z.array(z.string()), + signalDetected: z.boolean(), +}); +``` + +- [ ] **Step 3: Replace input schemas with described and normalized versions** + +In `context-tools.ts`, replace the input schema section from +`connectionListSchema` through `entityDetailsSchema` with: + +```typescript +const connectionListSchema = z.object({}); + +const knowledgeSearchSchema = z.object({ + query: z.string().min(1).describe('Natural-language wiki search query, e.g. "revenue recognition policy".'), + limit: z.number().int().min(1).max(50).default(10).describe('Maximum wiki pages to return. Defaults to 10.'), +}); + +const knowledgeReadSchema = z.object({ + key: z.string().min(1).describe('Wiki page key returned by wiki_search, e.g. "global/revenue".'), +}); + +const slReadSourceSchema = z.object({ + connectionId: connectionIdSchema.describe('Connection id that owns the semantic-layer source.'), + sourceName: z.string().min(1).describe('Semantic-layer source name without ".yaml", e.g. "orders".'), +}); + +const slQueryMeasureSchema = z.union([ + z.string().describe('Semantic-layer measure key, e.g. "orders.order_count".'), + z.object({ + expr: z.string().min(1).describe('Ad hoc aggregate expression, e.g. "sum(orders.amount)".'), + name: z.string().min(1).describe('Alias for the ad hoc measure, e.g. "gross_revenue".'), + }), +]); + +const slQueryDimensionSchema = z.preprocess( + (value) => { + if (typeof value === 'string') return { field: value }; + if (value && typeof value === 'object' && !Array.isArray(value)) { + const obj = { ...(value as Record) }; + if (!('field' in obj) && typeof obj.dimension === 'string') obj.field = obj.dimension; + return obj; + } + return value; + }, + z.object({ + field: z.string().min(1).describe('Dimension to group by, e.g. "orders.created_at" or "orders.status".'), + granularity: z.string().min(1).optional().describe('Time grain for time dimensions: day, week, month, quarter, or year.'), + }), +); +``` + +Keep the existing `slQueryOrderBySchema` preprocess and replace +`slQuerySchema` plus `entityDetailsTableRefSchema` with: + +```typescript +const slQuerySchema = z.object({ + connectionId: connectionIdSchema + .optional() + .describe('Connection id to query. Omit only when the project has exactly one configured connection.'), + measures: z.array(slQueryMeasureSchema).min(1).describe('Measures to select. Use semantic-layer keys when available.'), + dimensions: z.array(slQueryDimensionSchema).default([]).describe('Dimensions to group by. Strings and {dimension, granularity} are accepted.'), + filters: z.array(z.string().describe('Semantic-layer filter expression, e.g. "orders.status = paid".')).default([]), + segments: z.array(z.string().describe('Semantic-layer segment key to apply.')).default([]), + order_by: z.array(slQueryOrderBySchema).default([]).describe('Sort clauses. Strings and Cube-style {id, desc} are accepted.'), + limit: z.number().int().min(0).default(1000).describe('Maximum rows to return. Defaults to 1000.'), + include_empty: z.boolean().default(true).describe('Whether to include empty dimension groups. Defaults to true.'), +}); + +const entityDetailsTableRefSchema = z.preprocess( + (value) => { + if (value && typeof value === 'object' && !Array.isArray(value)) { + const obj = { ...(value as Record) }; + if (!('db' in obj) && typeof obj.schema === 'string') obj.db = obj.schema; + if (!('name' in obj) && typeof obj.table === 'string') obj.name = obj.table; + if (!('catalog' in obj)) obj.catalog = null; + return obj; + } + return value; + }, + z.object({ + catalog: z.string().nullable().describe('Catalog/project/database. Use null when not applicable.'), + db: z.string().nullable().describe('Schema/database/dataset. Use null when not applicable.'), + name: z.string().min(1).describe('Table name.'), + }), +); + +const entityDetailsSchema = z.object({ + connectionId: connectionIdSchema.describe('Connection id whose latest scan snapshot should be read.'), + entities: z + .array( + z.object({ + table: z + .union([z.string().min(1), entityDetailsTableRefSchema]) + .describe('Table display string or object ref. {schema, table} is accepted as an alias for {db, name}.'), + columns: z.array(z.string().min(1).describe('Column name to inspect.')).optional().describe('Optional column filter.'), + }), + ) + .min(1) + .max(20) + .describe('Tables or columns to inspect. Maximum 20 entities.'), +}); +``` + +Replace `dictionarySearchSchema`, `discoverDataSchema`, and +`sqlExecutionSchema` with: + +```typescript +const dictionarySearchSchema = z.object({ + values: z + .array(z.string().min(1).describe('Business value to locate, e.g. "Acme Corp" or "enterprise".')) + .min(1) + .max(20) + .describe('Values to search for in sampled warehouse dictionaries.'), + connectionId: connectionIdSchema + .optional() + .describe('Optional connection id. Pass it when user intent pins a specific warehouse.'), +}); + +const discoverDataKindSchema = z.enum(['wiki', 'sl_source', 'sl_measure', 'sl_dimension', 'table', 'column']); + +const discoverDataSchema = z.object({ + query: z.string().min(1).describe('Natural-language discovery query, e.g. "monthly orders by customer".'), + connectionId: connectionIdSchema + .optional() + .describe('Optional connection id. Pass it when user intent pins a specific warehouse.'), + kinds: z.array(discoverDataKindSchema.describe('Reference kind to include.')).optional().describe('Optional kind filter.'), + limit: z.number().int().min(1).max(50).default(15).optional().describe('Maximum refs to return. Defaults to 15.'), +}); + +const sqlExecutionSchema = z.object({ + connectionId: connectionIdSchema.describe('Connection id to execute against. Required for raw SQL.'), + sql: z.string().min(1).describe('Parser-validated read-only SQL, e.g. "select count(*) from public.orders".'), + maxRows: z.number().int().min(1).max(10_000).default(1000).optional().describe('Maximum rows to return. Defaults to 1000.'), +}); +``` + +- [ ] **Step 4: Replace `jsonToolResult`, `formatToolError`, and `registerParsedTool`** + +Replace `jsonToolResult`, `jsonErrorToolResult`, and `registerParsedTool` +with: + +```typescript +export function jsonToolResult(structuredContent: T): KtxMcpToolResult { + return { + content: [{ type: 'text', text: JSON.stringify(structuredContent, null, 2) }], + structuredContent, + }; +} + +export function jsonErrorToolResult(text: string): KtxMcpToolResult> { + return { + content: [{ type: 'text', text }], + isError: true, + }; +} + +function formatToolError(error: unknown): string { + if (error instanceof z.ZodError) { + return error.issues + .map((issue) => `${issue.path.length > 0 ? issue.path.join('.') : ''}: ${issue.message}`) + .join('\n'); + } + return error instanceof Error ? error.message : String(error); +} + +function mcpProgressCallback(context?: KtxMcpToolHandlerContext): KtxMcpProgressCallback | undefined { + const progressToken = context?._meta?.progressToken; + if (progressToken === undefined || !context?.sendNotification) { + return undefined; + } + return async (event) => { + await context.sendNotification?.({ + method: 'notifications/progress', + params: { + progressToken, + progress: event.progress, + ...(event.total !== undefined ? { total: event.total } : {}), + message: event.message, + }, + }); + }; +} + +function registerParsedTool( + server: KtxMcpServerLike, + name: string, + config: { + title: string; + description: string; + inputSchema: unknown; + outputSchema: unknown; + annotations: ToolAnnotations; + }, + schema: TSchema, + handler: (input: z.infer, context?: KtxMcpToolHandlerContext) => Promise, +): void { + server.registerTool(name, config, async (input, context) => { + try { + return await handler(schema.parse(input), context); + } catch (error) { + return jsonErrorToolResult(formatToolError(error)); + } + }); +} +``` + +- [ ] **Step 5: Update every registration config** + +For each `registerParsedTool` call, add `annotations` and `outputSchema`. +For example, replace the `connection_list` config with: + +```typescript + { + title: toolAnnotations.connection_list.title!, + description: toolDescriptions.connection_list, + inputSchema: connectionListSchema.shape, + outputSchema: connectionListOutputSchema, + annotations: toolAnnotations.connection_list, + }, +``` + +Use these exact output schemas: + +```typescript +connection_list -> connectionListOutputSchema +wiki_search -> wikiSearchOutputSchema +wiki_read -> wikiReadOutputSchema +sl_read_source -> slReadSourceOutputSchema +sl_query -> slQueryOutputSchema +entity_details -> entityDetailsOutputSchema +dictionary_search -> dictionarySearchOutputSchema +discover_data -> discoverDataOutputSchema +sql_execution -> sqlExecutionOutputSchema +memory_ingest -> memoryIngestOutputSchema +memory_ingest_status -> memoryIngestStatusOutputSchema +``` + +Use `toolAnnotations.` and `toolDescriptions.` for the +matching tool. + +- [ ] **Step 6: Remove the local sql_execution catch and wire progress callbacks** + +Replace the `sql_execution` handler with: + +```typescript + async (input, context) => { + const onProgress = mcpProgressCallback(context); + return jsonToolResult( + await sqlExecution.execute( + { + connectionId: input.connectionId, + sql: input.sql, + maxRows: input.maxRows ?? 1000, + }, + onProgress ? { onProgress } : undefined, + ), + ); + }, +``` + +Replace the `sl_query` handler with: + +```typescript + async (input, context) => { + const onProgress = mcpProgressCallback(context); + return jsonToolResult( + await semanticLayer.query( + { + connectionId: input.connectionId, + query: { + measures: input.measures, + dimensions: input.dimensions, + filters: input.filters, + segments: input.segments, + order_by: input.order_by, + limit: input.limit, + include_empty: input.include_empty, + }, + }, + onProgress ? { onProgress } : undefined, + ), + ); + }, +``` + +- [ ] **Step 7: Run MCP tests and update the snapshot** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts -u +``` + +Expected: PASS. The new snapshot file is created at +`packages/context/src/mcp/__snapshots__/mcp-tools-list.json`. + +- [ ] **Step 8: Commit** + +```bash +git add packages/context/src/mcp/types.ts packages/context/src/mcp/context-tools.ts packages/context/src/mcp/server.test.ts packages/context/src/mcp/__snapshots__/mcp-tools-list.json +git commit -m "feat(context): polish mcp tool metadata" +``` + +### Task 3: Enforce resolved semantic-layer compute sources + +**Files:** + +- Modify: `packages/context/src/daemon/semantic-layer-compute.ts` +- Modify: `packages/context/src/sl/local-query.ts` + +- [ ] **Step 1: Narrow compute port source types and add invariant comments** + +In `packages/context/src/daemon/semantic-layer-compute.ts`, replace the import +from `../sl/index.js` with: + +```typescript +import type { ResolvedSemanticLayerSource, SemanticLayerQueryInput } from '../sl/types.js'; +``` + +Replace the `query` and `validateSources` signatures in +`KtxSemanticLayerComputePort` with: + +```typescript + /** + * Callers must pass sources sanitized through toResolvedWire. The Python + * daemon rejects authoring-only fields such as usage and inherits_columns_from. + */ + query(input: { + sources: ResolvedSemanticLayerSource[]; + query: SemanticLayerQueryInput; + dialect: string; + }): Promise; + + /** + * Callers must pass sources sanitized through toResolvedWire. The Python + * daemon rejects authoring-only fields such as usage and inherits_columns_from. + */ + validateSources(input: { + sources: ResolvedSemanticLayerSource[]; + dialect: string; + recentlyTouched?: string[]; + }): Promise; +``` + +- [ ] **Step 2: Remove the unnecessary cast in local query loading** + +In `packages/context/src/sl/local-query.ts`, replace `loadComputableSources` +with: + +```typescript +async function loadComputableSources( + project: KtxLocalProject, + connectionId: string, +): Promise[]> { + return (await loadLocalSlSourceRecords(project, { connectionId: assertSafeConnectionId(connectionId) })) + .filter((record) => record.source.table || record.source.sql) + .map((record) => toResolvedWire(record.source)); +} +``` + +- [ ] **Step 3: Run type-check and relevant semantic-layer tests** + +Run: + +```bash +pnpm --filter @ktx/context run type-check +pnpm --filter @ktx/context exec vitest run src/sl/local-query.test.ts +``` + +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +git add packages/context/src/daemon/semantic-layer-compute.ts packages/context/src/sl/local-query.ts +git commit -m "fix(context): enforce resolved semantic layer compute sources" +``` + +### Task 4: Add local progress stages for sl_query and sql_execution + +**Files:** + +- Modify: `packages/context/src/sl/local-query.ts` +- Modify: `packages/context/src/sl/local-query.test.ts` +- Modify: `packages/context/src/mcp/local-project-ports.ts` +- Modify: `packages/context/src/mcp/local-project-ports.test.ts` + +- [ ] **Step 1: Add failing local-query progress tests** + +In `packages/context/src/sl/local-query.test.ts`, add a test that calls +`compileLocalSlQuery` with execution enabled and captures events: + +```typescript + it('emits progress while compiling and executing a local semantic-layer query', async () => { + const progress: Array<{ progress: number; message: string }> = []; + const queryExecutor = { + execute: vi.fn(async () => ({ + headers: ['status', 'order_count'], + rows: [['paid', 2]], + totalRows: 1, + command: 'SELECT', + rowCount: 1, + })), + }; + + const result = await compileLocalSlQuery(project, { + connectionId: 'warehouse', + query: { + measures: ['orders.order_count'], + dimensions: ['orders.status'], + limit: 25, + }, + compute, + execute: true, + maxRows: 10, + queryExecutor, + onProgress: (event) => progress.push({ progress: event.progress, message: event.message }), + }); + + expect(result.totalRows).toBe(1); + expect(progress).toEqual([ + { progress: 0, message: 'Compiling query' }, + { progress: 0.3, message: 'Generating SQL' }, + { progress: 0.6, message: 'Executing' }, + { progress: 1, message: 'Fetched 1 rows' }, + ]); + }); +``` + +- [ ] **Step 2: Implement local-query progress** + +In `packages/context/src/sl/local-query.ts`, import the progress type: + +```typescript +import type { KtxMcpProgressCallback } from '../mcp/types.js'; +``` + +Add the option: + +```typescript + onProgress?: KtxMcpProgressCallback; +``` + +In `compileLocalSlQuery`, emit stages in this order: + +```typescript + await options.onProgress?.({ progress: 0, message: 'Compiling query' }); + const connectionId = resolveLocalConnectionId(project, options.connectionId); + const dialect = dialectForDriver(project.config.connections[connectionId]?.driver); + const sources = await loadComputableSources(project, connectionId); + + await options.onProgress?.({ progress: 0.3, message: 'Generating SQL' }); + const response = await options.compute.query({ + sources, + dialect, + query: options.query, + }); +``` + +Before the query-executor call, add: + +```typescript + await options.onProgress?.({ progress: 0.6, message: 'Executing' }); +``` + +After the query-executor call, add: + +```typescript + await options.onProgress?.({ progress: 1, message: `Fetched ${execution.totalRows} rows` }); +``` + +In the compile-only branch, before returning, add: + +```typescript + await options.onProgress?.({ progress: 1, message: 'Fetched 0 rows' }); +``` + +- [ ] **Step 3: Add failing local SQL execution progress test** + +In `packages/context/src/mcp/local-project-ports.test.ts`, add: + +```typescript + it('emits sql_execution progress stages from local MCP ports', async () => { + const project = await initKtxProject({ projectDir: tempDir }); + project.config.connections.warehouse = { + driver: 'postgres', + url: 'env:DATABASE_URL', + }; + const connector = testConnector(testSnapshot(), { + headers: ['id'], + headerTypes: ['integer'], + rows: [[1]], + totalRows: 1, + rowCount: 1, + }); + const createConnector = vi.fn(async () => connector); + const sqlAnalysis = { + analyzeForFingerprint: vi.fn(), + analyzeBatch: vi.fn(), + validateReadOnly: vi.fn(async () => ({ ok: true, error: null })), + }; + const progress: Array<{ progress: number; message: string }> = []; + const ports = createLocalProjectMcpContextPorts(project, { + sqlAnalysis, + localScan: { + createConnector, + }, + }); + + const result = await ports.sqlExecution?.execute( + { connectionId: 'warehouse', sql: 'select id from public.orders', maxRows: 5 }, + { onProgress: (event) => progress.push({ progress: event.progress, message: event.message }) }, + ); + + expect(result?.rowCount).toBe(1); + expect(progress).toEqual([ + { progress: 0, message: 'Validating SQL' }, + { progress: 0.3, message: 'Executing' }, + { progress: 1, message: 'Fetched 1 rows' }, + ]); + }); +``` + +- [ ] **Step 4: Implement local SQL execution progress** + +In `packages/context/src/mcp/local-project-ports.ts`, import the progress type: + +```typescript +import type { KtxMcpContextPorts, KtxMcpProgressCallback, KtxSqlExecutionResponse } from './types.js'; +``` + +Change `executeValidatedReadOnlySql` to accept progress: + +```typescript +async function executeValidatedReadOnlySql( + project: KtxLocalProject, + options: CreateLocalProjectMcpContextPortsOptions, + input: { connectionId: string; sql: string; maxRows: number }, + onProgress?: KtxMcpProgressCallback, +): Promise { +``` + +At the start of the function, add: + +```typescript + await onProgress?.({ progress: 0, message: 'Validating SQL' }); +``` + +Immediately before `connector.executeReadOnly`, add: + +```typescript + await onProgress?.({ progress: 0.3, message: 'Executing' }); +``` + +Replace the direct return with: + +```typescript + const response = { + headers: result.headers, + ...(result.headerTypes ? { headerTypes: result.headerTypes } : {}), + rows: result.rows, + rowCount: result.rowCount ?? result.rows.length, + }; + await onProgress?.({ progress: 1, message: `Fetched ${response.rowCount} rows` }); + return response; +``` + +Pass progress through the port: + +```typescript + async execute(input, executionOptions) { + return executeValidatedReadOnlySql(project, options, input, executionOptions?.onProgress); + }, +``` + +Pass semantic-layer progress through: + +```typescript + return compileLocalSlQuery(project, { + connectionId: input.connectionId, + query: input.query, + compute: options.semanticLayerCompute, + execute: Boolean(options.queryExecutor), + maxRows: input.query.limit, + queryExecutor: options.queryExecutor, + onProgress: executionOptions?.onProgress, + }); +``` + +- [ ] **Step 5: Run local progress tests** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/sl/local-query.test.ts src/mcp/local-project-ports.test.ts +``` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add packages/context/src/sl/local-query.ts packages/context/src/sl/local-query.test.ts packages/context/src/mcp/local-project-ports.ts packages/context/src/mcp/local-project-ports.test.ts +git commit -m "feat(context): emit mcp query progress stages" +``` + +### Task 5: Final verification + +**Files:** + +- Verify: TypeScript workspace checks. + +- [ ] **Step 1: Run context tests** + +Run: + +```bash +pnpm --filter @ktx/context run test +pnpm --filter @ktx/context run test:slow +``` + +Expected: PASS. + +- [ ] **Step 2: Run type-checks** + +Run: + +```bash +pnpm --filter @ktx/context run type-check +pnpm --filter @ktx/cli run type-check +``` + +Expected: PASS. + +- [ ] **Step 3: Run CLI tests** + +Run: + +```bash +pnpm --filter @ktx/cli run test +``` + +Expected: PASS. + +- [ ] **Step 4: Run dead-code checks** + +Run: + +```bash +pnpm run dead-code +``` + +Expected: PASS. + +- [ ] **Step 5: Inspect final diff** + +Run: + +```bash +git status --short +git diff --stat +git diff -- packages/context/src/mcp/types.ts packages/context/src/mcp/context-tools.ts packages/context/src/mcp/server.test.ts packages/context/src/daemon/semantic-layer-compute.ts packages/context/src/sl/local-query.ts packages/context/src/sl/local-query.test.ts packages/context/src/mcp/local-project-ports.ts packages/context/src/mcp/local-project-ports.test.ts +``` + +Expected: only intended MCP polish and progress files are changed. diff --git a/docs/superpowers/plans/2026-05-16-mcp-tool-polish-v1-surface-change.md b/docs/superpowers/plans/2026-05-16-mcp-tool-polish-v1-surface-change.md new file mode 100644 index 00000000..2e259463 --- /dev/null +++ b/docs/superpowers/plans/2026-05-16-mcp-tool-polish-v1-surface-change.md @@ -0,0 +1,1305 @@ +# MCP Tool Polish V1 Surface Change Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land the atomic MCP surface change from the MCP tool polish spec: +retain only the research-loop tools, replace `memory_capture` with +`memory_ingest`, and update the installed analytics skill in the same change. + +**Architecture:** Keep the existing context, memory, and CLI services, but make +the MCP server register only the v1 research surface. Move memory ingest into +`registerKtxContextTools` so the next polish plan can apply annotations, +`outputSchema`, descriptions, and in-band error handling through one path. + +**Tech Stack:** TypeScript, Zod, MCP SDK, Vitest, pnpm workspace commands. + +--- + +## Audit summary + +The original spec is +`docs/superpowers/specs/2026-05-16-mcp-tool-polish-design.md`. + +Implemented before this plan: + +- `discover_data` already returns an object shape: + `jsonToolResult({ refs: await discover.search(input) })`. +- `sl_query.order_by` already accepts bare strings and Cube-style + `{ id, desc }` objects through `z.preprocess`. +- The local `sl_query` path already sanitizes sources with `toResolvedWire`. + +Remaining v1 blockers: + +- The MCP server still registers the broad admin surface: + `connection_test`, `wiki_write`, `sl_list_sources`, `sl_write_source`, + `sl_validate`, `ingest_*`, and `scan_*`. +- The MCP memory tools are still `memory_capture` and + `memory_capture_status`, with `userMessage` and `assistantMessage` input. +- Memory tools are still registered directly in `server.ts`, bypassing + `registerParsedTool`. +- The analytics skill and agent client docs still say "memory capture." + +Remaining v1 blockers after this plan: + +- Per-tool polish kit: annotations, `outputSchema`, input field + descriptions, long tool descriptions, in-band error wrapping, union-drift + normalization, `jsonToolResult` type narrowing, and `toResolvedWire` + invariant enforcement for `validateSources`. +- Progress notifications for `sql_execution` and `sl_query`. + +Non-blocking items from the spec: + +- Deleting admin tool implementation code after a future `ktx-admin` skill + lands. +- MCP resources, MCP prompts, elicitation, sampling, tool icons, code + execution, multi-tenancy, telemetry, and rate limiting. +- Error-message redaction for `formatToolError`, which belongs to the polish + kit plan. + +## File structure + +- `packages/context/src/memory/memory-runs.ts`: rename the memory run service + API from capture to ingest with no compatibility wrapper. +- `packages/context/src/memory/local-memory.ts`: rename the local factory to + `createLocalProjectMemoryIngest`. +- `packages/context/src/memory/index.ts`: re-export the new memory ingest + names only. +- `packages/context/src/mcp/types.ts`: rename `MemoryCapturePort` to + `MemoryIngestPort`, add `memoryIngest` to `KtxMcpContextPorts`, and remove + MCP context ports for removed admin tool families. +- `packages/context/src/mcp/context-tools.ts`: remove removed tool + registrations and register `memory_ingest` plus `memory_ingest_status`. +- `packages/context/src/mcp/server.ts`: delete direct memory tool + registration and route all tools through `registerKtxContextTools`. +- `packages/context/src/mcp/local-project-ports.ts`: stop assembling MCP + ports for removed admin tools. +- `packages/cli/src/mcp-server-factory.ts`: create the local memory ingest + port and include it in `contextTools.memoryIngest`. +- `packages/cli/src/text-ingest.ts`: rename CLI text ingest dependency names + from capture to ingest while preserving behavior. +- `packages/cli/src/skills/analytics/SKILL.md`: replace memory capture + guidance with memory ingest guidance and add multi-connection routing. +- `docs-site/content/docs/integrations/agent-clients.mdx`: replace the + existing memory capture wording. +- Tests: + `packages/context/src/mcp/server.test.ts`, + `packages/context/src/memory/memory-runs.test.ts`, + `packages/context/src/memory/local-memory.test.ts`, + `packages/cli/src/text-ingest.test.ts`, + `packages/cli/src/setup-agents.test.ts`. + +### Task 1: Lock the new MCP surface with failing tests + +**Files:** + +- Modify: `packages/context/src/mcp/server.test.ts` + +- [ ] **Step 1: Update the imports for new memory names** + +In `packages/context/src/mcp/server.test.ts`, replace the memory imports at +the top with: + +```typescript +import { + createLocalProjectMemoryIngest, + detectCaptureSignals, + type MemoryAgentInput, +} from '../memory/index.js'; +``` + +In the MCP type import from `./types.js`, replace `MemoryCapturePort` with +`MemoryIngestPort`: + +```typescript +import type { + KtxDiscoverDataMcpPort, + KtxDictionarySearchMcpPort, + KtxEntityDetailsMcpPort, + KtxKnowledgeMcpPort, + KtxMcpContextPorts, + KtxSemanticLayerMcpPort, + KtxSqlExecutionMcpPort, + KtxSqlExecutionResponse, + MemoryIngestPort, +} from './types.js'; +``` + +- [ ] **Step 2: Replace the standalone memory capture test** + +Replace the test named +`registers memory capture tools without host app dependencies` with this test: + +```typescript + it('registers memory ingest tools through the context tool surface', async () => { + const fake = makeFakeServer(); + let receivedInput: MemoryAgentInput | undefined; + const ingest: MemoryIngestPort = { + ingest: vi.fn().mockImplementation(async (input) => { + receivedInput = input; + return { runId: 'run-1' }; + }), + status: vi.fn().mockResolvedValue({ + runId: 'run-1', + status: 'done', + stage: 'done', + done: true, + captured: { wiki: ['revenue'], sl: [], xrefs: [] }, + error: null, + commitHash: 'abc123', + skillsLoaded: ['wiki_capture'], + signalDetected: true, + }), + }; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'mcp-user' }, + contextTools: { memoryIngest: ingest }, + }); + + expect(fake.tools.map((tool) => tool.name).sort()).toEqual([ + 'memory_ingest', + 'memory_ingest_status', + ]); + + const content = [ + 'view: orders {', + ' sql_table_name: public.orders ;;', + ' measure: gross_revenue {', + ' type: sum', + ' sql: ${TABLE}.gross_revenue_cents ;;', + ' }', + '}', + ].join('\n'); + const memoryIngest = getTool(fake.tools, 'memory_ingest'); + await expect( + memoryIngest.handler({ + content, + connectionId: '00000000-0000-4000-8000-000000000001', + }), + ).resolves.toEqual({ + content: [{ type: 'text', text: JSON.stringify({ runId: 'run-1' }, null, 2) }], + structuredContent: { runId: 'run-1' }, + }); + expect(ingest.ingest).toHaveBeenCalledWith({ + userId: 'mcp-user', + chatId: expect.stringMatching(/^mcp-/), + userMessage: 'Ingest external knowledge into KTX memory.', + assistantMessage: content, + connectionId: '00000000-0000-4000-8000-000000000001', + sourceType: 'external_ingest', + }); + + const cliEquivalentInput: MemoryAgentInput = { + userId: 'mcp-user', + chatId: 'cli-text-ingest-test-1', + userMessage: 'Ingest external text artifact "orders lookml" into KTX memory.', + assistantMessage: content, + connectionId: '00000000-0000-4000-8000-000000000001', + sourceType: 'external_ingest', + }; + expect(detectCaptureSignals(receivedInput!)).toEqual(detectCaptureSignals(cliEquivalentInput)); + + const memoryStatus = getTool(fake.tools, 'memory_ingest_status'); + await expect(memoryStatus.handler({ runId: 'run-1' })).resolves.toEqual({ + content: [ + { + type: 'text', + text: JSON.stringify( + { + runId: 'run-1', + status: 'done', + stage: 'done', + done: true, + captured: { wiki: ['revenue'], sl: [], xrefs: [] }, + error: null, + commitHash: 'abc123', + skillsLoaded: ['wiki_capture'], + signalDetected: true, + }, + null, + 2, + ), + }, + ], + structuredContent: { + runId: 'run-1', + status: 'done', + stage: 'done', + done: true, + captured: { wiki: ['revenue'], sl: [], xrefs: [] }, + error: null, + commitHash: 'abc123', + skillsLoaded: ['wiki_capture'], + signalDetected: true, + }, + }); + }); +``` + +- [ ] **Step 3: Replace the missing memory run test** + +Replace the test that looks up `memory_capture_status` for a missing run with: + +```typescript + it('returns an in-band error when a memory ingest run is missing', async () => { + const fake = makeFakeServer(); + const ingest: MemoryIngestPort = { + ingest: vi.fn(), + status: vi.fn().mockResolvedValue(null), + }; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'mcp-user' }, + contextTools: { memoryIngest: ingest }, + }); + + const memoryStatus = getTool(fake.tools, 'memory_ingest_status'); + await expect(memoryStatus.handler({ runId: 'missing-run' })).resolves.toEqual({ + content: [{ type: 'text', text: 'Memory ingest run "missing-run" was not found.' }], + isError: true, + }); + }); +``` + +- [ ] **Step 4: Update the local project MCP memory test** + +Rename the test `runs MCP memory_capture against a local project memory port` +to `runs MCP memory_ingest against a local project memory port`. + +Inside that test, rename the factory call and handler calls: + +```typescript + const memoryIngest = createLocalProjectMemoryIngest(project, { + agentRunner, + llmProvider, + runIdFactory: () => 'memory-run-mcp', + }); + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local' }, + contextTools: { memoryIngest }, + }); + + const capture = await getTool(fake.tools, 'memory_ingest').handler({ + content: 'Revenue means paid order value.', + connectionId: 'warehouse', + }); + + await memoryIngest.waitForRun('memory-run-mcp'); + const status = await getTool(fake.tools, 'memory_ingest_status').handler({ + runId: 'memory-run-mcp', + }); +``` + +Keep the existing wiki assertion in the test. Update its expected memory-agent +input to use: + +```typescript +{ + userId: 'local', + chatId: expect.stringMatching(/^mcp-/), + userMessage: 'Ingest external knowledge into KTX memory.', + assistantMessage: 'Revenue means paid order value.', + connectionId: 'warehouse', + sourceType: 'external_ingest', +} +``` + +- [ ] **Step 5: Update the full-surface registration assertion** + +In the large registration test, replace the expected tool-name list with the +retained v1 list: + +```typescript + expect(fake.tools.map((tool) => tool.name).sort()).toEqual([ + 'connection_list', + 'dictionary_search', + 'discover_data', + 'entity_details', + 'memory_ingest', + 'memory_ingest_status', + 'sl_query', + 'sl_read_source', + 'sql_execution', + 'wiki_read', + 'wiki_search', + ]); +``` + +Delete assertions that call removed tools: +`connection_test`, `wiki_write`, `sl_list_sources`, `sl_write_source`, +`sl_validate`, `ingest_trigger`, `ingest_status`, `ingest_report`, +`ingest_replay`, `scan_trigger`, `scan_status`, `scan_report`, +`scan_list_artifacts`, and `scan_read_artifact`. + +- [ ] **Step 6: Run the MCP tests and confirm they fail** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts -t "memory ingest|registers all available" +``` + +Expected: FAIL. The current implementation still registers `memory_capture`, +accepts `userMessage` and `assistantMessage`, and exposes removed admin tools. + +### Task 2: Rename memory capture internals to memory ingest + +**Files:** + +- Modify: `packages/context/src/memory/memory-runs.ts` +- Modify: `packages/context/src/memory/memory-runs.test.ts` +- Modify: `packages/context/src/memory/local-memory.ts` +- Modify: `packages/context/src/memory/local-memory.test.ts` +- Modify: `packages/context/src/memory/index.ts` + +- [ ] **Step 1: Update memory run tests to the new API** + +In `packages/context/src/memory/memory-runs.test.ts`, replace the import with: + +```typescript +import { MemoryIngestService, type MemoryRunStorePort } from './memory-runs.js'; +``` + +Replace `MemoryCaptureService` with `MemoryIngestService`, rename local +variables from `capture` to `ingest`, and replace `.capture(` calls with +`.ingest(` calls. The shared test setup type becomes: + +```typescript +let ingest: MemoryIngestService; +``` + +The service construction becomes: + +```typescript +ingest = new MemoryIngestService({ memoryAgent, runs: store }); +``` + +- [ ] **Step 2: Update local memory tests to the new factory** + +In `packages/context/src/memory/local-memory.test.ts`, replace the import with: + +```typescript +import { createLocalProjectMemoryIngest } from './local-memory.js'; +``` + +Rename the describe block to: + +```typescript +describe('createLocalProjectMemoryIngest', () => { +``` + +Replace `createLocalProjectMemoryCapture(` with +`createLocalProjectMemoryIngest(` and replace local variables named `capture` +with `ingest`. + +- [ ] **Step 3: Run the renamed memory tests and confirm they fail** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/memory/memory-runs.test.ts src/memory/local-memory.test.ts +``` + +Expected: FAIL with missing exports and missing `.ingest()` method. + +- [ ] **Step 4: Rename the memory run service** + +In `packages/context/src/memory/memory-runs.ts`, replace the capture-specific +type and class declarations with: + +```typescript +export interface MemoryIngestServiceDeps { + memoryAgent: Pick; + runs: MemoryRunStorePort; +} + +export interface MemoryIngestStartResult { + runId: string; +} + +export interface MemoryIngestStatus { + runId: string; + status: MemoryRunStatus; + stage: string; + done: boolean; + captured: { + wiki: string[]; + sl: string[]; + xrefs: string[]; + }; + error: string | null; + commitHash: string | null; + skillsLoaded: string[]; + signalDetected: boolean; +} +``` + +Update `capturedKeys` to return the renamed status type: + +```typescript +function capturedKeys(actions: MemoryAction[]): MemoryIngestStatus['captured'] { +``` + +Replace the class with: + +```typescript +export class MemoryIngestService { + private readonly inFlight = new Map>(); + + constructor(private readonly deps: MemoryIngestServiceDeps) {} + + async ingest(input: MemoryAgentInput): Promise { + const row = await this.deps.runs.createRunning({ + inputHash: inputHash(input), + chatId: input.chatId, + }); + + await this.deps.runs.markRunning(row.id, 'ingesting'); + + const run = this.runIngest(row.id, input); + this.inFlight.set(row.id, run); + run.finally(() => this.inFlight.delete(row.id)).catch(() => undefined); + + return { runId: row.id }; + } + + async waitForRun(runId: string): Promise { + await this.inFlight.get(runId); + } + + private async runIngest(runId: string, input: MemoryAgentInput): Promise { + try { + const outputSummary = await this.deps.memoryAgent.ingest(input); + await this.deps.runs.markDone(runId, outputSummary); + } catch (error) { + await this.deps.runs.markError(runId, error instanceof Error ? error.message : String(error)); + } + } + + async status(runId: string): Promise { + const row = await this.deps.runs.findById(runId); + if (!row) { + return null; + } + + const output = row.outputSummary; + return { + runId: row.id, + status: row.status, + stage: row.stage, + done: row.status !== 'running', + captured: output ? capturedKeys(output.actions) : { wiki: [], sl: [], xrefs: [] }, + error: row.error, + commitHash: output?.commitHash ?? null, + skillsLoaded: output?.skillsLoaded ?? [], + signalDetected: output?.signalDetected ?? false, + }; + } +} +``` + +- [ ] **Step 5: Rename the local memory factory** + +In `packages/context/src/memory/local-memory.ts`, replace the service import: + +```typescript +import { MemoryIngestService } from './memory-runs.js'; +``` + +Rename the options interface and factory: + +```typescript +export interface CreateLocalProjectMemoryIngestOptions { + llmProvider?: KtxLlmProvider; + agentRunner?: AgentRunnerService; + memoryModel?: string; + semanticLayerCompute?: KtxSemanticLayerComputePort; + queryExecutor?: { execute(input: { connectionId: string; sql: string; maxRows?: number }): Promise }; + runIdFactory?: () => string; + logger?: KtxLogger; +} + +export function createLocalProjectMemoryIngest( + project: KtxLocalProject, + options: CreateLocalProjectMemoryIngestOptions = {}, +): MemoryIngestService { +``` + +Update the error string: + +```typescript +throw new Error('createLocalProjectMemoryIngest requires llm.provider.backend or an injected agentRunner'); +``` + +Return the renamed service: + +```typescript + return new MemoryIngestService({ + memoryAgent, + runs: new LocalMemoryRunStore({ projectDir: project.projectDir, idFactory: options.runIdFactory }), + }); +``` + +- [ ] **Step 6: Update memory exports** + +In `packages/context/src/memory/index.ts`, replace the memory run exports with: + +```typescript +export { createLocalProjectMemoryIngest, type CreateLocalProjectMemoryIngestOptions } from './local-memory.js'; +export { LocalMemoryRunStore, type LocalMemoryRunStoreOptions } from './local-memory-runs.js'; +export { + MemoryIngestService, + type MemoryIngestServiceDeps, + type MemoryIngestStartResult, + type MemoryIngestStatus, + type MemoryRunRecord, + type MemoryRunStatus, + type MemoryRunStorePort, +} from './memory-runs.js'; +``` + +- [ ] **Step 7: Run memory tests and commit** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/memory/memory-runs.test.ts src/memory/local-memory.test.ts +``` + +Expected: PASS. + +Commit: + +```bash +git add packages/context/src/memory/memory-runs.ts packages/context/src/memory/memory-runs.test.ts packages/context/src/memory/local-memory.ts packages/context/src/memory/local-memory.test.ts packages/context/src/memory/index.ts +git commit -m "refactor(context): rename memory capture service to ingest" +``` + +### Task 3: Move memory ingest into the shared MCP context tool path + +**Files:** + +- Modify: `packages/context/src/mcp/types.ts` +- Modify: `packages/context/src/mcp/context-tools.ts` +- Modify: `packages/context/src/mcp/server.ts` +- Modify: `packages/context/src/mcp/server.test.ts` + +- [ ] **Step 1: Update MCP types** + +In `packages/context/src/mcp/types.ts`, replace the memory import with: + +```typescript +import type { MemoryIngestService } from '../memory/index.js'; +``` + +Replace `MemoryCapturePort` with: + +```typescript +export interface MemoryIngestPort { + ingest: MemoryIngestService['ingest']; + status: MemoryIngestService['status']; +} +``` + +Reduce the retained MCP port interfaces to the v1 surface: + +```typescript +export interface KtxConnectionsMcpPort { + list(): Promise; +} + +export interface KtxKnowledgeMcpPort { + search(input: { userId: string; query: string; limit: number }): Promise; + read(input: { userId: string; key: string }): Promise; +} + +export interface KtxSemanticLayerMcpPort { + readSource(input: { connectionId: string; sourceName: string }): Promise; + query(input: { connectionId?: string; query: SemanticLayerQueryInput }): Promise; +} + +export interface KtxMcpContextPorts { + connections?: KtxConnectionsMcpPort; + knowledge?: KtxKnowledgeMcpPort; + semanticLayer?: KtxSemanticLayerMcpPort; + entityDetails?: KtxEntityDetailsMcpPort; + dictionarySearch?: KtxDictionarySearchMcpPort; + discover?: KtxDiscoverDataMcpPort; + sqlExecution?: KtxSqlExecutionMcpPort; + memoryIngest?: MemoryIngestPort; +} + +export interface KtxMcpServerDeps { + server: KtxMcpServerLike; + userContext: KtxMcpUserContext; + contextTools?: KtxMcpContextPorts; +} +``` + +- [ ] **Step 2: Add memory ingest schemas to `context-tools.ts`** + +At the top of `packages/context/src/mcp/context-tools.ts`, add: + +```typescript +import { randomUUID } from 'node:crypto'; +import type { MemoryAgentInput } from '../memory/index.js'; +``` + +After `sqlExecutionSchema`, add: + +```typescript +const memoryIngestSchema = z.object({ + content: z + .string() + .min(1) + .describe( + 'Free-form markdown to ingest. Include the knowledge itself plus any context (source, the user question, why this came up) that the memory agent should consider when triaging into wiki/SL.', + ), + connectionId: connectionIdSchema + .optional() + .describe( + 'Scope this memory to a specific connection. Required when the knowledge is warehouse-specific, including measure definitions, schema gotchas, or anything tied to a particular warehouse. Omit only for global wiki knowledge.', + ), +}); + +const memoryIngestStatusSchema = z.object({ + runId: z.string().min(1).describe('The memory ingest run id returned by memory_ingest.'), +}); +``` + +- [ ] **Step 3: Delete removed registration blocks** + +In `registerKtxContextTools`, delete the registration blocks for these tool +names: + +```text +connection_test +wiki_write +sl_list_sources +sl_write_source +sl_validate +ingest_trigger +ingest_status +ingest_report +ingest_replay +scan_trigger +scan_status +scan_report +scan_list_artifacts +scan_read_artifact +``` + +Also delete their now-unused input schemas from `context-tools.ts`: +`connectionTestSchema`, `historicSqlUsageFrontmatterSchema`, +`knowledgeWriteSchema`, `slListSourcesSchema`, `slWriteSourceSchema`, +`slValidateSchema`, `ingestTriggerSchema`, `ingestStatusSchema`, +`ingestReportSchema`, `ingestReplaySchema`, `scanTriggerSchema`, +`scanStatusSchema`, and `scanArtifactReadSchema`. + +- [ ] **Step 4: Register memory ingest through `registerParsedTool`** + +Add this block near the end of `registerKtxContextTools`, after +`sql_execution`: + +```typescript + if (ports.memoryIngest) { + const memoryIngest = ports.memoryIngest; + registerParsedTool( + server, + 'memory_ingest', + { + title: 'Memory Ingest', + description: + 'Ingest free-form markdown knowledge into KTX durable memory. Use this for business rules, metric definitions, schema gotchas, recurring findings, or explicit user requests to remember something.', + inputSchema: memoryIngestSchema.shape, + }, + memoryIngestSchema, + async (input) => { + const ingestInput: MemoryAgentInput = { + userId: userContext.userId, + chatId: `mcp-${randomUUID()}`, + userMessage: 'Ingest external knowledge into KTX memory.', + assistantMessage: input.content, + connectionId: input.connectionId, + sourceType: 'external_ingest', + }; + return jsonToolResult(await memoryIngest.ingest(ingestInput)); + }, + ); + + registerParsedTool( + server, + 'memory_ingest_status', + { + title: 'Memory Ingest Status', + description: 'Read the current or final status for a memory ingest run.', + inputSchema: memoryIngestStatusSchema.shape, + }, + memoryIngestStatusSchema, + async (input) => { + const status = await memoryIngest.status(input.runId); + return status ? jsonToolResult(status) : jsonErrorToolResult(`Memory ingest run "${input.runId}" was not found.`); + }, + ); + } +``` + +- [ ] **Step 5: Simplify `server.ts`** + +Replace `packages/context/src/mcp/server.ts` with: + +```typescript +import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import { registerKtxContextTools } from './context-tools.js'; +import type { KtxMcpServerDeps, KtxMcpServerLike } from './types.js'; + +export function createKtxMcpServer(deps: KtxMcpServerDeps): KtxMcpServerDeps['server'] { + if (deps.contextTools) { + registerKtxContextTools({ + server: deps.server, + ports: deps.contextTools, + userContext: deps.userContext, + }); + } + + return deps.server; +} + +export function createDefaultKtxMcpServer( + deps: Omit & { name?: string; version?: string }, +): McpServer { + const server = new McpServer({ + name: deps.name ?? 'ktx', + version: deps.version ?? '0.0.0-private', + }); + createKtxMcpServer({ + server: server as KtxMcpServerLike, + userContext: deps.userContext, + contextTools: deps.contextTools, + }); + return server; +} +``` + +- [ ] **Step 6: Run MCP tests and commit** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts -t "memory ingest|registers all available" +``` + +Expected: PASS for the new memory ingest and retained surface tests. + +Commit: + +```bash +git add packages/context/src/mcp/types.ts packages/context/src/mcp/context-tools.ts packages/context/src/mcp/server.ts packages/context/src/mcp/server.test.ts +git commit -m "feat(mcp): slim research tool surface" +``` + +### Task 4: Slim local MCP port assembly and CLI server factory + +**Files:** + +- Modify: `packages/context/src/mcp/local-project-ports.ts` +- Modify: `packages/context/src/mcp/local-project-ports.test.ts` +- Modify: `packages/cli/src/mcp-server-factory.ts` + +- [ ] **Step 1: Remove local admin MCP port assembly** + +In `packages/context/src/mcp/local-project-ports.ts`, remove the +`localIngest` option: + +```typescript +interface CreateLocalProjectMcpContextPortsOptions { + semanticLayerCompute?: KtxSemanticLayerComputePort; + queryExecutor?: KtxSqlQueryExecutorPort; + sqlAnalysis?: SqlAnalysisPort; + localScan?: LocalScanMcpOptions; + embeddingService?: KtxEmbeddingPort | null; +} +``` + +Inside `createLocalProjectMcpContextPorts`, remove these object members: + +```typescript + async test(input) { + return testLocalConnection(project, options, input.connectionId); + }, +``` + +```typescript + async write(input) { + const existing = await readLocalKnowledgePage(project, { + key: input.key, + userId: input.userId, + }); + await writeLocalKnowledgePage(project, { + key: input.key, + scope: 'GLOBAL', + userId: input.userId, + summary: input.summary, + content: input.content, + tags: input.tags, + refs: input.refs, + slRefs: input.slRefs, + source: input.source, + intent: input.intent, + tables: input.tables, + representativeSql: input.representativeSql, + usage: input.usage, + fingerprints: input.fingerprints, + }); + return { success: true, key: input.key, action: existing ? 'updated' : 'created' }; + }, +``` + +Remove `semanticLayer.listSources`, `semanticLayer.writeSource`, and +`semanticLayer.validate` from the returned semantic-layer port. Keep only +`readSource` and `query`. + +Delete the `if (options.localIngest) { ... }` block and the +`if (options.localScan) { ... }` block at the bottom of the function. Keep +the `options.localScan` value available to `sql_execution`, because +`executeValidatedReadOnlySql` still uses it. + +- [ ] **Step 2: Remove local-project helper code that became unused** + +In `packages/context/src/mcp/local-project-ports.ts`, delete these helper +functions when no references remain: + +```text +testLocalConnection +scanArtifactType +listArtifactsForReport +readScanArtifact +loadComputableSources +validateSourceRecord +localIngestSourceDir +rawFileCountFromIngestReport +statusFromIngestReport +``` + +Remove now-unused imports from `../ingest/index.js`, `../wiki/local-knowledge.js`, +`yaml`, and `./types.js`. Keep imports used by `connection_list`, +`wiki_search`, `wiki_read`, `sl_read_source`, `sl_query`, `entity_details`, +`dictionary_search`, `discover_data`, and `sql_execution`. + +- [ ] **Step 3: Update local-project port tests** + +In `packages/context/src/mcp/local-project-ports.test.ts`, remove assertions +that depend on `ports.connections.test`, `ports.knowledge.write`, +`ports.semanticLayer.listSources`, `ports.semanticLayer.writeSource`, +`ports.semanticLayer.validate`, `ports.ingest`, or `ports.scan`. + +Add this retained-surface assertion to the test that constructs local ports: + +```typescript +expect(Object.keys(ports).sort()).toEqual([ + 'connections', + 'dictionarySearch', + 'discover', + 'entityDetails', + 'knowledge', + 'semanticLayer', + 'sqlExecution', +]); +expect(Object.keys(ports.connections ?? {}).sort()).toEqual(['list']); +expect(Object.keys(ports.knowledge ?? {}).sort()).toEqual(['read', 'search']); +expect(Object.keys(ports.semanticLayer ?? {}).sort()).toEqual(['query', 'readSource']); +``` + +- [ ] **Step 4: Update the CLI MCP server factory** + +In `packages/cli/src/mcp-server-factory.ts`, replace the memory import: + +```typescript +import { createLocalProjectMemoryIngest } from '@ktx/context/memory'; +``` + +Remove the `localIngest` block from the call to +`createLocalProjectMcpContextPorts`. Keep `semanticLayerCompute`, +`queryExecutor`, `sqlAnalysis`, and `localScan`. + +Replace the memory creation block with: + +```typescript + let memoryIngest: ReturnType | undefined; + try { + memoryIngest = createLocalProjectMemoryIngest(input.project, { semanticLayerCompute, queryExecutor }); + } catch (error) { + input.io?.stderr.write(`KTX MCP memory_ingest disabled: ${error instanceof Error ? error.message : String(error)}\n`); + } +``` + +Pass memory ingest through the context tools object: + +```typescript + return () => + createDefaultKtxMcpServer({ + name: 'ktx', + version: input.cliVersion, + userContext: { userId: 'local' }, + contextTools: { + ...contextTools, + ...(memoryIngest ? { memoryIngest } : {}), + }, + }); +``` + +- [ ] **Step 5: Run local MCP and CLI factory tests and commit** + +Run: + +```bash +pnpm --filter @ktx/context exec vitest run src/mcp/local-project-ports.test.ts src/mcp/server.test.ts +pnpm --filter @ktx/cli exec vitest run src/commands/mcp-commands.test.ts src/mcp-http-server.test.ts src/managed-mcp-daemon.test.ts +``` + +Expected: PASS. + +Commit: + +```bash +git add packages/context/src/mcp/local-project-ports.ts packages/context/src/mcp/local-project-ports.test.ts packages/cli/src/mcp-server-factory.ts +git commit -m "refactor(mcp): remove admin ports from server factory" +``` + +### Task 5: Rename CLI text ingest dependencies + +**Files:** + +- Modify: `packages/cli/src/text-ingest.ts` +- Modify: `packages/cli/src/text-ingest.test.ts` + +- [ ] **Step 1: Update text-ingest tests** + +In `packages/cli/src/text-ingest.test.ts`, replace +`MemoryCaptureStatus` with `MemoryIngestStatus` and +`TextMemoryCapturePort` with `TextMemoryIngestPort`. + +Rename helper functions and dependency keys: + +```typescript +function createMemoryIngestStub( + status: MemoryIngestStatus | null, +): TextMemoryIngestPort { +``` + +Replace `createMemoryCapture` dependency uses with `createMemoryIngest`. + +- [ ] **Step 2: Run text ingest tests and confirm they fail** + +Run: + +```bash +pnpm --filter @ktx/cli exec vitest run src/text-ingest.test.ts +``` + +Expected: FAIL with missing `MemoryIngestStatus`, +`TextMemoryIngestPort`, and `createMemoryIngest`. + +- [ ] **Step 3: Update `text-ingest.ts` imports and types** + +In `packages/cli/src/text-ingest.ts`, replace the memory import with: + +```typescript +import { createLocalProjectMemoryIngest, type MemoryAgentInput, type MemoryIngestStatus } from '@ktx/context/memory'; +``` + +Replace the text port and dependency types with: + +```typescript +export interface TextMemoryIngestPort { + ingest(input: MemoryAgentInput): Promise<{ runId: string }>; + waitForRun(runId: string): Promise; + status(runId: string): Promise; +} +``` + +```typescript +export interface KtxTextIngestDeps { + loadProject?: (options: { projectDir: string }) => Promise; + createMemoryIngest?: (project: KtxLocalProject) => TextMemoryIngestPort; + readFile?: (path: string) => Promise; + readStdin?: () => Promise; + now?: () => number; +} +``` + +Update the default factory: + +```typescript +function defaultCreateMemoryIngest(project: KtxLocalProject): TextMemoryIngestPort { + return createLocalProjectMemoryIngest(project); +} +``` + +Replace `MemoryCaptureStatus` type references with `MemoryIngestStatus`. + +- [ ] **Step 4: Update the text ingest runtime calls** + +In `runKtxTextIngest`, replace: + +```typescript + const memoryCapture = (deps.createMemoryCapture ?? defaultCreateMemoryCapture)(project); +``` + +with: + +```typescript + const memoryIngest = (deps.createMemoryIngest ?? defaultCreateMemoryIngest)(project); +``` + +Replace the run block with: + +```typescript + const ingestInput: MemoryAgentInput = { + userId: args.userId, + chatId: `cli-text-ingest-${batchId}-${index + 1}`, + userMessage: `Ingest external text artifact ${artifactReference(item.label)} into KTX memory.`, + assistantMessage: item.content.trim(), + ...(args.connectionId ? { connectionId: args.connectionId } : {}), + sourceType: 'external_ingest', + }; + const ingest = await memoryIngest.ingest(ingestInput); + runId = ingest.runId; + await memoryIngest.waitForRun(runId); + const status = await memoryIngest.status(runId); + if (!status) { + throw new Error(`Memory ingest run "${runId}" was not found.`); + } +``` + +- [ ] **Step 5: Run text ingest tests and commit** + +Run: + +```bash +pnpm --filter @ktx/cli exec vitest run src/text-ingest.test.ts +``` + +Expected: PASS. + +Commit: + +```bash +git add packages/cli/src/text-ingest.ts packages/cli/src/text-ingest.test.ts +git commit -m "refactor(cli): rename text ingest memory port" +``` + +### Task 6: Update analytics skill and docs + +**Files:** + +- Modify: `packages/cli/src/skills/analytics/SKILL.md` +- Modify: `packages/cli/src/setup-agents.test.ts` +- Modify: `docs-site/content/docs/integrations/agent-clients.mdx` + +- [ ] **Step 1: Update the analytics skill text** + +In `packages/cli/src/skills/analytics/SKILL.md`, replace line 8 with: + +```markdown +You have access to KTX MCP tools for data discovery, semantic-layer analysis, raw read-only SQL, wiki context, and memory ingest. Follow this workflow. +``` + +Replace workflow step 7 with: + +```markdown +7. **Capture durable learnings** - call `memory_ingest` whenever a turn produces something worth remembering (business rules, metric definitions, schema gotchas, recurring findings) **or** whenever the user asks you to remember something. Pass markdown in `content` including any source context the memory agent should weigh. Each call is a feedback loop; better notes today mean smarter `discover_data` and `wiki_search` results tomorrow. +``` + +Add this rule under `` after the `dictionary_search` rule: + +```markdown +- When `connection_list` shows multiple connections, pass an explicit `connectionId` to every tool that takes one and where user intent pins a specific warehouse. Required: `entity_details`, `sl_read_source`, and `sql_execution`. Required when user intent is warehouse-specific, including wording like "in our warehouse" or "this warehouse": `memory_ingest`; without `connectionId`, the memory agent cannot update the semantic layer and the knowledge lands as wiki-only. Pass `connectionId` when intent pins a warehouse, otherwise omit for unscoped discovery: `sl_query`, `discover_data`, and `dictionary_search`. Never pass `connectionId` to `connection_list`, `wiki_search`, `wiki_read`, or `memory_ingest_status`. If intent is ambiguous for a required-or-scoped tool, ask the user which warehouse before calling. +``` + +In the first example, replace step 5 with: + +```markdown +5. `memory_ingest({ connectionId: "warehouse", content: "Acme Corp order analysis used orders_facts.order_count filtered by customers.name = 'Acme Corp'. Source: current analysis turn." })` captures the durable finding. +``` + +Add this example before ``: + +```markdown +--- + +**Input:** "Heads up: ARR is always reported in cents in our warehouse." + +**Workflow:** +1. If multiple connections exist, call `connection_list` and identify the warehouse the user means. Ask if ambiguous. +2. `memory_ingest({ connectionId: "warehouse", content: "ARR is reported in cents (not dollars) in this warehouse. Multiply by 0.01 for dollar amounts. Source: user clarification." })` remembers the warehouse-specific rule without running an analysis turn. +``` + +- [ ] **Step 2: Add setup-agent skill assertions** + +In `packages/cli/src/setup-agents.test.ts`, find the test that reads +`.agents/skills/ktx-analytics/SKILL.md` and currently asserts +`name: ktx-analytics`. Extend it with: + +```typescript +expect(analyticsSkill).toContain('memory_ingest'); +expect(analyticsSkill).toContain('ARR is reported in cents'); +expect(analyticsSkill).not.toContain('memory_capture'); +``` + +- [ ] **Step 3: Update docs-site memory wording** + +In `docs-site/content/docs/integrations/agent-clients.mdx`, replace: + +```markdown +semantic-layer queries, wiki search, SQL execution, and memory capture. The +``` + +with: + +```markdown +semantic-layer queries, wiki search, SQL execution, and memory ingest. The +``` + +- [ ] **Step 4: Run skill and docs tests and commit** + +Run: + +```bash +pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts +pnpm --filter ktx-docs run build +pnpm --filter ktx-docs run test +``` + +Expected: PASS. + +Commit: + +```bash +git add packages/cli/src/skills/analytics/SKILL.md packages/cli/src/setup-agents.test.ts docs-site/content/docs/integrations/agent-clients.mdx +git commit -m "docs: update analytics skill for memory ingest" +``` + +### Task 7: Full verification and cleanup + +**Files:** + +- Verify: all files changed in Tasks 1-6 + +- [ ] **Step 1: Check for stale capture names** + +Run: + +```bash +rg -n "memory_capture|memory_capture_status|MemoryCapture|createLocalProjectMemoryCapture|TextMemoryCapturePort|memoryCapture" packages/context/src packages/cli/src docs-site/content/docs/integrations/agent-clients.mdx +``` + +Expected: no matches in MCP, memory service, CLI setup, analytics skill, text +ingest, or docs-site files. Matches in historical `docs/superpowers/` files +are allowed and are intentionally excluded from the command. + +- [ ] **Step 2: Check retained MCP tool registration names** + +Run: + +```bash +rg -n "'(connection_test|wiki_write|sl_list_sources|sl_write_source|sl_validate|ingest_trigger|ingest_status|ingest_report|ingest_replay|scan_trigger|scan_status|scan_report|scan_list_artifacts|scan_read_artifact)'" packages/context/src/mcp packages/cli/src +``` + +Expected: no matches. + +- [ ] **Step 3: Run required context checks** + +Run: + +```bash +pnpm --filter @ktx/context run test +pnpm --filter @ktx/context run test:slow +pnpm --filter @ktx/context run type-check +``` + +Expected: PASS. + +- [ ] **Step 4: Run required CLI checks** + +Run: + +```bash +pnpm --filter @ktx/cli run type-check +pnpm --filter @ktx/cli run test +``` + +Expected: PASS. + +- [ ] **Step 5: Run docs-site checks** + +Run: + +```bash +pnpm --filter ktx-docs run build +pnpm --filter ktx-docs run test +``` + +Expected: PASS. + +- [ ] **Step 6: Run dead-code check** + +Run: + +```bash +pnpm run dead-code +``` + +Expected: PASS. If Knip reports only exports intentionally kept for future +admin CLI work, add narrow `knip.json` entries for the exact symbols. Delete +private unused MCP-only helpers instead of ignoring them. + +- [ ] **Step 7: Run pre-commit on changed files** + +Run this command with the actual changed files from `git diff --name-only`: + +```bash +uv run pre-commit run --files packages/context/src/memory/memory-runs.ts packages/context/src/memory/memory-runs.test.ts packages/context/src/memory/local-memory.ts packages/context/src/memory/local-memory.test.ts packages/context/src/memory/index.ts packages/context/src/mcp/types.ts packages/context/src/mcp/context-tools.ts packages/context/src/mcp/server.ts packages/context/src/mcp/server.test.ts packages/context/src/mcp/local-project-ports.ts packages/context/src/mcp/local-project-ports.test.ts packages/cli/src/mcp-server-factory.ts packages/cli/src/text-ingest.ts packages/cli/src/text-ingest.test.ts packages/cli/src/skills/analytics/SKILL.md packages/cli/src/setup-agents.test.ts docs-site/content/docs/integrations/agent-clients.mdx +``` + +Expected: PASS. If pre-commit reports missing local tool versions without +changing files, record the exact error in the final handoff and rely on the +passing package checks above. + +- [ ] **Step 8: Commit final verification cleanup** + +Run: + +```bash +git status --short +``` + +Expected: only intentional files from this plan are modified. + +If verification cleanup changed files, commit them: + +```bash +git add packages/context/src packages/cli/src docs-site/content/docs/integrations/agent-clients.mdx knip.json +git commit -m "chore: verify mcp surface rename" +``` + +If no files changed after the previous commits, do not create an empty commit. + +## Self-review + +- Spec coverage: This plan covers PR 1 from the spec: tool surface reduction, + `memory_capture` to `memory_ingest` rename, memory input contract, memory + registration through the shared context tool path, analytics skill updates, + docs-site wording, CLI factory wiring, text ingest naming, and tests. +- Deferred v1 coverage: PR 2 polish kit and PR 3 progress notifications remain + v1-blocking follow-up plans after this lands. +- Red-flag scan: The plan avoids deferred-work markers, migration shims, + compatibility wrappers, and incomplete implementation instructions. +- Type consistency: All new names use `MemoryIngestService`, + `MemoryIngestPort`, `MemoryIngestStatus`, + `createLocalProjectMemoryIngest`, `TextMemoryIngestPort`, + `memory_ingest`, and `memory_ingest_status`. diff --git a/docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md b/docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md index a8044076..b65c77b6 100644 --- a/docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md +++ b/docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md @@ -143,7 +143,9 @@ or `entity_details` tools. } ``` -**Output:** array of refs, each: +**Output:** `{ refs: Ref[] }` — the MCP protocol requires `structuredContent` +to be a JSON object, so the array of matches is wrapped under `refs`. Each +ref is shaped: ```typescript { diff --git a/docs/superpowers/specs/2026-05-16-mcp-tool-polish-design.md b/docs/superpowers/specs/2026-05-16-mcp-tool-polish-design.md new file mode 100644 index 00000000..6dda60a9 --- /dev/null +++ b/docs/superpowers/specs/2026-05-16-mcp-tool-polish-design.md @@ -0,0 +1,802 @@ +# MCP Tool Polish: Slim Research Surface + Spec Compliance + +**Date:** 2026-05-16 +**Author:** Andrey Avtomonov +**Status:** Design — pending implementation plan + +## Background + +KTX currently exposes 25 MCP tools across context, semantic-layer, ingest, and +scan ports (`packages/context/src/mcp/context-tools.ts`). The +`ktx-analytics` SKILL.md (`packages/cli/src/skills/analytics/SKILL.md`) +is installed into every supported MCP client by `ktx setup --agents` and +already describes a Douala-equivalent research methodology (Discover → Inspect +→ Resolve → Plan → Query → Validate → Capture) that references nine of those +tools. + +A recent in-session audit surfaced three real bug classes: + +- **`structuredContent` shape** — `discover_data` returned a bare array; MCP + requires an object. Fixed. +- **Union-shape LLM drift** — `sl_query.order_by` accepted + `{ field, direction }` but Claude emitted Cube-style `{ id, desc }`. Fixed + via a `z.preprocess` that normalizes the alt-shape before strict validation. +- **Contract leak to the Python daemon** — `compileLocalSlQuery` skipped + `toResolvedWire` and sent TS-only authoring fields (`usage`, + `inherits_columns_from`) to a Pydantic model with `extra="forbid"`. Fixed. + +The audit also identified systemic gaps applicable across the surface: no +per-field `.describe()` outside `memory_capture*`, no `outputSchema` declared +anywhere, no MCP tool annotations, lingering union-drift risk on +`slQueryDimensionSchema` (`{ dimension, granularity }`) and +`entityDetailsTableRefSchema` (`{ schema, table }`), and inconsistent error +handling (only `sql_execution` wraps thrown errors in-band per MCP spec; the +other 24 let exceptions propagate as JSON-RPC errors). + +The current MCP spec (2025-11-25) provides several mechanisms KTX does not +use: `outputSchema` with `structuredContent` validation, tool annotations +(`readOnlyHint`/`destructiveHint`/`idempotentHint`/`openWorldHint`/`title`), +progress notifications via `_meta.progressToken`, and a clear in-band error +contract (`isError: true` with text content). + +The 25-tool surface is also wider than the agent needs. The `ktx-analytics` +SKILL only orchestrates nine of them; the rest are admin/setup/maintenance +operations that are better served by a `ktx`-CLI flow (and a future +`ktx-admin` SKILL — out of scope for this spec). + +## Goals + +- Reduce the MCP-registered surface to **11 tools** focused on the research + loop: `connection_list`, `discover_data`, `wiki_search`, `wiki_read`, + `entity_details`, `dictionary_search`, `sl_read_source`, `sl_query`, + `sql_execution`, `memory_ingest` (new), `memory_ingest_status` (new). The + remaining 14 tools become CLI-only by removing their MCP registration; their + implementations stay in `packages/context/src/` to back the CLI. +- Replace `memory_capture` / `memory_capture_status` with `memory_ingest` / + `memory_ingest_status`. The new tool takes free-form markdown `content` + plus optional `connectionId`; the memory agent triages into wiki and SL as + before. +- Apply the per-tool polish kit on every retained tool: MCP tool annotations, + `outputSchema`, per-field `.describe()`, rewritten tool descriptions, + standardized in-band error handling, union-drift normalization on the two + remaining at-risk schemas, and a type-narrowed `jsonToolResult`. +- Emit MCP progress notifications from `sql_execution` and `sl_query`. +- Update `ktx-analytics` SKILL.md to use the renamed tool, broaden the capture + step, and document multi-connection routing. + +## Non-Goals + +- Admin CLI skill (separate spec). +- Deleting the source code of the admin tools (deferred follow-up gated by + the admin CLI skill landing). +- MCP resources (subscribable wiki / SL). +- MCP prompts pushed by the server (the analytics SKILL is the equivalent). +- Elicitation, sampling, tool icons. +- A code-execution tool / Python sandbox (separate spec; the analytics + workflow does not require one for the goals above). +- Per-client schema-feature workarounds beyond what the audit findings + already cover. Codex's no-header-auth limitation is unrelated to tool + shape and is left to `setup-agents.ts` to document. +- Multi-tenancy, telemetry, rate limiting. + +## Design + +### 1. Surface change + +#### 1.1 Retained tools (11) + +| # | Tool | Port | +|---|---|---| +| 1 | `connection_list` | `KtxConnectionsMcpPort.list` | +| 2 | `discover_data` | `KtxDiscoverDataMcpPort.search` | +| 3 | `wiki_search` | `KtxKnowledgeMcpPort.search` | +| 4 | `wiki_read` | `KtxKnowledgeMcpPort.read` | +| 5 | `entity_details` | `KtxEntityDetailsMcpPort.read` | +| 6 | `dictionary_search` | `KtxDictionarySearchMcpPort.search` | +| 7 | `sl_read_source` | `KtxSemanticLayerMcpPort.readSource` | +| 8 | `sl_query` | `KtxSemanticLayerMcpPort.query` | +| 9 | `sql_execution` | `KtxSqlExecutionMcpPort.execute` | +| 10 | `memory_ingest` | New port `KtxMemoryIngestMcpPort.ingest` | +| 11 | `memory_ingest_status` | `KtxMemoryIngestMcpPort.status` | + +`connection_list` is retained because in multi-connection projects, the agent +needs a way to enumerate available connections before issuing a `sql_execution` +or `sl_query` against a specific one. + +**`connectionId` resolution per retained tool** (auto-resolution exists today +only on the local SL path; do not broaden it as part of this spec): + +| Tool | `connectionId` | Auto-resolves to single connection if omitted? | +|---|---|---| +| `connection_list` | n/a | n/a | +| `discover_data` | optional | no — search is run unscoped when omitted | +| `wiki_search` | n/a | n/a | +| `wiki_read` | n/a | n/a | +| `entity_details` | required | no | +| `dictionary_search` | optional | no — search is run unscoped when omitted | +| `sl_read_source` | required | no | +| `sl_query` | optional | yes — `resolveLocalConnectionId` (`packages/context/src/sl/local-query.ts`) auto-resolves when the project has exactly one connection | +| `sql_execution` | required | no | +| `memory_ingest` | optional | no — omitted means "global" knowledge (wiki only — see below) | +| `memory_ingest_status` | n/a | n/a | + +The skill update in §3 must reflect this matrix: when `connection_list` shows +multiple connections, the agent always passes `connectionId` for the required +tools and for `sl_query`/`discover_data`/`dictionary_search` whenever the user +intent pins a specific warehouse. + +**`memory_ingest` connectionId semantics — important constraint.** The +underlying `MemoryAgentService.ingest` derives `hasSL = !!input.connectionId` +(`packages/context/src/memory/memory-agent.service.ts:55`) and only wires the +SL-capable toolset when `connectionId` is supplied +(`packages/context/src/memory/memory-agent.service.ts:116-118`). Therefore +`memory_ingest` can update the semantic layer **only** when `connectionId` is +provided. Omit `connectionId` only for genuinely global wiki-only knowledge +(company-wide policies, vocabulary, user preferences); supply `connectionId` +for any knowledge that touches a specific warehouse — including measure +definitions, schema gotchas, and any wording like "in our warehouse" or "this +warehouse". The §3 SKILL update and the worked example must enforce this. + +#### 1.2 Removed from MCP registration + +`connection_test`, `wiki_write`, `sl_list_sources`, `sl_write_source`, +`sl_validate`, `ingest_trigger`, `ingest_status`, `ingest_report`, +`ingest_replay`, `scan_trigger`, `scan_status`, `scan_report`, +`scan_list_artifacts`, `scan_read_artifact`, +plus `memory_capture` and `memory_capture_status` (replaced). + +The conditional registration blocks in `registerKtxContextTools` for these +ports are removed. The underlying `KtxIngestMcpPort`, `KtxScanMcpPort`, etc. +implementations stay; the `ktx` CLI uses them directly. The `KtxMcpContextPorts` +type drops the removed `ingest?`, `scan?`, etc. fields. `MemoryCapturePort` is +renamed to `MemoryIngestPort`. + +#### 1.3 New tool — `memory_ingest` + +Replaces `memory_capture`. The change is a rename + a slightly relaxed input +contract; the underlying `MemoryCaptureService` +(`packages/context/src/memory/memory-runs.ts:81`) is reused as-is and renamed to +`MemoryIngestService`. No alias, no migration shim — per the standing +no-back-compat rule, the rename is atomic with the SKILL update. + +**Final internal API shape after the rename — no compatibility wrappers:** + +| Old name | New name | +|---|---| +| `MemoryCaptureService` (class) | `MemoryIngestService` | +| `MemoryCaptureService.capture(input)` (method) | `MemoryIngestService.ingest(input)` | +| `MemoryCaptureServiceDeps` | `MemoryIngestServiceDeps` | +| `MemoryCaptureStartResult` | `MemoryIngestStartResult` | +| `MemoryCaptureStatus` (return type) | `MemoryIngestStatus` | +| `MemoryCapturePort` (in `mcp/types.ts`) | `MemoryIngestPort` (with `.ingest()` and `.status()`) | +| `MemoryCapturePort.capture()` | `MemoryIngestPort.ingest()` | +| `TextMemoryCapturePort` (CLI, `text-ingest.ts`) | `TextMemoryIngestPort` (with `.ingest()`, `.waitForRun()`, `.status()`) | +| `createLocalProjectMemoryCapture` factory | `createLocalProjectMemoryIngest` | + +Every internal call site (`packages/context/src/mcp/server.ts`, +`packages/context/src/mcp/local-project-ports.ts`, +`packages/cli/src/mcp-server-factory.ts`, `packages/cli/src/text-ingest.ts`, +their tests, and the `packages/context/src/memory/index.ts` re-exports) is +updated in lockstep. The agent-facing `MemoryAgentService.ingest` method and +its `MemoryAgentInput` type are unchanged. + +**Mapping `memory_ingest` input → `MemoryAgentInput`** (defined in +`packages/context/src/memory/types.ts`): + +| `MemoryAgentInput` field | Value supplied by `memory_ingest` handler | +|---|---| +| `userId` | `userContext.userId` (existing pattern) | +| `chatId` | `mcp-${randomUUID()}` (existing pattern) | +| `userMessage` | synthetic framing string, e.g. `Ingest external knowledge into KTX memory.` | +| `assistantMessage` | input.`content` | +| `connectionId` | input.`connectionId` (when provided) | +| `sourceType` | `'external_ingest'` | + +The free-form markdown is routed into `assistantMessage` (not `userMessage`) +with a synthetic framing `userMessage`, mirroring the existing CLI text-ingest +path (`packages/cli/src/text-ingest.ts:295-302`). This mapping is required so +that `detectCaptureSignals` +(`packages/context/src/memory/capture-signals.ts:14`) can fire its +`assistantMessage`-keyed cues — SQL aggregates, LookML structure, and +definition tables — for artifact-like content. Routing content into +`userMessage` would lose those signals and silently degrade triage parity with +CLI ingest. The existing memory-agent prompt and tests already expect this +shape; no changes to the memory agent itself are required. + +Acceptance criterion: an MCP `memory_ingest` call with the same markdown +content as a CLI `ktx ingest text` invocation must produce identical +`CaptureSignals` (knowledge / sl / dialect / reasons) — covered by a parity +test that feeds the same fixture content through both ingest entry points and +asserts equal `detectCaptureSignals` output. + +**Input schema:** + +```typescript +const memoryIngestSchema = z.object({ + content: z + .string() + .min(1) + .describe( + 'Free-form markdown to ingest. Include the knowledge itself plus any ' + + 'context (source, the user\'s question, why this came up) that the ' + + 'memory agent should consider when triaging into wiki/SL.', + ), + connectionId: z + .string() + .min(1) + .optional() + .describe( + 'Scope this memory to a specific connection. REQUIRED when the knowledge ' + + 'is warehouse-specific (measure definitions, schema gotchas, anything ' + + 'tied to a particular warehouse) — without it the memory agent cannot ' + + 'update the semantic layer and the knowledge will land as wiki-only. ' + + 'Omit only for genuinely global wiki knowledge (company-wide policies, ' + + 'vocabulary, user preferences).', + ), +}); +``` + +**Tool description:** + +> Ingest free-form knowledge into KTX's durable memory so it is available to +> future turns. Call this whenever a research turn produces something worth +> remembering — business rules, metric definitions, gotchas, schema +> explanations, recurring findings — **or** whenever the user asks you to +> remember something. Pass everything in `content` as markdown: the finding, +> plus any source or context that helps the memory agent triage. KTX's memory +> agent decides whether the content belongs in the wiki, the semantic layer, +> or both. Each call is a feedback loop — better notes here mean smarter +> `discover_data` and `wiki_search` results for everyone next time. + +**Returns:** `{ runId: string }` — same shape as today's `memory_capture`. + +**`memory_ingest_status`** mirrors today's `memory_capture_status` exactly, +renamed only. + +### 2. Per-tool polish kit + +#### 2.0 Registration topology — memory tools must share the polish path + +Today `memory_capture` / `memory_capture_status` are registered in +`packages/context/src/mcp/server.ts` via direct `deps.server.registerTool` +calls (`server.ts:23,45`), **bypassing** `registerParsedTool` in +`context-tools.ts`. If left as-is, the polish kit below +(annotations, `outputSchema`, in-band error wrapping, per-field `.describe()`) +would not apply to `memory_ingest` / `memory_ingest_status`, contradicting the +"all 11 tools" acceptance criteria. + +Therefore, as part of the polish-kit PR (PR 2), one of the following must +happen — the implementation plan picks: + +1. **Preferred:** Move `memory_ingest` and `memory_ingest_status` registration + into `registerKtxContextTools` so they go through `registerParsedTool` like + every other tool. The `MemoryIngestPort` becomes a `contextTools.memoryIngest` + port and the standalone `registerMemoryCaptureTools` helper in `server.ts` + is deleted. +2. **Acceptable fallback:** Keep `registerMemoryIngestTools` in `server.ts` + but rewrite it to call `registerParsedTool` (exported from + `context-tools.ts`) so the same annotations / `outputSchema` / + error-wrapping plumbing is applied uniformly. + +Either way, every checklist item in §§2.1–2.4 must apply to the two memory +tools, and the §Verification annotations and `outputSchema` tests must cover +them. + +#### 2.1 Tool annotations + +Every tool gets annotations and a `title`: + +| Tool | title | readOnly | destructive | idempotent | openWorld | +|---|---|:--:|:--:|:--:|:--:| +| `connection_list` | Connection List | ✓ | — | ✓ | — | +| `discover_data` | Discover Data | ✓ | — | — | — | +| `wiki_search` | Wiki Search | ✓ | — | — | — | +| `wiki_read` | Wiki Read | ✓ | — | ✓ | — | +| `entity_details` | Entity Details | ✓ | — | ✓ | — | +| `dictionary_search` | Dictionary Search | ✓ | — | — | — | +| `sl_read_source` | Semantic Layer Read Source | ✓ | — | ✓ | — | +| `sl_query` | Semantic Layer Query | ✓ | — | — | — | +| `sql_execution` | SQL Execution | ✓ | — | — | — | +| `memory_ingest` | Memory Ingest | — | ✓ | — | — | +| `memory_ingest_status` | Memory Ingest Status | ✓ | — | omit | — | + +`openWorldHint: false` for every tool — even `sql_execution` targets a +configured, bounded warehouse, not the web. `sql_execution` is `readOnlyHint: +true` because the server-side parser enforces read-only (`assertReadOnlySql`). +`destructiveHint` is omitted (defaults to `false`) for read-only tools per the +MCP spec; explicit `false` is fine but redundant. + +`ToolAnnotations` are static optional booleans per the MCP 2025-11-25 schema +(`title?`, `readOnlyHint?`, `destructiveHint?`, `idempotentHint?`, +`openWorldHint?` — no state-dependent variants). `idempotentHint` describes +whether repeated calls have additional environmental effect and is most +meaningful when `readOnlyHint` is `false`. For `memory_ingest_status`, which is +a polling read whose response shape changes while a run is active, leave +`idempotentHint` unset — the tool is read-only but not statically idempotent. + +`registerTool` accepts annotations in the `config` object today; this is a +plumbing change in `registerParsedTool` to forward them. + +#### 2.2 `outputSchema` on all 11 tools + +Per the MCP 2025-11-25 spec, clients SHOULD validate `structuredContent` +against `outputSchema` when declared. Authoring is mechanical: each response +shape already typed in `packages/context/src/mcp/types.ts` gets a parallel +Zod schema and is passed as `outputSchema` to `registerTool`. + +`registerParsedTool` is extended to accept an optional `outputSchema` arg and +forward it to `server.registerTool`. The Zod schemas live alongside the +input schemas in `context-tools.ts` (or a sibling `tool-output-schemas.ts` if +the file grows too large). + +Example for `discover_data`: + +```typescript +const discoverDataOutputSchema = z.object({ + refs: z.array( + z.object({ + kind: discoverDataKindSchema, + id: z.string(), + score: z.number(), + summary: z.string().nullable(), + snippet: z.string().nullable(), + matchedOn: z.enum(['name', 'display', 'description', 'comment', 'expr', 'sample_value', 'body']), + connectionId: z.string().optional(), + tableRef: z.object({ catalog: z.string().nullable(), db: z.string().nullable(), name: z.string() }).optional(), + columnName: z.string().optional(), + }), + ), +}); +``` + +#### 2.3 Per-field `.describe()` on every input + +Anthropic's documented mechanism for fighting model drift, already used in +`memory_capture*`. Applied to every input field on every retained tool. +Highest leverage: `sl_query`, `entity_details`, `dictionary_search`, +`sql_execution`, `memory_ingest`. Tool-level `description` strings are +rewritten to be longer with one concrete example shape inlined (the technique +that fixed `order_by` model drift in this session). + +#### 2.4 In-band error wrapping in `registerParsedTool` + +Per MCP spec, tools return handler/runtime errors as `isError: true` + text +content, not JSON-RPC errors. Move the try/catch into the `registerParsedTool` +helper so every tool consistently surfaces handler exceptions as +`jsonErrorToolResult`. `sql_execution`'s local try/catch is removed (the +helper handles it). + +**Scope — what becomes in-band vs. what stays JSON-RPC.** The MCP SDK +pre-validates incoming arguments against the registered `inputSchema` before +the tool callback runs, and surfaces validation failures as +`McpError(InvalidParams)` / JSON-RPC errors +(`@modelcontextprotocol/sdk/dist/esm/server/mcp.js` `validateToolInput`, +~line 166). KTX cannot intercept those without forking the SDK and we will not. +Therefore: + +- Schema-validation failures on input → remain JSON-RPC `InvalidParams` errors, + emitted by the SDK before our handler runs. This is the documented MCP + behavior; clients already handle it. +- Handler exceptions, port/driver errors, and any post-validation runtime + errors thrown inside the tool body → wrapped in-band as + `{ isError: true, content: [{ type: 'text', ... }] }` by + `registerParsedTool`'s catch. +- The redundant `inputSchema.parse(input)` inside `registerParsedTool` may be + kept as defense-in-depth (e.g., for the rare path where the SDK was given a + raw shape and a downstream change loosens validation) or removed; either is + acceptable. If kept, parse failures here are wrapped in-band as well, but in + practice they are unreachable for valid SDK registrations because the SDK + has already parsed against the same schema. + +```typescript +function registerParsedTool( + server: KtxMcpServerLike, + name: string, + config: { title: string; description: string; inputSchema: unknown; outputSchema?: unknown; annotations: ToolAnnotations }, + inputSchema: TInput, + handler: (input: z.infer) => Promise, + outputSchema?: TOutput, +): void { + server.registerTool(name, { ...config, outputSchema: outputSchema ? outputSchema : undefined }, async (input) => { + try { + return await handler(inputSchema.parse(input)); + } catch (error) { + return jsonErrorToolResult(formatToolError(error)); + } + }); +} +``` + +A small `formatToolError` helper renders Zod errors with `path: message` lines +and falls through to `error.message` / `String(error)` for non-Zod cases. + +Acceptance tests in §Verification must therefore split the error path: + +- Bad input shape (rejected by SDK pre-validation) → expect a thrown + `McpError`/`InvalidParams` JSON-RPC error, not `isError: true`. +- Handler-thrown / port-thrown error (e.g., unknown `connectionId`, driver + failure) → expect `{ isError: true, content: [{ type: 'text', ... }] }`. + +#### 2.5 Union-drift normalization + +Apply the same `z.preprocess` pattern used for `order_by` to the two remaining +at-risk unions: + +**`slQueryDimensionSchema`** — accept `{ dimension, granularity }` (Cube +convention) as an alias for `{ field, granularity }`. Bare strings continue to +work unchanged. + +```typescript +const slQueryDimensionSchema = z.preprocess( + (value) => { + if (typeof value === 'string') return { field: value }; + if (value && typeof value === 'object' && !Array.isArray(value)) { + const obj = { ...(value as Record) }; + if (!('field' in obj) && typeof obj.dimension === 'string') obj.field = obj.dimension; + return obj; + } + return value; + }, + z.object({ + field: z.string().min(1).describe('Dimension to group by, e.g. "orders.created_at" or a SL dimension key.'), + granularity: z.string().min(1).optional().describe('Time granularity for time dimensions: day, week, month, quarter, year.'), + }), +); +``` + +**`entityDetailsTableRefSchema`** — accept `{ schema, table }` (BigQuery / +SQL-style convention) as an alias for `{ db, name }`. Today's schema requires +`catalog`, `db`, and `name` (`packages/context/src/mcp/context-tools.ts:169`), +so the alias path must also default `catalog` to `null` to satisfy the +validator. Either of the two equivalent shapes below is acceptable; the +implementation plan picks one: + +1. Make `catalog` (and `db`) `.nullable().default(null)` so the alias path + doesn't have to set them, and bare `{ schema, table }` is accepted with + `catalog === null`, `db === schema`, `name === table`. +2. Have the preprocess unconditionally fill missing `catalog`/`db` with `null`. + +Acceptance criterion: a tool call with `{ table: { schema: "public", table: "orders" } }` parses successfully and resolves to `{ catalog: null, db: "public", name: "orders" }`. Existing callers passing `{ catalog, db, name }` continue to work unchanged. + +```typescript +const entityDetailsTableRefSchema = z.preprocess( + (value) => { + if (value && typeof value === 'object' && !Array.isArray(value)) { + const obj = { ...(value as Record) }; + if (!('db' in obj) && typeof obj.schema === 'string') obj.db = obj.schema; + if (!('name' in obj) && typeof obj.table === 'string') obj.name = obj.table; + if (!('catalog' in obj)) obj.catalog = null; + return obj; + } + return value; + }, + z.object({ + catalog: z.string().nullable().describe('Catalog/project (BigQuery project, Snowflake database). null when not applicable.'), + db: z.string().nullable().describe('Schema/database for the table. null when not applicable.'), + name: z.string().min(1).describe('Table name.'), + }), +); +``` + +`slQueryMeasureSchema` is **not** changed — bare strings cover the common case +and `{ expr, name }` matches Cube's measure-with-alias convention; no observed +drift. + +#### 2.6 Type-narrow `jsonToolResult` + +The goal is to forbid arrays at compile time without breaking the existing +`interface`-typed response shapes (e.g. `KtxKnowledgeSearchResponse` in +`packages/context/src/mcp/types.ts`). `Record` is **not** +acceptable as the constraint because TypeScript interfaces without an index +signature are not assignable to it; using it would cause widespread compile +failures on valid object responses. + +Use a non-array object constraint instead, e.g.: + +```typescript +type NonArrayObject = object & { length?: never }; + +export function jsonToolResult( + structuredContent: T, +): KtxMcpToolResult { ... } +``` + +Acceptance criteria: + +- A bare array literal at a call site fails to type-check (catches the + `discover_data` bug class). +- Every existing `interface`-typed response (`KtxKnowledgeSearchResponse`, + `KtxSemanticLayerQueryResponse`, `KtxSqlExecutionResponse`, + `KtxEntityDetailsResponse`, `KtxDictionarySearchResponse`, + `KtxDiscoverDataResponse`, `KtxConnectionTestResponse`, + `KtxSemanticLayerReadResponse`, `KtxSemanticLayerListResponse`, + `KtxKnowledgePage`, memory capture/status response shapes) continues to + type-check at every existing `jsonToolResult` call site without modification. + +Implementation plan can substitute an equivalent narrowing if it is more +idiomatic; the contract is "no arrays, no breaking interface assignability." +Pure defensive type change; no runtime effect on current code. + +#### 2.7 Enforce the `toResolvedWire` invariant + +Add a doc comment on `KtxSemanticLayerComputePort.query` and +`.validateSources` stating that callers must pass `toResolvedWire`-sanitized +sources to prevent the daemon `usage`-leak bug from regressing if a new code +path bypasses `SemanticLayerService`. + +The doc comment alone is not sufficient — there is already an unsanitized +caller. `loadComputableSources` in +`packages/context/src/mcp/local-project-ports.ts` (~line 311) parses YAML and +pushes the raw record into `validateSources` +(`packages/context/src/mcp/local-project-ports.ts:586`). It must be brought into +conformance by sanitizing each record with `toResolvedWire` before handing it +to `validateSources`, mirroring the existing sanitization in +`packages/context/src/sl/local-query.ts:76`. Acceptance criterion: every +`KtxSemanticLayerComputePort.query` and `.validateSources` call site in the +repo passes `toResolvedWire`-sanitized records, verified by code review and +covered by the existing `local-query.test.ts` / `local-project-ports.test.ts` +tests for the relevant paths. Note that `sl_validate` is removed from MCP +registration (§1.2) but the underlying port keeps backing the CLI, so the +invariant must hold for the CLI path too. + +#### 2.8 Progress notifications — `sql_execution` + `sl_query` + +Per MCP spec, the caller may include `params._meta.progressToken` in the +request; the server emits `notifications/progress` with `{ progressToken, +progress, total?, message }` at stage transitions. + +`KtxSqlExecutionMcpPort.execute` and `KtxSemanticLayerMcpPort.query` are +extended with an optional `onProgress?: (event: { progress: number; total?: number; message: string }) => void` parameter. The MCP tool handlers wire +`onProgress` to the SDK's notification channel via the handler-context object +passed by `server.registerTool`. Non-progress-supporting clients ignore the +events. + +Emitted stages: + +- `sql_execution`: `"Validating SQL"` (progress 0.0) → `"Executing"` (0.3) → `"Fetched N rows"` (1.0). +- `sl_query`: `"Compiling query"` (0.0) → `"Generating SQL"` (0.3) → `"Executing"` (0.6) → `"Fetched N rows"` (1.0). + +Progress emission is best-effort; if the underlying port can't report a stage +boundary (e.g., a driver doesn't expose progress callbacks), the stage is +simply skipped. + +`memory_ingest` does not emit progress — `runId` + `memory_ingest_status` +polling is the documented async pattern for it. + +### 3. `ktx-analytics` SKILL.md refinements + +File: `packages/cli/src/skills/analytics/SKILL.md`. + +**Step 7 rewrite** (current vs. new): + +Current: + +> 7. **Capture durable learnings** - at the end of the turn, call `memory_capture` when the investigation produced reusable business context, metric definitions, or schema knowledge. + +New: + +> 7. **Capture durable learnings** - call `memory_ingest` whenever a turn produces something worth remembering (business rules, metric definitions, schema gotchas, recurring findings) **or** whenever the user asks you to remember something. Pass markdown in `content` including any source context the memory agent should weigh. Each call is a feedback loop — better notes today mean smarter `discover_data` and `wiki_search` results tomorrow. + +**Tool-name updates** throughout: every `memory_capture` reference becomes +`memory_ingest`. + +**Multi-connection rule** added under `` — phrased to match the §1.1 +connection matrix so the agent does not over-scope unscoped tools: + +> When `connection_list` shows multiple connections, pass an explicit +> `connectionId` to every tool that takes one **and where user intent pins a +> specific warehouse**. The matrix is: +> +> - **Required:** `entity_details`, `sl_read_source`, `sql_execution`. +> - **Required when user intent is warehouse-specific (including any wording +> like "in our warehouse" / "this warehouse"):** `memory_ingest` — without +> `connectionId`, the memory agent cannot update the semantic layer and the +> knowledge will land as wiki-only. +> - **Pass when intent pins a warehouse, otherwise omit for unscoped +> discovery:** `sl_query`, `discover_data`, `dictionary_search`. +> - **Never pass `connectionId` (the tool does not accept one):** +> `connection_list`, `wiki_search`, `wiki_read`, `memory_ingest_status`. +> +> If intent is ambiguous for a required-or-scoped tool, ask the user which +> warehouse before calling — do not guess. + +**One new worked example** demonstrating user-driven ingest: + +> **Input:** "Heads up — ARR is always reported in cents in our warehouse." +> +> **Workflow:** +> 1. If multiple connections, call `connection_list` and pick the warehouse the +> user means (asking if ambiguous). Pass its id as `connectionId` so the +> memory agent can update the semantic layer, not just the wiki. +> 2. `memory_ingest({ connectionId: "", content: "ARR is reported in cents (not dollars) in this warehouse. Multiply by 0.01 for dollar amounts. Source: user clarification." })` — no analysis turn; just remember. + +The existing Discover → Inspect → Resolve → Plan → Query → Validate → Capture +workflow stays. The existing two examples are updated only to reflect the +`memory_capture` → `memory_ingest` rename. + +## Migration / sequencing + +Three landings, each independently mergeable, in this order: + +### PR 1 — Surface change (atomic) + +- Remove the 14 admin tools from `registerKtxContextTools` (conditional + registration blocks deleted). +- Rename `memory_capture` → `memory_ingest`, `memory_capture_status` → + `memory_ingest_status`. Rename `MemoryCapturePort` → `MemoryIngestPort`, + `MemoryCaptureService` → `MemoryIngestService`. Update the new tool's input + contract per §1.3. +- Update `packages/context/src/mcp/local-project-ports.ts` and + `packages/context/src/mcp/server.ts` to reflect the renames and the dropped + ports. +- Update `packages/cli/src/skills/analytics/SKILL.md` per §3 in the same + diff. +- Update all tests for the removed/renamed tools. +- Update `docs-site/content/docs/integrations/agent-clients.mdx` to replace + the existing "memory capture" wording (currently at line ~90) with + "memory ingest". This update is unconditional — the file already names the + tool family. +- Update `packages/cli/src/mcp-server-factory.ts` and + `packages/cli/src/text-ingest.ts` (plus their tests) to reflect the + `MemoryCapture*` → `MemoryIngest*` rename. Both files import + `createLocalProjectMemoryCapture` and use a `memoryCapture` variable, so the + rename does cross the CLI boundary even though the tool surface used by the + CLI is unchanged. Re-export rename in `packages/context/src/memory/index.ts` + is part of this PR. + +The CLI's runtime use of the removed admin-tool implementations is unchanged — +only the memory rename touches CLI code. + +### PR 2 — Polish kit + +Touches all 11 retained tools. Can be one PR or split per family +(annotations + outputSchema + descriptions + error wrapping + union-drift +fixes + `jsonToolResult` type narrowing + `toResolvedWire` doc comment). + +### PR 3 — Progress notifications + +Extends `KtxSqlExecutionMcpPort` and `KtxSemanticLayerMcpPort` with optional +`onProgress`, wires the MCP handler context's notification channel, emits at +the stage boundaries listed in §2.8. + +Eventual **deletion** of the 14 admin tool implementations is a separate +follow-up spec gated on the `ktx-admin` SKILL landing. Until then they remain +in `packages/context/src/` and are used only by the CLI. + +## Verification + +### Unit tests per retained tool + +For each of the 11 retained tools: + +- Input schema accepts canonical input. +- Input schema accepts each documented normalized alt-shape (Cube + `{ dimension, granularity }`, `{ schema, table }`, bare-string `order_by`, + Cube-style `{ id, desc }` `order_by`). +- Output schema accepts the response shape returned by the underlying port. +- Error path returns `{ isError: true, content: [{ type: 'text', ... }] }` + (not a thrown exception). + +### Schema snapshot test + +A `tools/list` snapshot test in `packages/context/src/mcp/server.test.ts` +captures the exact JSON Schema each client receives for every tool. Re-runs +across PRs catch accidental schema drift (e.g., a Zod change silently +broadening the contract). + +### Annotations test + +Assert every tool's registered config carries the expected `readOnlyHint`, +`destructiveHint`, `idempotentHint`, `openWorldHint`, and `title` per §2.1. + +### Multi-client end-to-end smoke + +Stdio (Claude Desktop) and Streamable HTTP (Claude Code) are the two +transports; the other four clients (Codex, Cursor, OpenCode, universal) share +one of these transports and their config files are static. Spin up `ktx mcp +stdio` and `ktx mcp start`, call each retained tool through both transports, +verify response shape against `outputSchema`. + +### Required commands + +The named test files include slow tests that are excluded from the default +`@ktx/context` `test` script and live in `test:slow` +(`packages/context/package.json:127-128`). Implementation must run both: + +```bash +pnpm --filter @ktx/context run test +pnpm --filter @ktx/context run test:slow +pnpm --filter @ktx/context run type-check +pnpm --filter @ktx/cli run type-check +pnpm --filter @ktx/cli run test +pnpm run dead-code +``` + +CLI checks are required because PR 1 renames cross the CLI boundary +(`packages/cli/src/mcp-server-factory.ts`, `packages/cli/src/text-ingest.ts`, +the `MemoryCapture*` re-exports, plus their vitest specs). +`pnpm --filter @ktx/cli run test:slow` should also be added if PR 1 ends up +touching any of the slow-test files enumerated in `packages/cli/package.json` +(`scan.test.ts`, `setup*.test.ts`, etc.); the rename diff today does not, but +the implementation plan must re-check before merging. + +When `docs-site/content/docs/integrations/agent-clients.mdx` is touched, also +run the docs-site scripts declared in `docs-site/package.json` — there is no +`lint` script: + +```bash +pnpm --filter ktx-docs run build +pnpm --filter ktx-docs run test +``` + +### Red-green regression + +For the union-drift fixes (§2.5): revert the preprocess in `local-query.ts` or +`context-tools.ts`, run the alt-shape test → expect failure, restore → +expect pass. Same pattern as the `order_by` and `usage`-leak fixes in this +session. + +## Risks + +- **Removed tools surprise a user who depended on them via MCP.** Mitigated + by the no-back-compat rule (KTX is pre-public) and by the SKILL update + landing atomically with the surface change. Users on `ktx setup --agents` + flow get the updated SKILL the next time they re-run setup. +- **`outputSchema` validation breaks a client that doesn't tolerate + unrecognized JSON Schema keywords.** Mitigated by emitting `outputSchema` + via the same `server.registerTool` path that already produces `inputSchema`, + so both schemas are serialized by the MCP SDK as JSON Schema 2020-12 (the + dialect the SDK's tool-list types declare — + `@modelcontextprotocol/sdk/.../types.js` `inputSchema` / `outputSchema` + fields, and Appendix B). The spec uses SHOULD-validate semantics, not MUST. + The snapshot test catches drift; the multi-client smoke confirms + compatibility. +- **Progress notifications increase notification volume for clients that + poll.** Mitigated by stage-based emission (3-4 events per call max). Clients + that don't support progress simply ignore the events. +- **Renaming `MemoryCapturePort`/`MemoryCaptureService` cascades through + internal callers.** Cascade is bounded to `packages/context/src/memory/`, + `packages/context/src/mcp/`, and their tests; type-checker catches missed + call sites. + +## Open Questions + +None at design time. Open items for the implementation plan: + +- Final wording for the rewritten tool descriptions on `discover_data`, + `entity_details`, `dictionary_search`, `sl_read_source`, `sl_query`, + `sql_execution`. (Drafts can be authored during PR 2.) +- Whether `formatToolError` should redact path elements for security + (probably not — these are local-only MCP servers, and the existing error + shape doesn't redact). +- Whether to split PR 2 into per-family sub-PRs or keep it monolithic. Default + is monolithic since the polish-kit changes touch the same files and tests. + +## Appendix A — File map + +| Change | File | +|---|---| +| Tool registration removed (14 tools) | `packages/context/src/mcp/context-tools.ts` | +| Memory rename + route memory_ingest tools through the shared polish path (§2.0) | `packages/context/src/mcp/server.ts`, `packages/context/src/mcp/context-tools.ts`, `packages/context/src/memory/*` | +| Local ports update | `packages/context/src/mcp/local-project-ports.ts` | +| Port types update | `packages/context/src/mcp/types.ts` | +| Annotations, outputSchema, describe, error wrapping, union preprocess | `packages/context/src/mcp/context-tools.ts` | +| `jsonToolResult` type narrowing | `packages/context/src/mcp/context-tools.ts` | +| `toResolvedWire` invariant comment | `packages/context/src/daemon/semantic-layer-compute.ts` | +| Progress callback plumbing | `packages/context/src/daemon/semantic-layer-compute.ts`, `packages/context/src/sl/local-query.ts`, `packages/context/src/mcp/local-project-ports.ts` (`executeValidatedReadOnlySql`) | +| Tests | `packages/context/src/mcp/server.test.ts`, `packages/context/src/mcp/local-project-ports.test.ts`, `packages/context/src/sl/local-query.test.ts` | +| Skill | `packages/cli/src/skills/analytics/SKILL.md` | +| Docs | `docs-site/content/docs/integrations/agent-clients.mdx` | + +## Appendix B — MCP-spec cross-reference + +- Tool annotations (`readOnlyHint`, `destructiveHint`, `idempotentHint`, + `openWorldHint`, `title`): MCP 2025-11-25 spec `/schema` "ToolAnnotations". +- `outputSchema` with `structuredContent` SHOULD-validate: spec + `/server/tools` "Tool Result > Structured Content". +- In-band error contract (`isError: true` + text content, not JSON-RPC + error): spec `/server/tools` "Tool Execution Error Example". +- Progress notifications via `params._meta.progressToken` → + `notifications/progress`: spec `/basic/utilities/progress`. +- `inputSchema.type: "object"` requirement and JSON Schema 2020-12 default + dialect: spec `/server/tools` "Tool > inputSchema" and SEP 1613. diff --git a/packages/cli/package.json b/packages/cli/package.json index 2f1c4313..a93b9b6c 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -48,6 +48,7 @@ "@ktx/llm": "workspace:*", "@modelcontextprotocol/sdk": "^1.29.0", "commander": "14.0.3", + "fflate": "^0.8.2", "ink": "^7.0.2", "react": "^19.2.6", "zod": "^4.4.3" diff --git a/packages/cli/src/cli-program.ts b/packages/cli/src/cli-program.ts index e82cf0b5..f4021be6 100644 --- a/packages/cli/src/cli-program.ts +++ b/packages/cli/src/cli-program.ts @@ -186,6 +186,10 @@ function shouldSuppressProjectDirLine(path: string[], options: Record { 'serve-internal', 'start', 'status', + 'stdio', 'stop', ]); expect( @@ -54,4 +55,48 @@ describe('registerMcpCommands', () => { ); expect(startDaemon).not.toHaveBeenCalled(); }); + + it('prints "already running" when startDaemon reports already-running', async () => { + const program = new Command().exitOverride().option('--project-dir '); + const startDaemon = vi.fn().mockResolvedValue({ + status: 'already-running', + url: 'http://127.0.0.1:7878/mcp', + state: { + schemaVersion: 1, + pid: 4242, + host: '127.0.0.1', + port: 7878, + tokenAuth: false, + projectDir: '/tmp/ktx-already', + startedAt: '2026-05-14T00:00:00.000Z', + logPath: '/tmp/ktx-already/.ktx/logs/mcp.log', + }, + }); + const context = makeContext({ deps: { mcp: { startDaemon } } }); + registerMcpCommands(program, context); + + await program.parseAsync(['--project-dir', '/tmp/ktx-already', 'mcp', 'start'], { from: 'user' }); + + expect(startDaemon).toHaveBeenCalledTimes(1); + expect(context.io.stdout.write).toHaveBeenCalledWith( + 'KTX MCP daemon already running: http://127.0.0.1:7878/mcp\n', + ); + }); + + it('runs the stdio server with the resolved project directory', async () => { + const program = new Command().exitOverride().option('--project-dir '); + const runStdioServer = vi.fn().mockResolvedValue(undefined); + const context = makeContext({ deps: { mcp: { runStdioServer } } }); + registerMcpCommands(program, context); + + await expect(program.parseAsync(['--project-dir', '/tmp/ktx6', 'mcp', 'stdio'], { from: 'user' })).resolves.toBe( + program, + ); + + expect(runStdioServer).toHaveBeenCalledWith({ + projectDir: '/tmp/ktx6', + cliVersion: '0.0.0-test', + io: context.io, + }); + }); }); diff --git a/packages/cli/src/commands/mcp-commands.ts b/packages/cli/src/commands/mcp-commands.ts index 7880f608..9e583ce4 100644 --- a/packages/cli/src/commands/mcp-commands.ts +++ b/packages/cli/src/commands/mcp-commands.ts @@ -15,6 +15,7 @@ import { stopKtxMcpDaemon, } from '../managed-mcp-daemon.js'; import { buildMcpSecurityConfig, runKtxMcpHttpServer } from '../mcp-http-server.js'; +import { runKtxMcpStdioServer } from '../mcp-stdio-server.js'; function tokenFromOption(value: string | undefined): string | undefined { return value ?? process.env.KTX_MCP_TOKEN; @@ -27,6 +28,17 @@ function binPath(): string { export function registerMcpCommands(program: Command, context: KtxCliCommandContext): void { const mcp = program.command('mcp').description('Run the KTX MCP HTTP server'); + mcp + .command('stdio') + .description('Run the KTX MCP server over stdio') + .action(async (_options, command) => { + await (context.deps.mcp?.runStdioServer ?? runKtxMcpStdioServer)({ + projectDir: resolveCommandProjectDir(command), + cliVersion: context.packageInfo.version, + io: context.io, + }); + }); + mcp .command('start') .description('Start the KTX MCP HTTP server') @@ -70,7 +82,11 @@ export function registerMcpCommands(program: Command, context: KtxCliCommandCont allowedOrigins: options.allowedOrigin, binPath: binPath(), }); - context.io.stdout.write(`KTX MCP daemon started: ${result.url}\n`); + context.io.stdout.write( + result.status === 'started' + ? `KTX MCP daemon started: ${result.url}\n` + : `KTX MCP daemon already running: ${result.url}\n`, + ); }); mcp diff --git a/packages/cli/src/commands/setup-commands.ts b/packages/cli/src/commands/setup-commands.ts index d09f8149..a2b2a050 100644 --- a/packages/cli/src/commands/setup-commands.ts +++ b/packages/cli/src/commands/setup-commands.ts @@ -218,6 +218,7 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo .addOption( new Option('--target ', 'Agent target').choices([ 'claude-code', + 'claude-desktop', 'codex', 'cursor', 'opencode', diff --git a/packages/cli/src/managed-mcp-daemon.test.ts b/packages/cli/src/managed-mcp-daemon.test.ts index 7ffc277c..e28c4a3e 100644 --- a/packages/cli/src/managed-mcp-daemon.test.ts +++ b/packages/cli/src/managed-mcp-daemon.test.ts @@ -94,6 +94,78 @@ describe('managed MCP daemon lifecycle', () => { ); }); + it('returns already-running without spawning when the daemon is alive at the same host/port', async () => { + await mkdir(join(projectDir, '.ktx'), { recursive: true }); + await writeFile(join(projectDir, '.ktx/mcp.json'), `${JSON.stringify(state(projectDir), null, 2)}\n`); + const spawnDaemon = vi.fn(() => child(9999)); + + const result = await startKtxMcpDaemon({ + projectDir, + cliVersion: '0.0.0-test', + host: '127.0.0.1', + port: 7878, + allowedHosts: [], + allowedOrigins: [], + binPath: '/repo/packages/cli/dist/bin.js', + spawnDaemon, + processAlive: vi.fn(() => true), + portAvailable: vi.fn(async () => true), + }); + + expect(result.status).toBe('already-running'); + expect(result.url).toBe('http://127.0.0.1:7878/mcp'); + expect(result.state.pid).toBe(4242); + expect(spawnDaemon).not.toHaveBeenCalled(); + }); + + it('throws when the recorded daemon uses a different host or port', async () => { + await mkdir(join(projectDir, '.ktx'), { recursive: true }); + await writeFile(join(projectDir, '.ktx/mcp.json'), `${JSON.stringify(state(projectDir), null, 2)}\n`); + const spawnDaemon = vi.fn(() => child(9999)); + + await expect( + startKtxMcpDaemon({ + projectDir, + cliVersion: '0.0.0-test', + host: '127.0.0.1', + port: 9000, + allowedHosts: [], + allowedOrigins: [], + binPath: '/repo/packages/cli/dist/bin.js', + spawnDaemon, + processAlive: vi.fn(() => true), + portAvailable: vi.fn(async () => true), + }), + ).rejects.toThrow(/different configuration[\s\S]*ktx mcp stop/); + expect(spawnDaemon).not.toHaveBeenCalled(); + }); + + it('throws when token-auth presence differs from the recorded daemon', async () => { + await mkdir(join(projectDir, '.ktx'), { recursive: true }); + await writeFile( + join(projectDir, '.ktx/mcp.json'), + `${JSON.stringify(state(projectDir, { tokenAuth: false }), null, 2)}\n`, + ); + const spawnDaemon = vi.fn(() => child(9999)); + + await expect( + startKtxMcpDaemon({ + projectDir, + cliVersion: '0.0.0-test', + host: '127.0.0.1', + port: 7878, + token: 'secret-token', + allowedHosts: [], + allowedOrigins: [], + binPath: '/repo/packages/cli/dist/bin.js', + spawnDaemon, + processAlive: vi.fn(() => true), + portAvailable: vi.fn(async () => true), + }), + ).rejects.toThrow(/different configuration/); + expect(spawnDaemon).not.toHaveBeenCalled(); + }); + it('reports running when the process is alive and health passes', async () => { await mkdir(join(projectDir, '.ktx'), { recursive: true }); await writeFile(join(projectDir, '.ktx/mcp.json'), `${JSON.stringify(state(projectDir), null, 2)}\n`); diff --git a/packages/cli/src/managed-mcp-daemon.ts b/packages/cli/src/managed-mcp-daemon.ts index 96394f69..ef3df2a9 100644 --- a/packages/cli/src/managed-mcp-daemon.ts +++ b/packages/cli/src/managed-mcp-daemon.ts @@ -121,11 +121,25 @@ export async function startKtxMcpDaemon(options: { portAvailable?: (host: string, port: number) => Promise; spawnDaemon?: typeof defaultSpawnDaemon; now?: () => Date; -}): Promise<{ status: 'started'; state: KtxMcpDaemonState; url: string }> { +}): Promise<{ status: 'started' | 'already-running'; state: KtxMcpDaemonState; url: string }> { const existing = await readState(options.projectDir).catch(() => undefined); const processAlive = options.processAlive ?? defaultProcessAlive; if (existing && processAlive(existing.pid)) { - throw new Error(`KTX MCP daemon is already recorded at http://${existing.host}:${existing.port}/mcp`); + const sameConfig = + existing.host === options.host && + existing.port === options.port && + existing.tokenAuth === Boolean(options.token); + if (sameConfig) { + return { + status: 'already-running', + state: existing, + url: `http://${existing.host}:${existing.port}/mcp`, + }; + } + throw new Error( + `KTX MCP daemon is already running at http://${existing.host}:${existing.port}/mcp ` + + 'with a different configuration. Run `ktx mcp stop` first, then start again.', + ); } const portAvailable = options.portAvailable ?? defaultPortAvailable; if (!(await portAvailable(options.host, options.port))) { diff --git a/packages/cli/src/mcp-http-server.ts b/packages/cli/src/mcp-http-server.ts index 68e8eb3b..0d4607f0 100644 --- a/packages/cli/src/mcp-http-server.ts +++ b/packages/cli/src/mcp-http-server.ts @@ -1,16 +1,11 @@ import { randomUUID } from 'node:crypto'; import { createServer, type IncomingHttpHeaders, type IncomingMessage, type Server, type ServerResponse } from 'node:http'; -import { createDefaultKtxMcpServer, createLocalProjectMcpContextPorts } from '@ktx/context/mcp'; -import { createLocalProjectMemoryCapture } from '@ktx/context/memory'; -import { loadKtxProject, type KtxLocalProject } from '@ktx/context/project'; +import { loadKtxProject } from '@ktx/context/project'; import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js'; import { isInitializeRequest } from '@modelcontextprotocol/sdk/types.js'; import type { KtxCliIo } from './cli-runtime.js'; -import { createKtxCliIngestQueryExecutor } from './ingest-query-executor.js'; -import { createKtxCliScanConnector } from './local-scan-connectors.js'; -import { createManagedPythonSemanticLayerComputePort } from './managed-python-command.js'; -import { createManagedDaemonSqlAnalysisPort } from './managed-python-http.js'; +import { createKtxMcpServerFactory } from './mcp-server-factory.js'; const DEFAULT_ALLOWED_HOSTS = ['localhost', '127.0.0.1', '::1'] as const; @@ -124,13 +119,6 @@ export interface RunKtxMcpHttpServerOptions extends McpSecurityConfigInput { loadProject?: typeof loadKtxProject; } -function noopIo(): KtxCliIo { - return { - stdout: { write() {} }, - stderr: { write() {} }, - }; -} - function writeJson(res: ServerResponse, status: number, body: object): void { const payload = `${JSON.stringify(body)}\n`; res.writeHead(status, { @@ -159,55 +147,6 @@ async function readJsonBody(req: IncomingMessage): Promise { return raw.trim().length === 0 ? undefined : (JSON.parse(raw) as unknown); } -async function defaultMcpServerFactory(input: { - project: KtxLocalProject; - projectDir: string; - cliVersion: string; - io?: KtxCliIo; -}): Promise<() => McpServer> { - const io = input.io ?? noopIo(); - const queryExecutor = createKtxCliIngestQueryExecutor(input.project); - const semanticLayerCompute = await createManagedPythonSemanticLayerComputePort({ - cliVersion: input.cliVersion, - installPolicy: 'auto', - io, - }); - const sqlAnalysis = createManagedDaemonSqlAnalysisPort({ - cliVersion: input.cliVersion, - projectDir: input.projectDir, - installPolicy: 'auto', - io, - }); - const contextTools = createLocalProjectMcpContextPorts(input.project, { - semanticLayerCompute, - queryExecutor, - sqlAnalysis, - localScan: { - createConnector: async (connectionId) => createKtxCliScanConnector(input.project, connectionId), - }, - localIngest: { - semanticLayerCompute, - queryExecutor, - }, - }); - - let memoryCapture: ReturnType | undefined; - try { - memoryCapture = createLocalProjectMemoryCapture(input.project, { semanticLayerCompute, queryExecutor }); - } catch (error) { - input.io?.stderr.write(`KTX MCP memory_capture disabled: ${error instanceof Error ? error.message : String(error)}\n`); - } - - return () => - createDefaultKtxMcpServer({ - name: 'ktx', - version: input.cliVersion, - userContext: { userId: 'local' }, - contextTools, - memoryCapture, - }); -} - function listenerPort(server: Server, fallback: number): number { const address = server.address(); return typeof address === 'object' && address ? address.port : fallback; @@ -233,7 +172,7 @@ export async function runKtxMcpHttpServer(options: RunKtxMcpHttpServerOptions): : undefined; const createMcpServer = options.createMcpServer ?? - (await defaultMcpServerFactory({ + (await createKtxMcpServerFactory({ project: project!, projectDir: options.projectDir, cliVersion: options.cliVersion ?? '0.0.0-private', diff --git a/packages/cli/src/mcp-server-factory.ts b/packages/cli/src/mcp-server-factory.ts new file mode 100644 index 00000000..5209f9b8 --- /dev/null +++ b/packages/cli/src/mcp-server-factory.ts @@ -0,0 +1,63 @@ +import { createDefaultKtxMcpServer, createLocalProjectMcpContextPorts } from '@ktx/context/mcp'; +import { createLocalProjectMemoryIngest } from '@ktx/context/memory'; +import type { KtxLocalProject } from '@ktx/context/project'; +import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import type { KtxCliIo } from './cli-runtime.js'; +import { createKtxCliIngestQueryExecutor } from './ingest-query-executor.js'; +import { createKtxCliScanConnector } from './local-scan-connectors.js'; +import { createManagedPythonSemanticLayerComputePort } from './managed-python-command.js'; +import { createManagedDaemonSqlAnalysisPort } from './managed-python-http.js'; + +function noopMcpIo(): KtxCliIo { + return { + stdout: { write() {} }, + stderr: { write() {} }, + }; +} + +export async function createKtxMcpServerFactory(input: { + project: KtxLocalProject; + projectDir: string; + cliVersion: string; + io?: KtxCliIo; +}): Promise<() => McpServer> { + const io = input.io ?? noopMcpIo(); + const queryExecutor = createKtxCliIngestQueryExecutor(input.project); + const semanticLayerCompute = await createManagedPythonSemanticLayerComputePort({ + cliVersion: input.cliVersion, + installPolicy: 'auto', + io, + }); + const sqlAnalysis = createManagedDaemonSqlAnalysisPort({ + cliVersion: input.cliVersion, + projectDir: input.projectDir, + installPolicy: 'auto', + io, + }); + const contextTools = createLocalProjectMcpContextPorts(input.project, { + semanticLayerCompute, + queryExecutor, + sqlAnalysis, + localScan: { + createConnector: async (connectionId) => createKtxCliScanConnector(input.project, connectionId), + }, + }); + + let memoryIngest: ReturnType | undefined; + try { + memoryIngest = createLocalProjectMemoryIngest(input.project, { semanticLayerCompute, queryExecutor }); + } catch (error) { + input.io?.stderr.write(`KTX MCP memory_ingest disabled: ${error instanceof Error ? error.message : String(error)}\n`); + } + + return () => + createDefaultKtxMcpServer({ + name: 'ktx', + version: input.cliVersion, + userContext: { userId: 'local' }, + contextTools: { + ...contextTools, + ...(memoryIngest ? { memoryIngest } : {}), + }, + }); +} diff --git a/packages/cli/src/mcp-stdio-server.ts b/packages/cli/src/mcp-stdio-server.ts new file mode 100644 index 00000000..a755c7ae --- /dev/null +++ b/packages/cli/src/mcp-stdio-server.ts @@ -0,0 +1,64 @@ +import process from 'node:process'; +import type { Readable, Writable } from 'node:stream'; +import { loadKtxProject } from '@ktx/context/project'; +import type { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; +import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js'; +import type { KtxCliIo } from './cli-runtime.js'; +import { createKtxMcpServerFactory } from './mcp-server-factory.js'; + +export interface RunKtxMcpStdioServerOptions { + projectDir: string; + cliVersion?: string; + io?: KtxCliIo; + createMcpServer?: () => McpServer; + loadProject?: typeof loadKtxProject; + stdin?: Readable; + stdout?: Writable; +} + +export async function runKtxMcpStdioServer(options: RunKtxMcpStdioServerOptions): Promise { + const project = + options.createMcpServer === undefined + ? await (options.loadProject ?? loadKtxProject)({ projectDir: options.projectDir }) + : undefined; + const protocolIo: KtxCliIo = { + stdout: { write() {} }, + stderr: options.io?.stderr ?? { write() {} }, + }; + const createMcpServer = + options.createMcpServer ?? + (await createKtxMcpServerFactory({ + project: project!, + projectDir: options.projectDir, + cliVersion: options.cliVersion ?? '0.0.0-private', + io: protocolIo, + })); + const stdin = options.stdin ?? process.stdin; + const transport = new StdioServerTransport(stdin, options.stdout); + + await new Promise((resolve, reject) => { + let settled = false; + const settle = (callback: () => void) => { + if (settled) return; + settled = true; + stdin.off('end', closeTransport); + stdin.off('close', closeTransport); + callback(); + }; + const closeTransport = () => { + transport.close().catch((error: unknown) => { + settle(() => reject(error instanceof Error ? error : new Error(String(error)))); + }); + }; + transport.onclose = () => settle(resolve); + transport.onerror = (error) => { + options.io?.stderr.write(`KTX MCP stdio transport error: ${error.message}\n`); + settle(() => reject(error)); + }; + stdin.once('end', closeTransport); + stdin.once('close', closeTransport); + createMcpServer().connect(transport).catch((error: unknown) => { + settle(() => reject(error instanceof Error ? error : new Error(String(error)))); + }); + }); +} diff --git a/packages/cli/src/memory-flow-tui.test.tsx b/packages/cli/src/memory-flow-tui.test.tsx index e1df900a..e525a834 100644 --- a/packages/cli/src/memory-flow-tui.test.tsx +++ b/packages/cli/src/memory-flow-tui.test.tsx @@ -27,7 +27,7 @@ function replayInput(): MemoryFlowReplayInput { { unitKey: 'customers', target: 'sl', action: 'updated', key: 'orbit_demo.customers', summary: 'customer metrics', rawFiles: ['customers'], status: 'success' }, ], provenance: [{ rawPath: 'orders', artifactKind: 'wiki', artifactKey: 'wiki/orders.md', actionType: 'wiki_written' }], - transcripts: [{ unitKey: 'orders', path: '/tmp/t.jsonl', toolCallCount: 2, errorCount: 0, toolNames: ['read_raw_span', 'wiki_write'] }], + transcripts: [{ unitKey: 'orders', path: '/tmp/t.jsonl', toolCallCount: 2, errorCount: 0, toolNames: ['read_raw_span', 'memory_ingest'] }], }, events: [ { type: 'source_acquired', adapter: 'live-database', trigger: 'manual_resync', fileCount: 2 }, diff --git a/packages/cli/src/setup-agents.test.ts b/packages/cli/src/setup-agents.test.ts index cccf1474..3a073ce3 100644 --- a/packages/cli/src/setup-agents.test.ts +++ b/packages/cli/src/setup-agents.test.ts @@ -2,6 +2,7 @@ import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { readKtxSetupState } from '@ktx/context/project'; +import { strFromU8, unzipSync } from 'fflate'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { formatInstallSummary, @@ -24,6 +25,44 @@ function makeIo() { }; } +async function readZipText(path: string, entry: string): Promise { + const archive = unzipSync(new Uint8Array(await readFile(path))); + const content = archive[entry]; + if (!content) throw new Error(`Missing zip entry: ${entry}`); + return strFromU8(content); +} + +function captureEnvKeys(env: NodeJS.ProcessEnv, keys: readonly string[]): Record { + const snapshot: Record = {}; + for (const key of keys) snapshot[key] = env[key]; + return snapshot; +} + +function clearEnvKeys(env: NodeJS.ProcessEnv, keys: readonly string[]): void { + for (const key of keys) delete env[key]; +} + +function captureKtxEnv(env: NodeJS.ProcessEnv): Record { + const snapshot: Record = {}; + for (const key of Object.keys(env)) { + if (key.startsWith('KTX_')) snapshot[key] = env[key]; + } + return snapshot; +} + +function clearKtxEnv(env: NodeJS.ProcessEnv): void { + for (const key of Object.keys(env)) { + if (key.startsWith('KTX_')) delete env[key]; + } +} + +function restoreEnvKeys(env: NodeJS.ProcessEnv, snapshot: Record): void { + for (const [key, value] of Object.entries(snapshot)) { + if (value === undefined) delete env[key]; + else env[key] = value; + } +} + describe('setup agents', () => { let tempDir: string; @@ -37,28 +76,54 @@ describe('setup agents', () => { await rm(tempDir, { recursive: true, force: true }); }); - it('plans project-scoped CLI and research files for every target', () => { - expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'claude-code', scope: 'project', mode: 'cli' })).toEqual([ + it('plans project-scoped MCP analytics files for every target', () => { + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'claude-code', scope: 'project', mode: 'mcp' })).toEqual([ + { kind: 'file', path: join(tempDir, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, + ]); + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'claude-desktop', scope: 'global', mode: 'mcp' })).toEqual([ + { kind: 'file', path: join(tempDir, '.ktx/agents/claude/ktx-plugin-runner.sh'), role: 'launcher' }, + { kind: 'file', path: join(tempDir, '.ktx/agents/claude/ktx-plugin.zip'), role: 'claude-plugin' }, + ]); + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'codex', scope: 'project', mode: 'mcp' })).toEqual([ + { kind: 'file', path: join(tempDir, '.agents/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, + ]); + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'cursor', scope: 'project', mode: 'mcp' })).toEqual([ + { kind: 'file', path: join(tempDir, '.cursor/rules/ktx-analytics.mdc'), role: 'analytics-skill' }, + ]); + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'opencode', scope: 'project', mode: 'mcp' })).toEqual([ + { kind: 'file', path: join(tempDir, '.opencode/commands/ktx-analytics.md'), role: 'analytics-skill' }, + ]); + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'universal', scope: 'project', mode: 'mcp' })).toEqual([ + { kind: 'file', path: join(tempDir, '.agents/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, + ]); + }); + + it('plans project-scoped admin CLI files for every target when requested', () => { + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'claude-code', scope: 'project', mode: 'mcp-cli' })).toEqual([ + { kind: 'file', path: join(tempDir, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, { kind: 'file', path: join(tempDir, '.claude/skills/ktx/SKILL.md'), role: 'skill' }, - { kind: 'file', path: join(tempDir, '.claude/skills/ktx-research/SKILL.md'), role: 'research-skill' }, { kind: 'file', path: join(tempDir, '.claude/rules/ktx.md'), role: 'rule' }, ]); - expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'codex', scope: 'project', mode: 'cli' })).toEqual([ + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'codex', scope: 'project', mode: 'mcp-cli' })).toEqual([ + { kind: 'file', path: join(tempDir, '.agents/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, { kind: 'file', path: join(tempDir, '.agents/skills/ktx/SKILL.md'), role: 'skill' }, - { kind: 'file', path: join(tempDir, '.agents/skills/ktx-research/SKILL.md'), role: 'research-skill' }, { kind: 'file', path: join(tempDir, '.codex/instructions/ktx.md'), role: 'rule' }, ]); - expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'cursor', scope: 'project', mode: 'cli' })).toEqual([ + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'cursor', scope: 'project', mode: 'mcp-cli' })).toEqual([ + { kind: 'file', path: join(tempDir, '.cursor/rules/ktx-analytics.mdc'), role: 'analytics-skill' }, { kind: 'file', path: join(tempDir, '.cursor/rules/ktx.mdc') }, - { kind: 'file', path: join(tempDir, '.cursor/rules/ktx-research.mdc'), role: 'research-skill' }, ]); - expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'opencode', scope: 'project', mode: 'cli' })).toEqual([ + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'opencode', scope: 'project', mode: 'mcp-cli' })).toEqual([ + { kind: 'file', path: join(tempDir, '.opencode/commands/ktx-analytics.md'), role: 'analytics-skill' }, { kind: 'file', path: join(tempDir, '.opencode/commands/ktx.md') }, - { kind: 'file', path: join(tempDir, '.opencode/commands/ktx-research.md'), role: 'research-skill' }, ]); - expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'universal', scope: 'project', mode: 'cli' })).toEqual([ + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'universal', scope: 'project', mode: 'mcp-cli' })).toEqual([ + { kind: 'file', path: join(tempDir, '.agents/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, { kind: 'file', path: join(tempDir, '.agents/skills/ktx/SKILL.md') }, - { kind: 'file', path: join(tempDir, '.agents/skills/ktx-research/SKILL.md'), role: 'research-skill' }, + ]); + expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'claude-desktop', scope: 'global', mode: 'mcp-cli' })).toEqual([ + { kind: 'file', path: join(tempDir, '.ktx/agents/claude/ktx-plugin-runner.sh'), role: 'launcher' }, + { kind: 'file', path: join(tempDir, '.ktx/agents/claude/ktx-plugin.zip'), role: 'claude-plugin' }, ]); }); @@ -74,7 +139,7 @@ describe('setup agents', () => { agents: true, target: 'universal', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -82,7 +147,7 @@ describe('setup agents', () => { ).resolves.toEqual({ status: 'ready', projectDir: tempDir, - installs: [{ target: 'universal', scope: 'project', mode: 'cli' }], + installs: [{ target: 'universal', scope: 'project', mode: 'mcp-cli' }], }); await expect(stat(join(tempDir, '.agents/skills/ktx/SKILL.md'))).resolves.toBeDefined(); @@ -99,13 +164,13 @@ describe('setup agents', () => { expect(await readKtxAgentInstallManifest(tempDir)).toMatchObject({ version: 1, projectDir: tempDir, - installs: [{ target: 'universal', scope: 'project', mode: 'cli' }], + installs: [{ target: 'universal', scope: 'project', mode: 'mcp-cli' }], }); expect(await readKtxSetupState(tempDir)).toEqual({ completed_steps: ['agents'] }); expect(io.stderr()).toBe(''); }); - it('installs the research skill from the runtime asset', async () => { + it('installs the analytics skill from the runtime asset', async () => { const io = makeIo(); await expect( @@ -117,17 +182,20 @@ describe('setup agents', () => { agents: true, target: 'universal', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, ), ).resolves.toMatchObject({ status: 'ready' }); - const researchSkill = await readFile(join(tempDir, '.agents/skills/ktx-research/SKILL.md'), 'utf-8'); - expect(researchSkill).toContain('name: ktx-research'); - expect(researchSkill).toContain('Always run `discover_data` before writing SQL.'); - expect(researchSkill).toContain('Treat a `dictionary_search` miss as non-authoritative.'); + const analyticsSkill = await readFile(join(tempDir, '.agents/skills/ktx-analytics/SKILL.md'), 'utf-8'); + expect(analyticsSkill).toContain('name: ktx-analytics'); + expect(analyticsSkill).toContain('Always run `discover_data` before writing SQL.'); + expect(analyticsSkill).toContain('Treat a `dictionary_search` miss as non-authoritative.'); + expect(analyticsSkill).toContain('memory_ingest'); + expect(analyticsSkill).toContain('ARR is reported in cents'); + expect(analyticsSkill).not.toContain(`memory_${'capture'}`); }); it('writes PATH-independent launcher commands for skills', async () => { @@ -142,7 +210,7 @@ describe('setup agents', () => { agents: true, target: 'universal', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -170,7 +238,7 @@ describe('setup agents', () => { agents: true, target: 'claude-code', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -187,6 +255,279 @@ describe('setup agents', () => { expect(io.stdout()).toContain('Run `ktx mcp start` to enable the configured KTX MCP server.'); }); + it('prompts for MCP-first client agent connection mode in interactive setup', async () => { + const io = makeIo(); + const prompts = { + select: vi.fn(async ({ message }: { message: string }) => (message.startsWith('Where') ? 'project' : 'mcp')), + multiselect: vi.fn(async () => ['claude-code']), + cancel: vi.fn(), + }; + + await expect( + runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'auto', + yes: false, + agents: true, + scope: 'project', + mode: 'mcp', + skipAgents: false, + }, + io.io, + { prompts }, + ), + ).resolves.toMatchObject({ + status: 'ready', + installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp' }], + }); + + expect(prompts.select).toHaveBeenCalledWith({ + message: 'How should client agents connect to this KTX project?', + options: [ + { value: 'mcp', label: 'MCP tools + analytics skill' }, + { value: 'mcp-cli', label: 'MCP tools + analytics skill + admin CLI skill' }, + ], + }); + expect(prompts.multiselect).toHaveBeenCalledWith( + expect.objectContaining({ + options: expect.arrayContaining([{ value: 'claude-desktop', label: 'Claude Desktop' }]), + }), + ); + }); + + it('prompts for global scope when every selected target supports it', async () => { + const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-')); + const previousHome = process.env.HOME; + process.env.HOME = home; + try { + const io = makeIo(); + const prompts = { + select: vi.fn(async ({ message }: { message: string }) => + message.startsWith('Where should') ? 'global' : 'mcp', + ), + multiselect: vi.fn(async () => ['claude-code']), + cancel: vi.fn(), + }; + + await expect( + runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'auto', + yes: false, + agents: true, + scope: 'project', + mode: 'mcp', + skipAgents: false, + }, + io.io, + { prompts }, + ), + ).resolves.toMatchObject({ + status: 'ready', + installs: [{ target: 'claude-code', scope: 'global', mode: 'mcp' }], + }); + + expect(prompts.select).toHaveBeenCalledWith({ + message: 'Where should KTX install supported agent config?', + options: [ + { value: 'project', label: 'Project' }, + { value: 'global', label: 'Global' }, + ], + }); + } finally { + process.env.HOME = previousHome; + await rm(home, { recursive: true, force: true }); + } + }); + + it('registers Claude Desktop MCP via claude_desktop_config.json and ships a skills-only plugin', async () => { + const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-')); + const previousHome = process.env.HOME; + const envSnapshot = captureEnvKeys(process.env, ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY']); + const ktxEnvSnapshot = captureKtxEnv(process.env); + process.env.HOME = home; + clearEnvKeys(process.env, ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY']); + clearKtxEnv(process.env); + try { + const io = makeIo(); + + await expect( + runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'disabled', + yes: true, + agents: true, + target: 'claude-desktop', + scope: 'project', + mode: 'mcp', + skipAgents: false, + }, + io.io, + ), + ).resolves.toMatchObject({ + status: 'ready', + installs: [{ target: 'claude-desktop', scope: 'global', mode: 'mcp' }], + }); + + const pluginPath = join(tempDir, '.ktx/agents/claude/ktx-plugin.zip'); + const launcherPath = join(tempDir, '.ktx/agents/claude/ktx-plugin-runner.sh'); + await expect(stat(pluginPath)).resolves.toBeDefined(); + const launcherStat = await stat(launcherPath); + expect(launcherStat.mode & 0o111).not.toBe(0); + const launcher = await readFile(launcherPath, 'utf-8'); + expect(launcher).toContain('KTX_CLI_BIN='); + expect(launcher).toContain('.nvm/versions/node'); + + const configPath = join(home, 'Library/Application Support/Claude/claude_desktop_config.json'); + const config = JSON.parse(await readFile(configPath, 'utf-8')) as { + mcpServers: { ktx: { command: string; args: string[]; env?: Record } }; + }; + expect(config.mcpServers.ktx).toEqual({ + command: launcherPath, + args: ['--project-dir', tempDir, 'mcp', 'stdio'], + }); + + expect(await readZipText(pluginPath, '.claude-plugin/plugin.json')).toContain('"name": "ktx"'); + await expect(readZipText(pluginPath, '.mcp.json')).rejects.toThrow('Missing zip entry'); + expect(await readZipText(pluginPath, 'skills/ktx-analytics/SKILL.md')).toContain('KTX Analytics Workflow'); + const setupMd = await readZipText(pluginPath, 'SETUP.md'); + expect(setupMd).not.toContain('ktx mcp start'); + expect(setupMd).toContain('claude_desktop_config.json'); + await expect(readZipText(pluginPath, 'skills/ktx/SKILL.md')).rejects.toThrow('Missing zip entry'); + + expect(io.stdout()).toContain('Claude plugin generated'); + expect(io.stdout()).toContain('.ktx/agents/claude/ktx-plugin.zip'); + expect(io.stdout()).toContain('KTX MCP server registered'); + expect(io.stdout()).toContain('claude_desktop_config.json'); + expect(io.stdout()).toContain('Restart Claude Desktop'); + expect(io.stdout()).not.toContain('Run `ktx mcp start`'); + } finally { + process.env.HOME = previousHome; + restoreEnvKeys(process.env, envSnapshot); + restoreEnvKeys(process.env, ktxEnvSnapshot); + await rm(home, { recursive: true, force: true }); + } + }); + + it('captures KTX_*, OPENAI_API_KEY, and ANTHROPIC_API_KEY into the Claude Desktop MCP env block', async () => { + const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-')); + const previousHome = process.env.HOME; + const envSnapshot = captureEnvKeys(process.env, [ + 'OPENAI_API_KEY', + 'ANTHROPIC_API_KEY', + 'KTX_LOG_LEVEL', + ]); + const ktxEnvSnapshot = captureKtxEnv(process.env); + process.env.HOME = home; + clearKtxEnv(process.env); + process.env.OPENAI_API_KEY = 'sk-test-openai'; // pragma: allowlist secret + process.env.ANTHROPIC_API_KEY = 'sk-ant-test'; // pragma: allowlist secret + process.env.KTX_LOG_LEVEL = 'debug'; + try { + const io = makeIo(); + await runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'disabled', + yes: true, + agents: true, + target: 'claude-desktop', + scope: 'project', + mode: 'mcp', + skipAgents: false, + }, + io.io, + ); + + const configPath = join(home, 'Library/Application Support/Claude/claude_desktop_config.json'); + const config = JSON.parse(await readFile(configPath, 'utf-8')) as { + mcpServers: { ktx: { env?: Record } }; + }; + expect(config.mcpServers.ktx.env).toEqual({ + OPENAI_API_KEY: 'sk-test-openai', // pragma: allowlist secret + ANTHROPIC_API_KEY: 'sk-ant-test', // pragma: allowlist secret + KTX_LOG_LEVEL: 'debug', + }); + } finally { + process.env.HOME = previousHome; + restoreEnvKeys(process.env, envSnapshot); + restoreEnvKeys(process.env, ktxEnvSnapshot); + await rm(home, { recursive: true, force: true }); + } + }); + + it('includes the admin CLI skill in the Claude Desktop plugin zip when requested', async () => { + const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-')); + const previousHome = process.env.HOME; + process.env.HOME = home; + try { + const io = makeIo(); + + await expect( + runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'disabled', + yes: true, + agents: true, + target: 'claude-desktop', + scope: 'project', + mode: 'mcp-cli', + skipAgents: false, + }, + io.io, + ), + ).resolves.toMatchObject({ + status: 'ready', + installs: [{ target: 'claude-desktop', scope: 'global', mode: 'mcp-cli' }], + }); + + const pluginPath = join(tempDir, '.ktx/agents/claude/ktx-plugin.zip'); + const adminSkill = await readZipText(pluginPath, 'skills/ktx/SKILL.md'); + expect(adminSkill).toContain(`--project-dir ${tempDir}`); + expect(adminSkill).toContain('status --json'); + expect(await readZipText(pluginPath, 'SETUP.md')).toContain('admin CLI skill'); + await expect(readZipText(pluginPath, '.mcp.json')).rejects.toThrow('Missing zip entry'); + } finally { + process.env.HOME = previousHome; + await rm(home, { recursive: true, force: true }); + } + }); + + it('installs MCP client config and analytics skill without admin CLI files', async () => { + const io = makeIo(); + + await expect( + runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'disabled', + yes: true, + agents: true, + target: 'claude-code', + scope: 'project', + mode: 'mcp', + skipAgents: false, + }, + io.io, + ), + ).resolves.toMatchObject({ + status: 'ready', + installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp' }], + }); + + const mcpJson = JSON.parse(await readFile(join(tempDir, '.mcp.json'), 'utf-8')) as { + mcpServers: { ktx: { type: string; url: string } }; + }; + expect(mcpJson.mcpServers.ktx).toEqual({ type: 'http', url: 'http://localhost:7878/mcp' }); + await expect(stat(join(tempDir, '.claude/skills/ktx-analytics/SKILL.md'))).resolves.toBeDefined(); + await expect(stat(join(tempDir, '.claude/skills/ktx/SKILL.md'))).rejects.toThrow(); + await expect(stat(join(tempDir, '.claude/rules/ktx.md'))).rejects.toThrow(); + }); + it('writes Cursor project MCP config', async () => { const io = makeIo(); @@ -198,7 +539,7 @@ describe('setup agents', () => { agents: true, target: 'cursor', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -210,7 +551,7 @@ describe('setup agents', () => { expect(cursorJson.mcpServers.ktx).toEqual({ url: 'http://localhost:7878/mcp' }); }); - it('prints Codex and opencode snippets without mutating printed-only config files', async () => { + it('prints Codex, opencode, and universal snippets without mutating printed-only config files', async () => { const codexIo = makeIo(); await runKtxSetupAgentsStep( { @@ -220,7 +561,7 @@ describe('setup agents', () => { agents: true, target: 'codex', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, codexIo.io, @@ -237,7 +578,7 @@ describe('setup agents', () => { agents: true, target: 'opencode', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, opencodeIo.io, @@ -245,6 +586,23 @@ describe('setup agents', () => { expect(opencodeIo.stdout()).toContain('"mcp"'); expect(opencodeIo.stdout()).toContain('"type": "remote"'); await expect(readFile(join(tempDir, 'opencode.json'), 'utf-8')).rejects.toThrow(); + + const universalIo = makeIo(); + await runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'disabled', + yes: true, + agents: true, + target: 'universal', + scope: 'project', + mode: 'mcp', + skipAgents: false, + }, + universalIo.io, + ); + expect(universalIo.stdout()).toContain('Universal MCP endpoint:'); + expect(universalIo.stdout()).toContain('http://localhost:7878/mcp'); }); it('uses MCP daemon state for port and token metadata without rendering literal tokens', async () => { @@ -280,7 +638,7 @@ describe('setup agents', () => { agents: true, target: 'claude-code', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -314,7 +672,7 @@ describe('setup agents', () => { agents: true, target: 'claude-code', scope: 'local', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -340,7 +698,7 @@ describe('setup agents', () => { agents: true, target: 'claude-code', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -355,6 +713,50 @@ describe('setup agents', () => { await expect(readKtxAgentInstallManifest(tempDir)).resolves.toEqual(null); }); + it('removes generated Claude Desktop plugin from the manifest', async () => { + const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-')); + const previousHome = process.env.HOME; + process.env.HOME = home; + try { + const io = makeIo(); + await runKtxSetupAgentsStep( + { + projectDir: tempDir, + inputMode: 'disabled', + yes: true, + agents: true, + target: 'claude-desktop', + scope: 'project', + mode: 'mcp-cli', + skipAgents: false, + }, + io.io, + ); + const pluginPath = join(tempDir, '.ktx/agents/claude/ktx-plugin.zip'); + const launcherPath = join(tempDir, '.ktx/agents/claude/ktx-plugin-runner.sh'); + const configPath = join(home, 'Library/Application Support/Claude/claude_desktop_config.json'); + await expect(stat(pluginPath)).resolves.toBeDefined(); + await expect(stat(launcherPath)).resolves.toBeDefined(); + const beforeConfig = JSON.parse(await readFile(configPath, 'utf-8')) as { + mcpServers: Record; + }; + expect(beforeConfig.mcpServers.ktx).toBeDefined(); + + await expect(removeKtxAgentInstall(tempDir, io.io)).resolves.toBe(0); + + await expect(stat(pluginPath)).rejects.toThrow(); + await expect(stat(launcherPath)).rejects.toThrow(); + const afterConfig = JSON.parse(await readFile(configPath, 'utf-8')) as { + mcpServers: Record; + }; + expect(afterConfig.mcpServers.ktx).toBeUndefined(); + await expect(readKtxAgentInstallManifest(tempDir)).resolves.toEqual(null); + } finally { + process.env.HOME = previousHome; + await rm(home, { recursive: true, force: true }); + } + }); + it('treats cancel as skip in interactive mode', async () => { const io = makeIo(); const prompts = { @@ -371,7 +773,7 @@ describe('setup agents', () => { yes: false, agents: true, scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -383,7 +785,7 @@ describe('setup agents', () => { it('explains how to select multiple agent targets in interactive mode', async () => { const io = makeIo(); const prompts = { - select: vi.fn(async () => 'cli'), + select: vi.fn(async () => 'mcp-cli'), multiselect: vi.fn(async () => ['back']), cancel: vi.fn(), }; @@ -396,7 +798,7 @@ describe('setup agents', () => { yes: false, agents: true, scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -423,7 +825,7 @@ describe('setup agents', () => { agents: true, target: 'claude-code', scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io.io, @@ -432,21 +834,28 @@ describe('setup agents', () => { const output = io.stdout(); expect(output).toContain('Agent integration complete'); expect(output).toContain('Claude Code'); - expect(output).toContain('+ Skill installed — teaches your agent which KTX commands to run'); + expect(output).toContain('+ Analytics skill installed — teaches your agent the KTX MCP analytics workflow'); + expect(output).toContain('.claude/skills/ktx-analytics/SKILL.md'); + expect(output).toContain('+ Skill installed — teaches admin agents which KTX CLI commands to run'); expect(output).toContain('.claude/skills/ktx/SKILL.md'); - expect(output).toContain('+ Rule installed — tells your agent when to use KTX'); + expect(output).toContain('+ Rule installed — tells admin agents when to use KTX CLI'); expect(output).toContain('.claude/rules/ktx.md'); }); it('formats summary with relative paths for project scope', () => { const summary = formatInstallSummary( - [{ target: 'cursor', scope: 'project', mode: 'cli' }], - [{ kind: 'file', path: join(tempDir, '.cursor/rules/ktx.mdc') }], + [{ target: 'cursor', scope: 'project', mode: 'mcp-cli' }], + [ + { kind: 'file', path: join(tempDir, '.cursor/rules/ktx-analytics.mdc'), role: 'analytics-skill' }, + { kind: 'file', path: join(tempDir, '.cursor/rules/ktx.mdc') }, + ], tempDir, ); expect(summary).toContain('Cursor'); - expect(summary).toContain('+ Rule installed — tells your agent when to use KTX'); + expect(summary).toContain('+ Analytics skill installed — teaches your agent the KTX MCP analytics workflow'); + expect(summary).toContain('.cursor/rules/ktx-analytics.mdc'); + expect(summary).toContain('+ Rule installed — tells admin agents when to use KTX CLI'); expect(summary).toContain('.cursor/rules/ktx.mdc'); expect(summary).not.toContain(tempDir); }); @@ -454,12 +863,14 @@ describe('setup agents', () => { it('formats summary with multiple agent targets', () => { const summary = formatInstallSummary( [ - { target: 'claude-code', scope: 'project', mode: 'cli' }, - { target: 'codex', scope: 'project', mode: 'cli' }, + { target: 'claude-code', scope: 'project', mode: 'mcp-cli' }, + { target: 'codex', scope: 'project', mode: 'mcp-cli' }, ], [ + { kind: 'file', path: join(tempDir, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, { kind: 'file', path: join(tempDir, '.claude/skills/ktx/SKILL.md'), role: 'skill' }, { kind: 'file', path: join(tempDir, '.claude/rules/ktx.md'), role: 'rule' }, + { kind: 'file', path: join(tempDir, '.agents/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, { kind: 'file', path: join(tempDir, '.agents/skills/ktx/SKILL.md'), role: 'skill' }, { kind: 'file', path: join(tempDir, '.codex/instructions/ktx.md'), role: 'rule' }, ], @@ -467,9 +878,11 @@ describe('setup agents', () => { ); expect(summary).toContain('Claude Code'); - expect(summary).toContain('+ Skill installed — teaches your agent which KTX commands to run'); - expect(summary).toContain('+ Rule installed — tells your agent when to use KTX'); + expect(summary).toContain('+ Analytics skill installed — teaches your agent the KTX MCP analytics workflow'); + expect(summary).toContain('+ Skill installed — teaches admin agents which KTX CLI commands to run'); + expect(summary).toContain('+ Rule installed — tells admin agents when to use KTX CLI'); expect(summary).toContain('Codex'); + expect(summary).toContain('.agents/skills/ktx-analytics/SKILL.md'); expect(summary).toContain('.agents/skills/ktx/SKILL.md'); }); }); diff --git a/packages/cli/src/setup-agents.ts b/packages/cli/src/setup-agents.ts index ae7e91dc..be66b8a9 100644 --- a/packages/cli/src/setup-agents.ts +++ b/packages/cli/src/setup-agents.ts @@ -1,5 +1,5 @@ import { existsSync } from 'node:fs'; -import { mkdir, readFile, rm, writeFile } from 'node:fs/promises'; +import { chmod, mkdir, readFile, rm, writeFile } from 'node:fs/promises'; import { dirname, join, relative, resolve } from 'node:path'; import { fileURLToPath } from 'node:url'; import { @@ -7,17 +7,20 @@ import { markKtxSetupStateStepComplete, serializeKtxProjectConfig, } from '@ktx/context/project'; +import { strToU8, zipSync } from 'fflate'; import type { KtxCliIo } from './cli-runtime.js'; +import { bold, dim, green } from './io/symbols.js'; import { withMultiselectNavigation } from './prompt-navigation.js'; import { createKtxSetupPromptAdapter, + createKtxSetupUiAdapter, type KtxSetupPromptOption, } from './setup-prompts.js'; import { readKtxMcpDaemonStatus } from './managed-mcp-daemon.js'; -export type KtxAgentTarget = 'claude-code' | 'codex' | 'cursor' | 'opencode' | 'universal'; +export type KtxAgentTarget = 'claude-code' | 'claude-desktop' | 'codex' | 'cursor' | 'opencode' | 'universal'; export type KtxAgentScope = 'project' | 'global' | 'local'; -export type KtxAgentInstallMode = 'cli'; +export type KtxAgentInstallMode = 'mcp' | 'mcp-cli'; export interface KtxSetupAgentsArgs { projectDir: string; @@ -47,7 +50,7 @@ export interface KtxAgentInstallManifest { installedAt: string; installs: Array<{ target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode }>; entries: Array< - | { kind: 'file'; path: string; role?: 'skill' | 'rule' | 'research-skill' } + | { kind: 'file'; path: string; role?: 'skill' | 'rule' | 'analytics-skill' | 'claude-plugin' | 'launcher' } | { kind: 'json-key'; path: string; jsonPath: string[] } >; } @@ -169,6 +172,14 @@ function opencodeSnippet(endpoint: KtxMcpEndpointInfo): string { ); } +function universalMcpSnippet(endpoint: KtxMcpEndpointInfo): string { + return [ + 'Universal MCP endpoint:', + endpoint.url, + ...(endpoint.tokenAuth ? ['Header: Authorization: Bearer ${KTX_MCP_TOKEN}'] : []), + ].join('\n'); +} + function claudeConfigPath(projectDir: string, scope: KtxAgentScope): { path: string; jsonPath: string[] } { const home = process.env.HOME ?? ''; if (scope === 'global') { @@ -188,16 +199,63 @@ function cursorConfigPath(projectDir: string, scope: KtxAgentScope): { path: str }; } +function claudeDesktopConfigPath(): { path: string; jsonPath: string[] } { + const home = process.env.HOME ?? ''; + const path = + process.platform === 'win32' + ? join(process.env.APPDATA ?? join(home, 'AppData/Roaming'), 'Claude/claude_desktop_config.json') + : join(home, 'Library/Application Support/Claude/claude_desktop_config.json'); + return { path, jsonPath: ['mcpServers', 'ktx'] }; +} + +const CLAUDE_DESKTOP_FORWARDED_ENV_KEYS = ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY'] as const; + +export function collectClaudeDesktopForwardedEnv(source: NodeJS.ProcessEnv): Record { + const captured: Record = {}; + for (const [key, value] of Object.entries(source)) { + if (value === undefined || value === '') continue; + if (key.startsWith('KTX_') || (CLAUDE_DESKTOP_FORWARDED_ENV_KEYS as readonly string[]).includes(key)) { + captured[key] = value; + } + } + return captured; +} + +function claudeDesktopMcpEntry(input: { + launcherPath: string; + projectDir: string; + env?: NodeJS.ProcessEnv; +}): Record { + const captured = collectClaudeDesktopForwardedEnv(input.env ?? process.env); + return { + command: input.launcherPath, + args: ['--project-dir', input.projectDir, 'mcp', 'stdio'], + ...(Object.keys(captured).length > 0 ? { env: captured } : {}), + }; +} + async function installMcpClientConfig(input: { projectDir: string; target: KtxAgentTarget; scope: KtxAgentScope; }): Promise { - const endpoint = await resolveMcpEndpoint(input.projectDir); const entries: InstallEntry[] = []; const snippets: string[] = []; const notices: string[] = []; + if (input.target === 'claude-desktop') { + const config = claudeDesktopConfigPath(); + const launcherPath = claudeDesktopLauncherPath(input.projectDir); + await writeJsonKey( + config.path, + config.jsonPath, + claudeDesktopMcpEntry({ launcherPath, projectDir: input.projectDir }), + ); + entries.push({ kind: 'json-key', path: config.path, jsonPath: config.jsonPath }); + return { entries, snippets, notices }; + } + + const endpoint = await resolveMcpEndpoint(input.projectDir); if (!endpoint.running) { notices.push('Run `ktx mcp start` to enable the configured KTX MCP server.'); } @@ -213,74 +271,141 @@ async function installMcpClientConfig(input: { } else if (input.target === 'codex') { snippets.push(`Codex MCP snippet for ~/.codex/config.toml:\n${codexSnippet(endpoint)}`); } else if (input.target === 'opencode') { - const path = input.scope === 'global' ? '~/.config/opencode/opencode.json' : relative(input.projectDir, join(input.projectDir, 'opencode.json')); + const path = + input.scope === 'global' + ? '~/.config/opencode/opencode.json' + : relative(input.projectDir, join(input.projectDir, 'opencode.json')); snippets.push(`opencode MCP snippet for ${path}:\n${opencodeSnippet(endpoint)}`); + } else if (input.target === 'universal') { + snippets.push(universalMcpSnippet(endpoint)); } return { entries, snippets, notices }; } +function plannedMcpJsonEntries(input: { + projectDir: string; + target: KtxAgentTarget; + scope: KtxAgentScope; +}): InstallEntry[] { + if (input.target === 'claude-code') { + const config = claudeConfigPath(input.projectDir, input.scope); + return [{ kind: 'json-key', path: config.path, jsonPath: config.jsonPath }]; + } + if (input.target === 'claude-desktop') { + const config = claudeDesktopConfigPath(); + return [{ kind: 'json-key', path: config.path, jsonPath: config.jsonPath }]; + } + if (input.target === 'cursor') { + const config = cursorConfigPath(input.projectDir, input.scope); + return [{ kind: 'json-key', path: config.path, jsonPath: config.jsonPath }]; + } + return []; +} + export function agentInstallManifestPath(projectDir: string): string { return join(resolve(projectDir), '.ktx/agents/install-manifest.json'); } +function claudeDesktopPluginPath(projectDir: string): string { + return join(resolve(projectDir), '.ktx/agents/claude/ktx-plugin.zip'); +} + +function claudeDesktopLauncherPath(projectDir: string): string { + return join(resolve(projectDir), '.ktx/agents/claude/ktx-plugin-runner.sh'); +} + export function plannedKtxAgentFiles(input: { projectDir: string; target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode; }): InstallEntry[] { + const withAdminCli = input.mode === 'mcp-cli'; + if (input.scope === 'global') { if (input.target === 'claude-code') { const home = process.env.HOME ?? ''; return [ - { kind: 'file', path: join(home, '.claude/skills/ktx/SKILL.md'), role: 'skill' as const }, - { kind: 'file', path: join(home, '.claude/skills/ktx-research/SKILL.md'), role: 'research-skill' as const }, - { kind: 'file', path: join(home, '.claude/rules/ktx.md'), role: 'rule' as const }, + { kind: 'file', path: join(home, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' as const }, + ...(withAdminCli + ? [ + { kind: 'file' as const, path: join(home, '.claude/skills/ktx/SKILL.md'), role: 'skill' as const }, + { kind: 'file' as const, path: join(home, '.claude/rules/ktx.md'), role: 'rule' as const }, + ] + : []), ]; } if (input.target === 'codex') { const codexHome = process.env.CODEX_HOME ?? join(process.env.HOME ?? '', '.codex'); return [ - { kind: 'file', path: join(codexHome, 'skills/ktx/SKILL.md'), role: 'skill' as const }, - { kind: 'file', path: join(codexHome, 'skills/ktx-research/SKILL.md'), role: 'research-skill' as const }, - { kind: 'file', path: join(codexHome, 'instructions/ktx.md'), role: 'rule' as const }, + { kind: 'file', path: join(codexHome, 'skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' as const }, + ...(withAdminCli + ? [ + { kind: 'file' as const, path: join(codexHome, 'skills/ktx/SKILL.md'), role: 'skill' as const }, + { kind: 'file' as const, path: join(codexHome, 'instructions/ktx.md'), role: 'rule' as const }, + ] + : []), ]; } if (input.target === 'cursor' || input.target === 'opencode') { return []; } + if (input.target === 'claude-desktop') { + return [ + { kind: 'file', path: claudeDesktopLauncherPath(input.projectDir), role: 'launcher' as const }, + { kind: 'file', path: claudeDesktopPluginPath(input.projectDir), role: 'claude-plugin' as const }, + ]; + } throw new Error(`Global ${input.target} installation is not supported; omit --global.`); } const root = resolve(input.projectDir); + const analyticsEntries: Partial> = { + 'claude-code': [ + { kind: 'file', path: join(root, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, + ], + codex: [ + { kind: 'file', path: join(root, '.agents/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, + ], + cursor: [ + { kind: 'file', path: join(root, '.cursor/rules/ktx-analytics.mdc'), role: 'analytics-skill' }, + ], + opencode: [ + { kind: 'file', path: join(root, '.opencode/commands/ktx-analytics.md'), role: 'analytics-skill' }, + ], + universal: [ + { kind: 'file', path: join(root, '.agents/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' }, + ], + 'claude-desktop': [], + }; const cliEntries: Partial> = { 'claude-code': [ { kind: 'file', path: join(root, '.claude/skills/ktx/SKILL.md'), role: 'skill' }, - { kind: 'file', path: join(root, '.claude/skills/ktx-research/SKILL.md'), role: 'research-skill' }, ], codex: [ { kind: 'file', path: join(root, '.agents/skills/ktx/SKILL.md'), role: 'skill' }, - { kind: 'file', path: join(root, '.agents/skills/ktx-research/SKILL.md'), role: 'research-skill' }, ], cursor: [ { kind: 'file', path: join(root, '.cursor/rules/ktx.mdc') }, - { kind: 'file', path: join(root, '.cursor/rules/ktx-research.mdc'), role: 'research-skill' }, ], opencode: [ { kind: 'file', path: join(root, '.opencode/commands/ktx.md') }, - { kind: 'file', path: join(root, '.opencode/commands/ktx-research.md'), role: 'research-skill' }, ], universal: [ { kind: 'file', path: join(root, '.agents/skills/ktx/SKILL.md') }, - { kind: 'file', path: join(root, '.agents/skills/ktx-research/SKILL.md'), role: 'research-skill' }, ], + 'claude-desktop': [], }; const ruleEntries: Partial> = { 'claude-code': { kind: 'file', path: join(root, '.claude/rules/ktx.md'), role: 'rule' }, codex: { kind: 'file', path: join(root, '.codex/instructions/ktx.md'), role: 'rule' }, }; - return [...(cliEntries[input.target] ?? []), ruleEntries[input.target]].filter( + return [ + ...(analyticsEntries[input.target] ?? []), + ...(withAdminCli ? (cliEntries[input.target] ?? []) : []), + ...(withAdminCli ? [ruleEntries[input.target]] : []), + ].filter( (entry): entry is InstallEntry => entry !== undefined, ); } @@ -292,8 +417,8 @@ function ktxCliLauncher(): KtxCliLauncher { }; } -async function readResearchSkillContent(): Promise { - const path = fileURLToPath(new URL('./skills/research/SKILL.md', import.meta.url)); +async function readAnalyticsSkillContent(): Promise { + const path = fileURLToPath(new URL('./skills/analytics/SKILL.md', import.meta.url)); const content = await readFile(path, 'utf-8'); return content.endsWith('\n') ? content : `${content}\n`; } @@ -305,6 +430,10 @@ function shellQuote(value: string): string { return `'${value.replaceAll("'", "'\\''")}'`; } +function shellScriptQuote(value: string): string { + return `'${value.replaceAll("'", "'\\''")}'`; +} + function ktxCommandLine(launcher: KtxCliLauncher, args: string[]): string { return [launcher.command, ...launcher.args, ...args].map(shellQuote).join(' '); } @@ -320,11 +449,14 @@ function cliInstructionContent(input: { projectDir: string; launcher: KtxCliLaun '', '# KTX Local Context', '', + 'This is an admin/developer CLI helper. End-user data agents should use the KTX MCP tools when available.', + '', `Use this project with \`--project-dir ${input.projectDir}\`.`, 'Commands are pinned to the local KTX CLI path that created this file, so agents do not need `ktx` in PATH.', 'If the CLI path no longer exists after moving this checkout or reinstalling KTX, rerun `ktx setup --agents`.', '', - 'Agents must not print secrets, credential references, environment variable values, or file contents from `.ktx/secrets`.', + 'Agents must not print secrets, credential references, environment variable values, or file contents from ' + + '`.ktx/secrets`.', '', 'Available commands:', '', @@ -352,9 +484,132 @@ function cliInstructionContent(input: { projectDir: string; launcher: KtxCliLaun ].join('\n'); } +function claudePluginJsonContent(): string { + return `${JSON.stringify( + { + name: 'ktx', + version: '0.0.0-local', + description: 'KTX analytics workflow guidance and local MCP tools.', + }, + null, + 2, + )}\n`; +} + +function claudePluginVersionContent(): string { + return `${JSON.stringify({ version: '0.0.0-local' }, null, 2)}\n`; +} + +function claudePluginSetupContent(input: { projectDir: string; withAdminCli: boolean }): string { + return [ + '# KTX Claude Plugin', + '', + 'Install this plugin ZIP from Claude Desktop to load the KTX analytics skill.', + '', + `KTX project: \`${input.projectDir}\``, + '', + 'Included:', + '', + '- `ktx-analytics` skill for the MCP analytics workflow', + ...(input.withAdminCli ? ['- `ktx` admin CLI skill for KTX maintenance commands'] : []), + '', + 'The KTX MCP server is registered separately in `claude_desktop_config.json` by `ktx setup` and runs as a local stdio child of Claude Desktop — no daemon to start.', + '', + 'If this checkout or project directory moves, rerun `ktx setup --agents` and reinstall the regenerated plugin.', + '', + ].join('\n'); +} + +function claudePluginLauncherContent(input: { launcher: KtxCliLauncher }): string { + const binPath = input.launcher.args[0]; + if (!binPath) { + throw new Error('Expected KTX CLI launcher to include a bin path.'); + } + const candidates = [ + input.launcher.command, + '/opt/homebrew/bin/node', + '/usr/local/bin/node', + '/usr/bin/node', + ]; + return [ + '#!/bin/sh', + 'set -eu', + '', + `KTX_CLI_BIN=${shellScriptQuote(binPath)}`, + '', + 'run_with_node() {', + ' node_bin=$1', + ' shift', + ' exec "$node_bin" "$KTX_CLI_BIN" "$@"', + '}', + '', + 'if [ -n "${KTX_NODE:-}" ] && [ -x "${KTX_NODE:-}" ]; then', + ' run_with_node "$KTX_NODE" "$@"', + 'fi', + '', + 'if [ -x "$HOME/.volta/bin/node" ]; then', + ' run_with_node "$HOME/.volta/bin/node" "$@"', + 'fi', + '', + ...candidates.map((candidate) => + [ + `if [ -x ${shellScriptQuote(candidate)} ]; then`, + ` run_with_node ${shellScriptQuote(candidate)} "$@"`, + 'fi', + ].join('\n'), + ), + '', + 'for candidate in "$HOME"/.nvm/versions/node/*/bin/node; do', + ' if [ -x "$candidate" ]; then', + ' run_with_node "$candidate" "$@"', + ' fi', + 'done', + '', + 'if command -v node >/dev/null 2>&1; then', + ' run_with_node "$(command -v node)" "$@"', + 'fi', + '', + 'echo "KTX plugin could not find Node.js. Set KTX_NODE to a Node executable and reinstall the plugin." >&2', + 'exit 127', + '', + ].join('\n'); +} + +async function writeClaudeDesktopPlugin(input: { + projectDir: string; + path: string; + mode: KtxAgentInstallMode; + launcher: KtxCliLauncher; +}): Promise { + const withAdminCli = input.mode === 'mcp-cli'; + const files: Record = { + '.claude-plugin/plugin.json': strToU8(claudePluginJsonContent()), + 'version.json': strToU8(claudePluginVersionContent()), + 'skills/ktx-analytics/SKILL.md': strToU8(await readAnalyticsSkillContent()), + 'SETUP.md': strToU8(claudePluginSetupContent({ projectDir: input.projectDir, withAdminCli })), + }; + if (withAdminCli) { + files['skills/ktx/SKILL.md'] = strToU8( + cliInstructionContent({ projectDir: input.projectDir, launcher: input.launcher }), + ); + } + await mkdir(dirname(input.path), { recursive: true }); + await writeFile(input.path, Buffer.from(zipSync(files))); +} + +async function writeClaudeDesktopLauncher(input: { + path: string; + launcher: KtxCliLauncher; +}): Promise { + await mkdir(dirname(input.path), { recursive: true }); + await writeFile(input.path, claudePluginLauncherContent({ launcher: input.launcher }), 'utf-8'); + await chmod(input.path, 0o755); +} + function ruleInstructionContent(input: { projectDir: string }): string { return [ - `Use the \`ktx\` CLI to query local semantic context and wiki knowledge for this project (\`--project-dir ${input.projectDir}\`).`, + `Use the \`ktx\` CLI to query local semantic context and wiki knowledge for this project ` + + `(\`--project-dir ${input.projectDir}\`).`, '', 'Use when the user asks about data schemas, metrics, dimensions, database structure, or wants to run SQL queries.', '', @@ -390,7 +645,9 @@ async function writeManifest(projectDir: string, manifest: KtxAgentInstallManife } function entryKey(entry: InstallEntry): string { - return entry.kind === 'json-key' ? `${entry.kind}:${entry.path}:${entry.jsonPath.join('.')}` : `${entry.kind}:${entry.path}`; + return entry.kind === 'json-key' + ? `${entry.kind}:${entry.path}:${entry.jsonPath.join('.')}` + : `${entry.kind}:${entry.path}`; } function mergeManifest( @@ -455,6 +712,7 @@ function createPromptAdapter(): KtxSetupAgentsPromptAdapter { const targetDisplayNames: Record = { 'claude-code': 'Claude Code', + 'claude-desktop': 'Claude Desktop', codex: 'Codex', cursor: 'Cursor', opencode: 'OpenCode', @@ -463,12 +721,25 @@ const targetDisplayNames: Record = { const fileEntryLabels: Record = { 'claude-code': 'Skill installed', + 'claude-desktop': 'Skill installed', codex: 'Skill installed', cursor: 'Rule installed', opencode: 'Command installed', universal: 'Skill installed', }; +function mcpEntryLabel(entry: Extract): string { + return `MCP config installed — connects client agents to KTX MCP tools (${entry.jsonPath.join('.')})`; +} + +function targetSupportsGlobalScope(target: KtxAgentTarget): boolean { + return target === 'claude-code' || target === 'codex'; +} + +function effectiveInstallScope(target: KtxAgentTarget, requestedScope: KtxAgentScope): KtxAgentScope { + return target === 'claude-desktop' ? 'global' : requestedScope; +} + export function formatInstallSummary( installs: Array<{ target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode }>, entries: InstallEntry[], @@ -486,11 +757,21 @@ export function formatInstallSummary( entries.filter((entry) => entry.kind === 'file' && plannedFilePaths.has(entry.path)), ); } + const mcpEntriesByTarget = new Map(); + for (const install of installs) { + const plannedMcpKeys = new Set(plannedMcpJsonEntries({ projectDir, ...install }).map(entryKey)); + mcpEntriesByTarget.set( + install.target, + entries.filter((entry) => entry.kind === 'json-key' && plannedMcpKeys.has(entryKey(entry))), + ); + } const fileHints: Record = { - skill: 'teaches your agent which KTX commands to run', - rule: 'tells your agent when to use KTX', - 'research-skill': 'teaches your agent the KTX MCP research workflow', + skill: 'teaches admin agents which KTX CLI commands to run', + rule: 'tells admin agents when to use KTX CLI', + 'analytics-skill': 'teaches your agent the KTX MCP analytics workflow', + 'claude-plugin': 'bundles KTX skills for Claude Desktop (MCP server is registered in claude_desktop_config.json)', + launcher: 'runs the local KTX CLI with an available Node.js for Claude Desktop', }; const lines: string[] = []; @@ -498,16 +779,34 @@ export function formatInstallSummary( const targetEntries = entriesByTarget.get(install.target) ?? []; lines.push(` ${targetDisplayNames[install.target]}`); for (const entry of targetEntries) { - const displayPath = - install.scope === 'global' ? entry.path : relative(projectDir, entry.path); if (entry.kind === 'file') { - const isRule = entry.role === 'rule' || fileEntryLabels[install.target] === 'Rule installed'; - const label = entry.role === 'research-skill' ? 'Research skill installed' : isRule ? 'Rule installed' : fileEntryLabels[install.target]; + const isRule = entry.role === 'rule' || (!entry.role && fileEntryLabels[install.target] === 'Rule installed'); + const label = + entry.role === 'analytics-skill' + ? 'Analytics skill installed' + : entry.role === 'claude-plugin' + ? 'Claude plugin generated' + : entry.role === 'launcher' + ? 'Launcher installed' + : isRule + ? 'Rule installed' + : fileEntryLabels[install.target]; const hint = fileHints[isRule ? 'rule' : (entry.role ?? 'skill')] ?? ''; lines.push(` + ${label} — ${hint}`); - lines.push(` ${displayPath}`); + if (entry.role !== 'claude-plugin') { + const displayPath = + install.scope === 'global' ? entry.path : relative(projectDir, entry.path); + lines.push(` ${displayPath}`); + } } } + for (const entry of mcpEntriesByTarget + .get(install.target) + ?.filter((entry): entry is Extract => entry.kind === 'json-key') ?? []) { + const displayPath = install.scope === 'global' ? entry.path : relative(projectDir, entry.path); + lines.push(` + ${mcpEntryLabel(entry)}`); + lines.push(` ${displayPath}`); + } } return lines.join('\n'); } @@ -522,11 +821,24 @@ async function installTarget(input: { const launcher = ktxCliLauncher(); for (const entry of entries) { if (entry.kind !== 'file') continue; + if (entry.role === 'launcher') { + await writeClaudeDesktopLauncher({ path: entry.path, launcher }); + continue; + } + if (entry.role === 'claude-plugin') { + await writeClaudeDesktopPlugin({ + projectDir: input.projectDir, + path: entry.path, + mode: input.mode, + launcher, + }); + continue; + } const content = entry.role === 'rule' ? ruleInstructionContent({ projectDir: input.projectDir }) - : entry.role === 'research-skill' - ? await readResearchSkillContent() + : entry.role === 'analytics-skill' + ? await readAnalyticsSkillContent() : cliInstructionContent({ projectDir: input.projectDir, launcher }); await mkdir(dirname(entry.path), { recursive: true }); await writeFile(entry.path, content, 'utf-8'); @@ -558,14 +870,13 @@ export async function runKtxSetupAgentsStep( args.inputMode === 'disabled' ? args.mode : ((await prompts.select({ - message: 'How should agents use this KTX project?', + message: 'How should client agents connect to this KTX project?', options: [ - { value: 'cli', label: 'CLI tools and skills' }, - { value: 'skip', label: 'Skip' }, + { value: 'mcp', label: 'MCP tools + analytics skill' }, + { value: 'mcp-cli', label: 'MCP tools + analytics skill + admin CLI skill' }, ], - })) as KtxAgentInstallMode | 'skip' | 'back'); + })) as KtxAgentInstallMode | 'back'); if (mode === 'back') return { status: 'skipped', projectDir: args.projectDir }; - if (mode === 'skip') return { status: 'skipped', projectDir: args.projectDir }; const targets = args.target !== undefined @@ -576,6 +887,7 @@ export async function runKtxSetupAgentsStep( message: withMultiselectNavigation('Which agent targets should KTX install?'), options: [ { value: 'claude-code', label: 'Claude Code' }, + { value: 'claude-desktop', label: 'Claude Desktop' }, { value: 'codex', label: 'Codex' }, { value: 'cursor', label: 'Cursor' }, { value: 'opencode', label: 'OpenCode' }, @@ -589,26 +901,80 @@ export async function runKtxSetupAgentsStep( return { status: 'missing-input', projectDir: args.projectDir }; } - const installs = targets.map((target) => ({ target, scope: args.scope, mode })); + const scopeTargets = targets.filter((target) => target !== 'claude-desktop'); + const selectedScope = + args.inputMode !== 'disabled' && + args.scope === 'project' && + scopeTargets.length > 0 && + scopeTargets.every(targetSupportsGlobalScope) + ? ((await prompts.select({ + message: 'Where should KTX install supported agent config?', + options: [ + { value: 'project', label: 'Project' }, + { value: 'global', label: 'Global' }, + ], + })) as KtxAgentScope | 'back') + : args.scope; + if (selectedScope === 'back') return { status: 'back', projectDir: args.projectDir }; + + const installs = targets.map((target) => ({ target, scope: effectiveInstallScope(target, selectedScope), mode })); const entries: InstallEntry[] = []; const snippets: string[] = []; const notices = new Set(); + let claudeDesktopTutorial: string | undefined; try { for (const install of installs) { - entries.push(...(await installTarget({ projectDir: args.projectDir, ...install }))); - const mcpResult = await installMcpClientConfig({ projectDir: args.projectDir, target: install.target, scope: install.scope }); + const targetEntries = await installTarget({ projectDir: args.projectDir, ...install }); + entries.push(...targetEntries); + const mcpResult = await installMcpClientConfig({ + projectDir: args.projectDir, + target: install.target, + scope: install.scope, + }); entries.push(...mcpResult.entries); for (const snippet of mcpResult.snippets) snippets.push(snippet); for (const notice of mcpResult.notices) notices.add(notice); + if (install.target === 'claude-desktop') { + const pluginEntry = targetEntries.find( + (entry): entry is Extract => + entry.kind === 'file' && entry.role === 'claude-plugin', + ); + const pluginPath = pluginEntry?.path ?? ''; + const configPath = claudeDesktopConfigPath().path; + claudeDesktopTutorial = [ + `${green('✓')} ${bold('KTX MCP server registered')}`, + ` ${dim(configPath)}`, + '', + bold('1. Restart Claude Desktop'), + ' Quit and reopen so it picks up the new MCP server.', + '', + bold('2. Install the KTX plugin'), + ' Open Claude Desktop → Settings → Plugins and install from file:', + ` 📦 ${dim(pluginPath)}`, + ].join('\n'); + } } - await writeManifest(args.projectDir, mergeManifest(args.projectDir, await readKtxAgentInstallManifest(args.projectDir), installs, entries)); + await writeManifest( + args.projectDir, + mergeManifest(args.projectDir, await readKtxAgentInstallManifest(args.projectDir), installs, entries), + ); await markAgentsComplete(args.projectDir); - io.stdout.write(`\nAgent integration complete\n\n${formatInstallSummary(installs, entries, args.projectDir)}\n`); - for (const snippet of snippets) { - io.stdout.write(`\n${snippet}\n`); + const setupUi = createKtxSetupUiAdapter(); + setupUi.note( + formatInstallSummary(installs, entries, args.projectDir), + 'Agent integration complete', + io, + ); + if (claudeDesktopTutorial) { + setupUi.note(claudeDesktopTutorial, 'Finish Claude Desktop setup', io, { + format: (line) => line, + }); } - for (const notice of notices) { - io.stdout.write(`\n${notice}\n`); + const nextStepBlocks: string[] = []; + for (const notice of notices) nextStepBlocks.push(notice); + for (const snippet of snippets) nextStepBlocks.push(snippet); + if (nextStepBlocks.length > 0) { + setupUi.note(nextStepBlocks.join('\n\n'), 'Next steps', io, { format: bold }); } return { status: 'ready', projectDir: args.projectDir, installs }; } catch (error) { diff --git a/packages/cli/src/setup-demo-tour.test.ts b/packages/cli/src/setup-demo-tour.test.ts index f571f91b..1d57b010 100644 --- a/packages/cli/src/setup-demo-tour.test.ts +++ b/packages/cli/src/setup-demo-tour.test.ts @@ -184,7 +184,7 @@ describe('runDemoTour', () => { const mockAgents = vi.fn().mockResolvedValue({ status: 'ready', projectDir: '/tmp/test', - installs: [{ target: 'claude-code', scope: 'project', mode: 'cli' }], + installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp-cli' }], } satisfies KtxSetupAgentsResult); const navigation = vi.fn().mockResolvedValue('forward'); diff --git a/packages/cli/src/setup-demo-tour.ts b/packages/cli/src/setup-demo-tour.ts index 35640026..b2acb9f1 100644 --- a/packages/cli/src/setup-demo-tour.ts +++ b/packages/cli/src/setup-demo-tour.ts @@ -375,7 +375,7 @@ export async function runDemoTour( yes: false, agents: true, scope: 'project', - mode: 'cli', + mode: 'mcp-cli', skipAgents: false, }, io, diff --git a/packages/cli/src/setup-prompts.ts b/packages/cli/src/setup-prompts.ts index ad97ec48..f5faacd8 100644 --- a/packages/cli/src/setup-prompts.ts +++ b/packages/cli/src/setup-prompts.ts @@ -138,9 +138,13 @@ export function createKtxSetupPromptAdapter(options: KtxSetupPromptAdapterOption }; } +interface KtxSetupNoteOptions { + format?: (line: string) => string; +} + export interface KtxSetupUiAdapter { intro(title: string, io: KtxCliIo): void; - note(message: string, title: string, io: KtxCliIo): void; + note(message: string, title: string, io: KtxCliIo, options?: KtxSetupNoteOptions): void; } function isWritableTtyOutput(output: KtxCliIo['stdout']): output is KtxCliIo['stdout'] & Writable { @@ -160,9 +164,12 @@ export function createKtxSetupUiAdapter(): KtxSetupUiAdapter { } io.stdout.write(`${title}\n`); }, - note(message, title, io) { + note(message, title, io, options) { if (isWritableTtyOutput(io.stdout)) { - note(message, title, { output: io.stdout }); + note(message, title, { + output: io.stdout, + ...(options?.format ? { format: options.format } : {}), + }); return; } io.stdout.write(`\n${title}:\n`); diff --git a/packages/cli/src/setup.test.ts b/packages/cli/src/setup.test.ts index 47d1aa83..23b46cbd 100644 --- a/packages/cli/src/setup.test.ts +++ b/packages/cli/src/setup.test.ts @@ -232,7 +232,10 @@ describe('setup status', () => { version: 1, projectDir: tempDir, installedAt: '2026-05-07T00:00:00.000Z', - installs: [{ target: 'codex', scope: 'project', mode: 'cli' }], + installs: [ + { target: 'codex', scope: 'project', mode: 'mcp' }, + { target: 'codex', scope: 'project', mode: 'mcp-cli' }, + ], entries: [], }, null, @@ -1514,7 +1517,7 @@ describe('setup status', () => { return { status: 'ready', projectDir: tempDir, - installs: [{ target: 'codex', scope: 'project', mode: 'cli' }], + installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }], }; }, }, @@ -1566,7 +1569,7 @@ describe('setup status', () => { agents: async () => ({ status: 'ready', projectDir: tempDir, - installs: [{ target: 'codex', scope: 'project', mode: 'cli' }], + installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }], }), }, ), @@ -1617,7 +1620,7 @@ describe('setup status', () => { return { status: 'ready', projectDir: tempDir, - installs: [{ target: 'codex', scope: 'project', mode: 'cli' }], + installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }], }; }, }, @@ -1632,7 +1635,7 @@ describe('setup status', () => { const agents = vi.fn(async () => ({ status: 'ready' as const, projectDir: tempDir, - installs: [{ target: 'codex' as const, scope: 'project' as const, mode: 'cli' as const }], + installs: [{ target: 'codex' as const, scope: 'project' as const, mode: 'mcp-cli' as const }], })); await writeFile(join(tempDir, 'ktx.yaml'), ['connections: {}', ''].join('\n'), 'utf-8'); @@ -1701,7 +1704,7 @@ describe('setup status', () => { version: 1, projectDir: tempDir, installedAt: '2026-05-07T00:00:00.000Z', - installs: [{ target: 'codex', scope: 'project', mode: 'cli' }], + installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }], entries: [], }, null, @@ -1764,7 +1767,7 @@ describe('setup status', () => { return { status: 'ready', projectDir: tempDir, - installs: [{ target: 'codex', scope: 'project', mode: 'cli' }], + installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }], }; }, }, @@ -1856,7 +1859,7 @@ describe('setup status', () => { return { status: 'ready', projectDir: tempDir, - installs: [{ target: 'codex', scope: 'project', mode: 'cli' }], + installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }], }; }, }, @@ -1873,7 +1876,7 @@ describe('setup status', () => { const agents = vi.fn(async () => ({ status: 'ready' as const, projectDir: tempDir, - installs: [{ target: 'universal' as const, scope: 'project' as const, mode: 'cli' as const }], + installs: [{ target: 'universal' as const, scope: 'project' as const, mode: 'mcp-cli' as const }], })); await expect( diff --git a/packages/cli/src/setup.ts b/packages/cli/src/setup.ts index a76ad6a3..b295e912 100644 --- a/packages/cli/src/setup.ts +++ b/packages/cli/src/setup.ts @@ -306,12 +306,15 @@ export async function readKtxSetupStatus(projectDir: string): Promise ({ + const agentMap = new Map(); + for (const install of manifest?.installs ?? []) { + agentMap.set(`${install.target}:${install.scope}`, { target: install.target, scope: install.scope, ready: true, - })) ?? []; + }); + } + const agents = [...agentMap.values()]; return { project: { path: resolvedProjectDir, ready: true, name: basename(project.projectDir) || project.projectDir }, @@ -657,7 +660,7 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup agents: true, ...(args.target ? { target: args.target } : {}), scope: args.agentScope ?? 'project', - mode: 'cli', + mode: 'mcp', skipAgents: false, }, io, @@ -702,16 +705,21 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup await commitSetupConfigChanges(projectResult.projectDir); const status = await readKtxSetupStatus(projectResult.projectDir); - io.stdout.write(formatKtxSetupStatus(status)); - setupUi.note( - formatSetupNextStepLines({ - setupReady: setupStatusReady(status), - hasContextTargets: setupHasContextTargets(status), - contextReady: setupContextReady(status), - agentIntegrationReady: status.agents.some((agent) => agent.ready), - }).join('\n'), - 'What you can do next', - io, - ); + const focusedOnAgents = args.agents || entryAction === 'agents'; + if (!focusedOnAgents) { + setupUi.note(formatKtxSetupStatus(status).trimEnd(), 'Project status', io, { + format: (line) => line, + }); + setupUi.note( + formatSetupNextStepLines({ + setupReady: setupStatusReady(status), + hasContextTargets: setupHasContextTargets(status), + contextReady: setupContextReady(status), + agentIntegrationReady: status.agents.some((agent) => agent.ready), + }).join('\n'), + 'What you can do next', + io, + ); + } return 0; } diff --git a/packages/cli/src/skills/analytics/SKILL.md b/packages/cli/src/skills/analytics/SKILL.md new file mode 100644 index 00000000..e4aa86d2 --- /dev/null +++ b/packages/cli/src/skills/analytics/SKILL.md @@ -0,0 +1,62 @@ +--- +name: ktx-analytics +description: Use when answering a question that needs data from a KTX-connected database - investigating, analyzing, "how many", "show me", "what's the breakdown of", finding records by value, exploring tables, comparing periods, explaining metrics, or any data-analysis request. Triggers even when the user does not say "analytics"; if the answer requires querying a configured KTX connection, this skill applies. +--- + +# KTX Analytics Workflow + +You have access to KTX MCP tools for data discovery, semantic-layer analysis, raw read-only SQL, wiki context, and memory ingest. Follow this workflow. + + +1. **Discover** - call `discover_data` first to see what exists across wiki pages, semantic-layer sources, metrics, dimensions, raw tables, and columns. Returns refs only. +2. **Inspect top hits in parallel** - for each promising ref: + - `kind: 'wiki'` -> `wiki_read` + - `kind: 'sl_source'`, `kind: 'sl_measure'`, or `kind: 'sl_dimension'` -> `sl_read_source` + - `kind: 'table'` or `kind: 'column'` -> `entity_details` +3. **Resolve business values** - if the user named a value such as "Acme Corp", "enterprise", or "status=shipped", call `dictionary_search` to find which column holds it. +4. **Plan the analysis** - identify the grain, metrics, dimensions, filters, time window, and expected row limits before querying. +5. **Query** - + - Prefer `sl_query` when the semantic layer covers the question. + - Use `sql_execution` only for questions the semantic layer does not cover. +6. **Validate and explain** - sanity-check totals, filters, null handling, and time zones. State the source tables or semantic-layer objects used. +7. **Capture durable learnings** - call `memory_ingest` whenever a turn produces something worth remembering (business rules, metric definitions, schema gotchas, recurring findings) **or** whenever the user asks you to remember something. Pass markdown in `content` including any source context the memory agent should weigh. Each call is a feedback loop; better notes today mean smarter `discover_data` and `wiki_search` results tomorrow. + + + +- Always run `discover_data` before writing SQL. Do not guess table names. +- Prefer the semantic layer over raw SQL when both can answer the question; measures are the source of truth. +- Read entity details before writing SQL against an unfamiliar table. Do not assume column names. +- Treat `sql_execution` as read-only. Writes are rejected by the server. +- Validate value mentions with `dictionary_search` instead of guessing case or spelling. Treat a `dictionary_search` miss as non-authoritative. The index is built from profile-sampled values, so a missing value may simply have been outside the sample. Follow up with `sql_execution` against the most plausible columns before concluding the value is absent. +- When `connection_list` shows multiple connections, pass an explicit `connectionId` to every tool that takes one and where user intent pins a specific warehouse. Required: `entity_details`, `sl_read_source`, and `sql_execution`. Required when user intent is warehouse-specific, including wording like "in our warehouse" or "this warehouse": `memory_ingest`; without `connectionId`, the memory agent cannot update the semantic layer and the knowledge lands as wiki-only. Pass `connectionId` when intent pins a warehouse, otherwise omit for unscoped discovery: `sl_query`, `discover_data`, and `dictionary_search`. Never pass `connectionId` to `connection_list`, `wiki_search`, `wiki_read`, or `memory_ingest_status`. If intent is ambiguous for a required-or-scoped tool, ask the user which warehouse before calling. +- Show compact result tables for small outputs. For broad results, summarize the top findings and mention the applied limit. +- Ask a concise clarification only when the metric, date range, entity, or grain is genuinely ambiguous and cannot be inferred from context. + + + +**Input:** "How many orders did Acme Corp place last month?" + +**Workflow:** +1. `dictionary_search({ values: ["Acme Corp"] })` finds `customers.name`. +2. `discover_data({ query: "orders customer monthly" })` finds an orders semantic-layer source. +3. `sl_read_source({ connectionId: "warehouse", sourceName: "orders_facts" })` confirms the source grain, measures, and dimensions. +4. `sl_query({ connectionId: "warehouse", measures: ["order_count"], filters: ["customer_name = 'Acme Corp'"] })` answers through the semantic layer. +5. `memory_ingest({ connectionId: "warehouse", content: "Acme Corp order analysis used orders_facts.order_count filtered by customers.name = 'Acme Corp'. Source: current analysis turn." })` captures the durable finding. + +--- + +**Input:** "What columns does the events table have?" + +**Workflow:** +1. `discover_data({ query: "events table" })` returns a `table` ref. +2. `entity_details({ connectionId: "warehouse", entities: [{ table: "analytics.events" }] })` returns columns, types, and foreign keys. +3. Answer directly. No query is needed. + +--- + +**Input:** "Heads up: ARR is always reported in cents in our warehouse." + +**Workflow:** +1. If multiple connections exist, call `connection_list` and identify the warehouse the user means. Ask if ambiguous. +2. `memory_ingest({ connectionId: "warehouse", content: "ARR is reported in cents (not dollars) in this warehouse. Multiply by 0.01 for dollar amounts. Source: user clarification." })` remembers the warehouse-specific rule without running an analysis turn. + diff --git a/packages/cli/src/skills/research/SKILL.md b/packages/cli/src/skills/research/SKILL.md deleted file mode 100644 index e8e354a3..00000000 --- a/packages/cli/src/skills/research/SKILL.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -name: ktx-research -description: Use when answering a question that needs data from a KTX-connected database - investigating, analyzing, "how many", "show me", "what's the breakdown of", finding records by value, exploring tables, comparing periods, or any data-investigation request. Triggers even when the user does not say "research"; if the answer requires querying a configured KTX connection, this skill applies. ---- - -# KTX Research Workflow - -You have access to KTX MCP tools for investigating data. Follow this workflow. - - -1. **Discover** - call `discover_data` first to see what exists across wiki, semantic-layer sources, and raw tables. Returns refs only. -2. **Inspect top hits in parallel** - for each promising ref: - - `kind: 'wiki'` -> `wiki_read` - - `kind: 'sl_source'`, `kind: 'sl_measure'`, or `kind: 'sl_dimension'` -> `sl_read_source` - - `kind: 'table'` or `kind: 'column'` -> `entity_details` -3. **Resolve literals** - if the user named a value such as "Acme Corp" or "status=shipped", call `dictionary_search` to find which column holds it. -4. **Query** - - - Prefer `sl_query` when the semantic layer covers the question. - - Use `sql_execution` only for questions the semantic layer does not cover. -5. **Capture learnings** - at the end of the turn, call `memory_capture` so future turns benefit. Skip when the answer carries no durable knowledge. - - - -- Always run `discover_data` before writing SQL. Do not guess table names. -- Prefer the semantic layer over raw SQL when both can answer the question; measures are the source of truth. -- Read entity details before writing SQL against an unfamiliar table. Do not assume column names. -- Treat `sql_execution` as read-only. Writes are rejected by the server. -- Validate value mentions with `dictionary_search` instead of guessing case or spelling. Treat a `dictionary_search` miss as non-authoritative. The index is built from profile-sampled values, so a missing value may simply have been outside the sample. Follow up with `sql_execution` against the most plausible columns before concluding the value is absent. - - - -**Input:** "How many orders did Acme Corp place last month?" - -**Workflow:** -1. `dictionary_search({ values: ["Acme Corp"] })` finds `customers.name`. -2. `discover_data({ query: "orders customer monthly" })` finds an orders semantic-layer source. -3. `sl_read_source({ connectionId: "warehouse", sourceName: "orders_facts" })` confirms the source grain, measures, and dimensions. -4. `sl_query({ connectionId: "warehouse", measures: ["order_count"], filters: ["customer_name = 'Acme Corp'"] })` answers through the semantic layer. -5. `memory_capture({ userMessage, assistantMessage })` captures the durable finding. - ---- - -**Input:** "What columns does the events table have?" - -**Workflow:** -1. `discover_data({ query: "events table" })` returns a `table` ref. -2. `entity_details({ connectionId: "warehouse", entities: [{ table: "analytics.events" }] })` returns columns, types, and foreign keys. -3. Answer directly. No query is needed. - diff --git a/packages/cli/src/text-ingest.test.ts b/packages/cli/src/text-ingest.test.ts index 55dbe9e3..04ff2e40 100644 --- a/packages/cli/src/text-ingest.test.ts +++ b/packages/cli/src/text-ingest.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it, vi } from 'vitest'; -import type { MemoryCaptureStatus } from '@ktx/context/memory'; +import type { MemoryIngestStatus } from '@ktx/context/memory'; import type { KtxLocalProject } from '@ktx/context/project'; -import { runKtxTextIngest, type TextMemoryCapturePort } from './text-ingest.js'; +import { runKtxTextIngest, type TextMemoryIngestPort } from './text-ingest.js'; function makeIo(options: { isTTY?: boolean } = {}) { let stdout = ''; @@ -25,18 +25,18 @@ function makeIo(options: { isTTY?: boolean } = {}) { }; } -function fakeCapture( +function fakeIngest( options: { failRunIds?: Set; missingStatusRunIds?: Set; events?: string[]; } = {}, -): TextMemoryCapturePort { +): TextMemoryIngestPort { let next = 1; return { - capture: vi.fn(async () => { + ingest: vi.fn(async () => { const runId = `run-${next++}`; - options.events?.push(`capture:${runId}`); + options.events?.push(`ingest:${runId}`); return { runId }; }), waitForRun: vi.fn(async (runId: string) => { @@ -51,26 +51,26 @@ function fakeCapture( return { runId, status: 'error', - stage: 'capturing', + stage: 'ingesting', done: true, captured: { wiki: [], sl: [], xrefs: [] }, error: `${runId} failed`, commitHash: null, skillsLoaded: [], signalDetected: false, - } satisfies MemoryCaptureStatus; + } satisfies MemoryIngestStatus; } return { runId, status: 'done', - stage: 'capturing', + stage: 'ingesting', done: true, captured: { wiki: [`wiki-${runId}`], sl: [`sl-${runId}`], xrefs: [] }, error: null, commitHash: `commit-${runId}`, skillsLoaded: ['wiki_capture', 'sl'], signalDetected: true, - } satisfies MemoryCaptureStatus; + } satisfies MemoryIngestStatus; }), }; } @@ -80,11 +80,11 @@ function fakeProject(projectDir = '/tmp/project'): KtxLocalProject { } describe('runKtxTextIngest', () => { - it('captures repeated inline text sequentially with generated internal chat ids', async () => { + it('ingests repeated inline text sequentially with generated internal chat ids', async () => { const io = makeIo(); const events: string[] = []; - const capture = fakeCapture({ events }); - const createMemoryCapture = vi.fn(() => capture); + const ingest = fakeIngest({ events }); + const createMemoryIngest = vi.fn(() => ingest); await expect( runKtxTextIngest( @@ -99,14 +99,14 @@ describe('runKtxTextIngest', () => { io.io, { loadProject: vi.fn(async () => fakeProject()), - createMemoryCapture, + createMemoryIngest, now: () => 1_700_000_000_000, }, ), ).resolves.toBe(0); - expect(createMemoryCapture).toHaveBeenCalledWith({ projectDir: '/tmp/project' }); - expect(capture.capture).toHaveBeenNthCalledWith( + expect(createMemoryIngest).toHaveBeenCalledWith({ projectDir: '/tmp/project' }); + expect(ingest.ingest).toHaveBeenNthCalledWith( 1, expect.objectContaining({ userId: 'local-cli', @@ -116,7 +116,7 @@ describe('runKtxTextIngest', () => { sourceType: 'external_ingest', }), ); - expect(capture.capture).toHaveBeenNthCalledWith( + expect(ingest.ingest).toHaveBeenNthCalledWith( 2, expect.objectContaining({ chatId: 'cli-text-ingest-1700000000000-2', @@ -124,8 +124,8 @@ describe('runKtxTextIngest', () => { assistantMessage: 'Orders are completed purchases.', }), ); - expect(capture.capture).not.toHaveBeenCalledWith(expect.objectContaining({ connectionId: expect.anything() })); - expect(events).toEqual(['capture:run-1', 'wait:run-1', 'status:run-1', 'capture:run-2', 'wait:run-2', 'status:run-2']); + expect(ingest.ingest).not.toHaveBeenCalledWith(expect.objectContaining({ connectionId: expect.anything() })); + expect(events).toEqual(['ingest:run-1', 'wait:run-1', 'status:run-1', 'ingest:run-2', 'wait:run-2', 'status:run-2']); expect(JSON.parse(io.stdout())).toMatchObject({ status: 'done', results: [ @@ -147,7 +147,7 @@ describe('runKtxTextIngest', () => { it('loads files and stdin as batch items and passes a global connection id', async () => { const io = makeIo(); - const capture = fakeCapture(); + const ingest = fakeIngest(); await expect( runKtxTextIngest( @@ -163,7 +163,7 @@ describe('runKtxTextIngest', () => { io.io, { loadProject: vi.fn(async () => fakeProject()), - createMemoryCapture: vi.fn(() => capture), + createMemoryIngest: vi.fn(() => ingest), readFile: vi.fn(async (path) => `file:${path}`), readStdin: vi.fn(async () => 'stdin content'), now: () => 10, @@ -171,7 +171,7 @@ describe('runKtxTextIngest', () => { ), ).resolves.toBe(0); - expect(capture.capture).toHaveBeenNthCalledWith( + expect(ingest.ingest).toHaveBeenNthCalledWith( 1, expect.objectContaining({ connectionId: 'warehouse', @@ -180,7 +180,7 @@ describe('runKtxTextIngest', () => { assistantMessage: 'file:/tmp/docs/revenue.md', }), ); - expect(capture.capture).toHaveBeenNthCalledWith( + expect(ingest.ingest).toHaveBeenNthCalledWith( 2, expect.objectContaining({ connectionId: 'warehouse', @@ -194,9 +194,9 @@ describe('runKtxTextIngest', () => { expect(io.stdout()).toContain('stdin'); }); - it('uses bounded inline text previews as labels in plain output and capture metadata', async () => { + it('uses bounded inline text previews as labels in plain output and ingest metadata', async () => { const io = makeIo(); - const capture = fakeCapture(); + const ingest = fakeIngest(); const longText = `This inline note is intentionally long ${'x'.repeat(120)}`; await expect( @@ -212,7 +212,7 @@ describe('runKtxTextIngest', () => { io.io, { loadProject: vi.fn(async () => fakeProject()), - createMemoryCapture: vi.fn(() => capture), + createMemoryIngest: vi.fn(() => ingest), now: () => 10, }, ), @@ -225,19 +225,19 @@ describe('runKtxTextIngest', () => { expect(output).not.toContain('text-1'); expect(output).not.toContain(longText); - expect(capture.capture).toHaveBeenNthCalledWith( + expect(ingest.ingest).toHaveBeenNthCalledWith( 1, expect.objectContaining({ userMessage: 'Ingest external text artifact "remember to call me Andrey" into KTX memory.', }), ); - expect(capture.capture).toHaveBeenNthCalledWith( + expect(ingest.ingest).toHaveBeenNthCalledWith( 2, expect.objectContaining({ userMessage: 'Ingest external text artifact "first line second line" into KTX memory.', }), ); - expect(capture.capture).toHaveBeenNthCalledWith( + expect(ingest.ingest).toHaveBeenNthCalledWith( 3, expect.objectContaining({ userMessage: 'Ingest external text artifact "This inline note is intentionally long xxxxxxxx..." into KTX memory.', @@ -247,7 +247,7 @@ describe('runKtxTextIngest', () => { it('continues after an item failure by default and stops when failFast is set', async () => { const continueIo = makeIo(); - const continueCapture = fakeCapture({ failRunIds: new Set(['run-1']) }); + const continueIngest = fakeIngest({ failRunIds: new Set(['run-1']) }); await expect( runKtxTextIngest( @@ -262,12 +262,12 @@ describe('runKtxTextIngest', () => { continueIo.io, { loadProject: vi.fn(async () => fakeProject()), - createMemoryCapture: vi.fn(() => continueCapture), + createMemoryIngest: vi.fn(() => continueIngest), }, ), ).resolves.toBe(1); - expect(continueCapture.capture).toHaveBeenCalledTimes(2); + expect(continueIngest.ingest).toHaveBeenCalledTimes(2); expect(JSON.parse(continueIo.stdout())).toMatchObject({ status: 'failed', results: [ @@ -277,7 +277,7 @@ describe('runKtxTextIngest', () => { }); const failFastIo = makeIo(); - const failFastCapture = fakeCapture({ failRunIds: new Set(['run-1']) }); + const failFastIngest = fakeIngest({ failRunIds: new Set(['run-1']) }); await expect( runKtxTextIngest( @@ -292,12 +292,12 @@ describe('runKtxTextIngest', () => { failFastIo.io, { loadProject: vi.fn(async () => fakeProject()), - createMemoryCapture: vi.fn(() => failFastCapture), + createMemoryIngest: vi.fn(() => failFastIngest), }, ), ).resolves.toBe(1); - expect(failFastCapture.capture).toHaveBeenCalledTimes(1); + expect(failFastIngest.ingest).toHaveBeenCalledTimes(1); expect(JSON.parse(failFastIo.stdout()).results).toHaveLength(1); }); @@ -314,7 +314,7 @@ describe('runKtxTextIngest', () => { failFast: false, }, noInputIo.io, - { loadProject: vi.fn(), createMemoryCapture: vi.fn() }, + { loadProject: vi.fn(), createMemoryIngest: vi.fn() }, ), ).resolves.toBe(1); expect(noInputIo.stderr()).toContain('Provide at least one text item'); @@ -331,7 +331,7 @@ describe('runKtxTextIngest', () => { failFast: false, }, emptyIo.io, - { loadProject: vi.fn(), createMemoryCapture: vi.fn() }, + { loadProject: vi.fn(), createMemoryIngest: vi.fn() }, ), ).resolves.toBe(1); expect(emptyIo.stderr()).toContain('Text item "text-1" is empty'); diff --git a/packages/cli/src/text-ingest.ts b/packages/cli/src/text-ingest.ts index d48ee24b..fe15244e 100644 --- a/packages/cli/src/text-ingest.ts +++ b/packages/cli/src/text-ingest.ts @@ -1,6 +1,6 @@ import { readFile as fsReadFile } from 'node:fs/promises'; import { basename, resolve } from 'node:path'; -import { createLocalProjectMemoryCapture, type MemoryAgentInput, type MemoryCaptureStatus } from '@ktx/context/memory'; +import { createLocalProjectMemoryIngest, type MemoryAgentInput, type MemoryIngestStatus } from '@ktx/context/memory'; import { loadKtxProject, type KtxLocalProject } from '@ktx/context/project'; import type { KtxCliIo } from './cli-runtime.js'; import { createRepainter, initViewState, renderContextBuildView, type ContextBuildTargetState } from './context-build-view.js'; @@ -17,10 +17,10 @@ export interface KtxTextIngestArgs { failFast: boolean; } -export interface TextMemoryCapturePort { - capture(input: MemoryAgentInput): Promise<{ runId: string }>; +export interface TextMemoryIngestPort { + ingest(input: MemoryAgentInput): Promise<{ runId: string }>; waitForRun(runId: string): Promise; - status(runId: string): Promise; + status(runId: string): Promise; } interface TextIngestItem { @@ -32,14 +32,14 @@ interface TextIngestResult { label: string; runId: string | null; status: 'done' | 'error'; - captured: MemoryCaptureStatus['captured']; + captured: MemoryIngestStatus['captured']; commitHash: string | null; error: string | null; } export interface KtxTextIngestDeps { loadProject?: (options: { projectDir: string }) => Promise; - createMemoryCapture?: (project: KtxLocalProject) => TextMemoryCapturePort; + createMemoryIngest?: (project: KtxLocalProject) => TextMemoryIngestPort; readFile?: (path: string) => Promise; readStdin?: () => Promise; now?: () => number; @@ -48,8 +48,8 @@ export interface KtxTextIngestDeps { const INLINE_TEXT_LABEL_MAX_LENGTH = 50; const ANSI_ESCAPE_PATTERN = /\x1B\[[0-?]*[ -/]*[@-~]/g; -function defaultCreateMemoryCapture(project: KtxLocalProject): TextMemoryCapturePort { - return createLocalProjectMemoryCapture(project); +function defaultCreateMemoryIngest(project: KtxLocalProject): TextMemoryIngestPort { + return createLocalProjectMemoryIngest(project); } async function defaultReadStdin(): Promise { @@ -65,7 +65,7 @@ async function defaultReadFile(path: string): Promise { return await fsReadFile(path, 'utf-8'); } -function emptyCaptured(): MemoryCaptureStatus['captured'] { +function emptyCaptured(): MemoryIngestStatus['captured'] { return { wiki: [], sl: [], xrefs: [] }; } @@ -182,7 +182,7 @@ function renderTextIngestView(state: ReturnType, styled: b }); } -function summarizeCaptured(captured: MemoryCaptureStatus['captured']): string { +function summarizeCaptured(captured: MemoryIngestStatus['captured']): string { const parts = [ `wiki=${captured.wiki.length}`, `sl=${captured.sl.length}`, @@ -191,7 +191,7 @@ function summarizeCaptured(captured: MemoryCaptureStatus['captured']): string { return parts.join(', '); } -function resultFromStatus(label: string, status: MemoryCaptureStatus): TextIngestResult { +function resultFromStatus(label: string, status: MemoryIngestStatus): TextIngestResult { return { label, runId: status.runId, @@ -251,7 +251,7 @@ export async function runKtxTextIngest( } const project = await (deps.loadProject ?? loadKtxProject)({ projectDir: args.projectDir }); - const memoryCapture = (deps.createMemoryCapture ?? defaultCreateMemoryCapture)(project); + const memoryIngest = (deps.createMemoryIngest ?? defaultCreateMemoryIngest)(project); const now = deps.now ?? (() => Date.now()); const batchId = now(); const state = initViewState(items.map((item) => makeTarget(item.label))); @@ -292,7 +292,7 @@ export async function runKtxTextIngest( let runId: string | null = null; let result: TextIngestResult; try { - const captureInput: MemoryAgentInput = { + const ingestInput: MemoryAgentInput = { userId: args.userId, chatId: `cli-text-ingest-${batchId}-${index + 1}`, userMessage: `Ingest external text artifact ${artifactReference(item.label)} into KTX memory.`, @@ -300,12 +300,12 @@ export async function runKtxTextIngest( ...(args.connectionId ? { connectionId: args.connectionId } : {}), sourceType: 'external_ingest', }; - const capture = await memoryCapture.capture(captureInput); - runId = capture.runId; - await memoryCapture.waitForRun(runId); - const status = await memoryCapture.status(runId); + const ingest = await memoryIngest.ingest(ingestInput); + runId = ingest.runId; + await memoryIngest.waitForRun(runId); + const status = await memoryIngest.status(runId); if (!status) { - throw new Error(`Memory capture run "${runId}" was not found.`); + throw new Error(`Memory ingest run "${runId}" was not found.`); } result = resultFromStatus(item.label, status); } catch (error) { diff --git a/packages/context/src/daemon/semantic-layer-compute.ts b/packages/context/src/daemon/semantic-layer-compute.ts index d6e2f0b1..a48239eb 100644 --- a/packages/context/src/daemon/semantic-layer-compute.ts +++ b/packages/context/src/daemon/semantic-layer-compute.ts @@ -2,7 +2,7 @@ import { request as httpRequest } from 'node:http'; import { request as httpsRequest } from 'node:https'; import { URL } from 'node:url'; import { spawn } from 'node:child_process'; -import type { SemanticLayerQueryInput, SemanticLayerSource } from '../sl/index.js'; +import type { ResolvedSemanticLayerSource, SemanticLayerQueryInput } from '../sl/types.js'; export interface KtxSemanticLayerComputeQueryResult { sql: string; @@ -54,13 +54,21 @@ export interface KtxSemanticLayerSourceGenerationResult { } export interface KtxSemanticLayerComputePort { + /** + * Callers must pass sources sanitized through toResolvedWire. The Python + * daemon rejects authoring-only fields such as usage and inherits_columns_from. + */ query(input: { - sources: Array | SemanticLayerSource>; + sources: ResolvedSemanticLayerSource[]; query: SemanticLayerQueryInput; dialect: string; }): Promise; + /** + * Callers must pass sources sanitized through toResolvedWire. The Python + * daemon rejects authoring-only fields such as usage and inherits_columns_from. + */ validateSources(input: { - sources: Array | SemanticLayerSource>; + sources: ResolvedSemanticLayerSource[]; dialect: string; recentlyTouched?: string[]; }): Promise; diff --git a/packages/context/src/llm/local-config.test.ts b/packages/context/src/llm/local-config.test.ts index 59ad34b7..539afe45 100644 --- a/packages/context/src/llm/local-config.test.ts +++ b/packages/context/src/llm/local-config.test.ts @@ -191,6 +191,36 @@ describe('local KTX embedding config', () => { expect(resolveLocalKtxEmbeddingConfig(config, {})).toBeNull(); }); + it('returns null when backend is openai but no apiKey is resolvable from env', () => { + const config: KtxProjectEmbeddingConfig = { + backend: 'openai', + model: 'text-embedding-3-small', + dimensions: 1536, + openai: { api_key: 'env:OPENAI_API_KEY' }, // pragma: allowlist secret + }; + + expect(resolveLocalKtxEmbeddingConfig(config, {})).toBeNull(); + }); + + it('resolves openai embedding config from env', () => { + const config: KtxProjectEmbeddingConfig = { + backend: 'openai', + model: 'text-embedding-3-small', + dimensions: 1536, + openai: { api_key: 'env:OPENAI_API_KEY' }, // pragma: allowlist secret + }; + + expect( + resolveLocalKtxEmbeddingConfig(config, { OPENAI_API_KEY: 'sk-test' }), // pragma: allowlist secret + ).toEqual({ + backend: 'openai', + model: 'text-embedding-3-small', + dimensions: 1536, + openai: { apiKey: 'sk-test' }, // pragma: allowlist secret + batchSize: undefined, + }); + }); + it('constructs deterministic embeddings from the default project config', () => { const createKtxEmbeddingProvider = vi.fn(() => ({}) as never); const provider = createLocalKtxEmbeddingProviderFromConfig( diff --git a/packages/context/src/llm/local-config.ts b/packages/context/src/llm/local-config.ts index 2709c4b7..e2ee45e0 100644 --- a/packages/context/src/llm/local-config.ts +++ b/packages/context/src/llm/local-config.ts @@ -145,11 +145,23 @@ export function resolveLocalKtxEmbeddingConfig( batchSize: config.batchSize, }; } + if (config.backend === 'openai') { + const openai = resolvedProviderConfig(config.openai, env); + if (!openai?.apiKey) { + return null; + } + return { + backend: config.backend, + model: config.model ?? 'deterministic', + dimensions: config.dimensions, + openai, + batchSize: config.batchSize, + }; + } return { backend: config.backend, model: config.model ?? 'deterministic', dimensions: config.dimensions, - ...(resolvedProviderConfig(config.openai, env) ? { openai: resolvedProviderConfig(config.openai, env) } : {}), ...(config.sentenceTransformers ? { sentenceTransformers: { diff --git a/packages/context/src/mcp/__snapshots__/mcp-tools-list.json b/packages/context/src/mcp/__snapshots__/mcp-tools-list.json new file mode 100644 index 00000000..db84b328 --- /dev/null +++ b/packages/context/src/mcp/__snapshots__/mcp-tools-list.json @@ -0,0 +1,1615 @@ +[ + { + "name": "connection_list", + "title": "Connection List", + "description": "List configured read-only data connections available to this KTX project. Use this before connection-scoped tools when the project may have multiple warehouses.", + "inputSchema": { + "type": "object", + "properties": {}, + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "connections": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "name": { + "type": "string" + }, + "connectionType": { + "type": "string" + } + }, + "required": [ + "id", + "name", + "connectionType" + ], + "additionalProperties": false + } + } + }, + "required": [ + "connections" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Connection List", + "readOnlyHint": true, + "idempotentHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "wiki_search", + "title": "Wiki Search", + "description": "Search KTX wiki pages for reusable business context. Example: wiki_search({ query: \"revenue recognition\", limit: 5 }).", + "inputSchema": { + "type": "object", + "properties": { + "query": { + "type": "string", + "minLength": 1, + "description": "Natural-language wiki search query, e.g. \"revenue recognition policy\"." + }, + "limit": { + "default": 10, + "description": "Maximum wiki pages to return. Defaults to 10.", + "type": "integer", + "minimum": 1, + "maximum": 50 + } + }, + "required": [ + "query" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "results": { + "type": "array", + "items": { + "type": "object", + "properties": { + "key": { + "type": "string" + }, + "path": { + "type": "string" + }, + "scope": { + "type": "string", + "enum": [ + "GLOBAL", + "USER" + ] + }, + "summary": { + "type": "string" + }, + "score": { + "type": "number" + }, + "matchReasons": { + "type": "array", + "items": { + "type": "string" + } + }, + "lanes": { + "type": "array", + "items": { + "type": "object", + "properties": { + "lane": { + "type": "string" + }, + "status": { + "type": "string" + }, + "requestedCandidatePoolLimit": { + "type": "number" + }, + "effectiveCandidatePoolLimit": { + "type": "number" + }, + "returnedCandidateCount": { + "type": "number" + }, + "weight": { + "type": "number" + }, + "reason": { + "type": "string" + } + }, + "required": [ + "lane", + "status", + "requestedCandidatePoolLimit", + "effectiveCandidatePoolLimit", + "returnedCandidateCount", + "weight" + ], + "additionalProperties": false + } + } + }, + "required": [ + "key", + "path", + "scope", + "summary", + "score" + ], + "additionalProperties": false + } + }, + "totalFound": { + "type": "number" + } + }, + "required": [ + "results", + "totalFound" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Wiki Search", + "readOnlyHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "wiki_read", + "title": "Wiki Read", + "description": "Read a KTX wiki page by key returned from wiki_search. Example: wiki_read({ key: \"global/revenue\" }).", + "inputSchema": { + "type": "object", + "properties": { + "key": { + "type": "string", + "minLength": 1, + "description": "Wiki page key returned by wiki_search, e.g. \"global/revenue\"." + } + }, + "required": [ + "key" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "key": { + "type": "string" + }, + "summary": { + "type": "string" + }, + "content": { + "type": "string" + }, + "scope": { + "type": "string", + "enum": [ + "GLOBAL", + "USER" + ] + }, + "tags": { + "type": "array", + "items": { + "type": "string" + } + }, + "refs": { + "type": "array", + "items": { + "type": "string" + } + }, + "slRefs": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "key", + "summary", + "content", + "scope" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Wiki Read", + "readOnlyHint": true, + "idempotentHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "sl_read_source", + "title": "Semantic Layer Read Source", + "description": "Read a semantic-layer YAML source by connection id and source name. Example: sl_read_source({ connectionId: \"warehouse\", sourceName: \"orders\" }).", + "inputSchema": { + "type": "object", + "properties": { + "connectionId": { + "type": "string", + "minLength": 1, + "description": "Connection id that owns the semantic-layer source." + }, + "sourceName": { + "type": "string", + "minLength": 1, + "description": "Semantic-layer source name without \".yaml\", e.g. \"orders\"." + } + }, + "required": [ + "connectionId", + "sourceName" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "sourceName": { + "type": "string" + }, + "yaml": { + "type": "string" + } + }, + "required": [ + "sourceName", + "yaml" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Semantic Layer Read Source", + "readOnlyHint": true, + "idempotentHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "sl_query", + "title": "Semantic Layer Query", + "description": "Execute a semantic-layer query and return rows, headers, generated SQL, and plan details. Example: sl_query({ connectionId: \"warehouse\", measures: [\"orders.order_count\"], dimensions: [{ dimension: \"orders.created_at\", granularity: \"month\" }] }).", + "inputSchema": { + "type": "object", + "properties": { + "connectionId": { + "description": "Connection id to query. Omit only when the project has exactly one configured connection.", + "type": "string", + "minLength": 1 + }, + "measures": { + "minItems": 1, + "type": "array", + "items": { + "anyOf": [ + { + "type": "string", + "description": "Semantic-layer measure key, e.g. \"orders.order_count\"." + }, + { + "type": "object", + "properties": { + "expr": { + "type": "string", + "minLength": 1, + "description": "Ad hoc aggregate expression, e.g. \"sum(orders.amount)\"." + }, + "name": { + "type": "string", + "minLength": 1, + "description": "Alias for the ad hoc measure, e.g. \"gross_revenue\"." + } + }, + "required": [ + "expr", + "name" + ] + } + ] + }, + "description": "Measures to select. Use semantic-layer keys when available." + }, + "dimensions": { + "description": "Dimensions to group by. Strings and {dimension, granularity} are accepted.", + "type": "array", + "items": { + "type": "object", + "properties": { + "field": { + "type": "string", + "minLength": 1, + "description": "Dimension to group by, e.g. \"orders.created_at\" or \"orders.status\"." + }, + "granularity": { + "description": "Time grain for time dimensions: day, week, month, quarter, or year.", + "type": "string", + "minLength": 1 + } + }, + "required": [ + "field" + ] + } + }, + "filters": { + "default": [], + "description": "Semantic-layer filter expressions to apply.", + "type": "array", + "items": { + "type": "string", + "description": "Semantic-layer filter expression, e.g. \"orders.status = paid\"." + } + }, + "segments": { + "default": [], + "description": "Semantic-layer segment keys to apply.", + "type": "array", + "items": { + "type": "string", + "description": "Semantic-layer segment key to apply." + } + }, + "order_by": { + "description": "Sort clauses. Strings and Cube-style {id, desc} are accepted.", + "type": "array", + "items": { + "type": "object", + "properties": { + "field": { + "type": "string", + "minLength": 1, + "description": "Field/measure/dimension id to order by, e.g. \"orders.created_at\", a dimension key like \"mart_nrr_quarterly.quarter_label\", or a measure alias." + }, + "direction": { + "default": "asc", + "description": "Sort direction: \"asc\" or \"desc\". Defaults to \"asc\".", + "type": "string", + "enum": [ + "asc", + "desc" + ] + } + }, + "required": [ + "field" + ] + } + }, + "limit": { + "default": 1000, + "description": "Maximum rows to return. Defaults to 1000.", + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "include_empty": { + "default": true, + "description": "Whether to include empty dimension groups. Defaults to true.", + "type": "boolean" + } + }, + "required": [ + "measures" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "connectionId": { + "type": "string" + }, + "dialect": { + "type": "string" + }, + "sql": { + "type": "string" + }, + "headers": { + "type": "array", + "items": { + "type": "string" + } + }, + "rows": { + "type": "array", + "items": { + "type": "array", + "items": {} + } + }, + "totalRows": { + "type": "number" + }, + "plan": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": {} + } + }, + "required": [ + "sql", + "headers", + "rows", + "totalRows" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Semantic Layer Query", + "readOnlyHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "entity_details", + "title": "Entity Details", + "description": "Read table and column metadata from the latest live-database scan snapshot. Example: entity_details({ connectionId: \"warehouse\", entities: [{ table: { schema: \"public\", table: \"orders\" }, columns: [\"id\"] }] }).", + "inputSchema": { + "type": "object", + "properties": { + "connectionId": { + "type": "string", + "minLength": 1, + "description": "Connection id whose latest scan snapshot should be read." + }, + "entities": { + "minItems": 1, + "maxItems": 20, + "type": "array", + "items": { + "type": "object", + "properties": { + "table": { + "anyOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "object", + "properties": { + "catalog": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Catalog/project/database. Use null when not applicable." + }, + "db": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Schema/database/dataset. Use null when not applicable." + }, + "name": { + "type": "string", + "minLength": 1, + "description": "Table name." + } + }, + "required": [ + "catalog", + "db", + "name" + ] + } + ], + "description": "Table display string or object ref. {schema, table} is accepted as an alias for {db, name}." + }, + "columns": { + "description": "Optional column filter.", + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "description": "Column name to inspect." + } + } + } + }, + "description": "Tables or columns to inspect. Maximum 20 entities." + } + }, + "required": [ + "connectionId", + "entities" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "results": { + "type": "array", + "items": { + "anyOf": [ + { + "type": "object", + "properties": { + "ok": { + "type": "boolean", + "const": true + }, + "connectionId": { + "type": "string" + }, + "tableRef": { + "type": "object", + "properties": { + "catalog": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "db": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "name": { + "type": "string" + } + }, + "required": [ + "catalog", + "db", + "name" + ], + "additionalProperties": false + }, + "display": { + "type": "string" + }, + "kind": { + "type": "string", + "enum": [ + "table", + "view", + "external", + "event_stream" + ] + }, + "comment": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "estimatedRows": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "columns": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "nativeType": { + "type": "string" + }, + "normalizedType": { + "type": "string" + }, + "dimensionType": { + "type": "string", + "enum": [ + "time", + "string", + "number", + "boolean" + ] + }, + "nullable": { + "type": "boolean" + }, + "primaryKey": { + "type": "boolean" + }, + "comment": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "name", + "nativeType", + "normalizedType", + "dimensionType", + "nullable", + "primaryKey", + "comment" + ], + "additionalProperties": false + } + }, + "foreignKeys": { + "type": "array", + "items": { + "type": "object", + "properties": { + "fromColumn": { + "type": "string" + }, + "toCatalog": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "toDb": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "toTable": { + "type": "string" + }, + "toColumn": { + "type": "string" + }, + "constraintName": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "fromColumn", + "toCatalog", + "toDb", + "toTable", + "toColumn", + "constraintName" + ], + "additionalProperties": false + } + }, + "snapshot": { + "type": "object", + "properties": { + "syncId": { + "type": "string" + }, + "extractedAt": { + "type": "string" + }, + "scanRunId": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "syncId", + "extractedAt", + "scanRunId" + ], + "additionalProperties": false + } + }, + "required": [ + "ok", + "connectionId", + "tableRef", + "display", + "kind", + "comment", + "estimatedRows", + "columns", + "foreignKeys", + "snapshot" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "ok": { + "type": "boolean", + "const": false + }, + "connectionId": { + "type": "string" + }, + "table": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "object", + "properties": { + "catalog": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "db": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "name": { + "type": "string" + } + }, + "required": [ + "catalog", + "db", + "name" + ], + "additionalProperties": false + } + ] + }, + "snapshot": { + "type": "object", + "properties": { + "syncId": { + "type": "string" + }, + "extractedAt": { + "type": "string" + }, + "scanRunId": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "syncId", + "extractedAt", + "scanRunId" + ], + "additionalProperties": false + }, + "error": { + "type": "object", + "properties": { + "code": { + "type": "string", + "enum": [ + "scan_missing", + "table_not_found", + "ambiguous_table", + "column_not_found" + ] + }, + "message": { + "type": "string" + }, + "candidates": { + "anyOf": [ + { + "type": "array", + "items": { + "type": "object", + "properties": { + "tableRef": { + "type": "object", + "properties": { + "catalog": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "db": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "name": { + "type": "string" + } + }, + "required": [ + "catalog", + "db", + "name" + ], + "additionalProperties": false + }, + "display": { + "type": "string" + } + }, + "required": [ + "tableRef", + "display" + ], + "additionalProperties": false + } + }, + { + "type": "array", + "items": { + "type": "string" + } + } + ] + } + }, + "required": [ + "code", + "message" + ], + "additionalProperties": false + } + }, + "required": [ + "ok", + "connectionId", + "table", + "error" + ], + "additionalProperties": false + } + ] + } + } + }, + "required": [ + "results" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Entity Details", + "readOnlyHint": true, + "idempotentHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "dictionary_search", + "title": "Dictionary Search", + "description": "Search profile-sampled warehouse values to locate likely source columns for business values. Example: dictionary_search({ values: [\"Acme Corp\"], connectionId: \"warehouse\" }).", + "inputSchema": { + "type": "object", + "properties": { + "values": { + "minItems": 1, + "maxItems": 20, + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "description": "Business value to locate, e.g. \"Acme Corp\" or \"enterprise\"." + }, + "description": "Values to search for in sampled warehouse dictionaries." + }, + "connectionId": { + "description": "Optional connection id. Pass it when user intent pins a specific warehouse.", + "type": "string", + "minLength": 1 + } + }, + "required": [ + "values" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "searched": { + "type": "array", + "items": { + "type": "object", + "properties": { + "connectionId": { + "type": "string" + }, + "coverage": { + "type": "object", + "properties": { + "sampledRows": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "valuesPerColumn": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + }, + "profiledColumns": { + "type": "number" + }, + "syncId": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "profiledAt": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "sampledRows", + "valuesPerColumn", + "profiledColumns", + "syncId", + "profiledAt" + ], + "additionalProperties": false + }, + "status": { + "type": "string", + "enum": [ + "ready", + "no_profile_artifact", + "no_candidate_columns" + ] + } + }, + "required": [ + "connectionId", + "coverage", + "status" + ], + "additionalProperties": false + } + }, + "results": { + "type": "array", + "items": { + "type": "object", + "properties": { + "value": { + "type": "string" + }, + "matches": { + "type": "array", + "items": { + "type": "object", + "properties": { + "connectionId": { + "type": "string" + }, + "sourceName": { + "type": "string" + }, + "columnName": { + "type": "string" + }, + "matchedValue": { + "type": "string" + }, + "cardinality": { + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "connectionId", + "sourceName", + "columnName", + "matchedValue", + "cardinality" + ], + "additionalProperties": false + } + }, + "misses": { + "type": "array", + "items": { + "type": "object", + "properties": { + "connectionId": { + "type": "string" + }, + "reason": { + "type": "string", + "enum": [ + "no_profile_artifact", + "no_candidate_columns", + "value_not_in_sample" + ] + } + }, + "required": [ + "connectionId", + "reason" + ], + "additionalProperties": false + } + } + }, + "required": [ + "value", + "matches", + "misses" + ], + "additionalProperties": false + } + } + }, + "required": [ + "searched", + "results" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Dictionary Search", + "readOnlyHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "discover_data", + "title": "Discover Data", + "description": "Search across KTX wiki pages, semantic-layer sources, measures, dimensions, raw tables, and columns. Example: discover_data({ query: \"monthly orders by customer\", connectionId: \"warehouse\", kinds: [\"sl_source\", \"table\"] }).", + "inputSchema": { + "type": "object", + "properties": { + "query": { + "type": "string", + "minLength": 1, + "description": "Natural-language discovery query, e.g. \"monthly orders by customer\"." + }, + "connectionId": { + "description": "Optional connection id. Pass it when user intent pins a specific warehouse.", + "type": "string", + "minLength": 1 + }, + "kinds": { + "description": "Optional kind filter.", + "type": "array", + "items": { + "type": "string", + "enum": [ + "wiki", + "sl_source", + "sl_measure", + "sl_dimension", + "table", + "column" + ], + "description": "Reference kind to include." + } + }, + "limit": { + "description": "Maximum refs to return. Defaults to 15.", + "default": 15, + "type": "integer", + "minimum": 1, + "maximum": 50 + } + }, + "required": [ + "query" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "refs": { + "type": "array", + "items": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "enum": [ + "wiki", + "sl_source", + "sl_measure", + "sl_dimension", + "table", + "column" + ] + }, + "id": { + "type": "string" + }, + "score": { + "type": "number" + }, + "summary": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "snippet": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "matchedOn": { + "type": "string", + "enum": [ + "name", + "display", + "description", + "comment", + "expr", + "sample_value", + "body" + ] + }, + "connectionId": { + "type": "string" + }, + "tableRef": { + "type": "object", + "properties": { + "catalog": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "db": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "name": { + "type": "string" + } + }, + "required": [ + "catalog", + "db", + "name" + ], + "additionalProperties": false + }, + "columnName": { + "type": "string" + } + }, + "required": [ + "kind", + "id", + "score", + "summary", + "snippet", + "matchedOn" + ], + "additionalProperties": false + } + } + }, + "required": [ + "refs" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Discover Data", + "readOnlyHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "sql_execution", + "title": "SQL Execution", + "description": "Execute one parser-validated read-only SQL query against a configured KTX connection. Example: sql_execution({ connectionId: \"warehouse\", sql: \"select count(*) from public.orders\", maxRows: 100 }).", + "inputSchema": { + "type": "object", + "properties": { + "connectionId": { + "type": "string", + "minLength": 1, + "description": "Connection id to execute against. Required for raw SQL." + }, + "sql": { + "type": "string", + "minLength": 1, + "description": "Parser-validated read-only SQL, e.g. \"select count(*) from public.orders\"." + }, + "maxRows": { + "description": "Maximum rows to return. Defaults to 1000.", + "default": 1000, + "type": "integer", + "minimum": 1, + "maximum": 10000 + } + }, + "required": [ + "connectionId", + "sql" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "headers": { + "type": "array", + "items": { + "type": "string" + } + }, + "headerTypes": { + "type": "array", + "items": { + "type": "string" + } + }, + "rows": { + "type": "array", + "items": { + "type": "array", + "items": {} + } + }, + "rowCount": { + "type": "number" + } + }, + "required": [ + "headers", + "rows", + "rowCount" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "SQL Execution", + "readOnlyHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "memory_ingest", + "title": "Memory Ingest", + "description": "Ingest free-form markdown knowledge into durable KTX memory. Use this for business rules, metric definitions, schema gotchas, recurring findings, or explicit user requests to remember something. Example: memory_ingest({ connectionId: \"warehouse\", content: \"ARR is reported in cents in this warehouse.\" }).", + "inputSchema": { + "type": "object", + "properties": { + "content": { + "type": "string", + "minLength": 1, + "description": "Free-form markdown to ingest. Include the knowledge itself plus any context (source, the user question, why this came up) that the memory agent should consider when triaging into wiki/SL." + }, + "connectionId": { + "description": "Scope this memory to a specific connection. Required when the knowledge is warehouse-specific, including measure definitions, schema gotchas, or anything tied to a particular warehouse. Omit only for global wiki knowledge.", + "type": "string", + "minLength": 1 + } + }, + "required": [ + "content" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "runId": { + "type": "string" + } + }, + "required": [ + "runId" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Memory Ingest", + "destructiveHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + }, + { + "name": "memory_ingest_status", + "title": "Memory Ingest Status", + "description": "Read the current or final status for a memory ingest run. Example: memory_ingest_status({ runId: \"memory-run-1\" }).", + "inputSchema": { + "type": "object", + "properties": { + "runId": { + "type": "string", + "minLength": 1, + "description": "The memory ingest run id returned by memory_ingest." + } + }, + "required": [ + "runId" + ], + "$schema": "http://json-schema.org/draft-07/schema#" + }, + "outputSchema": { + "type": "object", + "properties": { + "runId": { + "type": "string" + }, + "status": { + "type": "string", + "enum": [ + "running", + "done", + "error" + ] + }, + "stage": { + "type": "string" + }, + "done": { + "type": "boolean" + }, + "captured": { + "type": "object", + "properties": { + "wiki": { + "type": "array", + "items": { + "type": "string" + } + }, + "sl": { + "type": "array", + "items": { + "type": "string" + } + }, + "xrefs": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "wiki", + "sl", + "xrefs" + ], + "additionalProperties": false + }, + "error": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "commitHash": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "skillsLoaded": { + "type": "array", + "items": { + "type": "string" + } + }, + "signalDetected": { + "type": "boolean" + } + }, + "required": [ + "runId", + "status", + "stage", + "done", + "captured", + "error", + "commitHash", + "skillsLoaded", + "signalDetected" + ], + "$schema": "http://json-schema.org/draft-07/schema#", + "additionalProperties": false + }, + "annotations": { + "title": "Memory Ingest Status", + "readOnlyHint": true, + "openWorldHint": false + }, + "execution": { + "taskSupport": "forbidden" + } + } +] diff --git a/packages/context/src/mcp/context-tools.ts b/packages/context/src/mcp/context-tools.ts index 773155bf..e040df87 100644 --- a/packages/context/src/mcp/context-tools.ts +++ b/packages/context/src/mcp/context-tools.ts @@ -1,5 +1,16 @@ +import { randomUUID } from 'node:crypto'; +import type { ToolAnnotations } from '@modelcontextprotocol/sdk/types.js'; import { z } from 'zod'; -import type { KtxMcpContextPorts, KtxMcpServerLike, KtxMcpToolResult, KtxMcpUserContext } from './types.js'; +import type { MemoryAgentInput } from '../memory/index.js'; +import type { + KtxMcpContextPorts, + KtxMcpProgressCallback, + KtxMcpServerLike, + KtxMcpToolHandlerContext, + KtxMcpToolResult, + KtxMcpUserContext, + NonArrayObject, +} from './types.js'; export interface RegisterKtxContextToolsDeps { server: KtxMcpServerLike; @@ -8,181 +19,433 @@ export interface RegisterKtxContextToolsDeps { } const connectionIdSchema = z.string().min(1); +const unknownRecordSchema = z.record(z.string(), z.unknown()); +const tableRefSchema = z.object({ + catalog: z.string().nullable(), + db: z.string().nullable(), + name: z.string(), +}); + +const toolAnnotations = { + connection_list: { title: 'Connection List', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + discover_data: { title: 'Discover Data', readOnlyHint: true, openWorldHint: false }, + wiki_search: { title: 'Wiki Search', readOnlyHint: true, openWorldHint: false }, + wiki_read: { title: 'Wiki Read', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + entity_details: { title: 'Entity Details', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + dictionary_search: { title: 'Dictionary Search', readOnlyHint: true, openWorldHint: false }, + sl_read_source: { title: 'Semantic Layer Read Source', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + sl_query: { title: 'Semantic Layer Query', readOnlyHint: true, openWorldHint: false }, + sql_execution: { title: 'SQL Execution', readOnlyHint: true, openWorldHint: false }, + memory_ingest: { title: 'Memory Ingest', destructiveHint: true, openWorldHint: false }, + memory_ingest_status: { title: 'Memory Ingest Status', readOnlyHint: true, openWorldHint: false }, +} satisfies Record; + +const toolDescriptions = { + connection_list: + 'List configured read-only data connections available to this KTX project. Use this before connection-scoped tools when the project may have multiple warehouses.', + discover_data: + 'Search across KTX wiki pages, semantic-layer sources, measures, dimensions, raw tables, and columns. Example: discover_data({ query: "monthly orders by customer", connectionId: "warehouse", kinds: ["sl_source", "table"] }).', + wiki_search: + 'Search KTX wiki pages for reusable business context. Example: wiki_search({ query: "revenue recognition", limit: 5 }).', + wiki_read: 'Read a KTX wiki page by key returned from wiki_search. Example: wiki_read({ key: "global/revenue" }).', + entity_details: + 'Read table and column metadata from the latest live-database scan snapshot. Example: entity_details({ connectionId: "warehouse", entities: [{ table: { schema: "public", table: "orders" }, columns: ["id"] }] }).', + dictionary_search: + 'Search profile-sampled warehouse values to locate likely source columns for business values. Example: dictionary_search({ values: ["Acme Corp"], connectionId: "warehouse" }).', + sl_read_source: + 'Read a semantic-layer YAML source by connection id and source name. Example: sl_read_source({ connectionId: "warehouse", sourceName: "orders" }).', + sl_query: + 'Execute a semantic-layer query and return rows, headers, generated SQL, and plan details. Example: sl_query({ connectionId: "warehouse", measures: ["orders.order_count"], dimensions: [{ dimension: "orders.created_at", granularity: "month" }] }).', + sql_execution: + 'Execute one parser-validated read-only SQL query against a configured KTX connection. Example: sql_execution({ connectionId: "warehouse", sql: "select count(*) from public.orders", maxRows: 100 }).', + memory_ingest: + 'Ingest free-form markdown knowledge into durable KTX memory. Use this for business rules, metric definitions, schema gotchas, recurring findings, or explicit user requests to remember something. Example: memory_ingest({ connectionId: "warehouse", content: "ARR is reported in cents in this warehouse." }).', + memory_ingest_status: + 'Read the current or final status for a memory ingest run. Example: memory_ingest_status({ runId: "memory-run-1" }).', +} satisfies Record; const connectionListSchema = z.object({}); -const connectionTestSchema = z.object({ - connectionId: connectionIdSchema, -}); - const knowledgeSearchSchema = z.object({ - query: z.string().min(1), - limit: z.number().int().min(1).max(50).default(10), + query: z.string().min(1).describe('Natural-language wiki search query, e.g. "revenue recognition policy".'), + limit: z.number().int().min(1).max(50).default(10).describe('Maximum wiki pages to return. Defaults to 10.'), }); const knowledgeReadSchema = z.object({ - key: z.string().min(1), -}); - -const historicSqlUsageFrontmatterSchema = z.object({ - executions: z.number().int().nonnegative(), - distinct_users: z.number().int().nonnegative(), - first_seen: z.string().min(1), - last_seen: z.string().min(1), - p50_runtime_ms: z.number().nonnegative().nullable(), - p95_runtime_ms: z.number().nonnegative().nullable(), - error_rate: z.number().min(0).max(1), - rows_produced: z.number().int().nonnegative().optional(), -}); - -const knowledgeWriteSchema = z.object({ - key: z.string().min(1).max(120), - summary: z.string().min(1).max(200), - content: z.string().min(1), - tags: z.array(z.string()).optional(), - refs: z.array(z.string()).optional(), - sl_refs: z.array(z.string()).optional(), - source: z.string().optional(), - intent: z.string().optional(), - tables: z.array(z.string()).optional(), - representative_sql: z.string().optional(), - usage: historicSqlUsageFrontmatterSchema.optional(), - fingerprints: z.array(z.string()).optional(), -}); - -const slListSourcesSchema = z.object({ - connectionId: connectionIdSchema.optional(), - query: z.string().min(1).optional(), + key: z.string().min(1).describe('Wiki page key returned by wiki_search, e.g. "global/revenue".'), }); const slReadSourceSchema = z.object({ - connectionId: connectionIdSchema, - sourceName: z.string().min(1), -}); - -const slWriteSourceSchema = z.object({ - connectionId: connectionIdSchema, - sourceName: z.string().regex(/^[a-z0-9][a-z0-9_]*$/, 'Source name must be snake_case'), - yaml: z.string().min(1).optional(), - source: z.record(z.string(), z.unknown()).optional(), - delete: z.boolean().optional(), -}); - -const slValidateSchema = z.object({ - connectionId: connectionIdSchema, - names: z.array(z.string().min(1)).optional(), + connectionId: connectionIdSchema.describe('Connection id that owns the semantic-layer source.'), + sourceName: z.string().min(1).describe('Semantic-layer source name without ".yaml", e.g. "orders".'), }); const slQueryMeasureSchema = z.union([ - z.string(), + z.string().describe('Semantic-layer measure key, e.g. "orders.order_count".'), z.object({ - expr: z.string().min(1), - name: z.string().min(1), + expr: z.string().min(1).describe('Ad hoc aggregate expression, e.g. "sum(orders.amount)".'), + name: z.string().min(1).describe('Alias for the ad hoc measure, e.g. "gross_revenue".'), }), ]); -const slQueryDimensionSchema = z.union([ - z.string(), +const slQueryDimensionSchema = z.preprocess( + (value) => { + if (typeof value === 'string') return { field: value }; + if (value && typeof value === 'object' && !Array.isArray(value)) { + const obj = { ...(value as Record) }; + if (!('field' in obj) && typeof obj.dimension === 'string') obj.field = obj.dimension; + return obj; + } + return value; + }, z.object({ - field: z.string().min(1), - granularity: z.string().min(1).optional(), + field: z.string().min(1).describe('Dimension to group by, e.g. "orders.created_at" or "orders.status".'), + granularity: z + .string() + .min(1) + .optional() + .describe('Time grain for time dimensions: day, week, month, quarter, or year.'), }), -]); +); -const slQueryOrderBySchema = z.union([ - z.string(), +const slQueryOrderBySchema = z.preprocess( + (value) => { + if (typeof value === 'string') { + return { field: value }; + } + if (value && typeof value === 'object' && !Array.isArray(value)) { + const obj = { ...(value as Record) }; + if (!('field' in obj) && typeof obj.id === 'string') { + obj.field = obj.id; + } + if (!('direction' in obj) && 'desc' in obj) { + obj.direction = obj.desc === true ? 'desc' : 'asc'; + } + return obj; + } + return value; + }, z.object({ - field: z.string().min(1), - direction: z.enum(['asc', 'desc']).default('asc'), + field: z + .string() + .min(1) + .describe( + 'Field/measure/dimension id to order by, e.g. "orders.created_at", a dimension key like "mart_nrr_quarterly.quarter_label", or a measure alias.', + ), + direction: z + .enum(['asc', 'desc']) + .default('asc') + .describe('Sort direction: "asc" or "desc". Defaults to "asc".'), }), -]); +); const slQuerySchema = z.object({ - connectionId: connectionIdSchema.optional(), - measures: z.array(slQueryMeasureSchema).min(1), - dimensions: z.array(slQueryDimensionSchema).default([]), - filters: z.array(z.string()).default([]), - segments: z.array(z.string()).default([]), - order_by: z.array(slQueryOrderBySchema).default([]), - limit: z.number().int().min(0).default(1000), - include_empty: z.boolean().default(true), + connectionId: connectionIdSchema + .optional() + .describe('Connection id to query. Omit only when the project has exactly one configured connection.'), + measures: z.array(slQueryMeasureSchema).min(1).describe('Measures to select. Use semantic-layer keys when available.'), + dimensions: z + .array(slQueryDimensionSchema) + .default([]) + .describe('Dimensions to group by. Strings and {dimension, granularity} are accepted.'), + filters: z + .array(z.string().describe('Semantic-layer filter expression, e.g. "orders.status = paid".')) + .default([]) + .describe('Semantic-layer filter expressions to apply.'), + segments: z + .array(z.string().describe('Semantic-layer segment key to apply.')) + .default([]) + .describe('Semantic-layer segment keys to apply.'), + order_by: z + .array(slQueryOrderBySchema) + .default([]) + .describe('Sort clauses. Strings and Cube-style {id, desc} are accepted.'), + limit: z.number().int().min(0).default(1000).describe('Maximum rows to return. Defaults to 1000.'), + include_empty: z.boolean().default(true).describe('Whether to include empty dimension groups. Defaults to true.'), }); -const ingestTriggerSchema = z.object({ - adapter: z.string().min(1), - connectionId: connectionIdSchema, - config: z.unknown().optional(), - trigger: z.enum(['upload', 'scheduled_pull', 'manual_resync']).default('manual_resync'), -}); - -const ingestStatusSchema = z.object({ - runId: z.string().min(1), -}); - -const ingestReportSchema = z.object({ - runId: z.string().min(1), -}); - -const ingestReplaySchema = z.object({ - runId: z.string().min(1), -}); - -const scanTriggerSchema = z.object({ - connectionId: connectionIdSchema, - mode: z.enum(['structural', 'relationships', 'enriched']).default('structural'), - detectRelationships: z.boolean().default(false), - dryRun: z.boolean().default(false), -}); - -const scanStatusSchema = z.object({ - runId: z.string().min(1), -}); - -const scanArtifactReadSchema = z.object({ - runId: z.string().min(1), - path: z.string().min(1), -}); - -const entityDetailsTableRefSchema = z.object({ - catalog: z.string().nullable(), - db: z.string().nullable(), - name: z.string().min(1), -}); +const entityDetailsTableRefSchema = z.preprocess( + (value) => { + if (value && typeof value === 'object' && !Array.isArray(value)) { + const obj = { ...(value as Record) }; + if (!('db' in obj) && typeof obj.schema === 'string') obj.db = obj.schema; + if (!('name' in obj) && typeof obj.table === 'string') obj.name = obj.table; + if (!('catalog' in obj)) obj.catalog = null; + return obj; + } + return value; + }, + z.object({ + catalog: z.string().nullable().describe('Catalog/project/database. Use null when not applicable.'), + db: z.string().nullable().describe('Schema/database/dataset. Use null when not applicable.'), + name: z.string().min(1).describe('Table name.'), + }), +); const entityDetailsSchema = z.object({ - connectionId: connectionIdSchema, + connectionId: connectionIdSchema.describe('Connection id whose latest scan snapshot should be read.'), entities: z .array( z.object({ - table: z.union([z.string().min(1), entityDetailsTableRefSchema]), - columns: z.array(z.string().min(1)).optional(), + table: z + .union([z.string().min(1), entityDetailsTableRefSchema]) + .describe('Table display string or object ref. {schema, table} is accepted as an alias for {db, name}.'), + columns: z + .array(z.string().min(1).describe('Column name to inspect.')) + .optional() + .describe('Optional column filter.'), }), ) .min(1) - .max(20), + .max(20) + .describe('Tables or columns to inspect. Maximum 20 entities.'), }); const dictionarySearchSchema = z.object({ - values: z.array(z.string().min(1)).min(1).max(20), - connectionId: connectionIdSchema.optional(), + values: z + .array(z.string().min(1).describe('Business value to locate, e.g. "Acme Corp" or "enterprise".')) + .min(1) + .max(20) + .describe('Values to search for in sampled warehouse dictionaries.'), + connectionId: connectionIdSchema + .optional() + .describe('Optional connection id. Pass it when user intent pins a specific warehouse.'), }); const discoverDataKindSchema = z.enum(['wiki', 'sl_source', 'sl_measure', 'sl_dimension', 'table', 'column']); const discoverDataSchema = z.object({ - query: z.string().min(1), - connectionId: connectionIdSchema.optional(), - kinds: z.array(discoverDataKindSchema).optional(), - limit: z.number().int().min(1).max(50).default(15).optional(), + query: z.string().min(1).describe('Natural-language discovery query, e.g. "monthly orders by customer".'), + connectionId: connectionIdSchema + .optional() + .describe('Optional connection id. Pass it when user intent pins a specific warehouse.'), + kinds: z.array(discoverDataKindSchema.describe('Reference kind to include.')).optional().describe('Optional kind filter.'), + limit: z.number().int().min(1).max(50).default(15).optional().describe('Maximum refs to return. Defaults to 15.'), }); const sqlExecutionSchema = z.object({ - connectionId: connectionIdSchema, - sql: z.string().min(1), - maxRows: z.number().int().min(1).max(10_000).default(1000).optional(), + connectionId: connectionIdSchema.describe('Connection id to execute against. Required for raw SQL.'), + sql: z.string().min(1).describe('Parser-validated read-only SQL, e.g. "select count(*) from public.orders".'), + maxRows: z.number().int().min(1).max(10_000).default(1000).optional().describe('Maximum rows to return. Defaults to 1000.'), }); -export function jsonToolResult(structuredContent: T): KtxMcpToolResult { +const memoryIngestSchema = z.object({ + content: z + .string() + .min(1) + .describe( + 'Free-form markdown to ingest. Include the knowledge itself plus any context (source, the user question, why this came up) that the memory agent should consider when triaging into wiki/SL.', + ), + connectionId: connectionIdSchema + .optional() + .describe( + 'Scope this memory to a specific connection. Required when the knowledge is warehouse-specific, including measure definitions, schema gotchas, or anything tied to a particular warehouse. Omit only for global wiki knowledge.', + ), +}); + +const memoryIngestStatusSchema = z.object({ + runId: z.string().min(1).describe('The memory ingest run id returned by memory_ingest.'), +}); + +const connectionListOutputSchema = z.object({ + connections: z.array( + z.object({ + id: z.string(), + name: z.string(), + connectionType: z.string(), + }), + ), +}); + +const wikiSearchOutputSchema = z.object({ + results: z.array( + z.object({ + key: z.string(), + path: z.string(), + scope: z.enum(['GLOBAL', 'USER']), + summary: z.string(), + score: z.number(), + matchReasons: z.array(z.string()).optional(), + lanes: z + .array( + z.object({ + lane: z.string(), + status: z.string(), + requestedCandidatePoolLimit: z.number(), + effectiveCandidatePoolLimit: z.number(), + returnedCandidateCount: z.number(), + weight: z.number(), + reason: z.string().optional(), + }), + ) + .optional(), + }), + ), + totalFound: z.number(), +}); + +const wikiReadOutputSchema = z.object({ + key: z.string(), + summary: z.string(), + content: z.string(), + scope: z.enum(['GLOBAL', 'USER']), + tags: z.array(z.string()).optional(), + refs: z.array(z.string()).optional(), + slRefs: z.array(z.string()).optional(), +}); + +const slReadSourceOutputSchema = z.object({ + sourceName: z.string(), + yaml: z.string(), +}); + +const slQueryOutputSchema = z.object({ + connectionId: z.string().optional(), + dialect: z.string().optional(), + sql: z.string(), + headers: z.array(z.string()), + rows: z.array(z.array(z.unknown())), + totalRows: z.number(), + plan: unknownRecordSchema.optional(), +}); + +const entityDetailsSnapshotOutputSchema = z.object({ + syncId: z.string(), + extractedAt: z.string(), + scanRunId: z.string().nullable(), +}); + +const entityDetailsColumnOutputSchema = z.object({ + name: z.string(), + nativeType: z.string(), + normalizedType: z.string(), + dimensionType: z.enum(['time', 'string', 'number', 'boolean']), + nullable: z.boolean(), + primaryKey: z.boolean(), + comment: z.string().nullable(), +}); + +const entityDetailsForeignKeyOutputSchema = z.object({ + fromColumn: z.string(), + toCatalog: z.string().nullable(), + toDb: z.string().nullable(), + toTable: z.string(), + toColumn: z.string(), + constraintName: z.string().nullable(), +}); + +const entityDetailsOutputSchema = z.object({ + results: z.array( + z.union([ + z.object({ + ok: z.literal(true), + connectionId: z.string(), + tableRef: tableRefSchema, + display: z.string(), + kind: z.enum(['table', 'view', 'external', 'event_stream']), + comment: z.string().nullable(), + estimatedRows: z.number().nullable(), + columns: z.array(entityDetailsColumnOutputSchema), + foreignKeys: z.array(entityDetailsForeignKeyOutputSchema), + snapshot: entityDetailsSnapshotOutputSchema, + }), + z.object({ + ok: z.literal(false), + connectionId: z.string(), + table: z.union([z.string(), tableRefSchema]), + snapshot: entityDetailsSnapshotOutputSchema.optional(), + error: z.object({ + code: z.enum(['scan_missing', 'table_not_found', 'ambiguous_table', 'column_not_found']), + message: z.string(), + candidates: z + .union([z.array(z.object({ tableRef: tableRefSchema, display: z.string() })), z.array(z.string())]) + .optional(), + }), + }), + ]), + ), +}); + +const dictionarySearchOutputSchema = z.object({ + searched: z.array( + z.object({ + connectionId: z.string(), + coverage: z.object({ + sampledRows: z.number().nullable(), + valuesPerColumn: z.number().nullable(), + profiledColumns: z.number(), + syncId: z.string().nullable(), + profiledAt: z.string().nullable(), + }), + status: z.enum(['ready', 'no_profile_artifact', 'no_candidate_columns']), + }), + ), + results: z.array( + z.object({ + value: z.string(), + matches: z.array( + z.object({ + connectionId: z.string(), + sourceName: z.string(), + columnName: z.string(), + matchedValue: z.string(), + cardinality: z.number().nullable(), + }), + ), + misses: z.array( + z.object({ + connectionId: z.string(), + reason: z.enum(['no_profile_artifact', 'no_candidate_columns', 'value_not_in_sample']), + }), + ), + }), + ), +}); + +const discoverDataOutputSchema = z.object({ + refs: z.array( + z.object({ + kind: discoverDataKindSchema, + id: z.string(), + score: z.number(), + summary: z.string().nullable(), + snippet: z.string().nullable(), + matchedOn: z.enum(['name', 'display', 'description', 'comment', 'expr', 'sample_value', 'body']), + connectionId: z.string().optional(), + tableRef: tableRefSchema.optional(), + columnName: z.string().optional(), + }), + ), +}); + +const sqlExecutionOutputSchema = z.object({ + headers: z.array(z.string()), + headerTypes: z.array(z.string()).optional(), + rows: z.array(z.array(z.unknown())), + rowCount: z.number(), +}); + +const memoryIngestOutputSchema = z.object({ + runId: z.string(), +}); + +const memoryIngestStatusOutputSchema = z.object({ + runId: z.string(), + status: z.enum(['running', 'done', 'error']), + stage: z.string(), + done: z.boolean(), + captured: z.object({ + wiki: z.array(z.string()), + sl: z.array(z.string()), + xrefs: z.array(z.string()), + }), + error: z.string().nullable(), + commitHash: z.string().nullable(), + skillsLoaded: z.array(z.string()), + signalDetected: z.boolean(), +}); + +export function jsonToolResult(structuredContent: T): KtxMcpToolResult { return { content: [{ type: 'text', text: JSON.stringify(structuredContent, null, 2) }], structuredContent, @@ -196,14 +459,53 @@ export function jsonErrorToolResult(text: string): KtxMcpToolResult `${issue.path.length > 0 ? issue.path.join('.') : ''}: ${issue.message}`) + .join('\n'); + } + return error instanceof Error ? error.message : String(error); +} + +function mcpProgressCallback(context?: KtxMcpToolHandlerContext): KtxMcpProgressCallback | undefined { + const progressToken = context?._meta?.progressToken; + if (progressToken === undefined || !context?.sendNotification) { + return undefined; + } + return async (event) => { + await context.sendNotification?.({ + method: 'notifications/progress', + params: { + progressToken, + progress: event.progress, + ...(event.total !== undefined ? { total: event.total } : {}), + message: event.message, + }, + }); + }; +} + function registerParsedTool( server: KtxMcpServerLike, name: string, - config: { title: string; description: string; inputSchema: unknown }, + config: { + title: string; + description: string; + inputSchema: unknown; + outputSchema: unknown; + annotations: ToolAnnotations; + }, schema: TSchema, - handler: (input: z.infer) => Promise, + handler: (input: z.infer, context?: KtxMcpToolHandlerContext) => Promise, ): void { - server.registerTool(name, config, async (input) => handler(schema.parse(input))); + server.registerTool(name, config, async (input, context) => { + try { + return await handler(schema.parse(input), context); + } catch (error) { + return jsonErrorToolResult(formatToolError(error)); + } + }); } export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void { @@ -215,32 +517,15 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void server, 'connection_list', { - title: 'Connection List', - description: 'List configured read-only data connections available to the KTX project.', + title: toolAnnotations.connection_list.title!, + description: toolDescriptions.connection_list, inputSchema: connectionListSchema.shape, + outputSchema: connectionListOutputSchema, + annotations: toolAnnotations.connection_list, }, connectionListSchema, async () => jsonToolResult({ connections: await connections.list() }), ); - - if (connections.test) { - registerParsedTool( - server, - 'connection_test', - { - title: 'Connection Test', - description: 'Test a configured standalone KTX connection through the host-provided scan connector.', - inputSchema: connectionTestSchema.shape, - }, - connectionTestSchema, - async (input) => { - const result = await connections.test?.({ connectionId: input.connectionId }); - return result - ? jsonToolResult(result) - : jsonErrorToolResult(`Connection "${input.connectionId}" was not found.`); - }, - ); - } } if (ports.knowledge) { @@ -249,9 +534,11 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void server, 'wiki_search', { - title: 'Wiki Search', - description: 'Search KTX wiki pages and return ranked summaries.', + title: toolAnnotations.wiki_search.title!, + description: toolDescriptions.wiki_search, inputSchema: knowledgeSearchSchema.shape, + outputSchema: wikiSearchOutputSchema, + annotations: toolAnnotations.wiki_search, }, knowledgeSearchSchema, async (input) => @@ -268,9 +555,11 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void server, 'wiki_read', { - title: 'Wiki Read', - description: 'Read a KTX wiki page by key.', + title: toolAnnotations.wiki_read.title!, + description: toolDescriptions.wiki_read, inputSchema: knowledgeReadSchema.shape, + outputSchema: wikiReadOutputSchema, + annotations: toolAnnotations.wiki_read, }, knowledgeReadSchema, async (input) => { @@ -278,58 +567,19 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void return page ? jsonToolResult(page) : jsonErrorToolResult(`Wiki page "${input.key}" was not found.`); }, ); - - registerParsedTool( - server, - 'wiki_write', - { - title: 'Wiki Write', - description: 'Create or replace a KTX wiki page and its SL references.', - inputSchema: knowledgeWriteSchema.shape, - }, - knowledgeWriteSchema, - async (input) => - jsonToolResult( - await knowledge.write({ - userId: userContext.userId, - key: input.key, - summary: input.summary, - content: input.content, - tags: input.tags, - refs: input.refs, - slRefs: input.sl_refs, - source: input.source, - intent: input.intent, - tables: input.tables, - representativeSql: input.representative_sql, - usage: input.usage, - fingerprints: input.fingerprints, - }), - ), - ); } if (ports.semanticLayer) { const semanticLayer = ports.semanticLayer; - registerParsedTool( - server, - 'sl_list_sources', - { - title: 'Semantic Layer List Sources', - description: 'List semantic-layer sources, optionally filtered by connection or search query.', - inputSchema: slListSourcesSchema.shape, - }, - slListSourcesSchema, - async (input) => jsonToolResult(await semanticLayer.listSources(input)), - ); - registerParsedTool( server, 'sl_read_source', { - title: 'Semantic Layer Read Source', - description: 'Read a semantic-layer YAML source by connection id and source name.', + title: toolAnnotations.sl_read_source.title!, + description: toolDescriptions.sl_read_source, inputSchema: slReadSourceSchema.shape, + outputSchema: slReadSourceOutputSchema, + annotations: toolAnnotations.sl_read_source, }, slReadSourceSchema, async (input) => { @@ -340,63 +590,37 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void }, ); - registerParsedTool( - server, - 'sl_write_source', - { - title: 'Semantic Layer Write Source', - description: 'Create, replace, or delete a semantic-layer source.', - inputSchema: slWriteSourceSchema.shape, - }, - slWriteSourceSchema, - async (input) => - jsonToolResult( - await semanticLayer.writeSource({ - connectionId: input.connectionId, - sourceName: input.sourceName, - yaml: input.yaml, - source: input.source, - delete: input.delete, - }), - ), - ); - - registerParsedTool( - server, - 'sl_validate', - { - title: 'Semantic Layer Validate', - description: 'Validate semantic-layer sources for a connection.', - inputSchema: slValidateSchema.shape, - }, - slValidateSchema, - async (input) => jsonToolResult(await semanticLayer.validate(input)), - ); - registerParsedTool( server, 'sl_query', { - title: 'Semantic Layer Query', - description: 'Execute a semantic-layer query and return rows, headers, SQL, and the query plan.', + title: toolAnnotations.sl_query.title!, + description: toolDescriptions.sl_query, inputSchema: slQuerySchema.shape, + outputSchema: slQueryOutputSchema, + annotations: toolAnnotations.sl_query, }, slQuerySchema, - async (input) => - jsonToolResult( - await semanticLayer.query({ - connectionId: input.connectionId, - query: { - measures: input.measures, - dimensions: input.dimensions, - filters: input.filters, - segments: input.segments, - order_by: input.order_by, - limit: input.limit, - include_empty: input.include_empty, + async (input, context) => { + const onProgress = mcpProgressCallback(context); + return jsonToolResult( + await semanticLayer.query( + { + connectionId: input.connectionId, + query: { + measures: input.measures, + dimensions: input.dimensions, + filters: input.filters, + segments: input.segments, + order_by: input.order_by, + limit: input.limit, + include_empty: input.include_empty, + }, }, - }), - ), + onProgress ? { onProgress } : undefined, + ), + ); + }, ); } @@ -406,9 +630,11 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void server, 'entity_details', { - title: 'Entity Details', - description: 'Read raw table and column metadata from the latest KTX live-database scan snapshot.', + title: toolAnnotations.entity_details.title!, + description: toolDescriptions.entity_details, inputSchema: entityDetailsSchema.shape, + outputSchema: entityDetailsOutputSchema, + annotations: toolAnnotations.entity_details, }, entityDetailsSchema, async (input) => jsonToolResult(await entityDetails.read(input)), @@ -421,10 +647,11 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void server, 'dictionary_search', { - title: 'Dictionary Search', - description: - 'Search profile-sampled warehouse values and report matching connection/source/column locations plus non-authoritative miss reasons.', + title: toolAnnotations.dictionary_search.title!, + description: toolDescriptions.dictionary_search, inputSchema: dictionarySearchSchema.shape, + outputSchema: dictionarySearchOutputSchema, + annotations: toolAnnotations.dictionary_search, }, dictionarySearchSchema, async (input) => jsonToolResult(await dictionarySearch.search(input)), @@ -437,13 +664,14 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void server, 'discover_data', { - title: 'Discover Data', - description: - 'Search across KTX wiki pages, semantic-layer sources/measures/dimensions, and raw warehouse schema refs.', + title: toolAnnotations.discover_data.title!, + description: toolDescriptions.discover_data, inputSchema: discoverDataSchema.shape, + outputSchema: discoverDataOutputSchema, + annotations: toolAnnotations.discover_data, }, discoverDataSchema, - async (input) => jsonToolResult(await discover.search(input)), + async (input) => jsonToolResult({ refs: await discover.search(input) }), ); } @@ -453,171 +681,70 @@ export function registerKtxContextTools(deps: RegisterKtxContextToolsDeps): void server, 'sql_execution', { - title: 'SQL Execution', - description: - 'Execute one parser-validated read-only SQL query against a configured KTX connection and return structured rows.', + title: toolAnnotations.sql_execution.title!, + description: toolDescriptions.sql_execution, inputSchema: sqlExecutionSchema.shape, + outputSchema: sqlExecutionOutputSchema, + annotations: toolAnnotations.sql_execution, }, sqlExecutionSchema, - async (input) => { - try { - return jsonToolResult( - await sqlExecution.execute({ + async (input, context) => { + const onProgress = mcpProgressCallback(context); + return jsonToolResult( + await sqlExecution.execute( + { connectionId: input.connectionId, sql: input.sql, maxRows: input.maxRows ?? 1000, - }), - ); - } catch (error) { - return jsonErrorToolResult(error instanceof Error ? error.message : String(error)); - } + }, + onProgress ? { onProgress } : undefined, + ), + ); }, ); } - if (ports.ingest) { - const ingest = ports.ingest; + if (ports.memoryIngest) { + const memoryIngest = ports.memoryIngest; registerParsedTool( server, - 'ingest_trigger', + 'memory_ingest', { - title: 'Ingest Trigger', - description: 'Trigger a KTX ingest run for an adapter and connection.', - inputSchema: ingestTriggerSchema.shape, + title: toolAnnotations.memory_ingest.title!, + description: toolDescriptions.memory_ingest, + inputSchema: memoryIngestSchema.shape, + outputSchema: memoryIngestOutputSchema, + annotations: toolAnnotations.memory_ingest, }, - ingestTriggerSchema, - async (input) => jsonToolResult(await ingest.trigger(input)), - ); - - registerParsedTool( - server, - 'ingest_status', - { - title: 'Ingest Status', - description: - 'Read the current or final status for an ingest run, including local diff and work-unit summaries when available.', - inputSchema: ingestStatusSchema.shape, - }, - ingestStatusSchema, + memoryIngestSchema, async (input) => { - const status = await ingest.status(input); - return status ? jsonToolResult(status) : jsonErrorToolResult(`Ingest run "${input.runId}" was not found.`); + const ingestInput: MemoryAgentInput = { + userId: userContext.userId, + chatId: `mcp-${randomUUID()}`, + userMessage: 'Ingest external knowledge into KTX memory.', + assistantMessage: input.content, + connectionId: input.connectionId, + sourceType: 'external_ingest', + }; + return jsonToolResult(await memoryIngest.ingest(ingestInput)); }, ); - if (ingest.report) { - registerParsedTool( - server, - 'ingest_report', - { - title: 'Ingest Report', - description: 'Read the stored canonical KTX ingest report for a local run id, job id, or report id.', - inputSchema: ingestReportSchema.shape, - }, - ingestReportSchema, - async (input) => { - const report = await ingest.report?.(input); - return report ? jsonToolResult(report) : jsonErrorToolResult(`Ingest report "${input.runId}" was not found.`); - }, - ); - } - - if (ingest.replay) { - registerParsedTool( - server, - 'ingest_replay', - { - title: 'Ingest Replay', - description: 'Read the memory-flow replay snapshot for a stored canonical KTX ingest run.', - inputSchema: ingestReplaySchema.shape, - }, - ingestReplaySchema, - async (input) => { - const replay = await ingest.replay?.(input); - return replay ? jsonToolResult(replay) : jsonErrorToolResult(`Ingest replay "${input.runId}" was not found.`); - }, - ); - } - } - - if (ports.scan) { - const scan = ports.scan; - registerParsedTool( - server, - 'scan_trigger', - { - title: 'Scan Trigger', - description: 'Run a standalone KTX structural connection scan and return its report summary.', - inputSchema: scanTriggerSchema.shape, - }, - scanTriggerSchema, - async (input) => jsonToolResult(await scan.trigger(input)), - ); - registerParsedTool( server, - 'scan_status', + 'memory_ingest_status', { - title: 'Scan Status', - description: 'Read the current or final status for a standalone KTX scan run.', - inputSchema: scanStatusSchema.shape, + title: toolAnnotations.memory_ingest_status.title!, + description: toolDescriptions.memory_ingest_status, + inputSchema: memoryIngestStatusSchema.shape, + outputSchema: memoryIngestStatusOutputSchema, + annotations: toolAnnotations.memory_ingest_status, }, - scanStatusSchema, + memoryIngestStatusSchema, async (input) => { - const status = await scan.status(input); - return status ? jsonToolResult(status) : jsonErrorToolResult(`Scan run "${input.runId}" was not found.`); + const status = await memoryIngest.status(input.runId); + return status ? jsonToolResult(status) : jsonErrorToolResult(`Memory ingest run "${input.runId}" was not found.`); }, ); - - registerParsedTool( - server, - 'scan_report', - { - title: 'Scan Report', - description: 'Read a standalone KTX scan report by run id.', - inputSchema: scanStatusSchema.shape, - }, - scanStatusSchema, - async (input) => { - const report = await scan.report(input); - return report ? jsonToolResult(report) : jsonErrorToolResult(`Scan report "${input.runId}" was not found.`); - }, - ); - - if (scan.listArtifacts) { - registerParsedTool( - server, - 'scan_list_artifacts', - { - title: 'Scan List Artifacts', - description: 'List report, raw-source, manifest, and enrichment artifact paths for a standalone KTX scan run.', - inputSchema: scanStatusSchema.shape, - }, - scanStatusSchema, - async (input) => { - const result = await scan.listArtifacts?.({ runId: input.runId }); - return result ? jsonToolResult(result) : jsonErrorToolResult(`Scan run "${input.runId}" was not found.`); - }, - ); - } - - if (scan.readArtifact) { - registerParsedTool( - server, - 'scan_read_artifact', - { - title: 'Scan Read Artifact', - description: 'Read one artifact that belongs to a standalone KTX scan run.', - inputSchema: scanArtifactReadSchema.shape, - }, - scanArtifactReadSchema, - async (input) => { - const result = await scan.readArtifact?.({ runId: input.runId, path: input.path }); - return result - ? jsonToolResult(result) - : jsonErrorToolResult(`Scan artifact "${input.path}" was not found for run "${input.runId}".`); - }, - ); - } } } diff --git a/packages/context/src/mcp/index.ts b/packages/context/src/mcp/index.ts index df1bc6c5..f241c68e 100644 --- a/packages/context/src/mcp/index.ts +++ b/packages/context/src/mcp/index.ts @@ -8,29 +8,18 @@ export type { KtxDiscoverDataMcpPort, KtxDictionarySearchMcpPort, KtxEntityDetailsMcpPort, - KtxIngestDiffSummary, - KtxIngestMcpPort, - KtxIngestStatusResponse, - KtxIngestTriggerKind, - KtxIngestTriggerResponse, - KtxIngestWorkUnitSummary, KtxKnowledgeMcpPort, KtxKnowledgePage, KtxKnowledgeSearchResponse, KtxKnowledgeSearchResult, - KtxKnowledgeWriteResponse, KtxMcpContextPorts, KtxMcpServerDeps, KtxMcpServerLike, KtxMcpTextContent, KtxMcpToolResult, KtxMcpUserContext, - KtxSemanticLayerListResponse, KtxSemanticLayerMcpPort, KtxSemanticLayerQueryResponse, KtxSemanticLayerReadResponse, - KtxSemanticLayerSourceSummary, - KtxSemanticLayerValidationResponse, - KtxSemanticLayerWriteResponse, - MemoryCapturePort, + MemoryIngestPort, } from './types.js'; diff --git a/packages/context/src/mcp/local-project-ports.test.ts b/packages/context/src/mcp/local-project-ports.test.ts index 0c000831..119e901d 100644 --- a/packages/context/src/mcp/local-project-ports.test.ts +++ b/packages/context/src/mcp/local-project-ports.test.ts @@ -1,9 +1,7 @@ -import { access, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import { AgentRunnerService } from '../agent/index.js'; -import { FakeSourceAdapter, type MemoryFlowReplayInput } from '../ingest/index.js'; import { initKtxProject } from '../project/index.js'; import { createKtxConnectorCapabilities, @@ -14,14 +12,6 @@ import { import { writeLocalSlSource } from '../sl/index.js'; import { createLocalProjectMcpContextPorts } from './local-project-ports.js'; -class TestAgentRunner extends AgentRunnerService { - override runLoop = vi.fn().mockResolvedValue({ stopReason: 'natural' as const }); - - constructor() { - super({ llmProvider: { getModel: () => ({}) as never } as never }); - } -} - describe('createLocalProjectMcpContextPorts', () => { let tempDir: string; @@ -178,7 +168,7 @@ describe('createLocalProjectMcpContextPorts', () => { ); } - it('lists local project connections from ktx.yaml', async () => { + it('lists local project connections and exposes only retained research ports', async () => { const project = await initKtxProject({ projectDir: tempDir }); project.config.connections.warehouse = { driver: 'postgres', @@ -186,48 +176,23 @@ describe('createLocalProjectMcpContextPorts', () => { }; const ports = createLocalProjectMcpContextPorts(project); + expect(Object.keys(ports).sort()).toEqual([ + 'connections', + 'dictionarySearch', + 'discover', + 'entityDetails', + 'knowledge', + 'semanticLayer', + ]); + expect(Object.keys(ports.connections ?? {}).sort()).toEqual(['list']); + expect(Object.keys(ports.knowledge ?? {}).sort()).toEqual(['read', 'search']); + expect(Object.keys(ports.semanticLayer ?? {}).sort()).toEqual(['query', 'readSource']); await expect(ports.connections?.list()).resolves.toEqual([ { id: 'warehouse', name: 'warehouse', connectionType: 'POSTGRESQL' }, ]); }); - it('tests a local project connection through the native scan connector factory', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - project.config.connections.warehouse = { - driver: 'postgres', - url: 'env:DATABASE_URL', - }; - const connector = testConnector(); - const createConnector = vi.fn(async () => connector); - const ports = createLocalProjectMcpContextPorts(project, { - localScan: { - createConnector, - }, - }); - - await expect(ports.connections?.test?.({ connectionId: 'warehouse' })).resolves.toEqual({ - id: 'warehouse', - connectionType: 'POSTGRESQL', - ok: true, - tableCount: 1, - message: 'Connection test passed.', - warnings: [], - }); - expect(createConnector).toHaveBeenCalledWith('warehouse'); - expect(connector.introspect).toHaveBeenCalledWith( - { - connectionId: 'warehouse', - driver: 'postgres', - mode: 'structural', - dryRun: true, - detectRelationships: false, - }, - { runId: 'connection-test-warehouse' }, - ); - expect(connector.cleanup).toHaveBeenCalled(); - }); - - it('executes MCP SQL only after parser-backed validation passes', async () => { + it('adds sql_execution when parser validation and a SQL-capable connector are configured', async () => { const project = await initKtxProject({ projectDir: tempDir }); project.config.connections.warehouse = { driver: 'postgres', @@ -253,6 +218,7 @@ describe('createLocalProjectMcpContextPorts', () => { }, }); + expect(Object.keys(ports).sort()).toContain('sqlExecution'); await expect( ports.sqlExecution?.execute({ connectionId: 'warehouse', @@ -278,6 +244,50 @@ describe('createLocalProjectMcpContextPorts', () => { expect(connector.cleanup).toHaveBeenCalled(); }); + it('emits sql_execution progress stages from local MCP ports', async () => { + const project = await initKtxProject({ projectDir: tempDir }); + project.config.connections.warehouse = { + driver: 'postgres', + url: 'env:DATABASE_URL', + }; + const connector = testConnector(testSnapshot(), { + headers: ['id'], + headerTypes: ['integer'], + rows: [[1]], + totalRows: 1, + rowCount: 1, + }); + const createConnector = vi.fn(async () => connector); + const sqlAnalysis = { + analyzeForFingerprint: vi.fn(), + analyzeBatch: vi.fn(), + validateReadOnly: vi.fn(async () => ({ ok: true, error: null })), + }; + const progress: Array<{ progress: number; message: string }> = []; + const ports = createLocalProjectMcpContextPorts(project, { + sqlAnalysis, + localScan: { + createConnector, + }, + }); + + const result = await ports.sqlExecution?.execute( + { connectionId: 'warehouse', sql: 'select id from public.orders', maxRows: 5 }, + { + onProgress: (event) => { + progress.push({ progress: event.progress, message: event.message }); + }, + }, + ); + + expect(result?.rowCount).toBe(1); + expect(progress).toEqual([ + { progress: 0, message: 'Validating SQL' }, + { progress: 0.3, message: 'Executing' }, + { progress: 1, message: 'Fetched 1 rows' }, + ]); + }); + it('rejects MCP SQL before connector execution when parser validation fails', async () => { const project = await initKtxProject({ projectDir: tempDir }); project.config.connections.warehouse = { @@ -603,241 +613,30 @@ describe('createLocalProjectMcpContextPorts', () => { ); }); - it('triggers canonical bundle ingest and reads status, report, and replay through MCP ports', async () => { + it('reads and searches seeded global wiki pages', async () => { const project = await initKtxProject({ projectDir: tempDir }); - project.config.connections.warehouse = { - driver: 'postgres', - }; - project.config.ingest.adapters = ['fake']; - project.config.ingest.embeddings = { - backend: 'deterministic', - dimensions: 8, - batchSize: 64, - }; - project.config.llm = { - provider: { backend: 'none' }, - models: {}, - }; - - const sourceDir = join(tempDir, 'source'); - await mkdir(join(sourceDir, 'orders'), { recursive: true }); - await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\n', 'utf-8'); - - const agentRunner = new TestAgentRunner(); - const ports = createLocalProjectMcpContextPorts(project, { - localIngest: { - adapters: [new FakeSourceAdapter()], - jobIdFactory: () => 'mcp-full-1', - agentRunner, - }, - }); - - const trigger = await ports.ingest?.trigger({ - adapter: 'fake', - connectionId: 'warehouse', - trigger: 'manual_resync', - config: { sourceDir }, - }); - - expect(trigger).toMatchObject({ - runId: expect.any(String), - jobId: 'mcp-full-1', - reportId: expect.any(String), - }); - expect(trigger?.runId).not.toBe('mcp-full-1'); - expect(agentRunner.runLoop).toHaveBeenCalledTimes(1); - - await expect(ports.ingest?.status({ runId: trigger?.jobId ?? '' })).resolves.toMatchObject({ - runId: trigger?.runId, - jobId: 'mcp-full-1', - reportId: trigger?.reportId, - status: 'done', - stage: 'done', - progress: 1, - done: true, - adapter: 'fake', - connectionId: 'warehouse', - sourceDir: null, - diffSummary: { added: 1, modified: 0, deleted: 0, unchanged: 0 }, - rawFileCount: 1, - workUnitCount: 1, - workUnits: [ - { - unitKey: 'fake-orders', - rawFiles: ['orders/orders.json'], - peerFileIndex: [], - dependencyPaths: [], - }, - ], - evictionDeletedRawPaths: [], - errors: [], - }); - - await expect(ports.ingest?.report?.({ runId: trigger?.reportId ?? '' })).resolves.toMatchObject({ - id: trigger?.reportId, - runId: trigger?.runId, - jobId: 'mcp-full-1', - connectionId: 'warehouse', - sourceKey: 'fake', - }); - - const replay = (await ports.ingest?.replay?.({ runId: trigger?.runId ?? '' })) as MemoryFlowReplayInput | null; - expect(replay).toMatchObject({ - runId: trigger?.runId, - reportId: trigger?.reportId, - reportPath: trigger?.reportId, - status: 'done', - adapter: 'fake', - connectionId: 'warehouse', - syncId: expect.stringContaining('mcp-full-1'), - }); - expect(replay?.events).toEqual( - expect.arrayContaining([ - { type: 'work_unit_finished', unitKey: 'fake-orders', status: 'success' }, - { type: 'report_created', runId: trigger?.runId, reportPath: trigger?.reportId }, - ]), + await project.fileStore.writeFile( + 'wiki/global/revenue.md', + [ + '---', + 'summary: Revenue definition', + 'tags: [finance]', + 'refs: [docs/revenue.md]', + 'sl_refs: [warehouse.orders]', + 'usage_mode: auto', + '---', + '', + '# Revenue', + '', + 'Revenue is net of refunds.', + '', + ].join('\n'), + 'ktx', + 'ktx@example.com', + 'Seed wiki', ); - }); - - it('returns child run metadata for local Metabase fan-out triggers', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - project.config.connections = { - 'prod-metabase': { - driver: 'metabase', - api_url: 'https://metabase.example.com', - }, - warehouse_a: { driver: 'postgres', url: 'postgres://localhost/a' }, - warehouse_b: { driver: 'postgres', url: 'postgres://localhost/b' }, - }; - project.config.ingest.adapters = ['metabase']; - const reportA = { - id: 'report-a', - runId: 'run-a', - jobId: 'child-a', - connectionId: 'warehouse_a', - sourceKey: 'metabase', - createdAt: '2026-05-04T12:00:00.000Z', - body: { - syncId: 'sync-a', - diffSummary: { added: 1, modified: 0, deleted: 0, unchanged: 0 }, - commitSha: null, - workUnits: [], - failedWorkUnits: [], - reconciliationSkipped: false, - conflictsResolved: [], - evictionsApplied: [], - unmappedFallbacks: [], - evictionInputs: [], - unresolvedCards: [], - supersededBy: null, - overrideOf: null, - provenanceRows: [], - toolTranscripts: [], - }, - }; - const reportB = { - ...reportA, - id: 'report-b', - runId: 'run-b', - jobId: 'child-b', - connectionId: 'warehouse_b', - body: { ...reportA.body, syncId: 'sync-b' }, - }; - - const ports = createLocalProjectMcpContextPorts(project, { - localIngest: { - runLocalMetabaseIngest: async () => ({ - metabaseConnectionId: 'prod-metabase', - status: 'all_succeeded', - totals: { workUnits: 2, failedWorkUnits: 0 }, - children: [ - { - jobId: 'child-a', - metabaseConnectionId: 'prod-metabase', - metabaseDatabaseId: 1, - targetConnectionId: 'warehouse_a', - result: { - jobId: 'child-a', - runId: 'run-a', - syncId: 'sync-a', - diffSummary: { added: 1, modified: 0, deleted: 0, unchanged: 0 }, - workUnitCount: 0, - failedWorkUnits: [], - artifactsWritten: 0, - commitSha: null, - }, - report: reportA, - }, - { - jobId: 'child-b', - metabaseConnectionId: 'prod-metabase', - metabaseDatabaseId: 2, - targetConnectionId: 'warehouse_b', - result: { - jobId: 'child-b', - runId: 'run-b', - syncId: 'sync-b', - diffSummary: { added: 1, modified: 0, deleted: 0, unchanged: 0 }, - workUnitCount: 0, - failedWorkUnits: [], - artifactsWritten: 0, - commitSha: null, - }, - report: reportB, - }, - ], - }), - }, - }); - - await expect( - ports.ingest?.trigger({ - adapter: 'metabase', - connectionId: 'prod-metabase', - trigger: 'manual_resync', - }), - ).resolves.toEqual({ - runId: 'metabase-fanout:prod-metabase', - jobId: undefined, - reportId: undefined, - fanout: { - status: 'all_succeeded', - children: [ - { - runId: 'run-a', - jobId: 'child-a', - reportId: 'report-a', - targetConnectionId: 'warehouse_a', - metabaseDatabaseId: 1, - }, - { - runId: 'run-b', - jobId: 'child-b', - reportId: 'report-b', - targetConnectionId: 'warehouse_b', - metabaseDatabaseId: 2, - }, - ], - }, - }); - }); - - it('writes, reads, and searches global wiki pages', async () => { - const project = await initKtxProject({ projectDir: tempDir }); const ports = createLocalProjectMcpContextPorts(project); - await expect( - ports.knowledge?.write({ - userId: 'local-user', - key: 'revenue', - summary: 'Revenue definition', - content: '# Revenue\n\nRevenue is net of refunds.', - tags: ['finance'], - refs: ['docs/revenue.md'], - slRefs: ['warehouse.orders'], - }), - ).resolves.toMatchObject({ success: true, key: 'revenue', action: 'created' }); - await expect(ports.knowledge?.read({ userId: 'local-user', key: 'revenue' })).resolves.toMatchObject({ key: 'revenue', scope: 'GLOBAL', @@ -863,231 +662,32 @@ describe('createLocalProjectMcpContextPorts', () => { totalFound: 1, }); expect(search?.results[0]?.score).toBeGreaterThan(0); - await expect(access(join(project.projectDir, '.ktx', 'db.sqlite'))).resolves.toBeUndefined(); }); - it('writes, lists, reads, and validates semantic-layer sources', async () => { + it('reads seeded semantic-layer sources', async () => { const project = await initKtxProject({ projectDir: tempDir }); + await writeLocalSlSource(project, { + connectionId: 'warehouse', + sourceName: 'orders', + yaml: [ + 'name: orders', + 'table: public.orders', + 'grain:', + ' - id', + 'columns:', + ' - name: id', + ' type: number', + '', + ].join('\n'), + }); const ports = createLocalProjectMcpContextPorts(project); - await expect( - ports.semanticLayer?.writeSource({ - connectionId: 'warehouse', - sourceName: 'orders', - source: { - name: 'orders', - table: 'public.orders', - grain: ['id'], - columns: [{ name: 'id', type: 'number' }], - joins: [], - measures: [{ name: 'order_count', expr: 'count(*)' }], - }, - }), - ).resolves.toMatchObject({ success: true, sourceName: 'orders' }); - - await expect(ports.semanticLayer?.listSources({ connectionId: 'warehouse' })).resolves.toEqual({ - sources: [ - { - connectionId: 'warehouse', - connectionName: 'warehouse', - name: 'orders', - columnCount: 1, - measureCount: 1, - joinCount: 0, - }, - ], - totalSources: 1, - }); - - await expect( - ports.semanticLayer?.listSources({ connectionId: 'warehouse', query: 'order_count' }), - ).resolves.toEqual({ - sources: [ - expect.objectContaining({ - connectionId: 'warehouse', - connectionName: 'warehouse', - name: 'orders', - columnCount: 1, - measureCount: 1, - joinCount: 0, - score: expect.any(Number), - matchReasons: expect.arrayContaining(['lexical']), - }), - ], - totalSources: 1, - }); - await expect(access(join(project.projectDir, '.ktx/db.sqlite'))).resolves.toBeUndefined(); - await expect( ports.semanticLayer?.readSource({ connectionId: 'warehouse', sourceName: 'orders' }), ).resolves.toMatchObject({ sourceName: 'orders', yaml: expect.stringContaining('name: orders'), }); - - await expect(ports.semanticLayer?.validate({ connectionId: 'warehouse' })).resolves.toEqual({ - success: true, - errors: [], - warnings: ['Local stdio validation checks YAML shape only; Python semantic validation is not configured.'], - }); - }); - - it('returns semantic-layer hybrid search metadata through local project ports', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - await writeLocalSlSource(project, { - connectionId: 'warehouse', - sourceName: 'orders', - yaml: [ - 'name: orders', - 'table: public.orders', - 'grain:', - ' - order_id', - 'columns:', - ' - name: order_id', - ' type: string', - ' - name: status', - ' type: string', - '', - ].join('\n'), - }); - await project.fileStore.writeFile( - 'raw-sources/warehouse/live-database/sync-1/enrichment/relationship-profile.json', - `${JSON.stringify( - { - connectionId: 'warehouse', - driver: 'postgres', - sqlAvailable: true, - queryCount: 2, - tables: [], - columns: { - 'orders.status': { - table: { catalog: null, db: 'public', name: 'orders' }, - column: 'status', - nativeType: 'text', - normalizedType: 'string', - rowCount: 10, - nullCount: 0, - distinctCount: 2, - uniquenessRatio: 0.2, - nullRate: 0, - sampleValues: ['paid', 'refunded'], - minTextLength: 4, - maxTextLength: 8, - }, - }, - warnings: [], - }, - null, - 2, - )}\n`, - 'ktx', - 'ktx@example.com', - 'Seed dictionary profile', - ); - - const ports = createLocalProjectMcpContextPorts(project); - await expect(ports.semanticLayer?.listSources({ connectionId: 'warehouse', query: 'paid' })).resolves.toEqual({ - sources: [ - expect.objectContaining({ - connectionId: 'warehouse', - connectionName: 'warehouse', - name: 'orders', - score: expect.any(Number), - matchReasons: expect.arrayContaining(['dictionary']), - dictionaryMatches: [{ column: 'status', values: ['paid'] }], - }), - ], - totalSources: 1, - }); - }); - - it('returns historic SQL usage frequency and snippet through semantic-layer list search', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - await project.fileStore.writeFile( - 'semantic-layer/warehouse/_schema/public.yaml', - `tables: - orders: - table: public.orders - usage: - narrative: Analysts inspect paid order lifecycle by customer segment. - frequencyTier: high - commonFilters: - - status - commonGroupBys: - - customer_segment - commonJoins: - - table: public.customers - on: - - customer_id - columns: - - name: order_id - type: string - - name: status - type: string -`, - 'ktx', - 'ktx@example.com', - 'Seed usage-backed manifest shard', - ); - - const ports = createLocalProjectMcpContextPorts(project); - await expect( - ports.semanticLayer?.listSources({ connectionId: 'warehouse', query: 'paid order lifecycle' }), - ).resolves.toEqual({ - sources: [ - expect.objectContaining({ - connectionId: 'warehouse', - connectionName: 'warehouse', - name: 'orders', - frequencyTier: 'high', - snippet: expect.stringContaining(''), - score: expect.any(Number), - matchReasons: expect.arrayContaining(['lexical']), - }), - ], - totalSources: 1, - }); - }); - - it('uses configured local embeddings for semantic-layer search when available', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - project.config.ingest.embeddings = { backend: 'none', dimensions: 2 }; - await writeLocalSlSource(project, { - connectionId: 'warehouse', - sourceName: 'orders', - yaml: [ - 'name: orders', - 'descriptions:', - ' user: Revenue facts', - 'table: public.orders', - 'grain:', - ' - order_id', - 'columns:', - ' - name: order_id', - ' type: string', - '', - ].join('\n'), - }); - - const ports = createLocalProjectMcpContextPorts(project, { - embeddingService: { - maxBatchSize: 8, - async computeEmbedding(text: string) { - return text.includes('cash collection') ? [1, 0] : [0, 1]; - }, - async computeEmbeddingsBulk(texts: string[]) { - return texts.map((text) => (text.includes('Revenue facts') ? [1, 0] : [0, 1])); - }, - }, - }); - - const result = await ports.semanticLayer?.listSources({ connectionId: 'warehouse', query: 'cash collection' }); - - expect(result?.sources[0]).toMatchObject({ - name: 'orders', - matchReasons: expect.arrayContaining(['semantic']), - lanes: expect.arrayContaining([expect.objectContaining({ lane: 'semantic', status: 'available' })]), - }); }); it('rejects path traversal keys before touching the project directory', async () => { @@ -1109,36 +709,35 @@ describe('createLocalProjectMcpContextPorts', () => { ).rejects.toThrow('Unsafe semantic-layer source name'); }); - it('uses semantic compute for validation and compile-only sl_query when supplied', async () => { + it('uses semantic compute for compile-only sl_query when supplied', async () => { const project = await initKtxProject({ projectDir: tempDir }); project.config.connections.warehouse = { driver: 'postgres', url: 'env:DATABASE_URL', }; - const shapeOnlyPorts = createLocalProjectMcpContextPorts(project); - await shapeOnlyPorts.semanticLayer?.writeSource({ + await writeLocalSlSource(project, { connectionId: 'warehouse', sourceName: 'orders', - source: { - name: 'orders', - table: 'public.orders', - grain: ['id'], - columns: [ - { name: 'id', type: 'number' }, - { name: 'status', type: 'string' }, - ], - joins: [], - measures: [{ name: 'order_count', expr: 'count(*)' }], - }, + yaml: [ + 'name: orders', + 'table: public.orders', + 'grain:', + ' - id', + 'columns:', + ' - name: id', + ' type: number', + ' - name: status', + ' type: string', + 'joins: []', + 'measures:', + ' - name: order_count', + ' expr: count(*)', + '', + ].join('\n'), }); const semanticLayerCompute = { - validateSources: vi.fn(async () => ({ - valid: true, - errors: [], - warnings: ['python validation ran'], - perSourceWarnings: {}, - })), + validateSources: vi.fn(), query: vi.fn(async () => ({ sql: 'select status, count(*) as order_count from public.orders group by status', dialect: 'postgres', @@ -1149,29 +748,6 @@ describe('createLocalProjectMcpContextPorts', () => { }; const ports = createLocalProjectMcpContextPorts(project, { semanticLayerCompute }); - await expect(ports.semanticLayer?.validate({ connectionId: 'warehouse', names: ['orders'] })).resolves.toEqual({ - success: true, - errors: [], - warnings: ['python validation ran'], - }); - expect(semanticLayerCompute.validateSources).toHaveBeenCalledWith({ - sources: [ - { - name: 'orders', - table: 'public.orders', - grain: ['id'], - columns: [ - { name: 'id', type: 'number' }, - { name: 'status', type: 'string' }, - ], - joins: [], - measures: [{ name: 'order_count', expr: 'count(*)' }], - }, - ], - dialect: 'postgres', - recentlyTouched: ['orders'], - }); - await expect( ports.semanticLayer?.query({ connectionId: 'warehouse', @@ -1201,18 +777,23 @@ describe('createLocalProjectMcpContextPorts', () => { driver: 'postgres', url: 'env:DATABASE_URL', }; - const shapeOnlyPorts = createLocalProjectMcpContextPorts(project); - await shapeOnlyPorts.semanticLayer?.writeSource({ + await writeLocalSlSource(project, { connectionId: 'warehouse', sourceName: 'orders', - source: { - name: 'orders', - table: 'public.orders', - grain: ['id'], - columns: [{ name: 'id', type: 'number' }], - joins: [], - measures: [{ name: 'order_count', expr: 'count(*)' }], - }, + yaml: [ + 'name: orders', + 'table: public.orders', + 'grain:', + ' - id', + 'columns:', + ' - name: id', + ' type: number', + 'joins: []', + 'measures:', + ' - name: order_count', + ' expr: count(*)', + '', + ].join('\n'), }); const compute = { validateSources: vi.fn(), @@ -1252,378 +833,4 @@ describe('createLocalProjectMcpContextPorts', () => { }), ); }); - - it('exposes detailed local ingest trigger and status ports when local ingest is enabled', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - project.config.connections.warehouse = { driver: 'postgres' }; - project.config.ingest.adapters = ['fake']; - project.config.ingest.embeddings = { - backend: 'deterministic', - dimensions: 8, - batchSize: 64, - }; - project.config.llm = { - provider: { backend: 'none' }, - models: {}, - }; - const sourceDir = join(project.projectDir, 'upload'); - await mkdir(join(sourceDir, 'orders'), { recursive: true }); - await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\n', 'utf-8'); - - let nextJob = 0; - const agentRunner = new TestAgentRunner(); - const ports = createLocalProjectMcpContextPorts(project, { - localIngest: { - adapters: [new FakeSourceAdapter()], - jobIdFactory: () => `mcp-local-run-${++nextJob}`, - agentRunner, - }, - }); - - const firstTrigger = await ports.ingest?.trigger({ - adapter: 'fake', - connectionId: 'warehouse', - trigger: 'manual_resync', - config: { sourceDir }, - }); - - expect(firstTrigger).toMatchObject({ - runId: expect.any(String), - jobId: 'mcp-local-run-1', - reportId: expect.any(String), - }); - expect(firstTrigger?.runId).not.toBe('mcp-local-run-1'); - - await expect(ports.ingest?.status({ runId: 'mcp-local-run-1' })).resolves.toMatchObject({ - runId: firstTrigger?.runId, - jobId: 'mcp-local-run-1', - reportId: firstTrigger?.reportId, - status: 'done', - stage: 'done', - done: true, - progress: 1, - adapter: 'fake', - connectionId: 'warehouse', - sourceDir: null, - syncId: expect.stringContaining('mcp-local-run-1'), - startedAt: expect.any(String), - completedAt: expect.any(String), - previousRunId: null, - diffSummary: { - added: 1, - modified: 0, - deleted: 0, - unchanged: 0, - }, - rawFileCount: 1, - workUnitCount: 1, - workUnits: [ - { - unitKey: 'fake-orders', - rawFiles: ['orders/orders.json'], - peerFileIndex: [], - dependencyPaths: [], - }, - ], - evictionDeletedRawPaths: [], - errors: [], - }); - - const secondTrigger = await ports.ingest?.trigger({ - adapter: 'fake', - connectionId: 'warehouse', - trigger: 'manual_resync', - config: { sourceDir }, - }); - - expect(secondTrigger).toMatchObject({ - runId: expect.any(String), - jobId: 'mcp-local-run-2', - reportId: expect.any(String), - }); - expect(secondTrigger?.runId).not.toBe('mcp-local-run-2'); - - await expect(ports.ingest?.status({ runId: 'mcp-local-run-2' })).resolves.toMatchObject({ - runId: secondTrigger?.runId, - jobId: 'mcp-local-run-2', - reportId: secondTrigger?.reportId, - status: 'done', - stage: 'done', - done: true, - progress: 1, - adapter: 'fake', - connectionId: 'warehouse', - sourceDir: null, - syncId: expect.stringContaining('mcp-local-run-2'), - startedAt: expect.any(String), - completedAt: expect.any(String), - previousRunId: null, - diffSummary: { - added: 0, - modified: 0, - deleted: 0, - unchanged: 1, - }, - rawFileCount: 0, - workUnitCount: 0, - workUnits: [], - evictionDeletedRawPaths: [], - errors: [], - }); - expect(agentRunner.runLoop).toHaveBeenCalledTimes(1); - }); - - it('passes local ingest pull-config options into runLocalIngest', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - project.config.connections.warehouse = { driver: 'postgres' }; - project.config.ingest.adapters = ['looker']; - const runLocalIngest = vi.fn(async () => ({ - result: { ok: true }, - report: { - id: 'report-1', - runId: 'run-1', - jobId: 'job-1', - sourceKey: 'looker', - connectionId: 'warehouse', - body: { - syncId: 'sync-1', - workUnits: [], - failedWorkUnits: [], - diffSummary: { added: 0, modified: 0, deleted: 0, unchanged: 0 }, - provenanceRows: [], - }, - }, - }) as never); - const ports = createLocalProjectMcpContextPorts(project, { - localIngest: { - adapters: [ - { source: 'looker', skillNames: [], detect: async () => true, chunk: async () => ({ workUnits: [] }) }, - ], - pullConfigOptions: { - looker: { - daemonBaseUrl: 'http://127.0.0.1:61234', - }, - }, - runLocalIngest, - }, - }); - - await expect( - ports.ingest?.trigger({ - adapter: 'looker', - connectionId: 'warehouse', - trigger: 'manual_resync', - config: {}, - }), - ).resolves.toMatchObject({ - runId: 'run-1', - jobId: 'job-1', - reportId: 'report-1', - }); - - expect(runLocalIngest).toHaveBeenCalledWith( - expect.objectContaining({ - pullConfigOptions: { - looker: { - daemonBaseUrl: 'http://127.0.0.1:61234', - }, - }, - }), - ); - }); - - it('triggers fetch-capable local ingest without sourceDir config', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - project.config.connections.warehouse = { - driver: 'postgres', - url: 'postgres://localhost:5432/warehouse', - }; - project.config.ingest.adapters = ['live-database']; - project.config.llm = { - provider: { backend: 'none' }, - models: {}, - }; - const agentRunner = new TestAgentRunner(); - const ports = createLocalProjectMcpContextPorts(project, { - localIngest: { - adapters: [ - { - source: 'live-database', - skillNames: ['live_database_ingest'], - async fetch(_pullConfig, stagedDir) { - await mkdir(join(stagedDir, 'tables'), { recursive: true }); - await writeFile(join(stagedDir, 'connection.json'), '{"connectionId":"warehouse"}\n', 'utf-8'); - await writeFile(join(stagedDir, 'foreign-keys.json'), '{"foreignKeys":[]}\n', 'utf-8'); - await writeFile( - join(stagedDir, 'tables', 'orders.json'), - '{"name":"orders","db":"public","columns":[]}\n', - 'utf-8', - ); - }, - async detect() { - return true; - }, - async chunk() { - return { - workUnits: [ - { - unitKey: 'live-database-public-orders', - rawFiles: ['tables/orders.json'], - dependencyPaths: ['connection.json', 'foreign-keys.json'], - peerFileIndex: [], - }, - ], - }; - }, - }, - ], - jobIdFactory: () => 'local-live-db-mcp', - agentRunner, - }, - }); - - const result = await ports.ingest?.trigger({ - adapter: 'live-database', - connectionId: 'warehouse', - trigger: 'manual_resync', - config: {}, - }); - - expect(result).toMatchObject({ - runId: expect.any(String), - jobId: 'local-live-db-mcp', - reportId: expect.any(String), - }); - expect(result?.runId).not.toBe('local-live-db-mcp'); - await expect(ports.ingest?.status({ runId: 'local-live-db-mcp' })).resolves.toMatchObject({ - runId: result?.runId, - jobId: 'local-live-db-mcp', - reportId: result?.reportId, - adapter: 'live-database', - sourceDir: null, - rawFileCount: 1, - workUnitCount: 1, - }); - expect(agentRunner.runLoop).toHaveBeenCalledTimes(1); - }); - - it('lists and reads only artifacts that belong to a local scan report', async () => { - const project = await initKtxProject({ projectDir: tempDir }); - project.config.connections.warehouse = { - driver: 'postgres', - url: 'env:DATABASE_URL', - }; - project.config.ingest.adapters = ['live-database']; - const ports = createLocalProjectMcpContextPorts(project, { - localScan: { - adapters: [ - { - source: 'live-database', - skillNames: ['live_database_ingest'], - async fetch(_pullConfig, stagedDir) { - await mkdir(join(stagedDir, 'tables'), { recursive: true }); - await writeFile(join(stagedDir, 'connection.json'), '{"connectionId":"warehouse"}\n', 'utf-8'); - await writeFile(join(stagedDir, 'foreign-keys.json'), '{"foreignKeys":[]}\n', 'utf-8'); - await writeFile( - join(stagedDir, 'tables', 'orders.json'), - '{"name":"orders","db":"public","columns":[]}\n', - 'utf-8', - ); - }, - async detect() { - return true; - }, - async chunk() { - return { - workUnits: [ - { - unitKey: 'live-database-public-orders', - rawFiles: ['tables/orders.json'], - dependencyPaths: ['connection.json', 'foreign-keys.json'], - peerFileIndex: [], - }, - ], - }; - }, - }, - ], - jobIdFactory: () => 'local-scan-artifacts', - now: () => new Date('2026-04-29T12:00:00.000Z'), - }, - }); - - const trigger = await ports.scan?.trigger({ - connectionId: 'warehouse', - mode: 'structural', - detectRelationships: false, - dryRun: false, - }); - - expect(trigger?.runId).toBe('local-scan-artifacts'); - const syncId = '2026-04-29-120000-local-scan-artifacts'; - await expect(ports.scan?.listArtifacts?.({ runId: 'local-scan-artifacts' })).resolves.toEqual({ - runId: 'local-scan-artifacts', - artifacts: [ - { - path: `raw-sources/warehouse/live-database/${syncId}/connection.json`, - type: 'raw_source', - size: 29, - }, - { - path: `raw-sources/warehouse/live-database/${syncId}/foreign-keys.json`, - type: 'raw_source', - size: 19, - }, - { - path: `raw-sources/warehouse/live-database/${syncId}/scan-report.json`, - type: 'report', - size: expect.any(Number), - }, - { - path: `raw-sources/warehouse/live-database/${syncId}/tables/orders.json`, - type: 'raw_source', - size: 45, - }, - { - path: 'semantic-layer/warehouse/_schema/public.yaml', - type: 'manifest_shard', - size: expect.any(Number), - }, - ], - }); - - await expect( - ports.scan?.readArtifact?.({ - runId: 'local-scan-artifacts', - path: `raw-sources/warehouse/live-database/${syncId}/tables/orders.json`, - }), - ).resolves.toEqual({ - runId: 'local-scan-artifacts', - path: `raw-sources/warehouse/live-database/${syncId}/tables/orders.json`, - type: 'raw_source', - size: 45, - content: '{"name":"orders","db":"public","columns":[]}\n', - }); - - await expect( - ports.scan?.readArtifact?.({ - runId: 'local-scan-artifacts', - path: 'semantic-layer/warehouse/_schema/public.yaml', - }), - ).resolves.toMatchObject({ - runId: 'local-scan-artifacts', - path: 'semantic-layer/warehouse/_schema/public.yaml', - type: 'manifest_shard', - content: expect.stringContaining('orders:'), - }); - - await expect( - ports.scan?.readArtifact?.({ - runId: 'local-scan-artifacts', - path: 'ktx.yaml', - }), - ).resolves.toBeNull(); - await expect(ports.scan?.listArtifacts?.({ runId: 'missing' })).resolves.toBeNull(); - await expect(readFile(join(project.projectDir, 'ktx.yaml'), 'utf-8')).resolves.not.toContain('project:'); - }); }); diff --git a/packages/context/src/mcp/local-project-ports.ts b/packages/context/src/mcp/local-project-ports.ts index 8088f27a..073b042d 100644 --- a/packages/context/src/mcp/local-project-ports.ts +++ b/packages/context/src/mcp/local-project-ports.ts @@ -1,65 +1,19 @@ -import YAML from 'yaml'; -import { - type KtxSqlQueryExecutorPort, - localConnectionInfoFromConfig, - localConnectionTypeForConfig, -} from '../connections/index.js'; +import { type KtxSqlQueryExecutorPort, localConnectionInfoFromConfig } from '../connections/index.js'; import type { KtxEmbeddingPort } from '../core/index.js'; import type { KtxSemanticLayerComputePort } from '../daemon/index.js'; -import { - createDefaultLocalIngestAdapters, - getLocalIngestStatus, - type IngestReportSnapshot, - ingestReportToMemoryFlowReplay, - type LocalIngestMcpOptions, - runLocalIngest, - runLocalMetabaseIngest, -} from '../ingest/index.js'; import { createLocalKtxEmbeddingProviderFromConfig, KtxIngestEmbeddingPortAdapter } from '../llm/index.js'; import type { KtxLocalProject } from '../project/index.js'; -import { - createKtxEntityDetailsService, - getLocalScanReport, - getLocalScanStatus, - type KtxConnectionDriver, - type KtxScanConnector, - type KtxScanReport, - type LocalScanMcpOptions, - runLocalScan, -} from '../scan/index.js'; +import { createKtxEntityDetailsService, type KtxScanConnector, type LocalScanMcpOptions } from '../scan/index.js'; import { createKtxDiscoverDataService } from '../search/index.js'; import type { SqlAnalysisDialect, SqlAnalysisPort } from '../sql-analysis/index.js'; -import { - compileLocalSlQuery, - createKtxDictionarySearchService, - type LocalSlSourceSearchResult, - type LocalSlSourceSummary, - listLocalSlSources, - searchLocalSlSources, - sourceDefinitionSchema, - sourceOverlaySchema, -} from '../sl/index.js'; -import { readLocalKnowledgePage, searchLocalKnowledgePages, writeLocalKnowledgePage } from '../wiki/local-knowledge.js'; -import type { - KtxConnectionTestResponse, - KtxIngestStatusResponse, - KtxMcpContextPorts, - KtxScanArtifactListResponse, - KtxScanArtifactReadResponse, - KtxScanArtifactSummary, - KtxScanArtifactType, - KtxSqlExecutionResponse, -} from './types.js'; - -const LOCAL_AUTHOR = 'ktx'; -const LOCAL_AUTHOR_EMAIL = 'ktx@example.com'; -const SL_SHAPE_WARNING = 'Local stdio validation checks YAML shape only; Python semantic validation is not configured.'; +import { compileLocalSlQuery, createKtxDictionarySearchService } from '../sl/index.js'; +import { readLocalKnowledgePage, searchLocalKnowledgePages } from '../wiki/local-knowledge.js'; +import type { KtxMcpContextPorts, KtxMcpProgressCallback, KtxSqlExecutionResponse } from './types.js'; interface CreateLocalProjectMcpContextPortsOptions { semanticLayerCompute?: KtxSemanticLayerComputePort; queryExecutor?: KtxSqlQueryExecutorPort; sqlAnalysis?: SqlAnalysisPort; - localIngest?: LocalIngestMcpOptions; localScan?: LocalScanMcpOptions; embeddingService?: KtxEmbeddingPort | null; } @@ -115,284 +69,23 @@ function assertSafeSourceName(sourceName: string): string { return assertSafePathToken('semantic-layer source name', sourceName); } -function normalizeScanDriver(driver: string | undefined): KtxConnectionDriver { - const normalized = (driver ?? '').toLowerCase(); - if ( - normalized === 'postgres' || - normalized === 'postgresql' || - normalized === 'sqlite' || - normalized === 'sqlite3' || - normalized === 'mysql' || - normalized === 'clickhouse' || - normalized === 'sqlserver' || - normalized === 'bigquery' || - normalized === 'snowflake' - ) { - return normalized === 'sqlite3' ? 'sqlite' : normalized; - } - return 'postgres'; -} - async function cleanupConnector(connector: KtxScanConnector | null): Promise { if (connector?.cleanup) { await connector.cleanup(); } } -async function testLocalConnection( - project: KtxLocalProject, - options: CreateLocalProjectMcpContextPortsOptions, - connectionId: string, -): Promise { - const safeConnectionId = assertSafeConnectionId(connectionId); - const connection = project.config.connections[safeConnectionId]; - if (!connection) { - return null; - } - const connectionType = localConnectionTypeForConfig(safeConnectionId, connection); - const createConnector = options.localScan?.createConnector; - if (!createConnector) { - return { - id: safeConnectionId, - connectionType, - ok: true, - tableCount: null, - message: 'Connection is configured; no native scan connector is available for live testing.', - warnings: ['ktx serve was not configured with a local scan connector factory.'], - }; - } - - let connector: KtxScanConnector | null = null; - try { - connector = await createConnector(safeConnectionId); - const snapshot = await connector.introspect( - { - connectionId: safeConnectionId, - driver: normalizeScanDriver(connection.driver), - mode: 'structural', - dryRun: true, - detectRelationships: false, - }, - { runId: `connection-test-${safeConnectionId}` }, - ); - return { - id: safeConnectionId, - connectionType, - ok: true, - tableCount: snapshot.tables.length, - message: 'Connection test passed.', - warnings: [], - }; - } catch (error) { - return { - id: safeConnectionId, - connectionType, - ok: false, - tableCount: null, - message: error instanceof Error ? error.message : String(error), - warnings: [], - }; - } finally { - await cleanupConnector(connector); - } -} - -function scanArtifactType(path: string, report: KtxScanReport): KtxScanArtifactType { - if (path === report.artifactPaths.reportPath) { - return 'report'; - } - if (report.artifactPaths.manifestShards.includes(path)) { - return 'manifest_shard'; - } - if (report.artifactPaths.enrichmentArtifacts.includes(path)) { - return 'enrichment_artifact'; - } - return 'raw_source'; -} - -async function artifactSize(project: KtxLocalProject, path: string): Promise { - try { - const result = await project.fileStore.readFile(path); - return typeof result.size === 'number' ? result.size : undefined; - } catch { - return undefined; - } -} - -async function listArtifactsForReport( - project: KtxLocalProject, - runId: string, - report: KtxScanReport, -): Promise { - const paths = new Set(); - if (report.artifactPaths.rawSourcesDir) { - const listed = await project.fileStore.listFiles(report.artifactPaths.rawSourcesDir); - for (const file of listed.files) { - paths.add(file); - } - } - if (report.artifactPaths.reportPath) { - paths.add(report.artifactPaths.reportPath); - } - for (const path of report.artifactPaths.manifestShards) { - paths.add(path); - } - for (const path of report.artifactPaths.enrichmentArtifacts) { - paths.add(path); - } - - const artifacts: KtxScanArtifactSummary[] = []; - for (const path of [...paths].sort()) { - const size = await artifactSize(project, path); - artifacts.push({ - path, - type: scanArtifactType(path, report), - ...(size === undefined ? {} : { size }), - }); - } - return { runId, artifacts }; -} - -async function readScanArtifact( - project: KtxLocalProject, - runId: string, - path: string, -): Promise { - const report = await getLocalScanReport(project, runId); - if (!report) { - return null; - } - const listed = await listArtifactsForReport(project, runId, report); - const artifact = listed.artifacts.find((candidate) => candidate.path === path); - if (!artifact) { - return null; - } - const result = await project.fileStore.readFile(path); - return { - runId, - path, - type: artifact.type, - ...(typeof result.size === 'number' ? { size: result.size } : {}), - content: result.content, - }; -} - function slPath(connectionId: string, sourceName: string): string { return `semantic-layer/${assertSafeConnectionId(connectionId)}/${assertSafeSourceName(sourceName)}.yaml`; } -function sourceNameFromPath(path: string): string { - return ( - path - .split('/') - .at(-1) - ?.replace(/\.ya?ml$/, '') ?? path - ); -} - -function isRecord(value: unknown): value is Record { - return typeof value === 'object' && value !== null && !Array.isArray(value); -} - -function parseYamlRecord(raw: string): Record { - const parsed = YAML.parse(raw) as unknown; - if (!isRecord(parsed)) { - throw new Error('Semantic-layer source YAML must contain an object'); - } - return parsed; -} - -async function listSlPaths(project: KtxLocalProject, connectionId?: string): Promise { - const root = connectionId ? `semantic-layer/${assertSafeConnectionId(connectionId)}` : 'semantic-layer'; - const listed = await project.fileStore.listFiles(root); - return listed.files.filter((file) => file.endsWith('.yaml') || file.endsWith('.yml')).sort(); -} - -async function loadComputableSources( - project: KtxLocalProject, - connectionId: string, -): Promise[]> { - const paths = await listSlPaths(project, connectionId); - const sources: Record[] = []; - for (const path of paths) { - const raw = await project.fileStore.readFile(path); - const source = parseYamlRecord(raw.content); - if (source.table || source.sql) { - sources.push(source); - } - } - return sources; -} - -function validateSourceRecord(sourceName: string, source: Record): string[] { - const namedSource = { ...source, name: typeof source.name === 'string' ? source.name : sourceName }; - const definition = sourceDefinitionSchema.safeParse(namedSource); - if (definition.success) { - return []; - } - const overlay = sourceOverlaySchema.safeParse(namedSource); - if (overlay.success) { - return []; - } - return definition.error.issues.map((issue) => `${sourceName}: ${issue.path.join('.') || 'source'} ${issue.message}`); -} - -function localIngestSourceDir(config: unknown): string | undefined { - if (!isRecord(config) || config.sourceDir === undefined) { - return undefined; - } - if (typeof config.sourceDir !== 'string' || config.sourceDir.trim().length === 0) { - throw new Error('Local ingest config sourceDir must be a non-empty string when provided'); - } - return config.sourceDir; -} - -function rawFileCountFromIngestReport(report: IngestReportSnapshot): number { - return new Set(report.body.workUnits.flatMap((workUnit) => workUnit.rawFiles)).size; -} - -function hasSlSearchMetadata( - source: LocalSlSourceSummary | LocalSlSourceSearchResult, -): source is LocalSlSourceSearchResult { - return 'score' in source; -} - -function statusFromIngestReport(report: IngestReportSnapshot): KtxIngestStatusResponse { - const failedWorkUnits = report.body.failedWorkUnits; - return { - runId: report.runId, - jobId: report.jobId, - reportId: report.id, - status: failedWorkUnits.length > 0 ? 'error' : 'done', - stage: 'done', - progress: 1, - errors: failedWorkUnits, - done: true, - adapter: report.sourceKey, - connectionId: report.connectionId, - sourceDir: null, - syncId: report.body.syncId, - startedAt: report.createdAt, - completedAt: report.createdAt, - previousRunId: null, - diffSummary: report.body.diffSummary, - workUnitCount: report.body.workUnits.length, - rawFileCount: rawFileCountFromIngestReport(report), - workUnits: report.body.workUnits.map((workUnit) => ({ - unitKey: workUnit.unitKey, - rawFiles: [...workUnit.rawFiles], - peerFileIndex: [], - dependencyPaths: [], - })), - evictionDeletedRawPaths: [...report.body.evictionInputs], - }; -} - async function executeValidatedReadOnlySql( project: KtxLocalProject, options: CreateLocalProjectMcpContextPortsOptions, input: { connectionId: string; sql: string; maxRows: number }, + onProgress?: KtxMcpProgressCallback, ): Promise { + await onProgress?.({ progress: 0, message: 'Validating SQL' }); const connectionId = assertSafeConnectionId(input.connectionId); const connection = project.config.connections[connectionId]; if (!connection) { @@ -416,6 +109,7 @@ async function executeValidatedReadOnlySql( if (!connector.capabilities.readOnlySql || !connector.executeReadOnly) { throw new Error(`Connection "${connectionId}" does not support read-only SQL execution.`); } + await onProgress?.({ progress: 0.3, message: 'Executing' }); const result = await connector.executeReadOnly( { connectionId, @@ -424,12 +118,14 @@ async function executeValidatedReadOnlySql( }, { runId: 'mcp-sql-execution' }, ); - return { + const response = { headers: result.headers, ...(result.headerTypes ? { headerTypes: result.headerTypes } : {}), rows: result.rows, rowCount: result.rowCount ?? result.rows.length, }; + await onProgress?.({ progress: 1, message: `Fetched ${response.rowCount} rows` }); + return response; } finally { await cleanupConnector(connector); } @@ -453,9 +149,6 @@ export function createLocalProjectMcpContextPorts( ) .sort((a, b) => a.id.localeCompare(b.id)); }, - async test(input) { - return testLocalConnection(project, options, input.connectionId); - }, }, knowledge: { async search(input) { @@ -495,58 +188,8 @@ export function createLocalProjectMcpContextPorts( } : null; }, - async write(input) { - const existing = await readLocalKnowledgePage(project, { - key: input.key, - userId: input.userId, - }); - await writeLocalKnowledgePage(project, { - key: input.key, - scope: 'GLOBAL', - userId: input.userId, - summary: input.summary, - content: input.content, - tags: input.tags, - refs: input.refs, - slRefs: input.slRefs, - source: input.source, - intent: input.intent, - tables: input.tables, - representativeSql: input.representativeSql, - usage: input.usage, - fingerprints: input.fingerprints, - }); - return { success: true, key: input.key, action: existing ? 'updated' : 'created' }; - }, }, semanticLayer: { - async listSources(input) { - const listed: Array = input.query - ? await searchLocalSlSources(project, { - connectionId: input.connectionId, - query: input.query, - embeddingService, - }) - : await listLocalSlSources(project, { connectionId: input.connectionId }); - const sources = listed.map((source) => ({ - connectionId: source.connectionId, - connectionName: source.connectionId, - name: source.name, - description: source.description, - columnCount: source.columnCount, - measureCount: source.measureCount, - joinCount: source.joinCount, - ...(hasSlSearchMetadata(source) && source.frequencyTier ? { frequencyTier: source.frequencyTier } : {}), - ...(hasSlSearchMetadata(source) && source.snippet ? { snippet: source.snippet } : {}), - ...(hasSlSearchMetadata(source) ? { score: source.score } : {}), - ...(hasSlSearchMetadata(source) && source.matchReasons ? { matchReasons: source.matchReasons } : {}), - ...(hasSlSearchMetadata(source) && source.dictionaryMatches - ? { dictionaryMatches: source.dictionaryMatches } - : {}), - ...(hasSlSearchMetadata(source) && source.lanes ? { lanes: source.lanes } : {}), - })); - return { sources, totalSources: sources.length }; - }, async readSource(input) { const path = slPath(input.connectionId, input.sourceName); try { @@ -556,71 +199,9 @@ export function createLocalProjectMcpContextPorts( return null; } }, - async writeSource(input) { - const path = slPath(input.connectionId, input.sourceName); - if (input.delete) { - const deleted = await project.fileStore.deleteFile( - path, - LOCAL_AUTHOR, - LOCAL_AUTHOR_EMAIL, - `Remove semantic-layer source: ${input.sourceName}`, - ); - return { success: Boolean(deleted), sourceName: input.sourceName }; - } - - const yaml = - input.yaml ?? YAML.stringify({ ...input.source, name: input.sourceName }, { indent: 2, lineWidth: 0, version: '1.1' }); - parseYamlRecord(yaml); - await project.fileStore.writeFile( - path, - `${yaml.trimEnd()}\n`, - LOCAL_AUTHOR, - LOCAL_AUTHOR_EMAIL, - `Update semantic-layer source: ${input.sourceName}`, - ); - return { success: true, sourceName: input.sourceName, yaml: `${yaml.trimEnd()}\n` }; - }, - async validate(input) { - if (options.semanticLayerCompute) { - const connectionId = assertSafeConnectionId(input.connectionId); - const result = await options.semanticLayerCompute.validateSources({ - sources: await loadComputableSources(project, connectionId), - dialect: dialectForDriver(project.config.connections[connectionId]?.driver), - recentlyTouched: input.names, - }); - return { - success: result.valid, - errors: result.errors, - warnings: result.warnings, - }; - } - - const names = new Set(input.names ?? []); - const paths = await listSlPaths(project, input.connectionId); - const errors: string[] = []; - for (const path of paths) { - const sourceName = sourceNameFromPath(path); - if (names.size > 0 && !names.has(sourceName)) { - continue; - } - try { - const raw = await project.fileStore.readFile(path); - errors.push(...validateSourceRecord(sourceName, parseYamlRecord(raw.content))); - } catch (error) { - errors.push(`${sourceName}: ${error instanceof Error ? error.message : String(error)}`); - } - } - return { - success: errors.length === 0, - errors, - warnings: [SL_SHAPE_WARNING], - }; - }, - async query(input) { + async query(input, executionOptions) { if (!options.semanticLayerCompute) { - throw new Error( - 'sl_query requires a semantic-layer query adapter. Local stdio MCP exposes file-backed SL CRUD only.', - ); + throw new Error('sl_query requires a semantic-layer query adapter.'); } return compileLocalSlQuery(project, { connectionId: input.connectionId, @@ -629,6 +210,7 @@ export function createLocalProjectMcpContextPorts( execute: Boolean(options.queryExecutor), maxRows: input.query.limit, queryExecutor: options.queryExecutor, + onProgress: executionOptions?.onProgress, }); }, }, @@ -651,114 +233,8 @@ export function createLocalProjectMcpContextPorts( if (options.sqlAnalysis && options.localScan?.createConnector) { ports.sqlExecution = { - async execute(input) { - return executeValidatedReadOnlySql(project, options, input); - }, - }; - } - - if (options.localIngest) { - ports.ingest = { - async trigger(input) { - const sourceDir = localIngestSourceDir(input.config); - if (input.adapter === 'metabase' && !sourceDir) { - const result = await (options.localIngest?.runLocalMetabaseIngest ?? runLocalMetabaseIngest)({ - project, - adapters: options.localIngest?.adapters ?? createDefaultLocalIngestAdapters(project), - metabaseConnectionId: input.connectionId, - trigger: input.trigger, - jobIdFactory: options.localIngest?.jobIdFactory, - pullConfigOptions: options.localIngest?.pullConfigOptions, - agentRunner: options.localIngest?.agentRunner, - llmProvider: options.localIngest?.llmProvider, - memoryModel: options.localIngest?.memoryModel, - semanticLayerCompute: options.localIngest?.semanticLayerCompute ?? options.semanticLayerCompute, - queryExecutor: options.localIngest?.queryExecutor ?? options.queryExecutor, - logger: options.localIngest?.logger, - }); - return { - runId: `metabase-fanout:${result.metabaseConnectionId}`, - jobId: undefined, - reportId: undefined, - fanout: { - status: result.status, - children: result.children.map((child) => ({ - runId: child.report.runId, - jobId: child.report.jobId, - reportId: child.report.id, - targetConnectionId: child.targetConnectionId, - metabaseDatabaseId: child.metabaseDatabaseId, - })), - }, - }; - } - - const executeLocalIngest = options.localIngest?.runLocalIngest ?? runLocalIngest; - const result = await executeLocalIngest({ - project, - adapters: options.localIngest?.adapters ?? createDefaultLocalIngestAdapters(project), - adapter: input.adapter, - connectionId: input.connectionId, - sourceDir, - pullConfigOptions: options.localIngest?.pullConfigOptions, - trigger: input.trigger, - jobId: options.localIngest?.jobIdFactory?.(), - agentRunner: options.localIngest?.agentRunner, - llmProvider: options.localIngest?.llmProvider, - memoryModel: options.localIngest?.memoryModel, - semanticLayerCompute: options.localIngest?.semanticLayerCompute ?? options.semanticLayerCompute, - queryExecutor: options.localIngest?.queryExecutor ?? options.queryExecutor, - logger: options.localIngest?.logger, - }); - return { - runId: result.report.runId, - jobId: result.report.jobId, - reportId: result.report.id, - }; - }, - async status(input) { - const report = await getLocalIngestStatus(project, input.runId); - return report ? statusFromIngestReport(report) : null; - }, - async report(input) { - return getLocalIngestStatus(project, input.runId); - }, - async replay(input) { - const report = await getLocalIngestStatus(project, input.runId); - return report ? ingestReportToMemoryFlowReplay(report) : null; - }, - }; - } - - if (options.localScan) { - ports.scan = { - async trigger(input) { - return runLocalScan({ - project, - connectionId: input.connectionId, - mode: input.mode, - detectRelationships: input.detectRelationships, - dryRun: input.dryRun, - trigger: 'mcp', - adapters: options.localScan?.adapters, - databaseIntrospectionUrl: options.localScan?.databaseIntrospectionUrl, - createConnector: options.localScan?.createConnector, - jobId: options.localScan?.jobIdFactory?.(), - now: options.localScan?.now, - }); - }, - async status(input) { - return getLocalScanStatus(project, input.runId); - }, - async report(input) { - return getLocalScanReport(project, input.runId); - }, - async listArtifacts(input) { - const report = await getLocalScanReport(project, input.runId); - return report ? listArtifactsForReport(project, input.runId, report) : null; - }, - async readArtifact(input) { - return readScanArtifact(project, input.runId, input.path); + async execute(input, executionOptions) { + return executeValidatedReadOnlySql(project, options, input, executionOptions?.onProgress); }, }; } diff --git a/packages/context/src/mcp/server.test.ts b/packages/context/src/mcp/server.test.ts index abf678bb..f02b4adb 100644 --- a/packages/context/src/mcp/server.test.ts +++ b/packages/context/src/mcp/server.test.ts @@ -1,28 +1,40 @@ import { access, mkdtemp, readFile, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; +import { Client } from '@modelcontextprotocol/sdk/client/index.js'; +import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js'; import { describe, expect, it, vi } from 'vitest'; -import { createLocalProjectMemoryCapture } from '../memory/index.js'; +import { + createLocalProjectMemoryIngest, + detectCaptureSignals, + type MemoryAgentInput, +} from '../memory/index.js'; import { initKtxProject } from '../project/index.js'; -import { createKtxMcpServer } from './server.js'; +import { jsonToolResult } from './context-tools.js'; +import { createDefaultKtxMcpServer, createKtxMcpServer } from './server.js'; import type { KtxDiscoverDataMcpPort, KtxDictionarySearchMcpPort, KtxEntityDetailsMcpPort, - KtxIngestMcpPort, KtxKnowledgeMcpPort, KtxMcpContextPorts, - KtxScanMcpPort, + KtxMcpToolHandlerContext, KtxSemanticLayerMcpPort, KtxSqlExecutionMcpPort, KtxSqlExecutionResponse, - MemoryCapturePort, + MemoryIngestPort, } from './types.js'; type RegisteredTool = { name: string; - config: { title?: string; description?: string; inputSchema: unknown }; - handler: (input: Record) => Promise; + config: { + title?: string; + description?: string; + inputSchema: unknown; + outputSchema?: unknown; + annotations?: Record; + }; + handler: (input: Record, context?: KtxMcpToolHandlerContext) => Promise; }; function makeFakeServer() { @@ -45,7 +57,155 @@ function getTool(tools: RegisteredTool[], name: string): RegisteredTool { return found; } +const retainedToolNames = [ + 'connection_list', + 'dictionary_search', + 'discover_data', + 'entity_details', + 'memory_ingest', + 'memory_ingest_status', + 'sl_query', + 'sl_read_source', + 'sql_execution', + 'wiki_read', + 'wiki_search', +] as const; + +function makeAllContextTools(): KtxMcpContextPorts { + return { + connections: { + list: vi.fn().mockResolvedValue([{ id: 'warehouse', name: 'Warehouse', connectionType: 'POSTGRES' }]), + }, + knowledge: { + search: vi.fn().mockResolvedValue({ results: [], totalFound: 0 }), + read: vi.fn().mockResolvedValue({ + key: 'revenue', + summary: 'Paid order value', + content: '# Revenue', + scope: 'GLOBAL', + tags: ['finance'], + refs: [], + slRefs: ['orders'], + }), + }, + semanticLayer: { + readSource: vi.fn().mockResolvedValue({ + sourceName: 'orders', + yaml: 'name: orders\n', + }), + query: vi.fn().mockResolvedValue({ + sql: 'select 1', + headers: ['count'], + rows: [[1]], + totalRows: 1, + plan: { sources: ['orders'] }, + }), + }, + entityDetails: { + read: vi.fn().mockResolvedValue({ results: [] }), + }, + dictionarySearch: { + search: vi.fn().mockResolvedValue({ searched: [], results: [] }), + }, + discover: { + search: vi.fn().mockResolvedValue([]), + }, + sqlExecution: { + execute: vi.fn().mockResolvedValue({ + headers: ['count'], + headerTypes: ['integer'], + rows: [[1]], + rowCount: 1, + }), + }, + memoryIngest: { + ingest: vi.fn().mockResolvedValue({ runId: 'run-1' }), + status: vi.fn().mockResolvedValue({ + runId: 'run-1', + status: 'done', + stage: 'done', + done: true, + captured: { wiki: [], sl: [], xrefs: [] }, + error: null, + commitHash: null, + skillsLoaded: [], + signalDetected: false, + }), + }, + }; +} + +async function listToolsThroughSdk(contextTools: KtxMcpContextPorts) { + const server = createDefaultKtxMcpServer({ + name: 'ktx-test', + version: '0.0.0-test', + userContext: { userId: 'mcp-user' }, + contextTools, + }); + const client = new Client({ name: 'ktx-test-client', version: '0.0.0-test' }); + const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); + + await Promise.all([server.connect(serverTransport), client.connect(clientTransport)]); + try { + return await client.listTools(); + } finally { + await client.close(); + await server.close(); + } +} + describe('createKtxMcpServer', () => { + it('registers annotations and output schemas for every retained tool', async () => { + const fake = makeFakeServer(); + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'mcp-user' }, + contextTools: makeAllContextTools(), + }); + + expect(fake.tools.map((tool) => tool.name).sort()).toEqual([...retainedToolNames].sort()); + + const expectedAnnotations: Record> = { + connection_list: { title: 'Connection List', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + discover_data: { title: 'Discover Data', readOnlyHint: true, openWorldHint: false }, + wiki_search: { title: 'Wiki Search', readOnlyHint: true, openWorldHint: false }, + wiki_read: { title: 'Wiki Read', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + entity_details: { title: 'Entity Details', readOnlyHint: true, idempotentHint: true, openWorldHint: false }, + dictionary_search: { title: 'Dictionary Search', readOnlyHint: true, openWorldHint: false }, + sl_read_source: { + title: 'Semantic Layer Read Source', + readOnlyHint: true, + idempotentHint: true, + openWorldHint: false, + }, + sl_query: { title: 'Semantic Layer Query', readOnlyHint: true, openWorldHint: false }, + sql_execution: { title: 'SQL Execution', readOnlyHint: true, openWorldHint: false }, + memory_ingest: { title: 'Memory Ingest', destructiveHint: true, openWorldHint: false }, + memory_ingest_status: { title: 'Memory Ingest Status', readOnlyHint: true, openWorldHint: false }, + }; + + for (const toolName of retainedToolNames) { + const tool = getTool(fake.tools, toolName); + expect(tool.config.title).toBe(expectedAnnotations[toolName]?.title); + expect(tool.config.annotations).toEqual(expectedAnnotations[toolName]); + expect(tool.config.outputSchema).toBeDefined(); + const inputShape = tool.config.inputSchema as Record; + for (const inputSchema of Object.values(inputShape)) { + expect(inputSchema.description).toEqual(expect.any(String)); + } + } + }); + + it('exposes annotations and output schemas through the SDK tools/list response', async () => { + const result = await listToolsThroughSdk(makeAllContextTools()); + const toolNames = result.tools.map((tool) => tool.name).sort(); + expect(toolNames).toEqual([...retainedToolNames].sort()); + + await expect(`${JSON.stringify(result.tools, null, 2)}\n`).toMatchFileSnapshot( + '__snapshots__/mcp-tools-list.json', + ); + }); + it('registers context tools without memory capture tools when memory capture is omitted', async () => { const fake = makeFakeServer(); @@ -119,11 +279,14 @@ describe('createKtxMcpServer', () => { rowCount: 1, }, }); - expect(sqlExecution.execute).toHaveBeenCalledWith({ - connectionId: 'warehouse', - sql: 'select status, count(*) from public.orders group by status', - maxRows: 50, - }); + expect(sqlExecution.execute).toHaveBeenCalledWith( + { + connectionId: 'warehouse', + sql: 'select status, count(*) from public.orders group by status', + maxRows: 50, + }, + undefined, + ); }); it('registers entity_details when the host provides an entity-details port', async () => { @@ -256,6 +419,162 @@ describe('createKtxMcpServer', () => { }); }); + it('sl_query normalizes order_by from cube-style {id, desc} and bare strings to {field, direction}', async () => { + const fake = makeFakeServer(); + const semanticLayer: KtxSemanticLayerMcpPort = { + readSource: vi.fn(), + query: vi.fn().mockResolvedValue({ + sql: '', + headers: [], + rows: [], + totalRows: 0, + }), + }; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { semanticLayer }, + }); + + await getTool(fake.tools, 'sl_query').handler({ + connectionId: 'warehouse', + measures: ['orders.count'], + order_by: [ + { field: 'orders.total', direction: 'desc' }, + { id: 'orders.quarter_label', desc: false }, + { id: 'orders.created_at', desc: true }, + 'orders.segment', + ], + }); + + expect(semanticLayer.query).toHaveBeenCalledWith( + { + connectionId: 'warehouse', + query: expect.objectContaining({ + order_by: [ + { field: 'orders.total', direction: 'desc' }, + { field: 'orders.quarter_label', direction: 'asc' }, + { field: 'orders.created_at', direction: 'desc' }, + { field: 'orders.segment', direction: 'asc' }, + ], + }), + }, + undefined, + ); + }); + + it('sl_query normalizes cube-style dimensions to field dimensions', async () => { + const fake = makeFakeServer(); + const semanticLayer = makeAllContextTools().semanticLayer!; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { semanticLayer }, + }); + + await getTool(fake.tools, 'sl_query').handler({ + connectionId: 'warehouse', + measures: ['orders.count'], + dimensions: [{ dimension: 'orders.created_at', granularity: 'month' }, 'orders.status'], + }); + + expect(semanticLayer.query).toHaveBeenCalledWith( + { + connectionId: 'warehouse', + query: expect.objectContaining({ + dimensions: [{ field: 'orders.created_at', granularity: 'month' }, { field: 'orders.status' }], + }), + }, + undefined, + ); + }); + + it('entity_details normalizes sql-style schema table refs', async () => { + const fake = makeFakeServer(); + const entityDetails = makeAllContextTools().entityDetails!; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { entityDetails }, + }); + + await getTool(fake.tools, 'entity_details').handler({ + connectionId: 'warehouse', + entities: [{ table: { schema: 'public', table: 'orders' }, columns: ['id'] }], + }); + + expect(entityDetails.read).toHaveBeenCalledWith({ + connectionId: 'warehouse', + entities: [{ table: { catalog: null, db: 'public', name: 'orders' }, columns: ['id'] }], + }); + }); + + it('wraps handler exceptions in-band for non-sql tools', async () => { + const fake = makeFakeServer(); + const knowledge: KtxKnowledgeMcpPort = { + search: vi.fn().mockRejectedValue(new Error('wiki index unavailable')), + read: vi.fn(), + }; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { knowledge }, + }); + + await expect(getTool(fake.tools, 'wiki_search').handler({ query: 'revenue' })).resolves.toEqual({ + content: [{ type: 'text', text: 'wiki index unavailable' }], + isError: true, + }); + }); + + it('wires sql_execution progress to MCP notifications when a progress token is present', async () => { + const fake = makeFakeServer(); + const notifications: unknown[] = []; + const sqlExecution: KtxSqlExecutionMcpPort = { + execute: vi.fn().mockImplementation(async (_input, options) => { + await options?.onProgress?.({ progress: 0, message: 'Validating SQL' }); + await options?.onProgress?.({ progress: 0.3, message: 'Executing' }); + await options?.onProgress?.({ progress: 1, message: 'Fetched 1 rows' }); + return { headers: ['count'], rows: [[1]], rowCount: 1 }; + }), + }; + + createKtxMcpServer({ + server: fake.server, + userContext: { userId: 'local-user' }, + contextTools: { sqlExecution }, + }); + + await getTool(fake.tools, 'sql_execution').handler( + { connectionId: 'warehouse', sql: 'select 1' }, + { + _meta: { progressToken: 'progress-1' }, + sendNotification: async (notification) => { + notifications.push(notification); + }, + }, + ); + + expect(notifications).toEqual([ + { + method: 'notifications/progress', + params: { progressToken: 'progress-1', progress: 0, message: 'Validating SQL' }, + }, + { + method: 'notifications/progress', + params: { progressToken: 'progress-1', progress: 0.3, message: 'Executing' }, + }, + { + method: 'notifications/progress', + params: { progressToken: 'progress-1', progress: 1, message: 'Fetched 1 rows' }, + }, + ]); + }); + it('registers discover_data when the host provides a discover port', async () => { const fake = makeFakeServer(); const discover: KtxDiscoverDataMcpPort = { @@ -288,14 +607,16 @@ describe('createKtxMcpServer', () => { limit: 5, }), ).resolves.toMatchObject({ - structuredContent: [ - { - kind: 'table', - id: 'public.orders', - connectionId: 'warehouse', - tableRef: { catalog: null, db: 'public', name: 'orders' }, - }, - ], + structuredContent: { + refs: [ + { + kind: 'table', + id: 'public.orders', + connectionId: 'warehouse', + tableRef: { catalog: null, db: 'public', name: 'orders' }, + }, + ], + }, }); expect(discover.search).toHaveBeenCalledWith({ query: 'orders', @@ -305,11 +626,15 @@ describe('createKtxMcpServer', () => { }); }); - it('registers memory capture tools without host app dependencies', async () => { + it('registers memory ingest tools through the context tool surface', async () => { const fake = makeFakeServer(); - const capture: MemoryCapturePort = { - capture: vi.fn().mockResolvedValue({ runId: 'run-1' }), - status: vi.fn().mockResolvedValue({ + let receivedInput: MemoryAgentInput | undefined; + const ingest: MemoryIngestPort = { + ingest: vi.fn().mockImplementation(async (input) => { + receivedInput = input; + return { runId: 'run-1' }; + }), + status: vi.fn().mockResolvedValue({ runId: 'run-1', status: 'done', stage: 'done', @@ -324,33 +649,51 @@ describe('createKtxMcpServer', () => { createKtxMcpServer({ server: fake.server, - memoryCapture: capture, userContext: { userId: 'mcp-user' }, + contextTools: { memoryIngest: ingest }, }); - expect(fake.tools.map((tool) => tool.name).sort()).toEqual(['memory_capture', 'memory_capture_status']); + expect(fake.tools.map((tool) => tool.name).sort()).toEqual(['memory_ingest', 'memory_ingest_status']); - const memoryCapture = getTool(fake.tools, 'memory_capture'); + const content = [ + 'view: orders {', + ' sql_table_name: public.orders ;;', + ' measure: gross_revenue {', + ' type: sum', + ' sql: ${TABLE}.gross_revenue_cents ;;', + ' }', + '}', + ].join('\n'); + const memoryIngest = getTool(fake.tools, 'memory_ingest'); await expect( - memoryCapture.handler({ - userMessage: 'Revenue means paid order value.', - assistantMessage: 'Captured.', + memoryIngest.handler({ + content, connectionId: '00000000-0000-4000-8000-000000000001', }), ).resolves.toEqual({ content: [{ type: 'text', text: JSON.stringify({ runId: 'run-1' }, null, 2) }], structuredContent: { runId: 'run-1' }, }); - expect(capture.capture).toHaveBeenCalledWith({ + expect(ingest.ingest).toHaveBeenCalledWith({ userId: 'mcp-user', chatId: expect.stringMatching(/^mcp-/), - userMessage: 'Revenue means paid order value.', - assistantMessage: 'Captured.', + userMessage: 'Ingest external knowledge into KTX memory.', + assistantMessage: content, connectionId: '00000000-0000-4000-8000-000000000001', sourceType: 'external_ingest', }); - const memoryStatus = getTool(fake.tools, 'memory_capture_status'); + const cliEquivalentInput: MemoryAgentInput = { + userId: 'mcp-user', + chatId: 'cli-text-ingest-test-1', + userMessage: 'Ingest external text artifact "orders lookml" into KTX memory.', + assistantMessage: content, + connectionId: '00000000-0000-4000-8000-000000000001', + sourceType: 'external_ingest', + }; + expect(detectCaptureSignals(receivedInput!)).toEqual(detectCaptureSignals(cliEquivalentInput)); + + const memoryStatus = getTool(fake.tools, 'memory_ingest_status'); await expect(memoryStatus.handler({ runId: 'run-1' })).resolves.toEqual({ content: [ { @@ -386,38 +729,38 @@ describe('createKtxMcpServer', () => { }); }); - it('returns an MCP error payload for missing run ids', async () => { + it('returns an in-band error when a memory ingest run is missing', async () => { const fake = makeFakeServer(); - const capture: MemoryCapturePort = { - capture: vi.fn(), - status: vi.fn().mockResolvedValue(null), + const ingest: MemoryIngestPort = { + ingest: vi.fn(), + status: vi.fn().mockResolvedValue(null), }; createKtxMcpServer({ server: fake.server, - memoryCapture: capture, userContext: { userId: 'mcp-user' }, + contextTools: { memoryIngest: ingest }, }); - const memoryStatus = getTool(fake.tools, 'memory_capture_status'); - await expect(memoryStatus.handler({ runId: 'missing' })).resolves.toEqual({ - content: [{ type: 'text', text: 'Memory capture run "missing" was not found.' }], + const memoryStatus = getTool(fake.tools, 'memory_ingest_status'); + await expect(memoryStatus.handler({ runId: 'missing-run' })).resolves.toEqual({ + content: [{ type: 'text', text: 'Memory ingest run "missing-run" was not found.' }], isError: true, }); }); - it('runs MCP memory_capture against a local project memory port', async () => { - const tempDir = await mkdtemp(join(tmpdir(), 'ktx-mcp-local-memory-')); - try { - const project = await initKtxProject({ projectDir: tempDir }); - const agentRunner = { - runLoop: async ({ - toolSet, - }: { - toolSet: Record Promise }>; - }) => { - await toolSet.load_skill.execute({ name: 'wiki_capture' }); - await toolSet.wiki_write.execute( + it('runs MCP memory_ingest against a local project memory port', async () => { + const tempDir = await mkdtemp(join(tmpdir(), 'ktx-mcp-local-memory-')); + try { + const project = await initKtxProject({ projectDir: tempDir }); + const agentRunner = { + runLoop: async ({ + toolSet, + }: { + toolSet: Record Promise }>; + }) => { + await toolSet.load_skill.execute({ name: 'wiki_capture' }); + await toolSet.wiki_write.execute( { key: 'arr', summary: 'ARR definition', @@ -428,29 +771,38 @@ describe('createKtxMcpServer', () => { return { stopReason: 'natural' as const }; }, }; - const memoryCapture = createLocalProjectMemoryCapture(project, { + const memoryIngest = createLocalProjectMemoryIngest(project, { agentRunner: agentRunner as never, runIdFactory: () => 'memory-run-mcp', }); + const ingestSpy = vi.spyOn(memoryIngest, 'ingest'); const fake = makeFakeServer(); createKtxMcpServer({ server: fake.server, - memoryCapture, - userContext: { userId: 'mcp-user' }, + userContext: { userId: 'local' }, + contextTools: { memoryIngest }, }); - const capture = await getTool(fake.tools, 'memory_capture').handler({ - userMessage: 'define ARR as annual recurring revenue', - assistantMessage: 'Captured.', + const capture = await getTool(fake.tools, 'memory_ingest').handler({ + content: 'Revenue means paid order value.', + connectionId: 'warehouse', }); expect(capture).toMatchObject({ structuredContent: { runId: 'memory-run-mcp' }, }); - await memoryCapture.waitForRun('memory-run-mcp'); + await memoryIngest.waitForRun('memory-run-mcp'); + expect(ingestSpy).toHaveBeenCalledWith({ + userId: 'local', + chatId: expect.stringMatching(/^mcp-/), + userMessage: 'Ingest external knowledge into KTX memory.', + assistantMessage: 'Revenue means paid order value.', + connectionId: 'warehouse', + sourceType: 'external_ingest', + }); await expect( - getTool(fake.tools, 'memory_capture_status').handler({ runId: 'memory-run-mcp' }), + getTool(fake.tools, 'memory_ingest_status').handler({ runId: 'memory-run-mcp' }), ).resolves.toMatchObject({ structuredContent: { runId: 'memory-run-mcp', @@ -471,10 +823,6 @@ describe('createKtxMcpServer', () => { it('registers KTX context MCP tools when context ports are supplied', async () => { const fake = makeFakeServer(); - const capture: MemoryCapturePort = { - capture: vi.fn().mockResolvedValue({ runId: 'run-1' }), - status: vi.fn().mockResolvedValue(null), - }; const contextTools: KtxMcpContextPorts = { connections: { list: vi.fn().mockResolvedValue([ @@ -484,14 +832,6 @@ describe('createKtxMcpServer', () => { connectionType: 'POSTGRES', }, ]), - test: vi.fn().mockResolvedValue({ - id: 'warehouse', - connectionType: 'postgres', - ok: true, - tableCount: 2, - message: 'Connection test passed.', - warnings: [], - }), }, knowledge: { search: vi.fn().mockResolvedValue({ @@ -516,42 +856,12 @@ describe('createKtxMcpServer', () => { refs: [], slRefs: ['orders'], }), - write: vi.fn().mockResolvedValue({ - success: true, - key: 'revenue', - action: 'updated', - }), }, semanticLayer: { - listSources: vi.fn().mockResolvedValue({ - sources: [ - { - connectionId: '00000000-0000-4000-8000-000000000001', - connectionName: 'Warehouse', - name: 'orders', - description: 'Order facts', - columnCount: 2, - measureCount: 1, - joinCount: 0, - }, - ], - totalSources: 1, - }), readSource: vi.fn().mockResolvedValue({ sourceName: 'orders', yaml: 'name: orders\n', }), - writeSource: vi.fn().mockResolvedValue({ - success: true, - sourceName: 'orders', - yaml: 'name: orders\n', - commitHash: 'abc123', - }), - validate: vi.fn().mockResolvedValue({ - success: true, - errors: [], - warnings: [], - }), query: vi.fn().mockResolvedValue({ sql: 'select 1', headers: ['count'], @@ -560,221 +870,50 @@ describe('createKtxMcpServer', () => { plan: { sources: ['orders'] }, }), }, - ingest: { - trigger: vi.fn().mockResolvedValue({ - runId: 'run-42', - jobId: 'job-42', - reportId: 'report-42', - }), - status: vi.fn().mockResolvedValue({ - runId: 'run-42', - jobId: 'job-42', - reportId: 'report-42', - status: 'done', - stage: 'done', - progress: 1, - done: true, - adapter: 'fake', - connectionId: 'warehouse', - sourceDir: '/tmp/upload', - syncId: '2026-04-27-120000-run-42', - startedAt: '2026-04-27T12:00:00.000Z', - completedAt: '2026-04-27T12:00:01.000Z', - previousRunId: 'run-41', - diffSummary: { - added: 0, - modified: 1, - deleted: 0, - unchanged: 3, - }, - rawFileCount: 4, - workUnitCount: 1, - workUnits: [ - { - unitKey: 'fake-orders', - rawFiles: ['orders/orders.json'], - peerFileIndex: [], - dependencyPaths: [], - }, - ], - evictionDeletedRawPaths: [], - errors: [], - }), - report: vi.fn>().mockResolvedValue({ - id: 'report-42', - runId: 'run-42', - jobId: 'job-42', - connectionId: 'warehouse', - sourceKey: 'fake', - createdAt: '2026-04-27T12:00:01.000Z', - body: { - syncId: '2026-04-27-120000-run-42', - diffSummary: { added: 0, modified: 1, deleted: 0, unchanged: 3 }, - commitSha: null, - workUnits: [], - failedWorkUnits: [], - reconciliationSkipped: false, - conflictsResolved: [], - evictionsApplied: [], - unmappedFallbacks: [], - evictionInputs: [], - unresolvedCards: [], - supersededBy: null, - overrideOf: null, - provenanceRows: [], - toolTranscripts: [], - }, - }), - replay: vi.fn>().mockResolvedValue({ - runId: 'run-42', - reportId: 'report-42', - reportPath: 'report-42', - connectionId: 'warehouse', - adapter: 'fake', - status: 'done', - sourceDir: null, - syncId: '2026-04-27-120000-run-42', - errors: [], - events: [{ type: 'report_created', runId: 'run-42', reportPath: 'report-42' }], - plannedWorkUnits: [], - details: { actions: [], provenance: [], transcripts: [] }, + entityDetails: { + read: vi.fn().mockResolvedValue({ results: [] }), + }, + dictionarySearch: { + search: vi.fn().mockResolvedValue({ + searched: [], + results: [], }), }, - scan: { - trigger: vi.fn().mockResolvedValue({ - runId: 'scan-run-1', - status: 'done', - done: true, - connectionId: 'warehouse', - mode: 'structural', - dryRun: false, - syncId: 'sync-1', - report: { - connectionId: 'warehouse', - driver: 'postgres', - syncId: 'sync-1', - runId: 'scan-run-1', - trigger: 'mcp', - mode: 'structural', - dryRun: false, - artifactPaths: { - rawSourcesDir: 'raw-sources/warehouse/live-database/sync-1', - reportPath: 'raw-sources/warehouse/live-database/sync-1/scan-report.json', - manifestShards: [], - enrichmentArtifacts: [], - }, - diffSummary: { - tablesAdded: 1, - tablesModified: 0, - tablesDeleted: 0, - tablesUnchanged: 0, - columnsAdded: 0, - columnsModified: 0, - columnsDeleted: 0, - }, - manifestShardsWritten: 0, - structuralSyncStats: { - tablesCreated: 0, - tablesUpdated: 0, - tablesDeleted: 0, - columnsCreated: 0, - columnsUpdated: 0, - columnsDeleted: 0, - }, - enrichment: { - dataDictionary: 'skipped', - tableDescriptions: 'skipped', - columnDescriptions: 'skipped', - embeddings: 'skipped', - deterministicRelationships: 'skipped', - llmRelationshipValidation: 'skipped', - statisticalValidation: 'skipped', - }, - capabilityGaps: [], - warnings: [], - relationships: { accepted: 0, review: 0, rejected: 0, skipped: 0 }, - enrichmentState: { - resumedStages: [], - completedStages: [], - failedStages: [], - }, - createdAt: '2026-04-29T09:00:00.000Z', - }, - }), - status: vi.fn().mockResolvedValue({ - runId: 'scan-run-1', - status: 'done', - done: true, - connectionId: 'warehouse', - mode: 'structural', - dryRun: false, - syncId: 'sync-1', - progress: 1, - startedAt: '2026-04-29T09:00:00.000Z', - completedAt: '2026-04-29T09:00:01.000Z', - reportPath: 'raw-sources/warehouse/live-database/sync-1/scan-report.json', - warnings: [], - }), - report: vi.fn().mockResolvedValue(null), - listArtifacts: vi.fn>().mockResolvedValue({ - runId: 'scan-run-1', - artifacts: [ - { - path: 'raw-sources/warehouse/live-database/sync-1/scan-report.json', - type: 'report', - size: 128, - }, - { - path: 'raw-sources/warehouse/live-database/sync-1/tables/orders.json', - type: 'raw_source', - size: 64, - }, - ], - }), - readArtifact: vi.fn>().mockImplementation(async (input) => { - if (input.path !== 'raw-sources/warehouse/live-database/sync-1/tables/orders.json') { - return null; - } - return { - runId: input.runId, - path: input.path, - type: 'raw_source', - size: 64, - content: '{"name":"orders"}\n', - }; + discover: { + search: vi.fn().mockResolvedValue([]), + }, + sqlExecution: { + execute: vi.fn().mockResolvedValue({ + headers: ['count'], + headerTypes: ['integer'], + rows: [[1]], + rowCount: 1, }), }, + memoryIngest: { + ingest: vi.fn().mockResolvedValue({ runId: 'run-1' }), + status: vi.fn().mockResolvedValue(null), + }, }; createKtxMcpServer({ server: fake.server, - memoryCapture: capture, userContext: { userId: 'mcp-user' }, contextTools, }); expect(fake.tools.map((tool) => tool.name).sort()).toEqual([ 'connection_list', - 'connection_test', - 'ingest_replay', - 'ingest_report', - 'ingest_status', - 'ingest_trigger', - 'memory_capture', - 'memory_capture_status', - 'scan_list_artifacts', - 'scan_read_artifact', - 'scan_report', - 'scan_status', - 'scan_trigger', - 'sl_list_sources', + 'dictionary_search', + 'discover_data', + 'entity_details', + 'memory_ingest', + 'memory_ingest_status', 'sl_query', 'sl_read_source', - 'sl_validate', - 'sl_write_source', + 'sql_execution', 'wiki_read', 'wiki_search', - 'wiki_write', ]); await expect(getTool(fake.tools, 'connection_list').handler({})).resolves.toEqual({ @@ -807,35 +946,6 @@ describe('createKtxMcpServer', () => { }, }); - await expect(getTool(fake.tools, 'connection_test').handler({ connectionId: 'warehouse' })).resolves.toEqual({ - content: [ - { - type: 'text', - text: JSON.stringify( - { - id: 'warehouse', - connectionType: 'postgres', - ok: true, - tableCount: 2, - message: 'Connection test passed.', - warnings: [], - }, - null, - 2, - ), - }, - ], - structuredContent: { - id: 'warehouse', - connectionType: 'postgres', - ok: true, - tableCount: 2, - message: 'Connection test passed.', - warnings: [], - }, - }); - expect(contextTools.connections?.test).toHaveBeenCalledWith({ connectionId: 'warehouse' }); - await getTool(fake.tools, 'wiki_search').handler({ query: 'revenue', limit: 5 }); expect(contextTools.knowledge?.search).toHaveBeenCalledWith({ userId: 'mcp-user', @@ -849,33 +959,6 @@ describe('createKtxMcpServer', () => { key: 'revenue', }); - await getTool(fake.tools, 'wiki_write').handler({ - key: 'revenue', - summary: 'Paid order value', - content: '# Revenue', - tags: ['finance'], - refs: ['gross-margin'], - sl_refs: ['orders'], - }); - expect(contextTools.knowledge?.write).toHaveBeenCalledWith({ - userId: 'mcp-user', - key: 'revenue', - summary: 'Paid order value', - content: '# Revenue', - tags: ['finance'], - refs: ['gross-margin'], - slRefs: ['orders'], - }); - - await getTool(fake.tools, 'sl_list_sources').handler({ - connectionId: '00000000-0000-4000-8000-000000000001', - query: 'orders', - }); - expect(contextTools.semanticLayer?.listSources).toHaveBeenCalledWith({ - connectionId: '00000000-0000-4000-8000-000000000001', - query: 'orders', - }); - await getTool(fake.tools, 'sl_read_source').handler({ connectionId: 'warehouse', sourceName: 'orders', @@ -885,28 +968,6 @@ describe('createKtxMcpServer', () => { sourceName: 'orders', }); - await getTool(fake.tools, 'sl_write_source').handler({ - connectionId: '00000000-0000-4000-8000-000000000001', - sourceName: 'orders', - source: { name: 'orders', table: 'public.orders', grain: ['id'], columns: [], joins: [], measures: [] }, - }); - expect(contextTools.semanticLayer?.writeSource).toHaveBeenCalledWith({ - connectionId: '00000000-0000-4000-8000-000000000001', - sourceName: 'orders', - source: { name: 'orders', table: 'public.orders', grain: ['id'], columns: [], joins: [], measures: [] }, - yaml: undefined, - delete: undefined, - }); - - await getTool(fake.tools, 'sl_validate').handler({ - connectionId: '00000000-0000-4000-8000-000000000001', - names: ['orders'], - }); - expect(contextTools.semanticLayer?.validate).toHaveBeenCalledWith({ - connectionId: '00000000-0000-4000-8000-000000000001', - names: ['orders'], - }); - await getTool(fake.tools, 'sl_query').handler({ connectionId: '00000000-0000-4000-8000-000000000001', measures: ['orders.count'], @@ -914,197 +975,29 @@ describe('createKtxMcpServer', () => { filters: ['orders.status = paid'], limit: 25, }); - expect(contextTools.semanticLayer?.query).toHaveBeenCalledWith({ - connectionId: '00000000-0000-4000-8000-000000000001', - query: { - measures: ['orders.count'], - dimensions: ['orders.created_at'], - filters: ['orders.status = paid'], - segments: [], - order_by: [], - limit: 25, - include_empty: true, + expect(contextTools.semanticLayer?.query).toHaveBeenCalledWith( + { + connectionId: '00000000-0000-4000-8000-000000000001', + query: { + measures: ['orders.count'], + dimensions: [{ field: 'orders.created_at' }], + filters: ['orders.status = paid'], + segments: [], + order_by: [], + limit: 25, + include_empty: true, + }, }, - }); - - await getTool(fake.tools, 'ingest_trigger').handler({ - adapter: 'lookml', - connectionId: '00000000-0000-4000-8000-000000000001', - trigger: 'scheduled_pull', - config: { repoUrl: 'https://github.com/acme/looker.git' }, - }); - expect(contextTools.ingest?.trigger).toHaveBeenCalledWith({ - adapter: 'lookml', - connectionId: '00000000-0000-4000-8000-000000000001', - trigger: 'scheduled_pull', - config: { repoUrl: 'https://github.com/acme/looker.git' }, - }); - - expect(getTool(fake.tools, 'ingest_status').config.description).toBe( - 'Read the current or final status for an ingest run, including local diff and work-unit summaries when available.', + undefined, ); + }); - await expect(getTool(fake.tools, 'ingest_status').handler({ runId: 'run-42' })).resolves.toMatchObject({ - structuredContent: { - runId: 'run-42', - status: 'done', - stage: 'done', - progress: 1, - done: true, - adapter: 'fake', - connectionId: 'warehouse', - sourceDir: '/tmp/upload', - syncId: '2026-04-27-120000-run-42', - previousRunId: 'run-41', - diffSummary: { - added: 0, - modified: 1, - deleted: 0, - unchanged: 3, - }, - rawFileCount: 4, - workUnitCount: 1, - workUnits: [ - { - unitKey: 'fake-orders', - rawFiles: ['orders/orders.json'], - peerFileIndex: [], - dependencyPaths: [], - }, - ], - evictionDeletedRawPaths: [], - errors: [], - }, - }); - expect(contextTools.ingest?.status).toHaveBeenCalledWith({ runId: 'run-42' }); + it('keeps jsonToolResult typed to non-array objects', () => { + expect(jsonToolResult({ ok: true }).structuredContent).toEqual({ ok: true }); - await expect(getTool(fake.tools, 'ingest_report').handler({ runId: 'report-42' })).resolves.toMatchObject({ - structuredContent: { - id: 'report-42', - runId: 'run-42', - jobId: 'job-42', - sourceKey: 'fake', - }, - }); - expect(contextTools.ingest?.report).toHaveBeenCalledWith({ runId: 'report-42' }); - - await expect(getTool(fake.tools, 'ingest_replay').handler({ runId: 'run-42' })).resolves.toMatchObject({ - structuredContent: { - runId: 'run-42', - reportId: 'report-42', - status: 'done', - adapter: 'fake', - }, - }); - expect(contextTools.ingest?.replay).toHaveBeenCalledWith({ runId: 'run-42' }); - - await getTool(fake.tools, 'scan_trigger').handler({ - connectionId: 'warehouse', - mode: 'structural', - dryRun: true, - }); - expect(contextTools.scan?.trigger).toHaveBeenCalledWith({ - connectionId: 'warehouse', - mode: 'structural', - detectRelationships: false, - dryRun: true, - }); - - await getTool(fake.tools, 'scan_trigger').handler({ - connectionId: 'warehouse', - mode: 'relationships', - detectRelationships: true, - dryRun: false, - }); - expect(contextTools.scan?.trigger).toHaveBeenCalledWith({ - connectionId: 'warehouse', - mode: 'relationships', - detectRelationships: true, - dryRun: false, - }); - - await expect(getTool(fake.tools, 'scan_status').handler({ runId: 'scan-run-1' })).resolves.toMatchObject({ - structuredContent: { - runId: 'scan-run-1', - status: 'done', - connectionId: 'warehouse', - }, - }); - - await expect(getTool(fake.tools, 'scan_report').handler({ runId: 'missing' })).resolves.toEqual({ - content: [{ type: 'text', text: 'Scan report "missing" was not found.' }], - isError: true, - }); - - await expect(getTool(fake.tools, 'scan_list_artifacts').handler({ runId: 'scan-run-1' })).resolves.toEqual({ - content: [ - { - type: 'text', - text: JSON.stringify( - { - runId: 'scan-run-1', - artifacts: [ - { - path: 'raw-sources/warehouse/live-database/sync-1/scan-report.json', - type: 'report', - size: 128, - }, - { - path: 'raw-sources/warehouse/live-database/sync-1/tables/orders.json', - type: 'raw_source', - size: 64, - }, - ], - }, - null, - 2, - ), - }, - ], - structuredContent: { - runId: 'scan-run-1', - artifacts: [ - { - path: 'raw-sources/warehouse/live-database/sync-1/scan-report.json', - type: 'report', - size: 128, - }, - { - path: 'raw-sources/warehouse/live-database/sync-1/tables/orders.json', - type: 'raw_source', - size: 64, - }, - ], - }, - }); - expect(contextTools.scan?.listArtifacts).toHaveBeenCalledWith({ runId: 'scan-run-1' }); - - await expect( - getTool(fake.tools, 'scan_read_artifact').handler({ - runId: 'scan-run-1', - path: 'raw-sources/warehouse/live-database/sync-1/tables/orders.json', - }), - ).resolves.toMatchObject({ - structuredContent: { - runId: 'scan-run-1', - path: 'raw-sources/warehouse/live-database/sync-1/tables/orders.json', - type: 'raw_source', - content: '{"name":"orders"}\n', - }, - }); - expect(contextTools.scan?.readArtifact).toHaveBeenCalledWith({ - runId: 'scan-run-1', - path: 'raw-sources/warehouse/live-database/sync-1/tables/orders.json', - }); - - await expect( - getTool(fake.tools, 'scan_read_artifact').handler({ - runId: 'scan-run-1', - path: 'ktx.yaml', - }), - ).resolves.toEqual({ - content: [{ type: 'text', text: 'Scan artifact "ktx.yaml" was not found for run "scan-run-1".' }], - isError: true, - }); + if (false) { + // @ts-expect-error bare arrays are not valid MCP structuredContent objects in KTX + jsonToolResult([]); + } }); }); diff --git a/packages/context/src/mcp/server.ts b/packages/context/src/mcp/server.ts index ba11c086..8931650f 100644 --- a/packages/context/src/mcp/server.ts +++ b/packages/context/src/mcp/server.ts @@ -1,71 +1,8 @@ -import { randomUUID } from 'node:crypto'; import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; -import { z } from 'zod'; -import type { MemoryAgentInput } from '../memory/index.js'; -import { jsonErrorToolResult, jsonToolResult, registerKtxContextTools } from './context-tools.js'; -import type { KtxMcpServerDeps, KtxMcpServerLike, MemoryCapturePort } from './types.js'; - -const memoryCaptureInputSchema = { - userMessage: z.string().min(1).describe('The user message that may contain durable knowledge.'), - assistantMessage: z.string().optional().describe('The assistant response that concluded the exchange.'), - connectionId: z.string().min(1).optional().describe('Optional connection id for semantic-layer capture.'), -}; - -const memoryCaptureStatusInputSchema = { - runId: z.string().min(1).describe('The memory capture run id returned by memory_capture.'), -}; - -function registerMemoryCaptureTools(deps: { - server: KtxMcpServerLike; - memoryCapture: MemoryCapturePort; - userContext: KtxMcpServerDeps['userContext']; -}): void { - deps.server.registerTool( - 'memory_capture', - { - title: 'Memory Capture', - description: - 'Capture durable knowledge and semantic-layer updates from the final user/assistant exchange. Returns a run id for polling.', - inputSchema: memoryCaptureInputSchema, - }, - async (input) => { - const captureInput: MemoryAgentInput = { - userId: deps.userContext.userId, - chatId: `mcp-${randomUUID()}`, - userMessage: String(input.userMessage), - assistantMessage: typeof input.assistantMessage === 'string' ? input.assistantMessage : undefined, - connectionId: typeof input.connectionId === 'string' ? input.connectionId : undefined, - sourceType: 'external_ingest', - }; - const result = await deps.memoryCapture.capture(captureInput); - return jsonToolResult(result); - }, - ); - - deps.server.registerTool( - 'memory_capture_status', - { - title: 'Memory Capture Status', - description: 'Read the current or final status for a memory capture run.', - inputSchema: memoryCaptureStatusInputSchema, - }, - async (input) => { - const runId = String(input.runId); - const status = await deps.memoryCapture.status(runId); - return status ? jsonToolResult(status) : jsonErrorToolResult(`Memory capture run "${runId}" was not found.`); - }, - ); -} +import { registerKtxContextTools } from './context-tools.js'; +import type { KtxMcpServerDeps, KtxMcpServerLike } from './types.js'; export function createKtxMcpServer(deps: KtxMcpServerDeps): KtxMcpServerDeps['server'] { - if (deps.memoryCapture) { - registerMemoryCaptureTools({ - server: deps.server, - memoryCapture: deps.memoryCapture, - userContext: deps.userContext, - }); - } - if (deps.contextTools) { registerKtxContextTools({ server: deps.server, @@ -86,7 +23,6 @@ export function createDefaultKtxMcpServer( }); createKtxMcpServer({ server: server as KtxMcpServerLike, - memoryCapture: deps.memoryCapture, userContext: deps.userContext, contextTools: deps.contextTools, }); diff --git a/packages/context/src/mcp/types.ts b/packages/context/src/mcp/types.ts index ab53f56e..49bf7b55 100644 --- a/packages/context/src/mcp/types.ts +++ b/packages/context/src/mcp/types.ts @@ -1,16 +1,7 @@ -import type { IngestReportSnapshot, MemoryFlowReplayInput, TableUsageOutput } from '../ingest/index.js'; -import type { MemoryCaptureService } from '../memory/index.js'; +import type { MemoryIngestService } from '../memory/index.js'; import type { KtxEntityDetailsInput, KtxEntityDetailsResponse } from '../scan/entity-details.js'; -import type { KtxScanMode, KtxScanReport } from '../scan/index.js'; import type { KtxDiscoverDataInput, KtxDiscoverDataResponse } from '../search/index.js'; -import type { - KtxDictionarySearchInput, - KtxDictionarySearchResponse, - SemanticLayerQueryInput, - SlDictionaryMatch, - SlSearchLaneSummary, - SlSearchMatchReason, -} from '../sl/index.js'; +import type { KtxDictionarySearchInput, KtxDictionarySearchResponse, SemanticLayerQueryInput } from '../sl/index.js'; import type { WikiSearchLaneSummary, WikiSearchMatchReason } from '../wiki/index.js'; export interface KtxMcpTextContent { @@ -18,15 +9,38 @@ export interface KtxMcpTextContent { text: string; } -export interface KtxMcpToolResult { +export type NonArrayObject = object & { length?: never }; + +export interface KtxMcpToolResult { content: KtxMcpTextContent[]; structuredContent?: T; isError?: true; } -export interface MemoryCapturePort { - capture: MemoryCaptureService['capture']; - status: MemoryCaptureService['status']; +interface KtxMcpProgressEvent { + progress: number; + total?: number; + message: string; +} + +export type KtxMcpProgressCallback = (event: KtxMcpProgressEvent) => void | Promise; + +export interface KtxMcpToolHandlerContext { + _meta?: { progressToken?: string | number; [key: string]: unknown }; + sendNotification?: (notification: { + method: 'notifications/progress'; + params: { + progressToken: string | number; + progress: number; + total?: number; + message?: string; + }; + }) => Promise; +} + +export interface MemoryIngestPort { + ingest: MemoryIngestService['ingest']; + status: MemoryIngestService['status']; } export interface KtxMcpUserContext { @@ -40,8 +54,10 @@ export interface KtxMcpServerLike { title?: string; description?: string; inputSchema: unknown; + outputSchema?: unknown; + annotations?: Record; }, - handler: (input: Record) => Promise, + handler: (input: Record, context?: KtxMcpToolHandlerContext) => Promise, ): void; } @@ -51,18 +67,8 @@ export interface KtxConnectionSummary { connectionType: string; } -export interface KtxConnectionTestResponse { - id: string; - connectionType: string; - ok: boolean; - tableCount: number | null; - message: string; - warnings: string[]; -} - export interface KtxConnectionsMcpPort { list(): Promise; - test?(input: { connectionId: string }): Promise; } export interface KtxKnowledgeSearchResult { @@ -90,62 +96,9 @@ export interface KtxKnowledgePage { slRefs?: string[]; } -interface KtxHistoricSqlKnowledgeUsage { - executions: number; - distinct_users: number; - first_seen: string; - last_seen: string; - p50_runtime_ms: number | null; - p95_runtime_ms: number | null; - error_rate: number; - rows_produced?: number; -} - -export interface KtxKnowledgeWriteResponse { - success: boolean; - key: string; - action: 'created' | 'updated'; -} - export interface KtxKnowledgeMcpPort { search(input: { userId: string; query: string; limit: number }): Promise; read(input: { userId: string; key: string }): Promise; - write(input: { - userId: string; - key: string; - summary: string; - content: string; - tags?: string[]; - refs?: string[]; - slRefs?: string[]; - source?: string; - intent?: string; - tables?: string[]; - representativeSql?: string; - usage?: KtxHistoricSqlKnowledgeUsage; - fingerprints?: string[]; - }): Promise; -} - -export interface KtxSemanticLayerSourceSummary { - connectionId: string; - connectionName: string; - name: string; - description?: string; - columnCount: number; - measureCount: number; - joinCount: number; - frequencyTier?: TableUsageOutput['frequencyTier']; - snippet?: string; - score?: number; - matchReasons?: SlSearchMatchReason[]; - dictionaryMatches?: SlDictionaryMatch[]; - lanes?: SlSearchLaneSummary[]; -} - -export interface KtxSemanticLayerListResponse { - sources: KtxSemanticLayerSourceSummary[]; - totalSources: number; } export interface KtxSemanticLayerReadResponse { @@ -153,21 +106,6 @@ export interface KtxSemanticLayerReadResponse { yaml: string; } -export interface KtxSemanticLayerWriteResponse { - success: boolean; - sourceName: string; - yaml?: string; - errors?: string[]; - warnings?: string[]; - commitHash?: string; -} - -export interface KtxSemanticLayerValidationResponse { - success: boolean; - errors: string[]; - warnings: string[]; -} - export interface KtxSemanticLayerQueryResponse { sql: string; headers: string[]; @@ -177,143 +115,11 @@ export interface KtxSemanticLayerQueryResponse { } export interface KtxSemanticLayerMcpPort { - listSources(input: { connectionId?: string; query?: string }): Promise; readSource(input: { connectionId: string; sourceName: string }): Promise; - writeSource(input: { - connectionId: string; - sourceName: string; - yaml?: string; - source?: Record; - delete?: boolean; - }): Promise; - validate(input: { connectionId: string; names?: string[] }): Promise; - query(input: { connectionId?: string; query: SemanticLayerQueryInput }): Promise; -} - -export type KtxIngestTriggerKind = 'upload' | 'scheduled_pull' | 'manual_resync'; - -interface KtxIngestTriggerFanoutChild { - runId: string; - jobId: string; - reportId: string; - targetConnectionId: string; - metabaseDatabaseId: number; -} - -export interface KtxIngestTriggerResponse { - runId: string; - jobId?: string; - reportId?: string; - fanout?: { - status: 'all_succeeded' | 'partial_failure' | 'all_failed'; - children: KtxIngestTriggerFanoutChild[]; - }; -} - -export interface KtxIngestDiffSummary { - added: number; - modified: number; - deleted: number; - unchanged: number; -} - -export interface KtxIngestWorkUnitSummary { - unitKey: string; - rawFiles: string[]; - peerFileIndex: string[]; - dependencyPaths: string[]; -} - -export interface KtxIngestStatusResponse { - runId: string; - jobId?: string; - reportId?: string; - status: string; - stage?: string; - progress?: number; - errors?: string[]; - done: boolean; - adapter?: string; - connectionId?: string; - sourceDir?: string | null; - syncId?: string; - startedAt?: string; - completedAt?: string; - previousRunId?: string | null; - diffSummary?: KtxIngestDiffSummary; - workUnitCount?: number; - rawFileCount?: number; - workUnits?: KtxIngestWorkUnitSummary[]; - evictionDeletedRawPaths?: string[]; -} - -export interface KtxIngestMcpPort { - trigger(input: { - adapter: string; - connectionId: string; - config?: unknown; - trigger: KtxIngestTriggerKind; - }): Promise; - status(input: { runId: string }): Promise; - report?(input: { runId: string }): Promise; - replay?(input: { runId: string }): Promise; -} - -interface KtxScanTriggerResponse { - runId: string; - status: 'done'; - done: true; - connectionId: string; - mode: KtxScanMode; - dryRun: boolean; - syncId: string; - report: KtxScanReport; -} - -interface KtxScanStatusResponse { - runId: string; - status: string; - done: boolean; - connectionId: string; - mode: KtxScanMode; - dryRun: boolean; - syncId: string; - progress: number; - startedAt: string; - completedAt: string; - reportPath: string | null; - warnings: KtxScanReport['warnings']; -} - -export type KtxScanArtifactType = 'report' | 'raw_source' | 'manifest_shard' | 'enrichment_artifact'; - -export interface KtxScanArtifactSummary { - path: string; - type: KtxScanArtifactType; - size?: number; -} - -export interface KtxScanArtifactListResponse { - runId: string; - artifacts: KtxScanArtifactSummary[]; -} - -export interface KtxScanArtifactReadResponse extends KtxScanArtifactSummary { - runId: string; - content: string; -} - -export interface KtxScanMcpPort { - trigger(input: { - connectionId: string; - mode?: KtxScanMode; - detectRelationships: boolean; - dryRun: boolean; - }): Promise; - status(input: { runId: string }): Promise; - report(input: { runId: string }): Promise; - listArtifacts?(input: { runId: string }): Promise; - readArtifact?(input: { runId: string; path: string }): Promise; + query( + input: { connectionId?: string; query: SemanticLayerQueryInput }, + options?: { onProgress?: KtxMcpProgressCallback }, + ): Promise; } export interface KtxEntityDetailsMcpPort { @@ -336,7 +142,10 @@ export interface KtxSqlExecutionResponse { } export interface KtxSqlExecutionMcpPort { - execute(input: { connectionId: string; sql: string; maxRows: number }): Promise; + execute( + input: { connectionId: string; sql: string; maxRows: number }, + options?: { onProgress?: KtxMcpProgressCallback }, + ): Promise; } export interface KtxMcpContextPorts { @@ -347,13 +156,11 @@ export interface KtxMcpContextPorts { dictionarySearch?: KtxDictionarySearchMcpPort; discover?: KtxDiscoverDataMcpPort; sqlExecution?: KtxSqlExecutionMcpPort; - ingest?: KtxIngestMcpPort; - scan?: KtxScanMcpPort; + memoryIngest?: MemoryIngestPort; } export interface KtxMcpServerDeps { server: KtxMcpServerLike; - memoryCapture?: MemoryCapturePort; userContext: KtxMcpUserContext; contextTools?: KtxMcpContextPorts; } diff --git a/packages/context/src/memory/index.ts b/packages/context/src/memory/index.ts index a23c4ed5..28efcd2e 100644 --- a/packages/context/src/memory/index.ts +++ b/packages/context/src/memory/index.ts @@ -8,13 +8,13 @@ export { stepBudgetFor, } from './capture-signals.js'; export { MemoryAgentService } from './memory-agent.service.js'; -export { createLocalProjectMemoryCapture, type CreateLocalProjectMemoryCaptureOptions } from './local-memory.js'; +export { createLocalProjectMemoryIngest, type CreateLocalProjectMemoryIngestOptions } from './local-memory.js'; export { LocalMemoryRunStore, type LocalMemoryRunStoreOptions } from './local-memory-runs.js'; export { - MemoryCaptureService, - type MemoryCaptureServiceDeps, - type MemoryCaptureStartResult, - type MemoryCaptureStatus, + MemoryIngestService, + type MemoryIngestServiceDeps, + type MemoryIngestStartResult, + type MemoryIngestStatus, type MemoryRunRecord, type MemoryRunStatus, type MemoryRunStorePort, diff --git a/packages/context/src/memory/local-memory.test.ts b/packages/context/src/memory/local-memory.test.ts index f0b870eb..24e0df14 100644 --- a/packages/context/src/memory/local-memory.test.ts +++ b/packages/context/src/memory/local-memory.test.ts @@ -3,7 +3,7 @@ import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { initKtxProject } from '../project/index.js'; -import { createLocalProjectMemoryCapture } from './local-memory.js'; +import { createLocalProjectMemoryIngest } from './local-memory.js'; import { LocalMemoryRunStore } from './local-memory-runs.js'; vi.mock('ai', () => ({ @@ -77,7 +77,7 @@ describe('LocalMemoryRunStore', () => { }); }); -describe('createLocalProjectMemoryCapture', () => { +describe('createLocalProjectMemoryIngest', () => { let tempDir: string; beforeEach(async () => { @@ -110,13 +110,13 @@ describe('createLocalProjectMemoryCapture', () => { }, }; - const capture = createLocalProjectMemoryCapture(project, { + const ingest = createLocalProjectMemoryIngest(project, { agentRunner: agentRunner as never, runIdFactory: () => 'memory-run-1', }); await expect( - capture.capture({ + ingest.ingest({ userId: 'local-user', chatId: 'chat-1', userMessage: 'define revenue as paid order value net of refunds', @@ -124,12 +124,12 @@ describe('createLocalProjectMemoryCapture', () => { sourceType: 'external_ingest', }), ).resolves.toEqual({ runId: 'memory-run-1' }); - await capture.waitForRun('memory-run-1'); + await ingest.waitForRun('memory-run-1'); await expect(access(join(project.projectDir, '.ktx/db.sqlite'))).resolves.toBeUndefined(); await expectPathMissing(join(project.projectDir, '.ktx/memory-runs/memory-run-1.json')); - await expect(capture.status('memory-run-1')).resolves.toMatchObject({ + await expect(ingest.status('memory-run-1')).resolves.toMatchObject({ runId: 'memory-run-1', status: 'done', done: true, @@ -172,12 +172,12 @@ describe('createLocalProjectMemoryCapture', () => { }, }; - const capture = createLocalProjectMemoryCapture(project, { + const ingest = createLocalProjectMemoryIngest(project, { agentRunner: agentRunner as never, runIdFactory: () => 'memory-run-2', }); - await capture.capture({ + await ingest.ingest({ userId: 'local-user', chatId: 'chat-2', userMessage: 'going forward define orders count as count of public orders', @@ -185,12 +185,12 @@ describe('createLocalProjectMemoryCapture', () => { connectionId: 'warehouse', sourceType: 'external_ingest', }); - await capture.waitForRun('memory-run-2'); + await ingest.waitForRun('memory-run-2'); await expect(access(join(project.projectDir, '.ktx/db.sqlite'))).resolves.toBeUndefined(); await expectPathMissing(join(project.projectDir, '.ktx/memory-runs/memory-run-2.json')); - await expect(capture.status('memory-run-2')).resolves.toMatchObject({ + await expect(ingest.status('memory-run-2')).resolves.toMatchObject({ runId: 'memory-run-2', status: 'done', captured: { wiki: [], sl: ['orders'], xrefs: [] }, diff --git a/packages/context/src/memory/local-memory.ts b/packages/context/src/memory/local-memory.ts index 3cc9d324..c1b1e4bd 100644 --- a/packages/context/src/memory/local-memory.ts +++ b/packages/context/src/memory/local-memory.ts @@ -47,7 +47,7 @@ import { } from '../wiki/index.js'; import { LocalMemoryRunStore } from './local-memory-runs.js'; import { MemoryAgentService } from './memory-agent.service.js'; -import { MemoryCaptureService } from './memory-runs.js'; +import { MemoryIngestService } from './memory-runs.js'; import type { MemoryConnectionPort, MemoryFileStorePort, @@ -60,9 +60,9 @@ import type { const promptsDir = fileURLToPath(new URL('../../prompts', import.meta.url)); const skillsDir = fileURLToPath(new URL('../../skills', import.meta.url)); const LOCAL_AUTHOR = { name: 'KTX Local', email: 'local@ktx.local' }; -const LOCAL_SHAPE_WARNING = 'Local memory capture validates semantic-layer YAML shape only.'; +const LOCAL_SHAPE_WARNING = 'Local memory ingest validates semantic-layer YAML shape only.'; -export interface CreateLocalProjectMemoryCaptureOptions { +export interface CreateLocalProjectMemoryIngestOptions { llmProvider?: KtxLlmProvider; agentRunner?: AgentRunnerService; memoryModel?: string; @@ -72,10 +72,10 @@ export interface CreateLocalProjectMemoryCaptureOptions { logger?: KtxLogger; } -export function createLocalProjectMemoryCapture( +export function createLocalProjectMemoryIngest( project: KtxLocalProject, - options: CreateLocalProjectMemoryCaptureOptions = {}, -): MemoryCaptureService { + options: CreateLocalProjectMemoryIngestOptions = {}, +): MemoryIngestService { const logger = options.logger ?? noopLogger; const rootFileStore = new LocalMemoryFileStore(project.fileStore); const embedding = new NoopEmbeddingPort(); @@ -137,7 +137,7 @@ export function createLocalProjectMemoryCapture( toolsetFactory, logger, }); - return new MemoryCaptureService({ + return new MemoryIngestService({ memoryAgent, runs: new LocalMemoryRunStore({ projectDir: project.projectDir, idFactory: options.runIdFactory }), }); @@ -145,7 +145,7 @@ export function createLocalProjectMemoryCapture( function requireLlmProvider(provider: KtxLlmProvider | null | undefined): KtxLlmProvider { if (!provider) { - throw new Error('createLocalProjectMemoryCapture requires llm.provider.backend or an injected agentRunner'); + throw new Error('createLocalProjectMemoryIngest requires llm.provider.backend or an injected agentRunner'); } return provider; } diff --git a/packages/context/src/memory/memory-runs.test.ts b/packages/context/src/memory/memory-runs.test.ts index 049936ad..9fd9c36d 100644 --- a/packages/context/src/memory/memory-runs.test.ts +++ b/packages/context/src/memory/memory-runs.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, vi } from 'vitest'; import type { MemoryAgentInput, MemoryAgentResult, MemoryAgentService } from './index.js'; -import { MemoryCaptureService, type MemoryRunStorePort } from './memory-runs.js'; +import { MemoryIngestService, type MemoryRunStorePort } from './memory-runs.js'; class InMemoryRunStore implements MemoryRunStorePort { readonly rows = new Map< @@ -74,32 +74,32 @@ function deferred() { } function buildService(): { - capture: MemoryCaptureService; + ingest: MemoryIngestService; store: InMemoryRunStore; - ingest: ReturnType; + memoryAgentIngest: ReturnType; run: ReturnType>; } { const store = new InMemoryRunStore(); const run = deferred(); - const ingest = vi.fn().mockReturnValue(run.promise); - const memoryAgent = { ingest }; + const memoryAgentIngest = vi.fn().mockReturnValue(run.promise); + const memoryAgent = { ingest: memoryAgentIngest }; return { - capture: new MemoryCaptureService({ memoryAgent, runs: store }), + ingest: new MemoryIngestService({ memoryAgent, runs: store }), store, - ingest, + memoryAgentIngest, run, }; } -describe('MemoryCaptureService', () => { - it('creates a run, executes memory capture, and stores a done summary', async () => { +describe('MemoryIngestService', () => { + it('creates a run, executes memory ingest, and stores a done summary', async () => { const result: MemoryAgentResult = { signalDetected: true, actions: [{ target: 'wiki', type: 'created', key: 'revenue', detail: 'captured revenue definition' }], skillsLoaded: ['wiki_capture'], commitHash: 'abc123', }; - const { capture, store, ingest, run } = buildService(); + const { ingest, store, memoryAgentIngest, run } = buildService(); const input: MemoryAgentInput = { userId: 'user-1', @@ -109,21 +109,21 @@ describe('MemoryCaptureService', () => { connectionId: '00000000-0000-0000-0000-000000000001', }; - const started = await capture.capture(input); + const started = await ingest.ingest(input); expect(started.runId).toBe('run-1'); - expect(ingest).toHaveBeenCalledWith(input); - await expect(capture.status(started.runId)).resolves.toMatchObject({ + expect(memoryAgentIngest).toHaveBeenCalledWith(input); + await expect(ingest.status(started.runId)).resolves.toMatchObject({ runId: 'run-1', status: 'running', - stage: 'capturing', + stage: 'ingesting', done: false, }); run.resolve(result); - await capture.waitForRun(started.runId); + await ingest.waitForRun(started.runId); - const status = await capture.status(started.runId); + const status = await ingest.status(started.runId); expect(status).toEqual({ runId: 'run-1', stage: 'done', @@ -142,10 +142,10 @@ describe('MemoryCaptureService', () => { expect(store.rows.get('run-1')?.inputHash).toHaveLength(64); }); - it('stores no-signal captures as done with empty captured arrays', async () => { - const { capture, run } = buildService(); + it('stores no-signal ingests as done with empty captured arrays', async () => { + const { ingest, run } = buildService(); - const started = await capture.capture({ + const started = await ingest.ingest({ userId: 'user-1', chatId: 'chat-2', userMessage: 'Thanks.', @@ -157,9 +157,9 @@ describe('MemoryCaptureService', () => { skillsLoaded: [], commitHash: null, }); - await capture.waitForRun(started.runId); + await ingest.waitForRun(started.runId); - await expect(capture.status(started.runId)).resolves.toMatchObject({ + await expect(ingest.status(started.runId)).resolves.toMatchObject({ done: true, status: 'done', captured: { wiki: [], sl: [], xrefs: [] }, @@ -172,16 +172,16 @@ describe('MemoryCaptureService', () => { const memoryAgent = { ingest: vi.fn().mockRejectedValue(new Error('LLM provider missing')), }; - const capture = new MemoryCaptureService({ memoryAgent, runs: store }); + const ingest = new MemoryIngestService({ memoryAgent, runs: store }); - const started = await capture.capture({ + const started = await ingest.ingest({ userId: 'user-1', chatId: 'chat-3', userMessage: 'Remember this.', }); - await capture.waitForRun(started.runId); + await ingest.waitForRun(started.runId); - await expect(capture.status(started.runId)).resolves.toMatchObject({ + await expect(ingest.status(started.runId)).resolves.toMatchObject({ done: true, status: 'error', stage: 'error', @@ -191,8 +191,8 @@ describe('MemoryCaptureService', () => { }); it('returns null for an unknown run id', async () => { - const { capture } = buildService(); + const { ingest } = buildService(); - await expect(capture.status('missing')).resolves.toBeNull(); + await expect(ingest.status('missing')).resolves.toBeNull(); }); }); diff --git a/packages/context/src/memory/memory-runs.ts b/packages/context/src/memory/memory-runs.ts index 5550d5d0..2b34657e 100644 --- a/packages/context/src/memory/memory-runs.ts +++ b/packages/context/src/memory/memory-runs.ts @@ -21,16 +21,16 @@ export interface MemoryRunStorePort { findById(id: string): Promise; } -export interface MemoryCaptureServiceDeps { +export interface MemoryIngestServiceDeps { memoryAgent: Pick; runs: MemoryRunStorePort; } -export interface MemoryCaptureStartResult { +export interface MemoryIngestStartResult { runId: string; } -export interface MemoryCaptureStatus { +export interface MemoryIngestStatus { runId: string; status: MemoryRunStatus; stage: string; @@ -55,7 +55,7 @@ function inputHash(input: MemoryAgentInput): string { return createHash('sha256').update(stableInput).digest('hex'); } -function capturedKeys(actions: MemoryAction[]): MemoryCaptureStatus['captured'] { +function capturedKeys(actions: MemoryAction[]): MemoryIngestStatus['captured'] { const wiki = new Set(); const sl = new Set(); const xrefs = new Set(); @@ -78,20 +78,20 @@ function capturedKeys(actions: MemoryAction[]): MemoryCaptureStatus['captured'] }; } -export class MemoryCaptureService { +export class MemoryIngestService { private readonly inFlight = new Map>(); - constructor(private readonly deps: MemoryCaptureServiceDeps) {} + constructor(private readonly deps: MemoryIngestServiceDeps) {} - async capture(input: MemoryAgentInput): Promise { + async ingest(input: MemoryAgentInput): Promise { const row = await this.deps.runs.createRunning({ inputHash: inputHash(input), chatId: input.chatId, }); - await this.deps.runs.markRunning(row.id, 'capturing'); + await this.deps.runs.markRunning(row.id, 'ingesting'); - const run = this.runCapture(row.id, input); + const run = this.runIngest(row.id, input); this.inFlight.set(row.id, run); run.finally(() => this.inFlight.delete(row.id)).catch(() => undefined); @@ -102,7 +102,7 @@ export class MemoryCaptureService { await this.inFlight.get(runId); } - private async runCapture(runId: string, input: MemoryAgentInput): Promise { + private async runIngest(runId: string, input: MemoryAgentInput): Promise { try { const outputSummary = await this.deps.memoryAgent.ingest(input); await this.deps.runs.markDone(runId, outputSummary); @@ -111,7 +111,7 @@ export class MemoryCaptureService { } } - async status(runId: string): Promise { + async status(runId: string): Promise { const row = await this.deps.runs.findById(runId); if (!row) { return null; diff --git a/packages/context/src/sl/local-query.test.ts b/packages/context/src/sl/local-query.test.ts index 2852b35a..b4703fe6 100644 --- a/packages/context/src/sl/local-query.test.ts +++ b/packages/context/src/sl/local-query.test.ts @@ -182,6 +182,46 @@ grain: [] }); }); + it('strips authoring-only fields (usage, inherits_columns_from) before sending sources to the daemon', async () => { + await project.fileStore.writeFile( + 'semantic-layer/warehouse/_schema/public.yaml', + `tables: + invoices: + table: public.invoices + columns: + - name: invoice_id + type: number + pk: true + - name: amount + type: number + usage: + narrative: Activation policy windows table for invoice analytics. + frequencyTier: mid + commonFilters: + - amount + commonGroupBys: [] + commonJoins: [] + staleSince: null +`, + 'ktx', + 'ktx@example.com', + 'Add manifest shard with usage', + ); + + await compileLocalSlQuery(project, { + connectionId: 'warehouse', + query: { measures: ['sum(invoices.amount)'], dimensions: [] }, + compute, + }); + + const lastCall = (compute.query as ReturnType).mock.calls.at(-1)?.[0]; + const invoices = lastCall?.sources.find((s: Record) => s.name === 'invoices'); + expect(invoices).toBeDefined(); + expect(invoices).not.toHaveProperty('usage'); + expect(invoices).not.toHaveProperty('inherits_columns_from'); + expect(invoices).not.toHaveProperty('source_type'); + }); + it('resolves the only configured connection when connectionId is omitted', async () => { await compileLocalSlQuery(project, { query: { measures: ['orders.order_count'], dimensions: [] }, @@ -236,6 +276,43 @@ grain: [] }); }); + it('emits progress while compiling and executing a local semantic-layer query', async () => { + const progress: Array<{ progress: number; message: string }> = []; + const queryExecutor = { + execute: vi.fn(async () => ({ + headers: ['status', 'order_count'], + rows: [['paid', 2]], + totalRows: 1, + command: 'SELECT', + rowCount: 1, + })), + }; + + const result = await compileLocalSlQuery(project, { + connectionId: 'warehouse', + query: { + measures: ['orders.order_count'], + dimensions: ['orders.status'], + limit: 25, + }, + compute, + execute: true, + maxRows: 10, + queryExecutor, + onProgress: (event) => { + progress.push({ progress: event.progress, message: event.message }); + }, + }); + + expect(result.totalRows).toBe(1); + expect(progress).toEqual([ + { progress: 0, message: 'Compiling query' }, + { progress: 0.3, message: 'Generating SQL' }, + { progress: 0.6, message: 'Executing' }, + { progress: 1, message: 'Fetched 1 rows' }, + ]); + }); + it('requires a query executor for executed mode', async () => { await expect( compileLocalSlQuery(project, { diff --git a/packages/context/src/sl/local-query.ts b/packages/context/src/sl/local-query.ts index a6f224dd..05a264fc 100644 --- a/packages/context/src/sl/local-query.ts +++ b/packages/context/src/sl/local-query.ts @@ -1,7 +1,9 @@ import type { KtxSqlQueryExecutorPort } from '../connections/index.js'; import type { KtxSemanticLayerComputePort } from '../daemon/index.js'; +import type { KtxMcpProgressCallback } from '../mcp/types.js'; import type { KtxLocalProject } from '../project/index.js'; import { loadLocalSlSourceRecords } from './local-sl.js'; +import { toResolvedWire } from './semantic-layer.service.js'; import type { SemanticLayerQueryExecutionResult, SemanticLayerQueryInput } from './types.js'; const COMPILE_ONLY_REASON = @@ -14,6 +16,7 @@ export interface CompileLocalSlQueryOptions { execute?: boolean; maxRows?: number; queryExecutor?: KtxSqlQueryExecutorPort; + onProgress?: KtxMcpProgressCallback; } export interface CompileLocalSlQueryResult extends SemanticLayerQueryExecutionResult { @@ -75,10 +78,10 @@ function resolveLocalConnectionId(project: KtxLocalProject, requested: string | async function loadComputableSources( project: KtxLocalProject, connectionId: string, -): Promise[]> { +): Promise[]> { return (await loadLocalSlSourceRecords(project, { connectionId: assertSafeConnectionId(connectionId) })) - .map((record) => ({ ...record.source })) - .filter((source) => source.table || source.sql); + .filter((record) => record.source.table || record.source.sql) + .map((record) => toResolvedWire(record.source)); } function headersFromColumns(columns: Array>): string[] { @@ -91,15 +94,20 @@ export async function compileLocalSlQuery( project: KtxLocalProject, options: CompileLocalSlQueryOptions, ): Promise { + await options.onProgress?.({ progress: 0, message: 'Compiling query' }); const connectionId = resolveLocalConnectionId(project, options.connectionId); const dialect = dialectForDriver(project.config.connections[connectionId]?.driver); + const sources = await loadComputableSources(project, connectionId); + + await options.onProgress?.({ progress: 0.3, message: 'Generating SQL' }); const response = await options.compute.query({ - sources: await loadComputableSources(project, connectionId), + sources, dialect, query: options.query, }); if (!options.execute) { + await options.onProgress?.({ progress: 1, message: 'Fetched 0 rows' }); return { connectionId, dialect: response.dialect, @@ -122,6 +130,7 @@ export async function compileLocalSlQuery( } const maxRows = options.maxRows ?? options.query.limit; + await options.onProgress?.({ progress: 0.6, message: 'Executing' }); const execution = await options.queryExecutor.execute({ connectionId, projectDir: project.projectDir, @@ -129,6 +138,7 @@ export async function compileLocalSlQuery( sql: response.sql, maxRows, }); + await options.onProgress?.({ progress: 1, message: `Fetched ${execution.totalRows} rows` }); return { connectionId, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 22e0c035..39b25c9b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -112,6 +112,9 @@ importers: commander: specifier: 14.0.3 version: 14.0.3 + fflate: + specifier: ^0.8.2 + version: 0.8.2 ink: specifier: ^7.0.2 version: 7.0.2(@types/react@19.2.14)(react@19.2.6) @@ -3191,6 +3194,9 @@ packages: resolution: {integrity: sha512-7yAQpD2UMJzLi1Dqv7qFYnPbaPx7ZfFK6PiIxQ4PfkGPyNyl2Ugx+a/umUonmKqjhM4DnfbMvdX6otXq83soQQ==} engines: {node: ^12.20 || >= 14.13} + fflate@0.8.2: + resolution: {integrity: sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==} + file-uri-to-path@1.0.0: resolution: {integrity: sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==} @@ -8355,6 +8361,8 @@ snapshots: node-domexception: 1.0.0 web-streams-polyfill: 3.3.3 + fflate@0.8.2: {} + file-uri-to-path@1.0.0: {} finalhandler@2.1.1: