apunkt/ktx
1
0
Fork 0
mirror of https://github.com/Kaelio/ktx.git synced 2026-06-07 07:55:13 +02:00
Code Issues Projects Releases Packages Wiki Activity

feat(mcp):added MCP server (#97)

Browse source 
* docs(specs): design research-agent MCP tools and ktx mcp daemon

Adds the 2026-05-14 design spec for exposing four new MCP tools
(discover_data, entity_details, dictionary_search, sql_execution),
shipping a ktx-research skill, and introducing an HTTP-only ktx mcp
daemon so external agents can use KTX as a research-capable context
layer.

* Refine research-agent MCP tools spec after adversarial review iteration 1

* Refine research-agent MCP tools spec after adversarial review iteration 2

* Refine research-agent MCP tools spec after adversarial review iteration 3

* Refine spec: drop connectionName compat carve-out and ground summary/snippet provenance per kind

* feat(daemon): validate read-only SQL with sqlglot

* feat(context): expose read-only SQL validation port

* feat(context): register MCP sql execution tool

* feat(context): execute MCP SQL through validated connector path

* test(context): update SQL analysis port fixtures

* docs: add research-agent MCP sql execution foundation plan

* feat(context): add scan-backed entity details service

* feat(context): register MCP entity details tool

* feat(context): expose local MCP entity details

* test(context): align entity details scan fixtures

* docs: add research-agent MCP entity_details plan

* feat(context): add dictionary search service

* feat(context): register MCP dictionary search tool

* feat(context): expose local MCP dictionary search

* docs: add research-agent MCP dictionary_search plan

* feat: add MCP discover data service

* feat: expose discover data MCP tool

* feat: wire local discover data MCP port

* docs: add research-agent MCP discover_data plan

* feat(cli): add mcp http security helpers

* feat(cli): host mcp over streamable http

* feat(cli): manage mcp daemon lifecycle

* feat(cli): add ktx mcp commands

* fix(cli): stabilize mcp daemon verification

* docs: add research-agent MCP http daemon plan

* feat(cli): install KTX research skill

* feat(cli): configure MCP clients in setup agents

* feat(cli): support Claude local MCP setup scope

* docs: add research-agent MCP setup-agents plan

* refactor(context): use connectionId in warehouse verification tools

* docs(context): update ingest verification prompts for connectionId

* docs: add research-agent MCP ingest contract convergence plan

* chore: build runtime artifacts in conductor setup

---------

Co-authored-by: Andrey Avtomonov <7889985+andreybavt@users.noreply.github.com>
This commit is contained in:
Andrey Avtomonov 2026-05-15 02:35:09 +02:00 • committed by GitHub
parent c7b64379bf
commit b759a4a286
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
78 changed files with 13689 additions and 190 deletions
Download patch file Download diff file Expand all files Collapse all files

939
docs/superpowers/plans/2026-05-14-research-agent-mcp-dictionary-search.md Normal file
View file

@ -0,0 +1,939 @@
# Research Agent MCP Dictionary Search Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add the MCP-shaped `dictionary_search` tool so external research agents can resolve user-mentioned literal values to profile-sampled warehouse columns.
**Architecture:** Reuse the existing relationship-profile dictionary extraction as the source of truth, add a focused local dictionary-search service that reports coverage and non-authoritative misses per connection, then register the service through the MCP context tool surface and local project ports. The service re-reads the latest profile artifact on each call instead of keeping a long-lived cache, so scan freshness is correct for the MCP daemon v1.
**Tech Stack:** TypeScript, Vitest, Zod, KTX local file store, relationship-profile artifacts, KTX MCP context ports.
---
## Current Audit
Original spec: `docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md`
Implemented v1 slices:
- `docs/superpowers/plans/2026-05-14-research-agent-mcp-sql-execution-foundation.md` is implemented. Current source has sqlglot read-only validation in `python/ktx-daemon/src/ktx_daemon/sql_analysis.py`, `SqlAnalysisPort.validateReadOnly()` in `packages/context/src/sql-analysis/ports.ts`, MCP `sql_execution` registration in `packages/context/src/mcp/context-tools.ts`, and local connector execution gated by validation in `packages/context/src/mcp/local-project-ports.ts`.
- `docs/superpowers/plans/2026-05-14-research-agent-mcp-entity-details.md` is implemented. Current source has `packages/context/src/scan/entity-details.ts`, MCP `entity_details` registration in `packages/context/src/mcp/context-tools.ts`, and local project wiring in `packages/context/src/mcp/local-project-ports.ts`.
V1-blocking gaps remaining against the original spec:
- `dictionary_search` is not registered on the MCP surface and `KtxMcpContextPorts` has no dictionary-search port.
- `discover_data` is not registered on the MCP surface and the unified ranked result shape is not implemented.
- The ingest-side warehouse-verification tools still use `connectionName` / `targets` / `rowLimit` contracts and have not been fully converged with shared MCP-shaped services.
- `ktx mcp start|stop|status|logs` and the HTTP Streamable MCP daemon do not exist.
- `ktx setup-agents` does not install MCP client config entries or the `ktx-research` skill.
This plan covers only the next focused blocker: MCP `dictionary_search`. Later plans still need to cover `discover_data`, ingest contract convergence, the HTTP daemon, and setup-agent/research-skill installation.
Non-blocking or explicitly out-of-scope gaps:
- Python code execution over MCP.
- Stdio MCP transport.
- OS-level auto-start.
- Native TLS, audit logging, rate limiting, per-tool authorization, and multi-project daemon routing.
- Streaming SQL results.
## File Structure
Create:
- `packages/context/src/sl/dictionary-search.ts`
- Reads the latest `relationship-profile.json` per searched connection.
- Uses `loadLatestSlDictionaryEntries()` for dictionary entries.
- Returns spec-shaped `searched` coverage records, matches, and per-value miss reasons.
- Re-reads artifacts per call rather than caching, satisfying MCP freshness for v1.
- `packages/context/src/sl/dictionary-search.test.ts`
- Covers matches, non-authoritative misses, missing profile artifacts, no candidate columns, case-insensitive substring matching, and connection scoping.
Modify:
- `packages/context/src/sl/index.ts`
- Export the new service and response types.
- `packages/context/src/mcp/types.ts`
- Add `KtxDictionarySearchMcpPort` and include `dictionarySearch` in `KtxMcpContextPorts`.
- `packages/context/src/mcp/context-tools.ts`
- Add the `dictionary_search` Zod schema and registration.
- `packages/context/src/mcp/server.test.ts`
- Assert MCP registration and structured output for `dictionary_search`.
- `packages/context/src/mcp/local-project-ports.ts`
- Wire local project dictionary search to the new service.
- `packages/context/src/mcp/local-project-ports.test.ts`
- Cover local-port `dictionary_search` success and missing-profile behavior.
- `packages/context/src/mcp/index.ts`
- Export the new MCP port type if it is not already covered by existing barrel exports.
## Task 1: Add The Dictionary Search Service
**Files:**
- Create: `packages/context/src/sl/dictionary-search.test.ts`
- Create: `packages/context/src/sl/dictionary-search.ts`
- Modify: `packages/context/src/sl/index.ts`
- [ ] **Step 1: Write failing service tests**
Create `packages/context/src/sl/dictionary-search.test.ts`:
```typescript
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { initKtxProject, type KtxLocalProject } from '../project/index.js';
import { createKtxDictionarySearchService } from './dictionary-search.js';
describe('createKtxDictionarySearchService', () => {
let tempDir: string;
let project: KtxLocalProject;
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-dictionary-search-'));
project = await initKtxProject({ projectDir: join(tempDir, 'project'), projectName: 'warehouse' });
project.config.connections.warehouse = { driver: 'postgres', url: 'env:DATABASE_URL' };
project.config.connections.billing = { driver: 'postgres', url: 'env:BILLING_DATABASE_URL' };
});
afterEach(async () => {
await rm(tempDir, { recursive: true, force: true });
});
async function seedProfile(input: {
connectionId: string;
syncId: string;
columns: Record<string, unknown>;
}): Promise<void> {
await project.fileStore.writeFile(
`raw-sources/${input.connectionId}/live-database/${input.syncId}/enrichment/relationship-profile.json`,
`${JSON.stringify(
{
connectionId: input.connectionId,
driver: 'postgres',
sqlAvailable: true,
queryCount: 4,
tables: [],
columns: input.columns,
warnings: [],
},
null,
2,
)}\n`,
'ktx',
'ktx@example.com',
'Seed relationship profile',
);
}
it('returns matches and non-authoritative misses across configured connections', async () => {
await seedProfile({
connectionId: 'warehouse',
syncId: 'sync-1',
columns: {
'orders.status': {
table: { catalog: null, db: 'public', name: 'orders' },
column: 'status',
nativeType: 'text',
normalizedType: 'string',
distinctCount: 3,
sampleValues: ['paid', 'refunded', 'pending'],
},
},
});
await seedProfile({
connectionId: 'billing',
syncId: 'sync-2',
columns: {
'customers.name': {
table: { catalog: null, db: 'public', name: 'customers' },
column: 'name',
nativeType: 'text',
normalizedType: 'string',
distinctCount: 4,
sampleValues: ['Acme Corp', 'Globex'],
},
},
});
const service = createKtxDictionarySearchService(project);
await expect(service.search({ values: ['PAID', 'missing'] })).resolves.toEqual({
searched: [
{
connectionId: 'billing',
coverage: {
sampledRows: null,
valuesPerColumn: null,
profiledColumns: 1,
syncId: 'sync-2',
profiledAt: null,
},
status: 'ready',
},
{
connectionId: 'warehouse',
coverage: {
sampledRows: null,
valuesPerColumn: null,
profiledColumns: 1,
syncId: 'sync-1',
profiledAt: null,
},
status: 'ready',
},
],
results: [
{
value: 'PAID',
matches: [
{
connectionId: 'warehouse',
sourceName: 'orders',
columnName: 'status',
matchedValue: 'paid',
cardinality: 3,
},
],
misses: [{ connectionId: 'billing', reason: 'value_not_in_sample' }],
},
{
value: 'missing',
matches: [],
misses: [
{ connectionId: 'billing', reason: 'value_not_in_sample' },
{ connectionId: 'warehouse', reason: 'value_not_in_sample' },
],
},
],
});
});
it('distinguishes missing profile artifacts from profiles with no candidate columns', async () => {
await seedProfile({
connectionId: 'billing',
syncId: 'sync-empty',
columns: {
'events.id': {
table: { catalog: null, db: 'public', name: 'events' },
column: 'id',
nativeType: 'integer',
normalizedType: 'integer',
distinctCount: 100,
sampleValues: [1, 2, 3],
},
},
});
const service = createKtxDictionarySearchService(project);
await expect(service.search({ values: ['Acme'] })).resolves.toEqual({
searched: [
{
connectionId: 'billing',
coverage: {
sampledRows: null,
valuesPerColumn: null,
profiledColumns: 0,
syncId: 'sync-empty',
profiledAt: null,
},
status: 'no_candidate_columns',
},
{
connectionId: 'warehouse',
coverage: {
sampledRows: null,
valuesPerColumn: null,
profiledColumns: 0,
syncId: null,
profiledAt: null,
},
status: 'no_profile_artifact',
},
],
results: [
{
value: 'Acme',
matches: [],
misses: [
{ connectionId: 'billing', reason: 'no_candidate_columns' },
{ connectionId: 'warehouse', reason: 'no_profile_artifact' },
],
},
],
});
});
it('scopes search to the requested connection', async () => {
await seedProfile({
connectionId: 'warehouse',
syncId: 'sync-1',
columns: {
'orders.status': {
table: { catalog: null, db: 'public', name: 'orders' },
column: 'status',
nativeType: 'text',
normalizedType: 'string',
distinctCount: 3,
sampleValues: ['paid'],
},
},
});
await seedProfile({
connectionId: 'billing',
syncId: 'sync-2',
columns: {
'invoices.status': {
table: { catalog: null, db: 'public', name: 'invoices' },
column: 'status',
nativeType: 'text',
normalizedType: 'string',
distinctCount: 2,
sampleValues: ['paid'],
},
},
});
const service = createKtxDictionarySearchService(project);
await expect(service.search({ connectionId: 'billing', values: ['paid'] })).resolves.toMatchObject({
searched: [{ connectionId: 'billing', status: 'ready' }],
results: [
{
value: 'paid',
matches: [{ connectionId: 'billing', sourceName: 'invoices', columnName: 'status', matchedValue: 'paid' }],
misses: [],
},
],
});
});
});
```
- [ ] **Step 2: Run service tests to verify they fail**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/sl/dictionary-search.test.ts
```
Expected: FAIL with `Cannot find module './dictionary-search.js'`.
- [ ] **Step 3: Implement the dictionary search service**
Create `packages/context/src/sl/dictionary-search.ts`:
```typescript
import type { KtxLocalProject } from '../project/index.js';
import { loadLatestSlDictionaryEntries, type SlDictionaryEntry } from './sl-dictionary-profile.js';
export type KtxDictionarySearchStatus = 'ready' | 'no_profile_artifact' | 'no_candidate_columns';
export type KtxDictionarySearchMissReason = 'no_profile_artifact' | 'no_candidate_columns' | 'value_not_in_sample';
export interface KtxDictionarySearchInput {
values: string[];
connectionId?: string;
}
export interface KtxDictionarySearchCoverage {
sampledRows: number | null;
valuesPerColumn: number | null;
profiledColumns: number;
syncId: string | null;
profiledAt: string | null;
}
export interface KtxDictionarySearchSearchedConnection {
connectionId: string;
coverage: KtxDictionarySearchCoverage;
status: KtxDictionarySearchStatus;
}
export interface KtxDictionarySearchMatch {
connectionId: string;
sourceName: string;
columnName: string;
matchedValue: string;
cardinality: number | null;
}
export interface KtxDictionarySearchMiss {
connectionId: string;
reason: KtxDictionarySearchMissReason;
}
export interface KtxDictionarySearchValueResult {
value: string;
matches: KtxDictionarySearchMatch[];
misses: KtxDictionarySearchMiss[];
}
export interface KtxDictionarySearchResponse {
searched: KtxDictionarySearchSearchedConnection[];
results: KtxDictionarySearchValueResult[];
}
interface RelationshipProfileArtifact {
connectionId?: string;
profileSampleRows?: unknown;
sampleValuesPerColumn?: unknown;
profiledAt?: unknown;
extractedAt?: unknown;
}
function uniqueSorted(values: Iterable<string>): string[] {
return [...new Set([...values].filter((value) => value.trim().length > 0))].sort((left, right) =>
left.localeCompare(right),
);
}
function latestProfileSyncId(path: string): string | null {
const parts = path.split('/');
return parts.at(-3) ?? null;
}
function optionalNumber(value: unknown): number | null {
return typeof value === 'number' && Number.isFinite(value) ? value : null;
}
function optionalString(value: unknown): string | null {
return typeof value === 'string' && value.trim().length > 0 ? value : null;
}
async function latestProfilePath(project: KtxLocalProject, connectionId: string): Promise<string | null> {
const root = `raw-sources/${connectionId}/live-database`;
let files: string[];
try {
files = (await project.fileStore.listFiles(root)).files;
} catch {
return null;
}
return files
.filter((path) => path.endsWith('/enrichment/relationship-profile.json'))
.sort((left, right) => left.localeCompare(right))
.at(-1) ?? null;
}
async function readProfile(project: KtxLocalProject, path: string): Promise<RelationshipProfileArtifact> {
const raw = await project.fileStore.readFile(path);
const parsed = JSON.parse(raw.content) as unknown;
return typeof parsed === 'object' && parsed !== null && !Array.isArray(parsed)
? (parsed as RelationshipProfileArtifact)
: {};
}
function profiledColumnCount(entries: readonly SlDictionaryEntry[]): number {
return new Set(entries.map((entry) => `${entry.sourceName}\u001f${entry.columnName}`)).size;
}
async function searchedConnection(
project: KtxLocalProject,
connectionId: string,
entries: readonly SlDictionaryEntry[],
): Promise<KtxDictionarySearchSearchedConnection> {
const path = await latestProfilePath(project, connectionId);
if (!path) {
return {
connectionId,
coverage: {
sampledRows: null,
valuesPerColumn: null,
profiledColumns: 0,
syncId: null,
profiledAt: null,
},
status: 'no_profile_artifact',
};
}
const profile = await readProfile(project, path);
const count = profiledColumnCount(entries);
return {
connectionId,
coverage: {
sampledRows: optionalNumber(profile.profileSampleRows),
valuesPerColumn: optionalNumber(profile.sampleValuesPerColumn),
profiledColumns: count,
syncId: latestProfileSyncId(path),
profiledAt: optionalString(profile.profiledAt) ?? optionalString(profile.extractedAt),
},
status: count > 0 ? 'ready' : 'no_candidate_columns',
};
}
function entryMatchesValue(entry: SlDictionaryEntry, value: string): boolean {
return entry.value.toLowerCase().includes(value.toLowerCase());
}
function toMatch(entry: SlDictionaryEntry): KtxDictionarySearchMatch {
return {
connectionId: entry.connectionId,
sourceName: entry.sourceName,
columnName: entry.columnName,
matchedValue: entry.value,
cardinality: entry.cardinality,
};
}
function sortMatches(matches: KtxDictionarySearchMatch[]): KtxDictionarySearchMatch[] {
return matches.sort(
(left, right) =>
left.connectionId.localeCompare(right.connectionId) ||
left.sourceName.localeCompare(right.sourceName) ||
left.columnName.localeCompare(right.columnName) ||
left.matchedValue.localeCompare(right.matchedValue),
);
}
function missReason(status: KtxDictionarySearchStatus): KtxDictionarySearchMissReason {
return status === 'ready' ? 'value_not_in_sample' : status;
}
export function createKtxDictionarySearchService(project: KtxLocalProject) {
return {
async search(input: KtxDictionarySearchInput): Promise<KtxDictionarySearchResponse> {
const connectionIds = input.connectionId ? [input.connectionId] : uniqueSorted(Object.keys(project.config.connections));
const entries = await loadLatestSlDictionaryEntries(project, connectionIds);
const entriesByConnection = new Map<string, SlDictionaryEntry[]>();
for (const connectionId of connectionIds) {
entriesByConnection.set(
connectionId,
entries.filter((entry) => entry.connectionId === connectionId),
);
}
const searched = (
await Promise.all(
connectionIds.map((connectionId) =>
searchedConnection(project, connectionId, entriesByConnection.get(connectionId) ?? []),
),
)
).sort((left, right) => left.connectionId.localeCompare(right.connectionId));
const searchedByConnection = new Map(searched.map((connection) => [connection.connectionId, connection]));
return {
searched,
results: input.values.map((value) => {
const matches = sortMatches(entries.filter((entry) => entryMatchesValue(entry, value)).map(toMatch));
const matchedConnections = new Set(matches.map((match) => match.connectionId));
return {
value,
matches,
misses: searched
.filter((connection) => !matchedConnections.has(connection.connectionId))
.map((connection) => ({
connectionId: connection.connectionId,
reason: missReason(searchedByConnection.get(connection.connectionId)?.status ?? 'no_profile_artifact'),
})),
};
}),
};
},
};
}
```
- [ ] **Step 4: Export the service**
In `packages/context/src/sl/index.ts`, add:
```typescript
export {
createKtxDictionarySearchService,
} from './dictionary-search.js';
export type {
KtxDictionarySearchCoverage,
KtxDictionarySearchInput,
KtxDictionarySearchMatch,
KtxDictionarySearchMiss,
KtxDictionarySearchMissReason,
KtxDictionarySearchResponse,
KtxDictionarySearchSearchedConnection,
KtxDictionarySearchStatus,
KtxDictionarySearchValueResult,
} from './dictionary-search.js';
```
- [ ] **Step 5: Run service tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/sl/dictionary-search.test.ts
```
Expected: PASS.
- [ ] **Step 6: Commit the service slice**
Run:
```bash
git add packages/context/src/sl/dictionary-search.ts packages/context/src/sl/dictionary-search.test.ts packages/context/src/sl/index.ts
git commit -m "feat(context): add dictionary search service"
```
## Task 2: Register The MCP `dictionary_search` Tool
**Files:**
- Modify: `packages/context/src/mcp/types.ts`
- Modify: `packages/context/src/mcp/context-tools.ts`
- Modify: `packages/context/src/mcp/server.test.ts`
- Modify: `packages/context/src/mcp/index.ts`
- [ ] **Step 1: Add MCP port types**
In `packages/context/src/mcp/types.ts`, extend the imports:
```typescript
import type { KtxDictionarySearchInput, KtxDictionarySearchResponse } from '../sl/index.js';
```
Add this interface near the other MCP port interfaces:
```typescript
export interface KtxDictionarySearchMcpPort {
search(input: KtxDictionarySearchInput): Promise<KtxDictionarySearchResponse>;
}
```
Add the new optional port to `KtxMcpContextPorts`:
```typescript
export interface KtxMcpContextPorts {
connections?: KtxConnectionsMcpPort;
knowledge?: KtxKnowledgeMcpPort;
semanticLayer?: KtxSemanticLayerMcpPort;
entityDetails?: KtxEntityDetailsMcpPort;
dictionarySearch?: KtxDictionarySearchMcpPort;
sqlExecution?: KtxSqlExecutionMcpPort;
ingest?: KtxIngestMcpPort;
scan?: KtxScanMcpPort;
}
```
- [ ] **Step 2: Write failing MCP registration test**
In `packages/context/src/mcp/server.test.ts`, update the type import list to include:
```typescript
KtxDictionarySearchMcpPort,
```
Add this test after the `entity_details` registration test:
```typescript
it('registers dictionary_search when the host provides a dictionary-search port', async () => {
const fake = makeFakeServer();
const dictionarySearch: KtxDictionarySearchMcpPort = {
search: vi.fn<KtxDictionarySearchMcpPort['search']>().mockResolvedValue({
searched: [
{
connectionId: 'warehouse',
coverage: {
sampledRows: null,
valuesPerColumn: null,
profiledColumns: 1,
syncId: 'sync-1',
profiledAt: null,
},
status: 'ready',
},
],
results: [
{
value: 'paid',
matches: [
{
connectionId: 'warehouse',
sourceName: 'orders',
columnName: 'status',
matchedValue: 'paid',
cardinality: 3,
},
],
misses: [],
},
],
}),
};
createKtxMcpServer({
server: fake.server,
userContext: { userId: 'local-user' },
contextTools: { dictionarySearch },
});
expect(fake.tools.map((tool) => tool.name)).toEqual(['dictionary_search']);
await expect(
getTool(fake.tools, 'dictionary_search').handler({
connectionId: 'warehouse',
values: ['paid'],
}),
).resolves.toMatchObject({
structuredContent: {
searched: [{ connectionId: 'warehouse', status: 'ready' }],
results: [
{
value: 'paid',
matches: [{ connectionId: 'warehouse', sourceName: 'orders', columnName: 'status' }],
misses: [],
},
],
},
});
expect(dictionarySearch.search).toHaveBeenCalledWith({
connectionId: 'warehouse',
values: ['paid'],
});
});
```
- [ ] **Step 3: Run failing MCP registration test**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts -t "dictionary_search"
```
Expected: FAIL because `dictionary_search` is not registered.
- [ ] **Step 4: Add the MCP schema and registration**
In `packages/context/src/mcp/context-tools.ts`, add the input schema near the other research schemas:
```typescript
const dictionarySearchSchema = z.object({
values: z.array(z.string().min(1)).min(1).max(20),
connectionId: connectionIdSchema.optional(),
});
```
Add this registration block after `entity_details` and before `sql_execution`:
```typescript
if (ports.dictionarySearch) {
const dictionarySearch = ports.dictionarySearch;
registerParsedTool(
server,
'dictionary_search',
{
title: 'Dictionary Search',
description:
'Search profile-sampled warehouse values and report matching connection/source/column locations plus non-authoritative miss reasons.',
inputSchema: dictionarySearchSchema.shape,
},
dictionarySearchSchema,
async (input) => jsonToolResult(await dictionarySearch.search(input)),
);
}
```
- [ ] **Step 5: Confirm MCP barrel exports**
Open `packages/context/src/mcp/index.ts`. If it exports from `./types.js`, no change is needed. If it lists named type exports, add `KtxDictionarySearchMcpPort` to that list.
- [ ] **Step 6: Run MCP registration test**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts -t "dictionary_search"
```
Expected: PASS.
- [ ] **Step 7: Commit MCP registration**
Run:
```bash
git add packages/context/src/mcp/types.ts packages/context/src/mcp/context-tools.ts packages/context/src/mcp/server.test.ts packages/context/src/mcp/index.ts
git commit -m "feat(context): register MCP dictionary search tool"
```
## Task 3: Wire Local Project MCP Ports
**Files:**
- Modify: `packages/context/src/mcp/local-project-ports.ts`
- Modify: `packages/context/src/mcp/local-project-ports.test.ts`
- [ ] **Step 1: Write failing local-port tests**
In `packages/context/src/mcp/local-project-ports.test.ts`, add this test after the entity-details local-port tests:
```typescript
it('exposes local dictionary search through MCP ports', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
};
await project.fileStore.writeFile(
'raw-sources/warehouse/live-database/sync-1/enrichment/relationship-profile.json',
`${JSON.stringify(
{
connectionId: 'warehouse',
driver: 'postgres',
sqlAvailable: true,
queryCount: 4,
tables: [],
columns: {
'orders.status': {
table: { catalog: null, db: 'public', name: 'orders' },
column: 'status',
nativeType: 'text',
normalizedType: 'string',
distinctCount: 2,
sampleValues: ['paid', 'refunded'],
},
},
warnings: [],
},
null,
2,
)}\n`,
'ktx',
'ktx@example.com',
'Seed dictionary profile',
);
const ports = createLocalProjectMcpContextPorts(project);
await expect(ports.dictionarySearch?.search({ values: ['paid'] })).resolves.toMatchObject({
searched: [{ connectionId: 'warehouse', status: 'ready' }],
results: [
{
value: 'paid',
matches: [{ connectionId: 'warehouse', sourceName: 'orders', columnName: 'status', matchedValue: 'paid' }],
misses: [],
},
],
});
});
it('reports missing local dictionary profiles through MCP ports', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
};
const ports = createLocalProjectMcpContextPorts(project);
await expect(ports.dictionarySearch?.search({ values: ['paid'] })).resolves.toEqual({
searched: [
{
connectionId: 'warehouse',
coverage: {
sampledRows: null,
valuesPerColumn: null,
profiledColumns: 0,
syncId: null,
profiledAt: null,
},
status: 'no_profile_artifact',
},
],
results: [
{
value: 'paid',
matches: [],
misses: [{ connectionId: 'warehouse', reason: 'no_profile_artifact' }],
},
],
});
});
```
- [ ] **Step 2: Run failing local-port tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/local-project-ports.test.ts -t "dictionary"
```
Expected: FAIL because `ports.dictionarySearch` is undefined.
- [ ] **Step 3: Wire the local port**
In `packages/context/src/mcp/local-project-ports.ts`, update the SL import block to include:
```typescript
createKtxDictionarySearchService,
```
Add this port to the `ports` object returned by `createLocalProjectMcpContextPorts()` near `entityDetails`:
```typescript
dictionarySearch: {
async search(input) {
return createKtxDictionarySearchService(project).search(input);
},
},
```
- [ ] **Step 4: Run local-port tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/local-project-ports.test.ts -t "dictionary"
```
Expected: PASS.
- [ ] **Step 5: Commit local-port wiring**
Run:
```bash
git add packages/context/src/mcp/local-project-ports.ts packages/context/src/mcp/local-project-ports.test.ts
git commit -m "feat(context): expose local MCP dictionary search"
```
## Task 4: Final Verification
**Files:**
- Verify all files changed in Tasks 1-3.
- [ ] **Step 1: Run focused tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/sl/dictionary-search.test.ts src/mcp/server.test.ts src/mcp/local-project-ports.test.ts
```
Expected: PASS for dictionary-search service, MCP registration, and local-port coverage.
- [ ] **Step 2: Run context type-check**
Run:
```bash
pnpm --filter @ktx/context run type-check
```
Expected: PASS.
- [ ] **Step 3: Inspect diff**
Run:
```bash
git status --short
git diff --stat HEAD
```
Expected: only the dictionary-search service, MCP type/registration, tests, and exports changed.
- [ ] **Step 4: Commit verification note if needed**
If the previous tasks already committed all source changes, do not create an empty commit. If a small follow-up fix was required during verification, commit only those files:
```bash
git add packages/context/src/sl/dictionary-search.ts packages/context/src/sl/dictionary-search.test.ts packages/context/src/sl/index.ts packages/context/src/mcp/types.ts packages/context/src/mcp/context-tools.ts packages/context/src/mcp/server.test.ts packages/context/src/mcp/index.ts packages/context/src/mcp/local-project-ports.ts packages/context/src/mcp/local-project-ports.test.ts
git commit -m "test(context): cover MCP dictionary search"
```

1315
docs/superpowers/plans/2026-05-14-research-agent-mcp-discover-data.md Normal file
View file

File diff suppressed because it is too large Load diff

1175
docs/superpowers/plans/2026-05-14-research-agent-mcp-entity-details.md Normal file
View file

File diff suppressed because it is too large Load diff

1561
docs/superpowers/plans/2026-05-14-research-agent-mcp-http-daemon.md Normal file
View file

File diff suppressed because it is too large Load diff

804
docs/superpowers/plans/2026-05-14-research-agent-mcp-ingest-contract-convergence.md Normal file
View file

@ -0,0 +1,804 @@
# Research Agent MCP Ingest Contract Convergence Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Finish the v1 research-agent MCP spec by converging the existing ingest warehouse-verification tools on `connectionId` terminology and a shared raw-schema catalog service.
**Architecture:** Move the existing warehouse catalog reader out of the ingest-only tool folder into `packages/context/src/scan/warehouse-catalog.ts`, rename its public contract from `connectionName` to `connectionId`, and make the ingest adapters consume that shared service. Keep the ingest tools' ingest-specific output shape (`markdown` plus `structured`) and their existing `targets` / `rowLimit` controls; the v1 blocker is the divergent connection parameter and stale prompt guidance, not changing ingest output into the MCP pure-structured shape.
**Tech Stack:** TypeScript, Zod, Vitest, existing KTX local file-store scan artifacts, existing ingest BaseTool framework.
---
## Audit Summary
Implemented and no longer v1-blocking:
- MCP `sql_execution`, `entity_details`, `dictionary_search`, and `discover_data` are registered in `packages/context/src/mcp/context-tools.ts` and wired through local project ports.
- `sql_execution` is parser-gated through the Python sqlglot validator before reaching local scan connectors.
- The HTTP-only `ktx mcp` daemon exists with Streamable HTTP `POST`, `GET`, and `DELETE` handling, session tracking, host/origin checks, token checks for `/mcp`, lifecycle state, and CLI commands.
- `ktx setup-agents` installs the `ktx-research` skill, writes Claude/Cursor JSON MCP config entries, and prints Codex/opencode snippets.
Remaining v1 blocker:
- The ingest warehouse-verification tools still expose and teach `connectionName` while the spec requires `connectionId` across `warehouse-verification/*.tool.ts`, `WarehouseCatalogService`, callers, tests, and prompt assets.
Non-blocking follow-ups not covered here:
- `ktx mcp status` does not print `startedAt` as a separate line, although the state file records it.
- `ktx setup-agents` writes safe `${KTX_MCP_TOKEN}` references for shared project configs, but it does not offer the spec's optional skip prompt when token auth is active.
- `discover_data` sample-value snippets use ASCII `" - samples: "` instead of the spec prose's middle-dot separator.
## File Structure
- Move: `packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.ts` to `packages/context/src/scan/warehouse-catalog.ts`
- Shared live-database scan catalog reader, display resolver, raw schema search, and table detail source of truth.
- Modify: `packages/context/src/scan/index.ts`
- Export the shared warehouse catalog service and public types.
- Modify: `packages/context/src/ingest/tools/warehouse-verification/entity-details.tool.ts`
- Accept `connectionId`, call shared catalog service, and emit connectionId-shaped markdown and structured output.
- Modify: `packages/context/src/ingest/tools/warehouse-verification/discover-data.tool.ts`
- Accept optional `connectionId`, search raw schema via shared catalog service, and teach follow-up calls with `connectionId`.
- Modify: `packages/context/src/ingest/tools/warehouse-verification/sql-execution.tool.ts`
- Accept `connectionId`, keep `rowLimit`, and pass `connectionId` to `SlConnectionCatalogPort.executeQuery`.
- Modify tests:
- `packages/context/src/ingest/tools/warehouse-verification/entity-details.tool.test.ts`
- `packages/context/src/ingest/tools/warehouse-verification/discover-data.tool.test.ts`
- `packages/context/src/ingest/tools/warehouse-verification/sql-execution.tool.test.ts`
- `packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.test.ts`
- Rename the service test file to `packages/context/src/scan/warehouse-catalog.test.ts`.
- Modify prompt assets:
- `packages/context/skills/_shared/identifier-verification.md`
- `packages/context/skills/dbt_ingest/SKILL.md`
- `packages/context/skills/historic_sql_patterns/SKILL.md`
- `packages/context/skills/historic_sql_table_digest/SKILL.md`
- `packages/context/skills/live_database_ingest/SKILL.md`
- `packages/context/skills/looker_ingest/SKILL.md`
- `packages/context/skills/lookml_ingest/SKILL.md`
- `packages/context/skills/metabase_ingest/SKILL.md`
- `packages/context/skills/metricflow_ingest/SKILL.md`
- `packages/context/skills/notion_synthesize/SKILL.md`
- `packages/context/skills/sl_capture/SKILL.md`
- `packages/context/skills/wiki_capture/SKILL.md`
- Preserve Looker/LookML prose where `connectionName` refers to a Looker runtime field, not a KTX tool parameter.
## Task 1: Add Failing Contract Tests
**Files:**
- Modify: `packages/context/src/ingest/tools/warehouse-verification/entity-details.tool.test.ts`
- Modify: `packages/context/src/ingest/tools/warehouse-verification/discover-data.tool.test.ts`
- Modify: `packages/context/src/ingest/tools/warehouse-verification/sql-execution.tool.test.ts`
- Modify: `packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.test.ts`
- Modify: `packages/context/src/ingest/ingest-runtime-assets.test.ts`
- [ ] **Step 1: Add entity_details input-contract coverage**
Add this test inside the existing `describe('EntityDetailsTool', ...)` block:
```typescript
it('uses connectionId as the public input field', async () => {
expect(
tool.parseInput({
connectionId: 'warehouse',
targets: [{ display: 'public.orders' }],
}),
).toEqual({
connectionId: 'warehouse',
targets: [{ display: 'public.orders' }],
});
expect(() =>
tool.parseInput({
connectionName: 'warehouse',
targets: [{ display: 'public.orders' }],
}),
).toThrow();
});
```
Update the existing `tool.call(...)` inputs in the same test file from `connectionName` to `connectionId`. For example:
```typescript
const result = await tool.call({ connectionId: 'warehouse', targets: [{ display: 'public.orders' }] }, context);
```
- [ ] **Step 2: Add sql_execution input-contract coverage**
Add this test inside `packages/context/src/ingest/tools/warehouse-verification/sql-execution.tool.test.ts`:
```typescript
it('uses connectionId as the public input field', () => {
expect(
tool.parseInput({
connectionId: 'warehouse',
sql: 'select 1',
rowLimit: 5,
}),
).toEqual({
connectionId: 'warehouse',
sql: 'select 1',
rowLimit: 5,
});
expect(() =>
tool.parseInput({
connectionName: 'warehouse',
sql: 'select 1',
rowLimit: 5,
}),
).toThrow();
});
```
Update the existing `tool.call(...)` inputs in the same test file from `connectionName` to `connectionId`.
- [ ] **Step 3: Add discover_data input and hint coverage**
Update the existing discover tests so the first case calls:
```typescript
const result = await tool.call({ query: 'orders', connectionId: 'warehouse', limit: 5 }, context);
```
Change the routing-hint assertions to:
```typescript
expect(result.markdown).toContain('use `entity_details({connectionId, targets: [{display}]})`');
```
In the multi-connection test, use a `connectionId` hit field and assert the follow-up call is connectionId-shaped:
```typescript
catalog.searchByName.mockImplementation(async (connectionId: string, query: string) => [
{
kind: 'table',
connectionId,
ref: { catalog: null, db: 'public', name: `${connectionId}_${query}` },
display: `public.${connectionId}_${query}`,
matchedOn: 'name',
},
]);
const result = await tool.call({ query: 'orders', limit: 10 }, multiConnectionContext);
expect(catalog.searchByName).toHaveBeenCalledWith('analytics', 'orders', 10);
expect(catalog.searchByName).toHaveBeenCalledWith('warehouse', 'orders', 10);
expect(result.markdown).toContain('connectionId=analytics');
expect(result.markdown).toContain('connectionId=warehouse');
expect(result.markdown).toContain(
'entity_details({connectionId: "analytics", targets: [{display: "public.analytics_orders"}]})',
);
expect(result.structured.raw?.hits.map((hit) => hit.connectionId)).toEqual(['analytics', 'warehouse']);
```
Add a parse contract test:
```typescript
it('uses connectionId as the optional connection filter', () => {
expect(tool.parseInput({ query: 'orders', connectionId: 'warehouse', limit: 5 })).toEqual({
query: 'orders',
connectionId: 'warehouse',
limit: 5,
});
expect(() => tool.parseInput({ query: 'orders', connectionName: 'warehouse', limit: 5 })).toThrow();
});
```
- [ ] **Step 4: Add shared catalog output coverage**
Rename `packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.test.ts` to `packages/context/src/scan/warehouse-catalog.test.ts`.
Update the import to:
```typescript
import { WarehouseCatalogService } from './warehouse-catalog.js';
```
Update the main detail assertion to use `connectionId`:
```typescript
const detail = await catalog.getTable({ connectionId: 'warehouse', catalog: null, db: 'public', name: 'orders' });
expect(detail).toMatchObject({
connectionId: 'warehouse',
display: 'public.orders',
});
expect(detail).not.toHaveProperty('connectionName');
```
Add raw hit coverage:
```typescript
const hits = await catalog.searchByName('warehouse', 'orders', 5);
expect(hits[0]).toMatchObject({
kind: 'table',
connectionId: 'warehouse',
display: 'public.orders',
});
expect(hits[0]).not.toHaveProperty('connectionName');
```
- [ ] **Step 5: Update prompt-asset test expectations first**
In `packages/context/src/ingest/ingest-runtime-assets.test.ts`, change the identifier verification expectations to:
```typescript
expect(shared).toContain('sql_execution({connectionId, sql: "SELECT DISTINCT');
expect(shared).toContain('sql_execution({connectionId, sql: "SELECT 1 FROM');
expect(shared).not.toContain('entity_details({connectionName');
expect(shared).not.toContain('sql_execution({connectionName');
```
- [ ] **Step 6: Run focused tests and verify they fail**
Run:
```bash
pnpm --filter @ktx/context exec vitest run \
src/ingest/tools/warehouse-verification/entity-details.tool.test.ts \
src/ingest/tools/warehouse-verification/discover-data.tool.test.ts \
src/ingest/tools/warehouse-verification/sql-execution.tool.test.ts \
src/scan/warehouse-catalog.test.ts \
src/ingest/ingest-runtime-assets.test.ts
```
Expected: FAIL because schemas still require `connectionName`, the catalog service still returns `connectionName`, and the prompt asset still contains old tool-call examples.
## Task 2: Move And Rename The Shared Warehouse Catalog Service
**Files:**
- Move: `packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.ts` to `packages/context/src/scan/warehouse-catalog.ts`
- Modify: `packages/context/src/scan/index.ts`
- Delete: `packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.ts`
- [ ] **Step 1: Move the service into the scan package**
Run:
```bash
git mv packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.ts packages/context/src/scan/warehouse-catalog.ts
```
- [ ] **Step 2: Fix imports for the new location**
In `packages/context/src/scan/warehouse-catalog.ts`, change the imports at the top to:
```typescript
import { getDialectForDriver } from '../connections/index.js';
import type { KtxFileStorePort } from '../core/index.js';
import type {
KtxConnectionDriver,
KtxSchemaColumn,
KtxSchemaForeignKey,
KtxSchemaTable,
KtxTableRef,
} from './types.js';
```
- [ ] **Step 3: Rename public catalog fields and method parameters**
In `packages/context/src/scan/warehouse-catalog.ts`, rename the service's public contract to this shape:
```typescript
export interface TableDetail {
connectionId: string;
catalog: string | null;
db: string | null;
name: string;
display: string;
kind: string;
comment: string | null;
description: string | null;
rowCount: number | null;
columns: WarehouseColumnDetail[];
foreignKeys: KtxSchemaForeignKey[];
}
export type RawSchemaHit =
| {
kind: 'table';
connectionId: string;
ref: KtxTableRef;
display: string;
matchedOn: 'name' | 'db' | 'comment' | 'description';
}
| {
kind: 'column';
connectionId: string;
ref: KtxTableRef & { column: string };
display: string;
matchedOn: 'name' | 'comment' | 'description';
};
interface ConnectionCatalog {
connectionId: string;
syncId: string;
driver: CatalogDriver;
tables: KtxSchemaTable[];
profile: RelationshipProfileArtifact | null;
}
```
Update the method signatures to:
```typescript
async hasScan(connectionId: string): Promise<boolean>
async getLatestSyncId(connectionId: string): Promise<string | null>
async listTables(connectionId: string): Promise<KtxTableRef[]>
async getTable(ref: { connectionId: string } & KtxTableRef): Promise<TableDetail | null>
async resolveDisplay(connectionId: string, display: string): Promise<{ resolved: KtxTableRef | null; candidates: KtxTableRef[]; dialect: string }>
async resolveDisplayTarget(connectionId: string, display: string): Promise<DisplayTargetResolution>
async searchByName(connectionId: string, query: string, limit: number): Promise<RawSchemaHit[]>
private loadCatalog(connectionId: string): Promise<ConnectionCatalog | null>
private async readCatalog(connectionId: string): Promise<ConnectionCatalog | null>
```
Within those methods, use `connectionId` for the cache key, raw artifact root, returned `TableDetail.connectionId`, and returned `RawSchemaHit.connectionId`.
- [ ] **Step 4: Export the shared service**
Add these exports to `packages/context/src/scan/index.ts` near the existing entity-details exports:
```typescript
export type {
DisplayTargetResolution,
RawSchemaHit,
TableDetail,
WarehouseCatalogServiceDeps,
} from './warehouse-catalog.js';
export { WarehouseCatalogService } from './warehouse-catalog.js';
```
- [ ] **Step 5: Run the catalog test**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/scan/warehouse-catalog.test.ts
```
Expected: PASS.
- [ ] **Step 6: Commit the shared catalog move**
Run:
```bash
git add packages/context/src/scan/warehouse-catalog.ts packages/context/src/scan/warehouse-catalog.test.ts packages/context/src/scan/index.ts packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.ts
git commit -m "refactor(context): share warehouse catalog service"
```
## Task 3: Rename Ingest Warehouse-Verification Tool Inputs
**Files:**
- Modify: `packages/context/src/ingest/tools/warehouse-verification/entity-details.tool.ts`
- Modify: `packages/context/src/ingest/tools/warehouse-verification/discover-data.tool.ts`
- Modify: `packages/context/src/ingest/tools/warehouse-verification/sql-execution.tool.ts`
- Modify: `packages/context/src/ingest/tools/warehouse-verification/index.ts`
- [ ] **Step 1: Update imports from the shared scan service**
In `entity-details.tool.ts`, use:
```typescript
import { WarehouseCatalogService, type TableDetail } from '../../../scan/warehouse-catalog.js';
```
In `discover-data.tool.ts`, use:
```typescript
import { WarehouseCatalogService, type RawSchemaHit } from '../../../scan/warehouse-catalog.js';
```
In `index.ts`, use:
```typescript
import { WarehouseCatalogService } from '../../../scan/warehouse-catalog.js';
```
- [ ] **Step 2: Rename entity_details input and calls**
In `entity-details.tool.ts`, update the schema:
```typescript
const entityDetailsInputSchema = z.object({
connectionId: z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]*$/),
targets: z.array(targetSchema).min(1).max(50),
});
```
Update `resolveTarget`:
```typescript
async function resolveTarget(
catalog: WarehouseCatalogService,
connectionId: string,
target: EntityDetailsTarget,
): Promise<{ resolved: (KtxTableRef & { column?: string }) | null; candidates: KtxTableRef[] }> {
if ('display' in target) {
return catalog.resolveDisplayTarget(connectionId, target.display);
}
const candidateResolution = await catalog.resolveDisplayTarget(connectionId, targetLabel(target));
return {
resolved: {
catalog: target.catalog,
db: target.db,
name: target.name,
column: target.column,
},
candidates: candidateResolution.candidates,
};
}
```
Update the start of `call`:
```typescript
async call(input: EntityDetailsInput, context: ToolContext): Promise<ToolOutput<EntityDetailsStructured>> {
const allowed = allowedConnectionNames(context);
if (allowed && !allowed.has(input.connectionId)) {
return {
markdown: `Connection "${input.connectionId}" is not available to this ingest stage.`,
structured: { resolved: [], missing: [], scanAvailable: false },
};
}
const catalog = this.catalogFactory(context);
const scanAvailable = await catalog.hasScan(input.connectionId);
if (!scanAvailable) {
return {
markdown: `No live-database scan available for connection "${input.connectionId}"; run \`ktx scan\` first.`,
structured: { resolved: [], missing: [], scanAvailable: false },
};
}
```
Update the table lookup:
```typescript
const resolution = await resolveTarget(catalog, input.connectionId, target);
const detail = await catalog.getTable({ connectionId: input.connectionId, ...resolution.resolved });
```
- [ ] **Step 3: Rename sql_execution input and calls**
In `sql-execution.tool.ts`, update the schema:
```typescript
const sqlExecutionInputSchema = z.object({
connectionId: z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]*$/),
sql: z.string().min(1),
rowLimit: z.number().int().positive().max(1000).optional().default(100),
});
```
Update the allowed-connection guard:
```typescript
const allowed = context.session?.allowedConnectionNames;
if (allowed && !allowed.has(input.connectionId)) {
return {
markdown: `Connection "${input.connectionId}" is not available to this ingest stage.`,
structured: {
headers: [],
rows: [],
rowCount: 0,
truncated: false,
sql: input.sql,
wrappedSql: '',
error: 'connection_not_allowed',
},
};
}
```
Update execution:
```typescript
const result = await this.connections.executeQuery(input.connectionId, wrappedSql);
```
- [ ] **Step 4: Rename discover_data input, raw hits, and routing hints**
In `discover-data.tool.ts`, update the schema:
```typescript
const discoverDataInputSchema = z.object({
query: z.string().optional(),
connectionId: z.string().regex(/^[a-zA-Z0-9][a-zA-Z0-9_-]*$/).optional(),
limit: z.number().int().positive().max(50).optional().default(10),
sourceName: z.string().optional(),
});
```
Update the out-of-scope check:
```typescript
if (input.connectionId && allowed && !allowed.has(input.connectionId)) {
return {
markdown: `Connection "${input.connectionId}" is not available to this ingest stage.`,
structured: { wiki: null, sl: null, raw: null },
};
}
```
Update the source inspect mode:
```typescript
const sl = await this.deps.slDiscoverTool.call(
{ sourceName: input.sourceName, connectionId: input.connectionId },
context,
);
```
Update the SL discover call:
```typescript
const slResult = await this.deps.slDiscoverTool.call(
{ query: query || undefined, connectionId: input.connectionId },
context,
);
```
Update the raw search loop and hints:
```typescript
const connections = input.connectionId ? [input.connectionId] : [...(allowed ?? [])].sort();
const rawHits: RawSchemaHit[] = [];
for (const connectionId of connections) {
rawHits.push(...(await catalog.searchByName(connectionId, query, limit)));
}
if (rawHits.length > 0) {
parts.push(
'## Raw Warehouse Schema',
'> use `entity_details({connectionId, targets: [{display}]})` for full DDL + sample values',
);
parts.push(
rawHits
.slice(0, limit)
.map(
(hit) =>
`- ${hit.kind}: ${hit.display} [connectionId=${hit.connectionId}] (matched on ${hit.matchedOn}) - ` +
`follow up with \`entity_details({connectionId: "${hit.connectionId}", targets: [{display: "${hit.display}"}]})\``,
)
.join('\n'),
);
raw = { hits: rawHits.slice(0, limit) };
}
```
- [ ] **Step 5: Run focused tool tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run \
src/ingest/tools/warehouse-verification/entity-details.tool.test.ts \
src/ingest/tools/warehouse-verification/discover-data.tool.test.ts \
src/ingest/tools/warehouse-verification/sql-execution.tool.test.ts
```
Expected: PASS.
- [ ] **Step 6: Commit the ingest tool contract rename**
Run:
```bash
git add packages/context/src/ingest/tools/warehouse-verification/entity-details.tool.ts packages/context/src/ingest/tools/warehouse-verification/discover-data.tool.ts packages/context/src/ingest/tools/warehouse-verification/sql-execution.tool.ts packages/context/src/ingest/tools/warehouse-verification/index.ts packages/context/src/ingest/tools/warehouse-verification/*.test.ts
git commit -m "refactor(context): use connectionId in warehouse verification tools"
```
## Task 4: Update Prompt Assets And Runtime Tests
**Files:**
- Modify: `packages/context/skills/_shared/identifier-verification.md`
- Modify: `packages/context/skills/dbt_ingest/SKILL.md`
- Modify: `packages/context/skills/historic_sql_patterns/SKILL.md`
- Modify: `packages/context/skills/historic_sql_table_digest/SKILL.md`
- Modify: `packages/context/skills/live_database_ingest/SKILL.md`
- Modify: `packages/context/skills/looker_ingest/SKILL.md`
- Modify: `packages/context/skills/lookml_ingest/SKILL.md`
- Modify: `packages/context/skills/metabase_ingest/SKILL.md`
- Modify: `packages/context/skills/metricflow_ingest/SKILL.md`
- Modify: `packages/context/skills/notion_synthesize/SKILL.md`
- Modify: `packages/context/skills/sl_capture/SKILL.md`
- Modify: `packages/context/skills/wiki_capture/SKILL.md`
- Modify: `packages/context/src/ingest/ingest-runtime-assets.test.ts`
- [ ] **Step 1: Update the shared identifier verification protocol**
Replace the tool-call examples in `packages/context/skills/_shared/identifier-verification.md` with:
```markdown
2. `entity_details({connectionId, targets: [{display: "<identifier>"}]})` -
confirm the identifier resolves; inspect native types, FK/PK, and
sampleValues.
3. For literal values from the source, such as status codes or plan tiers,
check whether they appear in `entity_details` sampleValues for the relevant
column. If sampleValues is short or the sample may have missed real values,
run a `sql_execution` probe with the same warehouse connection id:
`sql_execution({connectionId, sql: "SELECT DISTINCT <col> FROM <ref> LIMIT 50"})`.
4. If the candidate identifier still does not resolve, do one of:
- Use `sql_execution({connectionId, sql: "SELECT 1 FROM <ref> LIMIT 0"})`.
If it errors, the identifier is fictional.
```
- [ ] **Step 2: Update copied skill assets**
In the listed `packages/context/skills/*/SKILL.md` files, replace only KTX tool-call examples:
```text
entity_details({connectionName, targets:
```
with:
```text
entity_details({connectionId, targets:
```
Replace:
```text
sql_execution({connectionName, sql:
```
with:
```text
sql_execution({connectionId, sql:
```
Replace concrete KTX tool-call examples like:
```text
sql_execution({connectionName: "warehouse", sql:
```
with:
```text
sql_execution({connectionId: "warehouse", sql:
```
In `packages/context/skills/sl_capture/SKILL.md`, replace the JSON field inside the example object:
```yaml
connectionName: "warehouse",
```
with:
```yaml
connectionId: "warehouse",
```
Do not change `packages/context/skills/looker_ingest/SKILL.md` text that defines Looker runtime `connectionName`, and do not change LookML parser docs where `connectionName` names a LookML model property.
- [ ] **Step 3: Update runtime asset tests**
In `packages/context/src/ingest/ingest-runtime-assets.test.ts`, ensure the identifier test asserts the new examples:
```typescript
expect(shared).toContain('sql_execution({connectionId, sql: "SELECT DISTINCT');
expect(shared).toContain('sql_execution({connectionId, sql: "SELECT 1 FROM');
expect(shared).not.toContain('entity_details({connectionName');
expect(shared).not.toContain('sql_execution({connectionName');
```
- [ ] **Step 4: Run prompt asset checks**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/ingest/ingest-runtime-assets.test.ts
```
Expected: PASS.
- [ ] **Step 5: Verify stale tool-call examples are gone**
Run:
```bash
rg -n "entity_details\\(\\{connectionName|sql_execution\\(\\{connectionName|connectionName=" packages/context/skills packages/context/src/ingest/ingest-runtime-assets.test.ts
```
Expected: no output. If this reports Looker/LookML prose that is not a KTX tool-call example, narrow the regex and keep the Looker/LookML prose unchanged.
- [ ] **Step 6: Commit prompt asset updates**
Run:
```bash
git add packages/context/skills packages/context/src/ingest/ingest-runtime-assets.test.ts
git commit -m "docs(context): update ingest verification prompts for connectionId"
```
## Task 5: Final Verification
**Files:**
- Verify all files changed in Tasks 1-4.
- [ ] **Step 1: Run focused research-agent ingest tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run \
src/scan/warehouse-catalog.test.ts \
src/ingest/tools/warehouse-verification/entity-details.tool.test.ts \
src/ingest/tools/warehouse-verification/discover-data.tool.test.ts \
src/ingest/tools/warehouse-verification/sql-execution.tool.test.ts \
src/ingest/ingest-runtime-assets.test.ts
```
Expected: PASS.
- [ ] **Step 2: Run context type-check**
Run:
```bash
pnpm --filter @ktx/context run type-check
```
Expected: PASS.
- [ ] **Step 3: Run dead-code checks after TypeScript changes**
Run:
```bash
pnpm run dead-code
```
Expected: PASS. If Knip reports unrelated pre-existing findings, record the exact unrelated findings in the implementation handoff and do not add broad ignores.
- [ ] **Step 4: Verify the v1-blocking old contract is gone**
Run:
```bash
rg -n "connectionName" packages/context/src/ingest/tools/warehouse-verification packages/context/src/scan/warehouse-catalog.ts packages/context/src/scan/warehouse-catalog.test.ts
```
Expected: no output.
Run:
```bash
rg -n "entity_details\\(\\{connectionName|sql_execution\\(\\{connectionName|connectionName=" packages/context/skills packages/context/src/ingest/ingest-runtime-assets.test.ts
```
Expected: no output.
- [ ] **Step 5: Inspect git status**
Run:
```bash
git status --short
```
Expected: only the intended scan catalog move, warehouse-verification tools/tests, prompt assets, and ingest runtime asset test changes are present.
- [ ] **Step 6: Commit final fixes if verification required any**
If Steps 1-5 required follow-up edits, commit those edits:
```bash
git add packages/context/src packages/context/skills
git commit -m "test(context): verify warehouse verification connectionId contract"
```
If `git status --short` is empty after the earlier task commits, skip this commit.
## Self-Review
- Spec coverage: This plan covers the remaining v1 requirement that ingest-side warehouse verification uses `connectionId` and shares the raw-schema catalog service instead of preserving a divergent `connectionName` contract.
- Placeholder scan: The plan contains no deferred-work marker phrases.
- Type consistency: The plan uses `connectionId` consistently in public tool inputs, `TableDetail`, `RawSchemaHit`, `WarehouseCatalogService` method parameters, tests, and prompt assets.

938
docs/superpowers/plans/2026-05-14-research-agent-mcp-setup-agents.md Normal file
View file

@ -0,0 +1,938 @@
# Research Agent MCP Setup Agents Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make `ktx setup-agents` install the `ktx-research` skill and configure or print MCP client entries that point agents at the local `ktx mcp` HTTP endpoint.
**Architecture:** Keep `packages/cli/src/setup-agents.ts` as the setup orchestration point. Add a small MCP-client config planner/writer in the same module, backed by `.ktx/mcp.json` when present, and install the research skill from a copied runtime asset so source checkouts and published CLI builds use the same `SKILL.md`.
**Tech Stack:** TypeScript, Vitest, Node fs/path APIs, Commander setup options, KTX MCP daemon state, JSON config writers.
---
## Current Audit
Original spec: `docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md`
Implemented v1 slices confirmed in current source:
- MCP `sql_execution`, `entity_details`, `dictionary_search`, and `discover_data` are registered in `packages/context/src/mcp/context-tools.ts`.
- Local project MCP ports wire all four tools in `packages/context/src/mcp/local-project-ports.ts`.
- Parser-backed SQL validation exists in `python/ktx-daemon/src/ktx_daemon/sql_analysis.py` and is exposed through `POST /sql/validate-read-only`.
- `ktx mcp start|stop|status|logs` exists in `packages/cli/src/commands/mcp-commands.ts`, with HTTP hosting in `packages/cli/src/mcp-http-server.ts` and daemon state in `packages/cli/src/managed-mcp-daemon.ts`.
- Targeted verification passed:
- `pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts src/search/discover.test.ts src/scan/entity-details.test.ts src/sl/dictionary-search.test.ts`
- `pnpm --filter @ktx/cli exec vitest run src/mcp-http-server.test.ts src/managed-mcp-daemon.test.ts src/commands/mcp-commands.test.ts src/setup-agents.test.ts`
V1-blocking gaps remaining against the original spec:
- `ktx setup-agents` still installs only the existing `ktx` agent files; it does not install `ktx-research`.
- `ktx setup-agents` does not write Claude Code or Cursor MCP JSON config entries.
- `ktx setup-agents` does not print Codex or opencode copy-paste snippets.
- `ktx setup-agents --remove` cannot remove written MCP JSON keys because none are written or tracked.
- The ingest-side warehouse-verification tools still use `connectionName`, `targets`, and `rowLimit`, and `WarehouseCatalogService` still exposes connection-name terminology. That is a separate v1-blocking subsystem and is not mixed into this setup-agent plan.
Non-blocking or explicitly out-of-scope gaps:
- Python code execution over MCP.
- Stdio MCP transport.
- OS-level auto-start.
- Native TLS, audit logging, rate limiting, per-tool authorization, and multi-project daemon routing.
- Streaming SQL results.
## File Structure
Create:
- `packages/cli/src/skills/research/SKILL.md`
- Canonical research skill body from the spec.
- Copied into `dist/skills/research/SKILL.md` during `@ktx/cli` build.
- `packages/cli/scripts/copy-runtime-assets.mjs`
- Copies `src/skills` into `dist/skills` after TypeScript compilation.
Modify:
- `packages/cli/package.json`
- Append the runtime asset copy step to the `build` script.
- `packages/cli/src/setup-agents.ts`
- Add `local` agent scope for Claude Code's per-project private config path.
- Add `research-skill` file entries in `plannedKtxAgentFiles()`.
- Read the research skill asset when writing research-skill entries.
- Add MCP endpoint resolution from `.ktx/mcp.json`, falling back to `http://localhost:7878/mcp`.
- Add JSON writers for Claude Code and Cursor MCP entries.
- Add printed snippets for Codex and opencode.
- Track written JSON keys in the install manifest.
- Print the daemon-start hint when the daemon is not currently running.
- `packages/cli/src/setup-agents.test.ts`
- Cover research skill install paths, MCP JSON writers, snippets, manifest removal, token handling, and no literal-token rendering.
- `packages/cli/src/commands/setup-commands.ts`
- Add `--local` for Claude Code local-scope setup.
- Reject `--local` with non-Claude targets and reject `--local --global`.
- `packages/cli/src/setup.ts`
- No behavior change beyond accepting `KtxAgentScope` with the new `local` value.
- `packages/cli/src/cli-program.ts`
- Keep the default bare setup `agentScope: 'project'`; no code change needed unless TypeScript requires the widened scope type in nearby annotations.
## Task 1: Add The Research Skill Runtime Asset
**Files:**
- Create: `packages/cli/src/skills/research/SKILL.md`
- Create: `packages/cli/scripts/copy-runtime-assets.mjs`
- Modify: `packages/cli/package.json`
- Modify: `packages/cli/src/setup-agents.test.ts`
- Modify: `packages/cli/src/setup-agents.ts`
- [ ] **Step 1: Write the failing research-skill install tests**
In `packages/cli/src/setup-agents.test.ts`, update the first test to expect `ktx-research` entries. Replace the project-scoped assertions with:
```typescript
it('plans project-scoped CLI and research files for every target', () => {
expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'claude-code', scope: 'project', mode: 'cli' })).toEqual([
{ kind: 'file', path: join(tempDir, '.claude/skills/ktx/SKILL.md'), role: 'skill' },
{ kind: 'file', path: join(tempDir, '.claude/skills/ktx-research/SKILL.md'), role: 'research-skill' },
{ kind: 'file', path: join(tempDir, '.claude/rules/ktx.md'), role: 'rule' },
]);
expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'codex', scope: 'project', mode: 'cli' })).toEqual([
{ kind: 'file', path: join(tempDir, '.agents/skills/ktx/SKILL.md'), role: 'skill' },
{ kind: 'file', path: join(tempDir, '.agents/skills/ktx-research/SKILL.md'), role: 'research-skill' },
{ kind: 'file', path: join(tempDir, '.codex/instructions/ktx.md'), role: 'rule' },
]);
expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'cursor', scope: 'project', mode: 'cli' })).toEqual([
{ kind: 'file', path: join(tempDir, '.cursor/rules/ktx.mdc') },
{ kind: 'file', path: join(tempDir, '.cursor/rules/ktx-research.mdc'), role: 'research-skill' },
]);
expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'opencode', scope: 'project', mode: 'cli' })).toEqual([
{ kind: 'file', path: join(tempDir, '.opencode/commands/ktx.md') },
{ kind: 'file', path: join(tempDir, '.opencode/commands/ktx-research.md'), role: 'research-skill' },
]);
expect(plannedKtxAgentFiles({ projectDir: tempDir, target: 'universal', scope: 'project', mode: 'cli' })).toEqual([
{ kind: 'file', path: join(tempDir, '.agents/skills/ktx/SKILL.md') },
{ kind: 'file', path: join(tempDir, '.agents/skills/ktx-research/SKILL.md'), role: 'research-skill' },
]);
});
```
Add this test after `installs target files, writes a manifest, and marks agents complete`:
```typescript
it('installs the research skill from the runtime asset', async () => {
const io = makeIo();
await expect(
runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'universal',
scope: 'project',
mode: 'cli',
skipAgents: false,
},
io.io,
),
).resolves.toMatchObject({ status: 'ready' });
const researchSkill = await readFile(join(tempDir, '.agents/skills/ktx-research/SKILL.md'), 'utf-8');
expect(researchSkill).toContain('name: ktx-research');
expect(researchSkill).toContain('Always run `discover_data` before writing SQL.');
expect(researchSkill).toContain('Treat a `dictionary_search` miss as non-authoritative.');
});
```
- [ ] **Step 2: Run tests to verify they fail**
Run:
```bash
pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts
```
Expected: FAIL because `plannedKtxAgentFiles()` does not return `ktx-research` entries and the installed research skill file does not exist.
- [ ] **Step 3: Add the research skill asset**
Create `packages/cli/src/skills/research/SKILL.md`:
```markdown
---
name: ktx-research
description: Use when answering a question that needs data from a KTX-connected database - investigating, analyzing, "how many", "show me", "what's the breakdown of", finding records by value, exploring tables, comparing periods, or any data-investigation request. Triggers even when the user does not say "research"; if the answer requires querying a configured KTX connection, this skill applies.
---
# KTX Research Workflow
You have access to KTX MCP tools for investigating data. Follow this workflow.
<workflow>
1. **Discover** - call `discover_data` first to see what exists across wiki, semantic-layer sources, and raw tables. Returns refs only.
2. **Inspect top hits in parallel** - for each promising ref:
- `kind: 'wiki'` -> `wiki_read`
- `kind: 'sl_source'`, `kind: 'sl_measure'`, or `kind: 'sl_dimension'` -> `sl_read_source`
- `kind: 'table'` or `kind: 'column'` -> `entity_details`
3. **Resolve literals** - if the user named a value such as "Acme Corp" or "status=shipped", call `dictionary_search` to find which column holds it.
4. **Query** -
- Prefer `sl_query` when the semantic layer covers the question.
- Use `sql_execution` only for questions the semantic layer does not cover.
5. **Capture learnings** - at the end of the turn, call `memory_capture` so future turns benefit. Skip when the answer carries no durable knowledge.
</workflow>
<rules>
- Always run `discover_data` before writing SQL. Do not guess table names.
- Prefer the semantic layer over raw SQL when both can answer the question; measures are the source of truth.
- Read entity details before writing SQL against an unfamiliar table. Do not assume column names.
- Treat `sql_execution` as read-only. Writes are rejected by the server.
- Validate value mentions with `dictionary_search` instead of guessing case or spelling. Treat a `dictionary_search` miss as non-authoritative. The index is built from profile-sampled values, so a missing value may simply have been outside the sample. Follow up with `sql_execution` against the most plausible columns before concluding the value is absent.
</rules>
<examples>
**Input:** "How many orders did Acme Corp place last month?"
**Workflow:**
1. `dictionary_search({ values: ["Acme Corp"] })` finds `customers.name`.
2. `discover_data({ query: "orders customer monthly" })` finds an orders semantic-layer source.
3. `sl_read_source({ connectionId: "warehouse", sourceName: "orders_facts" })` confirms the source grain, measures, and dimensions.
4. `sl_query({ connectionId: "warehouse", measures: ["order_count"], filters: ["customer_name = 'Acme Corp'"] })` answers through the semantic layer.
5. `memory_capture({ userMessage, assistantMessage })` captures the durable finding.
---
**Input:** "What columns does the events table have?"
**Workflow:**
1. `discover_data({ query: "events table" })` returns a `table` ref.
2. `entity_details({ connectionId: "warehouse", entities: [{ table: "analytics.events" }] })` returns columns, types, and foreign keys.
3. Answer directly. No query is needed.
</examples>
```
- [ ] **Step 4: Copy skill assets during CLI build**
Create `packages/cli/scripts/copy-runtime-assets.mjs`:
```javascript
import { cp, mkdir, rm } from 'node:fs/promises';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
const packageRoot = fileURLToPath(new URL('..', import.meta.url));
const skillsSource = join(packageRoot, 'src', 'skills');
const skillsTarget = join(packageRoot, 'dist', 'skills');
await rm(skillsTarget, { recursive: true, force: true });
await mkdir(dirname(skillsTarget), { recursive: true });
await cp(skillsSource, skillsTarget, { recursive: true });
```
Modify `packages/cli/package.json`:
```json
"build": "node -e \"fs.rmSync('dist', { recursive: true, force: true })\" && tsc -p tsconfig.json && node scripts/copy-runtime-assets.mjs && node ../../scripts/prepare-cli-bin.mjs"
```
- [ ] **Step 5: Add research-skill install entries and content loading**
In `packages/cli/src/setup-agents.ts`, update the manifest entry role type:
```typescript
| { kind: 'file'; path: string; role?: 'skill' | 'rule' | 'research-skill' }
```
Add this helper near `ktxCliLauncher()`:
```typescript
async function readResearchSkillContent(): Promise<string> {
const path = fileURLToPath(new URL('./skills/research/SKILL.md', import.meta.url));
const content = await readFile(path, 'utf-8');
return content.endsWith('\n') ? content : `${content}\n`;
}
```
Update `plannedKtxAgentFiles()` so every supported project target includes the `ktx-research` entry shown in Step 1. For global targets, return:
```typescript
if (input.scope === 'global') {
if (input.target === 'claude-code') {
const home = process.env.HOME ?? '';
return [
{ kind: 'file', path: join(home, '.claude/skills/ktx/SKILL.md'), role: 'skill' as const },
{ kind: 'file', path: join(home, '.claude/skills/ktx-research/SKILL.md'), role: 'research-skill' as const },
{ kind: 'file', path: join(home, '.claude/rules/ktx.md'), role: 'rule' as const },
];
}
if (input.target === 'codex') {
const codexHome = process.env.CODEX_HOME ?? join(process.env.HOME ?? '', '.codex');
return [
{ kind: 'file', path: join(codexHome, 'skills/ktx/SKILL.md'), role: 'skill' as const },
{ kind: 'file', path: join(codexHome, 'skills/ktx-research/SKILL.md'), role: 'research-skill' as const },
{ kind: 'file', path: join(codexHome, 'instructions/ktx.md'), role: 'rule' as const },
];
}
if (input.target === 'cursor' || input.target === 'opencode') {
return [];
}
throw new Error(`Global ${input.target} installation is not supported; omit --global.`);
}
```
In `installTarget()`, switch the file content selection to:
```typescript
const content =
entry.role === 'rule'
? ruleInstructionContent({ projectDir: input.projectDir })
: entry.role === 'research-skill'
? await readResearchSkillContent()
: cliInstructionContent({ projectDir: input.projectDir, launcher });
```
- [ ] **Step 6: Run tests to verify the research skill passes**
Run:
```bash
pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts
```
Expected: PASS for the research skill install tests. MCP config tests are added in the next task and will fail until implemented.
- [ ] **Step 7: Commit**
```bash
git add packages/cli/src/skills/research/SKILL.md packages/cli/scripts/copy-runtime-assets.mjs packages/cli/package.json packages/cli/src/setup-agents.ts packages/cli/src/setup-agents.test.ts
git commit -m "feat(cli): install KTX research skill"
```
## Task 2: Add MCP Client Config Planning And Rendering
**Files:**
- Modify: `packages/cli/src/setup-agents.test.ts`
- Modify: `packages/cli/src/setup-agents.ts`
- [ ] **Step 1: Write failing MCP config planner tests**
In `packages/cli/src/setup-agents.test.ts`, add these tests before `removes only manifest-listed files`:
```typescript
it('writes Claude Code project MCP config and tracks the json key', async () => {
const io = makeIo();
await expect(
runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'claude-code',
scope: 'project',
mode: 'cli',
skipAgents: false,
},
io.io,
),
).resolves.toMatchObject({ status: 'ready' });
const mcpJson = JSON.parse(await readFile(join(tempDir, '.mcp.json'), 'utf-8')) as {
mcpServers: { ktx: { type: string; url: string; headers?: Record<string, string> } };
};
expect(mcpJson.mcpServers.ktx).toEqual({ type: 'http', url: 'http://localhost:7878/mcp' });
expect(await readKtxAgentInstallManifest(tempDir)).toMatchObject({
entries: expect.arrayContaining([{ kind: 'json-key', path: join(tempDir, '.mcp.json'), jsonPath: ['mcpServers', 'ktx'] }]),
});
expect(io.stdout()).toContain('Run `ktx mcp start` to enable the configured KTX MCP server.');
});
it('writes Cursor project MCP config', async () => {
const io = makeIo();
await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'cursor',
scope: 'project',
mode: 'cli',
skipAgents: false,
},
io.io,
);
const cursorJson = JSON.parse(await readFile(join(tempDir, '.cursor/mcp.json'), 'utf-8')) as {
mcpServers: { ktx: { url: string; headers?: Record<string, string> } };
};
expect(cursorJson.mcpServers.ktx).toEqual({ url: 'http://localhost:7878/mcp' });
});
it('prints Codex and opencode snippets without mutating printed-only config files', async () => {
const codexIo = makeIo();
await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'codex',
scope: 'project',
mode: 'cli',
skipAgents: false,
},
codexIo.io,
);
expect(codexIo.stdout()).toContain('[mcp_servers.ktx]');
expect(codexIo.stdout()).toContain('url = "http://localhost:7878/mcp"');
const opencodeIo = makeIo();
await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'opencode',
scope: 'project',
mode: 'cli',
skipAgents: false,
},
opencodeIo.io,
);
expect(opencodeIo.stdout()).toContain('"mcp"');
expect(opencodeIo.stdout()).toContain('"type": "remote"');
await expect(readFile(join(tempDir, 'opencode.json'), 'utf-8')).rejects.toThrow();
});
it('uses MCP daemon state for port and token metadata without rendering literal tokens', async () => {
await mkdir(join(tempDir, '.ktx'), { recursive: true });
await writeFile(
join(tempDir, '.ktx/mcp.json'),
`${JSON.stringify(
{
schemaVersion: 1,
pid: 999999,
host: '127.0.0.1',
port: 8787,
tokenAuth: true,
projectDir: tempDir,
startedAt: '2026-05-14T00:00:00.000Z',
logPath: join(tempDir, '.ktx/logs/mcp.log'),
},
null,
2,
)}\n`,
'utf-8',
);
const io = makeIo();
const previousToken = process.env.KTX_MCP_TOKEN;
process.env.KTX_MCP_TOKEN = 'secret-token';
try {
await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'claude-code',
scope: 'project',
mode: 'cli',
skipAgents: false,
},
io.io,
);
const rendered = JSON.stringify(JSON.parse(await readFile(join(tempDir, '.mcp.json'), 'utf-8')));
expect(rendered).toContain('http://127.0.0.1:8787/mcp');
expect(rendered).toContain('Bearer ${KTX_MCP_TOKEN}');
expect(rendered).not.toContain('secret-token');
expect(io.stdout()).toContain('Run `ktx mcp start` to enable the configured KTX MCP server.');
} finally {
process.env.KTX_MCP_TOKEN = previousToken;
}
});
```
- [ ] **Step 2: Run tests to verify they fail**
Run:
```bash
pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts
```
Expected: FAIL because no MCP config writer or snippet renderer exists.
- [ ] **Step 3: Add JSON helpers and MCP endpoint resolution**
In `packages/cli/src/setup-agents.ts`, add `existsSync` and `readKtxMcpDaemonStatus` imports:
```typescript
import { existsSync } from 'node:fs';
import { readKtxMcpDaemonStatus } from './managed-mcp-daemon.js';
```
Add these types and helpers after `type InstallEntry`:
```typescript
interface KtxMcpEndpointInfo {
url: string;
tokenAuth: boolean;
running: boolean;
}
interface KtxMcpClientInstallResult {
entries: InstallEntry[];
snippets: string[];
notices: string[];
}
async function readJsonObject(path: string): Promise<Record<string, unknown>> {
if (!existsSync(path)) return {};
const parsed = JSON.parse(await readFile(path, 'utf-8')) as unknown;
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error(`Expected JSON object in ${path}`);
}
return parsed as Record<string, unknown>;
}
function objectAtPath(root: Record<string, unknown>, jsonPath: string[]): Record<string, unknown> {
let cursor = root;
for (const segment of jsonPath) {
const current = cursor[segment];
if (!current || typeof current !== 'object' || Array.isArray(current)) {
cursor[segment] = {};
}
cursor = cursor[segment] as Record<string, unknown>;
}
return cursor;
}
async function writeJsonKey(path: string, jsonPath: string[], value: unknown): Promise<void> {
const root = await readJsonObject(path);
const parent = objectAtPath(root, jsonPath.slice(0, -1));
parent[jsonPath.at(-1) as string] = value;
await mkdir(dirname(path), { recursive: true });
await writeFile(path, `${JSON.stringify(root, null, 2)}\n`, 'utf-8');
}
async function resolveMcpEndpoint(projectDir: string): Promise<KtxMcpEndpointInfo> {
const status = await readKtxMcpDaemonStatus({ projectDir }).catch(() => null);
if (status?.kind === 'running') {
return {
url: status.url,
tokenAuth: status.state.tokenAuth,
running: true,
};
}
if (status?.kind === 'stale' && status.state) {
return {
url: `http://${status.state.host}:${status.state.port}/mcp`,
tokenAuth: status.state.tokenAuth || Boolean(process.env.KTX_MCP_TOKEN),
running: false,
};
}
return {
url: 'http://localhost:7878/mcp',
tokenAuth: Boolean(process.env.KTX_MCP_TOKEN),
running: false,
};
}
```
- [ ] **Step 4: Add MCP entry renderers**
Add these helpers after `resolveMcpEndpoint()`:
```typescript
function tokenHeaders(endpoint: KtxMcpEndpointInfo): Record<string, string> | undefined {
return endpoint.tokenAuth ? { Authorization: 'Bearer ${KTX_MCP_TOKEN}' } : undefined;
}
function claudeMcpEntry(endpoint: KtxMcpEndpointInfo): Record<string, unknown> {
return {
type: 'http',
url: endpoint.url,
...(tokenHeaders(endpoint) ? { headers: tokenHeaders(endpoint) } : {}),
};
}
function cursorMcpEntry(endpoint: KtxMcpEndpointInfo): Record<string, unknown> {
return {
url: endpoint.url,
...(tokenHeaders(endpoint) ? { headers: tokenHeaders(endpoint) } : {}),
};
}
function codexSnippet(endpoint: KtxMcpEndpointInfo): string {
if (endpoint.tokenAuth) {
return [
'Codex MCP config does not currently document HTTP headers.',
'Run KTX on loopback without token auth for Codex, or configure headers after Codex documents support.',
].join('\n');
}
return [`[mcp_servers.ktx]`, `url = "${endpoint.url}"`].join('\n');
}
function opencodeSnippet(endpoint: KtxMcpEndpointInfo): string {
return JSON.stringify(
{
mcp: {
ktx: {
type: 'remote',
url: endpoint.url,
enabled: true,
...(tokenHeaders(endpoint) ? { headers: tokenHeaders(endpoint) } : {}),
},
},
},
null,
2,
);
}
function claudeConfigPath(projectDir: string, scope: KtxAgentScope): { path: string; jsonPath: string[] } {
const home = process.env.HOME ?? '';
if (scope === 'global') {
return { path: join(home, '.claude.json'), jsonPath: ['mcpServers', 'ktx'] };
}
if (scope === 'local') {
return { path: join(home, '.claude.json'), jsonPath: ['projects', resolve(projectDir), 'mcpServers', 'ktx'] };
}
return { path: join(resolve(projectDir), '.mcp.json'), jsonPath: ['mcpServers', 'ktx'] };
}
function cursorConfigPath(projectDir: string, scope: KtxAgentScope): { path: string; jsonPath: string[] } {
const home = process.env.HOME ?? '';
return {
path: scope === 'global' ? join(home, '.cursor/mcp.json') : join(resolve(projectDir), '.cursor/mcp.json'),
jsonPath: ['mcpServers', 'ktx'],
};
}
```
- [ ] **Step 5: Add the MCP client install planner**
Add this function after the snippet helpers:
```typescript
async function installMcpClientConfig(input: {
projectDir: string;
target: KtxAgentTarget;
scope: KtxAgentScope;
}): Promise<KtxMcpClientInstallResult> {
const endpoint = await resolveMcpEndpoint(input.projectDir);
const entries: InstallEntry[] = [];
const snippets: string[] = [];
const notices: string[] = [];
if (!endpoint.running) {
notices.push('Run `ktx mcp start` to enable the configured KTX MCP server.');
}
if (input.target === 'claude-code') {
const config = claudeConfigPath(input.projectDir, input.scope);
await writeJsonKey(config.path, config.jsonPath, claudeMcpEntry(endpoint));
entries.push({ kind: 'json-key', path: config.path, jsonPath: config.jsonPath });
} else if (input.target === 'cursor') {
const config = cursorConfigPath(input.projectDir, input.scope);
await writeJsonKey(config.path, config.jsonPath, cursorMcpEntry(endpoint));
entries.push({ kind: 'json-key', path: config.path, jsonPath: config.jsonPath });
} else if (input.target === 'codex') {
snippets.push(`Codex MCP snippet for ~/.codex/config.toml:\n${codexSnippet(endpoint)}`);
} else if (input.target === 'opencode') {
const path =
input.scope === 'global' ? '~/.config/opencode/opencode.json' : `${relative(input.projectDir, join(input.projectDir, 'opencode.json'))}`;
snippets.push(`opencode MCP snippet for ${path}:\n${opencodeSnippet(endpoint)}`);
}
return { entries, snippets, notices };
}
```
- [ ] **Step 6: Call the MCP planner during setup**
Keep `installTarget()` responsible only for writing agent files and returning those file entries.
In `runKtxSetupAgentsStep()`, replace the current install loop:
```typescript
const entries: InstallEntry[] = [];
for (const install of installs) entries.push(...(await installTarget({ projectDir: args.projectDir, ...install })));
```
with:
```typescript
const entries: InstallEntry[] = [];
const snippets: string[] = [];
const notices = new Set<string>();
for (const install of installs) {
entries.push(...(await installTarget({ projectDir: args.projectDir, ...install })));
const mcpResult = await installMcpClientConfig({ projectDir: args.projectDir, target: install.target, scope: install.scope });
entries.push(...mcpResult.entries);
for (const snippet of mcpResult.snippets) snippets.push(snippet);
for (const notice of mcpResult.notices) notices.add(notice);
}
```
After the install summary write:
```typescript
for (const snippet of snippets) {
io.stdout.write(`\n${snippet}\n`);
}
for (const notice of notices) {
io.stdout.write(`\n${notice}\n`);
}
```
- [ ] **Step 7: Run tests to verify MCP config passes**
Run:
```bash
pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts
```
Expected: PASS for research-skill and MCP config tests.
- [ ] **Step 8: Commit**
```bash
git add packages/cli/src/setup-agents.ts packages/cli/src/setup-agents.test.ts
git commit -m "feat(cli): configure MCP clients in setup agents"
```
## Task 3: Add Claude Local Scope
**Files:**
- Modify: `packages/cli/src/commands/setup-commands.ts`
- Modify: `packages/cli/src/setup-agents.ts`
- Modify: `packages/cli/src/setup-agents.test.ts`
- Modify: `packages/cli/src/setup.test.ts`
- Modify: `packages/cli/src/index.test.ts`
- [ ] **Step 1: Write failing local-scope tests**
Add this test to `packages/cli/src/setup-agents.test.ts`:
```typescript
it('writes Claude Code local MCP config under the project key in ~/.claude.json', async () => {
const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-'));
const previousHome = process.env.HOME;
process.env.HOME = home;
try {
const io = makeIo();
await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'claude-code',
scope: 'local',
mode: 'cli',
skipAgents: false,
},
io.io,
);
const config = JSON.parse(await readFile(join(home, '.claude.json'), 'utf-8')) as {
projects: Record<string, { mcpServers: { ktx: { type: string; url: string } } }>;
};
expect(config.projects[tempDir].mcpServers.ktx).toEqual({ type: 'http', url: 'http://localhost:7878/mcp' });
} finally {
process.env.HOME = previousHome;
await rm(home, { recursive: true, force: true });
}
});
```
Add these command-level tests after the existing `dispatches setup agent flags` test in `packages/cli/src/index.test.ts`:
```typescript
it('rejects --local with non-Claude targets', async () => {
const setup = vi.fn(async () => 0);
const setupIo = makeIo();
await expect(
runKtxCli(
['--project-dir', tempDir, 'setup', '--agents', '--target', 'cursor', '--local', '--no-input'],
setupIo.io,
{ setup },
),
).resolves.toBe(0);
expect(setupIo.stderr()).toContain('--local is only supported with --target claude-code');
expect(setup).not.toHaveBeenCalled();
});
it('rejects --local and --global together', async () => {
const setup = vi.fn(async () => 0);
const setupIo = makeIo();
await expect(
runKtxCli(
['--project-dir', tempDir, 'setup', '--agents', '--target', 'claude-code', '--local', '--global', '--no-input'],
setupIo.io,
{ setup },
),
).resolves.toBe(0);
expect(setupIo.stderr()).toContain('Choose only one agent scope: --local or --global.');
expect(setup).not.toHaveBeenCalled();
});
```
- [ ] **Step 2: Run tests to verify they fail**
Run:
```bash
pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts src/index.test.ts
```
Expected: FAIL because `KtxAgentScope` does not include `local` and the setup command has no `--local` option.
- [ ] **Step 3: Add the local scope type and command option**
In `packages/cli/src/setup-agents.ts`, change:
```typescript
export type KtxAgentScope = 'project' | 'global';
```
to:
```typescript
export type KtxAgentScope = 'project' | 'global' | 'local';
```
In `packages/cli/src/commands/setup-commands.ts`, add `local` to `isOnlyAgentOptions()`:
```typescript
'local',
```
Add the command option after `--global`:
```typescript
.option('--local', 'Install Claude Code MCP config into the private per-project ~/.claude.json scope', false)
```
In the setup action before `const mode = ...`, add:
```typescript
if (options.local && options.global) {
context.io.stderr.write('Choose only one agent scope: --local or --global.\n');
context.setExitCode(1);
return;
}
if (options.local && options.target && options.target !== 'claude-code') {
context.io.stderr.write('--local is only supported with --target claude-code.\n');
context.setExitCode(1);
return;
}
```
Replace:
```typescript
const resolvedAgentScope = options.global ? 'global' : 'project';
```
with:
```typescript
const resolvedAgentScope = options.local ? 'local' : options.global ? 'global' : 'project';
```
- [ ] **Step 4: Run local-scope tests**
Run:
```bash
pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts src/index.test.ts
```
Expected: PASS for the new local-scope coverage.
- [ ] **Step 5: Commit**
```bash
git add packages/cli/src/commands/setup-commands.ts packages/cli/src/setup-agents.ts packages/cli/src/setup-agents.test.ts packages/cli/src/setup.test.ts packages/cli/src/index.test.ts
git commit -m "feat(cli): support Claude local MCP setup scope"
```
## Task 4: Final Verification
**Files:**
- Verify all files changed in Tasks 1-3.
- [ ] **Step 1: Run focused CLI tests**
Run:
```bash
pnpm --filter @ktx/cli exec vitest run src/setup-agents.test.ts src/commands/mcp-commands.test.ts src/mcp-http-server.test.ts src/managed-mcp-daemon.test.ts
```
Expected: all selected test files pass.
- [ ] **Step 2: Run CLI type-check**
Run:
```bash
pnpm --filter @ktx/cli run type-check
```
Expected: TypeScript completes with no errors.
- [ ] **Step 3: Run CLI build**
Run:
```bash
pnpm --filter @ktx/cli run build
```
Expected: build succeeds and `packages/cli/dist/skills/research/SKILL.md` exists.
- [ ] **Step 4: Run dead-code check for the changed TypeScript surface**
Run:
```bash
pnpm run dead-code
```
Expected: Biome and Knip complete with no new findings from the setup-agent changes.
- [ ] **Step 5: Inspect git status**
Run:
```bash
git status --short
```
Expected: only intended setup-agent, skill asset, package script, and test files are modified.
## Self-Review
Spec coverage:
- Covers `ktx-research` skill installation paths for Claude Code, Codex, Cursor, opencode, and universal project targets.
- Covers Claude Code and Cursor JSON MCP writers.
- Covers Codex and opencode printed snippets.
- Covers token handling with `${KTX_MCP_TOKEN}` and no literal token rendering.
- Covers `.ktx/mcp.json` port selection and daemon-start hint.
- Covers manifest tracking for written JSON keys and removal through existing `json-key` cleanup.
Known v1 gap not covered by this plan:
- Ingest warehouse-verification contract convergence from `connectionName` to `connectionId`, shared service extraction, and caller/test updates remains v1-blocking and needs its own focused plan after this setup-agent slice lands.

999
docs/superpowers/plans/2026-05-14-research-agent-mcp-sql-execution-foundation.md Normal file
View file

@ -0,0 +1,999 @@
# Research Agent MCP SQL Execution Foundation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add the parser-backed safety prerequisite and MCP `sql_execution` surface needed before the research-agent MCP tools can safely execute warehouse SQL.
**Architecture:** Keep connector `executeReadOnly()` as the execution path, but make the MCP adapter require a sqlglot-backed validator before calling any connector. Extend the existing Python SQL-analysis daemon with a read-only validation endpoint, expose it through the TypeScript SQL-analysis port, then register an MCP `sql_execution` tool only when the host provides that validator and a local scan connector factory.
**Tech Stack:** TypeScript, Vitest, Zod, Python, pytest, FastAPI, sqlglot, KTX MCP context ports, KTX scan connectors.
---
## Audit Summary
Original spec: `docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md`
Implemented plans that overlap with the spec:
- `docs/superpowers/plans/2026-05-11-managed-agent-mcp-semantic-runtime.md` is implemented for the existing in-process MCP semantic runtime. Current evidence: `packages/context/src/mcp/context-tools.ts` registers `connection_*`, `wiki_*`, `sl_*`, `ingest_*`, and `scan_*` tools, and `packages/context/src/mcp/local-project-ports.ts` provides local ports for those surfaces.
- `docs/superpowers/plans/2026-05-12-warehouse-verification-tools.md` plus its May 12 and May 13 closure plans are implemented for ingest-only warehouse verification. Current evidence: `packages/context/src/ingest/tools/warehouse-verification/{discover-data,entity-details,sql-execution,warehouse-catalog.service}.ts` exist and are wired for ingest agents.
V1-blocking gaps remaining against the original spec:
- The public MCP research tools are not registered. `KtxMcpContextPorts` has no `discover`, `entityDetails`, `dictionarySearch`, or `sqlExecution` ports.
- The existing ingest `discover_data`, `entity_details`, and `sql_execution` tools use `connectionName`, `targets`, and `rowLimit`, and return markdown plus structured output. The spec requires MCP-shaped `connectionId`, `entities` / `maxRows`, and pure structured outputs.
- `sql_execution` cannot be safely exposed yet: `packages/context/src/connections/read-only-sql.ts` still uses first-token regex checks. The spec requires a sqlglot/AST-backed guard or connector-side read-only session before MCP registration.
- `packages/context/src/scan/entity-details.ts`, `packages/context/src/sl/dictionary-search.ts`, and `packages/context/src/search/discover.ts` do not exist.
- `WarehouseCatalogService` caches by connection only and does not invalidate when latest scan artifact identity advances.
- `dictionary_search` has no MCP service, no coverage metadata, and no per-connection miss reasons.
- `discover_data` has no unified ranked MCP result shape with `summary`, `snippet`, `matchedOn`, `kind`, `tableRef`, and RRF fusion across wiki, SL, and raw schema.
- `ktx mcp start|stop|status|logs` does not exist, and no HTTP Streamable MCP daemon exists.
- `ktx setup-agents` installs only the existing `ktx` CLI skill/rules; it does not install `ktx-research` or MCP client config entries/snippets.
Non-blocking or explicitly out-of-scope gaps:
- Python code execution over MCP.
- Stdio MCP transport.
- OS-level auto-start.
- Native TLS, audit logging, rate limiting, per-tool authorization, and multi-project daemon routing.
- Streaming SQL results.
- Full DDL-style ingest `entity_details` markdown formatting and hard write-time validation in ingest writer tools.
This plan covers the first prerequisite blocker: parser-backed SQL validation and MCP `sql_execution`. The remaining v1-blocking tool, daemon, and setup-agent work stays visible for subsequent plans.
## File Structure
Create no new files.
Modify these files:
- `python/ktx-daemon/src/ktx_daemon/sql_analysis.py`: add a sqlglot-backed read-only SQL validator.
- `python/ktx-daemon/src/ktx_daemon/app.py`: expose `POST /sql/validate-read-only`.
- `python/ktx-daemon/tests/test_sql_analysis.py`: cover accepted SELECT/WITH and rejected CTE-DML, multi-statement, command, pragma, and parse-error payloads.
- `python/ktx-daemon/tests/test_app.py`: cover the new HTTP endpoint.
- `packages/context/src/sql-analysis/ports.ts`: add `validateReadOnly()` to `SqlAnalysisPort`.
- `packages/context/src/sql-analysis/http-sql-analysis-port.ts`: call `/sql/validate-read-only` and map its response.
- `packages/context/src/sql-analysis/http-sql-analysis-port.test.ts`: cover request and response mapping.
- `packages/context/src/mcp/types.ts`: add `KtxSqlExecutionMcpPort` and `sqlExecution` to `KtxMcpContextPorts`.
- `packages/context/src/mcp/context-tools.ts`: add the MCP `sql_execution` schema and registration.
- `packages/context/src/mcp/server.test.ts`: assert MCP registration and structured output for `sql_execution`.
- `packages/context/src/mcp/local-project-ports.ts`: expose local project SQL execution only when both `SqlAnalysisPort.validateReadOnly()` and a local scan connector factory are available.
- `packages/context/src/mcp/local-project-ports.test.ts`: cover validator success and validator rejection.
### Task 1: Add sqlglot Read-Only Validation
**Files:**
- Modify: `python/ktx-daemon/tests/test_sql_analysis.py`
- Modify: `python/ktx-daemon/src/ktx_daemon/sql_analysis.py`
- Modify: `python/ktx-daemon/tests/test_app.py`
- Modify: `python/ktx-daemon/src/ktx_daemon/app.py`
- [ ] **Step 1: Write failing sqlglot validator tests**
In `python/ktx-daemon/tests/test_sql_analysis.py`, update the import block to include the new request model and function:
```python
from ktx_daemon.sql_analysis import (
AnalyzeSqlBatchItem,
AnalyzeSqlBatchRequest,
ValidateReadOnlySqlRequest,
_columns_from_nodes,
analyze_sql_batch_response,
validate_read_only_sql_response,
)
```
Add these tests after `test_columns_from_nodes_ignores_non_expression_clause_values`:
```python
def test_validate_read_only_sql_accepts_select_and_with_queries() -> None:
select_response = validate_read_only_sql_response(
ValidateReadOnlySqlRequest(
dialect="postgres",
sql="select id, status from public.orders where status = 'paid'",
)
)
with_response = validate_read_only_sql_response(
ValidateReadOnlySqlRequest(
dialect="postgres",
sql=(
"with paid as (select * from public.orders where status = 'paid') "
"select count(*) from paid"
),
)
)
assert select_response.ok is True
assert select_response.error is None
assert with_response.ok is True
assert with_response.error is None
def test_validate_read_only_sql_rejects_cte_dml() -> None:
response = validate_read_only_sql_response(
ValidateReadOnlySqlRequest(
dialect="postgres",
sql="with x as (insert into audit.events values (1) returning *) select * from x",
)
)
assert response.ok is False
assert response.error == "SQL contains read/write operation: Insert"
def test_validate_read_only_sql_rejects_multi_statement_payloads() -> None:
response = validate_read_only_sql_response(
ValidateReadOnlySqlRequest(
dialect="postgres",
sql="select * from public.orders; delete from public.orders",
)
)
assert response.ok is False
assert response.error == "Only one SQL statement can be executed."
def test_validate_read_only_sql_rejects_commands_and_pragmas() -> None:
command_response = validate_read_only_sql_response(
ValidateReadOnlySqlRequest(dialect="postgres", sql="call refresh_stats()")
)
pragma_response = validate_read_only_sql_response(
ValidateReadOnlySqlRequest(dialect="sqlite", sql="pragma table_info(users)")
)
assert command_response.ok is False
assert command_response.error == "SQL contains read/write operation: Command"
assert pragma_response.ok is False
assert pragma_response.error == "SQL contains read/write operation: Pragma"
def test_validate_read_only_sql_reports_parse_errors() -> None:
response = validate_read_only_sql_response(
ValidateReadOnlySqlRequest(dialect="postgres", sql="select * from where")
)
assert response.ok is False
assert response.error is not None
assert "Invalid expression" in response.error
```
- [ ] **Step 2: Run failing Python validator tests**
Run:
```bash
source .venv/bin/activate && uv run pytest python/ktx-daemon/tests/test_sql_analysis.py -q
```
Expected: FAIL with an import error for `ValidateReadOnlySqlRequest` or `validate_read_only_sql_response`.
- [ ] **Step 3: Implement the sqlglot validator**
In `python/ktx-daemon/src/ktx_daemon/sql_analysis.py`, add this model after `AnalyzeSqlBatchResponse`:
```python
class ValidateReadOnlySqlRequest(BaseModel):
dialect: str
sql: str
class ValidateReadOnlySqlResponse(BaseModel):
ok: bool
error: str | None = None
```
Add this constant after the model definitions:
```python
_READ_ONLY_ROOT_TYPES = (exp.Select, exp.Union)
_READ_WRITE_NODE_TYPES = (
exp.Alter,
exp.Analyze,
exp.Cache,
exp.Command,
exp.Commit,
exp.Copy,
exp.Create,
exp.Delete,
exp.Describe,
exp.Drop,
exp.Execute,
exp.Grant,
exp.Insert,
exp.Merge,
exp.Pragma,
exp.Refresh,
exp.Revoke,
exp.Rollback,
exp.Set,
exp.Show,
exp.Transaction,
exp.TruncateTable,
exp.Uncache,
exp.Update,
exp.Use,
)
```
Add this function after `_analyze_payload`:
```python
def validate_read_only_sql_response(
request: ValidateReadOnlySqlRequest,
) -> ValidateReadOnlySqlResponse:
try:
statements = sqlglot.parse(request.sql, read=request.dialect)
except sqlglot.errors.SqlglotError as exc:
return ValidateReadOnlySqlResponse(ok=False, error=str(exc))
if len(statements) != 1:
return ValidateReadOnlySqlResponse(
ok=False,
error="Only one SQL statement can be executed.",
)
tree = statements[0]
if tree is None:
return ValidateReadOnlySqlResponse(ok=False, error="SQL did not parse to a statement.")
if not isinstance(tree, _READ_ONLY_ROOT_TYPES):
return ValidateReadOnlySqlResponse(
ok=False,
error=f"SQL contains read/write operation: {type(tree).__name__}",
)
for node in tree.walk():
if isinstance(node, _READ_WRITE_NODE_TYPES):
return ValidateReadOnlySqlResponse(
ok=False,
error=f"SQL contains read/write operation: {type(node).__name__}",
)
return ValidateReadOnlySqlResponse(ok=True, error=None)
```
- [ ] **Step 4: Run Python validator tests**
Run:
```bash
source .venv/bin/activate && uv run pytest python/ktx-daemon/tests/test_sql_analysis.py -q
```
Expected: PASS.
- [ ] **Step 5: Write failing HTTP endpoint test**
In `python/ktx-daemon/tests/test_app.py`, add this test after `test_sql_parse_table_identifier_endpoint`:
```python
def test_sql_validate_read_only_endpoint() -> None:
client = TestClient(create_app())
ok_response = client.post(
"/sql/validate-read-only",
json={"dialect": "postgres", "sql": "select * from public.orders"},
)
bad_response = client.post(
"/sql/validate-read-only",
json={
"dialect": "postgres",
"sql": "with x as (insert into audit.events values (1) returning *) select * from x",
},
)
assert ok_response.status_code == 200
assert ok_response.json() == {"ok": True, "error": None}
assert bad_response.status_code == 200
assert bad_response.json() == {
"ok": False,
"error": "SQL contains read/write operation: Insert",
}
```
- [ ] **Step 6: Run failing HTTP endpoint test**
Run:
```bash
source .venv/bin/activate && uv run pytest python/ktx-daemon/tests/test_app.py -q -k validate_read_only
```
Expected: FAIL with HTTP 404 for `/sql/validate-read-only`.
- [ ] **Step 7: Register the HTTP endpoint**
In `python/ktx-daemon/src/ktx_daemon/app.py`, update the SQL-analysis import to include the new symbols:
```python
from ktx_daemon.sql_analysis import (
AnalyzeSqlBatchRequest,
AnalyzeSqlBatchResponse,
ValidateReadOnlySqlRequest,
ValidateReadOnlySqlResponse,
analyze_sql_batch_response,
validate_read_only_sql_response,
)
```
Add this endpoint immediately before the existing `@app.post("/sql/analyze-batch", ...)` route:
```python
@app.post("/sql/validate-read-only", response_model=ValidateReadOnlySqlResponse)
async def sql_validate_read_only(
request: ValidateReadOnlySqlRequest,
) -> ValidateReadOnlySqlResponse:
try:
return validate_read_only_sql_response(request)
except Exception as error:
logger.exception("SQL read-only validation failed: %s", error)
raise HTTPException(
status_code=500,
detail=f"SQL read-only validation failed: {error}",
) from error
```
- [ ] **Step 8: Run Python HTTP endpoint test**
Run:
```bash
source .venv/bin/activate && uv run pytest python/ktx-daemon/tests/test_app.py -q -k validate_read_only
```
Expected: PASS.
- [ ] **Step 9: Commit Python validator**
Run:
```bash
git add python/ktx-daemon/src/ktx_daemon/sql_analysis.py python/ktx-daemon/src/ktx_daemon/app.py python/ktx-daemon/tests/test_sql_analysis.py python/ktx-daemon/tests/test_app.py
git commit -m "feat(daemon): validate read-only SQL with sqlglot"
```
### Task 2: Expose Read-Only Validation Through the TypeScript SQL-Analysis Port
**Files:**
- Modify: `packages/context/src/sql-analysis/ports.ts`
- Modify: `packages/context/src/sql-analysis/http-sql-analysis-port.test.ts`
- Modify: `packages/context/src/sql-analysis/http-sql-analysis-port.ts`
- [ ] **Step 1: Add the port contract**
In `packages/context/src/sql-analysis/ports.ts`, add this interface after `SqlAnalysisBatchResult`:
```typescript
export interface SqlReadOnlyValidationResult {
ok: boolean;
error?: string | null;
}
```
Update `SqlAnalysisPort` to include the new method:
```typescript
export interface SqlAnalysisPort {
analyzeForFingerprint(sql: string, dialect: SqlAnalysisDialect): Promise<SqlAnalysisFingerprintResult>;
analyzeBatch(
items: SqlAnalysisBatchItem[],
dialect: SqlAnalysisDialect,
): Promise<Map<string, SqlAnalysisBatchResult>>;
validateReadOnly(sql: string, dialect: SqlAnalysisDialect): Promise<SqlReadOnlyValidationResult>;
}
```
- [ ] **Step 2: Write failing HTTP port tests**
In `packages/context/src/sql-analysis/http-sql-analysis-port.test.ts`, add this test inside the existing `describe('createHttpSqlAnalysisPort', ...)` block:
```typescript
it('maps read-only SQL validation responses', async () => {
const requests: Array<{ path: string; payload: Record<string, unknown> }> = [];
const port = createHttpSqlAnalysisPort({
baseUrl: 'http://127.0.0.1:8765',
requestJson: async (path, payload) => {
requests.push({ path, payload });
return { ok: false, error: 'SQL contains read/write operation: Insert' };
},
});
await expect(port.validateReadOnly('with x as (insert into t values (1)) select * from x', 'postgres')).resolves.toEqual({
ok: false,
error: 'SQL contains read/write operation: Insert',
});
expect(requests).toEqual([
{
path: '/sql/validate-read-only',
payload: {
dialect: 'postgres',
sql: 'with x as (insert into t values (1)) select * from x',
},
},
]);
});
```
Add this test after it:
```typescript
it('rejects malformed read-only validation responses', async () => {
const port = createHttpSqlAnalysisPort({
baseUrl: 'http://127.0.0.1:8765',
requestJson: async () => ({ ok: 'yes' }),
});
await expect(port.validateReadOnly('select 1', 'postgres')).rejects.toThrow(
'sql analysis response is missing boolean field ok',
);
});
```
- [ ] **Step 3: Run failing HTTP port tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/sql-analysis/http-sql-analysis-port.test.ts
```
Expected: FAIL because `validateReadOnly` is not implemented.
- [ ] **Step 4: Implement HTTP response mapping**
In `packages/context/src/sql-analysis/http-sql-analysis-port.ts`, update the type import to include `SqlReadOnlyValidationResult`:
```typescript
SqlReadOnlyValidationResult,
```
Add this helper after `requiredStringArray`:
```typescript
function requiredBoolean(raw: Record<string, unknown>, field: string): boolean {
const value = raw[field];
if (typeof value !== 'boolean') {
throw new Error(`sql analysis response is missing boolean field ${field}`);
}
return value;
}
```
Add this mapper after `mapBatchResponse`:
```typescript
function mapReadOnlyValidation(raw: Record<string, unknown>): SqlReadOnlyValidationResult {
const error = optionalString(raw, 'error');
return {
ok: requiredBoolean(raw, 'ok'),
...(error !== undefined ? { error } : {}),
};
}
```
Add this method to the object returned by `createHttpSqlAnalysisPort`:
```typescript
async validateReadOnly(sql: string, dialect: SqlAnalysisDialect) {
const raw = await requestJson('/sql/validate-read-only', {
dialect,
sql,
});
return mapReadOnlyValidation(raw);
},
```
- [ ] **Step 5: Run HTTP port tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/sql-analysis/http-sql-analysis-port.test.ts
```
Expected: PASS.
- [ ] **Step 6: Commit TypeScript SQL-analysis port**
Run:
```bash
git add packages/context/src/sql-analysis/ports.ts packages/context/src/sql-analysis/http-sql-analysis-port.ts packages/context/src/sql-analysis/http-sql-analysis-port.test.ts
git commit -m "feat(context): expose read-only SQL validation port"
```
### Task 3: Register the MCP `sql_execution` Tool Contract
**Files:**
- Modify: `packages/context/src/mcp/types.ts`
- Modify: `packages/context/src/mcp/context-tools.ts`
- Modify: `packages/context/src/mcp/server.test.ts`
- [ ] **Step 1: Add the MCP SQL execution port types**
In `packages/context/src/mcp/types.ts`, add these interfaces immediately before `KtxMcpContextPorts`:
```typescript
export interface KtxSqlExecutionResponse {
headers: string[];
headerTypes?: string[];
rows: unknown[][];
rowCount: number;
}
export interface KtxSqlExecutionMcpPort {
execute(input: { connectionId: string; sql: string; maxRows: number }): Promise<KtxSqlExecutionResponse>;
}
```
Then add the new optional port to `KtxMcpContextPorts`:
```typescript
sqlExecution?: KtxSqlExecutionMcpPort;
```
- [ ] **Step 2: Write failing MCP registration test**
In `packages/context/src/mcp/server.test.ts`, update the type import from `./types.js` to include `KtxSqlExecutionMcpPort`.
Add this test in `describe('createKtxMcpServer', ...)` after the existing connection-list registration test:
```typescript
it('registers parser-gated sql_execution when the host provides a SQL execution port', async () => {
const fake = makeFakeServer();
const sqlExecution: KtxSqlExecutionMcpPort = {
execute: vi.fn<KtxSqlExecutionMcpPort['execute']>().mockResolvedValue({
headers: ['status', 'count'],
headerTypes: ['text', 'bigint'],
rows: [['paid', 42]],
rowCount: 1,
}),
};
createKtxMcpServer({
server: fake.server,
userContext: { userId: 'local-user' },
contextTools: {
sqlExecution,
},
});
expect(fake.tools.map((tool) => tool.name)).toEqual(['sql_execution']);
await expect(
getTool(fake.tools, 'sql_execution').handler({
connectionId: 'warehouse',
sql: 'select status, count(*) from public.orders group by status',
maxRows: 50,
}),
).resolves.toEqual({
content: [
{
type: 'text',
text: JSON.stringify(
{
headers: ['status', 'count'],
headerTypes: ['text', 'bigint'],
rows: [['paid', 42]],
rowCount: 1,
},
null,
2,
),
},
],
structuredContent: {
headers: ['status', 'count'],
headerTypes: ['text', 'bigint'],
rows: [['paid', 42]],
rowCount: 1,
},
});
expect(sqlExecution.execute).toHaveBeenCalledWith({
connectionId: 'warehouse',
sql: 'select status, count(*) from public.orders group by status',
maxRows: 50,
});
});
```
- [ ] **Step 3: Run failing MCP registration test**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts -t sql_execution
```
Expected: FAIL because `sql_execution` is not registered.
- [ ] **Step 4: Add the MCP schema and registration**
In `packages/context/src/mcp/context-tools.ts`, add this schema after `scanArtifactReadSchema`:
```typescript
const sqlExecutionSchema = z.object({
connectionId: connectionIdSchema,
sql: z.string().min(1),
maxRows: z.number().int().min(1).max(10_000).default(1000).optional(),
});
```
Add this registration block in `registerKtxContextTools`, after the semantic-layer block and before the ingest block:
```typescript
if (ports.sqlExecution) {
const sqlExecution = ports.sqlExecution;
registerParsedTool(
server,
'sql_execution',
{
title: 'SQL Execution',
description:
'Execute one parser-validated read-only SQL query against a configured KTX connection and return structured rows.',
inputSchema: sqlExecutionSchema.shape,
},
sqlExecutionSchema,
async (input) => {
try {
return jsonToolResult(
await sqlExecution.execute({
connectionId: input.connectionId,
sql: input.sql,
maxRows: input.maxRows ?? 1000,
}),
);
} catch (error) {
return jsonErrorToolResult(error instanceof Error ? error.message : String(error));
}
},
);
}
```
- [ ] **Step 5: Run MCP registration test**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/server.test.ts -t sql_execution
```
Expected: PASS.
- [ ] **Step 6: Commit MCP tool contract**
Run:
```bash
git add packages/context/src/mcp/types.ts packages/context/src/mcp/context-tools.ts packages/context/src/mcp/server.test.ts
git commit -m "feat(context): register MCP sql execution tool"
```
### Task 4: Implement Local Project SQL Execution With Parser Validation
**Files:**
- Modify: `packages/context/src/mcp/local-project-ports.ts`
- Modify: `packages/context/src/mcp/local-project-ports.test.ts`
- [ ] **Step 1: Write failing local-port success test**
In `packages/context/src/mcp/local-project-ports.test.ts`, update the imports from `../scan/index.js` to include `type KtxQueryResult`.
Replace the existing `testConnector` helper with this version so tests can opt into read-only SQL:
```typescript
function testConnector(
snapshot = testSnapshot(),
queryResult?: KtxQueryResult,
): KtxScanConnector {
return {
id: `test:${snapshot.connectionId}`,
driver: snapshot.driver,
capabilities: createKtxConnectorCapabilities({ readOnlySql: queryResult !== undefined }),
introspect: vi.fn(async () => snapshot),
executeReadOnly: queryResult === undefined ? undefined : vi.fn(async () => queryResult),
cleanup: vi.fn(async () => {}),
};
}
```
Add this test after `tests a local project connection through the native scan connector factory`:
```typescript
it('executes MCP SQL only after parser-backed validation passes', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
};
const connector = testConnector(testSnapshot(), {
headers: ['id'],
headerTypes: ['integer'],
rows: [[1]],
totalRows: 1,
rowCount: 1,
});
const createConnector = vi.fn(async () => connector);
const sqlAnalysis = {
analyzeForFingerprint: vi.fn(),
analyzeBatch: vi.fn(),
validateReadOnly: vi.fn(async () => ({ ok: true, error: null })),
};
const ports = createLocalProjectMcpContextPorts(project, {
sqlAnalysis,
localScan: {
createConnector,
},
});
await expect(
ports.sqlExecution?.execute({
connectionId: 'warehouse',
sql: 'select id from public.orders',
maxRows: 5,
}),
).resolves.toEqual({
headers: ['id'],
headerTypes: ['integer'],
rows: [[1]],
rowCount: 1,
});
expect(sqlAnalysis.validateReadOnly).toHaveBeenCalledWith('select id from public.orders', 'postgres');
expect(createConnector).toHaveBeenCalledWith('warehouse');
expect(connector.executeReadOnly).toHaveBeenCalledWith(
{
connectionId: 'warehouse',
sql: 'select id from public.orders',
maxRows: 5,
},
{ runId: 'mcp-sql-execution' },
);
expect(connector.cleanup).toHaveBeenCalled();
});
```
- [ ] **Step 2: Write failing local-port rejection test**
Add this test after the success test:
```typescript
it('rejects MCP SQL before connector execution when parser validation fails', async () => {
const project = await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
};
const connector = testConnector(testSnapshot(), {
headers: ['id'],
rows: [[1]],
totalRows: 1,
rowCount: 1,
});
const sqlAnalysis = {
analyzeForFingerprint: vi.fn(),
analyzeBatch: vi.fn(),
validateReadOnly: vi.fn(async () => ({
ok: false,
error: 'SQL contains read/write operation: Insert',
})),
};
const ports = createLocalProjectMcpContextPorts(project, {
sqlAnalysis,
localScan: {
createConnector: vi.fn(async () => connector),
},
});
await expect(
ports.sqlExecution?.execute({
connectionId: 'warehouse',
sql: 'with x as (insert into t values (1) returning *) select * from x',
maxRows: 1000,
}),
).rejects.toThrow('SQL contains read/write operation: Insert');
expect(connector.executeReadOnly).not.toHaveBeenCalled();
});
```
- [ ] **Step 3: Run failing local-port tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/local-project-ports.test.ts -t "MCP SQL"
```
Expected: FAIL because `CreateLocalProjectMcpContextPortsOptions` has no `sqlAnalysis` option and no `sqlExecution` port.
- [ ] **Step 4: Add SQL-analysis option and helper imports**
In `packages/context/src/mcp/local-project-ports.ts`, add this import with the other context imports:
```typescript
import type { SqlAnalysisDialect, SqlAnalysisPort } from '../sql-analysis/index.js';
```
Add `sqlAnalysis` to `CreateLocalProjectMcpContextPortsOptions`:
```typescript
sqlAnalysis?: SqlAnalysisPort;
```
Add this helper near `dialectForDriver`:
```typescript
function sqlAnalysisDialectForDriver(driver: string | undefined): SqlAnalysisDialect {
return dialectForDriver(driver) as SqlAnalysisDialect;
}
```
- [ ] **Step 5: Implement the local SQL execution port**
In `packages/context/src/mcp/local-project-ports.ts`, add this function before `createLocalProjectMcpContextPorts`:
```typescript
async function executeValidatedReadOnlySql(
project: KtxLocalProject,
options: CreateLocalProjectMcpContextPortsOptions,
input: { connectionId: string; sql: string; maxRows: number },
): Promise<{ headers: string[]; headerTypes?: string[]; rows: unknown[][]; rowCount: number }> {
const connectionId = assertSafeConnectionId(input.connectionId);
const connection = project.config.connections[connectionId];
if (!connection) {
throw new Error(`Connection "${connectionId}" is not configured in ktx.yaml`);
}
if (!options.sqlAnalysis) {
throw new Error('sql_execution requires parser-backed SQL validation.');
}
const validation = await options.sqlAnalysis.validateReadOnly(
input.sql,
sqlAnalysisDialectForDriver(connection.driver),
);
if (!validation.ok) {
throw new Error(validation.error ?? 'SQL is not read-only.');
}
const createConnector = options.localScan?.createConnector;
if (!createConnector) {
throw new Error('sql_execution requires a local scan connector factory.');
}
let connector: KtxScanConnector | null = null;
try {
connector = await createConnector(connectionId);
if (!connector.capabilities.readOnlySql || !connector.executeReadOnly) {
throw new Error(`Connection "${connectionId}" does not support read-only SQL execution.`);
}
const result = await connector.executeReadOnly(
{
connectionId,
sql: input.sql,
maxRows: input.maxRows,
},
{ runId: 'mcp-sql-execution' },
);
return {
headers: result.headers,
...(result.headerTypes ? { headerTypes: result.headerTypes } : {}),
rows: result.rows,
rowCount: result.rowCount ?? result.rows.length,
};
} finally {
await cleanupConnector(connector);
}
}
```
In `createLocalProjectMcpContextPorts`, add this conditional block immediately after the initial `ports` object is created and before the existing `if (options.localIngest)` block:
```typescript
if (options.sqlAnalysis && options.localScan?.createConnector) {
ports.sqlExecution = {
async execute(input) {
return executeValidatedReadOnlySql(project, options, input);
},
};
}
```
- [ ] **Step 6: Run local-port tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/mcp/local-project-ports.test.ts -t "MCP SQL"
```
Expected: PASS.
- [ ] **Step 7: Commit local MCP SQL execution**
Run:
```bash
git add packages/context/src/mcp/local-project-ports.ts packages/context/src/mcp/local-project-ports.test.ts
git commit -m "feat(context): execute MCP SQL through validated connector path"
```
### Task 5: Verification
**Files:**
- Verify: all modified files from Tasks 1-4
- [ ] **Step 1: Run Python SQL-analysis and app tests**
Run:
```bash
source .venv/bin/activate && uv run pytest python/ktx-daemon/tests/test_sql_analysis.py python/ktx-daemon/tests/test_app.py -q
```
Expected: PASS.
- [ ] **Step 2: Run focused TypeScript tests**
Run:
```bash
pnpm --filter @ktx/context exec vitest run src/sql-analysis/http-sql-analysis-port.test.ts src/mcp/server.test.ts src/mcp/local-project-ports.test.ts
```
Expected: PASS.
- [ ] **Step 3: Run type-check**
Run:
```bash
pnpm --filter @ktx/context run type-check
```
Expected: PASS.
- [ ] **Step 4: Run Python pre-commit on changed Python files**
Run:
```bash
source .venv/bin/activate && uv run pre-commit run --files python/ktx-daemon/src/ktx_daemon/sql_analysis.py python/ktx-daemon/src/ktx_daemon/app.py python/ktx-daemon/tests/test_sql_analysis.py python/ktx-daemon/tests/test_app.py
```
Expected: PASS. If the repository has no usable pre-commit configuration in the active environment, record the exact error and keep the pytest results above as the closest Python verification.
- [ ] **Step 5: Confirm the remaining v1 blockers are unchanged**
Run:
```bash
test -e packages/context/src/scan/entity-details.ts; printf 'entity-details:%s\n' "$?"
test -e packages/context/src/sl/dictionary-search.ts; printf 'dictionary-search:%s\n' "$?"
test -e packages/context/src/search/discover.ts; printf 'discover:%s\n' "$?"
test -e packages/cli/src/commands/mcp-commands.ts; printf 'mcp-commands:%s\n' "$?"
test -e packages/cli/src/skills/research/SKILL.md; printf 'research-skill:%s\n' "$?"
```
Expected:
```text
entity-details:1
dictionary-search:1
discover:1
mcp-commands:1
research-skill:1
```
These `1` exit-code markers confirm this plan landed only the SQL execution foundation and did not silently claim the remaining research-tool, daemon, or setup-agent v1 work.
- [ ] **Step 6: Commit verification notes if any test docs changed**
Run:
```bash
git status --short
```
Expected: no uncommitted source changes after the task commits. If verification required a small documentation note, commit only that note with:
```bash
git add docs/superpowers/plans/2026-05-14-research-agent-mcp-sql-execution-foundation.md
git commit -m "docs: record research MCP SQL execution plan"
```

931
docs/superpowers/specs/2026-05-14-research-agent-mcp-tools-design.md Normal file
View file

@ -0,0 +1,931 @@
# Research Agent MCP Tools Design
**Date:** 2026-05-14
**Author:** Andrey Avtomonov
**Status:** Design — pending implementation plan
## Background
KTX positions itself as a standalone context layer for database agents.
External agents — Claude Code, Cursor, Codex, opencode — should be able to
connect to a local KTX instance via MCP and perform research against
configured data connections.
The existing MCP surface (`packages/context/src/mcp/context-tools.ts`) already
exposes strong **context** primitives: wiki search/read/write, semantic-layer
list/read/write/validate/query, ingest and scan run management, memory
capture. What it is missing is the **active investigation** primitives a
research agent needs:
- The agent cannot run raw SQL against a connection. `sl_query` only covers
semantic-layer-defined queries.
- The agent cannot inspect raw table or column metadata for tables that are
not yet modeled in the semantic layer.
- The agent cannot find which column holds a literal value mentioned by the
user (e.g., "Acme Corp").
- The agent must call multiple separate search tools (`wiki_search`,
`sl_list_sources`) and reconcile results manually instead of getting a
unified ranked discovery view.
The Kaelio research agent (reference implementation at
`/Users/andrey/conductor/workspaces/kaelio-main2/douala/server/src/cores/research-execution.core.ts`)
addresses these gaps with tools named `sql_execution`, `entity_details`,
`dictionary_search`, and `discover_data`, used in a discovery → inspection →
query loop. The corresponding KTX infrastructure already exists in pieces:
- `KtxScanConnector.executeReadOnly` on every connector
(`packages/connector-postgres/src/connector.ts:447` and siblings) — read-only
SQL execution with `assertReadOnlySql` and `limitSqlForExecution`.
- `KtxSchemaSnapshot` from scan reports — full table/column/FK metadata.
- `SlDictionaryEntry` extraction over relationship-profiling artifacts
(`packages/context/src/sl/sl-dictionary-profile.ts`).
- Hybrid search core with Reciprocal Rank Fusion
(`packages/context/src/search/{hybrid-search-core,rrf}.ts`).
This design exposes those primitives as four new MCP tools, adds a research
skill to guide external agents, and introduces an HTTP-only `ktx mcp` daemon
to host the MCP server.
## Goals
- Expose four new MCP tools that turn KTX into a research-capable context
layer for any MCP-compatible client: `discover_data`, `entity_details`,
`dictionary_search`, `sql_execution`.
- Ship a `ktx-research` skill installable via `ktx setup-agents`, describing
the discover → inspect → query → capture workflow for external agents.
- Provide a `ktx mcp` CLI subtree that runs the MCP server over HTTP on
localhost, with the same lifecycle pattern as the existing managed Python
daemon (`packages/cli/src/managed-python-daemon.ts`).
- Make `ktx setup-agents` install MCP client configuration for the configured
targets pointing at the local HTTP endpoint. v1 splits this by client: for
claude-code and cursor (JSON config), `setup-agents` writes the entry
directly; for codex (TOML) and opencode (different JSON wrapper),
`setup-agents` prints a copy-pasteable snippet rather than writing the file.
See the client matrix below for full per-target behavior.
- Reuse existing infrastructure (connector `executeReadOnly`, schema
snapshots, dictionary profile, hybrid search + RRF) rather than building
parallel implementations.
## Non-goals
- This spec does not build an agent loop inside KTX. The system prompt, step
budget, tool dispatch, and methodology tracking remain in the external
client. KTX is a context provider, not an agent runner.
- This spec does not expose Python code execution. The `ktx-daemon`
`/code/execute` endpoint exists but is not surfaced via MCP. That is a
separate design with its own sandboxing and security considerations.
- This spec does not ship widget rendering, chart creation, or scheduled
report execution. Those are presentation concerns the external client owns.
- This spec does not implement stdio MCP transport. HTTP-only.
- This spec does not implement OS-level auto-start (launchd, systemd user
units). `ktx mcp start` must be run explicitly.
- This spec does not implement remote network exposure beyond loopback. Token
auth and non-`127.0.0.1` binding are supported but TLS, audit logging, and
multi-tenant isolation are out of scope for v1.
## Tool inventory
Four new MCP tools, registered in `packages/context/src/mcp/context-tools.ts`
alongside the existing tools.
### Relationship to existing warehouse-verification tools
KTX already ships ingest-side implementations of `sql_execution`,
`entity_details`, and `discover_data` at
`packages/context/src/ingest/tools/warehouse-verification/{sql-execution,entity-details,discover-data}.tool.ts`,
backed by `warehouse-catalog.service.ts`. Their contracts differ from the
MCP shapes proposed below in three concrete ways:
- They currently take `connectionName` (slug-shaped); this spec renames
them to `connectionId` in the same change (see below).
- They take `targets` (a discriminated `display` vs. `{catalog,db,name}`
union) and `rowLimit`, not `entities` / `maxRows`.
- They return `{ markdown, structured }` with scan availability, candidate
matches, and ingest-session-allowed-connection scoping, not the
MCP-shaped pure-structured outputs in this spec.
To avoid two divergent contracts for the same primitives, the MCP tools
**must be implemented by extracting the shared logic out of
`warehouse-verification/*` and into reusable services**
(e.g., `WarehouseCatalogService` as the source of truth for table/column
resolution and discovery, plus a shared read-only SQL executor that wraps
`assertReadOnlySql`/`limitSqlForExecution`). The ingest tools and the new
MCP tools then become thin adapters around those services with their own
input/output shapes appropriate to each surface.
KTX has no public users yet, so the same change that introduces the MCP
tools renames the ingest-side `connectionName` parameter to `connectionId`
across `warehouse-verification/*.tool.ts`, `warehouse-catalog.service.ts`,
and any callers. `connectionId` matches the rest of the in-process MCP
surface (`sl_query`, `sl_list_sources`, `scan_trigger`, etc.) and the new
MCP tool inputs. The ingest tools and the new MCP tools then share both
the service layer and the parameter name; only their input/output shapes
differ (markdown+structured for the ingest surface, pure structured for
the MCP surface).
### discover_data
Unified ranked search across wiki, semantic-layer sources/measures/dimensions,
and raw schema tables/columns. Returns refs only with a uniform shape; the
agent dereferences top hits using the existing `wiki_read`, `sl_read_source`,
or `entity_details` tools.
**Input schema:**
```typescript
{
query: z.string().min(1),
connectionId: z.string().optional(), // omit → all connections
kinds: z.array(z.enum([
'wiki', 'sl_source', 'sl_measure', 'sl_dimension', 'table', 'column',
])).optional(), // omit → all kinds
limit: z.number().int().min(1).max(50).default(15).optional(),
}
```
**Output:** array of refs, each:
```typescript
{
kind: 'wiki' | 'sl_source' | 'sl_measure' | 'sl_dimension' | 'table' | 'column',
id: string, // stable id: wiki key, source name, or driver-qualified table/column display string
score: number, // RRF fused score, 0-1 range
summary: string | null, // one-line description; null when no source field is populated
snippet: string | null, // short context snippet, ≤200 chars; null when nothing meaningful to show
matchedOn: // why this result matched (powers the snippet for non-description kinds)
| 'name' | 'display' | 'description' | 'comment' | 'expr' | 'sample_value' | 'body',
connectionId?: string, // present for non-wiki kinds
tableRef?: { // present for kind 'table' and 'column'
catalog: string | null,
db: string | null,
name: string,
},
columnName?: string, // present for kind 'column'
}
```
The structured `tableRef` mirrors the live `KtxSchemaTable` identity
(`packages/context/src/scan/types.ts:74-83`) so callers can pass refs into
`entity_details` without losing `catalog`/`db` qualification on drivers
that need it (BigQuery `project.dataset.table`, Snowflake/SQL Server
`database.schema.table`).
#### `summary` and `snippet` provenance per kind
Both fields are derived from existing source data, never invented or
LLM-generated. The resolver is pure and deterministic per kind. When no
source field exists for a given kind, the field is `null`; agents must
not assume a missing snippet means "no context" — they should dereference
the ref via `wiki_read`, `sl_read_source`, or `entity_details` to get
authoritative content.
| Kind | `summary` source | `snippet` source |
|---|---|---|
| `wiki` | `WikiFrontmatter.summary` (`packages/context/src/wiki/types.ts:15`) — populated at write time | Up to 200 chars from the wiki body around the match position; falls back to first 200 chars of body when `matchedOn === 'name'`/`'display'` |
| `sl_source` | `resolveDescription(source.descriptions, priority)` (`packages/context/src/sl/descriptions.ts:16-34`) over the `user|ai|dbt|db` priority chain (`packages/context/src/sl/types.ts:5`) | When `matchedOn === 'description'`/`'body'`: a window of the resolved description; otherwise the source's `name` + first 1–2 measure or dimension names as context |
| `sl_measure` | `measure.description` (`packages/context/src/sl/types.ts:37`) | `measure.expr` truncated to 200 chars — the calculation is the most informative one-line context for a measure |
| `sl_dimension` | `resolveDescription(column.descriptions, priority)` (same precedence as `sl_source`); when empty, fall back to `null` | `${column.name} (${column.type})` formatted exactly like the existing inline rendering in `sl-search.service.ts:29-41` |
| `table` | `firstDescription(table.descriptions)` then `table.comment` (precedence already used by `warehouse-catalog.service.ts:286-287`); `null` when both are empty | When `matchedOn === 'description'`/`'comment'`: a window of that string; when `matchedOn === 'name'`/`'display'`: a comma-joined list of up to 5 of the table's column names |
| `column` | `resolveDescription(column.descriptions)` then `column.comment` (`warehouse-catalog.service.ts:228-245`); `null` when both are empty | When `matchedOn === 'description'`/`'comment'`: that text; when `matchedOn === 'sample_value'`: `${column.nativeType} · samples: <up to 5 sampleValues>` formatted from `column.sampleValues` (`warehouse-catalog.service.ts:18-23`); otherwise `${column.nativeType}` |
The `matchedOn` field is the same concept as the existing
`RawSchemaHit.matchedOn` in `warehouse-catalog.service.ts:40-54`,
extended to the wiki and SL kinds. Snippets always come from a single
already-stored field; the resolver never concatenates across sources or
invents bridging text. Length cap is enforced at the producer side (≤200
chars after a single-pass slice; no ellipsis appended — clients render
one if they want).
**Implementation:** new module `packages/context/src/search/discover.ts`.
Composes three sub-searches in parallel:
1. Wiki search via the existing wiki search backend.
2. SL search over sources/measures/dimensions using existing
`sl-sources-index` (or a new lightweight index if needed for measure
granularity).
3. Raw schema search over tables and columns from `KtxSchemaSnapshot`,
indexed at scan time and stored alongside other scan artifacts.
Results from each sub-search are fused with `packages/context/src/search/rrf.ts`
using equal weights. The `kinds` filter constrains which sub-searches run.
### entity_details
Read structured metadata for one or more raw tables (and optionally specific
columns) from the latest scan snapshot. The raw-data equivalent of
`sl_read_source`.
**Input schema:**
```typescript
{
connectionId: z.string().min(1),
entities: z.array(z.object({
// table accepts either a driver-display string ("project.dataset.table",
// "schema.name", "db.schema.name") or a structured ref. The resolver
// returns a structured error when the input is ambiguous across multiple
// schemas/catalogs.
table: z.union([
z.string().min(1),
z.object({
catalog: z.string().nullable(),
db: z.string().nullable(),
name: z.string().min(1),
}),
]),
columns: z.array(z.string()).optional(), // omit → all columns
})).min(1).max(20),
}
```
**Output:** for each entity, a structured record:
```typescript
{
connectionId: string,
tableRef: { // structured identity, lossless on every driver
catalog: string | null, // BigQuery project, Snowflake/SQL Server database
db: string | null, // schema/dataset
name: string,
},
display: string, // driver-formatted display string
// (e.g. "project.dataset.table", "schema.name")
kind: 'table' | 'view' | 'external' | 'event_stream', // matches KtxSchemaTableKind
comment: string | null,
estimatedRows: number | null,
columns: Array<{
name: string,
nativeType: string,
normalizedType: string,
dimensionType: 'time' | 'string' | 'number' | 'boolean',
nullable: boolean,
primaryKey: boolean,
comment: string | null,
}>,
foreignKeys: Array<{
fromColumn: string,
toCatalog: string | null, // qualified FK target, preserves cross-db FKs
toDb: string | null,
toTable: string,
toColumn: string,
constraintName: string | null,
}>,
snapshot: { // freshness metadata, present on every response
syncId: string, // latest scan/sync identifier
extractedAt: string, // ISO-8601 UTC of the snapshot
scanRunId: string | null, // scan run id if available
},
}
```
Output fields mirror `KtxSchemaTable` / `KtxSchemaColumn` /
`KtxSchemaForeignKey` from `packages/context/src/scan/types.ts:51-82`. The
full `KtxSchemaTableKind` set is preserved so BigQuery `external` tables
and warehouses with event-stream sources are not silently coerced. FK
target qualification (`toCatalog`/`toDb`) carries through so agents can
write valid SQL for cross-schema or cross-database references without
re-resolving.
If `columns` is provided, only the requested columns appear in the `columns`
array (PKs and FKs still report on the full table).
**Implementation:** new module `packages/context/src/scan/entity-details.ts`.
Reads `KtxSchemaSnapshot` from the same store the existing `scan_*` tools
read. No new infrastructure. If the requested table is not in the latest
snapshot, the tool returns a structured error with a suggestion to run
`ktx ingest <connectionId>`.
**Cache freshness.** Today `WarehouseCatalogService` caches `ConnectionCatalog`
per connection name with no invalidation
(`packages/context/src/ingest/tools/warehouse-verification/warehouse-catalog.service.ts:248-249`,
`:404-411`). For an ingest tool that runs inside a single short-lived ingest
session that is acceptable, but the MCP daemon is long-lived and serves
clients across multiple `scan_trigger` / `ktx ingest` runs. The MCP adapter
**must** key its cache on the latest scan artifact identity (the `syncId`
derived from the artifact path, or the artifact file mtime) and re-read when
that identity advances. The same rule applies to the shared services backing
`discover_data` and `dictionary_search`. The implementation plan must
either:
1. Extend `WarehouseCatalogService` (and equivalent dictionary/discover
services) to invalidate cached entries when the underlying artifact
identity advances, or
2. Wrap those services in an MCP-adapter cache layer that performs the
identity check before returning cached values.
### dictionary_search
Find which connection, source, and column **profile-sampled** a given literal
value (or substring) such as "Acme Corp" or "shipped". Backed by the existing
`SlDictionaryEntry` extraction over relationship-profiling artifacts.
**Authoritativeness.** The dictionary index is built from *sampled* values
captured during relationship profiling — by default 5 values per column,
drawn from a sample of up to 10,000 rows
(`packages/context/src/scan/relationship-profiling.ts:409-410`,
`packages/context/src/sl/sl-dictionary-profile.ts:70`). A hit confirms a
column contains the value; a miss is **not** proof that the value is absent
from the column or warehouse — the value may simply have been outside the
profile sample. The tool must surface this distinction in its output and the
research skill must teach agents not to treat a miss as exhaustive.
**Input schema:**
```typescript
{
values: z.array(z.string().min(1)).min(1).max(20),
connectionId: z.string().optional(), // omit → all connections
}
```
**Output:** for each input value, the list of matching entries plus
per-connection provenance. Coverage and miss reasons are connection-scoped
because `loadLatestSlDictionaryEntries` iterates each connection's profile
artifact independently
(`packages/context/src/sl/sl-dictionary-profile.ts:96-112`); a single
all-connections call can mix `no_profile_artifact` (one connection never
ran an enriched scan), `value_not_in_sample` (another connection ran but
the literal was outside the sample), and matches in the same response.
```typescript
{
// The set of connections actually searched on this call. When the input
// omits connectionId this is every configured connection; otherwise it
// contains the single requested connection.
searched: Array<{
connectionId: string,
coverage: {
sampledRows: number | null, // profileSampleRows used at profile time
valuesPerColumn: number | null, // sampleValuesPerColumn used at profile time
profiledColumns: number, // count of columns in the dictionary index for this connection
syncId: string | null, // identifier of the profile artifact (null when missing)
profiledAt: string | null, // ISO-8601 UTC of the profile artifact (null when missing)
},
// Per-connection status, independent of any specific input value:
// ready — profile present with profiled columns
// no_profile_artifact — enriched scan never ran for this connection
// no_candidate_columns — profile present but no columns profile-eligible
status: 'ready' | 'no_profile_artifact' | 'no_candidate_columns',
}>,
results: Array<{
value: string, // input value
matches: Array<{
connectionId: string,
sourceName: string,
columnName: string,
matchedValue: string, // actual value found (may differ in case)
cardinality: number | null, // column cardinality if known
}>,
// Per-connection miss reasons for this value, present when that
// connection produced no match. Connections that matched do not appear
// in `misses`. For ready connections with no match, the reason is
// 'value_not_in_sample' (non-authoritative miss). For unready
// connections, the reason mirrors their `status` above.
misses: Array<{
connectionId: string,
reason:
| 'no_profile_artifact'
| 'no_candidate_columns'
| 'value_not_in_sample',
}>,
}>,
}
```
**Matching semantics:** case-insensitive substring match against the
profile-sampled values. Misses are never authoritative — they only state
that the value was not in the captured sample for the listed connection.
`misses[].reason` distinguishes "no enriched scan has run on this
connection" (`no_profile_artifact`), "enriched scan ran but no columns
were profile-eligible" (`no_candidate_columns`), and "scan ran but value
was not in the sample" (`value_not_in_sample`). The research skill must
direct agents to follow up a `value_not_in_sample` miss with
`sql_execution` against the most plausible columns, not to conclude the
value is absent.
**Cache freshness:** the dictionary index is keyed on the profile artifact
identity (the `syncId` derived from its path or the artifact mtime). When
that identity advances, the daemon re-reads the artifact on next call. See
the `entity_details` cache-freshness note above for the shared rule.
**Implementation:** new module `packages/context/src/sl/dictionary-search.ts`.
Loads `SlDictionaryEntry` records via the existing extraction code path,
builds a per-connection in-memory index on first call, caches it for the
lifetime of the MCP daemon. Invalidated on next ingest run (the daemon
watches `.ktx/db.sqlite` for changes, or simply re-reads on each call when
the artifact mtime advances).
### sql_execution
Execute a read-only SQL query against a configured connection and return the
result. The fallback path for questions the semantic layer does not cover.
**Input schema:**
```typescript
{
connectionId: z.string().min(1),
sql: z.string().min(1),
maxRows: z.number().int().min(1).max(10_000).default(1000).optional(),
}
```
**Output:**
```typescript
{
headers: string[],
headerTypes?: string[], // driver-mapped type names, one per header; optional
rows: Array<Array<unknown>>,
rowCount: number,
}
```
`headerTypes` is optional because not every connector exposes per-column
type metadata. The current contract makes it optional
(`KtxQueryResult.headerTypes` in `packages/context/src/scan/types.ts:272-277`),
and the SQLite connector currently omits it
(`packages/connector-sqlite/src/connector.ts:237-240`, `:301-308`). When a
connector returns header types, the MCP adapter passes them through
verbatim. When a connector does not, the MCP adapter omits the field rather
than fabricating values.
**Implementation:** delegates to `KtxScanConnector.executeReadOnly` on the
matching connector. The connector calls `assertReadOnlySql` and
`limitSqlForExecution` (`packages/context/src/connections/read-only-sql.ts`).
**Read-only enforcement is lexical, not parser-backed.** The current guard
inspects the first token with regex: it accepts queries whose first non-space
token is `SELECT` or `WITH`, and rejects queries whose first non-space token
matches a fixed mutating-verb list. Implications:
- A CTE that nests a data-modifying statement (e.g., `WITH x AS (INSERT ...
RETURNING *) SELECT ...`, valid in Postgres) passes the first-token check
and would reach the connector.
- Dialect-specific read/write constructs and procedure calls that do not
start with a listed verb are not caught.
Because `sql_execution` exposes this boundary to external MCP clients, the
tool **must not** be enabled until one of the following holds:
1. The guard is upgraded to a sqlglot/AST-based read-only check that
inspects every statement and CTE node, with explicit tests for CTE-DML,
`CALL`, `DO`, vendor pragmas, and multi-statement payloads; or
2. Connector-side execution forces a read-only transaction / session (e.g.,
`SET TRANSACTION READ ONLY` for Postgres, `READ ONLY` connection for
MySQL, equivalent for each connector), so the guard is defense-in-depth
rather than the sole boundary.
The implementation plan that follows this spec is required to choose and
land one of those before registering `sql_execution` in the MCP surface.
Errors from `assertReadOnlySql` (whichever implementation) are returned as
structured tool errors so the agent can correct the query and retry.
## Tool naming convention
Match the existing KTX MCP convention (no prefix): `discover_data`,
`entity_details`, `dictionary_search`, `sql_execution`. The existing tools
(`wiki_search`, `sl_list_sources`, `scan_trigger`, `memory_capture`) all use
unprefixed snake_case; the new tools follow suit.
## Connection model
- `sql_execution` and `entity_details` require `connectionId` — these tools
cannot operate without a target.
- `discover_data` and `dictionary_search` make `connectionId` optional. Omit
it to search across all configured connections; provide it to scope. This
matches the existing pattern for `sl_list_sources({ connectionId? })`.
- All tools are project-locked: the MCP daemon runs in one KTX project dir;
to operate on a different project, restart the daemon with a different
`--project-dir` or `cwd`.
## MCP daemon: `ktx mcp`
A new CLI subtree in `packages/cli/src/commands/mcp-commands.ts`, wired into
`cli-program.ts` alongside `setup`, `connection`, `ingest`, `wiki`, `sl`,
`status`, `dev`.
### Commands
```bash
ktx mcp start [--port <n>] [--host <h>] [--token <t>] [--foreground] \
[--allowed-host <h>...] [--allowed-origin <o>...]
ktx mcp stop
ktx mcp status
ktx mcp logs [--follow]
```
`--allowed-host` and `--allowed-origin` are repeatable. They extend (not
replace) the defaults defined in the security model below.
### `ktx mcp start`
Starts a long-lived HTTP MCP server bound to the configured host and port,
serving every tool registered by `createKtxMcpServer`. The server stays alive
until `ktx mcp stop` is invoked or the process is terminated.
- Default `--host` is `127.0.0.1`. Any value other than `127.0.0.1` or
`localhost` **requires** `--token` (or `KTX_MCP_TOKEN` in the environment);
the command refuses otherwise.
- Default `--port` is 7878. If the port is in use, the command exits with an
error explaining how to choose another. Allocated port is persisted to
`.ktx/mcp.json` for subsequent `status`, `stop`, `logs`, and
`setup-agents` calls.
- `--foreground` runs the server in the foreground and pipes all logs to
stdout, for debugging. Default is background.
- Background runs detach via the same pattern as the managed Python daemon
(`packages/cli/src/managed-python-daemon.ts`): spawn a detached child,
write `pid`, `port`, `startedAt` to `.ktx/mcp.json`, return immediately
with the URL the user should configure in their client.
- Logs go to `.ktx/logs/mcp.log` (matches existing log layout).
### `ktx mcp stop`
Reads `.ktx/mcp.json` for the daemon PID, sends SIGTERM, waits up to 10
seconds for graceful exit, then SIGKILLs if still running. Removes the state
file on success.
### `ktx mcp status`
Reads `.ktx/mcp.json`, checks the process is alive, hits the server's
`/health` endpoint, and reports:
- Running / stopped / stale (state file present but process not alive)
- Port, host, started-at, pid
- Whether token auth is enabled
- Configured project dir
### `ktx mcp logs`
Tails or follows `.ktx/logs/mcp.log`. Standard `--follow` flag.
### Lifecycle
Manual: the user runs `ktx mcp start` after each reboot or whenever they
want the server running. No auto-start on other `ktx` commands (matches the
explicit pattern established by the daemon model).
### Transport
HTTP-only via `StreamableHTTPServerTransport` from
`@modelcontextprotocol/sdk/server/streamableHttp.js`.
The `/mcp` endpoint must implement the full Streamable HTTP contract, not
just `POST`:
- `POST /mcp` — JSON-RPC requests (and the `initialize` handshake when no
session exists). On the first `initialize` post, the server allocates a
session id and returns it in the `Mcp-Session-Id` response header.
- `GET /mcp` — opens an SSE stream for server-initiated messages on an
existing session. Requires a valid `Mcp-Session-Id` header.
- `DELETE /mcp` — explicit session termination by the client. Requires a
valid `Mcp-Session-Id` header; the server must drop the session and any
associated SSE streams.
**Session model.** v1 ships **stateful** sessions: the server generates a
session id with `randomUUID()` on `initialize`, stores the transport in an
in-memory map keyed by session id, reuses it on subsequent
`POST`/`GET`/`DELETE` calls that carry the same `Mcp-Session-Id`, and
removes it on `DELETE` or transport close. Requests that carry an unknown
session id are rejected with HTTP 404 so the client knows to re-initialize.
Health: `GET /health` returns `{ status: 'ok', projectDir, port }` for
liveness checks. `/health` is separate from `/mcp` and is not subject to
session-id requirements (but is subject to host/origin validation; see
below).
### Security model
- `127.0.0.1` binding is the default and requires no token auth (loopback
only). Even on loopback, the server enforces **Host and Origin header
validation** on every `/mcp` and `/health` request to defend against
browser-driven DNS-rebinding attacks (the same defense the MCP SDK
exposes in `createMcpExpressApp` / `createMcpHonoApp`).
- **Host validation** compares the incoming `Host` header to the allowed-host
list after normalizing: lowercase, strip any port, strip surrounding
brackets from IPv6 literals (`[::1]:7878` → `::1`). Comparison is exact
on the normalized host string. The default allowed-host list is
`['localhost', '127.0.0.1', '::1']`. `--allowed-host` values are appended
after the same normalization.
- **Origin validation** compares the full browser `Origin` header (scheme +
host + port) to the allowed-origin list. The default allowed-origin list
is empty: any request that carries an `Origin` header is rejected unless
an explicit `--allowed-origin` entry matches. Non-browser clients that
do not send an `Origin` header (Claude Code, Cursor, Codex, opencode
HTTP transports) are accepted regardless of `Origin`. Each
`--allowed-origin` value must be a full origin string
(e.g., `http://localhost:7878`); KTX validates the format at startup.
- Non-loopback binding requires `--token <t>` or `KTX_MCP_TOKEN`. The
server checks `Authorization: Bearer <t>` on **every** `/mcp` method —
`POST`, `GET` (SSE), and `DELETE` — and rejects with HTTP 401 otherwise.
Token enforcement is independent of the session check; both must pass.
When `--host` is non-loopback, the allowed-host list expands to include
the normalized bound host plus any user-supplied `--allowed-host`
values.
- TLS is out of scope. For remote access, document running KTX behind a
reverse proxy (Caddy, nginx) that terminates TLS.
## Client config installation via `ktx setup-agents`
`ktx setup-agents` extends its existing per-target file installation
(`plannedKtxAgentFiles` in `packages/cli/src/setup-agents.ts:64`) to also
write MCP server entries.
The per-client config matrix is **not uniform**. Each client has its own
file location, scope semantics, and entry shape; `setup-agents` must
produce the correct shape per target rather than emit one JSON blob.
| Target | Scope | MCP config path | Writer behavior |
|---|---|---|---|
| claude-code | user (global) | `~/.claude.json` → root `mcpServers.ktx` | write JSON |
| claude-code | local (per-project, private) | `~/.claude.json` → `projects[<absProjectPath>].mcpServers.ktx` | write JSON |
| claude-code | project (shared, checked in) | `<projectDir>/.mcp.json` → `mcpServers.ktx` | write JSON |
| cursor | global | `~/.cursor/mcp.json` → `mcpServers.ktx` | write JSON |
| cursor | project | `<projectDir>/.cursor/mcp.json` → `mcpServers.ktx` | write JSON |
| codex | user (global) | `~/.codex/config.toml` → `[mcp_servers.ktx]` (TOML) | print instructions; do not auto-write in v1 |
| opencode | user (global) | `~/.config/opencode/opencode.json` → `mcp.ktx` | print instructions; do not auto-write in v1 |
| opencode | project | `<projectDir>/opencode.json` → `mcp.ktx` | print instructions; do not auto-write in v1 |
The shared global `~/.claude.json` and per-project `~/.claude.json` →
`projects[...]` scope are both supported because Claude Code's "user" vs.
"local" scopes write to different sub-trees of the same file; `setup-agents`
must select the scope explicitly per invocation.
Codex and opencode entries are **printed as copy-pasteable snippets** in v1
because their config formats (TOML for codex, a different JSON wrapper for
opencode) diverge enough from the JSON writers above that mixing them into
the same writer codepath risks silently producing invalid files. This is a
deliberate v1 scoping decision, not a permanent limitation.
#### Entry shapes by target
Claude Code (HTTP):
```jsonc
{
"mcpServers": {
"ktx": {
"type": "http",
"url": "http://localhost:7878/mcp"
// when token auth is active, env-var expansion only:
// "headers": { "Authorization": "Bearer ${KTX_MCP_TOKEN}" }
}
}
}
```
Cursor (HTTP, project `.cursor/mcp.json` or global `~/.cursor/mcp.json`):
```jsonc
{
"mcpServers": {
"ktx": {
"url": "http://localhost:7878/mcp"
// when token auth is active, env-var expansion only:
// "headers": { "Authorization": "Bearer ${KTX_MCP_TOKEN}" }
}
}
}
```
Codex (printed snippet, `~/.codex/config.toml`):
```toml
[mcp_servers.ktx]
url = "http://localhost:7878/mcp"
# Codex MCP config does not currently document a headers field; if token
# auth is active, instruct the user to either run KTX on loopback without a
# token or wait for codex header support before enabling.
```
opencode (printed snippet, `opencode.json`):
```jsonc
{
"mcp": {
"ktx": {
"type": "remote",
"url": "http://localhost:7878/mcp",
"enabled": true
// when token auth is active, env-var expansion only:
// "headers": { "Authorization": "Bearer ${KTX_MCP_TOKEN}" }
}
}
}
```
#### Token handling per client
When `--token` / `KTX_MCP_TOKEN` is active, `setup-agents` writes the bearer
token **only via environment-variable reference** (`Bearer ${KTX_MCP_TOKEN}`),
never as a literal token value. Claude Code, Cursor, and opencode all
support environment-variable expansion inside `headers` values; the
written entry references `${KTX_MCP_TOKEN}` and the user is responsible
for exporting it in the shell that launches the MCP client.
Rules:
- **No literal-token writes, anywhere.** Even the user-scope (private)
Claude Code / Cursor config receives env-var references, not the raw
token. This keeps the same writer codepath for every scope and avoids a
branch that materializes secrets.
- **Project-scope (shared, checked-in) configs are gated.** When a token is
active and the user requests a shared scope — `<projectDir>/.mcp.json`
for Claude Code, `<projectDir>/.cursor/mcp.json` for Cursor — `setup-agents`
prints a warning and offers a choice: (a) write the entry with the
`${KTX_MCP_TOKEN}` reference (the file is safe to commit; readers must
export the variable locally), or (b) skip the shared entry and rely on a
user-scope entry instead. The default is (a).
- **Verify header support per client before writing.** The matrix below
reflects the current state of each client's MCP config docs:
- claude-code: supports `headers` with `${VAR}` expansion on HTTP entries.
- cursor: supports `headers` with `${VAR}` expansion on HTTP entries.
- opencode: supports `headers` with `${VAR}` expansion on remote MCP
entries.
- codex: **not currently supported** in published config docs. When a
token is active and the user selects codex, `setup-agents` prints a
warning and skips the codex entry rather than writing an entry that
codex will silently ignore. The recommended workaround is to bind KTX
to loopback without a token for codex users.
- **Implementation acceptance test.** Setup-agents writer tests must assert
that no rendered output contains the literal token string for any
scope/target combination — only the `${KTX_MCP_TOKEN}` reference.
Port is read from `.ktx/mcp.json` if present, falling back to 7878. The
install manifest (`agentInstallManifestPath`,
`packages/cli/src/setup-agents.ts:60`) tracks each **written** entry so
`ktx setup-agents --remove` can roll back cleanly. The current manifest
entry kinds are `file` and `json-key`
(`packages/cli/src/setup-agents.ts:42-50`); the MCP client writers for
claude-code and cursor add `json-key` entries for their respective config
files. Printed-only snippets for codex and opencode are **not** tracked in
the manifest, and `--remove` does not attempt to mutate user-written
files for those targets; the printed instructions tell the user how to
remove the entry by hand.
If the daemon is not running when `setup-agents` runs, the command prints a
follow-up hint: "Run `ktx mcp start` to enable the configured KTX MCP
server." It does **not** auto-start the daemon (matches the manual
lifecycle decision).
## Research skill
A new skill source file at `packages/cli/src/skills/research/SKILL.md`,
installed by `ktx setup-agents` to all configured targets. The skill is
separate from the existing setup skill (different triggers: "work in a KTX
project" vs. "answer a data question") and lives in its own per-target
folder so global vs. project scope and removal stay clean.
`plannedKtxAgentFiles` in `packages/cli/src/setup-agents.ts:64` is extended
to return both the existing `ktx` entries and new `ktx-research` entries:
| Target | Scope | Path |
|---|---|---|
| claude-code | global | `~/.claude/skills/ktx-research/SKILL.md` |
| claude-code | project | `.claude/skills/ktx-research/SKILL.md` |
| codex | global | `${CODEX_HOME}/skills/ktx-research/SKILL.md` |
| codex | project | `.agents/skills/ktx-research/SKILL.md` |
| cursor | project | `.cursor/rules/ktx-research.mdc` |
| opencode | project | `.opencode/commands/ktx-research.md` |
| universal | project | `.agents/skills/ktx-research/SKILL.md` |
The skill body is identical across targets; only the wrapper format and
file path differ to match each target's convention.
### Skill content
```markdown
---
name: ktx-research
description: Use when answering a question that needs data from a KTX-connected database — investigating, analyzing, "how many", "show me", "what's the breakdown of", finding records by value, exploring tables, comparing periods, or any data-investigation request. Triggers even when the user does not say "research"; if the answer requires querying a configured KTX connection, this skill applies.
---
# KTX Research Workflow
You have access to KTX MCP tools for investigating data. Follow this workflow.
<workflow>
1. **Discover** — call `discover_data(query)` first to see what exists across wiki, semantic-layer sources, and raw tables. Returns refs only.
2. **Inspect top hits in parallel** — for each promising ref:
- `kind: 'wiki'` → `wiki_read(key)`
- `kind: 'sl_source'` / `'sl_measure'` / `'sl_dimension'` → `sl_read_source(connectionId, sourceName)`
- `kind: 'table'` / `'column'` → `entity_details(connectionId, entities)`
3. **Resolve literals** — if the user named a value (e.g., "Acme Corp", "status=shipped"), call `dictionary_search(values)` to find which column holds it.
4. **Query** —
- Prefer `sl_query` when the semantic layer covers the question (joins, measures pre-defined).
- Use `sql_execution` only for things the semantic layer doesn't cover.
5. **Capture learnings** — at the end of the turn, call `memory_capture(userMessage, assistantMessage)` so future turns benefit. Skip when the answer carries no durable knowledge (e.g., the user only asked for schema info).
</workflow>
<rules>
- Always run `discover_data` before writing SQL. Do not guess table names.
- Prefer the semantic layer over raw SQL when both can answer the question — measures are the source of truth.
- Read entity details before writing SQL against an unfamiliar table; do not assume column names.
- Treat `sql_execution` as read-only. Writes are rejected by the server.
- Validate value mentions with `dictionary_search` instead of guessing case/spelling — but treat a `dictionary_search` *miss* as non-authoritative. The index is built from profile-sampled values, so a missing value may simply have been outside the sample. Follow up with `sql_execution` against the most plausible columns before concluding the value is absent.
</rules>
<examples>
**Input:** "How many orders did Acme Corp place last month?"
**Output workflow:**
1. `dictionary_search(["Acme Corp"])` → finds `customers.name`
2. `discover_data("orders customer monthly")` → finds `orders_facts` SL source
3. `sl_read_source("warehouse", "orders_facts")` → confirms measure `order_count`, dim `customer_name`, dim `ordered_at`
4. `sl_query({ measures: ["order_count"], filters: ["customer_name = 'Acme Corp'", "ordered_at >= date_trunc('month', now() - interval '1 month')"], dimensions: [{ field: "ordered_at", granularity: "month" }] })`
5. `memory_capture(userMessage, assistantMessage)`
---
**Input:** "What columns does the events table have?"
**Output workflow:**
1. `discover_data("events table")` → top hit `kind: 'table', id: 'analytics.events'`
2. `entity_details("warehouse", [{ table: "analytics.events" }])` → returns columns, types, FKs
3. Answer directly. (No query needed; no `memory_capture` since no durable learning.)
</examples>
```
## Files
### New
- `packages/context/src/scan/entity-details.ts` — derives entity-detail
records from `KtxSchemaSnapshot`, sharing resolution logic with
`warehouse-verification/warehouse-catalog.service.ts` (refactored or
imported, not duplicated).
- `packages/context/src/sl/dictionary-search.ts` — builds and queries the
dictionary index over relationship-profiling artifacts.
- `packages/context/src/search/discover.ts` — composes wiki, SL, and raw
schema searches; fuses results via `rrf.ts`. Reuses the same wiki/SL/raw
search building blocks as `warehouse-verification/discover-data.tool.ts`.
- `packages/cli/src/commands/mcp-commands.ts` — `ktx mcp start|stop|status|logs`.
- `packages/cli/src/managed-mcp-daemon.ts` — daemon lifecycle (spawn,
pidfile, log management), mirroring `managed-python-daemon.ts`.
- `packages/cli/src/skills/research/SKILL.md` — research workflow skill.
- Tests for each new module following existing patterns
(`*.test.ts` siblings), including coverage of the per-client config
writer/printer matrix.
### Modified
- `packages/context/src/mcp/context-tools.ts` — register the four new tools
with their Zod schemas.
- `packages/context/src/mcp/server.ts` — extend `KtxMcpContextPorts` with the
new ports (`sqlExecution`, `entityDetails`, `dictionarySearch`, `discover`).
- `packages/context/src/mcp/types.ts` — add the new port interface
definitions.
- `packages/cli/src/cli-program.ts` — register the `mcp` command subtree.
- `packages/cli/src/setup-agents.ts` — install the research skill and write
MCP client config entries to each configured target.
## Testing strategy
- Unit tests for each new module (`entity-details.ts`,
`dictionary-search.ts`, `discover.ts`) using existing fixture patterns.
- MCP-level integration test in `packages/context/src/mcp/server.test.ts`
that registers a fake server, invokes each tool, and asserts the
responses.
- CLI integration test for `ktx mcp start|stop|status` lifecycle following
the pattern in `managed-python-daemon.test.ts`.
- Setup-agents tests verifying behavior per target: claude-code and cursor
writers add the correct JSON entry and a corresponding `json-key`
manifest entry that `--remove` cleans up; codex and opencode targets
produce printed snippet output and do not mutate any user config file
or add manifest entries in v1.
- Verification commands per CLAUDE.md: `pnpm --filter @ktx/context run test`
and `pnpm --filter @ktx/cli run test` for the affected packages, plus
`pnpm run type-check`.
## Out of scope / follow-ups
- **Python code execution via MCP.** The daemon's `/code/execute` endpoint
exists; surfacing it via MCP is a separate design with sandbox/security
considerations.
- **Stdio MCP transport.** HTTP-only for now. Stdio can be added later as an
additional transport mode without changing the tool surface.
- **OS-level auto-start.** Manual `ktx mcp start` only. Adding launchd /
systemd unit installation is a UX polish for a later release.
- **TLS in the daemon itself.** Reverse proxy is the documented path. Native
TLS support if/when demand emerges.
- **Multi-project / project-switching MCP.** One daemon per project. A
cross-project model would require per-call `projectDir` arguments or a
`set_project_dir` tool and is deferred.
- **Audit logging, rate limiting, per-tool authorization.** Not in scope for
v1; the security boundary is loopback or bearer token.
## Open trade-offs
- **`dictionary_search` requires `--deep` (enriched) scan to have run.** The
relationship-profiling artifact that powers the dictionary index is only
produced by enriched scans. The tool reports this distinctly when missing,
but the dependency is real: without enriched scan, the tool returns
empty.
- **`entity_details` reads from the latest snapshot, not live.** If the
database schema changes after the last scan, the tool will reflect the
scan state, not reality. Surfacing this clearly in the tool's response
(snapshot timestamp) is part of the implementation.
- **No streaming for `sql_execution`.** Large results are capped at
`maxRows` (default 1000, max 10k). The tool returns the full result set
in one response. Streaming partial results is left for a later iteration
if real workloads demand it.