apunkt/ktx
1
0
Fork 0
mirror of https://github.com/Kaelio/ktx.git synced 2026-07-25 12:01:03 +02:00
Code Issues Projects Releases Packages Wiki Activity

Merge origin/main into divergent-implementation-audit

Browse source 
Reconciles this branch's audit fixes with main's independently-landed work
where they overlapped:

- MCP Python-runtime install: main's #297 (createLazyManagedPython-
  SemanticLayerComputePort + self-provisioned uv, shipped in 0.12.0)
  supersedes this branch's redundant 96fb94c8. Adopted main's canonical lazy
  port; dropped the duplicate single-function lazy fix.
- mcp-server-factory.ts: kept main's lazy semantic-compute port AND this
  branch's resolveSqlAnalysisPort (honors KTX_SQL_ANALYSIS_URL/KTX_DAEMON_URL,
  a strict superset of main's createManagedDaemonSqlAnalysisPort).
- doctor.test.ts: configured embeddings.openai.api_key on the ready-path
  tests to match 7692d92e's corrected status semantics (status no longer
  reports openai embeddings ready from a bare OPENAI_API_KEY env var).

Verified: type-check, build, default suite (3010 tests), connection/sl/
mcp-port excluded tests (67 tests), dead-code gate — all green.
This commit is contained in:
Andrey Avtomonov 2026-06-18 14:22:31 +02:00
commit 6b7701487a
95 changed files with 4485 additions and 587 deletions
Download patch file Download diff file Expand all files Collapse all files

2
assets/star-history.svg
View file

File diff suppressed because one or more lines are too long
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 63 KiB

After

Width:  |  Height:  |  Size: 63 KiB

Before After
Before After

5
docs-site/content/docs/cli-reference/ktx-admin.mdx
View file

@ -48,6 +48,11 @@ directory. Use it from any directory to generate editor or agent schema files.
| `stop` | Stop the **ktx** daemon |
| `status` | Show managed Python runtime status and readiness checks |
`install` is self-contained: **ktx** downloads its own pinned, checksum-verified
`uv` build under the runtime root and uses it to provision Python and the
runtime wheel. Nothing needs to be installed on `PATH` first; the host only
needs network access to `github.com` during the first install.
## `admin runtime` Options
| Flag | Description | Default |

1
docs-site/content/docs/cli-reference/ktx-mcp.mdx
View file

@ -68,3 +68,4 @@ hosts and origins for browser clients.
| No **ktx** project found | Current directory has no `ktx.yaml` and `KTX_PROJECT_DIR` is unset | Run from a **ktx** project or pass `--project-dir <path>` |
| Non-loopback host rejected | The server needs token auth before binding beyond localhost | Pass `--token <token>` or set `KTX_MCP_TOKEN` |
| Client cannot connect | Host, port, token, allowed host, or allowed origin does not match the client | Check `ktx mcp status`, then restart with explicit `--host`, `--port`, `--allowed-host`, and `--allowed-origin` values |
| A Python-backed tool reports a runtime install failure | A tool that needs the managed Python runtime (metric compute, query-history SQL analysis) ran on a host that cannot reach `github.com` to download the pinned `uv` and Python | The server still starts and serves catalog and search tools. Restore network access and retry, or pre-build the runtime where network is available: `ktx admin runtime install --yes` |

1
docs-site/content/docs/cli-reference/ktx-setup.mdx
View file

@ -29,6 +29,7 @@ below.
| `--agents` | Install agent configuration and rules only | `false` |
| `--target <target>` | Agent target: `claude-code`, `claude-desktop`, `codex`, `cursor`, `opencode`, or `universal` | - |
| `--global` | Install agent integration into the global target scope for `claude-code` or `codex` | `false` |
| `--install-dir <path>` | Directory to install project-scoped agent config into. Defaults to the ktx project directory; resolved against the current directory and created if missing. Use it to install `.claude/`, `.mcp.json`, and rules where you open your agent (e.g. `--install-dir .`). Mutually exclusive with `--global` and `--local` | ktx project dir |
| `--yes` | Accept project creation and runtime install defaults where setup asks for confirmation | `false` |
| `--no-input` | Disable interactive terminal input | - |

154
docs-site/content/docs/concepts/cross-database-federation.mdx Normal file
View file

@ -0,0 +1,154 @@
---
title: Cross-database federation
description: How ktx federates postgres, mysql, and sqlite connections so a single read-only SQL query can join across them without copying data.
---
Cross-database federation lets a single read-only SQL query join tables that
live in different databases. **ktx** achieves this by embedding DuckDB and
using its `ATTACH` mechanism to connect each member database read-only. The
join executes inside DuckDB at query time — live data, no ETL, no copy.
You run federated queries as raw SQL against the `_ktx_federated` connection
(see [Querying the federated connection
directly](#querying-the-federated-connection-directly)). Semantic-layer queries
(`ktx sl query` / the `sl_query` tool) stay per-connection; pointing one at
`_ktx_federated` returns an error telling you to use raw SQL instead.
Federation activates automatically when a `ktx.yaml` file declares two or more
attach-compatible connections. There is nothing to configure and no federation
block to add. With zero or one compatible connection the behavior is unchanged.
## Which connections participate
The v1 federation engine supports three drivers:
| Driver | Participates in federation |
|--------|---------------------------|
| `postgres` | Yes |
| `mysql` | Yes |
| `sqlite` | Yes |
| `snowflake` | No — standalone connection |
| `bigquery` | No — standalone connection |
| `clickhouse` | No — standalone connection |
| `sqlserver` | No — standalone connection |
Non-participating connections continue to work exactly as they did. They are
queried independently; they do not appear as federation members.
## How it activates
**ktx** inspects the connections in `ktx.yaml` at startup. When it finds two or
more connections whose driver is `postgres`, `mysql`, or `sqlite`, it
instantiates the DuckDB federation engine and attaches each one read-only.
There is no `federation:` key, no opt-in flag, and no connection-level setting
to enable. The engine is derived entirely from what is already declared.
A minimal `ktx.yaml` that triggers federation:
```yaml
connections:
pg_books:
driver: postgres
url: "postgres://user:pass@localhost:5432/books" # pragma: allowlist secret
sqlite_reviews:
driver: sqlite
path: ./data/reviews.db
```
Two attach-compatible connections are present, so federation is active.
## Table naming in federated queries
Inside a federated query, postgres and mysql tables use a three-part name:
`connectionId.schema.table`. SQLite tables, which have no schema layer in
DuckDB, use the two-part form `connectionId.table`. In both cases the
connection's `id` field in `ktx.yaml` becomes the catalog name inside DuckDB.
If a connection `id` is not a bare SQL identifier — for example it contains a
hyphen, like `books-db` — double-quote it in the query the same way DuckDB
quotes any identifier: `"books-db".public.books`. Writing it unquoted
(`books-db.public.books`) is a SQL syntax error, not a federation feature.
For the example above:
- `pg_books.public.books` — the `books` table in the `public` schema of the
postgres connection
- `sqlite_reviews.reviews` — the `reviews` table in the sqlite connection
These fully qualified names are what you write when you query the federated
connection with raw SQL (see [Querying the federated connection
directly](#querying-the-federated-connection-directly)). A source file's own
`table:` field is not prefixed this way — see [Source files keep member-native
table refs](#source-files-keep-member-native-table-refs) below.
## Source names in the federated view
When you list or search semantic-layer sources under the federated connection,
each source's `name` is prefixed with its member connection id — for example
`pg_books.books` and `sqlite_reviews.reviews`. The prefix keeps names unique
when two members own a table with the same name: a `users` table in each of
`pg_app` and `sqlite_app` surfaces as `pg_app.users` and `sqlite_app.users`
rather than colliding on a bare `users`.
## Source files keep member-native table refs
A source file's physical `table:` field is not prefixed with the connection id.
It stays the member-native reference the connector uses on its own —
`public.books` for the postgres member, `reviews` for the sqlite member —
because the same file backs a per-connection semantic-layer query against that
member, which runs on the member's own driver where a `pg_books.` catalog prefix
would point at a database that does not exist. The connection-id prefix is a
DuckDB catalog name that appears only in raw federated SQL; the member prefix on
the source `name` (above) is independent of it.
## Cross-database joins
Write a cross-database join as raw SQL against `_ktx_federated` — see
[Querying the federated connection
directly](#querying-the-federated-connection-directly) below for a runnable
example. DuckDB attaches both members and resolves the join live at query time.
Declaring the join in a source file's `joins:` block is not supported yet. The
semantic layer plans each connection on its own, so a `joins:` entry whose `to:`
points at a table in another member is not resolved across the federation
boundary. Until that lands, express cross-database joins as raw SQL.
## Querying the federated connection directly
The federated connection is addressable by its id,
`_ktx_federated`, anywhere **ktx** runs read-only SQL. The same id works for the
`ktx sql` command and for a data agent calling the `sql_execution` MCP tool, so
both surfaces can run a cross-database query without a source file:
```bash
ktx sql -c _ktx_federated \
"SELECT b.title, avg(r.rating) AS avg_rating
FROM pg_books.public.books b
JOIN sqlite_reviews.reviews r ON b.id = r.book_id
GROUP BY b.title"
```
Table names follow the rules from
[Table naming in federated queries](#table-naming-in-federated-queries):
three-part `connectionId.schema.table` for postgres and mysql, two-part
`connectionId.table` for sqlite. The `_ktx_federated` id is virtual — it is
never written to `ktx.yaml` and only exists when two or more attach-compatible
connections are declared. It surfaces in `ktx connection` and in the agent's
connection list so the id is discoverable. Querying a single member database
directly with its own connection id (`ktx sql -c pg_books ...`) is unchanged.
## Federated queries are read-only
DuckDB attaches every member database with read-only access. Federated queries
are `SELECT`/`WITH` only. No writes, no DDL, and no mutations reach any member
database through the federation engine.
## Current limitations
- **Raw SQL joins only.** Cross-database joins are written as raw SQL; declaring
them in a source's `joins:` block and automatic discovery of cross-database
relationships are not available yet. Intra-database relationship discovery for
each member connection is unchanged.
- **postgres, mysql, and sqlite only.** Other drivers (snowflake, bigquery,
clickhouse, sqlserver) do not participate in federation in this version. They
remain usable as standalone connections.

2
docs-site/content/docs/concepts/meta.json
View file

@ -1,5 +1,5 @@
{
"title": "Concepts",
"defaultOpen": true,
"pages": ["the-context-layer", "semantic-layer-internals", "wiki-retrieval"]
"pages": ["the-context-layer", "semantic-layer-internals", "cross-database-federation", "wiki-retrieval"]
}

19
docs-site/content/docs/integrations/agent-clients.mdx
View file

@ -68,19 +68,30 @@ If you choose an install mode, it then asks which targets to install:
└
```
When every selected target supports both project and global setup, the command
also asks where to install supported agent config:
When at least one selected target supports project-scoped setup, the command
asks where to install agent config:
```txt
◆ Where should ktx install supported agent config?
◆ Where should ktx install agent config?
│
│ ktx project: /path/to/your/ktx-project
│
│ ○ Project scope (ktx project directory)
│ ○ ktx project directory /path/to/your/ktx-project
│ ○ Current directory /path/to/where/you/ran/ktx
│ ○ Custom directory… (enter a path)
│ ○ Global scope (user config)
└
```
The first three choices write project-scoped files (`.claude/`, `.mcp.json`,
`.cursor/`, skills, and rules) into the chosen directory while still pointing
them at this ktx project. Use **Current directory** or **Custom directory…**
when you open your coding agent from somewhere other than the ktx project
directory. **Current directory** is hidden when it is already the ktx project
directory, and **Global scope** appears only when every selected target
supports global setup. Non-interactive runs pass `--install-dir <path>` (for
example `--install-dir .`) for the same result.
## Generated files
**ktx** writes MCP client configuration and analytics guidance by default. It writes

2
package.json
View file

@ -1,6 +1,6 @@
{
"name": "ktx-workspace",
"version": "0.11.0",
"version": "0.12.0",
"description": "Workspace root for ktx packages",
"private": true,
"type": "module",

3
packages/cli/package.json
View file

@ -1,6 +1,6 @@
{
"name": "@kaelio/ktx",
"version": "0.11.0",
"version": "0.12.0",
"description": "Standalone ktx context layer for data agents",
"author": {
"name": "Kaelio",
@ -55,6 +55,7 @@
"@clack/prompts": "1.4.0",
"@clickhouse/client": "^1.18.5",
"@commander-js/extra-typings": "14.0.0",
"@duckdb/node-api": "1.5.3-r.3",
"@google-cloud/bigquery": "^8.3.1",
"@looker/sdk": "^26.8.0",
"@looker/sdk-node": "^26.8.0",

17
packages/cli/src/commands/setup-commands.ts
View file

@ -89,6 +89,7 @@ function shouldShowSetupEntryMenu(
target?: string;
global?: boolean;
local?: boolean;
installDir?: string;
skipAgents?: boolean;
yes?: boolean;
input?: boolean;
@ -159,6 +160,7 @@ function shouldShowSetupEntryMenu(
'target',
'global',
'local',
'installDir',
'skipAgents',
'yes',
'input',
@ -217,6 +219,10 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
)
.option('--global', 'Install agent integration into the global target scope', false)
.option('--local', 'Install Claude Code MCP config into the private per-project ~/.claude.json scope', false)
.option(
'--install-dir <path>',
'Directory to install project-scoped agent config into (defaults to the ktx project directory)',
)
.addOption(new Option('--skip-agents', 'Leave agent integration incomplete for now').hideHelp().default(false))
.option('--yes', 'Accept project creation and runtime install defaults where setup confirms', false)
.option('--no-input', 'Disable interactive terminal input')
@ -394,6 +400,16 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
context.setExitCode(1);
return;
}
if (options.installDir && (options.global || options.local)) {
context.io.stderr.write('Choose either --install-dir or a scope flag (--global / --local), not both.\n');
context.setExitCode(1);
return;
}
if (options.installDir && options.target === 'claude-desktop') {
context.io.stderr.write('--install-dir does not apply to --target claude-desktop, which is always global.\n');
context.setExitCode(1);
return;
}
const creatingDatabaseConnection = options.database.length > 0 || options.databaseUrl !== undefined;
if (creatingDatabaseConnection && options.databaseConnectionId.length > 1) {
@ -412,6 +428,7 @@ export function registerSetupCommands(program: Command, context: KtxCliCommandCo
agents: options.agents === true,
...(options.target ? { target: options.target } : {}),
agentScope: resolvedAgentScope,
...(options.installDir ? { installRoot: options.installDir } : {}),
skipAgents: options.skipAgents === true,
inputMode: options.input === false ? 'disabled' : 'auto',
...(debugEnabled ? { debug: true } : {}),

17
packages/cli/src/connection.ts
View file

@ -8,6 +8,7 @@ import { type NotionBotInfo, NotionClient } from './context/ingest/adapters/noti
import { createLocalLookerCredentialResolver } from './context/ingest/adapters/looker/local-looker.adapter.js';
import { metabaseRuntimeConfigFromLocalConnection } from './context/ingest/adapters/metabase/local-metabase.adapter.js';
import { testRepoConnection } from './context/ingest/repo-fetch.js';
import { federatedConnectionListing } from './context/connections/federation.js';
import { getDriverRegistration } from './context/connections/drivers.js';
import { parseNotionConnectionConfig, resolveNotionConnectionAuthToken } from './context/connections/notion-config.js';
import { resolveKtxConfigReference } from './context/core/config-reference.js';
@ -447,15 +448,23 @@ export async function runKtxConnection(
io.stdout.write('No connections configured. Run `ktx setup` to add one.\n');
return 0;
}
const idWidth = Math.max('ID'.length, ...entries.map(([id]) => id.length));
const driverWidth = Math.max(
'DRIVER'.length,
const federated = federatedConnectionListing(project.config.connections, args.projectDir);
const idCandidates = [...entries.map(([id]) => id), ...(federated ? [federated.id] : [])];
const driverLengths = [
...entries.map(([, c]) => (c.driver ?? 'unknown').length),
);
...(federated ? [federated.driver.length] : []),
];
const idWidth = Math.max('ID'.length, ...idCandidates.map((id) => id.length));
const driverWidth = Math.max('DRIVER'.length, ...driverLengths);
io.stdout.write(`${'ID'.padEnd(idWidth)} ${'DRIVER'.padEnd(driverWidth)}\n`);
for (const [id, connection] of entries) {
io.stdout.write(`${id.padEnd(idWidth)} ${(connection.driver ?? 'unknown').padEnd(driverWidth)}\n`);
}
if (federated) {
io.stdout.write(`${federated.id.padEnd(idWidth)} ${federated.driver.padEnd(driverWidth)}\n`);
io.stdout.write(` federates: ${federated.members.join(', ')}\n`);
io.stdout.write(` ${federated.hint}\n`);
}
return 0;
}

16
packages/cli/src/connectors/bigquery/connector.ts
View file

@ -26,9 +26,7 @@ import {
type KtxTableSampleInput,
type KtxTableSampleResult,
} from '../../context/scan/types.js';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import { resolveStringReference } from '../shared/string-reference.js';
export interface KtxBigQueryConnectionConfig {
driver?: string;
@ -138,18 +136,6 @@ class DefaultBigQueryClientFactory implements KtxBigQueryClientFactory {
}
}
function resolveStringReference(value: string, env: NodeJS.ProcessEnv): string {
if (value.startsWith('env:')) {
return env[value.slice('env:'.length)] ?? '';
}
if (value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}
function stringConfigValue(
connection: KtxBigQueryConnectionConfig | undefined,
key: keyof KtxBigQueryConnectionConfig,

17
packages/cli/src/connectors/clickhouse/connector.ts
View file

@ -3,10 +3,8 @@ import { getDialectForDriver } from '../../context/connections/dialects.js';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { connectorTestFailure, createKtxConnectorCapabilities, type KtxConnectorTestResult, type KtxColumnSampleInput, type KtxColumnSampleResult, type KtxColumnStatsInput, type KtxColumnStatsResult, type KtxQueryResult, type KtxReadOnlyQueryInput, type KtxScanConnector, type KtxScanContext, type KtxScanInput, type KtxSchemaColumn, type KtxSchemaSnapshot, type KtxSchemaTable, type KtxTableRef, type KtxTableSampleInput, type KtxTableListEntry, type KtxTableSampleResult } from '../../context/scan/types.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
import { readFileSync } from 'node:fs';
import { resolveStringReference } from '../shared/string-reference.js';
import { Agent as HttpsAgent } from 'node:https';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
export interface KtxClickHouseConnectionConfig {
driver?: string;
@ -142,19 +140,6 @@ function stringConfigValue(
return typeof value === 'string' && value.trim().length > 0 ? resolveStringReference(value.trim(), env) : undefined;
}
function resolveStringReference(value: string, env: NodeJS.ProcessEnv): string {
if (value.startsWith('env:')) {
const envName = value.slice('env:'.length);
return env[envName] ?? '';
}
if (value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}
function maybeNumber(value: unknown): number | undefined {
return typeof value === 'number' && Number.isFinite(value) ? value : undefined;
}

90
packages/cli/src/connectors/duckdb/federated-attach.ts Normal file
View file

@ -0,0 +1,90 @@
import { sqliteDatabasePathFromConfig, type KtxSqliteConnectionConfig } from '../sqlite/connector.js';
import { postgresPoolConfigFromConfig, type KtxPostgresConnectionConfig } from '../postgres/connector.js';
import {
mysqlConnectionPoolConfigFromConfig,
type KtxMysqlConnectionConfig,
} from '../mysql/connector.js';
import type { FederatedMember } from '../../context/connections/federation.js';
function kvKeyword(value: string): string {
// libpq/DuckDB key-value values quote with single quotes and backslash-escape.
return /[\s'\\]/.test(value) ? `'${value.replaceAll('\\', '\\\\').replaceAll("'", "\\'")}'` : value;
}
function withRequiredSslMode(connectionString: string): string {
// DuckDB passes this libpq URL straight to the server, so an ssl:true member
// must carry sslmode in the URL itself; keep a stronger mode the URL already pins.
const url = new URL(connectionString);
if (url.searchParams.has('sslmode')) {
return connectionString;
}
url.searchParams.set('sslmode', 'require');
return url.toString();
}
function postgresAttachString(member: FederatedMember, env: NodeJS.ProcessEnv): string {
const cfg = postgresPoolConfigFromConfig({
connectionId: member.connectionId,
connection: member.connection as KtxPostgresConnectionConfig,
env,
});
if (cfg.connectionString) {
return cfg.ssl ? withRequiredSslMode(cfg.connectionString) : cfg.connectionString;
}
const parts: string[] = [];
if (cfg.host) parts.push(`host=${kvKeyword(cfg.host)}`);
if (cfg.port) parts.push(`port=${cfg.port}`);
if (cfg.database) parts.push(`dbname=${kvKeyword(cfg.database)}`);
if (cfg.user) parts.push(`user=${kvKeyword(cfg.user)}`);
if (cfg.password) parts.push(`password=${kvKeyword(cfg.password)}`);
if (cfg.ssl) {
parts.push('sslmode=require');
}
if (cfg.options) {
parts.push(`options=${kvKeyword(cfg.options)}`);
}
return parts.join(' ');
}
function mysqlAttachString(member: FederatedMember, env: NodeJS.ProcessEnv): string {
const cfg = mysqlConnectionPoolConfigFromConfig({
connectionId: member.connectionId,
connection: member.connection as KtxMysqlConnectionConfig,
env,
});
const parts: string[] = [
`host=${kvKeyword(cfg.host)}`,
`port=${cfg.port}`,
`database=${kvKeyword(cfg.database)}`,
`user=${kvKeyword(cfg.user)}`,
];
if (cfg.password) {
parts.push(`password=${kvKeyword(cfg.password)}`);
}
if (cfg.ssl) {
parts.push('ssl_mode=REQUIRED');
}
return parts.join(' ');
}
/**
* Resolves a federated member's ktx.yaml config into the connection target
* DuckDB's ATTACH wants for that driver, reusing each connector's canonical
* resolver so federation and standalone scans agree on config interpretation.
*/
export function federatedAttachTarget(member: FederatedMember, env: NodeJS.ProcessEnv): string {
switch (member.driver.toLowerCase()) {
case 'sqlite':
return sqliteDatabasePathFromConfig({
connectionId: member.connectionId,
projectDir: member.projectDir,
connection: member.connection as KtxSqliteConnectionConfig,
});
case 'postgres':
return postgresAttachString(member, env);
case 'mysql':
return mysqlAttachString(member, env);
default:
throw new Error(`Driver "${member.driver}" cannot be attached by DuckDB federation.`);
}
}

78
packages/cli/src/connectors/duckdb/federated-executor.ts Normal file
View file

@ -0,0 +1,78 @@
import { DuckDBInstance } from '@duckdb/node-api';
import { federatedAttachTarget } from './federated-attach.js';
import type {
KtxSqlQueryExecutionInput,
KtxSqlQueryExecutionResult,
} from '../../context/connections/query-executor.js';
import { normalizeQueryRows } from '../../context/connections/query-executor.js';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { attachTypeForDriver, type FederatedMember } from '../../context/connections/federation.js';
function quoteDuckdbIdentifier(id: string): string {
return `"${id.replaceAll('"', '""')}"`;
}
const MIN_SAFE_BIGINT = BigInt(Number.MIN_SAFE_INTEGER);
const MAX_SAFE_BIGINT = BigInt(Number.MAX_SAFE_INTEGER);
// DuckDB returns integer columns as JS bigint (unserializable by JSON). Values
// in Number's safe range become Number; larger magnitudes become strings so a
// BIGINT beyond 2^53 keeps its exact value instead of silently rounding.
function jsonSafeBigint(value: bigint): number | string {
return value >= MIN_SAFE_BIGINT && value <= MAX_SAFE_BIGINT ? Number(value) : value.toString();
}
function toJsonSafeRows(rows: unknown[][]): unknown[][] {
return rows.map((row) => row.map((cell) => (typeof cell === 'bigint' ? jsonSafeBigint(cell) : cell)));
}
/** @internal */
export function buildAttachStatements(members: FederatedMember[], env: NodeJS.ProcessEnv): string[] {
const attachments = members.map((member) => ({
type: attachTypeForDriver(member.driver),
url: federatedAttachTarget(member, env),
alias: member.connectionId,
}));
const loadStatements = [...new Set(attachments.map((a) => a.type))].map(
(type) => `INSTALL ${type}; LOAD ${type};`,
);
const attachStatements = attachments.map(
({ type, url, alias }) =>
`ATTACH '${url.replaceAll("'", "''")}' AS ${quoteDuckdbIdentifier(alias)} (TYPE ${type}, READ_ONLY);`,
);
return [...loadStatements, ...attachStatements];
}
export async function executeFederatedQuery(
members: FederatedMember[],
input: KtxSqlQueryExecutionInput,
env: NodeJS.ProcessEnv = process.env,
): Promise<KtxSqlQueryExecutionResult> {
const sql = limitSqlForExecution(assertReadOnlySql(input.sql), input.maxRows);
const attachStatements = buildAttachStatements(members, env);
const instance = await DuckDBInstance.create(':memory:');
try {
const connection = await instance.connect();
try {
for (const statement of attachStatements) {
await connection.run(statement);
}
const reader = await connection.runAndReadAll(sql);
const rows = toJsonSafeRows(normalizeQueryRows(reader.getRows()));
const headers = reader.columnNames();
return {
headers,
rows,
totalRows: rows.length,
command: 'SELECT',
rowCount: rows.length,
};
} finally {
connection.closeSync();
}
} finally {
instance.closeSync();
}
}

17
packages/cli/src/connectors/mysql/connector.ts
View file

@ -1,8 +1,6 @@
import mysql, { type FieldPacket, type Pool, type RowDataPacket } from 'mysql2/promise';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import { getDialectForDriver } from '../../context/connections/dialects.js';
import { resolveStringReference } from '../shared/string-reference.js';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import {
constraintDiscoveryWarning,
@ -183,19 +181,6 @@ function stringConfigValue(
return typeof value === 'string' && value.trim().length > 0 ? resolveStringReference(value.trim(), env) : undefined;
}
function resolveStringReference(value: string, env: NodeJS.ProcessEnv): string {
if (value.startsWith('env:')) {
const envName = value.slice('env:'.length);
return env[envName] ?? '';
}
if (value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}
function maybeNumber(value: unknown): number | undefined {
return typeof value === 'number' && Number.isFinite(value) ? value : undefined;
}

15
packages/cli/src/connectors/postgres/connector.ts
View file

@ -1,6 +1,4 @@
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import { resolveStringReference } from '../shared/string-reference.js';
import { getDialectForDriver } from '../../context/connections/dialects.js';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
@ -281,17 +279,6 @@ function stringConfigValue(
return typeof value === 'string' && value.trim().length > 0 ? resolveStringReference(value.trim(), env) : undefined;
}
function resolveStringReference(value: string, env: NodeJS.ProcessEnv): string {
if (value.startsWith('env:')) {
return env[value.slice('env:'.length)] ?? '';
}
if (value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}
function numberValue(value: unknown): number | undefined {
return typeof value === 'number' && Number.isFinite(value) ? value : undefined;

20
packages/cli/src/connectors/shared/string-reference.ts Normal file
View file

@ -0,0 +1,20 @@
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
/**
* Resolves a config string that may reference an environment variable
* (`env:NAME`) or a file (`file:/path`, `~` expands to the home dir).
* Plain values pass through unchanged.
*/
export function resolveStringReference(value: string, env: NodeJS.ProcessEnv): string {
if (value.startsWith('env:')) {
return env[value.slice('env:'.length)] ?? '';
}
if (value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(rawPath[1] === '/' ? 2 : 1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}

16
packages/cli/src/connectors/snowflake/connector.ts
View file

@ -1,8 +1,6 @@
import { createPrivateKey } from 'node:crypto';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import { getDialectForDriver } from '../../context/connections/dialects.js';
import { resolveStringReference } from '../shared/string-reference.js';
import { assertReadOnlySql, limitSqlForExecution } from '../../context/connections/read-only-sql.js';
import { tryConstraintQuery } from '../../context/scan/constraint-discovery.js';
import { scopedTableNames } from '../../context/scan/table-ref.js';
@ -135,18 +133,6 @@ export interface KtxSnowflakeColumnDistinctValuesResult {
const DATE_TYPES = ['DATE', 'TIMESTAMP', 'TIMESTAMP_LTZ', 'TIMESTAMP_NTZ', 'TIMESTAMP_TZ', 'TIME'];
function resolveStringReference(value: string, env: NodeJS.ProcessEnv): string {
if (value.startsWith('env:')) {
return env[value.slice('env:'.length)] ?? '';
}
if (value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}
function stringConfigValue(
connection: KtxSnowflakeConnectionConfig | undefined,
key: keyof KtxSnowflakeConnectionConfig,

16
packages/cli/src/connectors/sqlserver/connector.ts
View file

@ -25,10 +25,8 @@ import {
type KtxTableSampleInput,
type KtxTableSampleResult,
} from '../../context/scan/types.js';
import { readFileSync } from 'node:fs';
import { homedir } from 'node:os';
import { resolve } from 'node:path';
import sql from 'mssql';
import { resolveStringReference } from '../shared/string-reference.js';
export interface KtxSqlServerConnectionConfig {
driver?: string;
@ -208,18 +206,6 @@ function stringConfigValue(
return typeof value === 'string' && value.trim().length > 0 ? resolveStringReference(value.trim(), env) : undefined;
}
function resolveStringReference(value: string, env: NodeJS.ProcessEnv): string {
if (value.startsWith('env:')) {
return env[value.slice('env:'.length)] ?? '';
}
if (value.startsWith('file:')) {
const rawPath = value.slice('file:'.length);
const path = rawPath.startsWith('~') ? resolve(homedir(), rawPath.slice(1)) : rawPath;
return readFileSync(path, 'utf-8').trim();
}
return value;
}
function parseSqlServerUrl(url: string): Partial<KtxSqlServerConnectionConfig> {
const parsed = new URL(url);
return {

83
packages/cli/src/context/connections/federation.ts Normal file
View file

@ -0,0 +1,83 @@
import type { KtxProjectConnectionConfig } from '../project/config.js';
/** Stable id for the runtime-derived federated connection. Never written to ktx.yaml. */
export const FEDERATED_CONNECTION_ID = '_ktx_federated';
/**
* Drivers DuckDB can ATTACH for federation. The driver name doubles as the
* DuckDB extension/TYPE name, so this set is the single source of truth for
* both membership (a driver participates iff it appears here) and attach type.
*/
const ATTACH_COMPATIBLE_DRIVERS = new Set(['postgres', 'mysql', 'sqlite']);
export function attachTypeForDriver(driver: string): string {
const normalized = driver.toLowerCase();
if (!ATTACH_COMPATIBLE_DRIVERS.has(normalized)) {
throw new Error(`Driver "${driver}" cannot be attached by DuckDB federation.`);
}
return normalized;
}
export interface FederatedMember {
connectionId: string;
driver: string;
projectDir: string;
connection: KtxProjectConnectionConfig;
}
export interface FederatedConnectionDescriptor {
id: typeof FEDERATED_CONNECTION_ID;
driver: 'duckdb';
members: FederatedMember[];
}
/**
* Derives a virtual federated connection when a project declares 2+
* attach-compatible databases. Returns null otherwise — single-DB and
* incompatible projects are unaffected.
*/
export function deriveFederatedConnection(
connections: Record<string, KtxProjectConnectionConfig>,
projectDir: string,
): FederatedConnectionDescriptor | null {
const members: FederatedMember[] = Object.entries(connections)
.filter(([, config]) => ATTACH_COMPATIBLE_DRIVERS.has(config.driver.toLowerCase()))
.map(([connectionId, config]) => ({
connectionId,
driver: config.driver.toLowerCase(),
projectDir,
connection: config,
}));
if (members.length < 2) {
return null;
}
return { id: FEDERATED_CONNECTION_ID, driver: 'duckdb', members };
}
export interface FederatedConnectionListing {
id: typeof FEDERATED_CONNECTION_ID;
driver: 'duckdb';
members: string[];
hint: string;
}
/**
* Listing-facing view of the virtual federated connection for `ktx connection`
* and MCP `connection_list`. Derived from the same declared state as
* deriveFederatedConnection, so both surfaces describe one connection.
*/
export function federatedConnectionListing(
connections: Record<string, KtxProjectConnectionConfig>,
projectDir: string,
): FederatedConnectionListing | null {
const descriptor = deriveFederatedConnection(connections, projectDir);
if (!descriptor) {
return null;
}
return {
id: FEDERATED_CONNECTION_ID,
driver: 'duckdb',
members: descriptor.members.map((member) => member.connectionId),
hint: 'Cross-database queries run here. Name tables connectionId.schema.table (or connectionId.table for sqlite); double-quote any id that is not a bare SQL identifier, e.g. "books-db".public.books.',
};
}

2
packages/cli/src/context/connections/local-warehouse-descriptor.ts
View file

@ -16,6 +16,8 @@ export interface LocalConnectionInfo {
id: string;
name: string;
connectionType: string;
members?: string[];
hint?: string;
}
const DRIVER_TO_CONNECTION_TYPE: Record<string, ConnectionType> = {

58
packages/cli/src/context/connections/project-sql-executor.ts Normal file
View file

@ -0,0 +1,58 @@
import { executeFederatedQuery } from '../../connectors/duckdb/federated-executor.js';
import type { KtxLocalProject } from '../project/project.js';
import type { KtxScanConnector, KtxScanContext } from '../scan/types.js';
import { deriveFederatedConnection, FEDERATED_CONNECTION_ID } from './federation.js';
import type { KtxSqlQueryExecutionInput, KtxSqlQueryExecutionResult } from './query-executor.js';
export interface ExecuteProjectReadOnlySqlDeps {
project: KtxLocalProject;
input: KtxSqlQueryExecutionInput;
createConnector: (connectionId: string) => Promise<KtxScanConnector> | KtxScanConnector;
executeFederated?: typeof executeFederatedQuery;
runId?: string;
}
/**
* Single resolve-and-execute path for project read-only SQL. The federated
* connection is derived from declared state here so every executor entry point
* routes `_ktx_federated` identically; standard connections go through the
* scan connector.
*/
export async function executeProjectReadOnlySql(
deps: ExecuteProjectReadOnlySqlDeps,
): Promise<KtxSqlQueryExecutionResult> {
const { project, input } = deps;
if (input.connectionId === FEDERATED_CONNECTION_ID) {
const descriptor = deriveFederatedConnection(project.config.connections, project.projectDir);
if (!descriptor) {
throw new Error('Federated execution requested but fewer than 2 attach-compatible connections exist.');
}
const runFederated = deps.executeFederated ?? executeFederatedQuery;
return runFederated(descriptor.members, input);
}
let connector: KtxScanConnector | null = null;
try {
connector = await deps.createConnector(input.connectionId);
if (!connector.capabilities.readOnlySql || !connector.executeReadOnly) {
throw new Error(
`Connection "${input.connectionId}" driver "${connector.driver}" does not support read-only SQL execution.`,
);
}
const ctx: KtxScanContext = { runId: deps.runId ?? 'sql-execution' };
const result = await connector.executeReadOnly(
{ connectionId: input.connectionId, sql: input.sql, maxRows: input.maxRows },
ctx,
);
return {
headers: result.headers,
...(result.headerTypes ? { headerTypes: result.headerTypes } : {}),
rows: result.rows,
totalRows: result.totalRows,
command: 'SELECT',
rowCount: result.rowCount,
};
} finally {
await connector?.cleanup?.();
}
}

3
packages/cli/src/context/connections/query-executor.ts
View file

@ -8,8 +8,9 @@ export interface KtxSqlQueryExecutionInput {
maxRows?: number;
}
interface KtxSqlQueryExecutionResult {
export interface KtxSqlQueryExecutionResult {
headers: string[];
headerTypes?: string[];
rows: unknown[][];
totalRows: number;
command: string;

8
packages/cli/src/context/connections/read-only-sql.ts
View file

@ -1,3 +1,5 @@
import { KtxQueryError } from '../../errors.js';
const MUTATING_SQL =
/^\s*(insert|update|delete|merge|alter|drop|create|truncate|grant|revoke|copy|call|do|vacuum|analyze|refresh)\b/i;
const READ_SQL = /^\s*(select|with)\b/i;
@ -80,7 +82,7 @@ function assertSingleSqlStatement(sql: string): void {
if (sql[index] === ';') {
sawSemicolon = true;
} else if (sawSemicolon && !/\s/.test(sql[index])) {
throw new Error('Only one SQL statement can be executed.');
throw new KtxQueryError('Only one SQL statement can be executed.');
}
index += 1;
}
@ -89,7 +91,7 @@ function assertSingleSqlStatement(sql: string): void {
export function assertReadOnlySql(sql: string): string {
const trimmed = stripLeadingSqlComments(sql).trim();
if (!READ_SQL.test(trimmed) || MUTATING_SQL.test(trimmed)) {
throw new Error('Only read-only SELECT/WITH queries can be executed locally.');
throw new KtxQueryError('Only read-only SELECT/WITH queries can be executed locally.');
}
assertSingleSqlStatement(trimmed);
return trimmed;
@ -133,7 +135,7 @@ export function limitSqlForExecution(sql: string, maxRows: number | undefined):
return trimmed;
}
if (!Number.isInteger(maxRows) || maxRows <= 0) {
throw new Error('maxRows must be a positive integer.');
throw new KtxQueryError('maxRows must be a positive integer.');
}
return `select * from (${trimmed}) as ktx_query_result limit ${maxRows}`;
}

50
packages/cli/src/context/connections/resolve-connection.ts Normal file
View file

@ -0,0 +1,50 @@
import { KtxExpectedError } from '../../errors.js';
import type { KtxProjectConfig, KtxProjectConnectionConfig } from '../project/config.js';
function configuredConnectionIds(config: KtxProjectConfig): string[] {
return Object.keys(config.connections).sort();
}
function availableConnectionsHint(config: KtxProjectConfig): string {
const ids = configuredConnectionIds(config);
return ids.length === 0
? 'No connections are configured in ktx.yaml.'
: `Configured connections: ${ids.join(', ')}.`;
}
/**
* Look up a connection by id, throwing an expected (caller-driven) error that
* names the configured connections so an agent or CLI user can self-correct.
*/
export function resolveConfiguredConnection(
config: KtxProjectConfig,
connectionId: string,
): KtxProjectConnectionConfig {
const connection = config.connections[connectionId];
if (!connection) {
throw new KtxExpectedError(
`Connection "${connectionId}" is not configured in ktx.yaml. ${availableConnectionsHint(config)}`,
);
}
return connection;
}
/**
* Resolve the connection id to run against: validate a requested id against the
* configured connections, or default to the sole connection when none is given.
* Throws an expected error that lists the configured connections otherwise.
*/
export function resolveRequiredConnectionId(
config: KtxProjectConfig,
requested: string | undefined,
): string {
if (requested !== undefined) {
resolveConfiguredConnection(config, requested);
return requested;
}
const ids = configuredConnectionIds(config);
if (ids.length === 1) {
return ids[0];
}
throw new KtxExpectedError(`connectionId is required. ${availableConnectionsHint(config)}`);
}

6
packages/cli/src/context/core/git-env.ts
View file

@ -31,6 +31,10 @@ function sanitizedGitEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEn
* directory is an existing repo ktx did not create and the machine has no configured git
* identity (e.g. a fresh Mac with no ~/.gitconfig), without mutating the user's repo config.
* Explicit `--author` flags on individual commits still take precedence over GIT_AUTHOR_NAME.
*
* `commit.gpgsign=false` is injected as a per-invocation `-c` override so ktx's commits never
* attempt GPG signing: ktx commits under a synthetic identity that can never own a secret key, so
* a user's `commit.gpgsign=true` would otherwise fail every commit with "No secret key".
*/
export function createSimpleGit(baseDir: string, identity?: { name: string; email: string }): SimpleGit {
const env = sanitizedGitEnv();
@ -40,5 +44,5 @@ export function createSimpleGit(baseDir: string, identity?: { name: string; emai
env.GIT_COMMITTER_NAME = identity.name;
env.GIT_COMMITTER_EMAIL = identity.email;
}
return simpleGit({ baseDir, unsafe: { allowUnsafeAskPass: true } }).env(env);
return simpleGit({ baseDir, config: ['commit.gpgsign=false'], unsafe: { allowUnsafeAskPass: true } }).env(env);
}

39
packages/cli/src/context/ingest/adapters/live-database/manifest.ts
View file

@ -86,6 +86,9 @@ export interface BuildLiveDatabaseManifestShardsInput {
existingPreservedJoins?: Map<string, LiveDatabaseManifestJoinEntry[]>;
existingDescriptions?: Map<string, LiveDatabaseManifestExistingDescriptions>;
existingUsage?: Map<string, TableUsageOutput>;
// Table refs owned by other federated members; declared cross-DB joins to
// these survive even though the target has no shard in this snapshot.
federatedSiblingTargets?: Set<string>;
}
export interface BuildLiveDatabaseManifestShardsResult {
@ -204,15 +207,20 @@ function joinCondition(
.join(' AND ');
}
function buildJoinsByTable(
/** @internal */
export function buildJoinsByTable(
tableNames: Set<string>,
joins: LiveDatabaseManifestJoinData[],
preservedJoins: Map<string, LiveDatabaseManifestJoinEntry[]>,
federatedSiblingTargets: Set<string> = new Set(),
): Map<string, LiveDatabaseManifestJoinEntry[]> {
const joinsByTable = new Map<string, LiveDatabaseManifestJoinEntry[]>();
for (const join of joins) {
if (!tableNames.has(join.fromTable) || !tableNames.has(join.toTable)) {
const fromLocal = tableNames.has(join.fromTable);
const toLocal = tableNames.has(join.toTable);
const toSibling = federatedSiblingTargets.has(join.toTable);
if (!fromLocal || (!toLocal && !toSibling)) {
continue;
}
const relationship = RELATIONSHIP_MAP[join.relationship] ?? join.relationship;
@ -223,13 +231,17 @@ function buildJoinsByTable(
source: join.source,
});
const reverseRelationship = RELATIONSHIP_INVERSE[relationship] ?? 'one_to_many';
addJoinOnce(joinsByTable, join.toTable, {
to: join.fromTable,
on: joinCondition(join.toTable, join.toColumns, join.fromTable, join.fromColumns),
relationship: reverseRelationship,
source: join.source,
});
// Reverse direction only when the target is a local table in THIS snapshot;
// a federated sibling has no shard here, so it gets no reverse entry.
if (toLocal) {
const reverseRelationship = RELATIONSHIP_INVERSE[relationship] ?? 'one_to_many';
addJoinOnce(joinsByTable, join.toTable, {
to: join.fromTable,
on: joinCondition(join.toTable, join.toColumns, join.fromTable, join.fromColumns),
relationship: reverseRelationship,
source: join.source,
});
}
}
for (const [tableName, tableJoins] of preservedJoins) {
@ -237,7 +249,7 @@ function buildJoinsByTable(
continue;
}
for (const join of tableJoins) {
if (tableNames.has(join.to)) {
if (tableNames.has(join.to) || federatedSiblingTargets.has(join.to)) {
addJoinOnce(joinsByTable, tableName, join);
}
}
@ -250,7 +262,12 @@ export function buildLiveDatabaseManifestShards(
input: BuildLiveDatabaseManifestShardsInput,
): BuildLiveDatabaseManifestShardsResult {
const tableNames = new Set(input.tables.map((table) => table.name));
const joinsByTable = buildJoinsByTable(tableNames, input.joins, input.existingPreservedJoins ?? new Map());
const joinsByTable = buildJoinsByTable(
tableNames,
input.joins,
input.existingPreservedJoins ?? new Map(),
input.federatedSiblingTargets ?? new Set(),
);
const shards = new Map<string, LiveDatabaseManifestShard>();
for (const table of input.tables) {

23
packages/cli/src/context/llm/claude-code-runtime.ts
View file

@ -151,7 +151,26 @@ function expectedMcpServerNames(tools: KtxRuntimeToolSet | undefined): Set<strin
return tools && Object.keys(tools).length > 0 ? new Set([KTX_MCP_SERVER_NAME]) : new Set();
}
const CLAUDE_RATE_LIMIT_ERROR_MARKERS = /\b429\b|rate limit|too many requests|quota exceeded|overloaded|max_retries/i;
// "session limit" is the Claude Code subscription cap ("You've hit your session
// limit · resets …"); the rest are transient 429-style throttling. All mean
// Claude Code authenticated successfully, so they must not be read as auth
// failures by the governor classifier or the auth probe.
const CLAUDE_RATE_LIMIT_ERROR_MARKERS =
/\b429\b|rate limit|session limit|usage limit|too many requests|quota exceeded|overloaded|max_retries/i;
// The subscription cap is its own case: re-authenticating and retrying both fail
// until reset, so it gets a distinct message from transient rate limiting.
const CLAUDE_SESSION_LIMIT_MARKERS = /session limit|usage limit/i;
function describeClaudeProbeFailure(message: string): string {
if (CLAUDE_SESSION_LIMIT_MARKERS.test(message)) {
return `Claude Code session limit reached. Wait for the reset shown, then rerun setup or the command. Details: ${message}`;
}
if (CLAUDE_RATE_LIMIT_ERROR_MARKERS.test(message)) {
return `Claude Code is rate limited. Retry shortly, then rerun setup or the command. Details: ${message}`;
}
return `Claude Code authentication is not usable. Authenticate Claude Code locally with the Claude Code CLI, then rerun setup or the command. ${message}`;
}
function normalizeClaudeResetAtMs(value: unknown): number | undefined {
if (typeof value === 'number' && Number.isFinite(value) && value > 0) {
@ -497,7 +516,7 @@ export async function runClaudeCodeAuthProbe(input: {
const message = error instanceof Error ? error.message : String(error);
return {
ok: false,
message: `Claude Code authentication is not usable. Authenticate Claude Code locally with the Claude Code CLI, then rerun setup or the command. ${message}`,
message: describeClaudeProbeFailure(message),
};
}
}

32
packages/cli/src/context/mcp/context-tools.ts
View file

@ -10,7 +10,7 @@ import {
shouldEmitMcpTelemetry,
} from '../../telemetry/index.js';
import { collectTelemetryRedactionSecrets } from '../../telemetry/redaction-secrets.js';
import { scrubErrorClass } from '../../telemetry/scrubber.js';
import { formatErrorDetail, scrubErrorClass } from '../../telemetry/scrubber.js';
import type {
KtxMcpClientInfo,
KtxMcpContextPorts,
@ -56,7 +56,7 @@ const toolAnnotations = {
const toolDescriptions = {
connection_list:
'List configured read-only data connections available to this ktx project. Use this before connection-scoped tools when the project may have multiple warehouses.',
'List configured read-only data connections available to this ktx project. Use this before connection-scoped tools when the project may have multiple warehouses. A "_ktx_federated" entry (when present) queries all its member databases together; use its id for cross-database joins.',
discover_data:
'Search across ktx wiki pages, semantic-layer sources, measures, dimensions, raw tables, and columns. Example: discover_data({ query: "monthly orders by customer", connectionId: "warehouse", kinds: ["sl_source", "table"] }).',
wiki_search:
@ -227,6 +227,8 @@ const connectionListOutputSchema = z.object({
id: z.string(),
name: z.string(),
connectionType: z.string(),
members: z.array(z.string()).optional(),
hint: z.string().optional(),
}),
),
});
@ -564,6 +566,28 @@ function clientTelemetryFields(
};
}
// Tools registered via registerParsedTool catch their own errors and return an
// isError result, so the telemetry layer never sees the thrown Error. Recover
// the failure message from the result's text content (the same string the agent
// reads) so the outcome event is self-diagnosing.
function mcpErrorResultDetail(result: unknown): string | undefined {
if (typeof result !== 'object' || result === null || !('content' in result)) {
return undefined;
}
const content = (result as { content?: unknown }).content;
if (!Array.isArray(content)) {
return undefined;
}
const text = content
.map((block) =>
typeof block === 'object' && block !== null && typeof (block as { text?: unknown }).text === 'string'
? (block as { text: string }).text
: '',
)
.join('\n');
return formatErrorDetail(text);
}
function instrumentMcpServer(
server: KtxMcpServerLike,
telemetry: { projectDir?: string; io?: KtxCliIo; getClientInfo?: () => KtxMcpClientInfo | undefined },
@ -577,6 +601,7 @@ function instrumentMcpServer(
if (telemetry.io && telemetry.projectDir && shouldEmitMcpTelemetry()) {
const isError =
typeof result === 'object' && result !== null && 'isError' in result && result.isError === true;
const errorDetail = isError ? mcpErrorResultDetail(result) : undefined;
await emitTelemetryEvent({
name: 'mcp_request_completed',
projectDir: telemetry.projectDir,
@ -586,6 +611,7 @@ function instrumentMcpServer(
outcome: isError ? 'error' : 'ok',
durationMs: Math.max(0, performance.now() - startedAt),
sampleRate: mcpTelemetrySampleRate(),
...(errorDetail ? { errorDetail } : {}),
...clientTelemetryFields(telemetry.getClientInfo),
},
});
@ -608,6 +634,7 @@ function instrumentMcpServer(
}
if (telemetry.io && telemetry.projectDir && shouldEmitMcpTelemetry()) {
const errorClass = scrubErrorClass(error);
const errorDetail = formatErrorDetail(error);
await emitTelemetryEvent({
name: 'mcp_request_completed',
projectDir: telemetry.projectDir,
@ -616,6 +643,7 @@ function instrumentMcpServer(
toolName: name,
outcome: 'error',
...(errorClass ? { errorClass } : {}),
...(errorDetail ? { errorDetail } : {}),
durationMs: Math.max(0, performance.now() - startedAt),
sampleRate: mcpTelemetrySampleRate(),
...clientTelemetryFields(telemetry.getClientInfo),

122
packages/cli/src/context/mcp/local-project-ports.ts
View file

@ -1,11 +1,16 @@
import type { KtxSqlQueryExecutorPort } from '../../context/connections/query-executor.js';
import { KtxQueryError, isNativeProgrammingFault } from '../../errors.js';
import { localConnectionInfoFromConfig } from '../../context/connections/local-warehouse-descriptor.js';
import { KtxExpectedError, KtxQueryError, isNativeProgrammingFault } from '../../errors.js';
import { executeProjectReadOnlySql } from '../../context/connections/project-sql-executor.js';
import { FEDERATED_CONNECTION_ID, federatedConnectionListing } from '../../context/connections/federation.js';
import { resolveConfiguredConnection } from '../../context/connections/resolve-connection.js';
import {
type LocalConnectionInfo,
localConnectionInfoFromConfig,
} from '../../context/connections/local-warehouse-descriptor.js';
import type { KtxEmbeddingPort } from '../../context/core/embedding.js';
import type { KtxSemanticLayerComputePort } from '../../context/daemon/semantic-layer-compute.js';
import type { KtxLocalProject } from '../../context/project/project.js';
import { createKtxEntityDetailsService } from '../../context/scan/entity-details.js';
import type { KtxScanConnector } from '../../context/scan/types.js';
import type { LocalScanMcpOptions } from '../../context/scan/local-scan.js';
import { createKtxDiscoverDataService } from '../../context/search/discover.js';
import { sqlAnalysisDialectForDriver } from '../../context/sql-analysis/dialect.js';
@ -25,12 +30,6 @@ interface CreateLocalProjectMcpContextPortsOptions {
embeddingService: KtxEmbeddingPort | null;
}
async function cleanupConnector(connector: KtxScanConnector | null): Promise<void> {
if (connector?.cleanup) {
await connector.cleanup();
}
}
async function executeValidatedReadOnlySql(
project: KtxLocalProject,
options: CreateLocalProjectMcpContextPortsOptions,
@ -38,61 +37,57 @@ async function executeValidatedReadOnlySql(
onProgress?: KtxMcpProgressCallback,
): Promise<KtxSqlExecutionResponse> {
await onProgress?.({ progress: 0, message: 'Validating SQL' });
const connectionId = assertSafeConnectionId(input.connectionId);
const connection = project.config.connections[connectionId];
if (!connection) {
throw new Error(`Connection "${connectionId}" is not configured in ktx.yaml`);
}
if (!options.sqlAnalysis) {
throw new Error('sql_execution requires parser-backed SQL validation.');
}
const validation = await options.sqlAnalysis.validateReadOnly(input.sql, sqlAnalysisDialectForDriver(connection.driver));
if (!validation.ok) {
throw new Error(validation.error ?? 'SQL is not read-only.');
}
const createConnector = options.localScan?.createConnector;
if (!createConnector) {
throw new Error('sql_execution requires a local scan connector factory.');
}
let connector: KtxScanConnector | null = null;
try {
connector = await createConnector(connectionId);
if (!connector.capabilities.readOnlySql || !connector.executeReadOnly) {
throw new Error(`Connection "${connectionId}" does not support read-only SQL execution.`);
}
await onProgress?.({ progress: 0.3, message: 'Executing' });
const result = await connector
.executeReadOnly(
{
connectionId,
sql: input.sql,
maxRows: input.maxRows,
},
{ runId: 'mcp-sql-execution' },
)
.catch((error: unknown) => {
// A warehouse/driver rejection (e.g. the agent's SQL failed to compile)
// is a surfaced operational outcome, not a ktx fault: mark it expected
// while preserving the warehouse's own diagnostics. A native JS error
// (TypeError, etc.) signals a bug in connector code — let it propagate
// unchanged so Error Tracking still sees it.
if (isNativeProgrammingFault(error)) {
throw error;
}
throw new KtxQueryError(error instanceof Error ? error.message : String(error), { cause: error });
});
const response = {
headers: result.headers,
...(result.headerTypes ? { headerTypes: result.headerTypes } : {}),
rows: result.rows,
rowCount: result.rowCount ?? result.rows.length,
};
await onProgress?.({ progress: 1, message: `Fetched ${response.rowCount} rows` });
return response;
} finally {
await cleanupConnector(connector);
const isFederated = input.connectionId === FEDERATED_CONNECTION_ID;
const connectionId = isFederated ? input.connectionId : assertSafeConnectionId(input.connectionId);
const connection = isFederated ? undefined : resolveConfiguredConnection(project.config, connectionId);
const dialect = sqlAnalysisDialectForDriver(isFederated ? 'duckdb' : connection!.driver);
const validation = await options.sqlAnalysis.validateReadOnly(input.sql, dialect);
if (!validation.ok) {
// A read-only guard rejecting the agent's SQL is an expected outcome, not a
// ktx fault: classify it so reportException keeps it out of Error Tracking.
throw new KtxQueryError(validation.error ?? 'SQL is not read-only.');
}
await onProgress?.({ progress: 0.3, message: 'Executing' });
const result = await executeProjectReadOnlySql({
project,
input: {
connectionId,
projectDir: project.projectDir,
connection,
sql: input.sql,
maxRows: input.maxRows,
},
createConnector,
runId: 'mcp-sql-execution',
}).catch((error: unknown) => {
// A warehouse/driver rejection (e.g. the agent's SQL failed to compile) is a
// surfaced operational outcome, not a ktx fault: mark it expected while
// preserving the warehouse's own diagnostics. A native JS error (TypeError,
// etc.) signals a bug in connector code — let it propagate unchanged so Error
// Tracking still sees it.
if (isNativeProgrammingFault(error) || error instanceof KtxExpectedError) {
throw error;
}
throw new KtxQueryError(error instanceof Error ? error.message : String(error), { cause: error });
});
const response = {
headers: result.headers,
...(result.headerTypes ? { headerTypes: result.headerTypes } : {}),
rows: result.rows,
rowCount: result.rowCount ?? result.rows.length,
};
await onProgress?.({ progress: 1, message: `Fetched ${response.rowCount} rows` });
return response;
}
export function createLocalProjectMcpContextPorts(
@ -103,12 +98,21 @@ export function createLocalProjectMcpContextPorts(
const ports: KtxMcpContextPorts = {
connections: {
async list() {
return Object.entries(project.config.connections)
const configured = Object.entries(project.config.connections)
.map(([id, config]) => localConnectionInfoFromConfig(id, config))
.filter(
(connection): connection is { id: string; name: string; connectionType: string } => connection !== null,
)
.filter((connection): connection is LocalConnectionInfo => connection !== null)
.sort((a, b) => a.id.localeCompare(b.id));
const federated = federatedConnectionListing(project.config.connections, project.projectDir);
if (federated) {
configured.push({
id: federated.id,
name: federated.id,
connectionType: 'DUCKDB',
members: federated.members,
hint: federated.hint,
});
}
return configured;
},
},
knowledge: {

2
packages/cli/src/context/mcp/types.ts
View file

@ -78,6 +78,8 @@ interface KtxConnectionSummary {
id: string;
name: string;
connectionType: string;
members?: string[];
hint?: string;
}
interface KtxConnectionsMcpPort {

45
packages/cli/src/context/scan/local-enrichment-artifacts.ts
View file

@ -4,6 +4,7 @@ import type { TableUsageOutput } from '../../context/ingest/adapters/historic-sq
import type { KtxScanRelationshipConfig } from '../project/config.js';
import type { KtxLocalProject } from '../../context/project/project.js';
import { isSlYamlPath } from '../../context/sl/source-files.js';
import { deriveFederatedConnection } from '../connections/federation.js';
import type { KtxLocalScanEnrichmentResult } from './local-enrichment.js';
import {
buildKtxRelationshipArtifacts,
@ -193,10 +194,43 @@ function joinReferencesExistingColumns(
return true;
}
async function federatedSiblingTargets(
project: KtxLocalProject,
connectionId: string,
): Promise<Set<string>> {
const descriptor = deriveFederatedConnection(project.config.connections, project.projectDir);
if (!descriptor) {
return new Set();
}
const siblings = descriptor.members.filter((member) => member.connectionId !== connectionId);
const perSibling = await Promise.all(siblings.map((sibling) => siblingJoinTargets(project, sibling.connectionId)));
return new Set(perSibling.flat());
}
async function siblingJoinTargets(project: KtxLocalProject, connectionId: string): Promise<string[]> {
const listed = await project.fileStore.listFiles(schemaDir(connectionId)).catch(() => ({ files: [] }));
const files = listed.files.filter(isSlYamlPath);
const perFile = await Promise.all(
files.map(async (file) => {
const shard = await project.fileStore
.readFile(file)
.then(({ content }) => YAML.parse(content) as LiveDatabaseManifestShard | null)
.catch(() => null);
// entry.table is buildTableRef's member-local ref (1-3 parts:
// table / schema.table / catalog.schema.table), never connectionId-
// prefixed — so prefixing with the member id yields the fully-qualified
// `to:` form authored in cross-DB joins.
return Object.values(shard?.tables ?? {}).map((entry) => `${connectionId}.${entry.table}`);
}),
);
return perFile.flat();
}
async function loadExistingManifestState(
project: KtxLocalProject,
connectionId: string,
snapshot: KtxSchemaSnapshot,
siblingTargets: Set<string>,
): Promise<ExistingManifestState> {
const descriptions = new Map<string, LiveDatabaseManifestExistingDescriptions>();
const preservedJoins = new Map<string, LiveDatabaseManifestJoinEntry[]>();
@ -236,7 +270,7 @@ async function loadExistingManifestState(
const joins = (entry.joins ?? []).filter((join) => {
return (
(join.source === 'manual' || join.source === 'inferred') &&
validTableNames.has(join.to) &&
(validTableNames.has(join.to) || siblingTargets.has(join.to)) &&
joinReferencesExistingColumns(join, columnsByTable)
);
});
@ -277,7 +311,13 @@ export async function writeLocalScanManifestShards(
};
}
const existing = await loadExistingManifestState(input.project, input.connectionId, input.snapshot);
const siblingTargets = await federatedSiblingTargets(input.project, input.connectionId);
const existing = await loadExistingManifestState(
input.project,
input.connectionId,
input.snapshot,
siblingTargets,
);
const { shards } = buildLiveDatabaseManifestShards({
connectionType: input.driver.toUpperCase(),
tables: snapshotTablesToManifestData(input.snapshot, input.descriptionUpdates),
@ -285,6 +325,7 @@ export async function writeLocalScanManifestShards(
existingDescriptions: existing.descriptions,
existingPreservedJoins: existing.preservedJoins,
existingUsage: existing.usage,
federatedSiblingTargets: siblingTargets,
mapColumnType: (dimensionType) => dimensionType,
});

51
packages/cli/src/context/sl/local-query.ts
View file

@ -2,15 +2,23 @@ import type { KtxSqlQueryExecutorPort } from '../../context/connections/query-ex
import type { KtxSemanticLayerComputePort } from '../../context/daemon/semantic-layer-compute.js';
import type { KtxMcpProgressCallback } from '../mcp/types.js';
import type { KtxLocalProject } from '../../context/project/project.js';
import { FEDERATED_CONNECTION_ID } from '../connections/federation.js';
import { resolveRequiredConnectionId } from '../connections/resolve-connection.js';
import { sqlAnalysisDialectForDriver } from '../sql-analysis/dialect.js';
import { loadLocalSlSourceRecords } from './local-sl.js';
import { toResolvedWire } from './semantic-layer.service.js';
import { assertSafeConnectionId } from './source-files.js';
import type { SemanticLayerQueryExecutionResult, SemanticLayerQueryInput } from './types.js';
import type { SemanticLayerQueryExecutionResult, SemanticLayerQueryInput, SemanticLayerSource } from './types.js';
const COMPILE_ONLY_REASON =
'Local semantic-layer query compiled SQL but no data-source execution adapter is configured.';
const FEDERATED_SL_QUERY_UNSUPPORTED =
`Semantic-layer queries are per-connection and cannot target the federated connection '${FEDERATED_CONNECTION_ID}'. ` +
`Run a cross-database query as read-only SQL instead — ktx sql -c ${FEDERATED_CONNECTION_ID} "SELECT ..." or the sql_execution tool — ` +
'using catalog-qualified table names (connectionId.schema.table, or connectionId.table for sqlite; ' +
'double-quote ids that are not bare identifiers, e.g. "books-db".public.books).';
export interface CompileLocalSlQueryOptions {
connectionId?: string;
query: SemanticLayerQueryInput;
@ -27,23 +35,36 @@ export interface CompileLocalSlQueryResult extends SemanticLayerQueryExecutionRe
}
function resolveLocalConnectionId(project: KtxLocalProject, requested: string | undefined): string {
if (requested) {
return assertSafeConnectionId(requested);
return assertSafeConnectionId(resolveRequiredConnectionId(project.config, requested));
}
// The planner rejects a source set carrying a join whose `to` names a source
// outside that set, which would break every query for this connection. Keep only
// joins resolvable within the connection's own sources; a cross-database join
// (its `to` qualified by a sibling connection id) is just one such unresolvable
// target and runs as raw SQL instead. Membership is the test, not a connection-id
// prefix match, so a same-connection target whose name collides with a sibling
// connection id is preserved.
function withResolvableJoinsOnly(
source: SemanticLayerSource,
knownSourceNames: ReadonlySet<string>,
): SemanticLayerSource {
if (source.joins.length === 0) {
return source;
}
const ids = Object.keys(project.config.connections).sort();
if (ids.length === 1) {
return assertSafeConnectionId(ids[0]);
}
throw new Error('connectionId is required when the local project has zero or multiple connections.');
const joins = source.joins.filter((join) => knownSourceNames.has(join.to));
return joins.length === source.joins.length ? source : { ...source, joins };
}
async function loadComputableSources(
project: KtxLocalProject,
connectionId: string,
): Promise<ReturnType<typeof toResolvedWire>[]> {
return (await loadLocalSlSourceRecords(project, { connectionId: assertSafeConnectionId(connectionId) }))
.filter((record) => record.source.table || record.source.sql)
.map((record) => toResolvedWire(record.source));
const records = (await loadLocalSlSourceRecords(project, { connectionId })).filter(
(record) => record.source.table || record.source.sql,
);
const knownSourceNames = new Set(records.map((record) => record.source.name));
return records.map((record) => toResolvedWire(withResolvableJoinsOnly(record.source, knownSourceNames)));
}
function headersFromColumns(columns: Array<Record<string, unknown>>): string[] {
@ -56,9 +77,13 @@ export async function compileLocalSlQuery(
project: KtxLocalProject,
options: CompileLocalSlQueryOptions,
): Promise<CompileLocalSlQueryResult> {
if (options.connectionId === FEDERATED_CONNECTION_ID) {
throw new Error(FEDERATED_SL_QUERY_UNSUPPORTED);
}
await options.onProgress?.({ progress: 0, message: 'Compiling query' });
const connectionId = resolveLocalConnectionId(project, options.connectionId);
const dialect = sqlAnalysisDialectForDriver(project.config.connections[connectionId]?.driver);
const driver = project.config.connections[connectionId]?.driver;
const dialect = sqlAnalysisDialectForDriver(driver);
const sources = await loadComputableSources(project, connectionId);
await options.onProgress?.({ progress: 0.3, message: 'Generating SQL' });
@ -113,7 +138,7 @@ export async function compileLocalSlQuery(
...response.plan,
execution: {
mode: 'executed',
driver: project.config.connections[connectionId]?.driver ?? 'unknown',
driver: driver ?? 'unknown',
maxRows,
rowCount: execution.rowCount,
},

34
packages/cli/src/context/sl/local-sl.ts
View file

@ -2,6 +2,7 @@ import { join } from 'node:path';
import YAML from 'yaml';
import { z } from 'zod';
import type { KtxEmbeddingPort } from '../../context/core/embedding.js';
import { deriveFederatedConnection, FEDERATED_CONNECTION_ID } from '../connections/federation.js';
import type { KtxLocalProject } from '../../context/project/project.js';
import { HybridSearchCore } from '../../context/search/hybrid-search-core.js';
import type { SearchCandidateGenerator } from '../../context/search/types.js';
@ -169,7 +170,38 @@ export async function loadLocalSlSourceRecords(
project: KtxLocalProject,
input: { connectionId: string },
): Promise<LocalSlSourceRecord[]> {
const connectionId = assertSafeConnectionId(input.connectionId);
if (input.connectionId === FEDERATED_CONNECTION_ID) {
const descriptor = deriveFederatedConnection(project.config.connections, project.projectDir);
if (!descriptor) {
return [];
}
const perMember = await Promise.all(
descriptor.members.map(async (member) => {
const records = await loadSingleConnectionSourceRecords(project, member.connectionId);
return records.map((record) => {
// The federated view is one virtual connection: rows carry its id and a
// member-prefixed name, so a listing/search row round-trips to
// `ktx sl -c _ktx_federated read <name>`. Member origin lives in the name.
const name = `${member.connectionId}.${record.name}`;
return {
...record,
connectionId: FEDERATED_CONNECTION_ID,
name,
source: { ...record.source, name },
};
});
}),
);
return perMember.flat();
}
return loadSingleConnectionSourceRecords(project, input.connectionId);
}
async function loadSingleConnectionSourceRecords(
project: KtxLocalProject,
rawConnectionId: string,
): Promise<LocalSlSourceRecord[]> {
const connectionId = assertSafeConnectionId(rawConnectionId);
const dir = `semantic-layer/${connectionId}`;
const schemaDir = `${dir}/_schema`;
const listed = await project.fileStore.listFiles(dir);

8
packages/cli/src/context/sl/source-files.ts
View file

@ -23,7 +23,15 @@ function assertSafePathToken(kind: string, value: string): string {
return value;
}
/** @internal */
export function isReservedConnectionId(connectionId: string): boolean {
return connectionId.startsWith('_ktx_');
}
export function assertSafeConnectionId(connectionId: string): string {
if (isReservedConnectionId(connectionId)) {
throw new Error(`Connection id "${connectionId}" uses the reserved "_ktx_" prefix.`);
}
if (!isSafeConnectionId(connectionId)) {
throw new Error(`Unsafe connection id: ${connectionId}`);
}

39
packages/cli/src/ingest-query-executor.ts
View file

@ -1,16 +1,14 @@
import { executeFederatedQuery } from './connectors/duckdb/federated-executor.js';
import type { KtxSqlQueryExecutionInput, KtxSqlQueryExecutorPort } from './context/connections/query-executor.js';
import { executeProjectReadOnlySql } from './context/connections/project-sql-executor.js';
import type { KtxLocalProject } from './context/project/project.js';
import type { KtxScanConnector, KtxScanContext } from './context/scan/types.js';
import { createKtxCliScanConnector } from './local-scan-connectors.js';
type CreateConnector = typeof createKtxCliScanConnector;
export interface KtxCliIngestQueryExecutorDeps {
createConnector?: CreateConnector;
}
async function cleanupConnector(connector: KtxScanConnector | null): Promise<void> {
await connector?.cleanup?.();
executeFederated?: typeof executeFederatedQuery;
}
export function createKtxCliIngestQueryExecutor(
@ -20,30 +18,13 @@ export function createKtxCliIngestQueryExecutor(
const createConnector = deps.createConnector ?? createKtxCliScanConnector;
return {
async execute(input: KtxSqlQueryExecutionInput) {
let connector: KtxScanConnector | null = null;
try {
connector = await createConnector(project, input.connectionId);
if (!connector.capabilities.readOnlySql || !connector.executeReadOnly) {
throw new Error(
`Connection "${input.connectionId}" driver "${connector.driver}" does not support read-only SQL execution.`,
);
}
const ctx: KtxScanContext = { runId: 'ingest-sql-execution' };
const result = await connector.executeReadOnly(
{ connectionId: input.connectionId, sql: input.sql, maxRows: input.maxRows },
ctx,
);
return {
headers: result.headers,
rows: result.rows,
totalRows: result.totalRows,
command: 'SELECT',
rowCount: result.rowCount,
};
} finally {
await cleanupConnector(connector);
}
return executeProjectReadOnlySql({
project,
input,
createConnector: (connectionId) => createConnector(project, connectionId),
executeFederated: deps.executeFederated,
runId: 'ingest-sql-execution',
});
},
};
}

80
packages/cli/src/managed-python-command.ts
View file

@ -62,7 +62,7 @@ export function managedRuntimeInstallCommand(feature: KtxRuntimeFeature): string
function installPrompt(feature: KtxRuntimeFeature): string {
const label = feature === 'local-embeddings' ? 'local embeddings Python runtime' : 'core Python runtime';
return `ktx needs to install the ${label}. This downloads Python dependencies with uv. Continue?`;
return `ktx needs to install the ${label}. This downloads a pinned, checksum-verified uv build, Python, and dependencies. Continue?`;
}
function runtimeRequiredMessage(feature: KtxRuntimeFeature): string {
@ -121,43 +121,55 @@ export async function ensureManagedPythonCommandRuntime(
}
}
export function createManagedPythonSemanticLayerComputePort(
export async function createManagedPythonSemanticLayerComputePort(
options: ManagedPythonSemanticLayerComputeOptions,
): Promise<KtxSemanticLayerComputePort> {
const runtime = await ensureManagedPythonCommandRuntime({
cliVersion: options.cliVersion,
installPolicy: options.installPolicy,
io: options.io,
feature: 'core',
...(options.readStatus ? { readStatus: options.readStatus } : {}),
...(options.installRuntime ? { installRuntime: options.installRuntime } : {}),
...(options.confirmInstall ? { confirmInstall: options.confirmInstall } : {}),
...(options.spinner ? { spinner: options.spinner } : {}),
});
const createPythonCompute = options.createPythonCompute ?? createPythonSemanticLayerComputePort;
const projectId = options.projectDir
? await readExistingTelemetryProjectId({ projectDir: options.projectDir })
: undefined;
return createPythonCompute({
command: runtime.manifest.python.daemonExecutable,
args: [],
...(projectId ? { projectId } : {}),
});
}
/**
* Defers the managed-runtime install to the first semantic-layer call so a
* long-lived server (the MCP server) can start and serve context tools that
* need no Python even when uv is absent. Caches on success only, so a runtime
* installed mid-session is picked up on the next call.
*/
export function createLazyManagedPythonSemanticLayerComputePort(
options: ManagedPythonSemanticLayerComputeOptions,
): KtxSemanticLayerComputePort {
const createPythonCompute = options.createPythonCompute ?? createPythonSemanticLayerComputePort;
let cachedPort: KtxSemanticLayerComputePort | undefined;
// Runtime install is deferred to the first compute call so an MCP session
// that only searches never installs the Python runtime at boot. Cached on
// success, so a failed install retries on the next call.
const resolvePort = async (): Promise<KtxSemanticLayerComputePort> => {
if (cachedPort) {
return cachedPort;
let cached: KtxSemanticLayerComputePort | undefined;
const resolve = async (): Promise<KtxSemanticLayerComputePort> => {
if (!cached) {
cached = await createManagedPythonSemanticLayerComputePort(options);
}
const runtime = await ensureManagedPythonCommandRuntime({
cliVersion: options.cliVersion,
installPolicy: options.installPolicy,
io: options.io,
feature: 'core',
...(options.readStatus ? { readStatus: options.readStatus } : {}),
...(options.installRuntime ? { installRuntime: options.installRuntime } : {}),
...(options.confirmInstall ? { confirmInstall: options.confirmInstall } : {}),
...(options.spinner ? { spinner: options.spinner } : {}),
});
const projectId = options.projectDir
? await readExistingTelemetryProjectId({ projectDir: options.projectDir })
: undefined;
cachedPort = createPythonCompute({
command: runtime.manifest.python.daemonExecutable,
args: [],
...(projectId ? { projectId } : {}),
});
return cachedPort;
return cached;
};
return {
query: async (input) => (await resolvePort()).query(input),
validateSources: async (input) => (await resolvePort()).validateSources(input),
generateSources: async (input) => (await resolvePort()).generateSources(input),
async query(input) {
return (await resolve()).query(input);
},
async validateSources(input) {
return (await resolve()).validateSources(input);
},
async generateSources(input) {
return (await resolve()).generateSources(input);
},
};
}

199
packages/cli/src/managed-python-runtime.ts
View file

@ -1,12 +1,19 @@
import { execFile } from 'node:child_process';
import { createHash } from 'node:crypto';
import { access, appendFile, mkdir, readFile, rm, writeFile } from 'node:fs/promises';
import { access, appendFile, chmod, mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises';
import { homedir } from 'node:os';
import { basename, join } from 'node:path';
import { basename, dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { promisify } from 'node:util';
import { strFromU8, unzipSync } from 'fflate';
import { gunzipSync, strFromU8, unzipSync } from 'fflate';
import { z } from 'zod';
import { KtxExpectedError } from './errors.js';
import {
MANAGED_UV_ARTIFACTS,
MANAGED_UV_VERSION,
type ManagedUvArtifact,
type ManagedUvPlatformKey,
} from './managed-uv-release.js';
const execFileAsync = promisify(execFile);
@ -96,6 +103,7 @@ export interface ManagedPythonRuntimeInstallOptions extends ManagedPythonRuntime
features: KtxRuntimeFeature[];
force?: boolean;
exec?: ManagedPythonRuntimeExec;
fetchUvArtifact?: ManagedUvFetchArtifact;
}
export interface ManagedPythonRuntimeInstallResult {
@ -122,9 +130,29 @@ export interface ManagedPythonRuntimeDoctorCheck {
fix?: string;
}
export type ManagedUvFetchArtifact = (url: string) => Promise<Uint8Array>;
/** @internal */
export const MISSING_UV_RUNTIME_INSTALL_MESSAGE =
'uv is required to install the ktx Python runtime. ktx does not download uv automatically. Install uv, make sure it is on PATH, and retry: ktx admin runtime install --yes';
export interface ManagedUvRelease {
version: string;
artifacts: Partial<Record<ManagedUvPlatformKey, ManagedUvArtifact>>;
}
const PINNED_UV_RELEASE: ManagedUvRelease = {
version: MANAGED_UV_VERSION,
artifacts: MANAGED_UV_ARTIFACTS,
};
/** @internal */
export interface EnsureManagedUvOptions {
platform?: NodeJS.Platform;
arch?: string;
env?: NodeJS.ProcessEnv;
homeDir?: string;
runtimeRoot?: string;
fetchArtifact?: ManagedUvFetchArtifact;
release?: ManagedUvRelease;
}
function defaultAssetDir(): string {
return fileURLToPath(new URL('../assets/python/', import.meta.url));
@ -347,12 +375,145 @@ function managedRuntimeUvEnv(baseEnv: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
return { ...baseEnv, UV_NO_CONFIG: '1' };
}
async function ensureUv(exec: ManagedPythonRuntimeExec, env?: NodeJS.ProcessEnv): Promise<string> {
function managedUvBinaryName(platform: NodeJS.Platform): string {
return platform === 'win32' ? 'uv.exe' : 'uv';
}
/** @internal */
export function managedUvPath(options: EnsureManagedUvOptions = {}): string {
const platform = options.platform ?? process.platform;
const env = options.env ?? process.env;
const homeDir = options.homeDir ?? homedir();
const runtimeRoot = options.runtimeRoot ?? runtimeRootFor({ env, homeDir });
const version = (options.release ?? PINNED_UV_RELEASE).version;
return join(runtimeRoot, 'uv', version, managedUvBinaryName(platform));
}
async function defaultFetchUvArtifact(url: string): Promise<Uint8Array> {
const response = await fetch(url);
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
return new Uint8Array(await response.arrayBuffer());
}
function readTarField(block: Uint8Array, start: number, length: number): string {
const field = block.subarray(start, start + length);
const end = field.indexOf(0);
return strFromU8(end < 0 ? field : field.subarray(0, end));
}
function findTarEntry(archive: Uint8Array, matches: (name: string) => boolean): Uint8Array | undefined {
let offset = 0;
while (offset + 512 <= archive.length) {
const block = archive.subarray(offset, offset + 512);
const name = readTarField(block, 0, 100);
if (!name) {
return undefined;
}
const size = Number.parseInt(readTarField(block, 124, 12).trim() || '0', 8);
if (matches(name)) {
return archive.subarray(offset + 512, offset + 512 + size);
}
offset += 512 + Math.ceil(size / 512) * 512;
}
return undefined;
}
function extractUvFromArchive(input: { file: string; contents: Uint8Array; binaryName: string }): Uint8Array {
const entry = input.file.endsWith('.zip')
? unzipSync(input.contents)[input.binaryName]
: findTarEntry(gunzipSync(input.contents), (name) => name === input.binaryName || name.endsWith(`/${input.binaryName}`));
if (!entry) {
throw new Error(`uv archive ${input.file} is missing the ${input.binaryName} binary`);
}
return entry;
}
/**
* ktx provisions its own pinned uv under the runtime root; uv on PATH is never
* consulted, so runtime installs behave identically on every machine. All
* failures here are environment outcomes (offline host, intercepting proxy,
* unsupported platform) and stay out of Error Tracking via KtxExpectedError —
* except a pin/layout mismatch inside a checksum-verified archive, which is a
* ktx release fault and must reach Error Tracking.
* @internal
*/
export async function ensureManagedUv(options: EnsureManagedUvOptions = {}): Promise<string> {
const platform = options.platform ?? process.platform;
const arch = options.arch ?? process.arch;
const release = options.release ?? PINNED_UV_RELEASE;
const binaryName = managedUvBinaryName(platform);
const uvPath = managedUvPath(options);
if (await pathExists(uvPath)) {
return uvPath;
}
const artifact = release.artifacts[`${platform}-${arch}` as ManagedUvPlatformKey];
if (!artifact) {
throw new KtxExpectedError(
`ktx does not bundle uv for ${platform}-${arch}. Place a uv ${release.version} binary at ${uvPath} and retry: ktx admin runtime install --yes`,
);
}
const url = `https://github.com/astral-sh/uv/releases/download/${release.version}/${artifact.file}`;
let contents: Uint8Array;
try {
const result = await exec('uv', ['--version'], { env });
return result.stdout.trim() || 'uv available';
} catch {
throw new Error(MISSING_UV_RUNTIME_INSTALL_MESSAGE);
contents = await (options.fetchArtifact ?? defaultFetchUvArtifact)(url);
} catch (error) {
throw new KtxExpectedError(
`ktx could not download uv ${release.version} (required to install the ktx Python runtime). ` +
'Check network access to github.com and retry: ktx admin runtime install --yes. ' +
`Air-gapped hosts: place the uv binary at ${uvPath}.`,
{ cause: error },
);
}
const sha256 = createHash('sha256').update(contents).digest('hex');
if (sha256 !== artifact.sha256) {
throw new KtxExpectedError(
`Downloaded uv ${release.version} failed checksum verification (a proxy or captive portal may have altered the download). Retry: ktx admin runtime install --yes`,
);
}
const binary = extractUvFromArchive({ file: artifact.file, contents, binaryName });
await mkdir(dirname(uvPath), { recursive: true });
const stagedPath = `${uvPath}.${process.pid}.download`;
await writeFile(stagedPath, binary);
await chmod(stagedPath, 0o755);
try {
await rename(stagedPath, uvPath);
} catch (error) {
// On Windows a concurrent install may have won the rename; the binary at
// uvPath is checksum-pinned identical, so reuse it.
await rm(stagedPath, { force: true });
if (!(await pathExists(uvPath))) {
throw error;
}
}
return uvPath;
}
async function ensureUv(input: {
exec: ManagedPythonRuntimeExec;
uvEnv: NodeJS.ProcessEnv;
options: ManagedPythonRuntimeLayoutOptions & { fetchUvArtifact?: ManagedUvFetchArtifact };
}): Promise<{ uvPath: string; version: string }> {
const uvPath = await ensureManagedUv({
platform: input.options.platform,
env: input.options.env,
homeDir: input.options.homeDir,
runtimeRoot: input.options.runtimeRoot,
fetchArtifact: input.options.fetchUvArtifact,
});
try {
const result = await input.exec(uvPath, ['--version'], { env: input.uvEnv });
return { uvPath, version: result.stdout.trim() || `uv ${MANAGED_UV_VERSION}` };
} catch (error) {
throw new KtxExpectedError(
`Managed uv at ${uvPath} failed to run. Delete it and retry: ktx admin runtime install --yes`,
{ cause: error },
);
}
}
@ -377,21 +538,23 @@ export async function installManagedPythonRuntime(
return { status: 'ready', layout, asset, manifest: existing };
}
// uv is acquired before the version dir is wiped, so a failed acquisition
// never destroys a previously installed runtime.
const { uvPath } = await ensureUv({ exec, uvEnv, options });
await rm(layout.versionDir, { recursive: true, force: true });
await mkdir(layout.versionDir, { recursive: true });
await writeFile(layout.installLogPath, '');
await ensureUv(exec, uvEnv);
await runLogged({
exec,
logPath: layout.installLogPath,
command: 'uv',
command: uvPath,
args: ['python', 'install', asset.requiresPython.minimumVersion],
env: uvEnv,
});
await runLogged({
exec,
logPath: layout.installLogPath,
command: 'uv',
command: uvPath,
args: ['venv', '--python', asset.requiresPython.minimumVersion, layout.venvDir],
env: uvEnv,
});
@ -399,7 +562,7 @@ export async function installManagedPythonRuntime(
await runLogged({
exec,
logPath: layout.installLogPath,
command: 'uv',
command: uvPath,
args: ['pip', 'install', '--python', layout.pythonPath, wheelSpec],
env: uvEnv,
});
@ -462,20 +625,20 @@ function check(
}
export async function doctorManagedPythonRuntime(
options: ManagedPythonRuntimeLayoutOptions & { exec?: ManagedPythonRuntimeExec },
options: ManagedPythonRuntimeLayoutOptions & { exec?: ManagedPythonRuntimeExec; fetchUvArtifact?: ManagedUvFetchArtifact },
): Promise<ManagedPythonRuntimeDoctorCheck[]> {
const exec = options.exec ?? defaultExec;
const checks: ManagedPythonRuntimeDoctorCheck[] = [];
try {
const version = await ensureUv(exec, managedRuntimeUvEnv(options.env ?? process.env));
checks.push(check('pass', { id: 'uv', label: 'uv', detail: version }));
const uv = await ensureUv({ exec, uvEnv: managedRuntimeUvEnv(options.env ?? process.env), options });
checks.push(check('pass', { id: 'uv', label: 'uv', detail: `${uv.version} (managed: ${uv.uvPath})` }));
} catch (error) {
checks.push(
check('fail', {
id: 'uv',
label: 'uv',
detail: error instanceof Error ? error.message : String(error),
fix: 'Install uv, make sure it is on PATH, and run: ktx admin runtime install --yes',
fix: 'Check network access to github.com and run: ktx admin runtime install --yes',
}),
);
}

26
packages/cli/src/managed-uv-release.ts Normal file
View file

@ -0,0 +1,26 @@
// Generated by scripts/refresh-uv-manifest.mjs. Do not edit by hand.
// Regenerate with: node scripts/refresh-uv-manifest.mjs [<uv-version>]
export type ManagedUvPlatformKey =
| 'darwin-arm64'
| 'darwin-x64'
| 'linux-arm64'
| 'linux-x64'
| 'win32-arm64'
| 'win32-x64';
export interface ManagedUvArtifact {
file: string;
sha256: string;
}
export const MANAGED_UV_VERSION = '0.11.21';
export const MANAGED_UV_ARTIFACTS: Record<ManagedUvPlatformKey, ManagedUvArtifact> = {
'darwin-arm64': { file: 'uv-aarch64-apple-darwin.tar.gz', sha256: '1f921d491ba5ffeea774eb04d6681ecee379101341cbb1500394993b541bf3f4' }, // pragma: allowlist secret
'darwin-x64': { file: 'uv-x86_64-apple-darwin.tar.gz', sha256: 'f3c8e5708a84b920c18b691214d54d2b0da6b984789caae95d47c95120cb7765' }, // pragma: allowlist secret
'linux-arm64': { file: 'uv-aarch64-unknown-linux-musl.tar.gz', sha256: 'e71badaed2a2c3a404a0a00974b51c7ed5f5bc7be947916846005b739c68a5a2' }, // pragma: allowlist secret
'linux-x64': { file: 'uv-x86_64-unknown-linux-musl.tar.gz', sha256: '9dadff5b9e7b1d2d011e41852a1cbca713d9d5d88194f2eb6bd240fa4fb0a719' }, // pragma: allowlist secret
'win32-arm64': { file: 'uv-aarch64-pc-windows-msvc.zip', sha256: '74e443f8004022dde57a1bd0d10c097830f9ea8feb4ec927db52cd5d805c2f48' }, // pragma: allowlist secret
'win32-x64': { file: 'uv-x86_64-pc-windows-msvc.zip', sha256: 'ace861f360c6de2babedc1607d0f454b6b09a820dbc8182dc15af927e4df9589' }, // pragma: allowlist secret
};

4
packages/cli/src/mcp-server-factory.ts
View file

@ -8,7 +8,7 @@ import type { KtxCliIo } from './cli-runtime.js';
import { resolveProjectEmbeddingProvider } from './embedding-resolution.js';
import { createKtxCliIngestQueryExecutor } from './ingest-query-executor.js';
import { createKtxCliScanConnector } from './local-scan-connectors.js';
import { createManagedPythonSemanticLayerComputePort } from './managed-python-command.js';
import { createLazyManagedPythonSemanticLayerComputePort } from './managed-python-command.js';
import { resolveSqlAnalysisPort } from './managed-python-http.js';
function noopMcpIo(): KtxCliIo {
@ -26,7 +26,7 @@ export async function createKtxMcpServerFactory(input: {
}): Promise<() => McpServer> {
const io = input.io ?? noopMcpIo();
const queryExecutor = createKtxCliIngestQueryExecutor(input.project);
const semanticLayerCompute = createManagedPythonSemanticLayerComputePort({
const semanticLayerCompute = createLazyManagedPythonSemanticLayerComputePort({
cliVersion: input.cliVersion,
installPolicy: 'auto',
io,

206
packages/cli/src/setup-agents.ts
View file

@ -1,5 +1,5 @@
import { existsSync } from 'node:fs';
import { mkdir, readFile, rm, writeFile } from 'node:fs/promises';
import { mkdir, readFile, rm, stat, writeFile } from 'node:fs/promises';
import { dirname, join, relative, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { styleText } from 'node:util';
@ -35,13 +35,26 @@ export interface KtxSetupAgentsArgs {
mode: KtxAgentInstallMode;
skipAgents: boolean;
showNextActions?: boolean;
installRoot?: string;
cwd?: string;
}
/** The directory project-scoped agent files land in; equals projectDir unless an install root is chosen. */
interface KtxAgentInstall {
target: KtxAgentTarget;
scope: KtxAgentScope;
mode: KtxAgentInstallMode;
installRoot: string;
}
/** Install shape for formatting helpers; installRoot falls back to projectDir when absent. */
type KtxAgentInstallLike = Omit<KtxAgentInstall, 'installRoot'> & { installRoot?: string };
export type KtxSetupAgentsResult =
| {
status: 'ready';
projectDir: string;
installs: Array<{ target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode }>;
installs: KtxAgentInstall[];
nextActions?: string;
}
| { status: 'skipped'; projectDir: string }
@ -53,7 +66,7 @@ export interface KtxAgentInstallManifest {
version: 1;
projectDir: string;
installedAt: string;
installs: Array<{ target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode }>;
installs: KtxAgentInstall[];
entries: Array<
| {
kind: 'file';
@ -258,7 +271,11 @@ function universalMcpSnippet(endpoint: KtxMcpEndpointInfo): string {
].join('\n');
}
function claudeConfigPath(projectDir: string, scope: KtxAgentScope): { path: string; jsonPath: string[] } {
function claudeConfigPath(
projectDir: string,
installRoot: string,
scope: KtxAgentScope,
): { path: string; jsonPath: string[] } {
const home = process.env.HOME ?? '';
if (scope === 'global') {
return { path: join(home, '.claude.json'), jsonPath: ['mcpServers', 'ktx'] };
@ -266,13 +283,13 @@ function claudeConfigPath(projectDir: string, scope: KtxAgentScope): { path: str
if (scope === 'local') {
return { path: join(home, '.claude.json'), jsonPath: ['projects', resolve(projectDir), 'mcpServers', 'ktx'] };
}
return { path: join(resolve(projectDir), '.mcp.json'), jsonPath: ['mcpServers', 'ktx'] };
return { path: join(resolve(installRoot), '.mcp.json'), jsonPath: ['mcpServers', 'ktx'] };
}
function cursorConfigPath(projectDir: string, scope: KtxAgentScope): { path: string; jsonPath: string[] } {
function cursorConfigPath(installRoot: string, scope: KtxAgentScope): { path: string; jsonPath: string[] } {
const home = process.env.HOME ?? '';
return {
path: scope === 'global' ? join(home, '.cursor/mcp.json') : join(resolve(projectDir), '.cursor/mcp.json'),
path: scope === 'global' ? join(home, '.cursor/mcp.json') : join(resolve(installRoot), '.cursor/mcp.json'),
jsonPath: ['mcpServers', 'ktx'],
};
}
@ -311,6 +328,7 @@ function claudeDesktopMcpEntry(input: { projectDir: string; env?: NodeJS.Process
async function installMcpClientConfig(input: {
projectDir: string;
installRoot: string;
target: KtxAgentTarget;
scope: KtxAgentScope;
}): Promise<KtxMcpClientInstallResult> {
@ -335,11 +353,11 @@ async function installMcpClientConfig(input: {
}
if (input.target === 'claude-code') {
const config = claudeConfigPath(input.projectDir, input.scope);
const config = claudeConfigPath(input.projectDir, input.installRoot, input.scope);
await writeJsonKey(config.path, config.jsonPath, claudeMcpEntry(endpoint));
entries.push({ kind: 'json-key', path: config.path, jsonPath: config.jsonPath });
} else if (input.target === 'cursor') {
const config = cursorConfigPath(input.projectDir, input.scope);
const config = cursorConfigPath(input.installRoot, input.scope);
await writeJsonKey(config.path, config.jsonPath, cursorMcpEntry(endpoint));
entries.push({ kind: 'json-key', path: config.path, jsonPath: config.jsonPath });
} else if (input.target === 'codex') {
@ -348,7 +366,7 @@ async function installMcpClientConfig(input: {
const path =
input.scope === 'global'
? '~/.config/opencode/opencode.json'
: relative(input.projectDir, join(input.projectDir, 'opencode.json'));
: relative(input.installRoot, join(input.installRoot, 'opencode.json'));
snippets.push(`Add this OpenCode MCP snippet to ${path}:\n${opencodeSnippet(endpoint)}`);
} else if (input.target === 'universal') {
snippets.push(`Use this universal MCP endpoint with unsupported MCP clients:\n${universalMcpSnippet(endpoint)}`);
@ -359,11 +377,12 @@ async function installMcpClientConfig(input: {
function plannedMcpJsonEntries(input: {
projectDir: string;
installRoot: string;
target: KtxAgentTarget;
scope: KtxAgentScope;
}): InstallEntry[] {
if (input.target === 'claude-code') {
const config = claudeConfigPath(input.projectDir, input.scope);
const config = claudeConfigPath(input.projectDir, input.installRoot, input.scope);
return [{ kind: 'json-key', path: config.path, jsonPath: config.jsonPath }];
}
if (input.target === 'claude-desktop') {
@ -371,7 +390,7 @@ function plannedMcpJsonEntries(input: {
return [{ kind: 'json-key', path: config.path, jsonPath: config.jsonPath }];
}
if (input.target === 'cursor') {
const config = cursorConfigPath(input.projectDir, input.scope);
const config = cursorConfigPath(input.installRoot, input.scope);
return [{ kind: 'json-key', path: config.path, jsonPath: config.jsonPath }];
}
return [];
@ -395,6 +414,7 @@ export function plannedKtxAgentFiles(input: {
target: KtxAgentTarget;
scope: KtxAgentScope;
mode: KtxAgentInstallMode;
installRoot?: string;
}): InstallEntry[] {
const withAdminCli = input.mode === 'mcp-cli';
@ -447,7 +467,7 @@ export function plannedKtxAgentFiles(input: {
throw new Error(`Global ${input.target} installation is not supported; omit --global.`);
}
const root = resolve(input.projectDir);
const root = resolve(input.installRoot ?? input.projectDir);
const analyticsEntries: Partial<Record<KtxAgentTarget, InstallEntry[]>> = {
'claude-code': [
{ kind: 'file', path: join(root, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' },
@ -650,7 +670,8 @@ function mergeManifest(
): KtxAgentInstallManifest {
const installMap = new Map<string, KtxAgentInstallManifest['installs'][number]>();
for (const install of [...(existing?.installs ?? []), ...installs]) {
installMap.set(`${install.target}:${install.scope}:${install.mode}`, install);
const installRoot = install.installRoot ?? resolve(projectDir);
installMap.set(`${install.target}:${install.scope}:${install.mode}:${installRoot}`, { ...install, installRoot });
}
const entryMap = new Map<string, InstallEntry>();
for (const entry of [...(existing?.entries ?? []), ...entries]) {
@ -688,6 +709,7 @@ interface KtxSetupAgentsPromptAdapter {
options: KtxSetupPromptOption[];
required?: boolean;
}): Promise<string[]>;
text(options: { message: string; placeholder?: string }): Promise<string | undefined>;
cancel(message: string): void;
}
@ -786,16 +808,29 @@ function formatInlinePath(path: string): string {
return path;
}
function installSummaryTitle(install: KtxAgentInstallLike, projectDir: string): string {
const name = targetDisplayName(install.target);
if (install.scope !== 'project') {
return `${name} · ${scopeDisplayName(install.scope)}`;
}
const installRoot = resolve(install.installRoot ?? projectDir);
if (installRoot === resolve(projectDir)) {
return `${name} · ${scopeDisplayName('project')}`;
}
return `${name} · ${formatInlinePath(installRoot)}`;
}
/** @internal */
export function formatInstallSummaryLines(
installs: Array<{ target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode }>,
installs: KtxAgentInstallLike[],
entries: InstallEntry[],
projectDir: string,
): InstallSummaryEntry[] {
const entriesByTarget = new Map<KtxAgentTarget, InstallEntry[]>();
for (const install of installs) {
const installRoot = install.installRoot ?? projectDir;
const plannedFilePaths = new Set(
plannedKtxAgentFiles({ projectDir, ...install })
plannedKtxAgentFiles({ projectDir, ...install, installRoot })
.filter((entry) => entry.kind === 'file')
.map((entry) => entry.path),
);
@ -806,7 +841,10 @@ export function formatInstallSummaryLines(
}
const mcpEntriesByTarget = new Map<KtxAgentTarget, InstallEntry[]>();
for (const install of installs) {
const plannedMcpKeys = new Set(plannedMcpJsonEntries({ projectDir, ...install }).map(entryKey));
const installRoot = install.installRoot ?? projectDir;
const plannedMcpKeys = new Set(
plannedMcpJsonEntries({ projectDir, installRoot, target: install.target, scope: install.scope }).map(entryKey),
);
mcpEntriesByTarget.set(
install.target,
entries.filter((entry) => entry.kind === 'json-key' && plannedMcpKeys.has(entryKey(entry))),
@ -856,7 +894,7 @@ export function formatInstallSummaryLines(
}
return {
title: `${targetDisplayName(install.target)} · ${scopeDisplayName(install.scope)}`,
title: installSummaryTitle(install, projectDir),
lines,
};
});
@ -864,7 +902,7 @@ export function formatInstallSummaryLines(
function claudeDesktopSkillBundlePathsForInstalls(
projectDir: string,
installs: Array<{ target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode }>,
installs: KtxAgentInstallLike[],
): string[] {
return installs
.filter((install) => install.target === 'claude-desktop')
@ -939,9 +977,13 @@ function manualActionFromSnippet(snippet: string): {
};
}
function openFromDirectoryLabel(installRoot: string, projectDir: string): string {
return resolve(installRoot) === resolve(projectDir) ? 'the ktx project directory' : 'the install directory';
}
function formatAgentNextActions(input: {
projectDir: string;
installs: Array<{ target: KtxAgentTarget; scope: KtxAgentScope; mode: KtxAgentInstallMode }>;
installs: KtxAgentInstallLike[];
notices: string[];
snippets: string[];
}): string {
@ -983,10 +1025,11 @@ function formatAgentNextActions(input: {
if (claudeCodeInstall) {
lines.push(`${step}. Open Claude Code`);
if (claudeCodeInstall.scope === 'project') {
lines.push(' Open Claude Code from the ktx project directory:');
const installRoot = resolve(claudeCodeInstall.installRoot ?? projectDir);
lines.push(` Open Claude Code from ${openFromDirectoryLabel(installRoot, projectDir)}:`);
lines.push('');
lines.push(' RUN:');
lines.push(` cd ${shellScriptQuote(projectDir)}`);
lines.push(` cd ${shellScriptQuote(installRoot)}`);
lines.push(' claude');
} else {
lines.push(' RUN:');
@ -1000,10 +1043,11 @@ function formatAgentNextActions(input: {
if (cursorInstall) {
lines.push(`${step}. Open Cursor`);
if (cursorInstall.scope === 'project') {
lines.push(' Open Cursor from the ktx project directory:');
const installRoot = resolve(cursorInstall.installRoot ?? projectDir);
lines.push(` Open Cursor from ${openFromDirectoryLabel(installRoot, projectDir)}:`);
lines.push('');
lines.push(' OPEN:');
lines.push(` ${projectDir}`);
lines.push(` ${installRoot}`);
} else {
lines.push(' Open Cursor.');
}
@ -1041,6 +1085,7 @@ function formatAgentNextActions(input: {
async function installTarget(input: {
projectDir: string;
installRoot: string;
target: KtxAgentTarget;
scope: KtxAgentScope;
mode: KtxAgentInstallMode;
@ -1076,6 +1121,72 @@ async function markAgentsComplete(projectDir: string): Promise<void> {
await markKtxSetupStateStepComplete(projectDir, 'agents');
}
// A typed path never passes through a shell, so expand a leading ~ here; HOME
// matches formatInlinePath so the ~/… hints shown in the menu round-trip.
function resolveTypedInstallDir(cwd: string, raw: string): string {
const home = process.env.HOME;
if (home && (raw === '~' || raw.startsWith('~/'))) {
return resolve(home, raw.slice(2));
}
return resolve(cwd, raw);
}
async function ensureInstallDir(resolvedPath: string): Promise<string> {
if (existsSync(resolvedPath)) {
if (!(await stat(resolvedPath)).isDirectory()) {
throw new Error(`Install directory path is a file, not a directory: ${resolvedPath}`);
}
return resolvedPath;
}
await mkdir(resolvedPath, { recursive: true });
return resolvedPath;
}
async function promptInstallDirectory(input: {
prompts: KtxSetupAgentsPromptAdapter;
io: KtxCliIo;
cwd: string;
projectRoot: string;
scopeTargets: KtxAgentTarget[];
}): Promise<{ scope: KtxAgentScope; installRoot: string } | 'back'> {
const { prompts, io, cwd, projectRoot, scopeTargets } = input;
const options: KtxSetupPromptOption[] = [
{ value: 'project', label: 'ktx project directory', hint: formatInlinePath(projectRoot) },
...(cwd !== projectRoot
? [{ value: 'current', label: 'Current directory', hint: formatInlinePath(cwd) }]
: []),
{ value: 'custom', label: 'Custom directory…', hint: 'Enter a path' },
...(scopeTargets.every(targetSupportsGlobalScope)
? [
{
value: 'global',
label: 'Global scope (user config)',
hint: 'Agents can load this ktx project from any working directory.',
},
]
: []),
];
const choice = await prompts.select({
message: `Where should ktx install agent config?\n\nktx project: ${projectRoot}`,
options,
});
if (choice === 'back') return 'back';
if (choice === 'global') return { scope: 'global', installRoot: projectRoot };
if (choice === 'current') return { scope: 'project', installRoot: cwd };
if (choice === 'project') return { scope: 'project', installRoot: projectRoot };
while (true) {
const typed = await prompts.text({ message: 'Enter the directory to install agent config into' });
if (typed === undefined) return 'back';
const trimmed = typed.trim();
if (trimmed === '') continue;
try {
return { scope: 'project', installRoot: await ensureInstallDir(resolveTypedInstallDir(cwd, trimmed)) };
} catch (error) {
io.stderr.write(`${errorMessage(error)}\n`);
}
}
}
export async function runKtxSetupAgentsStep(
args: KtxSetupAgentsArgs,
io: KtxCliIo,
@ -1146,31 +1257,31 @@ export async function runKtxSetupAgentsStep(
return { status: 'missing-input', projectDir: args.projectDir };
}
const cwd = resolve(args.cwd ?? process.cwd());
const projectRoot = resolve(args.projectDir);
const scopeTargets = targets.filter((target) => target !== 'claude-desktop');
const selectedScope =
args.inputMode !== 'disabled' &&
args.scope === 'project' &&
scopeTargets.length > 0 &&
scopeTargets.every(targetSupportsGlobalScope)
? ((await prompts.select({
message: `Where should ktx install supported agent config?\n\nktx project: ${resolve(args.projectDir)}`,
options: [
{
value: 'project',
label: 'Project scope (ktx project directory)',
hint: 'Only agents opened from this ktx project path load the project-scoped config.',
},
{
value: 'global',
label: 'Global scope (user config)',
hint: 'Agents can load this ktx project from any working directory.',
},
],
})) as KtxAgentScope | 'back')
: args.scope;
if (selectedScope === 'back') return { status: 'back', projectDir: args.projectDir };
const installs = targets.map((target) => ({ target, scope: effectiveInstallScope(target, selectedScope), mode }));
let selectedScope: KtxAgentScope = args.scope;
let installRoot = projectRoot;
if (args.installRoot !== undefined) {
try {
installRoot = await ensureInstallDir(resolveTypedInstallDir(cwd, args.installRoot));
} catch (error) {
writePrefixedLines((chunk) => io.stderr.write(chunk), errorMessage(error));
return { status: 'failed', projectDir: args.projectDir };
}
selectedScope = 'project';
} else if (args.inputMode !== 'disabled' && args.scope === 'project' && scopeTargets.length > 0) {
const decision = await promptInstallDirectory({ prompts, io, cwd, projectRoot, scopeTargets });
if (decision === 'back') return { status: 'back', projectDir: args.projectDir };
selectedScope = decision.scope;
installRoot = decision.installRoot;
}
const installs: KtxAgentInstall[] = targets.map((target) => {
const scope = effectiveInstallScope(target, selectedScope);
return { target, scope, mode, installRoot: scope === 'project' ? installRoot : projectRoot };
});
const entries: InstallEntry[] = [];
const snippets: string[] = [];
const notices = new Set<string>();
@ -1180,6 +1291,7 @@ export async function runKtxSetupAgentsStep(
entries.push(...targetEntries);
const mcpResult = await installMcpClientConfig({
projectDir: args.projectDir,
installRoot: install.installRoot,
target: install.target,
scope: install.scope,
});

21
packages/cli/src/setup-databases.ts
View file

@ -3,6 +3,7 @@ import { readFile, writeFile } from 'node:fs/promises';
import { delimiter, dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { promisify } from 'node:util';
import { deriveFederatedConnection, FEDERATED_CONNECTION_ID } from './context/connections/federation.js';
import { getDriverRegistration } from './context/connections/drivers.js';
import { createLocalKtxLlmRuntimeFromConfig } from './context/llm/local-config.js';
import type { KtxLlmRuntimePort } from './context/llm/runtime-port.js';
@ -1171,6 +1172,26 @@ async function writeConnectionConfig(input: {
if (queryHistory?.enabled === true) {
await ensureHistoricSqlIngestDefaults(input.projectDir);
}
if (input.io) {
const federationNotice = federationNoticeFor(config.connections, input.projectDir);
if (federationNotice) {
writeSetupSection(input.io, 'Federated connection available', [federationNotice]);
}
}
}
/** @internal */
export function federationNoticeFor(
connections: Record<string, KtxProjectConnectionConfig>,
projectDir: string,
): string | null {
const descriptor = deriveFederatedConnection(connections, projectDir);
if (!descriptor) {
return null;
}
const names = descriptor.members.map((m) => m.connectionId).join(', ');
return `Detected ${descriptor.members.length} attach-compatible databases (${names}). Run a cross-database join as read-only SQL against \`${FEDERATED_CONNECTION_ID}\` (ktx sql -c ${FEDERATED_CONNECTION_ID} "SELECT ..."), using catalog-qualified table names.`;
}
async function disableConnectionQueryHistory(projectDir: string, connectionId: string): Promise<void> {

7
packages/cli/src/setup-sources.ts
View file

@ -20,6 +20,7 @@ import type { KtxCliIo } from './cli-runtime.js';
import { createCliSpinner, errorMessage, writePrefixedLines } from './clack.js';
import { pickNotionRootPages } from './notion-page-picker.js';
import { runKtxSourceMapping } from './source-mapping.js';
import { assertSafeConnectionId } from './context/sl/source-files.js';
import {
runConnectionSetupWithRecovery,
type ConfigureResult,
@ -206,12 +207,6 @@ async function promptText(
return await prompts.text({ ...options, message: withTextInputNavigation(options.message) });
}
function assertSafeConnectionId(connectionId: string): void {
if (!/^[a-zA-Z0-9][a-zA-Z0-9_-]*$/.test(connectionId)) {
throw new Error(`Unsafe connection id: ${connectionId}`);
}
}
function credentialRef(value: string | undefined, label: string): string {
const ref = value?.trim();
if (!ref) {

2
packages/cli/src/setup.ts
View file

@ -80,6 +80,7 @@ export type KtxSetupArgs =
agents: boolean;
target?: KtxAgentTarget;
agentScope?: KtxAgentScope;
installRoot?: string;
skipAgents?: boolean;
inputMode: 'auto' | 'disabled';
debug?: boolean;
@ -919,6 +920,7 @@ async function runKtxSetupInner(args: KtxSetupArgs, io: KtxCliIo, deps: KtxSetup
agents: true,
...(args.target ? { target: args.target } : {}),
scope: args.agentScope ?? 'project',
...(args.installRoot ? { installRoot: args.installRoot } : {}),
mode: 'mcp',
skipAgents: false,
showNextActions: agentsRequested,

4
packages/cli/src/sl.ts
View file

@ -80,7 +80,7 @@ interface KtxSlDeps {
installPolicy: KtxManagedPythonInstallPolicy;
io: KtxSlIo;
projectDir?: string;
}) => KtxSemanticLayerComputePort;
}) => Promise<KtxSemanticLayerComputePort>;
createQueryExecutor?: (project: KtxLocalProject) => KtxSqlQueryExecutorPort;
}
@ -315,7 +315,7 @@ export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: Ktx
queryForTelemetry = query;
const compute = deps.createSemanticLayerCompute
? deps.createSemanticLayerCompute()
: (deps.createManagedSemanticLayerCompute ?? createManagedPythonSemanticLayerComputePort)({
: await (deps.createManagedSemanticLayerCompute ?? createManagedPythonSemanticLayerComputePort)({
cliVersion: args.cliVersion,
installPolicy: args.runtimeInstallPolicy,
io,

102
packages/cli/src/sql.ts
View file

@ -1,5 +1,10 @@
import { executeFederatedQuery } from './connectors/duckdb/federated-executor.js';
import { FEDERATED_CONNECTION_ID } from './context/connections/federation.js';
import { executeProjectReadOnlySql } from './context/connections/project-sql-executor.js';
import type { KtxSqlQueryExecutionResult } from './context/connections/query-executor.js';
import { resolveConfiguredConnection } from './context/connections/resolve-connection.js';
import { loadKtxProject, type KtxLocalProject } from './context/project/project.js';
import type { KtxQueryResult, KtxScanConnector } from './context/scan/types.js';
import { sqlAnalysisDialectForDriver } from './context/sql-analysis/dialect.js';
import type { SqlAnalysisDialect, SqlAnalysisPort } from './context/sql-analysis/ports.js';
import type { KtxCliIo } from './cli-runtime.js';
import { type KtxOutputMode, resolveOutputMode } from './io/mode.js';
@ -30,6 +35,7 @@ export interface KtxSqlDeps {
loadProject?: typeof loadKtxProject;
createSqlAnalysis?: () => SqlAnalysisPort;
createScanConnector?: typeof createKtxCliScanConnector;
executeFederated?: typeof executeFederatedQuery;
}
interface SqlExecutionOutput {
@ -40,20 +46,6 @@ interface SqlExecutionOutput {
rowCount: number;
}
function sqlAnalysisDialectForDriver(driver: string | undefined): SqlAnalysisDialect {
const normalized = String(driver ?? '').trim().toLowerCase();
const map: Record<string, SqlAnalysisDialect> = {
postgres: 'postgres',
bigquery: 'bigquery',
snowflake: 'snowflake',
mysql: 'mysql',
sqlserver: 'tsql',
sqlite: 'sqlite',
clickhouse: 'clickhouse',
};
return map[normalized] ?? 'postgres';
}
function queryVerb(sql: string): 'select' | 'explain' | 'show' | 'with' | 'other' {
const first = sql.trim().split(/\s+/, 1)[0]?.toLowerCase();
if (first === 'select' || first === 'explain' || first === 'show' || first === 'with') {
@ -123,13 +115,7 @@ function printSqlResult(output: SqlExecutionOutput, mode: KtxSqlOutputMode, io:
printPretty(output, io);
}
async function cleanupConnector(connector: KtxScanConnector | null): Promise<void> {
if (connector?.cleanup) {
await connector.cleanup();
}
}
function resultOutput(connectionId: string, result: KtxQueryResult): SqlExecutionOutput {
function resultOutput(connectionId: string, result: KtxSqlQueryExecutionResult): SqlExecutionOutput {
return {
connectionId,
headers: result.headers,
@ -146,12 +132,10 @@ export async function runKtxSql(args: KtxSqlArgs, io: KtxCliIo = process, deps:
let project: KtxLocalProject | undefined;
try {
project = await (deps.loadProject ?? loadKtxProject)({ projectDir: args.projectDir });
const connection = project.config.connections[args.connectionId];
if (!connection) {
throw new Error(`Connection "${args.connectionId}" is not configured in ktx.yaml`);
}
driver = String(connection.driver ?? 'unknown').toLowerCase();
demoConnection = isDemoConnection(args.connectionId, connection);
const isFederated = args.connectionId === FEDERATED_CONNECTION_ID;
const connection = isFederated ? undefined : resolveConfiguredConnection(project.config, args.connectionId);
driver = isFederated ? 'duckdb' : String(connection?.driver ?? 'unknown').toLowerCase();
demoConnection = isFederated ? false : isDemoConnection(args.connectionId, connection);
const createSqlAnalysis =
deps.createSqlAnalysis ??
@ -163,7 +147,7 @@ export async function runKtxSql(args: KtxSqlArgs, io: KtxCliIo = process, deps:
io,
}));
const analysisPort = createSqlAnalysis();
const dialect = sqlAnalysisDialectForDriver(connection.driver);
const dialect: SqlAnalysisDialect = isFederated ? 'duckdb' : sqlAnalysisDialectForDriver(connection?.driver);
const validation = await analysisPort.validateReadOnly(args.sql, dialect);
if (!validation.ok) {
throw new Error(validation.error ?? 'SQL is not read-only.');
@ -171,39 +155,35 @@ export async function runKtxSql(args: KtxSqlArgs, io: KtxCliIo = process, deps:
const referencedTableCount = await safeReferencedTableCount(analysisPort, args.sql, dialect);
const createScanConnector = deps.createScanConnector ?? createKtxCliScanConnector;
let connector: KtxScanConnector | null = null;
try {
connector = await createScanConnector(project, args.connectionId);
if (!connector.capabilities.readOnlySql || !connector.executeReadOnly) {
throw new Error(`Connection "${args.connectionId}" does not support read-only SQL execution.`);
}
const result = await connector.executeReadOnly(
{
connectionId: args.connectionId,
sql: args.sql,
maxRows: args.maxRows,
},
{ runId: 'cli-sql' },
);
const mode = resolveOutputMode({ explicit: args.output, json: args.json, io });
printSqlResult(resultOutput(args.connectionId, result), mode, io);
await emitTelemetryEvent({
name: 'sql_completed',
const result = await executeProjectReadOnlySql({
project,
input: {
connectionId: args.connectionId,
projectDir: args.projectDir,
io,
fields: {
driver,
isDemoConnection: demoConnection,
queryVerb: queryVerb(args.sql),
referencedTableCount,
durationMs: Math.max(0, performance.now() - startedAt),
outcome: 'ok',
},
});
return 0;
} finally {
await cleanupConnector(connector);
}
connection,
sql: args.sql,
maxRows: args.maxRows,
},
createConnector: (connectionId) => createScanConnector(project!, connectionId),
executeFederated: deps.executeFederated,
runId: 'cli-sql',
});
const mode = resolveOutputMode({ explicit: args.output, json: args.json, io });
printSqlResult(resultOutput(args.connectionId, result), mode, io);
await emitTelemetryEvent({
name: 'sql_completed',
projectDir: args.projectDir,
io,
fields: {
driver,
isDemoConnection: demoConnection,
queryVerb: queryVerb(args.sql),
referencedTableCount,
durationMs: Math.max(0, performance.now() - startedAt),
outcome: 'ok',
},
});
return 0;
} catch (error) {
const errorClass = scrubErrorClass(error);
await emitTelemetryEvent({

5
packages/cli/src/telemetry/events.schema.json
View file

@ -162,6 +162,7 @@
"outcome",
"durationMs",
"errorClass",
"errorDetail",
"sampleRate",
"mcpClientName",
"mcpClientVersion"
@ -1167,6 +1168,10 @@
"errorClass": {
"type": "string"
},
"errorDetail": {
"type": "string",
"maxLength": 1000
},
"sampleRate": {
"type": "number",
"const": 1

12
packages/cli/src/telemetry/events.ts
View file

@ -161,6 +161,7 @@ const mcpRequestCompletedSchema = telemetryCommonEnvelopeSchema
outcome: outcomeSchema,
durationMs: z.number().nonnegative(),
errorClass: z.string().optional(),
errorDetail: z.string().max(1000).optional(),
sampleRate: z.literal(1),
// Raw, client-tool-controlled identity from the MCP initialize handshake
// (clientInfo.name/version). Optional: clients may omit clientInfo. Stored
@ -349,7 +350,16 @@ export const telemetryEventCatalog = [
{
name: 'mcp_request_completed',
description: 'Emitted for sampled MCP tool requests.',
fields: ['toolName', 'outcome', 'durationMs', 'errorClass', 'sampleRate', 'mcpClientName', 'mcpClientVersion'],
fields: [
'toolName',
'outcome',
'durationMs',
'errorClass',
'errorDetail',
'sampleRate',
'mcpClientName',
'mcpClientVersion',
],
},
{
name: 'daemon_started',

66
packages/cli/test/connection-list-federated.test.ts Normal file
View file

@ -0,0 +1,66 @@
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { runKtxConnection } from '../src/connection.js';
import { initKtxProject } from '../src/context/project/project.js';
import { parseKtxProjectConfig, serializeKtxProjectConfig } from '../src/context/project/config.js';
import type { KtxProjectConnectionConfig } from '../src/context/project/config.js';
function makeIo() {
const out: string[] = [];
return {
io: {
stdout: { isTTY: false, write: (c: string) => { out.push(c); return true; } },
stderr: { write: () => true },
},
stdout: () => out.join(''),
};
}
async function writeConnections(
projectDir: string,
connections: Record<string, KtxProjectConnectionConfig>,
): Promise<void> {
const config = parseKtxProjectConfig(await readFile(join(projectDir, 'ktx.yaml'), 'utf-8'));
await writeFile(join(projectDir, 'ktx.yaml'), serializeKtxProjectConfig({ ...config, connections }), 'utf-8');
}
describe('ktx connection list federated entry', () => {
let tempDir: string;
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-conn-fed-'));
});
afterEach(async () => {
await rm(tempDir, { recursive: true, force: true });
});
it('shows _ktx_federated when 2+ attach-compatible connections exist', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir });
await writeConnections(projectDir, {
books_db: { driver: 'sqlite' },
reviews_db: { driver: 'sqlite' },
});
const io = makeIo();
const code = await runKtxConnection({ command: 'list', projectDir }, io.io);
const printed = io.stdout();
expect(code).toBe(0);
expect(printed).toContain('_ktx_federated');
expect(printed).toContain('books_db, reviews_db');
expect(printed).toContain('Cross-database queries run here');
});
it('omits _ktx_federated with a single connection', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir });
await writeConnections(projectDir, {
books_db: { driver: 'sqlite' },
});
const io = makeIo();
await runKtxConnection({ command: 'list', projectDir }, io.io);
expect(io.stdout()).not.toContain('_ktx_federated');
});
});

143
packages/cli/test/connectors/duckdb/federated-attach.test.ts Normal file
View file

@ -0,0 +1,143 @@
import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { describe, expect, it } from 'vitest';
import { federatedAttachTarget } from '../../../src/connectors/duckdb/federated-attach.js';
import type { FederatedMember } from '../../../src/context/connections/federation.js';
const member = (over: Partial<FederatedMember>): FederatedMember => ({
connectionId: 'm',
driver: 'sqlite',
projectDir: '/proj',
connection: { driver: 'sqlite' },
...over,
});
describe('federatedAttachTarget', () => {
it('resolves a sqlite path: config to an absolute filesystem path against projectDir', () => {
const dir = mkdtempSync(join(tmpdir(), 'ktx-attach-'));
writeFileSync(join(dir, 'reviews.db'), '');
try {
const target = federatedAttachTarget(
member({ driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: './reviews.db' } }),
{},
);
expect(target).toBe(join(dir, 'reviews.db'));
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('resolves a sqlite file:// url to a filesystem path', () => {
const target = federatedAttachTarget(
member({ driver: 'sqlite', connection: { driver: 'sqlite', url: 'file:///data/reviews.db' } }),
{},
);
expect(target).toBe('/data/reviews.db');
});
it('builds a libpq connection string for postgres from host/database/user', () => {
const target = federatedAttachTarget(
member({
driver: 'postgres',
connection: { driver: 'postgres', host: 'h', port: 5433, database: 'books', username: 'u', password: 'p' },
}),
{},
);
expect(target).toContain('host=h');
expect(target).toContain('port=5433');
expect(target).toContain('dbname=books');
expect(target).toContain('user=u');
expect(target).toContain('password=p');
});
it('passes a postgres url through as the connection string', () => {
const target = federatedAttachTarget(
member({ driver: 'postgres', connection: { driver: 'postgres', url: 'env:PG_URL' } }),
{ PG_URL: 'postgresql://localhost/books' },
);
expect(target).toBe('postgresql://localhost/books');
});
it('adds sslmode=require to a postgres url when the member sets ssl', () => {
const target = federatedAttachTarget(
member({ driver: 'postgres', connection: { driver: 'postgres', url: 'env:PG_URL', ssl: true } }),
{ PG_URL: 'postgresql://localhost/books' },
);
expect(target).toContain('sslmode=require');
});
it('keeps a stronger sslmode already pinned in a postgres url', () => {
const target = federatedAttachTarget(
member({ driver: 'postgres', connection: { driver: 'postgres', url: 'env:PG_URL', ssl: true } }),
{ PG_URL: 'postgresql://localhost/books?sslmode=verify-full' },
);
expect(target).toContain('sslmode=verify-full');
expect(target).not.toContain('sslmode=require');
});
it('builds a mysql connection string from host/database/user', () => {
const target = federatedAttachTarget(
member({
driver: 'mysql',
connection: { driver: 'mysql', host: 'h', port: 3307, database: 'app', username: 'u', password: 'p' },
}),
{},
);
expect(target).toContain('host=h');
expect(target).toContain('port=3307');
expect(target).toContain('database=app');
expect(target).toContain('user=u');
expect(target).toContain('password=p');
});
it('quotes mysql values containing spaces', () => {
const target = federatedAttachTarget(
member({
driver: 'mysql',
connection: { driver: 'mysql', host: 'h', database: 'app', username: 'u', password: 'pass word' }, // pragma: allowlist secret
}),
{},
);
expect(target).toContain("password='pass word'"); // pragma: allowlist secret
});
it('emits sslmode=require for a postgres member configured with discrete fields and ssl', () => {
const target = federatedAttachTarget(
member({
driver: 'postgres',
connection: { driver: 'postgres', host: 'h', database: 'db', username: 'u', ssl: true },
}),
{},
);
expect(target).toContain('sslmode=require');
});
it('passes through the postgres search_path as options', () => {
const target = federatedAttachTarget(
member({
driver: 'postgres',
connection: { driver: 'postgres', host: 'h', database: 'db', username: 'u', schema: 'analytics' },
}),
{},
);
expect(target).toContain('search_path=analytics');
});
it('emits ssl_mode=REQUIRED for a mysql member with ssl', () => {
const target = federatedAttachTarget(
member({
driver: 'mysql',
connection: { driver: 'mysql', host: 'h', database: 'db', username: 'u', ssl: true },
}),
{},
);
expect(target).toContain('ssl_mode=REQUIRED');
});
it('throws for an unsupported driver', () => {
expect(() => federatedAttachTarget(member({ driver: 'snowflake', connection: { driver: 'snowflake' } }), {})).toThrow(
/cannot be attached/i,
);
});
});

70
packages/cli/test/connectors/duckdb/federated-executor.test.ts Normal file
View file

@ -0,0 +1,70 @@
import { describe, expect, it } from 'vitest';
import { buildAttachStatements } from '../../../src/connectors/duckdb/federated-executor.js';
import { attachTypeForDriver, type FederatedMember } from '../../../src/context/connections/federation.js';
const member = (
connectionId: string,
driver: string,
connection: FederatedMember['connection'],
): FederatedMember => ({ connectionId, driver, projectDir: '/proj', connection });
describe('attachTypeForDriver', () => {
it('maps drivers to DuckDB attach extension types', () => {
expect(attachTypeForDriver('postgres')).toBe('postgres');
expect(attachTypeForDriver('mysql')).toBe('mysql');
expect(attachTypeForDriver('sqlite')).toBe('sqlite');
});
it('throws for an unsupported driver', () => {
expect(() => attachTypeForDriver('snowflake')).toThrow(/cannot be attached/i);
});
});
describe('buildAttachStatements', () => {
it('loads each driver type once, then emits READ_ONLY ATTACH aliased by connectionId, resolving env refs', () => {
const stmts = buildAttachStatements(
[
member('pg_books', 'postgres', { driver: 'postgres', url: 'env:PG_URL' }),
member('sqlite_reviews', 'sqlite', { driver: 'sqlite', path: '/data/reviews.db' }),
],
{ PG_URL: 'postgresql://localhost/books' },
);
expect(stmts).toEqual([
'INSTALL postgres; LOAD postgres;',
'INSTALL sqlite; LOAD sqlite;',
'ATTACH \'postgresql://localhost/books\' AS "pg_books" (TYPE postgres, READ_ONLY);',
'ATTACH \'/data/reviews.db\' AS "sqlite_reviews" (TYPE sqlite, READ_ONLY);',
]);
});
it('loads a shared driver type only once across members', () => {
const stmts = buildAttachStatements(
[
member('pg_a', 'postgres', { driver: 'postgres', url: 'postgresql://h/a' }),
member('pg_b', 'postgres', { driver: 'postgres', url: 'postgresql://h/b' }),
],
{},
);
expect(stmts).toEqual([
'INSTALL postgres; LOAD postgres;',
'ATTACH \'postgresql://h/a\' AS "pg_a" (TYPE postgres, READ_ONLY);',
'ATTACH \'postgresql://h/b\' AS "pg_b" (TYPE postgres, READ_ONLY);',
]);
});
it('quotes a hyphenated connection id as a DuckDB identifier', () => {
const stmts = buildAttachStatements(
[member('postgres-warehouse', 'postgres', { driver: 'postgres', url: 'postgresql://h/db' })],
{},
);
expect(stmts.at(-1)).toBe(`ATTACH 'postgresql://h/db' AS "postgres-warehouse" (TYPE postgres, READ_ONLY);`);
});
it('escapes single quotes in a resolved attach target', () => {
const stmts = buildAttachStatements(
[member('pg', 'postgres', { driver: 'postgres', url: "postgresql://u:it's@h/db" })],
{},
);
expect(stmts.at(-1)).toBe('ATTACH \'postgresql://u:it\'\'s@h/db\' AS "pg" (TYPE postgres, READ_ONLY);');
});
});

147
packages/cli/test/connectors/duckdb/federated-join.integration.test.ts Normal file
View file

@ -0,0 +1,147 @@
import { describe, expect, it } from 'vitest';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import Database from 'better-sqlite3';
import { executeFederatedQuery } from '../../../src/connectors/duckdb/federated-executor.js';
import type { FederatedMember } from '../../../src/context/connections/federation.js';
describe('federated cross-catalog join (live DuckDB)', () => {
it('joins two sqlite catalogs and enforces read-only', async () => {
const dir = mkdtempSync(join(tmpdir(), 'ktx-fed-'));
const booksPath = join(dir, 'books.db');
const reviewsPath = join(dir, 'reviews.db');
const books = new Database(booksPath);
books.exec("CREATE TABLE books (id INTEGER, title TEXT); INSERT INTO books VALUES (1, 'Dune'), (2, 'Foundation');");
books.close();
const reviews = new Database(reviewsPath);
reviews.exec('CREATE TABLE reviews (book_id INTEGER, stars INTEGER); INSERT INTO reviews VALUES (1, 5), (1, 4), (2, 2);');
reviews.close();
const members: FederatedMember[] = [
{ connectionId: 'books_db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: booksPath } },
{ connectionId: 'reviews_db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: reviewsPath } },
];
try {
const result = await executeFederatedQuery(members, {
connectionId: '_ktx_federated',
connection: undefined,
sql: 'SELECT b.title, AVG(r.stars) AS avg_stars FROM books_db.books b JOIN reviews_db.reviews r ON b.id = r.book_id GROUP BY b.title ORDER BY b.title',
});
expect(result.headers).toEqual(['title', 'avg_stars']);
// ORDER BY title: Dune, Foundation
expect(result.rows.map((row) => row[0])).toEqual(['Dune', 'Foundation']);
expect(Number(result.rows[0][1])).toBeCloseTo(4.5); // Dune: (5+4)/2
expect(Number(result.rows[1][1])).toBeCloseTo(2.0); // Foundation: 2/1
await expect(
executeFederatedQuery(members, {
connectionId: '_ktx_federated',
connection: undefined,
sql: "INSERT INTO books_db.books VALUES (2, 'Hack')",
}),
).rejects.toThrow(/read-only/i);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('returns integer columns as JSON-safe numbers, not bigint', async () => {
const dir = mkdtempSync(join(tmpdir(), 'ktx-fed-bigint-'));
const booksPath = join(dir, 'books.db');
const reviewsPath = join(dir, 'reviews.db');
const books = new Database(booksPath);
books.exec("CREATE TABLE books (id INTEGER, title TEXT); INSERT INTO books VALUES (1, 'Dune'), (2, 'Foundation');");
books.close();
const reviews = new Database(reviewsPath);
reviews.exec('CREATE TABLE reviews (book_id INTEGER, stars INTEGER); INSERT INTO reviews VALUES (1, 5), (1, 4), (2, 2);');
reviews.close();
const members: FederatedMember[] = [
{ connectionId: 'books_db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: booksPath } },
{ connectionId: 'reviews_db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: reviewsPath } },
];
try {
const result = await executeFederatedQuery(members, {
connectionId: '_ktx_federated',
connection: undefined,
sql: 'SELECT b.id, count(*) AS n FROM books_db.books b JOIN reviews_db.reviews r ON b.id = r.book_id GROUP BY b.id ORDER BY b.id',
});
for (const row of result.rows) {
for (const cell of row) {
expect(typeof cell).not.toBe('bigint');
}
}
expect(() => JSON.stringify(result)).not.toThrow();
expect(result.rows[0][0]).toBe(1);
expect(Number(result.rows[0][1])).toBeGreaterThan(0);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('preserves a BIGINT beyond 2^53 as an exact string instead of rounding', async () => {
const dir = mkdtempSync(join(tmpdir(), 'ktx-fed-bigint-large-'));
const idsPath = join(dir, 'ids.db');
const otherPath = join(dir, 'other.db');
const ids = new Database(idsPath);
// 9007199254740993 = 2^53 + 1, which rounds to ...992 as a JS number; the
// literal lives in SQL text so sqlite stores it exactly.
ids.exec('CREATE TABLE ids (big_id INTEGER); INSERT INTO ids VALUES (9007199254740993);');
ids.close();
const other = new Database(otherPath);
other.exec('CREATE TABLE t (x INTEGER); INSERT INTO t VALUES (1);');
other.close();
const members: FederatedMember[] = [
{ connectionId: 'ids_db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: idsPath } },
{ connectionId: 'other_db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: otherPath } },
];
try {
const result = await executeFederatedQuery(members, {
connectionId: '_ktx_federated',
connection: undefined,
sql: 'SELECT big_id FROM ids_db.ids',
});
expect(result.rows[0][0]).toBe('9007199254740993');
expect(() => JSON.stringify(result)).not.toThrow();
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('joins catalogs whose connection ids contain hyphens', async () => {
const dir = mkdtempSync(join(tmpdir(), 'ktx-fed-hyphen-'));
const booksPath = join(dir, 'books.db');
const reviewsPath = join(dir, 'reviews.db');
const books = new Database(booksPath);
books.exec("CREATE TABLE books (id INTEGER, title TEXT); INSERT INTO books VALUES (1, 'Dune');");
books.close();
const reviews = new Database(reviewsPath);
reviews.exec('CREATE TABLE reviews (book_id INTEGER, stars INTEGER); INSERT INTO reviews VALUES (1, 5), (1, 3);');
reviews.close();
const members: FederatedMember[] = [
{ connectionId: 'books-db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: booksPath } },
{ connectionId: 'reviews-db', driver: 'sqlite', projectDir: dir, connection: { driver: 'sqlite', path: reviewsPath } },
];
try {
const result = await executeFederatedQuery(members, {
connectionId: '_ktx_federated',
connection: undefined,
sql: 'SELECT b.title, AVG(r.stars) AS avg_stars FROM "books-db".books b JOIN "reviews-db".reviews r ON b.id = r.book_id GROUP BY b.title',
});
expect(result.rows[0][0]).toBe('Dune');
expect(Number(result.rows[0][1])).toBeCloseTo(4.0);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});

41
packages/cli/test/connectors/shared/string-reference.test.ts Normal file
View file

@ -0,0 +1,41 @@
import { describe, expect, it } from 'vitest';
import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
import { homedir, tmpdir } from 'node:os';
import { join } from 'node:path';
import { resolveStringReference } from '../../../src/connectors/shared/string-reference.js';
describe('resolveStringReference', () => {
it('returns plain values unchanged', () => {
expect(resolveStringReference('postgres://localhost/db', {})).toBe('postgres://localhost/db');
});
it('resolves env: references from the provided env', () => {
expect(resolveStringReference('env:MY_URL', { MY_URL: 'resolved-url' })).toBe('resolved-url');
});
it('returns empty string for a missing env var', () => {
expect(resolveStringReference('env:NOPE', {})).toBe('');
});
it('resolves file: references and trims whitespace', () => {
const dir = mkdtempSync(join(tmpdir(), 'ktx-strref-'));
const file = join(dir, 'secret.txt');
writeFileSync(file, ' hunter2\n');
try {
expect(resolveStringReference(`file:${file}`, {})).toBe('hunter2');
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
it('expands ~ in file: references to the home directory', () => {
const name = `.ktx-strref-test-${process.pid}.txt`;
const abs = join(homedir(), name);
writeFileSync(abs, 'tilde-secret\n');
try {
expect(resolveStringReference(`file:~/${name}`, {})).toBe('tilde-secret');
} finally {
rmSync(abs, { force: true });
}
});
});

80
packages/cli/test/context/connections/federation.test.ts Normal file
View file

@ -0,0 +1,80 @@
import { describe, expect, it } from 'vitest';
import {
deriveFederatedConnection,
federatedConnectionListing,
FEDERATED_CONNECTION_ID,
} from '../../../src/context/connections/federation.js';
const conns = (entries: Record<string, { driver: string; [k: string]: unknown }>) => entries as never;
describe('deriveFederatedConnection', () => {
it('returns null with zero compatible members', () => {
expect(deriveFederatedConnection(conns({ snow: { driver: 'snowflake' } }), '/proj')).toBeNull();
});
it('returns null with exactly one compatible member', () => {
expect(deriveFederatedConnection(conns({ pg: { driver: 'postgres' } }), '/proj')).toBeNull();
});
it('derives a descriptor with two compatible members', () => {
const result = deriveFederatedConnection(
conns({ pg: { driver: 'postgres' }, lite: { driver: 'sqlite' } }),
'/proj',
);
expect(result).not.toBeNull();
expect(result?.id).toBe(FEDERATED_CONNECTION_ID);
expect(result?.driver).toBe('duckdb');
expect(result?.members.map((m) => m.connectionId).sort()).toEqual(['lite', 'pg']);
});
it('carries each member connection config and projectDir', () => {
const result = deriveFederatedConnection(
conns({ pg: { driver: 'postgres', host: 'h' }, lite: { driver: 'sqlite', path: './a.db' } }),
'/proj',
);
const pg = result?.members.find((m) => m.connectionId === 'pg');
expect(pg?.connection).toEqual({ driver: 'postgres', host: 'h' });
expect(pg?.projectDir).toBe('/proj');
});
it('excludes incompatible members from the group', () => {
const result = deriveFederatedConnection(
conns({ pg: { driver: 'postgres' }, my: { driver: 'mysql' }, snow: { driver: 'snowflake' } }),
'/proj',
);
expect(result?.members.map((m) => m.connectionId).sort()).toEqual(['my', 'pg']);
});
it('is case-insensitive on driver names', () => {
const result = deriveFederatedConnection(
conns({ pg: { driver: 'POSTGRES' }, lite: { driver: 'SQLite' } }),
'/proj',
);
expect(result?.members).toHaveLength(2);
});
});
describe('federatedConnectionListing', () => {
it('returns null with fewer than 2 attach-compatible connections', () => {
expect(
federatedConnectionListing({ books_db: { driver: 'sqlite', path: './b.db' } }, '/tmp/p'),
).toBeNull();
});
it('returns id, driver, member ids and a usage hint with 2+ members', () => {
const listing = federatedConnectionListing(
{
books_db: { driver: 'sqlite', path: './b.db' },
reviews_db: { driver: 'sqlite', path: './r.db' },
snow: { driver: 'snowflake', account: 'x' },
},
'/tmp/p',
);
expect(listing).not.toBeNull();
expect(listing!.id).toBe(FEDERATED_CONNECTION_ID);
expect(listing!.driver).toBe('duckdb');
expect(listing!.members).toEqual(['books_db', 'reviews_db']);
expect(listing!.hint).toContain('Cross-database');
expect(listing!.hint).toContain('connectionId.table');
});
});

57
packages/cli/test/context/connections/project-sql-executor.integration.test.ts Normal file
View file

@ -0,0 +1,57 @@
import { describe, expect, it } from 'vitest';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import Database from 'better-sqlite3';
import { executeProjectReadOnlySql } from '../../../src/context/connections/project-sql-executor.js';
import type { KtxLocalProject } from '../../../src/context/project/project.js';
function fakeProject(projectDir: string, connections: Record<string, { driver: string; path: string }>): KtxLocalProject {
return {
projectDir,
configPath: join(projectDir, 'ktx.yaml'),
config: { connections } as unknown as KtxLocalProject['config'],
coreConfig: {} as KtxLocalProject['coreConfig'],
git: {} as KtxLocalProject['git'],
fileStore: {} as KtxLocalProject['fileStore'],
};
}
describe('executeProjectReadOnlySql — federated integration (real DuckDB)', () => {
it('runs a federated cross-catalog join through the default executeFederatedQuery', async () => {
const dir = mkdtempSync(join(tmpdir(), 'ktx-fed-exec-'));
const booksPath = join(dir, 'books.db');
const reviewsPath = join(dir, 'reviews.db');
const books = new Database(booksPath);
books.exec("CREATE TABLE books (id INTEGER, title TEXT); INSERT INTO books VALUES (1, 'Dune'), (2, 'Foundation');");
books.close();
const reviews = new Database(reviewsPath);
reviews.exec('CREATE TABLE reviews (book_id INTEGER, stars INTEGER); INSERT INTO reviews VALUES (1, 5), (1, 4), (2, 2);');
reviews.close();
const project = fakeProject(dir, {
books_db: { driver: 'sqlite', path: booksPath },
reviews_db: { driver: 'sqlite', path: reviewsPath },
});
try {
const result = await executeProjectReadOnlySql({
project,
input: {
connectionId: '_ktx_federated',
connection: undefined,
sql: 'SELECT b.title, AVG(r.stars) AS avg_stars FROM books_db.books b JOIN reviews_db.reviews r ON b.id = r.book_id GROUP BY b.title ORDER BY b.title',
maxRows: 100,
},
createConnector: () => {
throw new Error('federated path must not create a scan connector');
},
});
expect(result.rows.map((row) => row[0])).toEqual(['Dune', 'Foundation']);
expect(Number(result.rows[0][1])).toBeCloseTo(4.5);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});

116
packages/cli/test/context/connections/project-sql-executor.test.ts Normal file
View file

@ -0,0 +1,116 @@
import { describe, expect, it, vi } from 'vitest';
import type { executeFederatedQuery } from '../../../src/connectors/duckdb/federated-executor.js';
import { executeProjectReadOnlySql } from '../../../src/context/connections/project-sql-executor.js';
import type { KtxLocalProject } from '../../../src/context/project/project.js';
import type { KtxScanConnector } from '../../../src/context/scan/types.js';
function fakeProject(connections: Record<string, { driver: string }>): KtxLocalProject {
return {
projectDir: '/tmp/proj',
configPath: '/tmp/proj/ktx.yaml',
config: { connections } as unknown as KtxLocalProject['config'],
coreConfig: {} as KtxLocalProject['coreConfig'],
git: {} as KtxLocalProject['git'],
fileStore: {} as KtxLocalProject['fileStore'],
};
}
describe('executeProjectReadOnlySql — federated routing', () => {
it('routes _ktx_federated through the federated executor with derived members', async () => {
const project = fakeProject({ pg: { driver: 'postgres' }, lite: { driver: 'sqlite' } });
const executeFederated = vi.fn<typeof executeFederatedQuery>(async () => ({
headers: ['x'],
rows: [[1]],
totalRows: 1,
command: 'SELECT',
rowCount: 1,
}));
const createConnector = vi.fn();
const result = await executeProjectReadOnlySql({
project,
input: { connectionId: '_ktx_federated', connection: undefined, sql: 'SELECT 1', maxRows: 100 },
createConnector: createConnector as never,
executeFederated,
});
expect(result.rows).toEqual([[1]]);
expect(executeFederated).toHaveBeenCalledOnce();
const members = executeFederated.mock.calls[0][0];
expect(members.map((m) => m.connectionId).sort()).toEqual(['lite', 'pg']);
expect(createConnector).not.toHaveBeenCalled();
});
it('throws when _ktx_federated requested but fewer than 2 compatible members', async () => {
const project = fakeProject({ pg: { driver: 'postgres' } });
await expect(
executeProjectReadOnlySql({
project,
input: { connectionId: '_ktx_federated', connection: undefined, sql: 'SELECT 1', maxRows: 100 },
createConnector: (() => {
throw new Error('should not be called');
}) as never,
executeFederated: vi.fn(),
}),
).rejects.toThrow(/fewer than 2/i);
});
it('routes a normal connection through the scan connector', async () => {
const project = fakeProject({ pg: { driver: 'postgres' } });
const connector = {
driver: 'postgres',
capabilities: { readOnlySql: true },
executeReadOnly: vi.fn(async () => ({ headers: ['a'], rows: [['v']], totalRows: 1, rowCount: 1 })),
cleanup: vi.fn(async () => {}),
};
const result = await executeProjectReadOnlySql({
project,
input: { connectionId: 'pg', connection: { driver: 'postgres' }, sql: 'SELECT a', maxRows: 50 },
createConnector: (async () => connector) as never,
executeFederated: vi.fn(),
});
expect(result.rows).toEqual([['v']]);
expect(connector.executeReadOnly).toHaveBeenCalledOnce();
expect(connector.cleanup).toHaveBeenCalledOnce();
});
});
function connectorReturning(result: {
headers: string[];
headerTypes?: string[];
rows: unknown[][];
totalRows: number;
rowCount: number | null;
}): KtxScanConnector {
return {
driver: 'sqlite',
capabilities: { readOnlySql: true },
async executeReadOnly() {
return result;
},
} as unknown as KtxScanConnector;
}
describe('executeProjectReadOnlySql headerTypes', () => {
it('forwards connector headerTypes on the non-federated branch', async () => {
const project = {
projectDir: '/tmp/p',
config: { connections: { books_db: { driver: 'sqlite', path: './b.db' } } },
} as never;
const result = await executeProjectReadOnlySql({
project,
input: { connectionId: 'books_db', connection: undefined, sql: 'SELECT 1', maxRows: 10 },
createConnector: () =>
connectorReturning({
headers: ['id'],
headerTypes: ['INTEGER'],
rows: [[1]],
totalRows: 1,
rowCount: 1,
}),
});
expect(result.headerTypes).toEqual(['INTEGER']);
});
});

15
packages/cli/test/context/connections/read-only-sql.test.ts
View file

@ -1,4 +1,5 @@
import { describe, expect, it } from 'vitest';
import { KtxExpectedError, KtxQueryError } from '../../../src/errors.js';
import {
assertReadOnlySql,
limitSqlForExecution,
@ -20,6 +21,20 @@ describe('assertReadOnlySql', () => {
);
});
// A guard refusing the agent's SQL is an expected outcome; classifying it as
// KtxQueryError keeps reportException from filing it as a ktx fault.
it('rejects with an expected KtxQueryError, not a bare Error', () => {
expect(() => assertReadOnlySql('delete from orders')).toThrow(KtxQueryError);
expect(() => assertReadOnlySql('describe orders')).toThrow(KtxQueryError);
expect(() => assertReadOnlySql('select 1; drop table orders')).toThrow(KtxQueryError);
try {
assertReadOnlySql('describe orders');
expect.unreachable('expected a throw');
} catch (error) {
expect(error).toBeInstanceOf(KtxExpectedError);
}
});
it('accepts read-only queries that begin with leading comments', () => {
expect(assertReadOnlySql('-- daily widget sales\nselect count(*) from public.widget_sales')).toBe(
'select count(*) from public.widget_sales',

80
packages/cli/test/context/connections/resolve-connection.test.ts Normal file
View file

@ -0,0 +1,80 @@
import { describe, expect, it } from 'vitest';
import {
buildDefaultKtxProjectConfig,
type KtxProjectConfig,
} from '../../../src/context/project/config.js';
import {
resolveConfiguredConnection,
resolveRequiredConnectionId,
} from '../../../src/context/connections/resolve-connection.js';
import { KtxExpectedError } from '../../../src/errors.js';
function configWith(ids: string[]): KtxProjectConfig {
const config = buildDefaultKtxProjectConfig();
for (const id of ids) {
config.connections[id] = { driver: 'postgres' };
}
return config;
}
describe('resolveConfiguredConnection', () => {
it('returns the connection config when the id is configured', () => {
const config = configWith(['warehouse']);
expect(resolveConfiguredConnection(config, 'warehouse')).toEqual({ driver: 'postgres' });
});
it('throws an expected error that lists the configured connections', () => {
const config = configWith(['analytics', 'warehouse']);
let error: unknown;
try {
resolveConfiguredConnection(config, 'ARK');
} catch (caught) {
error = caught;
}
expect(error).toBeInstanceOf(KtxExpectedError);
expect((error as Error).message).toBe(
'Connection "ARK" is not configured in ktx.yaml. Configured connections: analytics, warehouse.',
);
});
it('reports when no connections are configured at all', () => {
const config = configWith([]);
expect(() => resolveConfiguredConnection(config, 'warehouse')).toThrow(
'Connection "warehouse" is not configured in ktx.yaml. No connections are configured in ktx.yaml.',
);
});
});
describe('resolveRequiredConnectionId', () => {
it('returns the requested id when it is configured', () => {
const config = configWith(['warehouse']);
expect(resolveRequiredConnectionId(config, 'warehouse')).toBe('warehouse');
});
it('throws an expected error listing connections when the requested id is unknown', () => {
const config = configWith(['analytics', 'warehouse']);
expect(() => resolveRequiredConnectionId(config, 'DIG_SMART_REP')).toThrow(KtxExpectedError);
expect(() => resolveRequiredConnectionId(config, 'DIG_SMART_REP')).toThrow(
'Connection "DIG_SMART_REP" is not configured in ktx.yaml. Configured connections: analytics, warehouse.',
);
});
it('defaults to the only connection when the id is omitted', () => {
const config = configWith(['warehouse']);
expect(resolveRequiredConnectionId(config, undefined)).toBe('warehouse');
});
it('throws an expected error listing connections when the id is omitted and several exist', () => {
const config = configWith(['analytics', 'warehouse']);
let error: unknown;
try {
resolveRequiredConnectionId(config, undefined);
} catch (caught) {
error = caught;
}
expect(error).toBeInstanceOf(KtxExpectedError);
expect((error as Error).message).toBe(
'connectionId is required. Configured connections: analytics, warehouse.',
);
});
});

19
packages/cli/test/context/core/git.service.init-identity.test.ts
View file

@ -94,4 +94,23 @@ describe('GitService.initialize without a configured git identity', () => {
}).trim();
expect(localName).toBe('');
});
// Regression for KLO-735: a machine with commit.gpgsign=true makes git try to GPG-sign every
// commit, but ktx commits under a synthetic identity that can never own a secret key, so signing
// fails with "No secret key". ktx commits must succeed regardless of the user's signing config.
it('commits even when the global git config forces gpg signing', async () => {
// Force signing and point gpg at a program that always fails, mirroring a machine whose
// configured signing key does not match ktx's synthetic identity.
await writeFile(
join(homeDir, '.gitconfig'),
'[user]\n\tuseConfigOnly = true\n[commit]\n\tgpgsign = true\n[gpg]\n\tprogram = false\n',
'utf-8',
);
const service = new GitService(coreConfig(repoDir));
await expect(service.onModuleInit()).resolves.toBeUndefined();
const head = await service.revParseHead();
expect(head).toMatch(/^[0-9a-f]{40}$/);
});
});

6
packages/cli/test/context/core/git.service.repo-isolation.test.ts
View file

@ -21,7 +21,11 @@ function coreConfig(configDir: string): KtxCoreConfig {
}
function git(cwd: string, args: string[]): string {
return execFileSync('git', args, {
// `-c commit.gpgsign=false` keeps fixture commits deterministic regardless of the host's git
// config: a contributor with commit.gpgsign=true would otherwise fail these raw commits under a
// synthetic identity that owns no secret key.
const fixtureArgs = args[0] === 'commit' ? ['-c', 'commit.gpgsign=false', ...args] : args;
return execFileSync('git', fixtureArgs, {
cwd,
encoding: 'utf-8',
env: {

70
packages/cli/test/context/ingest/manifest-federated-join.test.ts Normal file
View file

@ -0,0 +1,70 @@
import { describe, expect, it } from 'vitest';
import { buildJoinsByTable, buildLiveDatabaseManifestShards } from '../../../src/context/ingest/adapters/live-database/manifest.js';
const joinData = (toTable: string) => ({
fromTable: 'books',
fromColumns: ['id'],
toTable,
toColumns: ['book_id'],
relationship: 'one_to_many',
source: 'manual' as const,
});
describe('buildJoinsByTable federated siblings', () => {
it('keeps a forward join whose target is a federated sibling table', () => {
const result = buildJoinsByTable(
new Set(['books']), // current snapshot
[joinData('sqlite_reviews.reviews')], // target NOT local
new Map(),
new Set(['sqlite_reviews.reviews']), // federated sibling targets
);
expect(result.get('books')?.map((j) => j.to)).toEqual(['sqlite_reviews.reviews']);
// The sibling target must NOT get a reverse entry (it has no shard in this snapshot)
expect(result.get('sqlite_reviews.reviews')).toBeUndefined();
});
it('still drops a join whose target is neither local nor a sibling', () => {
const result = buildJoinsByTable(new Set(['books']), [joinData('ghost')], new Map(), new Set());
expect(result.get('books')).toBeUndefined();
});
it('keeps both directions for a fully-local join (unchanged behavior)', () => {
const result = buildJoinsByTable(new Set(['books', 'authors']), [joinData('authors')], new Map(), new Set());
expect(result.get('books')?.map((j) => j.to)).toEqual(['authors']);
expect(result.get('authors')?.map((j) => j.to)).toEqual(['books']); // reverse still added for local joins
});
});
describe('buildLiveDatabaseManifestShards federated preserved joins', () => {
it('keeps a preserved manual join whose target is a federated sibling', () => {
const result = buildLiveDatabaseManifestShards({
connectionType: 'POSTGRES',
tables: [{ name: 'books', catalog: null, db: 'public', columns: [{ name: 'id', type: 'int' }] }],
joins: [],
existingPreservedJoins: new Map([
[
'books',
[{ to: 'sqlite_reviews.reviews', on: 'books.id = reviews.book_id', relationship: 'one_to_many', source: 'manual' }],
],
]),
federatedSiblingTargets: new Set(['sqlite_reviews.reviews']),
mapColumnType: (t) => t,
});
const shard = result.shards.get('public');
expect(shard?.tables.books?.joins?.map((j) => j.to)).toEqual(['sqlite_reviews.reviews']);
});
it('still drops a preserved join whose target is neither local nor a sibling', () => {
const result = buildLiveDatabaseManifestShards({
connectionType: 'POSTGRES',
tables: [{ name: 'books', catalog: null, db: 'public', columns: [{ name: 'id', type: 'int' }] }],
joins: [],
existingPreservedJoins: new Map([
['books', [{ to: 'ghost', on: 'books.id = ghost.id', relationship: 'one_to_many', source: 'manual' }]],
]),
federatedSiblingTargets: new Set(),
mapColumnType: (t) => t,
});
expect(result.shards.get('public')?.tables.books?.joins).toBeUndefined();
});
});

66
packages/cli/test/context/llm/claude-code-runtime.test.ts
View file

@ -738,4 +738,70 @@ describe('ClaudeCodeKtxLlmRuntime', () => {
message: 'Unsupported Claude Code model "gpt-5". Use sonnet, opus, haiku, or a claude-* model id.',
});
});
it('reports a Claude Code session limit as a session limit, not an auth or ktx failure', async () => {
const query = vi.fn((_input: any) =>
stream([
initMessage(),
resultMessage({
subtype: 'error_during_execution',
is_error: true,
errors: ["You've hit your session limit · resets 10:50pm (Asia/Saigon)"],
}),
]),
);
const result = await runClaudeCodeAuthProbe({ projectDir: '/tmp/project', model: 'sonnet', query, env: {} });
const message = (result as { message: string }).message;
expect(result.ok).toBe(false);
// Auth worked; the subscription is capped. Must not tell the user to re-authenticate.
expect(message).not.toContain('authentication is not usable');
// Frame it as Claude Code's session limit, tell them to wait, and preserve the reset text.
expect(message).toContain('Claude Code session limit reached');
expect(message).toContain('Wait for the reset shown');
expect(message).toContain('resets 10:50pm (Asia/Saigon)');
});
it('reports a Claude Code rate limit as a rate limit, not an auth failure', async () => {
const query = vi.fn((_input: any) =>
stream([
initMessage(),
resultMessage({
subtype: 'error_during_execution',
is_error: true,
errors: ['API Error: 429 Too Many Requests'],
}),
]),
);
const result = await runClaudeCodeAuthProbe({ projectDir: '/tmp/project', model: 'sonnet', query, env: {} });
const message = (result as { message: string }).message;
expect(result.ok).toBe(false);
expect(message).not.toContain('authentication is not usable');
expect(message).toContain('Claude Code is rate limited');
expect(message).toContain('429 Too Many Requests');
});
it('still reports a genuine auth failure as an auth failure', async () => {
const query = vi.fn((_input: any) =>
stream([
initMessage(),
resultMessage({
subtype: 'error_during_execution',
is_error: true,
errors: ['Invalid API key · Please run `claude login`'],
}),
]),
);
const result = await runClaudeCodeAuthProbe({ projectDir: '/tmp/project', model: 'sonnet', query, env: {} });
const message = (result as { message: string }).message;
expect(result.ok).toBe(false);
expect(message).toContain('authentication is not usable');
expect(message).not.toContain('session limit reached');
expect(message).not.toContain('rate limited');
});
});

11
packages/cli/test/context/mcp/__snapshots__/mcp-tools-list.json
View file

@ -2,7 +2,7 @@
{
"name": "connection_list",
"title": "Connection List",
"description": "List configured read-only data connections available to this ktx project. Use this before connection-scoped tools when the project may have multiple warehouses.",
"description": "List configured read-only data connections available to this ktx project. Use this before connection-scoped tools when the project may have multiple warehouses. A \"_ktx_federated\" entry (when present) queries all its member databases together; use its id for cross-database joins.",
"inputSchema": {
"type": "object",
"properties": {},
@ -24,6 +24,15 @@
},
"connectionType": {
"type": "string"
},
"members": {
"type": "array",
"items": {
"type": "string"
}
},
"hint": {
"type": "string"
}
},
"required": [

34
packages/cli/test/context/mcp/connection-list-federated.test.ts Normal file
View file

@ -0,0 +1,34 @@
import { describe, expect, it } from 'vitest';
import { createLocalProjectMcpContextPorts } from '../../../src/context/mcp/local-project-ports.js';
const project = {
projectDir: '/tmp/p',
config: {
connections: {
books_db: { driver: 'sqlite', path: './b.db' },
reviews_db: { driver: 'sqlite', path: './r.db' },
},
},
} as never;
describe('MCP connection_list federated entry', () => {
it('includes _ktx_federated with members and hint', async () => {
const ports = createLocalProjectMcpContextPorts(project, { embeddingService: null });
const list = await ports.connections!.list();
const federated = list.find((c) => c.id === '_ktx_federated');
expect(federated).toBeDefined();
expect(federated!.connectionType).toBe('DUCKDB');
expect(federated!.members).toEqual(['books_db', 'reviews_db']);
expect(federated!.hint).toContain('Cross-database');
});
it('omits _ktx_federated with a single connection', async () => {
const single = {
projectDir: '/tmp/p',
config: { connections: { books_db: { driver: 'sqlite', path: './b.db' } } },
} as never;
const ports = createLocalProjectMcpContextPorts(single, { embeddingService: null });
const list = await ports.connections!.list();
expect(list.find((c) => c.id === '_ktx_federated')).toBeUndefined();
});
});

99
packages/cli/test/context/mcp/local-project-ports-federated.integration.test.ts Normal file
View file

@ -0,0 +1,99 @@
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import Database from 'better-sqlite3';
import { describe, expect, it, vi } from 'vitest';
import { createLocalProjectMcpContextPorts } from '../../../src/context/mcp/local-project-ports.js';
import { initKtxProject } from '../../../src/context/project/project.js';
describe('MCP sql_execution — federated routing (live DuckDB)', () => {
it('routes _ktx_federated through the shared federated executor, validating with the duckdb dialect', async () => {
const dir = await mkdtemp(join(tmpdir(), 'ktx-mcp-fed-'));
try {
const booksPath = join(dir, 'books.db');
const reviewsPath = join(dir, 'reviews.db');
const books = new Database(booksPath);
books.exec("CREATE TABLE books (id INTEGER, title TEXT); INSERT INTO books VALUES (1, 'Dune');");
books.close();
const reviews = new Database(reviewsPath);
reviews.exec('CREATE TABLE reviews (book_id INTEGER, stars INTEGER); INSERT INTO reviews VALUES (1, 5), (1, 3);');
reviews.close();
const project = await initKtxProject({ projectDir: dir });
project.config.connections.books_db = { driver: 'sqlite', path: booksPath };
project.config.connections.reviews_db = { driver: 'sqlite', path: reviewsPath };
const validateReadOnly = vi.fn(async () => ({ ok: true, error: null }));
const ports = createLocalProjectMcpContextPorts(project, {
sqlAnalysis: {
analyzeForFingerprint: vi.fn(),
analyzeBatch: vi.fn(),
validateReadOnly,
} as never,
localScan: {
createConnector: () => {
throw new Error('federated path must not create a scan connector');
},
},
embeddingService: null,
});
const result = await ports.sqlExecution?.execute({
connectionId: '_ktx_federated',
sql: 'SELECT b.title, AVG(r.stars) AS avg_stars FROM books_db.books b JOIN reviews_db.reviews r ON b.id = r.book_id GROUP BY b.title',
maxRows: 100,
});
expect(result?.rows?.[0]?.[0]).toBe('Dune');
// Federated validation uses the duckdb dialect, not a member driver.
expect(validateReadOnly).toHaveBeenCalledWith(expect.any(String), 'duckdb');
} finally {
await rm(dir, { recursive: true, force: true });
}
});
it('serializes integer columns from a federated query without throwing on bigint', async () => {
const dir = await mkdtemp(join(tmpdir(), 'ktx-mcp-fed-int-'));
try {
const booksPath = join(dir, 'books.db');
const reviewsPath = join(dir, 'reviews.db');
const books = new Database(booksPath);
books.exec("CREATE TABLE books (id INTEGER, title TEXT); INSERT INTO books VALUES (1, 'Dune');");
books.close();
const reviews = new Database(reviewsPath);
reviews.exec('CREATE TABLE reviews (book_id INTEGER, stars INTEGER); INSERT INTO reviews VALUES (1, 5), (1, 3);');
reviews.close();
const project = await initKtxProject({ projectDir: dir });
project.config.connections.books_db = { driver: 'sqlite', path: booksPath };
project.config.connections.reviews_db = { driver: 'sqlite', path: reviewsPath };
const validateReadOnly = vi.fn(async () => ({ ok: true, error: null }));
const ports = createLocalProjectMcpContextPorts(project, {
sqlAnalysis: {
analyzeForFingerprint: vi.fn(),
analyzeBatch: vi.fn(),
validateReadOnly,
} as never,
localScan: {
createConnector: () => {
throw new Error('federated path must not create a scan connector');
},
},
embeddingService: null,
});
const result = await ports.sqlExecution?.execute({
connectionId: '_ktx_federated',
sql: 'SELECT b.title, count(*) AS n FROM books_db.books b JOIN reviews_db.reviews r ON b.id = r.book_id GROUP BY b.title',
maxRows: 100,
});
expect(() => JSON.stringify(result)).not.toThrow();
expect(result?.rows?.[0]?.[0]).toBe('Dune');
expect(result?.rows?.[0]?.[1]).toBe(2);
} finally {
await rm(dir, { recursive: true, force: true });
}
});
});

39
packages/cli/test/context/mcp/local-project-ports.test.ts
View file

@ -3,7 +3,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { initKtxProject } from '../../../src/context/project/project.js';
import { KtxQueryError } from '../../../src/errors.js';
import { KtxExpectedError, KtxQueryError } from '../../../src/errors.js';
import { createKtxConnectorCapabilities, type KtxQueryResult, type KtxScanConnector, type KtxSchemaSnapshot } from '../../../src/context/scan/types.js';
import { SemanticLayerService } from '../../../src/context/sl/semantic-layer.service.js';
import type { SemanticLayerSource } from '../../../src/context/sl/types.js';
@ -245,6 +245,43 @@ describe('createLocalProjectMcpContextPorts', () => {
expect(connector.cleanup).toHaveBeenCalled();
});
it('rejects sql_execution against an unconfigured connection with an actionable expected error', async () => {
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {
driver: 'postgres',
url: 'env:DATABASE_URL',
};
const connector = testConnector(testSnapshot(), {
headers: ['id'],
headerTypes: ['integer'],
rows: [[1]],
totalRows: 1,
rowCount: 1,
});
const createConnector = vi.fn(async () => connector);
const sqlAnalysis = {
analyzeForFingerprint: vi.fn(),
analyzeBatch: vi.fn(),
validateReadOnly: vi.fn(async () => ({ ok: true, error: null })),
};
const ports = createLocalProjectMcpContextPorts(project, {
sqlAnalysis,
localScan: { createConnector },
embeddingService: null,
});
const execution = ports.sqlExecution?.execute({
connectionId: 'DIG_SMART_REP',
sql: 'select 1',
maxRows: 5,
});
await expect(execution).rejects.toBeInstanceOf(KtxExpectedError);
await expect(execution).rejects.toThrow(
'Connection "DIG_SMART_REP" is not configured in ktx.yaml. Configured connections: warehouse.',
);
expect(createConnector).not.toHaveBeenCalled();
});
it('emits sql_execution progress stages from local MCP ports', async () => {
const project = await initKtxProject({ projectDir: tempDir });
project.config.connections.warehouse = {

29
packages/cli/test/context/mcp/server.test.ts
View file

@ -287,6 +287,35 @@ describe('createKtxMcpServer', () => {
expect(io.stderrText()).not.toContain('mcpClientVersion');
});
it('records the failure message as errorDetail when a tool returns an error', async () => {
vi.spyOn(Math, 'random').mockReturnValue(0);
vi.stubEnv('KTX_TELEMETRY_DEBUG', '1');
vi.stubEnv('CI', '');
const fake = makeFakeServer();
const io = makeIo();
createKtxMcpServer({
server: fake.server,
userContext: { userId: 'local-user' },
projectDir: '/tmp/ktx-mcp-error-detail',
io,
contextTools: {
knowledge: {
search: vi.fn<KtxKnowledgeMcpPort['search']>().mockRejectedValue(new Error('wiki search exploded')),
read: vi.fn<KtxKnowledgeMcpPort['read']>().mockResolvedValue(null),
},
},
});
await expect(getTool(fake.tools, 'wiki_search').handler({ query: 'revenue', limit: 5 })).resolves.toMatchObject({
isError: true,
});
expect(io.stderrText()).toContain('"event":"mcp_request_completed"');
expect(io.stderrText()).toContain('"outcome":"error"');
expect(io.stderrText()).toContain('"errorDetail":"wiki search exploded"');
});
it('reports MCP tool exceptions with a tool-derived source', async () => {
reportExceptionMock.mockClear();
vi.stubEnv('ANTHROPIC_API_KEY', 'mcp-anthropic-secret'); // pragma: allowlist secret

130
packages/cli/test/context/scan/local-enrichment-federated-join.test.ts Normal file
View file

@ -0,0 +1,130 @@
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import YAML from 'yaml';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { buildDefaultKtxProjectConfig } from '../../../src/context/project/config.js';
import type { GitService } from '../../../src/context/core/git.service.js';
import { LocalGitFileStore } from '../../../src/context/project/local-git-file-store.js';
import type { KtxLocalProject } from '../../../src/context/project/project.js';
import { writeLocalScanManifestShards } from '../../../src/context/scan/local-enrichment-artifacts.js';
import type { KtxSchemaSnapshot } from '../../../src/context/scan/types.js';
// `writeLocalScanManifestShards` commits its output via git; the file is
// already on disk before the commit call, so the stub only returns commit info.
const stubGitCommitFile: Pick<GitService, 'commitFile'> = {
commitFile: async () => ({
commitHash: 'stub',
shortHash: 'stub',
message: 'stub',
author: 'ktx',
authorEmail: 'ktx@example.com',
timestamp: new Date().toISOString(),
committedDate: new Date().toISOString(),
created: true,
}),
};
const stubGit = stubGitCommitFile as GitService;
function fakeProject(projectDir: string, connections: KtxLocalProject['config']['connections']): KtxLocalProject {
const fileStore = new LocalGitFileStore({ rootDir: projectDir, git: stubGit });
return {
projectDir,
configPath: join(projectDir, 'ktx.yaml'),
config: { ...buildDefaultKtxProjectConfig(), connections },
coreConfig: {} as KtxLocalProject['coreConfig'],
git: stubGit,
fileStore,
};
}
const EXISTING_BOOKS_SHARD = `tables:
books:
table: public.books
columns:
- name: id
type: number
pk: true
joins:
- to: sqlite_reviews.reviews
on: books.id = reviews.book_id
relationship: one_to_many
source: manual
`;
const booksSnapshot: KtxSchemaSnapshot = {
connectionId: 'pg_books',
driver: 'postgres',
extractedAt: new Date().toISOString(),
scope: {},
metadata: {},
tables: [
{
name: 'books',
catalog: null,
db: 'public',
kind: 'table',
comment: null,
estimatedRows: null,
columns: [
{
name: 'id',
nativeType: 'integer',
normalizedType: 'integer',
dimensionType: 'number',
nullable: false,
primaryKey: true,
comment: null,
},
],
foreignKeys: [],
},
],
};
describe('writeLocalScanManifestShards federated cross-DB joins', () => {
let tempDir: string;
let project: KtxLocalProject;
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-enrich-fed-'));
project = fakeProject(join(tempDir, 'project'), {
pg_books: { driver: 'postgres' },
sqlite_reviews: { driver: 'sqlite' },
});
await project.fileStore.writeFile(
'semantic-layer/pg_books/_schema/public.yaml',
EXISTING_BOOKS_SHARD,
'ktx',
'ktx@example.com',
'seed',
{ skipLock: true },
);
await project.fileStore.writeFile(
'semantic-layer/sqlite_reviews/_schema/main.yaml',
'tables:\n reviews:\n table: reviews\n columns:\n - name: book_id\n type: number\n',
'ktx',
'ktx@example.com',
'seed',
{ skipLock: true },
);
});
afterEach(async () => {
await rm(tempDir, { recursive: true, force: true });
});
it('preserves a manual cross-DB join to a sqlite sibling across a re-scan', async () => {
await writeLocalScanManifestShards({
project,
connectionId: 'pg_books',
syncId: 'sync1',
driver: 'postgres',
snapshot: booksSnapshot,
dryRun: false,
});
const { content } = await project.fileStore.readFile('semantic-layer/pg_books/_schema/public.yaml');
const shard = YAML.parse(content) as { tables: Record<string, { joins?: Array<{ to: string }> }> };
expect(shard.tables.books?.joins?.map((j) => j.to)).toEqual(['sqlite_reviews.reviews']);
});
});

111
packages/cli/test/context/sl/local-query-federated.integration.test.ts Normal file
View file

@ -0,0 +1,111 @@
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import Database from 'better-sqlite3';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { buildDefaultKtxProjectConfig } from '../../../src/context/project/config.js';
import { executeProjectReadOnlySql } from '../../../src/context/connections/project-sql-executor.js';
import type { GitService } from '../../../src/context/core/git.service.js';
import { LocalGitFileStore } from '../../../src/context/project/local-git-file-store.js';
import type { KtxLocalProject } from '../../../src/context/project/project.js';
import { loadLocalSlSourceRecords } from '../../../src/context/sl/local-sl.js';
const BOOKS_MANIFEST = `tables:
books:
table: main.books
columns:
- name: id
type: number
pk: true
- name: title
type: string
`;
const REVIEWS_MANIFEST = `tables:
reviews:
table: main.reviews
columns:
- name: book_id
type: number
pk: true
- name: stars
type: number
`;
// On-disk file store only (no git init/commit) so manifest seeding never hits
// the gpg-signing path; connections also carry real sqlite paths so the
// federated executor can attach them.
function fakeProject(projectDir: string, connections: KtxLocalProject['config']['connections']): KtxLocalProject {
const fileStore = new LocalGitFileStore({ rootDir: projectDir, git: {} as GitService });
const config = { ...buildDefaultKtxProjectConfig(), connections };
return {
projectDir,
configPath: join(projectDir, 'ktx.yaml'),
config,
coreConfig: {} as KtxLocalProject['coreConfig'],
git: {} as GitService,
fileStore,
};
}
async function seedManifest(project: KtxLocalProject, path: string, content: string): Promise<void> {
await project.fileStore.writeFile(path, content, 'ktx', 'ktx@example.com', 'seed manifest', { skipLock: true });
}
describe('federated SL source loading and physical execution (real DuckDB)', () => {
let tempDir: string;
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-query-fed-'));
});
afterEach(async () => {
await rm(tempDir, { recursive: true, force: true });
});
it('namespaces source names while keeping physical table refs, and executes against them', async () => {
const projectDir = join(tempDir, 'project');
const booksPath = join(tempDir, 'books.db');
const reviewsPath = join(tempDir, 'reviews.db');
const books = new Database(booksPath);
books.exec("CREATE TABLE books (id INTEGER, title TEXT); INSERT INTO books VALUES (1, 'Dune'), (2, 'Foundation');");
books.close();
const reviews = new Database(reviewsPath);
reviews.exec('CREATE TABLE reviews (book_id INTEGER, stars INTEGER); INSERT INTO reviews VALUES (1, 5), (1, 4), (2, 2);');
reviews.close();
const project = fakeProject(projectDir, {
sqlite_books: { driver: 'sqlite', path: booksPath },
sqlite_reviews: { driver: 'sqlite', path: reviewsPath },
});
await seedManifest(project, 'semantic-layer/sqlite_books/_schema/main.yaml', BOOKS_MANIFEST);
await seedManifest(project, 'semantic-layer/sqlite_reviews/_schema/main.yaml', REVIEWS_MANIFEST);
// (a) Name-vs-physical separation: federated loading namespaces source.name
// by member id while source.table stays the unprefixed physical ref.
const records = await loadLocalSlSourceRecords(project, { connectionId: '_ktx_federated' });
const byName = new Map(records.map((record) => [record.source.name, record.source.table]));
expect([...byName.keys()].sort()).toEqual(['sqlite_books.books', 'sqlite_reviews.reviews']);
expect(byName.get('sqlite_books.books')).toBe('main.books');
expect(byName.get('sqlite_reviews.reviews')).toBe('main.reviews');
// (b) Physical targeting end-to-end: a federated query joining the two
// attached catalogs by their connectionId-prefixed physical refs returns
// the correct joined rows through live DuckDB.
const result = await executeProjectReadOnlySql({
project,
input: {
connectionId: '_ktx_federated',
connection: undefined,
sql: 'SELECT b.title, AVG(r.stars) AS avg_stars FROM sqlite_books.books b JOIN sqlite_reviews.reviews r ON b.id = r.book_id GROUP BY b.title ORDER BY b.title',
maxRows: 100,
},
createConnector: () => {
throw new Error('federated path must not create a scan connector');
},
});
expect(result.rows.map((row) => row[0])).toEqual(['Dune', 'Foundation']);
expect(Number(result.rows[0][1])).toBeCloseTo(4.5);
});
});

207
packages/cli/test/context/sl/local-query-federated.test.ts Normal file
View file

@ -0,0 +1,207 @@
import { describe, expect, it, vi } from 'vitest';
import type { KtxSemanticLayerComputePort } from '../../../src/context/daemon/semantic-layer-compute.js';
import type { KtxLocalProject } from '../../../src/context/project/project.js';
import { compileLocalSlQuery } from '../../../src/context/sl/local-query.js';
function makeFakeProject(): KtxLocalProject {
const fileStore = {
listFiles: vi.fn(async () => ({ files: [] })),
readFile: vi.fn(async () => ({ content: '' })),
writeFile: vi.fn(async () => ({})),
deleteFile: vi.fn(async () => ({})),
fileHistory: vi.fn(async () => []),
headCommit: vi.fn(async () => null),
} as unknown as KtxLocalProject['fileStore'];
return {
projectDir: '/tmp/fake-ktx-project',
configPath: '/tmp/fake-ktx-project/ktx.yaml',
config: {
connections: {
pg_books: { driver: 'postgres' },
sqlite_reviews: { driver: 'sqlite' },
},
storage: { state: 'sqlite', search: 'sqlite-fts5', git: {} },
llm: {},
ingest: {},
agent: {},
scan: {},
} as unknown as KtxLocalProject['config'],
coreConfig: {} as KtxLocalProject['coreConfig'],
git: {} as KtxLocalProject['git'],
fileStore,
};
}
function makeFakeProjectWithFiles(
connections: Record<string, { driver: string }>,
files: Record<string, string>,
): KtxLocalProject {
const fileStore = {
listFiles: vi.fn(async (dir: string) => ({
files: Object.keys(files).filter((path) => path.startsWith(`${dir}/`)),
})),
readFile: vi.fn(async (path: string) => ({ content: files[path] ?? '' })),
writeFile: vi.fn(async () => ({})),
deleteFile: vi.fn(async () => ({})),
fileHistory: vi.fn(async () => []),
headCommit: vi.fn(async () => null),
} as unknown as KtxLocalProject['fileStore'];
return {
projectDir: '/tmp/fake-ktx-project',
configPath: '/tmp/fake-ktx-project/ktx.yaml',
config: {
connections,
storage: { state: 'sqlite', search: 'sqlite-fts5', git: {} },
llm: {},
ingest: {},
agent: {},
scan: {},
} as unknown as KtxLocalProject['config'],
coreConfig: {} as KtxLocalProject['coreConfig'],
git: {} as KtxLocalProject['git'],
fileStore,
};
}
function makeFakeCompute(): KtxSemanticLayerComputePort & {
lastDialect: string | undefined;
lastSources: Array<{ name: string; joins?: Array<{ to: string }> }> | undefined;
} {
const fake = {
lastDialect: undefined as string | undefined,
lastSources: undefined as Array<{ name: string; joins?: Array<{ to: string }> }> | undefined,
query: vi.fn(async (input: { dialect: string; query: unknown; sources: unknown[] }) => {
fake.lastDialect = input.dialect;
fake.lastSources = input.sources as Array<{ name: string; joins?: Array<{ to: string }> }>;
return {
sql: 'select 1',
dialect: input.dialect,
columns: [],
plan: { measures: [], dimensions: [] },
};
}),
validateSources: vi.fn(),
generateSources: vi.fn(),
};
return fake;
}
describe('compileLocalSlQuery — federated connection', () => {
it('rejects federated queries and points to raw SQL', async () => {
const project = makeFakeProject();
const compute = makeFakeCompute();
await expect(
compileLocalSlQuery(project, {
connectionId: '_ktx_federated',
query: { measures: [], dimensions: [] },
compute,
execute: false,
}),
).rejects.toThrow(/_ktx_federated[\s\S]*ktx sql/);
// The compute adapter must never be invoked for a federated query.
expect(compute.query).not.toHaveBeenCalled();
});
it('still uses the driver dialect for a normal connection', async () => {
const project = makeFakeProject();
const compute = makeFakeCompute();
await compileLocalSlQuery(project, {
connectionId: 'pg_books',
query: { measures: [], dimensions: [] },
compute,
execute: false,
});
expect(compute.lastDialect).toBe('postgres');
});
it('drops a cross-connection join target so a member query is not poisoned', async () => {
// A preserved cross-DB join (to: sqlite_reviews.reviews) would otherwise be
// an orphan target the planner rejects, breaking every pg_books SL query.
const manifest = `tables:
books:
table: public.books
columns:
- name: id
type: number
pk: true
- name: author_id
type: number
joins:
- to: sqlite_reviews.reviews
on: books.id = reviews.book_id
relationship: one_to_many
- to: authors
on: books.author_id = authors.id
relationship: many_to_one
authors:
table: public.authors
columns:
- name: id
type: number
pk: true
`;
const project = makeFakeProjectWithFiles(
{ pg_books: { driver: 'postgres' }, sqlite_reviews: { driver: 'sqlite' } },
{ 'semantic-layer/pg_books/_schema/public.yaml': manifest },
);
const compute = makeFakeCompute();
await compileLocalSlQuery(project, {
connectionId: 'pg_books',
query: { measures: [], dimensions: [] },
compute,
execute: false,
});
expect(compute.query).toHaveBeenCalledTimes(1);
const books = compute.lastSources?.find((source) => source.name === 'books');
// The same-connection join survives; only the federated-sibling target is dropped.
expect(books?.joins?.map((join) => join.to)).toEqual(['authors']);
});
it('keeps a same-connection join whose target name collides with another connection id', async () => {
// Connection ids and source names share a vocabulary, so a sibling connection
// can be named `authors` while a same-connection source is also `authors`. The
// join target resolves within the connection and must not be pruned.
const manifest = `tables:
books:
table: public.books
columns:
- name: id
type: number
pk: true
- name: author_id
type: number
joins:
- to: authors
on: books.author_id = authors.id
relationship: many_to_one
authors:
table: public.authors
columns:
- name: id
type: number
pk: true
`;
const project = makeFakeProjectWithFiles(
{ pg_books: { driver: 'postgres' }, authors: { driver: 'postgres' } },
{ 'semantic-layer/pg_books/_schema/public.yaml': manifest },
);
const compute = makeFakeCompute();
await compileLocalSlQuery(project, {
connectionId: 'pg_books',
query: { measures: [], dimensions: [] },
compute,
execute: false,
});
const books = compute.lastSources?.find((source) => source.name === 'books');
expect(books?.joins?.map((join) => join.to)).toEqual(['authors']);
});
});

14
packages/cli/test/context/sl/local-query.test.ts
View file

@ -324,7 +324,7 @@ grain: []
).rejects.toThrow('Local semantic-layer execution requires a query executor.');
});
it('requires connectionId when multiple connections are configured', async () => {
it('requires connectionId, listing the configured connections, when several exist', async () => {
project.config.connections.analytics = { driver: 'bigquery' };
await expect(
@ -332,6 +332,16 @@ grain: []
query: { measures: ['orders.order_count'], dimensions: [] },
compute,
}),
).rejects.toThrow('connectionId is required when the local project has zero or multiple connections.');
).rejects.toThrow('connectionId is required. Configured connections: analytics, warehouse.');
});
it('rejects a connectionId that is not configured, listing the configured connections', async () => {
await expect(
compileLocalSlQuery(project, {
connectionId: 'DIG_SMART_REP',
query: { measures: ['orders.order_count'], dimensions: [] },
compute,
}),
).rejects.toThrow('Connection "DIG_SMART_REP" is not configured in ktx.yaml. Configured connections: warehouse.');
});
});

108
packages/cli/test/context/sl/local-sl-federated.test.ts Normal file
View file

@ -0,0 +1,108 @@
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { buildDefaultKtxProjectConfig } from '../../../src/context/project/config.js';
import type { GitService } from '../../../src/context/core/git.service.js';
import { LocalGitFileStore } from '../../../src/context/project/local-git-file-store.js';
import type { KtxLocalProject } from '../../../src/context/project/project.js';
import { loadLocalSlSourceRecords } from '../../../src/context/sl/local-sl.js';
const BOOKS_MANIFEST = `tables:
books:
table: public.books
columns:
- name: book_id
type: number
pk: true
- name: title
type: string
`;
const REVIEWS_MANIFEST = `tables:
reviews:
table: main.reviews
columns:
- name: review_id
type: number
pk: true
- name: rating
type: number
`;
// Build a project backed only by an on-disk file store (no git init, no
// commit), so the fixture never hits the gpg-signing path during init.
function fakeProject(projectDir: string, connections: KtxLocalProject['config']['connections']): KtxLocalProject {
const fileStore = new LocalGitFileStore({ rootDir: projectDir, git: {} as GitService });
const config = { ...buildDefaultKtxProjectConfig(), connections };
return {
projectDir,
configPath: join(projectDir, 'ktx.yaml'),
config,
coreConfig: {} as KtxLocalProject['coreConfig'],
git: {} as GitService,
fileStore,
};
}
// `skipLock: true` writes the file to disk without committing, avoiding git.
async function seedManifest(project: KtxLocalProject, path: string, content: string): Promise<void> {
await project.fileStore.writeFile(path, content, 'ktx', 'ktx@example.com', 'seed manifest', { skipLock: true });
}
describe('federated semantic-layer source loading', () => {
let tempDir: string;
let project: KtxLocalProject;
let singleMemberProject: KtxLocalProject;
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-local-sl-fed-'));
project = fakeProject(join(tempDir, 'project'), {
pg_books: { driver: 'postgres' },
sqlite_reviews: { driver: 'sqlite' },
});
await seedManifest(project, 'semantic-layer/pg_books/_schema/public.yaml', BOOKS_MANIFEST);
await seedManifest(project, 'semantic-layer/sqlite_reviews/_schema/main.yaml', REVIEWS_MANIFEST);
singleMemberProject = fakeProject(join(tempDir, 'single'), {
pg_books: { driver: 'postgres' },
});
await seedManifest(singleMemberProject, 'semantic-layer/pg_books/_schema/public.yaml', BOOKS_MANIFEST);
});
afterEach(async () => {
await rm(tempDir, { recursive: true, force: true });
});
it('namespaces member source records by connection id for _ktx_federated', async () => {
const records = await loadLocalSlSourceRecords(project, { connectionId: '_ktx_federated' });
const names = records.map((r) => r.source.name).sort();
expect(names).toEqual(['pg_books.books', 'sqlite_reviews.reviews']);
});
it('keeps colliding member table names distinct via namespacing', async () => {
const collide = fakeProject(join(tempDir, 'collide'), {
pg_a: { driver: 'postgres' },
sqlite_b: { driver: 'sqlite' },
});
const usersManifest = `tables:\n users:\n table: public.users\n columns:\n - name: id\n type: number\n`;
await seedManifest(collide, 'semantic-layer/pg_a/_schema/public.yaml', usersManifest);
await seedManifest(collide, 'semantic-layer/sqlite_b/_schema/main.yaml', usersManifest);
const records = await loadLocalSlSourceRecords(collide, { connectionId: '_ktx_federated' });
expect(records.map((r) => r.source.name).sort()).toEqual(['pg_a.users', 'sqlite_b.users']);
});
it('tags member records with the virtual federated connection id so reads round-trip', async () => {
const records = await loadLocalSlSourceRecords(project, { connectionId: '_ktx_federated' });
// The federated connection owns no directory and is addressed by one virtual
// id; the member-prefixed names (asserted above) prove the union read from
// member dirs, so the (connectionId, name) pair resolves back via `sl read`.
expect(records.map((r) => r.connectionId)).toEqual(['_ktx_federated', '_ktx_federated']);
});
it('returns empty for _ktx_federated when fewer than 2 compatible members', async () => {
const records = await loadLocalSlSourceRecords(singleMemberProject, { connectionId: '_ktx_federated' });
expect(records).toEqual([]);
});
});

22
packages/cli/test/context/sl/source-files-reserved.test.ts Normal file
View file

@ -0,0 +1,22 @@
import { describe, expect, it } from 'vitest';
import { assertSafeConnectionId, isReservedConnectionId } from '../../../src/context/sl/source-files.js';
describe('reserved connection ids', () => {
it('flags _ktx_ prefixed ids as reserved', () => {
expect(isReservedConnectionId('_ktx_federated')).toBe(true);
expect(isReservedConnectionId('_ktx_anything')).toBe(true);
});
it('does not flag normal ids', () => {
expect(isReservedConnectionId('pg_books')).toBe(false);
expect(isReservedConnectionId('sqlite_reviews')).toBe(false);
});
it('rejects a user-supplied reserved id', () => {
expect(() => assertSafeConnectionId('_ktx_federated')).toThrow(/reserved/i);
});
it('still accepts normal ids', () => {
expect(assertSafeConnectionId('pg_books')).toBe('pg_books');
});
});

4
packages/cli/test/doctor.test.ts
View file

@ -457,6 +457,8 @@ describe('runKtxDoctor', () => {
' backend: openai',
' model: text-embedding-3-small',
' dimensions: 1536',
' openai:',
' api_key: env:OPENAI_API_KEY', // pragma: allowlist secret
'',
].join('\n'),
'utf-8',
@ -592,6 +594,8 @@ describe('runKtxDoctor', () => {
' backend: openai',
' model: text-embedding-3-small',
' dimensions: 1536',
' openai:',
' api_key: env:OPENAI_API_KEY', // pragma: allowlist secret
'',
].join('\n'),
'utf-8',

89
packages/cli/test/index.test.ts
View file

@ -526,6 +526,7 @@ describe('runKtxCli', () => {
expect(stdout).toContain('--target <target>');
expect(stdout).toContain('--global');
expect(stdout).toContain('--local');
expect(stdout).toContain('--install-dir <path>');
expect(stdout).toContain('--yes');
expect(stdout).toContain('--no-input');
expect(stdout).toContain('Global Options:');
@ -1486,6 +1487,94 @@ describe('runKtxCli', () => {
expect(setup).not.toHaveBeenCalled();
});
it('dispatches --install-dir as the agent install root', async () => {
const setup = vi.fn(async () => 0);
const setupIo = makeIo();
await expect(
runKtxCli(
['--project-dir', tempDir, 'setup', '--agents', '--target', 'claude-code', '--install-dir', '.', '--no-input'],
setupIo.io,
{ setup },
),
).resolves.toBe(0);
expect(setup).toHaveBeenCalledWith(
expect.objectContaining({ agents: true, target: 'claude-code', agentScope: 'project', installRoot: '.' }),
setupIo.io,
);
});
it('rejects --install-dir together with --global', async () => {
const setup = vi.fn(async () => 0);
const setupIo = makeIo();
await expect(
runKtxCli(
['--project-dir', tempDir, 'setup', '--agents', '--target', 'claude-code', '--install-dir', '.', '--global', '--no-input'],
setupIo.io,
{ setup },
),
).resolves.toBe(1);
expect(setupIo.stderr()).toContain('Choose either --install-dir or a scope flag (--global / --local), not both.');
expect(setup).not.toHaveBeenCalled();
});
it('rejects --install-dir together with --local', async () => {
const setup = vi.fn(async () => 0);
const setupIo = makeIo();
await expect(
runKtxCli(
['--project-dir', tempDir, 'setup', '--agents', '--target', 'claude-code', '--install-dir', '.', '--local', '--no-input'],
setupIo.io,
{ setup },
),
).resolves.toBe(1);
expect(setupIo.stderr()).toContain('Choose either --install-dir or a scope flag (--global / --local), not both.');
expect(setup).not.toHaveBeenCalled();
});
it('treats an empty --install-dir as not provided', async () => {
const setup = vi.fn(async () => 0);
const setupIo = makeIo();
await expect(
runKtxCli(
['--project-dir', tempDir, 'setup', '--agents', '--target', 'claude-code', '--install-dir', '', '--global', '--no-input'],
setupIo.io,
{ setup },
),
).resolves.toBe(0);
expect(setup).toHaveBeenCalledWith(
expect.objectContaining({ agents: true, target: 'claude-code', agentScope: 'global' }),
setupIo.io,
);
expect(setup).toHaveBeenCalledWith(
expect.not.objectContaining({ installRoot: expect.anything() }),
setupIo.io,
);
});
it('rejects --install-dir with --target claude-desktop', async () => {
const setup = vi.fn(async () => 0);
const setupIo = makeIo();
await expect(
runKtxCli(
['--project-dir', tempDir, 'setup', '--agents', '--target', 'claude-desktop', '--install-dir', '.', '--no-input'],
setupIo.io,
{ setup },
),
).resolves.toBe(1);
expect(setupIo.stderr()).toContain('--install-dir does not apply to --target claude-desktop, which is always global.');
expect(setup).not.toHaveBeenCalled();
});
it('rejects source-path with source-git-url', async () => {
const setup = vi.fn(async () => 0);
const testIo = makeIo();

36
packages/cli/test/ingest-query-executor-federated.test.ts Normal file
View file

@ -0,0 +1,36 @@
import { describe, expect, it, vi } from 'vitest';
import { createKtxCliIngestQueryExecutor } from '../src/ingest-query-executor.js';
describe('federated query executor routing', () => {
it('routes _ktx_federated to the DuckDB federated executor, not a single connector', async () => {
const project = {
projectDir: '/tmp/x',
config: { connections: { pg: { driver: 'postgres', url: 'env:PG' }, lite: { driver: 'sqlite', url: '/x.db' } } },
} as never;
const federatedSpy = vi.fn(async () => ({
headers: ['n'], rows: [[1]], totalRows: 1, command: 'SELECT', rowCount: 1,
}));
const executor = createKtxCliIngestQueryExecutor(project, { executeFederated: federatedSpy });
const result = await executor.execute({
connectionId: '_ktx_federated',
connection: undefined,
sql: 'select 1 as n',
});
expect(federatedSpy).toHaveBeenCalledOnce();
expect(result.totalRows).toBe(1);
});
it('throws if _ktx_federated requested but fewer than 2 compatible members', async () => {
const project = {
projectDir: '/tmp/x',
config: { connections: { pg: { driver: 'postgres', url: 'env:PG' } } },
} as never;
const executor = createKtxCliIngestQueryExecutor(project, { executeFederated: vi.fn() });
await expect(
executor.execute({ connectionId: '_ktx_federated', connection: undefined, sql: 'select 1' }),
).rejects.toThrow(/2 attach-compatible/i);
});
});

263
packages/cli/test/managed-python-command.test.ts
View file

@ -1,5 +1,7 @@
import { describe, expect, it, vi } from 'vitest';
import { KtxExpectedError } from '../src/errors.js';
import {
createLazyManagedPythonSemanticLayerComputePort,
createManagedPythonSemanticLayerComputePort,
ensureManagedPythonCommandRuntime,
managedRuntimeInstallCommand,
@ -186,111 +188,64 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
]);
});
it('does not touch the runtime when the port is only constructed', () => {
const io = makeIo();
const readStatus = vi.fn(async () => readyStatus());
const installRuntime = vi.fn(async () => installResult());
const createPythonCompute = vi.fn(() => ({ query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }));
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'auto',
io: io.io,
readStatus,
installRuntime,
createPythonCompute,
});
expect(readStatus).not.toHaveBeenCalled();
expect(installRuntime).not.toHaveBeenCalled();
expect(createPythonCompute).not.toHaveBeenCalled();
expect(io.stderr()).toBe('');
});
it('uses the managed ktx-daemon executable on the first compute call', async () => {
it('uses the managed ktx-daemon executable when the runtime is ready', async () => {
const io = makeIo();
const compute = { query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() };
const createPythonCompute = vi.fn(() => compute);
const port = createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'never',
io: io.io,
readStatus: vi.fn(async () => readyStatus()),
installRuntime: vi.fn(),
createPythonCompute,
});
await port.validateSources({ sources: [], dialect: 'postgres' });
await expect(
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'never',
io: io.io,
readStatus: vi.fn(async () => readyStatus()),
installRuntime: vi.fn(),
createPythonCompute,
}),
).resolves.toBe(compute);
expect(createPythonCompute).toHaveBeenCalledWith({
command: '/runtime/0.2.0/.venv/bin/ktx-daemon',
args: [],
});
expect(compute.validateSources).toHaveBeenCalledWith({ sources: [], dialect: 'postgres' });
expect(io.stderr()).toBe('');
});
it('resolves the runtime once across multiple compute calls', async () => {
const io = makeIo();
const readStatus = vi.fn(async () => readyStatus());
const createPythonCompute = vi.fn(() => ({ query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }));
const port = createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'never',
io: io.io,
readStatus,
installRuntime: vi.fn(),
createPythonCompute,
});
await port.validateSources({ sources: [], dialect: 'postgres' });
await port.validateSources({ sources: [], dialect: 'postgres' });
expect(readStatus).toHaveBeenCalledTimes(1);
expect(createPythonCompute).toHaveBeenCalledTimes(1);
});
it('fails with a preparation command on first use when input is disabled and the runtime is missing', async () => {
it('fails with a preparation command when input is disabled and the runtime is missing', async () => {
const io = makeIo();
const installRuntime = vi.fn();
const port = createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'never',
io: io.io,
readStatus: vi.fn(async () => missingStatus()),
installRuntime,
});
await expect(port.validateSources({ sources: [], dialect: 'postgres' })).rejects.toThrow(
'ktx Python runtime is required for this command. Run: ktx admin runtime install --yes',
);
await expect(
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'never',
io: io.io,
readStatus: vi.fn(async () => missingStatus()),
installRuntime,
}),
).rejects.toThrow('ktx Python runtime is required for this command. Run: ktx admin runtime install --yes');
expect(installRuntime).not.toHaveBeenCalled();
});
it('installs the core runtime without prompting on first use when policy is auto', async () => {
it('installs the core runtime without prompting when policy is auto', async () => {
const io = makeIo();
const { events, spinner } = makeSpinnerEvents();
const compute = { query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() };
const createPythonCompute = vi.fn(() => compute);
const installRuntime = vi.fn(async () => installResult());
const port = createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'auto',
io: io.io,
readStatus: vi.fn(async () => missingStatus()),
installRuntime,
createPythonCompute,
spinner,
});
expect(installRuntime).not.toHaveBeenCalled();
await port.validateSources({ sources: [], dialect: 'postgres' });
await expect(
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'auto',
io: io.io,
readStatus: vi.fn(async () => missingStatus()),
installRuntime,
createPythonCompute,
spinner,
}),
).resolves.toBe(compute);
expect(installRuntime).toHaveBeenCalledWith({
cliVersion: '0.2.0',
@ -303,13 +258,13 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
]);
});
it('prompts before installing on first use when policy is prompt', async () => {
it('prompts before installing when policy is prompt', async () => {
const io = makeIo();
const { events, spinner } = makeSpinnerEvents();
const confirmInstall = vi.fn(async () => true);
const installRuntime = vi.fn(async () => installResult());
const port = createManagedPythonSemanticLayerComputePort({
await createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'prompt',
io: io.io,
@ -320,10 +275,8 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
spinner,
});
await port.validateSources({ sources: [], dialect: 'postgres' });
expect(confirmInstall).toHaveBeenCalledWith(
'ktx needs to install the core Python runtime. This downloads Python dependencies with uv. Continue?',
'ktx needs to install the core Python runtime. This downloads a pinned, checksum-verified uv build, Python, and dependencies. Continue?',
io.io,
);
expect(installRuntime).toHaveBeenCalledWith({
@ -341,21 +294,21 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
const installRuntime = vi.fn(async (): Promise<ManagedPythonRuntimeInstallResult> => installResult());
const confirmInstall = vi.fn(async () => true);
const port = createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'prompt',
io: io.io,
readStatus: async () => missingStatus(),
installRuntime,
confirmInstall,
createPythonCompute: () => compute,
spinner,
});
await port.validateSources({ sources: [], dialect: 'postgres' });
await expect(
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'prompt',
io: io.io,
readStatus: async () => missingStatus(),
installRuntime,
confirmInstall,
createPythonCompute: () => compute,
spinner,
}),
).resolves.toBe(compute);
expect(confirmInstall).toHaveBeenCalledWith(
'ktx needs to install the core Python runtime. This downloads Python dependencies with uv. Continue?',
'ktx needs to install the core Python runtime. This downloads a pinned, checksum-verified uv build, Python, and dependencies. Continue?',
io.io,
);
expect(events).toContainEqual('start:Installing ktx Python runtime (core) with uv...');
@ -365,17 +318,107 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
const io = makeIo();
Object.assign(io.io.stdout, { isTTY: false });
const port = createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'prompt',
io: io.io,
readStatus: async () => missingStatus(),
installRuntime: vi.fn(),
createPythonCompute: () => ({ query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }),
});
await expect(port.validateSources({ sources: [], dialect: 'postgres' })).rejects.toThrow(
'ktx Python runtime installation was cancelled',
);
await expect(
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'prompt',
io: io.io,
readStatus: async () => missingStatus(),
installRuntime: vi.fn(),
createPythonCompute: () => ({ query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }),
}),
).rejects.toThrow('ktx Python runtime installation was cancelled');
});
});
describe('createLazyManagedPythonSemanticLayerComputePort', () => {
it('does not touch the managed runtime at construction, so a server starts without uv', async () => {
const io = makeIo();
const readStatus = vi.fn(async () => missingStatus());
const installRuntime = vi.fn(async (): Promise<ManagedPythonRuntimeInstallResult> => {
throw new KtxExpectedError('uv missing');
});
const createPythonCompute = vi.fn(() => ({ query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }));
const port = createLazyManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'auto',
io: io.io,
readStatus,
installRuntime,
createPythonCompute,
});
expect(readStatus).not.toHaveBeenCalled();
expect(installRuntime).not.toHaveBeenCalled();
expect(createPythonCompute).not.toHaveBeenCalled();
await expect(port.query({ sources: [], query: {} as never, dialect: 'postgres' })).rejects.toBeInstanceOf(
KtxExpectedError,
);
expect(installRuntime).toHaveBeenCalledTimes(1);
});
it('resolves the runtime once and reuses it across calls', async () => {
const io = makeIo();
const readStatus = vi.fn(async () => readyStatus());
const compute = {
query: vi.fn(),
validateSources: vi.fn(),
generateSources: vi.fn(),
};
const createPythonCompute = vi.fn(() => compute);
const port = createLazyManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'never',
io: io.io,
readStatus,
installRuntime: vi.fn(),
createPythonCompute,
});
await port.query({ sources: [], query: {} as never, dialect: 'postgres' });
await port.validateSources({ sources: [], dialect: 'postgres' });
expect(readStatus).toHaveBeenCalledTimes(1);
expect(createPythonCompute).toHaveBeenCalledTimes(1);
expect(compute.query).toHaveBeenCalledTimes(1);
expect(compute.validateSources).toHaveBeenCalledTimes(1);
});
it('retries the runtime resolution after a failed attempt', async () => {
const io = makeIo();
const compute = {
query: vi.fn(),
validateSources: vi.fn(),
generateSources: vi.fn(),
};
const createPythonCompute = vi.fn(() => compute);
let attempt = 0;
const installRuntime = vi.fn(async (): Promise<ManagedPythonRuntimeInstallResult> => {
attempt += 1;
if (attempt === 1) {
throw new KtxExpectedError('uv missing');
}
return installResult();
});
const port = createLazyManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'auto',
io: io.io,
readStatus: vi.fn(async () => missingStatus()),
installRuntime,
createPythonCompute,
});
await expect(port.query({ sources: [], query: {} as never, dialect: 'postgres' })).rejects.toBeInstanceOf(
KtxExpectedError,
);
await port.query({ sources: [], query: {} as never, dialect: 'postgres' });
expect(installRuntime).toHaveBeenCalledTimes(2);
expect(compute.query).toHaveBeenCalledTimes(1);
});
});

265
packages/cli/test/managed-python-runtime.test.ts
View file

@ -1,19 +1,61 @@
import { createHash } from 'node:crypto';
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { strToU8, zipSync } from 'fflate';
import { dirname, join } from 'node:path';
import { gzipSync, strToU8, zipSync } from 'fflate';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { KtxExpectedError } from '../src/errors.js';
import {
MISSING_UV_RUNTIME_INSTALL_MESSAGE,
doctorManagedPythonRuntime,
ensureManagedUv,
installManagedPythonRuntime,
managedPythonDaemonLayout,
managedPythonRuntimeLayout,
managedUvPath,
readManagedPythonRuntimeStatus,
verifyRuntimeAsset,
type ManagedPythonRuntimeExec,
type ManagedUvRelease,
} from '../src/managed-python-runtime.js';
import type { ManagedUvPlatformKey } from '../src/managed-uv-release.js';
async function placeFakeUv(runtimeRoot: string): Promise<string> {
const uvPath = managedUvPath({ runtimeRoot });
await mkdir(dirname(uvPath), { recursive: true });
await writeFile(uvPath, '#!/bin/sh\n');
return uvPath;
}
function tarball(entries: Record<string, Uint8Array>): Uint8Array {
const blocks: Uint8Array[] = [];
for (const [name, data] of Object.entries(entries)) {
const header = new Uint8Array(512);
header.set(strToU8(name), 0);
header.set(strToU8('0000755\0'), 100);
header.set(strToU8(`${data.length.toString(8).padStart(11, '0')}\0`), 124);
blocks.push(header);
const padded = new Uint8Array(Math.ceil(data.length / 512) * 512);
padded.set(data);
blocks.push(padded);
}
blocks.push(new Uint8Array(1024));
const out = new Uint8Array(blocks.reduce((total, block) => total + block.length, 0));
let offset = 0;
for (const block of blocks) {
out.set(block, offset);
offset += block.length;
}
return out;
}
function releaseFor(file: string, contents: Uint8Array, key: ManagedUvPlatformKey): ManagedUvRelease {
return {
version: '9.9.9-test',
artifacts: {
[key]: { file, sha256: createHash('sha256').update(contents).digest('hex') },
},
};
}
function runtimeWheelContents(input: { label?: string; requiresPython?: string | null } = {}): Buffer {
const label = input.label ?? 'runtime-wheel';
@ -246,10 +288,11 @@ describe('installManagedPythonRuntime', () => {
it('creates a venv, installs the core wheel, and writes a manifest', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const commands: Array<{ command: string; args: string[] }> = [];
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => {
commands.push({ command, args });
return { stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.9.5\n' : '', stderr: '' };
return { stdout: command === uvPath && args[0] === '--version' ? 'uv 0.9.5\n' : '', stderr: '' };
});
const result = await installManagedPythonRuntime({
@ -262,11 +305,11 @@ describe('installManagedPythonRuntime', () => {
expect(result.status).toBe('installed');
expect(commands).toEqual([
{ command: 'uv', args: ['--version'] },
{ command: 'uv', args: ['python', 'install', '3.13'] },
{ command: 'uv', args: ['venv', '--python', '3.13', result.layout.venvDir] },
{ command: uvPath, args: ['--version'] },
{ command: uvPath, args: ['python', 'install', '3.13'] },
{ command: uvPath, args: ['venv', '--python', '3.13', result.layout.venvDir] },
{
command: 'uv',
command: uvPath,
args: ['pip', 'install', '--python', result.layout.pythonPath, result.asset.wheelPath],
},
]);
@ -283,10 +326,11 @@ describe('installManagedPythonRuntime', () => {
it('disables repo uv config for managed runtime uv commands', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const commands: Array<{ command: string; args: string[]; env?: NodeJS.ProcessEnv }> = [];
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args, options) => {
commands.push({ command, args, env: options?.env });
return { stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.11.13\n' : '', stderr: '' };
return { stdout: command === uvPath && args[0] === '--version' ? 'uv 0.11.13\n' : '', stderr: '' };
});
await installManagedPythonRuntime({
@ -299,19 +343,20 @@ describe('installManagedPythonRuntime', () => {
});
expect(commands.map((call) => [call.command, call.args[0], call.env?.UV_NO_CONFIG, call.env?.PATH])).toEqual([
['uv', '--version', '1', '/opt/homebrew/bin'],
['uv', 'python', '1', '/opt/homebrew/bin'],
['uv', 'venv', '1', '/opt/homebrew/bin'],
['uv', 'pip', '1', '/opt/homebrew/bin'],
[uvPath, '--version', '1', '/opt/homebrew/bin'],
[uvPath, 'python', '1', '/opt/homebrew/bin'],
[uvPath, 'venv', '1', '/opt/homebrew/bin'],
[uvPath, 'pip', '1', '/opt/homebrew/bin'],
]);
});
it('installs the local-embeddings extra when requested', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'embedding-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const commands: Array<{ command: string; args: string[] }> = [];
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => {
commands.push({ command, args });
return { stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.9.5\n' : '', stderr: '' };
return { stdout: command === uvPath && args[0] === '--version' ? 'uv 0.9.5\n' : '', stderr: '' };
});
const result = await installManagedPythonRuntime({
@ -323,38 +368,72 @@ describe('installManagedPythonRuntime', () => {
});
expect(commands.at(-1)).toEqual({
command: 'uv',
command: uvPath,
args: ['pip', 'install', '--python', result.layout.pythonPath, `${result.asset.wheelPath}[local-embeddings]`],
});
const manifest = JSON.parse(await readFile(result.layout.manifestPath, 'utf8')) as { features: string[] };
expect(manifest.features).toEqual(['core', 'local-embeddings']);
});
it('fails with the hard-prerequisite message when uv is missing', async () => {
it('attempts the pinned uv download from github.com and rejects checksum mismatches', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const commands: Array<{ command: string; args: string[] }> = [];
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => {
commands.push({ command, args });
throw new Error('spawn uv ENOENT');
const runtimeRoot = join(tempDir, 'runtime');
const archive = gzipSync(tarball({ 'uv-test/uv': strToU8('#!/bin/sh\necho uv\n') }));
const fetchUvArtifact = vi.fn(async () => archive);
const exec: ManagedPythonRuntimeExec = vi.fn(async () => ({ stdout: 'uv 9.9.9\n', stderr: '' }));
const error = await installManagedPythonRuntime({
cliVersion: '0.2.0',
runtimeRoot,
assetDir,
features: ['core'],
exec,
fetchUvArtifact,
}).catch((caught: unknown) => caught);
expect(fetchUvArtifact).toHaveBeenCalledTimes(1);
expect(fetchUvArtifact).toHaveBeenCalledWith(
expect.stringMatching(/^https:\/\/github\.com\/astral-sh\/uv\/releases\/download\//),
);
expect(error).toBeInstanceOf(KtxExpectedError);
expect((error as Error).message).toContain('failed checksum verification');
expect(exec).not.toHaveBeenCalled();
});
it('fails with download guidance and preserves the existing runtime when uv cannot be fetched', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const runtimeRoot = join(tempDir, 'runtime');
const exec: ManagedPythonRuntimeExec = vi.fn(async () => ({ stdout: '', stderr: '' }));
const fetchUvArtifact = vi.fn(async () => {
throw new Error('getaddrinfo ENOTFOUND github.com');
});
const survivingRuntimeFile = join(runtimeRoot, '0.2.0', 'install.log');
await mkdir(dirname(survivingRuntimeFile), { recursive: true });
await writeFile(survivingRuntimeFile, 'stale runtime contents\n');
await expect(
installManagedPythonRuntime({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
assetDir,
features: ['core'],
exec,
}),
).rejects.toThrow(MISSING_UV_RUNTIME_INSTALL_MESSAGE);
const error = await installManagedPythonRuntime({
cliVersion: '0.2.0',
runtimeRoot,
assetDir,
features: ['core'],
exec,
fetchUvArtifact,
}).catch((caught: unknown) => caught);
expect(commands).toEqual([{ command: 'uv', args: ['--version'] }]);
// KtxExpectedError keeps this user-environment outcome out of Error Tracking.
expect(error).toBeInstanceOf(KtxExpectedError);
expect((error as Error).message).toContain('could not download uv');
expect((error as Error).message).toContain('ktx admin runtime install --yes');
expect(exec).not.toHaveBeenCalled();
// A failed uv acquisition must not wipe whatever runtime is already on disk.
await expect(readFile(survivingRuntimeFile, 'utf8')).resolves.toContain('stale');
});
it('reuses an existing compatible runtime when force is false', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => ({
stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stdout: command === uvPath && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stderr: '',
}));
@ -383,14 +462,15 @@ describe('installManagedPythonRuntime', () => {
it('keeps failed install logs in the versioned runtime directory', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => {
if (command === 'uv' && args[0] === 'venv') {
if (command === uvPath && args[0] === 'venv') {
throw Object.assign(new Error('uv venv failed'), {
stdout: 'creating\n',
stderr: '× No solution found\n╰─▶ current Python version (3.12.3) does not satisfy Python>=3.13\n',
});
}
return { stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.9.5\n' : '', stderr: '' };
return { stdout: command === uvPath && args[0] === '--version' ? 'uv 0.9.5\n' : '', stderr: '' };
});
await expect(
@ -404,11 +484,98 @@ describe('installManagedPythonRuntime', () => {
).rejects.toThrow(/current Python version \(3\.12\.3\) does not satisfy Python>=3\.13/);
const log = await readFile(join(tempDir, 'runtime', '0.2.0', 'install.log'), 'utf8');
expect(log).toContain('$ uv venv --python 3.13');
expect(log).toContain(`$ ${uvPath} venv --python 3.13`);
expect(log).toContain('current Python version (3.12.3) does not satisfy Python>=3.13');
});
});
describe('ensureManagedUv', () => {
let tempDir: string;
beforeEach(async () => {
tempDir = await mkdtemp(join(tmpdir(), 'ktx-managed-uv-'));
});
afterEach(async () => {
await rm(tempDir, { recursive: true, force: true });
});
it('downloads, verifies, and extracts uv from a tar.gz artifact, then reuses the cached binary', async () => {
const binary = strToU8('#!/bin/sh\necho uv\n');
const archive = gzipSync(tarball({ 'uv-test/': new Uint8Array(0), 'uv-test/uvx': strToU8('x'), 'uv-test/uv': binary }));
const release = releaseFor('uv-test.tar.gz', archive, 'linux-x64');
const fetchArtifact = vi.fn(async () => archive);
const uvPath = await ensureManagedUv({
platform: 'linux',
arch: 'x64',
runtimeRoot: join(tempDir, 'runtime'),
fetchArtifact,
release,
});
expect(uvPath).toBe(join(tempDir, 'runtime', 'uv', '9.9.9-test', 'uv'));
await expect(readFile(uvPath, 'utf8')).resolves.toBe('#!/bin/sh\necho uv\n');
const again = await ensureManagedUv({
platform: 'linux',
arch: 'x64',
runtimeRoot: join(tempDir, 'runtime'),
fetchArtifact,
release,
});
expect(again).toBe(uvPath);
expect(fetchArtifact).toHaveBeenCalledTimes(1);
});
it('extracts uv.exe from a zip artifact on Windows', async () => {
const archive = zipSync({ 'uv.exe': strToU8('MZ-uv'), 'uvx.exe': strToU8('MZ-uvx') });
const release = releaseFor('uv-test.zip', archive, 'win32-x64');
const uvPath = await ensureManagedUv({
platform: 'win32',
arch: 'x64',
runtimeRoot: join(tempDir, 'runtime'),
fetchArtifact: vi.fn(async () => archive),
release,
});
expect(uvPath).toBe(join(tempDir, 'runtime', 'uv', '9.9.9-test', 'uv.exe'));
await expect(readFile(uvPath, 'utf8')).resolves.toBe('MZ-uv');
});
it('rejects an artifact whose checksum does not match the pin', async () => {
const archive = gzipSync(tarball({ 'uv-test/uv': strToU8('uv') }));
const release = releaseFor('uv-test.tar.gz', archive, 'linux-x64');
release.artifacts['linux-x64']!.sha256 = 'b'.repeat(64);
const error = await ensureManagedUv({
platform: 'linux',
arch: 'x64',
runtimeRoot: join(tempDir, 'runtime'),
fetchArtifact: vi.fn(async () => archive),
release,
}).catch((caught: unknown) => caught);
expect(error).toBeInstanceOf(KtxExpectedError);
expect((error as Error).message).toContain('failed checksum verification');
});
it('fails with manual-placement guidance on platforms without a pinned artifact', async () => {
const error = await ensureManagedUv({
platform: 'sunos',
arch: 'x64',
runtimeRoot: join(tempDir, 'runtime'),
fetchArtifact: vi.fn(),
release: { version: '9.9.9-test', artifacts: {} },
}).catch((caught: unknown) => caught);
expect(error).toBeInstanceOf(KtxExpectedError);
expect((error as Error).message).toContain('does not bundle uv for sunos-x64');
expect((error as Error).message).toContain(join(tempDir, 'runtime', 'uv', '9.9.9-test', 'uv'));
});
});
describe('readManagedPythonRuntimeStatus', () => {
let tempDir: string;
@ -433,8 +600,9 @@ describe('readManagedPythonRuntimeStatus', () => {
it('reports ready when manifest and executables exist', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => ({
stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stdout: command === uvPath && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stderr: '',
}));
const install = await installManagedPythonRuntime({
@ -460,8 +628,9 @@ describe('readManagedPythonRuntimeStatus', () => {
it('reports broken when an executable is missing', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => ({
stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stdout: command === uvPath && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stderr: '',
}));
await installManagedPythonRuntime({
@ -496,8 +665,9 @@ describe('doctorManagedPythonRuntime', () => {
it('checks uv, bundled assets, and installed runtime status', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const uvPath = await placeFakeUv(join(tempDir, 'runtime'));
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args) => ({
stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stdout: command === uvPath && args[0] === '--version' ? 'uv 0.9.5\n' : '',
stderr: '',
}));
@ -513,28 +683,27 @@ describe('doctorManagedPythonRuntime', () => {
['asset', 'pass'],
['runtime', 'fail'],
]);
expect(checks[0]?.detail).toBe(`uv 0.9.5 (managed: ${uvPath})`);
expect(checks[2]?.fix).toBe('Run: ktx admin runtime install --yes');
});
it('reports uv as a hard prerequisite when uv is missing', async () => {
it('fails the uv check with download guidance when uv cannot be acquired', async () => {
const { assetDir } = await writeAsset(tempDir, { label: 'core-wheel' });
const exec: ManagedPythonRuntimeExec = vi.fn(async () => {
throw new Error('spawn uv ENOENT');
});
const exec: ManagedPythonRuntimeExec = vi.fn(async () => ({ stdout: '', stderr: '' }));
const checks = await doctorManagedPythonRuntime({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
assetDir,
exec,
fetchUvArtifact: vi.fn(async () => {
throw new Error('getaddrinfo ENOTFOUND github.com');
}),
});
expect(checks[0]).toEqual({
id: 'uv',
label: 'uv',
status: 'fail',
detail: MISSING_UV_RUNTIME_INSTALL_MESSAGE,
fix: 'Install uv, make sure it is on PATH, and run: ktx admin runtime install --yes',
});
expect(checks[0]?.id).toBe('uv');
expect(checks[0]?.status).toBe('fail');
expect(checks[0]?.detail).toContain('could not download uv');
expect(checks[0]?.fix).toBe('Check network access to github.com and run: ktx admin runtime install --yes');
});
});

10
packages/cli/test/mcp-server-factory.test.ts
View file

@ -4,6 +4,7 @@ import { createLocalProjectMcpContextPorts } from '../src/context/mcp/local-proj
import { createLocalProjectMemoryIngest } from '../src/context/memory/local-memory.js';
import { resolveProjectEmbeddingProvider } from '../src/embedding-resolution.js';
import { createKtxCliScanConnector } from '../src/local-scan-connectors.js';
import { createLazyManagedPythonSemanticLayerComputePort } from '../src/managed-python-command.js';
import { createKtxMcpServerFactory } from '../src/mcp-server-factory.js';
type FakeEmbeddingProvider = {
@ -62,7 +63,7 @@ vi.mock('../src/local-scan-connectors.js', () => ({
}));
vi.mock('../src/managed-python-command.js', () => ({
createManagedPythonSemanticLayerComputePort: vi.fn(() => mocks.semanticLayerCompute),
createLazyManagedPythonSemanticLayerComputePort: vi.fn(() => mocks.semanticLayerCompute),
}));
vi.mock('../src/managed-python-http.js', () => ({
@ -124,6 +125,13 @@ describe('createKtxMcpServerFactory', () => {
expect(provider.embed).toHaveBeenCalledWith('gross revenue');
expect(provider.embedMany).toHaveBeenCalledWith(['gross revenue']);
expect(createKtxCliScanConnector).toHaveBeenCalledWith(project, 'warehouse');
// The server must wire the lazy compute port so startup never blocks on (or
// fails over) a missing managed Python runtime / uv.
expect(createLazyManagedPythonSemanticLayerComputePort).toHaveBeenCalledWith({
cliVersion: '0.5.0',
installPolicy: 'auto',
io,
});
expect(contextOptions).toMatchObject({
queryExecutor: mocks.queryExecutor,
semanticLayerCompute: mocks.semanticLayerCompute,

362
packages/cli/test/setup-agents.test.ts
View file

@ -383,6 +383,7 @@ describe('setup agents', () => {
const prompts = {
select: vi.fn(async ({ message }: { message: string }) => (message.startsWith('Where') ? 'project' : 'mcp')),
multiselect: vi.fn(async () => ['claude-code']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
@ -439,6 +440,7 @@ describe('setup agents', () => {
multiselect: vi.fn(async () => {
throw new Error('target selection should not run');
}),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
@ -495,6 +497,7 @@ describe('setup agents', () => {
message.startsWith('Where should') ? 'global' : 'mcp',
),
multiselect: vi.fn(async () => ['claude-code']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
@ -508,6 +511,7 @@ describe('setup agents', () => {
scope: 'project',
mode: 'mcp',
skipAgents: false,
cwd: tempDir,
},
io.io,
{ prompts },
@ -518,13 +522,10 @@ describe('setup agents', () => {
});
expect(prompts.select).toHaveBeenCalledWith({
message: `Where should ktx install supported agent config?\n\nktx project: ${tempDir}`,
message: `Where should ktx install agent config?\n\nktx project: ${tempDir}`,
options: [
{
value: 'project',
label: 'Project scope (ktx project directory)',
hint: 'Only agents opened from this ktx project path load the project-scoped config.',
},
{ value: 'project', label: 'ktx project directory', hint: tempDir },
{ value: 'custom', label: 'Custom directory…', hint: 'Enter a path' },
{
value: 'global',
label: 'Global scope (user config)',
@ -978,6 +979,7 @@ describe('setup agents', () => {
const prompts = {
select: vi.fn(async () => 'back'),
multiselect: vi.fn(async () => ['codex']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
@ -1003,6 +1005,7 @@ describe('setup agents', () => {
const prompts = {
select: vi.fn(async () => 'mcp-cli'),
multiselect: vi.fn(async () => ['back']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
@ -1136,6 +1139,7 @@ describe('setup agents', () => {
message.startsWith('Where should') ? 'project' : 'mcp',
),
multiselect: vi.fn(async () => ['claude-code', 'claude-desktop']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
@ -1228,8 +1232,11 @@ describe('setup agents', () => {
it('explains next actions for Codex, Cursor, OpenCode, and universal MCP targets', async () => {
const io = makeIo();
const prompts = {
select: vi.fn(async () => 'mcp-cli'),
select: vi.fn(async ({ message }: { message: string }) =>
message.startsWith('Where') ? 'project' : 'mcp-cli',
),
multiselect: vi.fn(async () => ['codex', 'cursor', 'opencode', 'universal']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
@ -1274,6 +1281,347 @@ describe('setup agents', () => {
expect(output).toContain('.agents guidance installed');
});
describe('install root', () => {
it('plans project-scoped files under installRoot, leaving projectDir as the default', () => {
const installRoot = join(tempDir, 'opened-here');
expect(
plannedKtxAgentFiles({
projectDir: tempDir,
installRoot,
target: 'claude-code',
scope: 'project',
mode: 'mcp-cli',
}),
).toEqual([
{ kind: 'file', path: join(installRoot, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' },
{ kind: 'file', path: join(installRoot, '.claude/skills/ktx/SKILL.md'), role: 'skill' },
{ kind: 'file', path: join(installRoot, '.claude/rules/ktx.md'), role: 'rule' },
]);
expect(
plannedKtxAgentFiles({ projectDir: tempDir, target: 'claude-code', scope: 'project', mode: 'mcp' }),
).toEqual([
{ kind: 'file', path: join(tempDir, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' },
]);
});
it('shows the install path in the summary title only when installRoot differs from projectDir', () => {
const installRoot = join(tempDir, 'app');
const custom = formatInstallSummaryLines(
[{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot }],
[
{ kind: 'file', path: join(installRoot, '.claude/skills/ktx-analytics/SKILL.md'), role: 'analytics-skill' },
{ kind: 'json-key', path: join(installRoot, '.mcp.json'), jsonPath: ['mcpServers', 'ktx'] },
],
tempDir,
);
expect(custom[0].title).toBe(`Claude Code · ${installRoot}`);
const same = formatInstallSummaryLines(
[{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot: tempDir }],
[],
tempDir,
);
expect(same[0].title).toBe('Claude Code · Project scope');
});
it('installs project files and next actions under an explicit installRoot', async () => {
const io = makeIo();
const installRoot = join(tempDir, 'workspace');
const result = await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'claude-code',
scope: 'project',
mode: 'mcp-cli',
skipAgents: false,
installRoot,
},
io.io,
);
expect(result).toMatchObject({
status: 'ready',
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp-cli', installRoot }],
});
await expect(stat(join(installRoot, '.claude/skills/ktx/SKILL.md'))).resolves.toBeDefined();
const mcp = JSON.parse(await readFile(join(installRoot, '.mcp.json'), 'utf-8')) as {
mcpServers?: Record<string, unknown>;
};
expect(mcp.mcpServers).toHaveProperty('ktx');
await expect(stat(join(tempDir, '.claude/skills/ktx/SKILL.md'))).rejects.toThrow();
const output = io.stdout();
expect(output).toContain('Open Claude Code from the install directory:');
expect(output).toContain(`cd '${installRoot}'`);
expect(output).toContain(`ktx mcp start --project-dir ${tempDir}`);
expect(await readKtxAgentInstallManifest(tempDir)).toMatchObject({
projectDir: tempDir,
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp-cli', installRoot }],
});
});
it('fails when an explicit installRoot points at an existing file', async () => {
const io = makeIo();
const filePath = join(tempDir, 'not-a-dir');
await writeFile(filePath, 'x', 'utf-8');
await expect(
runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'claude-code',
scope: 'project',
mode: 'mcp',
skipAgents: false,
installRoot: filePath,
},
io.io,
),
).resolves.toEqual({ status: 'failed', projectDir: tempDir });
expect(io.stderr()).toContain('is a file, not a directory');
});
it('installs into the current directory and records it in the manifest', async () => {
const io = makeIo();
const openedDir = join(tempDir, 'opened');
await mkdir(openedDir, { recursive: true });
const prompts = {
select: vi.fn(async ({ message }: { message: string }) =>
message.startsWith('Where') ? 'current' : 'mcp',
),
multiselect: vi.fn(async () => ['claude-code', 'cursor']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
const result = await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'auto',
yes: false,
agents: true,
scope: 'project',
mode: 'mcp',
skipAgents: false,
cwd: openedDir,
},
io.io,
{ prompts },
);
expect(result).toMatchObject({
status: 'ready',
installs: [
{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot: openedDir },
{ target: 'cursor', scope: 'project', mode: 'mcp', installRoot: openedDir },
],
});
await expect(stat(join(openedDir, '.claude/skills/ktx-analytics/SKILL.md'))).resolves.toBeDefined();
await expect(stat(join(openedDir, '.cursor/mcp.json'))).resolves.toBeDefined();
const output = io.stdout();
expect(output).toContain('Open Cursor from the install directory:');
expect(output).toContain(openedDir);
expect(await readKtxAgentInstallManifest(tempDir)).toMatchObject({
installs: [
{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot: openedDir },
{ target: 'cursor', scope: 'project', mode: 'mcp', installRoot: openedDir },
],
});
});
it('creates and installs into a typed custom directory', async () => {
const io = makeIo();
const customDir = join(tempDir, 'custom-target');
const prompts = {
select: vi.fn(async ({ message }: { message: string }) =>
message.startsWith('Where') ? 'custom' : 'mcp',
),
multiselect: vi.fn(async () => ['claude-code']),
text: vi.fn(async () => customDir),
cancel: vi.fn(),
};
const result = await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'auto',
yes: false,
agents: true,
scope: 'project',
mode: 'mcp',
skipAgents: false,
cwd: tempDir,
},
io.io,
{ prompts },
);
expect(result).toMatchObject({
status: 'ready',
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot: customDir }],
});
await expect(stat(join(customDir, '.claude/skills/ktx-analytics/SKILL.md'))).resolves.toBeDefined();
});
it('hides the current directory row when cwd equals the ktx project directory', async () => {
const io = makeIo();
const prompts = {
select: vi.fn(async ({ message }: { message: string; options: Array<{ value: string }> }) =>
message.startsWith('Where') ? 'project' : 'mcp',
),
multiselect: vi.fn(async () => ['claude-code']),
text: vi.fn(async () => undefined),
cancel: vi.fn(),
};
await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'auto',
yes: false,
agents: true,
scope: 'project',
mode: 'mcp',
skipAgents: false,
cwd: tempDir,
},
io.io,
{ prompts },
);
const directoryCall = prompts.select.mock.calls.find(([opts]) => opts.message.startsWith('Where'));
expect(directoryCall).toBeDefined();
expect(directoryCall?.[0].options.map((option) => option.value)).toEqual(['project', 'custom', 'global']);
});
it('re-prompts when a typed custom directory is an existing file', async () => {
const io = makeIo();
const filePath = join(tempDir, 'afile');
await writeFile(filePath, 'x', 'utf-8');
const validDir = join(tempDir, 'valid');
const prompts = {
select: vi.fn(async ({ message }: { message: string }) =>
message.startsWith('Where') ? 'custom' : 'mcp',
),
multiselect: vi.fn(async () => ['claude-code']),
text: vi.fn<() => Promise<string | undefined>>().mockResolvedValueOnce(filePath).mockResolvedValueOnce(validDir),
cancel: vi.fn(),
};
const result = await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'auto',
yes: false,
agents: true,
scope: 'project',
mode: 'mcp',
skipAgents: false,
cwd: tempDir,
},
io.io,
{ prompts },
);
expect(result).toMatchObject({
status: 'ready',
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot: validDir }],
});
expect(prompts.text).toHaveBeenCalledTimes(2);
expect(io.stderr()).toContain('is a file, not a directory');
await expect(stat(join(validDir, '.claude/skills/ktx-analytics/SKILL.md'))).resolves.toBeDefined();
});
it('expands a leading ~ in a typed custom directory', async () => {
const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-'));
const previousHome = process.env.HOME;
process.env.HOME = home;
try {
const io = makeIo();
const prompts = {
select: vi.fn(async ({ message }: { message: string }) =>
message.startsWith('Where') ? 'custom' : 'mcp',
),
multiselect: vi.fn(async () => ['claude-code']),
text: vi.fn(async () => '~/opened-here'),
cancel: vi.fn(),
};
const expected = join(home, 'opened-here');
const result = await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'auto',
yes: false,
agents: true,
scope: 'project',
mode: 'mcp',
skipAgents: false,
cwd: tempDir,
},
io.io,
{ prompts },
);
expect(result).toMatchObject({
status: 'ready',
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot: expected }],
});
await expect(stat(join(expected, '.claude/skills/ktx-analytics/SKILL.md'))).resolves.toBeDefined();
await expect(stat(join(tempDir, '~'))).rejects.toThrow();
} finally {
process.env.HOME = previousHome;
await rm(home, { recursive: true, force: true });
}
});
it('expands a leading ~ in an explicit installRoot', async () => {
const home = await mkdtemp(join(tmpdir(), 'ktx-setup-agents-home-'));
const previousHome = process.env.HOME;
process.env.HOME = home;
try {
const io = makeIo();
const expected = join(home, 'flagged');
const result = await runKtxSetupAgentsStep(
{
projectDir: tempDir,
inputMode: 'disabled',
yes: true,
agents: true,
target: 'claude-code',
scope: 'project',
mode: 'mcp',
skipAgents: false,
installRoot: '~/flagged',
cwd: tempDir,
},
io.io,
);
expect(result).toMatchObject({
status: 'ready',
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp', installRoot: expected }],
});
await expect(stat(join(expected, '.claude/skills/ktx-analytics/SKILL.md'))).resolves.toBeDefined();
} finally {
process.env.HOME = previousHome;
await rm(home, { recursive: true, force: true });
}
});
});
describe('createAgentNextActionsLineFormatter', () => {
function makeColorStdout(): { write: (chunk: string) => boolean; hasColors: () => boolean } {
return { write: () => true, hasColors: () => true };

28
packages/cli/test/setup-databases-federation-notice.test.ts Normal file
View file

@ -0,0 +1,28 @@
import { describe, expect, it } from 'vitest';
import { federationNoticeFor } from '../src/setup-databases.js';
describe('federationNoticeFor', () => {
it('returns a notice naming members when 2+ compatible exist', () => {
const notice = federationNoticeFor({
pg_books: { driver: 'postgres' },
sqlite_reviews: { driver: 'sqlite' },
} as never, '/proj');
expect(notice).toMatch(/pg_books/);
expect(notice).toMatch(/sqlite_reviews/);
expect(notice).toMatch(/cross-database/i);
// Cross-DB joins via a source's `joins:` list are unsupported; the notice
// must steer users to raw SQL against the federated connection instead.
expect(notice).toMatch(/_ktx_federated/);
expect(notice).not.toMatch(/joins:/);
});
it('returns null with fewer than 2 compatible', () => {
expect(federationNoticeFor({ pg: { driver: 'postgres' } } as never, '/proj')).toBeNull();
});
it('returns null when the second db is incompatible', () => {
expect(
federationNoticeFor({ pg: { driver: 'postgres' }, snow: { driver: 'snowflake' } } as never, '/proj'),
).toBeNull();
});
});

2
packages/cli/test/setup-demo-tour.test.ts
View file

@ -184,7 +184,7 @@ describe('runDemoTour', () => {
const mockAgents = vi.fn().mockResolvedValue({
status: 'ready',
projectDir: '/tmp/test',
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp-cli' }],
installs: [{ target: 'claude-code', scope: 'project', mode: 'mcp-cli', installRoot: '/tmp/test' }],
} satisfies KtxSetupAgentsResult);
const navigation = vi.fn().mockResolvedValue('forward');

18
packages/cli/test/setup.test.ts
View file

@ -2025,7 +2025,7 @@ describe('setup status', () => {
return {
status: 'ready',
projectDir: tempDir,
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }],
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli', installRoot: tempDir }],
};
},
},
@ -2078,7 +2078,7 @@ describe('setup status', () => {
agents: async () => ({
status: 'ready',
projectDir: tempDir,
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }],
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli', installRoot: tempDir }],
}),
},
),
@ -2133,7 +2133,7 @@ describe('setup status', () => {
return {
status: 'ready',
projectDir: tempDir,
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }],
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli', installRoot: tempDir }],
};
},
},
@ -2150,7 +2150,7 @@ describe('setup status', () => {
const agents = vi.fn(async () => ({
status: 'ready' as const,
projectDir: tempDir,
installs: [{ target: 'codex' as const, scope: 'project' as const, mode: 'mcp-cli' as const }],
installs: [{ target: 'codex' as const, scope: 'project' as const, mode: 'mcp-cli' as const, installRoot: tempDir }],
}));
await writeFile(join(tempDir, 'ktx.yaml'), ['connections: {}', ''].join('\n'), 'utf-8');
@ -2193,7 +2193,7 @@ describe('setup status', () => {
const agents = vi.fn(async () => ({
status: 'ready' as const,
projectDir: tempDir,
installs: [{ target: 'claude-code' as const, scope: 'project' as const, mode: 'mcp' as const }],
installs: [{ target: 'claude-code' as const, scope: 'project' as const, mode: 'mcp' as const, installRoot: tempDir }],
}));
await writeFile(join(tempDir, 'ktx.yaml'), ['connections: {}', ''].join('\n'), 'utf-8');
@ -2272,7 +2272,7 @@ describe('setup status', () => {
version: 1,
projectDir: tempDir,
installedAt: '2026-05-07T00:00:00.000Z',
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }],
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli', installRoot: tempDir }],
entries: [],
},
null,
@ -2347,7 +2347,7 @@ describe('setup status', () => {
return {
status: 'ready',
projectDir: tempDir,
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }],
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli', installRoot: tempDir }],
};
},
},
@ -2453,7 +2453,7 @@ describe('setup status', () => {
return {
status: 'ready',
projectDir: tempDir,
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli' }],
installs: [{ target: 'codex', scope: 'project', mode: 'mcp-cli', installRoot: tempDir }],
};
},
},
@ -2636,7 +2636,7 @@ describe('setup status', () => {
const agents = vi.fn(async () => ({
status: 'ready' as const,
projectDir: tempDir,
installs: [{ target: 'universal' as const, scope: 'project' as const, mode: 'mcp-cli' as const }],
installs: [{ target: 'universal' as const, scope: 'project' as const, mode: 'mcp-cli' as const, installRoot: tempDir }],
}));
await expect(

13
packages/cli/test/sl.test.ts
View file

@ -618,6 +618,15 @@ joins: []
vi.stubEnv('CI', '');
const projectDir = join(tempDir, 'project');
await seedSlSource({ projectDir });
const config = parseKtxProjectConfig(await readFile(join(projectDir, 'ktx.yaml'), 'utf-8'));
await writeFile(
join(projectDir, 'ktx.yaml'),
serializeKtxProjectConfig({
...config,
connections: { ...config.connections, warehouse: { driver: 'postgres' } },
}),
'utf-8',
);
const io = makeIo({ isTTY: true });
const createSemanticLayerCompute = vi.fn(() => ({
query: vi.fn(async () => ({
@ -654,6 +663,7 @@ joins: []
const projectDir = join(tempDir, 'project');
const project = await initKtxProject({ projectDir });
project.config.connections.warehouse = { driver: 'postgres' };
await writeFile(join(projectDir, 'ktx.yaml'), serializeKtxProjectConfig(project.config), 'utf-8');
await project.fileStore.writeFile(
'semantic-layer/warehouse/orders.yaml',
`name: orders
@ -721,6 +731,7 @@ joins: []
const projectDir = join(tempDir, 'project');
const project = await initKtxProject({ projectDir });
project.config.connections.warehouse = { driver: 'postgres' };
await writeFile(join(projectDir, 'ktx.yaml'), serializeKtxProjectConfig(project.config), 'utf-8');
await project.fileStore.writeFile(
'semantic-layer/warehouse/orders.yaml',
`name: orders
@ -751,7 +762,7 @@ joins: []
validateSources: vi.fn(),
generateSources: vi.fn(),
};
const createManagedSemanticLayerCompute = vi.fn(() => compute);
const createManagedSemanticLayerCompute = vi.fn(async () => compute);
await expect(
runKtxSl(

90
packages/cli/test/sql-federated.integration.test.ts Normal file
View file

@ -0,0 +1,90 @@
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import Database from 'better-sqlite3';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { parseKtxProjectConfig, serializeKtxProjectConfig } from '../src/context/project/config.js';
import { initKtxProject } from '../src/context/project/project.js';
import type { SqlAnalysisPort } from '../src/context/sql-analysis/ports.js';
import type { KtxCliIo } from '../src/cli-runtime.js';
import { runKtxSql } from '../src/sql.js';
function fakeIo(): { io: KtxCliIo; out: () => string; err: () => string } {
let out = '';
let err = '';
return {
io: {
stdout: { write: (chunk: string) => ((out += chunk), true) },
stderr: { write: (chunk: string) => ((err += chunk), true) },
} as unknown as KtxCliIo,
out: () => out,
err: () => err,
};
}
// Validation needs the Python daemon, unavailable in unit tests; execution is real.
const stubSqlAnalysis: SqlAnalysisPort = {
analyzeForFingerprint: async () => ({ fingerprint: '', normalizedSql: '', tablesTouched: [], literalSlots: [] }),
analyzeBatch: async () => new Map([['cli-sql', { tablesTouched: [], columnsByClause: {} }]]),
validateReadOnly: async () => ({ ok: true, error: null }),
};
describe('ktx sql federated integration', () => {
let dir: string;
beforeEach(async () => {
dir = await mkdtemp(join(tmpdir(), 'ktx-fed-int-'));
});
afterEach(async () => {
await rm(dir, { recursive: true, force: true });
});
it('joins books and reviews across two sqlite files', async () => {
const projectDir = join(dir, 'project');
await initKtxProject({ projectDir });
const books = new Database(join(projectDir, 'books.db'));
books.exec("CREATE TABLE books (id INTEGER PRIMARY KEY, title TEXT); INSERT INTO books VALUES (1, 'Clean Code');");
books.close();
const reviews = new Database(join(projectDir, 'reviews.db'));
reviews.exec('CREATE TABLE reviews (id INTEGER PRIMARY KEY, book_id INTEGER, rating INTEGER); INSERT INTO reviews VALUES (1, 1, 5);');
reviews.close();
const config = parseKtxProjectConfig(await readFile(join(projectDir, 'ktx.yaml'), 'utf-8'));
await writeFile(
join(projectDir, 'ktx.yaml'),
serializeKtxProjectConfig({
...config,
connections: {
books_db: { driver: 'sqlite', path: 'books.db' },
reviews_db: { driver: 'sqlite', path: 'reviews.db' },
},
}),
'utf-8',
);
const { io, out, err } = fakeIo();
const code = await runKtxSql(
{
command: 'execute',
projectDir,
connectionId: '_ktx_federated',
sql: 'SELECT b.title, r.rating FROM books_db.books b JOIN reviews_db.reviews r ON b.id = r.book_id',
maxRows: 100,
json: true,
cliVersion: 'test',
},
io,
{ createSqlAnalysis: () => stubSqlAnalysis },
);
expect(code, err()).toBe(0);
const payload = JSON.parse(out()) as { connectionId: string; headers: string[]; rows: unknown[][] };
expect(payload.connectionId).toBe('_ktx_federated');
expect(payload.headers).toEqual(['title', 'rating']);
expect(payload.rows).toHaveLength(1);
expect(payload.rows[0][0]).toBe('Clean Code');
expect(Number(payload.rows[0][1])).toBe(5);
});
});

58
packages/cli/test/sql.test.ts
View file

@ -306,7 +306,9 @@ describe('runKtxSql', () => {
),
).resolves.toBe(1);
expect(io.stderr()).toContain('Connection "warehouse" is not configured in ktx.yaml');
expect(io.stderr()).toContain(
'Connection "warehouse" is not configured in ktx.yaml. No connections are configured in ktx.yaml.',
);
});
it('rejects connectors without read-only SQL support and still cleans up', async () => {
@ -343,6 +345,58 @@ describe('runKtxSql', () => {
expect(connector.executeReadOnly).not.toHaveBeenCalled();
expect(connector.cleanup).toHaveBeenCalledTimes(1);
expect(io.stderr()).toContain('Connection "warehouse" does not support read-only SQL execution.');
expect(io.stderr()).toContain('does not support read-only SQL execution.');
});
it('routes _ktx_federated through the shared federated executor', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir });
await writeConnections(projectDir, {
books_db: { driver: 'sqlite', path: 'books.db' },
reviews_db: { driver: 'sqlite', path: 'reviews.db' },
});
const executeFederated = vi.fn(async () => ({
headers: ['title', 'rating'],
rows: [['Clean Code', 5]],
totalRows: 1,
command: 'SELECT',
rowCount: 1,
}));
const memberConnector = makeConnector({
executeReadOnly: vi.fn(async () => {
throw new Error('member connector must not be used for federated id');
}),
});
const io = makeIo();
await expect(
runKtxSql(
{
command: 'execute',
projectDir,
connectionId: '_ktx_federated',
sql: 'select 1',
maxRows: 100,
output: 'json',
json: true,
cliVersion: '0.0.0-test',
},
io.io,
{
createSqlAnalysis: () => makeSqlAnalysis({ ok: true, error: null }),
createScanConnector: vi.fn(async () => memberConnector),
executeFederated,
},
),
).resolves.toBe(0);
expect(executeFederated).toHaveBeenCalledTimes(1);
expect(memberConnector.executeReadOnly).not.toHaveBeenCalled();
expect(JSON.parse(io.stdout())).toEqual({
connectionId: '_ktx_federated',
headers: ['title', 'rating'],
rows: [['Clean Code', 5]],
rowCount: 1,
});
});
});

94
pnpm-lock.yaml generated
View file

@ -143,6 +143,9 @@ importers:
'@commander-js/extra-typings':
specifier: 14.0.0
version: 14.0.0(commander@14.0.3)
'@duckdb/node-api':
specifier: 1.5.3-r.3
version: 1.5.3-r.3
'@google-cloud/bigquery':
specifier: ^8.3.1
version: 8.3.1
@ -747,6 +750,56 @@ packages:
'@dabh/diagnostics@2.0.8':
resolution: {integrity: sha512-R4MSXTVnuMzGD7bzHdW2ZhhdPC/igELENcq5IjEverBvq5hn1SXCWcsi6eSsdWP0/Ur+SItRRjAktmdoX/8R/Q==}
'@duckdb/node-api@1.5.3-r.3':
resolution: {integrity: sha512-FzuL6sevuFfEFwkgiUMRMUAj4TaVqV//L0oo2FVZ9s9oYpLpALF9qZyQv2ucclTNQZwDCkm8+e6yLMc6t8IjlA==}
'@duckdb/node-bindings-darwin-arm64@1.5.3-r.3':
resolution: {integrity: sha512-ttD8QBesgzHu7Sc4qouuIGLM7PWedLW8GvFbnZEyMqk24mQz1HWFgaT0ivw6nDRaDPUQLB9QnAOq8MZUh1zWHQ==}
cpu: [arm64]
os: [darwin]
'@duckdb/node-bindings-darwin-x64@1.5.3-r.3':
resolution: {integrity: sha512-Vp9MYtoYf6zUWHdCmHXwUcJlHq3YaaIeULWeSiPUM1hsDflLiZKXtz5i250Ulz03VsfWBjpO4wdM99sjjrYKkg==}
cpu: [x64]
os: [darwin]
'@duckdb/node-bindings-linux-arm64-musl@1.5.3-r.3':
resolution: {integrity: sha512-IadRyx+98FEynKLXAk2MzReinFgduiDXgNd5Z8c5VKch+8FgBfqkEUYGOnBMMUPT8kuheKdLj23vpWXaCzOgoQ==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@duckdb/node-bindings-linux-arm64@1.5.3-r.3':
resolution: {integrity: sha512-3HLcrzQE83947JS51UVR7C9qnXQMltCOk4Dnhiz1CD+9u32DGLMgPTIIxclk7O+Q7EwfqzD8JV86Ud+LT1crcQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@duckdb/node-bindings-linux-x64-musl@1.5.3-r.3':
resolution: {integrity: sha512-5bulS16YhftXcarki4tvCufVslntpQDLOEF6RZ+FSMOGiv5d7SDXqklmVRy4DKW3C5ekgN7S2oYzuGL/ss9BuA==}
cpu: [x64]
os: [linux]
libc: [musl]
'@duckdb/node-bindings-linux-x64@1.5.3-r.3':
resolution: {integrity: sha512-TXndAL0ZoETq17Df6wB+SUZjLGDmOsKuDSySxB+wy6sHfpRtbDgQibyXRlajVeUkRDwSzBFC5ymy16YG0Fl4iw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@duckdb/node-bindings-win32-arm64@1.5.3-r.3':
resolution: {integrity: sha512-55Vu13S0jUudiAGlNWJd7UvlW1iKjwWehD8s93jBCNm0AdE/EJN4nz5rQ0IuWzPWXpMjAYuKu00yE7NdtbTyug==}
cpu: [arm64]
os: [win32]
'@duckdb/node-bindings-win32-x64@1.5.3-r.3':
resolution: {integrity: sha512-rlOc9ltWQNHuDq99Ah8XaD80nN1ucrSK5AcH/7ibSp9ogX/jswPYlRVE7ODFJAjnQNf8bVvs++Mp+wyGvuG7ag==}
cpu: [x64]
os: [win32]
'@duckdb/node-bindings@1.5.3-r.3':
resolution: {integrity: sha512-Dphw1a9kKXZnCiWX1YCEAJsQ7WJQO2Ikgxy7m8jy0QVXqAwB9esr5NGsuEL3vMKL7velZHeZCjGOMnHZEcIsdg==}
'@electric-sql/pglite-socket@0.1.5':
resolution: {integrity: sha512-/RAye+3EPKfO9nY4tljzxXmkT7yIpFDm0L3F+c28b+Z6uxPOjy/Zz/QEHYHXcrfuUC88/a9S72EO0+3E0j97wQ==}
hasBin: true
@ -6755,6 +6808,47 @@ snapshots:
enabled: 2.0.0
kuler: 2.0.0
'@duckdb/node-api@1.5.3-r.3':
dependencies:
'@duckdb/node-bindings': 1.5.3-r.3
'@duckdb/node-bindings-darwin-arm64@1.5.3-r.3':
optional: true
'@duckdb/node-bindings-darwin-x64@1.5.3-r.3':
optional: true
'@duckdb/node-bindings-linux-arm64-musl@1.5.3-r.3':
optional: true
'@duckdb/node-bindings-linux-arm64@1.5.3-r.3':
optional: true
'@duckdb/node-bindings-linux-x64-musl@1.5.3-r.3':
optional: true
'@duckdb/node-bindings-linux-x64@1.5.3-r.3':
optional: true
'@duckdb/node-bindings-win32-arm64@1.5.3-r.3':
optional: true
'@duckdb/node-bindings-win32-x64@1.5.3-r.3':
optional: true
'@duckdb/node-bindings@1.5.3-r.3':
dependencies:
detect-libc: 2.1.2
optionalDependencies:
'@duckdb/node-bindings-darwin-arm64': 1.5.3-r.3
'@duckdb/node-bindings-darwin-x64': 1.5.3-r.3
'@duckdb/node-bindings-linux-arm64': 1.5.3-r.3
'@duckdb/node-bindings-linux-arm64-musl': 1.5.3-r.3
'@duckdb/node-bindings-linux-x64': 1.5.3-r.3
'@duckdb/node-bindings-linux-x64-musl': 1.5.3-r.3
'@duckdb/node-bindings-win32-arm64': 1.5.3-r.3
'@duckdb/node-bindings-win32-x64': 1.5.3-r.3
'@electric-sql/pglite-socket@0.1.5(@electric-sql/pglite@0.4.5)':
dependencies:
'@electric-sql/pglite': 0.4.5

2
python/ktx-daemon/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "ktx-daemon"
version = "0.11.0"
version = "0.12.0"
description = "Portable compute package for ktx semantic-layer operations"
readme = "README.md"
requires-python = ">=3.13"

5
python/ktx-daemon/src/ktx_daemon/telemetry/events.schema.json
View file

@ -162,6 +162,7 @@
"outcome",
"durationMs",
"errorClass",
"errorDetail",
"sampleRate",
"mcpClientName",
"mcpClientVersion"
@ -1167,6 +1168,10 @@
"errorClass": {
"type": "string"
},
"errorDetail": {
"type": "string",
"maxLength": 1000
},
"sampleRate": {
"type": "number",
"const": 1

2
python/ktx-sl/pyproject.toml
View file

@ -1,6 +1,6 @@
[project]
name = "ktx-sl"
version = "0.11.0"
version = "0.12.0"
description = "Agent-first semantic layer engine with aggregate locality"
readme = "README.md"
requires-python = ">=3.13"

2
release-policy.json
View file

@ -19,7 +19,7 @@
},
"publishedPackageSmoke": {
"packageName": "@kaelio/ktx",
"version": "0.11.0",
"version": "0.12.0",
"registry": null
},
"runtimeInstaller": {

95
scripts/refresh-uv-manifest.mjs Normal file
View file

@ -0,0 +1,95 @@
#!/usr/bin/env node
import { writeFileSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
const scriptDir = path.dirname(fileURLToPath(import.meta.url));
const manifestModulePath = path.join(scriptDir, '..', 'packages', 'cli', 'src', 'managed-uv-release.ts');
// Linux always uses the musl-static build: it runs on both glibc and musl
// distributions, so the CLI never has to detect libc at runtime.
const PLATFORM_ARTIFACTS = {
'darwin-arm64': 'uv-aarch64-apple-darwin.tar.gz',
'darwin-x64': 'uv-x86_64-apple-darwin.tar.gz',
'linux-arm64': 'uv-aarch64-unknown-linux-musl.tar.gz',
'linux-x64': 'uv-x86_64-unknown-linux-musl.tar.gz',
'win32-arm64': 'uv-aarch64-pc-windows-msvc.zip',
'win32-x64': 'uv-x86_64-pc-windows-msvc.zip',
};
function parseSha256File(contents, file) {
const hash = contents.trim().split(/\s+/)[0]?.replace(/^\*/, '');
if (!/^[a-f0-9]{64}$/.test(hash ?? '')) {
throw new Error(`Unexpected sha256 file contents for ${file}: ${contents.slice(0, 120)}`);
}
return hash;
}
export function renderUvReleaseModule(version, artifacts) {
const keys = Object.keys(PLATFORM_ARTIFACTS);
for (const key of keys) {
if (!artifacts[key]?.file || !artifacts[key]?.sha256) {
throw new Error(`Missing artifact entry for ${key}`);
}
}
const entries = keys
.map(
(key) =>
` '${key}': { file: '${artifacts[key].file}', sha256: '${artifacts[key].sha256}' }, // pragma: allowlist secret`,
)
.join('\n');
return `// Generated by scripts/refresh-uv-manifest.mjs. Do not edit by hand.
// Regenerate with: node scripts/refresh-uv-manifest.mjs [<uv-version>]
export type ManagedUvPlatformKey =
${keys.map((key) => ` | '${key}'`).join('\n')};
export interface ManagedUvArtifact {
file: string;
sha256: string;
}
export const MANAGED_UV_VERSION = '${version}';
export const MANAGED_UV_ARTIFACTS: Record<ManagedUvPlatformKey, ManagedUvArtifact> = {
${entries}
};
`;
}
export async function refreshUvManifest(options = {}) {
const fetchImpl = options.fetch ?? fetch;
const writeFile = options.writeFile ?? writeFileSync;
const log = options.log ?? console.log;
const outputPath = options.outputPath ?? manifestModulePath;
let version = options.version;
if (!version) {
const response = await fetchImpl('https://api.github.com/repos/astral-sh/uv/releases/latest');
if (!response.ok) {
throw new Error(`Failed to resolve latest uv release: HTTP ${response.status}`);
}
version = (await response.json()).tag_name;
}
const artifacts = {};
for (const [key, file] of Object.entries(PLATFORM_ARTIFACTS)) {
const url = `https://github.com/astral-sh/uv/releases/download/${version}/${file}.sha256`;
const response = await fetchImpl(url);
if (!response.ok) {
throw new Error(`Failed to fetch ${url}: HTTP ${response.status}`);
}
artifacts[key] = { file, sha256: parseSha256File(await response.text(), file) };
}
writeFile(outputPath, renderUvReleaseModule(version, artifacts));
log(`Pinned uv ${version} into ${outputPath}`);
return { version, artifacts };
}
if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) {
refreshUvManifest({ version: process.argv[2] }).catch((error) => {
console.error(error);
process.exit(1);
});
}

69
scripts/refresh-uv-manifest.test.mjs Normal file
View file

@ -0,0 +1,69 @@
import assert from 'node:assert/strict';
import { describe, it } from 'node:test';
import { refreshUvManifest, renderUvReleaseModule } from './refresh-uv-manifest.mjs';
const ALL_KEYS = ['darwin-arm64', 'darwin-x64', 'linux-arm64', 'linux-x64', 'win32-arm64', 'win32-x64'];
function artifactsFixture() {
return Object.fromEntries(
ALL_KEYS.map((key, index) => [key, { file: `uv-${key}.tar.gz`, sha256: String(index).repeat(64).slice(0, 64) }]),
);
}
describe('renderUvReleaseModule', () => {
it('renders a typed module with every platform key and the pinned version', () => {
const module = renderUvReleaseModule('1.2.3', artifactsFixture());
assert.match(module, /MANAGED_UV_VERSION = '1\.2\.3'/);
assert.match(module, /Generated by scripts\/refresh-uv-manifest\.mjs/);
for (const key of ALL_KEYS) {
assert.match(module, new RegExp(`'${key}': \\{ file: 'uv-${key}\\.tar\\.gz', sha256: '[a-f0-9]{64}' \\}`));
}
});
it('rejects an incomplete platform map', () => {
const artifacts = artifactsFixture();
delete artifacts['win32-arm64'];
assert.throws(() => renderUvReleaseModule('1.2.3', artifacts), /Missing artifact entry for win32-arm64/);
});
});
describe('refreshUvManifest', () => {
it('fetches per-artifact sha256 files and writes the module', async () => {
const written = [];
const fetched = [];
const fetchStub = async (url) => {
fetched.push(url);
return {
ok: true,
text: async () => `${'a'.repeat(64)} *${url.split('/').at(-1).replace('.sha256', '')}\n`,
};
};
const result = await refreshUvManifest({
version: '1.2.3',
fetch: fetchStub,
writeFile: (path, contents) => written.push({ path, contents }),
log: () => {},
outputPath: '/tmp/managed-uv-release.ts',
});
assert.equal(result.version, '1.2.3');
assert.equal(fetched.length, ALL_KEYS.length);
assert.ok(fetched.every((url) => url.endsWith('.sha256') && url.includes('/download/1.2.3/')));
assert.equal(written.length, 1);
assert.match(written[0].contents, /MANAGED_UV_VERSION = '1\.2\.3'/);
});
it('rejects malformed sha256 file contents', async () => {
await assert.rejects(
refreshUvManifest({
version: '1.2.3',
fetch: async () => ({ ok: true, text: async () => '<html>proxy login</html>' }),
writeFile: () => {},
log: () => {},
}),
/Unexpected sha256 file contents/,
);
});
});

4
uv.lock generated
View file

@ -466,7 +466,7 @@ wheels = [
[[package]]
name = "ktx-daemon"
version = "0.11.0"
version = "0.12.0"
source = { editable = "python/ktx-daemon" }
dependencies = [
{ name = "fastapi" },
@ -523,7 +523,7 @@ dev = [
[[package]]
name = "ktx-sl"
version = "0.11.0"
version = "0.12.0"
source = { editable = "python/ktx-sl" }
dependencies = [
{ name = "pydantic" },