Merge remote-tracking branch 'origin/main' into luca-martial/schema-select-ux-text

# Conflicts:
#	packages/cli/src/demo.test.ts
#	packages/context/src/ingest/local-adapters.ts
This commit is contained in:
Andrey Avtomonov 2026-05-12 14:47:28 +02:00
commit d0f650f44a
123 changed files with 3739 additions and 933 deletions

View file

@ -12,7 +12,7 @@ refs:
## New Hire Week-One Onboarding Policy
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
**Owner:** Manager (not People Ops)
---

View file

@ -20,7 +20,7 @@ tables:
# Activation KPI Glossary
**Owner team:** Growth
**Owner team:** Growth
**Source:** Notion — Orbit Demo Home / Data Team - Onboarding / Activation KPI Glossary, last edited 2026-05-07
Use this when a question is about signup-to-habit behavior. Orbit uses activation language across Growth, Product, and CS conversations.
@ -62,4 +62,3 @@ Growth conversations typically use D7 and D14 Activation Rate. Product and CS ma
## Relationship to Account-Level Activation
This glossary defines **customer-level** activation (signup-to-habit). The **account-level** activation workflow (requester login → first approved purchase request → account activated) is a separate concept tracked in `mart_account_activity` and governed by the January 2026 policy change. See `orbit-activation-policy-change-jan-2026` for that definition.

View file

@ -14,9 +14,9 @@ sl_refs:
# Activation Policy Change — January 2026
**Governed metric key:** `activated_accounts`
**Owner team:** growth
**Notion:** `notion://notion_page_activation_policy_decision#policy-change`
**Governed metric key:** `activated_accounts`
**Owner team:** growth
**Notion:** `notion://notion_page_activation_policy_decision#policy-change`
**Sources:** `mart_account_activity`, `int_activation_policy_windows`, `stg_activation_events`
## Policy Boundary

View file

@ -15,9 +15,9 @@ sl_refs:
# ARR — Contract-First Definition
**Governed metric key:** `arr`
**Owner team:** finance
**Notion:** `notion://notion_page_arr_contract_reporting#arr-contract-first`
**Governed metric key:** `arr`
**Owner team:** finance
**Notion:** `notion://notion_page_arr_contract_reporting#arr-contract-first`
**Source:** `mart_arr_daily` (grain: `metric_date`)
## Rule

View file

@ -21,7 +21,7 @@ refs:
Orbit sells procurement workflow and spend-control software. The core value proposition: route purchase requests, collect approvals, onboard suppliers, and issue purchase orders without turning every exception into a status hunt.
**Primary buyers:** Finance, Procurement, Business Operations.
**Primary buyers:** Finance, Procurement, Business Operations.
**Daily users:** department admins, office managers, IT leads, legal ops partners — anyone who has to get a vendor through the building.
## Product Workflow
@ -69,4 +69,3 @@ Orbit sells procurement workflow and spend-control software. The core value prop
- "Supplier onboarding is split across three teams."
- "Renewals are visible too late."
- "People keep asking Finance for status because there is nowhere better to look."

View file

@ -14,9 +14,9 @@ sl_refs:
# Customer Health Risk Definition
**Governed metric key:** `active_customers`
**Owner team:** customer_success
**Notion:** `notion://notion_page_customer_health_playbook#risk-definition`
**Governed metric key:** `active_customers`
**Owner team:** customer_success
**Notion:** `notion://notion_page_customer_health_playbook#risk-definition`
**Sources:** `mart_customer_health`, `int_customer_health_signals`
## Risk Levels

View file

@ -18,8 +18,8 @@ tables:
# Orbit Customers Source
**Table:** `orbit_analytics.customer`
**Grain:** one row per signed-up customer
**Table:** `orbit_analytics.customer`
**Grain:** one row per signed-up customer
**Source:** Notion — Orbit Demo Home / Data Team - Onboarding / Orbit Customers Source, last edited 2026-05-07
Use this when a question needs customer identity, plan tier, signup timing, recent activity, or the standard customer joins.
@ -58,4 +58,3 @@ Always join through `customer.id`. Do not join on `email`.
- **Timezone:** `created_at` and `last_seen_at` are UTC. Confirm whether a question expects UTC or a local business day before filtering.
- **Paying vs. all:** `free` customers must be excluded from paying-customer follow-ups. Use `paying_customer_count`, not `customer_count`.
- **plan_tier values:** `free`, `pro`, `enterprise`. Note: `pro_plus` is a legacy alias for `growth` in the account/contract layer (see `orbit-plan-segment-normalization`), but `plan_tier` on this table uses `pro` not `pro_plus`.

View file

@ -42,4 +42,3 @@ Declared in `models/exposures.yml`. All exposures are type `dashboard` with matu
- **Owner:** Growth (growth@orbit-demo.example.com)
- **Depends on:** `mart_account_activity`
- **Description:** Activation policy comparison around the January 2026 workflow update.

View file

@ -22,10 +22,10 @@ sl_refs:
# Orbit dbt Project Overview
**Project name:** `kaelio_demo`
**dbt version:** 1.0.0
**Profile target:** Postgres (`orbit_analytics` schema, `kaelio_demo` database)
**Raw source schema:** `orbit_raw`
**Project name:** `kaelio_demo`
**dbt version:** 1.0.0
**Profile target:** Postgres (`orbit_analytics` schema, `kaelio_demo` database)
**Raw source schema:** `orbit_raw`
**Analytics schema:** `orbit_analytics` (all models materialised as views by default)
## Model Layers
@ -52,4 +52,3 @@ sl_refs:
## Raw Source Tables (`orbit_raw` schema)
accounts, account_hierarchy, plans, contracts, subscriptions, contract_discount_terms, arr_movements, invoices, invoice_line_items, refunds, plan_segment_mapping, users, activation_events, sessions, purchase_requests, approval_events, suppliers, supplier_onboarding_events, purchase_orders, support_tickets, account_owners.

View file

@ -20,7 +20,7 @@ tables:
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/106.json -->
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/107.json -->
**Table:** `orbit_analytics.mart_account_activity`
**Table:** `orbit_analytics.mart_account_activity`
**Grain:** one row per `policy_change_date`
## Columns
@ -47,4 +47,3 @@ tables:
- The January 2026 activation policy change (`policy_change_date = 2026-01-15`) is the primary boundary. `policy_version` in upstream events splits into `pre_2026_01_15` and `post_2026_01_15` cohorts.
- Rates are ratios (01); multiply by 100 for percentage display.
- See [orbit-activation-policy-change-jan-2026](orbit-activation-policy-change-jan-2026) for full policy context.

View file

@ -19,7 +19,7 @@ tables:
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/69.json -->
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/100.json -->
**Table:** `orbit_analytics.mart_account_segments`
**Table:** `orbit_analytics.mart_account_segments`
**Grain:** one row per `account_id`
## Columns
@ -53,4 +53,3 @@ tables:
- `normalized_plan_code` maps `pro_plus``growth`. Always use `normalized_plan_code` for plan-based reporting. See [orbit-plan-segment-normalization](orbit-plan-segment-normalization).
- `segment` is derived from `canonical_plan_code × size_band` via `stg_plan_segment_mapping`.
- `contract_arr_cents` is the contract-first ARR value. See [orbit-arr-contract-first-definition](orbit-arr-contract-first-definition).

View file

@ -18,7 +18,7 @@ tables:
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/56.json -->
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/96.json -->
**Table:** `orbit_analytics.mart_arr_daily`
**Table:** `orbit_analytics.mart_arr_daily`
**Grain:** one row per `metric_date`
## Columns
@ -44,4 +44,3 @@ tables:
- ARR is calculated contract-first: active contract ARR takes precedence over subscription ARR for any covered period. See [orbit-arr-contract-first-definition](orbit-arr-contract-first-definition).
- `display` is a formatted label for UI rendering; use `arr_cents` for all arithmetic.

View file

@ -20,7 +20,7 @@ tables:
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/98.json -->
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/103.json -->
**Table:** `orbit_analytics.mart_nrr_quarterly`
**Table:** `orbit_analytics.mart_nrr_quarterly`
**Grain:** one row per `quarter_label` × `segment`
## Columns
@ -53,4 +53,3 @@ tables:
- `net_revenue_retention` is a ratio, not a percentage. Multiply by 100 for display.
- Contraction includes discount expirations (classified as contraction, not churn). See [orbit-nrr-discount-expiration-treatment](orbit-nrr-discount-expiration-treatment).
- Enterprise is the primary executive reporting segment.

View file

@ -18,7 +18,7 @@ tables:
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/88.json -->
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/108.json -->
**Table:** `orbit_analytics.mart_procurement_activity`
**Table:** `orbit_analytics.mart_procurement_activity`
**Grain:** one row per `week_start_date` × `contract_arr_threshold_cents`
## Columns
@ -45,4 +45,3 @@ tables:
- `active_requesters` counts non-internal, non-test requesters on large active contracts. See [orbit-procurement-qualifying-actions](orbit-procurement-qualifying-actions).
- The standard threshold is `contract_arr_threshold_cents = 20000000` ($200k ARR).
- Always filter by `contract_arr_threshold_cents` — the table contains rows for multiple threshold values.

View file

@ -19,7 +19,7 @@ tables:
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/105.json -->
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/115.json -->
**Table:** `orbit_analytics.mart_retention_movement_breakout`
**Table:** `orbit_analytics.mart_retention_movement_breakout`
**Grain:** one row per `quarter_label` × `segment` × `movement_type` × `movement_reason`
## Columns
@ -53,4 +53,3 @@ tables:
- Contraction includes discount expirations, classified as contraction (not churn), tracked via `movement_reason`. See [orbit-nrr-discount-expiration-treatment](orbit-nrr-discount-expiration-treatment).
- This table is the row-level source for `mart_nrr_quarterly` aggregations.
- Only one of `expansion_arr_cents`, `contraction_arr_cents`, `churned_arr_cents` is non-zero per row.

View file

@ -20,7 +20,7 @@ tables:
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/102.json -->
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/104.json -->
**Table:** `orbit_analytics.mart_revenue_daily`
**Table:** `orbit_analytics.mart_revenue_daily`
**Grain:** one row per `revenue_date`
## Columns
@ -54,4 +54,3 @@ tables:
- `reconciliation_check` must be `true` on every row. Any `false` row indicates a data quality issue.
- Gross-to-net reconciliation: gross revenue credits refunds = net revenue. See [orbit-revenue-gross-to-net-reconciliation](orbit-revenue-gross-to-net-reconciliation).
- All amounts are in cents; divide by 100 for USD, by 100,000,000 for $M.

View file

@ -69,4 +69,3 @@ Card 48 is the canonical reference; card 55 is a filtered variant for large-cont
| 53 | Enterprise NRR quarter breakout | mart_nrr_quarterly | 0 |
| 54 | February credits drilldown | mart_revenue_daily | 0 |
| 55 | Large contract requesters | mart_account_segments | 0 |

View file

@ -15,9 +15,9 @@ sl_refs:
# NRR — Discount Expiration Treatment
**Governed metric key:** `net_revenue_retention`
**Owner team:** analytics
**Notion:** `notion://notion_page_retention_policy_current#nrr-definition` and `#discount-expiration-treatment`
**Governed metric key:** `net_revenue_retention`
**Owner team:** analytics
**Notion:** `notion://notion_page_retention_policy_current#nrr-definition` and `#discount-expiration-treatment`
**Sources:** `mart_nrr_quarterly`, `mart_retention_movement_breakout`
## NRR Definition

View file

@ -14,9 +14,9 @@ sl_refs:
# Plan & Segment Normalization
**Governed metric key:** `segment`
**Owner team:** sales_ops
**Notion:** `notion://notion_page_sales_ops_segmentation#growth-plan-normalization`
**Governed metric key:** `segment`
**Owner team:** sales_ops
**Notion:** `notion://notion_page_sales_ops_segmentation#growth-plan-normalization`
**Sources:** `mart_account_segments`, `stg_plan_segment_mapping`, `stg_plans`
## Canonical Plan Codes

View file

@ -14,9 +14,9 @@ sl_refs:
# Procurement — Qualifying Actions & Weekly Active Requesters
**Governed metric key:** `weekly_active_requesters`
**Owner team:** product
**Notion:** `notion://notion_page_procurement_instrumentation#qualifying-procurement-actions`
**Governed metric key:** `weekly_active_requesters`
**Owner team:** product
**Notion:** `notion://notion_page_procurement_instrumentation#qualifying-procurement-actions`
**Sources:** `mart_procurement_activity`, `int_procurement_qualifying_actions`
## Qualifying Action Definition

View file

@ -14,9 +14,9 @@ sl_refs:
# Revenue — Gross-to-Net Reconciliation
**Governed metric key:** `net_revenue`
**Owner team:** finance
**Notion:** `notion://notion_page_revenue_reporting_policy#gross-to-net-reconciliation`
**Governed metric key:** `net_revenue`
**Owner team:** finance
**Notion:** `notion://notion_page_revenue_reporting_policy#gross-to-net-reconciliation`
**Source:** `mart_revenue_daily` (grain: `revenue_date`)
## Formula

View file

@ -14,7 +14,7 @@ refs:
## Sales Ops → Customer Success Implementation Handoff
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
**Owner:** Sales Ops (sender), Customer Success (receiver)
---

View file

@ -1,4 +1,4 @@
import { spinner } from '@clack/prompts';
import { cancel, confirm, isCancel, log, spinner } from '@clack/prompts';
export interface KtxCliSpinner {
start(message: string): void;
@ -6,6 +6,62 @@ export interface KtxCliSpinner {
error(message: string): void;
}
export interface KtxCliPromptAdapter {
confirm(options: { message: string; initialValue?: boolean }): Promise<boolean>;
cancel(message: string): void;
log: {
info(message: string): void;
warn(message: string): void;
error(message: string): void;
success(message: string): void;
step(message: string): void;
};
spinner(): KtxCliSpinner;
}
export class KtxCliPromptCancelledError extends Error {
constructor(message = 'Operation cancelled.') {
super(message);
this.name = 'KtxCliPromptCancelledError';
}
}
export function createClackSpinner(): KtxCliSpinner {
return spinner();
}
export function createClackPromptAdapter(): KtxCliPromptAdapter {
return {
async confirm(options) {
const value = await confirm(options);
if (isCancel(value)) {
cancel('Operation cancelled.');
throw new KtxCliPromptCancelledError();
}
return value;
},
cancel(message) {
cancel(message);
},
log: {
info(message) {
log.info(message);
},
warn(message) {
log.warn(message);
},
error(message) {
log.error(message);
},
success(message) {
log.success(message);
},
step(message) {
log.step(message);
},
},
spinner() {
return createClackSpinner();
},
};
}

View file

@ -138,7 +138,7 @@ function makeIo(options: { isTTY?: boolean; stdinIsTTY?: boolean } = {}) {
describe('runKtxConnectionMetabaseSetup', () => {
const fakeMetabaseCredential = 'mb_example';
const existingMetabaseCredential = 'mb_existing';
const fakeAdminCredential = 'pw';
const fakeAdminCredential = 'admin-secret-value-123';
let tempDir: string;
let projectDir: string;

View file

@ -53,10 +53,12 @@ export function registerRuntimeCommands(program: Command, context: KtxCliCommand
runtime
.command('stop')
.description('Stop the KTX-managed Python HTTP daemon')
.action(async () => {
.option('--all', 'Stop all KTX daemon processes recorded or discoverable on this machine', false)
.action(async (options: { all?: boolean }) => {
await runRuntimeArgs(context, {
command: 'stop',
cliVersion: context.packageInfo.version,
all: options.all === true,
});
});

View file

@ -1,7 +1,8 @@
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { initKtxProject, parseKtxProjectConfig } from '@ktx/context/project';
import type { MetabaseRuntimeClient } from '@ktx/context/ingest';
import { initKtxProject, parseKtxProjectConfig, serializeKtxProjectConfig } from '@ktx/context/project';
import type { KtxConnectionDriver, KtxScanConnector, KtxSchemaSnapshot } from '@ktx/context/scan';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { runKtxConnection } from './connection.js';
@ -476,7 +477,7 @@ describe('runKtxConnection', () => {
force: false,
allowLiteralCredentials: false,
notion: {
authTokenRef: 'env:NOTION_AUTH_TOKEN',
authTokenRef: 'env:NOTION_TOKEN',
crawlMode: 'all_accessible',
rootPageIds: [],
rootDatabaseIds: [],
@ -492,7 +493,7 @@ describe('runKtxConnection', () => {
const yaml = await readFile(join(projectDir, 'ktx.yaml'), 'utf-8');
expect(yaml).toContain('driver: notion');
expect(yaml).toContain('auth_token_ref: env:NOTION_AUTH_TOKEN');
expect(yaml).toContain('auth_token_ref: env:NOTION_TOKEN');
expect(yaml).toContain('crawl_mode: all_accessible');
expect(yaml).toContain('max_pages_per_run: 50');
expect(yaml).not.toContain('ntn_');
@ -515,7 +516,7 @@ describe('runKtxConnection', () => {
force: false,
allowLiteralCredentials: false,
notion: {
authTokenRef: 'env:NOTION_AUTH_TOKEN',
authTokenRef: 'env:NOTION_TOKEN',
crawlMode: 'all_accessible',
rootPageIds: [],
rootDatabaseIds: ['database-1'],
@ -598,6 +599,61 @@ describe('runKtxConnection', () => {
expect(io.stdout()).toContain('Tables: 2');
});
it('tests a configured Metabase connection through the Metabase runtime client', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
const projectConfig = parseKtxProjectConfig(await readFile(join(projectDir, 'ktx.yaml'), 'utf-8'));
await writeFile(
join(projectDir, 'ktx.yaml'),
serializeKtxProjectConfig({
...projectConfig,
connections: {
...projectConfig.connections,
prod_metabase: {
driver: 'metabase',
api_url: 'http://metabase.example.test',
api_key: 'mb_test',
},
},
}),
'utf-8',
);
const testConnection = vi.fn(async () => ({ success: true as const }));
const getDatabases = vi.fn(async () => [
{ id: 1, name: 'Analytics', engine: 'postgres', details: {}, is_sample: false },
{ id: 2, name: 'Sample Database', engine: 'h2', details: {}, is_sample: true },
]);
const cleanup = vi.fn(async () => undefined);
const createMetabaseClient = vi.fn(
async (): Promise<Pick<MetabaseRuntimeClient, 'testConnection' | 'getDatabases' | 'cleanup'>> => ({
testConnection,
getDatabases,
cleanup,
}),
);
const createScanConnector = vi.fn(async () => {
throw new Error('native scanner should not be used for Metabase');
});
const io = makeIo();
await expect(
runKtxConnection({ command: 'test', projectDir, connectionId: 'prod_metabase' }, io.io, {
createScanConnector,
createMetabaseClient,
}),
).resolves.toBe(0);
expect(createScanConnector).not.toHaveBeenCalled();
expect(createMetabaseClient).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), 'prod_metabase');
expect(testConnection).toHaveBeenCalledTimes(1);
expect(getDatabases).toHaveBeenCalledTimes(1);
expect(cleanup).toHaveBeenCalledTimes(1);
expect(io.stdout()).toContain('Connection test passed: prod_metabase');
expect(io.stdout()).toContain('Driver: metabase');
expect(io.stdout()).toContain('Databases: 1');
expect(io.stderr()).toBe('');
});
it('cleans up the native scan connector when connection testing fails', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });

View file

@ -1,4 +1,10 @@
import { cancel, confirm, isCancel } from '@clack/prompts';
import {
DEFAULT_METABASE_CLIENT_CONFIG,
DefaultMetabaseConnectionClientFactory,
type MetabaseRuntimeClient,
metabaseRuntimeConfigFromLocalConnection,
} from '@ktx/context/ingest';
import { type KtxLocalProject, loadKtxProject, serializeKtxProjectConfig } from '@ktx/context/project';
import type { KtxScanConnector } from '@ktx/context/scan';
import type { KtxConnectionMappingArgs } from './commands/connection-mapping.js';
@ -61,6 +67,7 @@ interface KtxConnectionIo extends KtxCliIo {
interface KtxConnectionDeps {
createScanConnector?: typeof createKtxCliScanConnector;
createMetabaseClient?: typeof createDefaultMetabaseClient;
runMapping?: (argv: string[], io: KtxCliIo) => Promise<number>;
prompts?: KtxConnectionPromptAdapter;
}
@ -104,6 +111,12 @@ async function cleanupConnector(connector: KtxScanConnector | null): Promise<voi
}
}
function normalizedConnectionDriver(project: KtxLocalProject, connectionId: string): string {
return String(project.config.connections[connectionId]?.driver ?? '')
.trim()
.toLowerCase();
}
async function testNativeConnection(
project: KtxLocalProject,
connectionId: string,
@ -131,6 +144,48 @@ async function testNativeConnection(
}
}
async function createDefaultMetabaseClient(
project: KtxLocalProject,
connectionId: string,
): Promise<Pick<MetabaseRuntimeClient, 'testConnection' | 'getDatabases' | 'cleanup'>> {
const factory = new DefaultMetabaseConnectionClientFactory(
(metabaseConnectionId) =>
metabaseRuntimeConfigFromLocalConnection(
metabaseConnectionId,
project.config.connections[metabaseConnectionId],
),
DEFAULT_METABASE_CLIENT_CONFIG,
);
return factory.createClient(connectionId);
}
async function testMetabaseConnection(
project: KtxLocalProject,
connectionId: string,
createMetabaseClient: typeof createDefaultMetabaseClient,
): Promise<{ driver: 'metabase'; databaseCount: number }> {
let client: Pick<MetabaseRuntimeClient, 'testConnection' | 'getDatabases' | 'cleanup'> | null = null;
try {
client = await createMetabaseClient(project, connectionId);
const testResult = await client.testConnection();
if (!testResult.success) {
throw new Error(
`Metabase connection test failed: ${testResult.error ?? testResult.message ?? 'unknown error'}`,
);
}
const databases = await client.getDatabases();
const databaseCount = databases.filter((database) => database.is_sample !== true).length;
if (databaseCount === 0) {
throw new Error('Metabase auth worked but no usable databases were returned');
}
return { driver: 'metabase', databaseCount };
} finally {
await client?.cleanup();
}
}
interface BufferedIo extends KtxCliIo {
stdoutText(): string;
stderrText(): string;
@ -399,6 +454,18 @@ export async function runKtxConnection(
return 0;
}
if (normalizedConnectionDriver(project, args.connectionId) === 'metabase') {
const result = await testMetabaseConnection(
project,
args.connectionId,
deps.createMetabaseClient ?? createDefaultMetabaseClient,
);
io.stdout.write(`Connection test passed: ${args.connectionId}\n`);
io.stdout.write(`Driver: ${result.driver}\n`);
io.stdout.write(`Databases: ${result.databaseCount}\n`);
return 0;
}
const result = await testNativeConnection(
project,
args.connectionId,

View file

@ -11,6 +11,9 @@ import type { renderMemoryFlowTui } from './memory-flow-tui.js';
import { KTX_NEXT_STEP_DIRECT_COMMANDS } from './next-steps.js';
import { resetVizFallbackWarningsForTest } from './viz-fallback.js';
const SEEDED_DEMO_SEMANTIC_SOURCE_COUNT = 46;
const SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT = 28;
function makeIo(options: { isTTY?: boolean; columns?: number; rawMode?: boolean } = {}) {
let stdout = '';
let stderr = '';
@ -336,8 +339,14 @@ describe('runKtxDemo', () => {
notion: { pageCount: 8 },
},
generatedOutputs: {
semanticLayer: { manifestSourceCount: 46, fileCount: 46 },
knowledge: { manifestPageCount: 28, fileCount: 28 },
semanticLayer: {
manifestSourceCount: SEEDED_DEMO_SEMANTIC_SOURCE_COUNT,
fileCount: SEEDED_DEMO_SEMANTIC_SOURCE_COUNT,
},
knowledge: {
manifestPageCount: SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT,
fileCount: SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT,
},
links: { manifestLinkCount: 23, linkCount: 23 },
reports: { primaryPath: 'reports/seeded-demo-report.json', fileCount: 1 },
},
@ -636,10 +645,16 @@ describe('runKtxDemo', () => {
).resolves.toBe(0);
expect(seededIo.stdout()).toContain('Status: ready');
expect(seededIo.stdout()).toContain('Semantic-layer sources: 46 manifest, 46 files');
expect(seededIo.stdout()).toContain('Knowledge pages: 28 manifest, 28 files');
expect(seededIo.stdout()).toContain(
`Semantic-layer sources: ${SEEDED_DEMO_SEMANTIC_SOURCE_COUNT} manifest, ${SEEDED_DEMO_SEMANTIC_SOURCE_COUNT} files`,
);
expect(seededIo.stdout()).toContain(
`Knowledge pages: ${SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT} manifest, ${SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT} files`,
);
expect(seededIo.stdout()).not.toContain('Status: corrupt');
expect(seededIo.stdout()).not.toContain('Semantic-layer sources: 6 manifest, 0 files');
expect(seededIo.stdout()).not.toContain(
`Semantic-layer sources: ${SEEDED_DEMO_SEMANTIC_SOURCE_COUNT} manifest, 0 files`,
);
});
it('fails corrupted demo projects in no-input mode with reset guidance', async () => {

View file

@ -2,6 +2,7 @@ import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { createRequire } from 'node:module';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { initKtxProject } from '@ktx/context/project';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
@ -143,6 +144,7 @@ describe('runKtxCli', () => {
const installIo = makeIo();
const startIo = makeIo();
const stopIo = makeIo();
const stopAllIo = makeIo();
const statusIo = makeIo();
const doctorIo = makeIo();
const pruneIo = makeIo();
@ -156,6 +158,7 @@ describe('runKtxCli', () => {
runKtxCli(['runtime', 'start', '--feature', 'local-embeddings', '--force'], startIo.io, { runtime }),
).resolves.toBe(0);
await expect(runKtxCli(['runtime', 'stop'], stopIo.io, { runtime })).resolves.toBe(0);
await expect(runKtxCli(['runtime', 'stop', '--all'], stopAllIo.io, { runtime })).resolves.toBe(0);
await expect(runKtxCli(['runtime', 'status', '--json'], statusIo.io, { runtime })).resolves.toBe(0);
await expect(runKtxCli(['runtime', 'doctor'], doctorIo.io, { runtime })).resolves.toBe(0);
await expect(runKtxCli(['runtime', 'prune', '--dry-run'], pruneIo.io, { runtime })).resolves.toBe(0);
@ -185,11 +188,21 @@ describe('runKtxCli', () => {
{
command: 'stop',
cliVersion: '0.0.0-private',
all: false,
},
stopIo.io,
);
expect(runtime).toHaveBeenNthCalledWith(
4,
{
command: 'stop',
cliVersion: '0.0.0-private',
all: true,
},
stopAllIo.io,
);
expect(runtime).toHaveBeenNthCalledWith(
5,
{
command: 'status',
cliVersion: '0.0.0-private',
@ -198,7 +211,7 @@ describe('runKtxCli', () => {
statusIo.io,
);
expect(runtime).toHaveBeenNthCalledWith(
5,
6,
{
command: 'doctor',
cliVersion: '0.0.0-private',
@ -207,7 +220,7 @@ describe('runKtxCli', () => {
doctorIo.io,
);
expect(runtime).toHaveBeenNthCalledWith(
6,
7,
{
command: 'prune',
cliVersion: '0.0.0-private',
@ -218,6 +231,17 @@ describe('runKtxCli', () => {
);
});
it('documents runtime stop all in command help', async () => {
const testIo = makeIo();
await expect(runKtxCli(['runtime', 'stop', '--help'], testIo.io)).resolves.toBe(0);
expect(testIo.stdout()).toContain('--all');
expect(testIo.stdout()).toContain('Stop all KTX daemon processes recorded or discoverable');
expect(testIo.stdout()).toContain('on this machine');
expect(testIo.stderr()).toBe('');
});
it('routes sl query managed runtime install policies', async () => {
const sl = vi.fn(async () => 0);
@ -310,6 +334,23 @@ describe('runKtxCli', () => {
expect(testIo.stderr()).toBe('');
});
it('keeps representative JSON command stdout parseable', async () => {
const projectDir = join(tempDir, 'project');
await initKtxProject({ projectDir, projectName: 'warehouse' });
const commands = [
['--project-dir', projectDir, 'setup', 'status', '--json'],
['--project-dir', projectDir, 'sl', 'list', '--json'],
];
for (const argv of commands) {
const testIo = makeIo();
await expect(runKtxCli(argv, testIo.io)).resolves.toBe(0);
expect(() => JSON.parse(testIo.stdout())).not.toThrow();
expect(testIo.stderr()).toBe('');
}
});
it('starts setup for bare ktx in a TTY when no project is discoverable', async () => {
const { mkdtemp, realpath, rm } = await import('node:fs/promises');
const { tmpdir } = await import('node:os');
@ -1964,7 +2005,7 @@ describe('runKtxCli', () => {
'--project-dir',
tempDir,
'--token-env',
'NOTION_AUTH_TOKEN',
'NOTION_TOKEN',
'--crawl-mode',
'selected_roots',
'--root-page-id',
@ -1991,7 +2032,7 @@ describe('runKtxCli', () => {
force: false,
allowLiteralCredentials: false,
notion: {
authTokenRef: 'env:NOTION_AUTH_TOKEN',
authTokenRef: 'env:NOTION_TOKEN',
crawlMode: 'selected_roots',
rootPageIds: ['page-1'],
rootDatabaseIds: ['database-1'],

View file

@ -47,13 +47,18 @@ export { runKtxRuntime, type KtxRuntimeArgs, type KtxRuntimeDeps } from './runti
export {
allocateDaemonPort,
readManagedPythonDaemonStatus,
stopAllManagedPythonDaemons,
startManagedPythonDaemon,
stopManagedPythonDaemon,
} from './managed-python-daemon.js';
export type {
ManagedPythonDaemonProcessInfo,
ManagedPythonDaemonStartResult,
ManagedPythonDaemonState,
ManagedPythonDaemonStatus,
ManagedPythonDaemonStopAllEntry,
ManagedPythonDaemonStopAllFailure,
ManagedPythonDaemonStopAllResult,
ManagedPythonDaemonStopResult,
} from './managed-python-daemon.js';
export {

View file

@ -331,8 +331,9 @@ describe('runKtxIngest viz and replay', () => {
).resolves.toBe(0);
expect(runLocal).toHaveBeenCalledWith(expect.objectContaining({ memoryFlow: expect.anything() }));
expect(io.stdout()).toContain('[5%] Fetching source files for warehouse/fake');
expect(io.stderr()).toContain('[5%] Fetching source files for warehouse/fake');
expect(io.stdout()).toContain('Job: plain-run');
expect(io.stdout()).not.toContain('[5%]');
expect(io.stdout()).not.toContain('KTX memory flow');
});
@ -407,8 +408,9 @@ describe('runKtxIngest viz and replay', () => {
expect(startLiveMemoryFlow).not.toHaveBeenCalled();
expect(runLocal).toHaveBeenCalledWith(expect.objectContaining({ memoryFlow: expect.anything() }));
expect(io.stdout()).toContain('[5%] Fetching source files for warehouse/fake');
expect(io.stderr()).toContain('[5%] Fetching source files for warehouse/fake');
expect(io.stdout()).toContain('Job: raw-missing-viz-run');
expect(io.stdout()).not.toContain('[5%]');
expect(io.stdout()).not.toContain('KTX memory flow');
expect(io.stderr()).toContain(
'Visualization requested but stdin raw mode is unavailable; printing plain output.',

View file

@ -546,7 +546,7 @@ export async function runPublicMetabaseSyncModeCase(tempDir: string, input: Sync
),
).resolves.toBe(0);
expect(io.stderr()).toBe('');
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
expect(io.stdout()).toContain(`target=warehouse_a database=1 status=done job=${jobId}`);

View file

@ -14,6 +14,7 @@ import {
import { initKtxProject, ktxLocalStateDbPath, loadKtxProject } from '@ktx/context/project';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { type KtxIngestArgs, runKtxIngest } from './ingest.js';
import type { KtxCliLocalIngestAdaptersOptions } from './local-adapters.js';
import {
CliLookerSlWritingAgentRunner,
CliMetabaseAgentRunner,
@ -32,6 +33,7 @@ import {
writeWarehouseConfig,
} from './ingest.test-utils.js';
import { resetVizFallbackWarningsForTest } from './viz-fallback.js';
import { runKtxSetup } from './setup.js';
describe('runKtxIngest', () => {
let tempDir: string;
@ -105,6 +107,75 @@ describe('runKtxIngest', () => {
expect(statusIo.stderr()).toBe('');
});
it('prints provider setup guidance when a skip-llm setup project runs dev ingest', async () => {
const projectDir = join(tempDir, 'project');
const setupIo = makeIo();
await expect(
runKtxSetup(
{
command: 'run',
projectDir,
mode: 'new',
agents: false,
agentScope: 'project',
agentInstallMode: 'cli',
skipAgents: true,
inputMode: 'disabled',
yes: true,
cliVersion: '0.0.0-test',
skipLlm: true,
skipEmbeddings: true,
databaseDrivers: ['postgres'],
databaseConnectionId: 'warehouse',
databaseUrl: 'env:WAREHOUSE_URL',
databaseSchemas: [],
enableHistoricSql: true,
skipDatabases: false,
skipSources: true,
},
setupIo.io,
{
databasesDeps: {
testConnection: async (_projectDir, _connectionId, io) => {
io.stdout.write('Driver: postgres\nTables: 1\n');
return 0;
},
scanConnection: async () => 0,
historicSqlProbe: async () => ({ ok: true, lines: ['PASS Historic SQL probe skipped in test'] }),
},
context: async () => ({ status: 'skipped', projectDir }),
},
),
).resolves.toBe(0);
const sourceDir = join(tempDir, 'source');
await mkdir(join(sourceDir, 'orders'), { recursive: true });
await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\n', 'utf-8');
const runIo = makeIo();
await expect(
runKtxIngest(
{
command: 'run',
projectDir,
connectionId: 'warehouse',
adapter: 'historic-sql',
sourceDir,
outputMode: 'plain',
},
runIo.io,
),
).resolves.toBe(1);
expect(runIo.stdout()).toBe('');
expect(runIo.stderr()).toContain(
'ktx dev ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.',
);
expect(runIo.stderr()).toContain(
`ktx setup --project-dir ${projectDir} --anthropic-api-key-env ANTHROPIC_API_KEY --anthropic-model claude-sonnet-4-6 --no-input`,
);
});
it('routes metabase scheduled pulls to the fan-out runner and prints child summaries', async () => {
const projectDir = join(tempDir, 'project');
await writeMetabaseConfig(projectDir);
@ -159,7 +230,7 @@ describe('runKtxIngest', () => {
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
expect(io.stdout()).toContain('warehouse_a');
expect(io.stdout()).toContain('metabase-child-1');
expect(io.stderr()).toBe('');
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
});
it('returns a non-zero code when Metabase fan-out has failed children', async () => {
@ -229,7 +300,7 @@ describe('runKtxIngest', () => {
expect(io.stdout()).toContain('Metabase fan-out: partial_failure');
expect(io.stdout()).toContain('Failed work units: 1');
expect(io.stdout()).toContain('status=error');
expect(io.stderr()).toBe('');
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
});
it('prints Metabase fan-out progress before the final summary', async () => {
@ -303,12 +374,56 @@ describe('runKtxIngest', () => {
),
).resolves.toBe(0);
expect(io.stdout()).toContain('Metabase ingest: prod-metabase');
expect(io.stdout()).toContain('Targets: 1 mapped database');
expect(io.stdout()).toContain('- database=1 target=warehouse_a status=running job=metabase-child-1');
expect(io.stdout()).toContain('- database=1 target=warehouse_a status=done job=metabase-child-1');
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
expect(io.stderr()).toContain('Targets: 1 mapped database');
expect(io.stderr()).toContain('- database=1 target=warehouse_a status=running job=metabase-child-1');
expect(io.stderr()).toContain('- database=1 target=warehouse_a status=done job=metabase-child-1');
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
expect(io.stderr()).toBe('');
expect(io.stdout()).not.toContain('status=running job=metabase-child-1');
});
it('writes metabase fan-out progress to stderr and final result to stdout', async () => {
const projectDir = join(tempDir, 'project');
await writeMetabaseConfig(projectDir);
const io = makeIo({ isTTY: true });
await expect(
runKtxIngest(
{
command: 'run',
projectDir,
connectionId: 'prod-metabase',
adapter: 'metabase',
outputMode: 'plain',
},
io.io,
{
runLocalMetabaseIngest: async (input) => {
input.progress?.onMetabaseFanoutPlanned?.({
metabaseConnectionId: 'prod-metabase',
children: [{ metabaseDatabaseId: 1, targetConnectionId: 'warehouse_a' }],
});
input.progress?.onMetabaseChildStarted?.({
metabaseConnectionId: 'prod-metabase',
metabaseDatabaseId: 1,
targetConnectionId: 'warehouse_a',
jobId: 'metabase-child-1',
});
return {
metabaseConnectionId: 'prod-metabase',
status: 'all_succeeded',
totals: { workUnits: 0, failedWorkUnits: 0 },
children: [],
};
},
},
),
).resolves.toBe(0);
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
expect(io.stderr()).toContain('status=running job=metabase-child-1');
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
expect(io.stdout()).not.toContain('status=running job=metabase-child-1');
});
it('runs Metabase scheduled ingest through the public CLI command path with real fan-out', async () => {
@ -393,7 +508,8 @@ describe('runKtxIngest', () => {
),
).resolves.toBe(0);
expect(io.stderr()).toBe('');
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
expect(io.stderr()).toContain('Targets: 2 mapped databases');
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
expect(io.stdout()).toContain('Source: prod-metabase');
expect(io.stdout()).toContain('Children: 2');
@ -483,6 +599,46 @@ describe('runKtxIngest', () => {
expect(io.stderr()).toBe('');
});
it('keeps metabase JSON stdout free of operational adapter logs', async () => {
const projectDir = join(tempDir, 'project');
await writeMetabaseConfig(projectDir);
const io = makeIo();
let adapterOptions: KtxCliLocalIngestAdaptersOptions | undefined;
await expect(
runKtxIngest(
{
command: 'run',
projectDir,
connectionId: 'prod-metabase',
adapter: 'metabase',
outputMode: 'json',
},
io.io,
{
createAdapters: (_project, options) => {
adapterOptions = options;
options?.logger?.warn('adapter warning');
return [];
},
runLocalMetabaseIngest: async (input) => {
input.adapters.find((adapter) => adapter.source === 'metabase');
return {
metabaseConnectionId: 'prod-metabase',
status: 'all_succeeded',
totals: { workUnits: 0, failedWorkUnits: 0 },
children: [],
};
},
},
),
).resolves.toBe(0);
expect(adapterOptions?.logger).toEqual(expect.objectContaining({ warn: expect.any(Function) }));
expect(() => JSON.parse(io.stdout())).not.toThrow();
expect(io.stderr()).toBe('');
});
it('rejects source-dir uploads through the metabase fan-out route', async () => {
const projectDir = join(tempDir, 'project');
await writeMetabaseConfig(projectDir);
@ -694,17 +850,22 @@ describe('runKtxIngest', () => {
),
).resolves.toBe(0);
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
databaseIntrospectionUrl: 'http://127.0.0.1:8765',
});
expect(createAdapters).toHaveBeenCalledWith(
expect.objectContaining({ projectDir }),
expect.objectContaining({
databaseIntrospectionUrl: 'http://127.0.0.1:8765',
logger: expect.any(Object),
}),
);
expect(runLocal).toHaveBeenCalledWith(
expect.objectContaining({
adapters: createdAdapters,
adapter: 'fake',
connectionId: 'warehouse',
pullConfigOptions: {
pullConfigOptions: expect.objectContaining({
databaseIntrospectionUrl: 'http://127.0.0.1:8765',
},
logger: expect.any(Object),
}),
}),
);
});
@ -747,14 +908,19 @@ describe('runKtxIngest', () => {
installPolicy: 'auto',
io: io.io,
};
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
managedDaemon: expectedManagedDaemon,
});
expect(createAdapters).toHaveBeenCalledWith(
expect.objectContaining({ projectDir }),
expect.objectContaining({
managedDaemon: expectedManagedDaemon,
logger: expect.any(Object),
}),
);
expect(runLocal).toHaveBeenCalledWith(
expect.objectContaining({
pullConfigOptions: {
pullConfigOptions: expect.objectContaining({
managedDaemon: expectedManagedDaemon,
},
logger: expect.any(Object),
}),
}),
);
});
@ -808,9 +974,13 @@ describe('runKtxIngest', () => {
),
).resolves.toBe(0);
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
historicSqlConnectionId: 'warehouse',
});
expect(createAdapters).toHaveBeenCalledWith(
expect.objectContaining({ projectDir }),
expect.objectContaining({
historicSqlConnectionId: 'warehouse',
logger: expect.any(Object),
}),
);
expect(runLocal).toHaveBeenCalledWith(
expect.objectContaining({
adapters: createdAdapters,
@ -912,12 +1082,228 @@ describe('runKtxIngest', () => {
expect(stdout).toContain('[45%] Planned 1 work unit');
expect(stdout).toContain('[80%] Processed 1/1 work units');
expect(stdout).toContain('[100%] Ingest completed');
expect(stdout.indexOf('[5%] Fetching source files for warehouse/historic-sql')).toBeLessThan(
stdout.indexOf('Report: report-live-1'),
);
expect(stdout).toContain('Report: report-live-1');
expect(io.stderr()).toBe('');
});
it('writes plain TTY ingest progress and final report to stdout', async () => {
const projectDir = join(tempDir, 'project');
await writeWarehouseConfig(projectDir);
const sourceDir = join(tempDir, 'source');
await mkdir(join(sourceDir, 'orders'), { recursive: true });
await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\n', 'utf-8');
const runLocal = vi.fn(async (input: RunLocalIngestOptions) => completedLocalBundleRun(input, 'local-job-1'));
const io = makeIo({ isTTY: true });
await expect(
runKtxIngest(
{
command: 'run',
projectDir,
connectionId: 'warehouse',
adapter: 'fake',
sourceDir,
outputMode: 'plain',
},
io.io,
{
env: interactiveEnv(),
runLocalIngest: runLocal,
},
),
).resolves.toBe(0);
expect(io.stdout()).toContain('[5%] Fetching source files for warehouse/fake');
expect(io.stdout()).toContain('Report: report-live-1');
expect(io.stderr()).toBe('');
});
it('prints plain WorkUnit step progress during long-running local ingest', async () => {
const projectDir = join(tempDir, 'historic-sql-step-progress-project');
await mkdir(projectDir, { recursive: true });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: historic-sql-step-progress-project',
'connections:',
' warehouse:',
' driver: postgres',
' url: env:WAREHOUSE_DATABASE_URL',
' historicSql:',
' enabled: true',
' dialect: postgres',
' minExecutions: 2',
'ingest:',
' adapters:',
' - historic-sql',
'',
].join('\n'),
'utf-8',
);
const createdAdapters: SourceAdapter[] = [
{ source: 'historic-sql', skillNames: [], detect: async () => true, chunk: async () => ({ workUnits: [] }) },
];
const runLocal = vi.fn(async (input: RunLocalIngestOptions) => {
input.memoryFlow?.update({
plannedWorkUnits: [
{
unitKey: 'historic-sql-table-public-orders',
rawFiles: ['tables/public/orders.json'],
peerFileCount: 0,
dependencyCount: 0,
},
{
unitKey: 'historic-sql-table-public-customers',
rawFiles: ['tables/public/customers.json'],
peerFileCount: 0,
dependencyCount: 0,
},
],
});
input.memoryFlow?.emit({ type: 'chunks_planned', chunkCount: 2, workUnitCount: 2, evictionCount: 0 });
input.memoryFlow?.emit({
type: 'work_unit_started',
unitKey: 'historic-sql-table-public-orders',
skills: ['historic_sql_table_digest'],
stepBudget: 40,
});
input.memoryFlow?.emit({
type: 'work_unit_step',
unitKey: 'historic-sql-table-public-orders',
stepIndex: 7,
stepBudget: 40,
});
input.memoryFlow?.emit({
type: 'work_unit_finished',
unitKey: 'historic-sql-table-public-orders',
status: 'success',
});
input.memoryFlow?.finish('done');
return completedLocalBundleRun(input, input.jobId ?? 'historic-step-progress-job');
});
const io = makeIo({ isTTY: true });
await expect(
runKtxIngest(
{
command: 'run',
projectDir,
connectionId: 'warehouse',
adapter: 'historic-sql',
outputMode: 'plain',
},
io.io,
{
env: interactiveEnv(),
createAdapters: vi.fn(() => createdAdapters as never),
runLocalIngest: runLocal,
jobIdFactory: () => 'historic-step-progress-job',
},
),
).resolves.toBe(0);
const stdout = io.stdout();
expect(stdout).toContain('[45%] Planned 2 work units');
expect(stdout).toContain('[55%] Processing 1/2 work units: historic-sql-table-public-orders');
expect(stdout).toContain(
'\r[58%] Processing work units: 0/2 complete, 1 active; latest historic-sql-table-public-orders step 7/40\u001b[K',
);
expect(stdout).toContain('[68%] Processed 1/2 work units');
});
it('renders concurrent WorkUnit step progress as transient aggregate status', async () => {
const projectDir = join(tempDir, 'historic-sql-concurrent-progress-project');
await mkdir(projectDir, { recursive: true });
await writeFile(
join(projectDir, 'ktx.yaml'),
[
'project: historic-sql-concurrent-progress-project',
'connections:',
' warehouse:',
' driver: postgres',
' url: env:WAREHOUSE_DATABASE_URL',
' historicSql:',
' enabled: true',
' dialect: postgres',
' minExecutions: 2',
'ingest:',
' adapters:',
' - historic-sql',
'',
].join('\n'),
'utf-8',
);
const createdAdapters: SourceAdapter[] = [
{ source: 'historic-sql', skillNames: [], detect: async () => true, chunk: async () => ({ workUnits: [] }) },
];
const workUnitKeys = [
'historic-sql-table-public-orders',
'historic-sql-table-public-customers',
'historic-sql-table-public-line-items',
'historic-sql-table-public-payments',
'historic-sql-table-public-products',
'historic-sql-table-public-suppliers',
];
const runLocal = vi.fn(async (input: RunLocalIngestOptions) => {
input.memoryFlow?.update({
plannedWorkUnits: workUnitKeys.map((unitKey) => ({
unitKey,
rawFiles: [`tables/${unitKey}.json`],
peerFileCount: 0,
dependencyCount: 0,
})),
});
input.memoryFlow?.emit({
type: 'chunks_planned',
chunkCount: workUnitKeys.length,
workUnitCount: workUnitKeys.length,
evictionCount: 0,
});
for (const unitKey of workUnitKeys) {
input.memoryFlow?.emit({
type: 'work_unit_started',
unitKey,
skills: ['historic_sql_table_digest'],
stepBudget: 40,
});
}
for (const unitKey of workUnitKeys) {
input.memoryFlow?.emit({ type: 'work_unit_step', unitKey, stepIndex: 1, stepBudget: 40 });
}
input.memoryFlow?.finish('done');
return completedLocalBundleRun(input, input.jobId ?? 'historic-concurrent-progress-job');
});
const io = makeIo({ isTTY: true });
await expect(
runKtxIngest(
{
command: 'run',
projectDir,
connectionId: 'warehouse',
adapter: 'historic-sql',
outputMode: 'plain',
},
io.io,
{
env: interactiveEnv(),
createAdapters: vi.fn(() => createdAdapters as never),
runLocalIngest: runLocal,
jobIdFactory: () => 'historic-concurrent-progress-job',
},
),
).resolves.toBe(0);
const stdout = io.stdout();
expect(stdout).toContain(
'\r[56%] Processing work units: 0/6 complete, 6 active; latest historic-sql-table-public-suppliers step 1/40\u001b[K',
);
expect(stdout).not.toContain(
'\n[56%] Processing 6/6 work units: historic-sql-table-public-suppliers step 1/40\n',
);
expect(stdout).toContain('\n[100%] Ingest completed\n');
});
it('passes local Looker pull-config options and agent runner into scheduled ingest for Looker scheduled ingest', async () => {
const projectDir = join(tempDir, 'project');
await writeWarehouseConfig(projectDir);
@ -958,15 +1344,19 @@ describe('runKtxIngest', () => {
),
).resolves.toBe(0);
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
looker: {
parser: pullConfigOptions.looker.parser,
},
});
expect(createAdapters).toHaveBeenCalledWith(
expect.objectContaining({ projectDir }),
expect.objectContaining({
logger: expect.any(Object),
looker: {
parser: pullConfigOptions.looker.parser,
},
}),
);
expect(runLocal).toHaveBeenCalledWith(
expect.objectContaining({
agentRunner,
pullConfigOptions,
pullConfigOptions: expect.objectContaining(pullConfigOptions),
}),
);
});

View file

@ -18,6 +18,7 @@ import {
} from '@ktx/context/ingest';
import { loadKtxProject } from '@ktx/context/project';
import { readIngestReportSnapshotFile } from './ingest-report-file.js';
import { createCliOperationalLogger } from './io/logger.js';
import { createKtxCliLocalIngestAdapters } from './local-adapters.js';
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
import { type KtxMemoryFlowStdin, renderMemoryFlowInteractively } from './memory-flow-interactive.js';
@ -142,22 +143,22 @@ function createMetabaseFanoutProgress(
connectionId: string,
io: KtxIngestIo,
): LocalMetabaseFanoutProgress {
io.stdout.write(`Metabase ingest: ${connectionId}\n`);
io.stdout.write('Checking mappings and scheduled-pull targets...\n');
io.stderr.write(`Metabase ingest: ${connectionId}\n`);
io.stderr.write('Checking mappings and scheduled-pull targets...\n');
return {
onMetabaseFanoutPlanned(event) {
io.stdout.write(`Targets: ${pluralize(event.children.length, 'mapped database')}\n`);
io.stderr.write(`Targets: ${pluralize(event.children.length, 'mapped database')}\n`);
for (const child of event.children) {
io.stdout.write(`- database=${child.metabaseDatabaseId} target=${child.targetConnectionId} status=queued\n`);
io.stderr.write(`- database=${child.metabaseDatabaseId} target=${child.targetConnectionId} status=queued\n`);
}
},
onMetabaseChildStarted(event) {
io.stdout.write(
io.stderr.write(
`- database=${event.metabaseDatabaseId} target=${event.targetConnectionId} status=running job=${event.jobId}\n`,
);
},
onMetabaseChildCompleted(event) {
io.stdout.write(
io.stderr.write(
`- database=${event.metabaseDatabaseId} target=${event.targetConnectionId} status=${event.status} job=${event.jobId}\n`,
);
},
@ -168,14 +169,51 @@ function formatDiffProgress(event: Extract<MemoryFlowEvent, { type: 'diff_comput
return `+${event.added}/~${event.modified}/-${event.deleted}/=${event.unchanged}`;
}
function completedWorkUnitCount(snapshot: MemoryFlowReplayInput): number {
return snapshot.events.filter((event) => event.type === 'work_unit_finished').length;
function workUnitEventsThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): MemoryFlowEvent[] {
return snapshot.events.slice(0, eventIndex + 1);
}
function completedWorkUnitCountThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): number {
return workUnitEventsThrough(snapshot, eventIndex).filter((event) => event.type === 'work_unit_finished').length;
}
function activeWorkUnitCountThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): number {
const active = new Set<string>();
for (const event of workUnitEventsThrough(snapshot, eventIndex)) {
if (event.type === 'work_unit_started') {
active.add(event.unitKey);
}
if (event.type === 'work_unit_finished') {
active.delete(event.unitKey);
}
}
return active.size;
}
function plannedWorkUnitCountThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): number {
if (snapshot.plannedWorkUnits.length > 0) {
return snapshot.plannedWorkUnits.length;
}
const planEvent = workUnitEventsThrough(snapshot, eventIndex)
.filter((event) => event.type === 'chunks_planned')
.at(-1);
return planEvent?.workUnitCount ?? completedWorkUnitCountThrough(snapshot, eventIndex);
}
function workUnitOrdinalThrough(snapshot: MemoryFlowReplayInput, eventIndex: number, unitKey: string): number {
const events = workUnitEventsThrough(snapshot, eventIndex);
const startedIndex = events.findIndex((event) => event.type === 'work_unit_started' && event.unitKey === unitKey);
if (startedIndex === -1) {
return completedWorkUnitCountThrough(snapshot, eventIndex) + 1;
}
return events.slice(0, startedIndex + 1).filter((event) => event.type === 'work_unit_started').length;
}
function plainIngestEventProgress(
event: MemoryFlowEvent,
snapshot: MemoryFlowReplayInput,
): { percent: number; message: string } | null {
eventIndex: number,
): { percent: number; message: string; transient?: boolean } | null {
switch (event.type) {
case 'source_acquired':
return {
@ -196,11 +234,28 @@ function plainIngestEventProgress(
};
case 'stage_skipped':
return { percent: 45, message: `Skipped ${event.stage}: ${event.reason}` };
case 'work_unit_started':
return { percent: 55, message: `Processing ${event.unitKey}` };
case 'work_unit_started': {
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
const ordinal = workUnitOrdinalThrough(snapshot, eventIndex, event.unitKey);
const progress = total > 0 ? `${ordinal}/${total} work units: ` : '';
return { percent: 55, message: `Processing ${progress}${event.unitKey}` };
}
case 'work_unit_step': {
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
const completed = completedWorkUnitCountThrough(snapshot, eventIndex);
const active = activeWorkUnitCountThrough(snapshot, eventIndex);
const stepFraction = event.stepBudget > 0 ? Math.min(1, event.stepIndex / event.stepBudget) : 0;
const percent = total > 0 ? 55 + Math.ceil(((completed + stepFraction) / total) * 25) : 55;
const latest = `${event.unitKey} step ${event.stepIndex}/${event.stepBudget}`;
return {
percent,
message: `Processing work units: ${completed}/${total} complete, ${active} active; latest ${latest}`,
transient: true,
};
}
case 'work_unit_finished': {
const total = snapshot.plannedWorkUnits.length || completedWorkUnitCount(snapshot);
const completed = completedWorkUnitCount(snapshot);
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
const completed = completedWorkUnitCountThrough(snapshot, eventIndex);
const percent = total > 0 ? 55 + Math.round((completed / total) * 25) : 80;
return {
percent,
@ -225,7 +280,6 @@ function plainIngestEventProgress(
case 'report_created':
return { percent: 98, message: `Created ingest report ${event.reportPath ?? event.runId}` };
case 'scope_detected':
case 'work_unit_step':
case 'candidate_action':
return null;
}
@ -242,15 +296,31 @@ function shouldWritePlainIngestProgress(
function createPlainIngestProgressRenderer(
args: Extract<KtxIngestArgs, { command: 'run' }>,
io: KtxIngestIo,
): { start(): void; update(snapshot: MemoryFlowReplayInput): void } {
): { start(): void; update(snapshot: MemoryFlowReplayInput): void; flush(): void } {
let printedEvents = 0;
let lastPercent = 0;
let printedCompletion = false;
let hasPendingTransient = false;
const write = (percent: number, message: string) => {
const flush = () => {
if (!hasPendingTransient) {
return;
}
io.stdout.write('\n');
hasPendingTransient = false;
};
const write = (percent: number, message: string, options?: { transient?: boolean }) => {
const nextPercent = Math.max(lastPercent, Math.max(0, Math.min(100, percent)));
lastPercent = nextPercent;
io.stdout.write(`[${nextPercent}%] ${message}\n`);
const line = `[${nextPercent}%] ${message}`;
if (options?.transient === true) {
io.stdout.write(`\r${line}\u001b[K`);
hasPendingTransient = true;
return;
}
flush();
io.stdout.write(`${line}\n`);
};
return {
@ -259,13 +329,14 @@ function createPlainIngestProgressRenderer(
},
update(snapshot) {
while (printedEvents < snapshot.events.length) {
const eventIndex = printedEvents;
const event = snapshot.events[printedEvents++];
if (!event) {
continue;
}
const progress = plainIngestEventProgress(event, snapshot);
const progress = plainIngestEventProgress(event, snapshot, eventIndex);
if (progress) {
write(progress.percent, progress.message);
write(progress.percent, progress.message, progress.transient === true ? { transient: true } : undefined);
}
}
if (!printedCompletion && snapshot.status !== 'running') {
@ -273,6 +344,7 @@ function createPlainIngestProgressRenderer(
write(100, snapshot.status === 'done' ? 'Ingest completed' : 'Ingest failed');
}
},
flush,
};
}
@ -435,11 +507,13 @@ export async function runKtxIngest(
const executeLocalIngest = deps.runLocalIngest ?? runLocalIngest;
const localIngestOptions = deps.localIngestOptions ?? {};
const managedDaemon = managedDaemonOptionsForIngestRun(args, io);
const operationalLogger = createCliOperationalLogger(io, args.outputMode);
const adapterOptions = {
...(localIngestOptions.pullConfigOptions ?? {}),
...(args.databaseIntrospectionUrl ? { databaseIntrospectionUrl: args.databaseIntrospectionUrl } : {}),
...(managedDaemon ? { managedDaemon } : {}),
...(args.adapter === 'historic-sql' ? { historicSqlConnectionId: args.connectionId } : {}),
logger: operationalLogger,
};
if (args.adapter === 'metabase' && args.sourceDir) {
throw new Error('source-dir uploads are not supported for the Metabase fan-out adapter');
@ -524,6 +598,7 @@ export async function runKtxIngest(
io.stdout.write(formatMemoryFlowFinalSummary(latestMemoryFlowSnapshot));
return reportStatus(result.report) === 'done' ? 0 : 1;
}
plainProgress?.flush();
await writeReportRecord(result.report, runOutputMode, io, {
interactive: (args.inputMode ?? 'auto') === 'auto',
renderStoredMemoryFlow: deps.renderStoredMemoryFlow,
@ -531,6 +606,7 @@ export async function runKtxIngest(
});
return reportStatus(result.report) === 'done' ? 0 : 1;
} finally {
plainProgress?.flush();
liveTui?.close();
}
}

View file

@ -0,0 +1,65 @@
import { describe, expect, it, vi } from 'vitest';
import { createCliOperationalLogger, createNoopOperationalLogger } from './logger.js';
function makeIo() {
let stdout = '';
let stderr = '';
return {
io: {
stdout: {
write: (chunk: string) => {
stdout += chunk;
},
},
stderr: {
write: (chunk: string) => {
stderr += chunk;
},
},
},
stdout: () => stdout,
stderr: () => stderr,
};
}
describe('createCliOperationalLogger', () => {
it('routes operational messages to stderr outside JSON mode', () => {
const io = makeIo();
const logger = createCliOperationalLogger(io.io, 'plain');
logger.log('progress');
logger.warn('warning');
logger.error('failure');
logger.debug?.('debug');
expect(io.stdout()).toBe('');
expect(io.stderr()).toBe('progress\nwarning\nfailure\ndebug\n');
});
it('suppresses operational messages in JSON mode by default', () => {
const io = makeIo();
const logger = createCliOperationalLogger(io.io, 'json');
logger.log('progress');
logger.warn('warning');
logger.error('failure');
logger.debug?.('debug');
expect(io.stdout()).toBe('');
expect(io.stderr()).toBe('');
});
});
describe('createNoopOperationalLogger', () => {
it('never writes', () => {
const logger = createNoopOperationalLogger();
const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined);
logger.log('progress');
logger.warn('warning');
logger.error('failure');
logger.debug?.('debug');
expect(warn).not.toHaveBeenCalled();
});
});

View file

@ -0,0 +1,40 @@
import type { KtxCliIo } from '../cli-runtime.js';
import type { KtxOutputMode } from './mode.js';
export interface KtxOperationalLogger {
log(message: string): void;
warn(message: string): void;
error(message: string): void;
debug?(message: string): void;
}
export type KtxOperationalOutputMode = KtxOutputMode | 'viz';
function writeLine(io: KtxCliIo, message: string): void {
io.stderr.write(message.endsWith('\n') ? message : `${message}\n`);
}
export function createNoopOperationalLogger(): KtxOperationalLogger {
return {
log: () => undefined,
warn: () => undefined,
error: () => undefined,
debug: () => undefined,
};
}
export function createCliOperationalLogger(
io: KtxCliIo,
mode: KtxOperationalOutputMode,
): KtxOperationalLogger {
if (mode === 'json') {
return createNoopOperationalLogger();
}
return {
log: (message) => writeLine(io, message),
warn: (message) => writeLine(io, message),
error: (message) => writeLine(io, message),
debug: (message) => writeLine(io, message),
};
}

View file

@ -28,6 +28,16 @@ export interface PrintListArgs<Row> {
io: KtxCliIo;
}
export interface KtxJsonResultEnvelope<T> {
kind: string;
data: T;
meta?: Record<string, unknown>;
}
export function writeJsonResult<T>(io: KtxCliIo, envelope: KtxJsonResultEnvelope<T>): void {
io.stdout.write(`${JSON.stringify(envelope, null, 2)}\n`);
}
export function printList<Row extends object>(args: PrintListArgs<Row>): void {
switch (args.mode) {
case 'json':
@ -61,12 +71,11 @@ function printListPlain<Row extends object>(args: PrintListArgs<Row>): void {
}
function printListJson<Row extends object>(args: PrintListArgs<Row>): void {
const envelope = {
writeJsonResult(args.io, {
kind: 'list',
data: { items: args.rows },
meta: { command: args.command },
};
args.io.stdout.write(`${JSON.stringify(envelope, null, 2)}\n`);
});
}
function pluralize(count: number, singular: string): string {

View file

@ -35,6 +35,7 @@ import {
managedDaemonDatabaseIntrospectionOptions,
type ManagedPythonCoreDaemonOptions,
} from './managed-python-http.js';
import type { KtxOperationalLogger } from './io/logger.js';
function hasSnowflakeDriver(connection: unknown): boolean {
return (
@ -162,6 +163,7 @@ export interface KtxCliLocalIngestAdaptersOptions extends DefaultLocalIngestAdap
sqlAnalysis?: SqlAnalysisPort;
sqlAnalysisUrl?: string;
managedDaemon?: ManagedPythonCoreDaemonOptions;
logger?: KtxOperationalLogger;
}
function historicSqlRecord(connection: unknown): Record<string, unknown> | null {

View file

@ -214,6 +214,7 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
expect(confirmInstall).toHaveBeenCalledWith(
'KTX needs to install the core Python runtime. This downloads Python dependencies with uv. Continue?',
io.io,
);
expect(installRuntime).toHaveBeenCalledWith({
cliVersion: '0.2.0',
@ -221,4 +222,45 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
force: false,
});
});
it('uses injected runtime confirmation instead of reading process TTY directly', async () => {
const io = makeIo();
const compute = { query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() };
const installRuntime = vi.fn(async (): Promise<ManagedPythonRuntimeInstallResult> => installResult());
const confirmInstall = vi.fn(async () => true);
await expect(
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'prompt',
io: io.io,
readStatus: async () => missingStatus(),
installRuntime,
confirmInstall,
createPythonCompute: () => compute,
}),
).resolves.toBe(compute);
expect(confirmInstall).toHaveBeenCalledWith(
'KTX needs to install the core Python runtime. This downloads Python dependencies with uv. Continue?',
io.io,
);
expect(io.stderr()).toContain('Installing KTX Python runtime (core) with uv...');
});
it('can decide default runtime prompting from injected io capabilities', async () => {
const io = makeIo();
Object.assign(io.io.stdout, { isTTY: false });
await expect(
createManagedPythonSemanticLayerComputePort({
cliVersion: '0.2.0',
installPolicy: 'prompt',
io: io.io,
readStatus: async () => missingStatus(),
installRuntime: vi.fn(),
createPythonCompute: () => ({ query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }),
}),
).rejects.toThrow('KTX Python runtime installation was cancelled');
});
});

View file

@ -1,6 +1,6 @@
import { cancel, confirm, isCancel } from '@clack/prompts';
import { createPythonSemanticLayerComputePort, type KtxSemanticLayerComputePort } from '@ktx/context/daemon';
import type { KtxCliIo } from './cli-runtime.js';
import { createClackPromptAdapter } from './clack.js';
import {
installManagedPythonRuntime,
readManagedPythonRuntimeStatus,
@ -36,7 +36,7 @@ export interface ManagedPythonCommandRuntime {
export interface ManagedPythonCommandDeps {
readStatus?: (options: ManagedPythonRuntimeLayoutOptions) => Promise<ManagedPythonRuntimeStatus>;
installRuntime?: (options: ManagedPythonRuntimeInstallOptions) => Promise<ManagedPythonRuntimeInstallResult>;
confirmInstall?: (message: string) => Promise<boolean>;
confirmInstall?: (message: string, io: KtxCliIo) => Promise<boolean>;
}
export interface ManagedPythonCommandOptions extends ManagedPythonCommandDeps {
@ -69,16 +69,12 @@ function hasFeature(manifest: InstalledKtxRuntimeManifest, feature: KtxRuntimeFe
return manifest.features.includes(feature);
}
async function defaultConfirmInstall(message: string): Promise<boolean> {
if (process.stdin.isTTY !== true || process.stdout.isTTY !== true) {
async function defaultConfirmInstall(message: string, io: KtxCliIo): Promise<boolean> {
if (io.stdout.isTTY !== true) {
return false;
}
const response = await confirm({ message, initialValue: true });
if (isCancel(response)) {
cancel('Runtime installation cancelled.');
return false;
}
return response === true;
const prompts = createClackPromptAdapter();
return await prompts.confirm({ message, initialValue: true });
}
export async function ensureManagedPythonCommandRuntime(
@ -99,7 +95,7 @@ export async function ensureManagedPythonCommandRuntime(
if (options.installPolicy === 'prompt') {
const confirmInstall = options.confirmInstall ?? defaultConfirmInstall;
const confirmed = await confirmInstall(installPrompt(feature));
const confirmed = await confirmInstall(installPrompt(feature), options.io);
if (!confirmed) {
throw new Error(`KTX Python runtime installation was cancelled. Run: ${managedRuntimeInstallCommand(feature)}`);
}

View file

@ -5,9 +5,11 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
readManagedPythonDaemonStatus,
startManagedPythonDaemon,
stopAllManagedPythonDaemons,
stopManagedPythonDaemon,
type ManagedPythonDaemonChild,
type ManagedPythonDaemonFetch,
type ManagedPythonDaemonProcessInfo,
type ManagedPythonDaemonSpawn,
type ManagedPythonDaemonState,
} from './managed-python-daemon.js';
@ -105,6 +107,24 @@ function runningState(root: string, overrides: Partial<ManagedPythonDaemonState>
};
}
function daemonStatePath(root: string, version: string): string {
return join(root, 'runtime', version, 'daemon.json');
}
function runningStateForVersion(
root: string,
version: string,
overrides: Partial<ManagedPythonDaemonState> = {},
): ManagedPythonDaemonState {
return {
...runningState(root),
version,
stdoutLog: join(root, 'runtime', version, 'daemon.stdout.log'),
stderrLog: join(root, 'runtime', version, 'daemon.stderr.log'),
...overrides,
};
}
describe('managed Python daemon lifecycle', () => {
let tempDir: string;
@ -170,6 +190,41 @@ describe('managed Python daemon lifecycle', () => {
});
});
it('makes a final health probe before reporting startup failure', async () => {
const spawnDaemon = makeSpawn(5556);
const installRuntime = vi.fn(async () => installResult(tempDir));
const fetch = vi
.fn<ManagedPythonDaemonFetch>()
.mockRejectedValueOnce(new Error('fetch failed'))
.mockResolvedValueOnce({
ok: true,
status: 200,
json: async () => ({ status: 'healthy', version: '0.2.0' }),
text: async () => '',
});
const result = await startManagedPythonDaemon({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
features: ['core'],
installRuntime,
spawnDaemon,
fetch,
allocatePort: vi.fn(async () => 61234),
now: () => new Date('2026-05-11T00:00:00.000Z'),
startupTimeoutMs: 5,
pollIntervalMs: 20,
});
expect(result.status).toBe('started');
expect(fetch).toHaveBeenCalledTimes(2);
expect(JSON.parse(await readFile(layout(tempDir).daemonStatePath, 'utf8'))).toMatchObject({
pid: 5556,
port: 61234,
version: '0.2.0',
});
});
it('reuses a healthy daemon with the requested feature set', async () => {
await mkdir(layout(tempDir).versionDir, { recursive: true });
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
@ -236,4 +291,138 @@ describe('managed Python daemon lifecycle', () => {
expect(killProcess).toHaveBeenCalledWith(4242);
await expect(readFile(layout(tempDir).daemonStatePath, 'utf8')).rejects.toThrow();
});
it('stops all recorded daemon states across runtime versions and removes state files', async () => {
await mkdir(join(tempDir, 'runtime', '0.1.0'), { recursive: true });
await mkdir(join(tempDir, 'runtime', '0.2.0'), { recursive: true });
await writeFile(
daemonStatePath(tempDir, '0.1.0'),
`${JSON.stringify(runningStateForVersion(tempDir, '0.1.0', { pid: 1111, port: 61111 }), null, 2)}\n`,
);
await writeFile(
daemonStatePath(tempDir, '0.2.0'),
`${JSON.stringify(runningStateForVersion(tempDir, '0.2.0', { pid: 2222, port: 62222 }), null, 2)}\n`,
);
const alive = new Set([1111, 2222]);
const killProcess = vi.fn((pid: number) => {
alive.delete(pid);
});
const result = await stopAllManagedPythonDaemons({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
listProcesses: vi.fn(async () => []),
processAlive: vi.fn((pid) => alive.has(pid)),
killProcess,
stopGraceMs: 0,
});
expect(result.failed).toHaveLength(0);
expect(result.stopped.map((entry) => entry.pid).sort()).toEqual([1111, 2222]);
expect(killProcess).toHaveBeenCalledWith(1111, 'SIGTERM');
expect(killProcess).toHaveBeenCalledWith(2222, 'SIGTERM');
await expect(readFile(daemonStatePath(tempDir, '0.1.0'), 'utf8')).rejects.toThrow();
await expect(readFile(daemonStatePath(tempDir, '0.2.0'), 'utf8')).rejects.toThrow();
});
it('removes stale state when the recorded daemon process is no longer alive', async () => {
await mkdir(layout(tempDir).versionDir, { recursive: true });
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
const result = await stopAllManagedPythonDaemons({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
listProcesses: vi.fn(async () => []),
processAlive: vi.fn(() => false),
killProcess: vi.fn(),
stopGraceMs: 0,
});
expect(result.stopped).toHaveLength(0);
expect(result.stale.map((entry) => entry.pid)).toEqual([4242]);
await expect(readFile(layout(tempDir).daemonStatePath, 'utf8')).rejects.toThrow();
});
it('deduplicates a daemon found by state and process scan, preferring state metadata', async () => {
await mkdir(layout(tempDir).versionDir, { recursive: true });
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
const alive = new Set([4242]);
const killProcess = vi.fn((pid: number) => {
alive.delete(pid);
});
const result = await stopAllManagedPythonDaemons({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
listProcesses: vi.fn(async (): Promise<ManagedPythonDaemonProcessInfo[]> => [
{ pid: 4242, command: 'uv run ktx-daemon serve-http --host 127.0.0.1 --port 61234' },
]),
processAlive: vi.fn((pid) => alive.has(pid)),
killProcess,
stopGraceMs: 0,
});
expect(result.stopped).toHaveLength(1);
expect(result.stopped[0]).toMatchObject({
pid: 4242,
source: 'state',
url: 'http://127.0.0.1:58731',
});
expect(killProcess).toHaveBeenCalledTimes(1);
});
it('stops unrecorded ktx-daemon serve-http processes from process scan results', async () => {
const alive = new Set([3333, 5555]);
const killProcess = vi.fn((pid: number) => {
alive.delete(pid);
});
const result = await stopAllManagedPythonDaemons({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
listProcesses: vi.fn(async (): Promise<ManagedPythonDaemonProcessInfo[]> => [
{ pid: 3333, command: 'uv run ktx-daemon serve-http --host 127.0.0.1 --port 8765' },
{ pid: 4444, command: 'node server.js --port 8765' },
{ pid: 5555, command: 'grep ktx-daemon serve-http --port 8765' },
]),
processAlive: vi.fn((pid) => alive.has(pid)),
killProcess,
stopGraceMs: 0,
});
expect(result.failed).toHaveLength(0);
expect(result.stopped).toEqual([
expect.objectContaining({
pid: 3333,
source: 'process',
url: 'http://127.0.0.1:8765',
}),
]);
expect(killProcess).toHaveBeenCalledWith(3333, 'SIGTERM');
expect(killProcess).not.toHaveBeenCalledWith(4444, expect.anything());
expect(killProcess).not.toHaveBeenCalledWith(5555, expect.anything());
});
it('reports a failed stop when TERM and KILL leave a daemon running', async () => {
await mkdir(layout(tempDir).versionDir, { recursive: true });
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
const result = await stopAllManagedPythonDaemons({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
listProcesses: vi.fn(async () => []),
processAlive: vi.fn(() => true),
killProcess: vi.fn(),
stopGraceMs: 0,
});
expect(result.stopped).toHaveLength(0);
expect(result.failed).toEqual([
expect.objectContaining({
pid: 4242,
detail: 'Process still running after SIGKILL',
}),
]);
expect(await readFile(layout(tempDir).daemonStatePath, 'utf8')).toContain('"pid": 4242');
});
});

View file

@ -1,7 +1,9 @@
import { spawn } from 'node:child_process';
import { mkdir, open, readFile, rm, writeFile } from 'node:fs/promises';
import { execFile, spawn } from 'node:child_process';
import { mkdir, open, readdir, readFile, rm, writeFile } from 'node:fs/promises';
import { createServer } from 'node:net';
import { join } from 'node:path';
import { setTimeout as delay } from 'node:timers/promises';
import { promisify } from 'node:util';
import { z } from 'zod';
import {
installManagedPythonRuntime,
@ -44,6 +46,35 @@ export interface ManagedPythonDaemonStopResult {
state?: ManagedPythonDaemonState;
}
export interface ManagedPythonDaemonProcessInfo {
pid: number;
command: string;
}
export type ManagedPythonDaemonStopAllSource = 'state' | 'process';
export interface ManagedPythonDaemonStopAllEntry {
pid: number;
source: ManagedPythonDaemonStopAllSource;
url?: string;
health?: 'healthy' | 'unreachable';
version?: string;
command?: string;
statePaths: string[];
}
export interface ManagedPythonDaemonStopAllFailure extends ManagedPythonDaemonStopAllEntry {
detail: string;
}
export interface ManagedPythonDaemonStopAllResult {
runtimeRoot: string;
stopped: ManagedPythonDaemonStopAllEntry[];
stale: ManagedPythonDaemonStopAllEntry[];
failed: ManagedPythonDaemonStopAllFailure[];
scanErrors: string[];
}
export interface ManagedPythonDaemonChild {
pid?: number;
unref(): void;
@ -68,6 +99,8 @@ export type ManagedPythonDaemonFetch = (
text(): Promise<string>;
}>;
export type ManagedPythonDaemonKillProcess = (pid: number, signal?: NodeJS.Signals) => void;
export interface ManagedPythonDaemonStartOptions extends ManagedPythonRuntimeLayoutOptions {
features: KtxRuntimeFeature[];
force?: boolean;
@ -76,7 +109,7 @@ export interface ManagedPythonDaemonStartOptions extends ManagedPythonRuntimeLay
fetch?: ManagedPythonDaemonFetch;
allocatePort?: () => Promise<number>;
processAlive?: (pid: number) => boolean;
killProcess?: (pid: number) => void;
killProcess?: ManagedPythonDaemonKillProcess;
now?: () => Date;
startupTimeoutMs?: number;
pollIntervalMs?: number;
@ -89,9 +122,20 @@ export interface ManagedPythonDaemonStatusOptions extends ManagedPythonRuntimeLa
export interface ManagedPythonDaemonStopOptions extends ManagedPythonRuntimeLayoutOptions {
processAlive?: (pid: number) => boolean;
killProcess?: (pid: number) => void;
killProcess?: ManagedPythonDaemonKillProcess;
}
export interface ManagedPythonDaemonStopAllOptions extends ManagedPythonRuntimeLayoutOptions {
listProcesses?: () => Promise<ManagedPythonDaemonProcessInfo[]>;
processAlive?: (pid: number) => boolean;
killProcess?: ManagedPythonDaemonKillProcess;
stopGraceMs?: number;
pollIntervalMs?: number;
healthProbeMs?: number;
}
const execFileAsync = promisify(execFile);
const daemonStateSchema = z.object({
schemaVersion: z.literal(1),
pid: z.number().int().positive(),
@ -126,9 +170,9 @@ function defaultProcessAlive(pid: number): boolean {
}
}
function defaultKillProcess(pid: number): void {
function defaultKillProcess(pid: number, signal: NodeJS.Signals = 'SIGTERM'): void {
try {
process.kill(pid, 'SIGTERM');
process.kill(pid, signal);
} catch (error) {
const code = (error as { code?: unknown }).code;
if (code !== 'ESRCH') {
@ -273,6 +317,15 @@ async function waitForHealth(input: {
lastDetail = health.detail;
await delay(input.pollIntervalMs);
}
const finalHealth = await healthOk({
state: input.state,
cliVersion: input.cliVersion,
fetch: input.fetch,
});
if (finalHealth.ok) {
return;
}
lastDetail = finalHealth.detail;
throw new Error(`KTX Python daemon failed to start: ${lastDetail}. stderr: ${input.state.stderrLog}`);
}
@ -284,7 +337,7 @@ async function stopRecordedDaemon(input: {
layout: ManagedPythonRuntimeLayout;
state: ManagedPythonDaemonState;
processAlive: (pid: number) => boolean;
killProcess: (pid: number) => void;
killProcess: ManagedPythonDaemonKillProcess;
}): Promise<void> {
if (input.processAlive(input.state.pid)) {
input.killProcess(input.state.pid);
@ -292,6 +345,323 @@ async function stopRecordedDaemon(input: {
await removeState(input.layout);
}
function runtimeRootForStopAll(options: ManagedPythonRuntimeLayoutOptions): string {
return managedPythonRuntimeLayout(options).runtimeRoot;
}
async function removeStatePaths(paths: string[]): Promise<void> {
await Promise.all([...new Set(paths)].map((path) => rm(path, { force: true })));
}
interface ManagedPythonDaemonStopCandidate {
pid: number;
source: ManagedPythonDaemonStopAllSource;
host?: string;
port?: number;
version?: string;
command?: string;
statePaths: string[];
}
function candidateUrl(candidate: ManagedPythonDaemonStopCandidate): string | undefined {
if (!candidate.host || !candidate.port) {
return undefined;
}
return `http://${candidate.host}:${candidate.port}`;
}
function candidateEntry(candidate: ManagedPythonDaemonStopCandidate): ManagedPythonDaemonStopAllEntry {
return {
pid: candidate.pid,
source: candidate.source,
...(candidateUrl(candidate) ? { url: candidateUrl(candidate) } : {}),
...(candidate.version ? { version: candidate.version } : {}),
...(candidate.command ? { command: candidate.command } : {}),
statePaths: [...candidate.statePaths],
};
}
async function probeCandidateHealth(
candidate: ManagedPythonDaemonStopCandidate,
timeoutMs: number,
): Promise<'healthy' | 'unreachable' | undefined> {
const url = candidateUrl(candidate);
if (!url) {
return undefined;
}
const controller = new AbortController();
const timeout = setTimeout(() => {
controller.abort();
}, timeoutMs);
try {
const response = await fetch(`${url}/health`, { signal: controller.signal });
if (!response.ok) {
return 'unreachable';
}
const body = (await response.json()) as unknown;
if (!body || typeof body !== 'object' || Array.isArray(body)) {
return 'unreachable';
}
return (body as Record<string, unknown>).status === 'healthy' ? 'healthy' : 'unreachable';
} catch {
return 'unreachable';
} finally {
clearTimeout(timeout);
}
}
async function readStateCandidates(runtimeRoot: string): Promise<ManagedPythonDaemonStopCandidate[]> {
let entries;
try {
entries = await readdir(runtimeRoot, { withFileTypes: true });
} catch (error) {
const code = (error as { code?: unknown }).code;
if (code === 'ENOENT') {
return [];
}
throw error;
}
const candidates: ManagedPythonDaemonStopCandidate[] = [];
for (const entry of entries) {
if (!entry.isDirectory()) {
continue;
}
const statePath = join(runtimeRoot, entry.name, 'daemon.json');
let state: ManagedPythonDaemonState | undefined;
try {
state = await readState(statePath);
} catch {
continue;
}
if (!state) {
continue;
}
candidates.push({
pid: state.pid,
source: 'state',
host: state.host,
port: state.port,
version: state.version,
statePaths: [statePath],
});
}
return candidates;
}
function tokenizeCommand(command: string): string[] {
const tokens: string[] = [];
for (const match of command.matchAll(/"([^"]*)"|'([^']*)'|(\S+)/g)) {
tokens.push(match[1] ?? match[2] ?? match[3] ?? '');
}
return tokens;
}
function executableName(token: string): string {
return token.split(/[\\/]/).at(-1) ?? token;
}
function isKtxDaemonExecutable(token: string): boolean {
return executableName(token) === 'ktx-daemon' || executableName(token) === 'ktx-daemon.exe';
}
function normalizedExecutableName(token: string): string {
return executableName(token).replace(/\.exe$/i, '').toLowerCase();
}
function hasUvRunPrefix(tokens: string[], daemonIndex: number): boolean {
return normalizedExecutableName(tokens[0] ?? '') === 'uv' && tokens.slice(1, daemonIndex).includes('run');
}
function isPythonExecutable(token: string): boolean {
const name = normalizedExecutableName(token);
return name === 'python' || name === 'python3';
}
function hasPythonModulePrefix(tokens: string[], moduleFlagIndex: number): boolean {
if (moduleFlagIndex === 1 && isPythonExecutable(tokens[0] ?? '')) {
return true;
}
return (
normalizedExecutableName(tokens[0] ?? '') === 'uv' &&
tokens.slice(1, moduleFlagIndex).includes('run') &&
tokens.some((token, index) => index < moduleFlagIndex && isPythonExecutable(token))
);
}
function isKtxDaemonServeHttp(tokens: string[]): boolean {
for (let index = 0; index < tokens.length; index += 1) {
if (
isKtxDaemonExecutable(tokens[index] ?? '') &&
tokens[index + 1] === 'serve-http' &&
(index === 0 || hasUvRunPrefix(tokens, index))
) {
return true;
}
if (
tokens[index] === '-m' &&
tokens[index + 1] === 'ktx_daemon' &&
tokens[index + 2] === 'serve-http' &&
hasPythonModulePrefix(tokens, index)
) {
return true;
}
}
return false;
}
function parseCommandOption(tokens: string[], option: string): string | undefined {
for (let index = 0; index < tokens.length; index += 1) {
const token = tokens[index];
if (token === option) {
return tokens[index + 1];
}
if (token?.startsWith(`${option}=`)) {
return token.slice(option.length + 1);
}
}
return undefined;
}
function processCandidate(processInfo: ManagedPythonDaemonProcessInfo): ManagedPythonDaemonStopCandidate | undefined {
const tokens = tokenizeCommand(processInfo.command);
if (!isKtxDaemonServeHttp(tokens)) {
return undefined;
}
const host = parseCommandOption(tokens, '--host') ?? '127.0.0.1';
const rawPort = parseCommandOption(tokens, '--port');
const parsedPort = rawPort ? Number.parseInt(rawPort, 10) : 8765;
const port = Number.isInteger(parsedPort) && parsedPort >= 1 && parsedPort <= 65535 ? parsedPort : 8765;
return {
pid: processInfo.pid,
source: 'process',
host,
port,
command: processInfo.command,
statePaths: [],
};
}
function mergeCandidates(candidates: ManagedPythonDaemonStopCandidate[]): ManagedPythonDaemonStopCandidate[] {
const byPid = new Map<number, ManagedPythonDaemonStopCandidate>();
for (const candidate of candidates) {
const existing = byPid.get(candidate.pid);
if (!existing) {
byPid.set(candidate.pid, { ...candidate, statePaths: [...candidate.statePaths] });
continue;
}
existing.statePaths.push(...candidate.statePaths);
if (existing.source === 'process' && candidate.source === 'state') {
byPid.set(candidate.pid, {
...candidate,
statePaths: [...new Set([...existing.statePaths, ...candidate.statePaths])],
});
} else {
existing.statePaths = [...new Set(existing.statePaths)];
}
}
return [...byPid.values()].sort((left, right) => left.pid - right.pid);
}
function parsePosixProcessList(output: string): ManagedPythonDaemonProcessInfo[] {
const processes: ManagedPythonDaemonProcessInfo[] = [];
for (const line of output.split(/\r?\n/)) {
const match = line.match(/^\s*(\d+)\s+(.+)$/);
if (!match) {
continue;
}
processes.push({ pid: Number.parseInt(match[1], 10), command: match[2] });
}
return processes;
}
function parseWindowsProcessList(output: string): ManagedPythonDaemonProcessInfo[] {
if (!output.trim()) {
return [];
}
const parsed = JSON.parse(output) as unknown;
const records = Array.isArray(parsed) ? parsed : [parsed];
const processes: ManagedPythonDaemonProcessInfo[] = [];
for (const record of records) {
if (!record || typeof record !== 'object') {
continue;
}
const value = record as Record<string, unknown>;
const pid = value.ProcessId;
const command = value.CommandLine;
if (typeof pid === 'number' && typeof command === 'string' && command.length > 0) {
processes.push({ pid, command });
}
}
return processes;
}
async function defaultListProcesses(platform: NodeJS.Platform = process.platform): Promise<ManagedPythonDaemonProcessInfo[]> {
if (platform === 'win32') {
const command = [
'Get-CimInstance Win32_Process',
'| Where-Object { $_.CommandLine -ne $null }',
'| Select-Object ProcessId,CommandLine',
'| ConvertTo-Json -Compress',
].join(' ');
const { stdout } = await execFileAsync('powershell.exe', ['-NoProfile', '-Command', command], {
encoding: 'utf8',
maxBuffer: 10 * 1024 * 1024,
});
return parseWindowsProcessList(stdout);
}
const { stdout } = await execFileAsync('ps', ['-axo', 'pid=,command='], {
encoding: 'utf8',
maxBuffer: 10 * 1024 * 1024,
});
return parsePosixProcessList(stdout);
}
async function waitUntilStopped(input: {
pid: number;
processAlive: (pid: number) => boolean;
timeoutMs: number;
pollIntervalMs: number;
}): Promise<boolean> {
const deadline = Date.now() + input.timeoutMs;
do {
if (!input.processAlive(input.pid)) {
return true;
}
if (Date.now() >= deadline) {
break;
}
await delay(input.pollIntervalMs);
} while (Date.now() <= deadline);
return !input.processAlive(input.pid);
}
async function discoverStopAllCandidates(
options: ManagedPythonDaemonStopAllOptions,
): Promise<{
runtimeRoot: string;
candidates: ManagedPythonDaemonStopCandidate[];
scanErrors: string[];
}> {
const runtimeRoot = runtimeRootForStopAll(options);
const stateCandidates = await readStateCandidates(runtimeRoot);
const scanErrors: string[] = [];
let processCandidates: ManagedPythonDaemonStopCandidate[] = [];
try {
const processes = await (options.listProcesses ?? defaultListProcesses)();
processCandidates = processes.flatMap((processInfo) => {
const candidate = processCandidate(processInfo);
return candidate ? [candidate] : [];
});
} catch (error) {
scanErrors.push(error instanceof Error ? error.message : String(error));
}
return {
runtimeRoot,
candidates: mergeCandidates([...stateCandidates, ...processCandidates]),
scanErrors,
};
}
export async function startManagedPythonDaemon(
options: ManagedPythonDaemonStartOptions,
): Promise<ManagedPythonDaemonStartResult> {
@ -395,3 +765,63 @@ export async function stopManagedPythonDaemon(
});
return { status: 'stopped', layout, state };
}
export async function stopAllManagedPythonDaemons(
options: ManagedPythonDaemonStopAllOptions,
): Promise<ManagedPythonDaemonStopAllResult> {
const processAlive = options.processAlive ?? defaultProcessAlive;
const killProcess = options.killProcess ?? defaultKillProcess;
const stopGraceMs = options.stopGraceMs ?? 500;
const pollIntervalMs = options.pollIntervalMs ?? 50;
const healthProbeMs = options.healthProbeMs ?? 100;
const discovery = await discoverStopAllCandidates(options);
const stopped: ManagedPythonDaemonStopAllEntry[] = [];
const stale: ManagedPythonDaemonStopAllEntry[] = [];
const failed: ManagedPythonDaemonStopAllFailure[] = [];
for (const candidate of discovery.candidates) {
const health = await probeCandidateHealth(candidate, healthProbeMs);
const entry = { ...candidateEntry(candidate), ...(health ? { health } : {}) };
if (!processAlive(candidate.pid)) {
await removeStatePaths(candidate.statePaths);
stale.push(entry);
continue;
}
try {
killProcess(candidate.pid, 'SIGTERM');
if (
!(await waitUntilStopped({
pid: candidate.pid,
processAlive,
timeoutMs: stopGraceMs,
pollIntervalMs,
}))
) {
killProcess(candidate.pid, 'SIGKILL');
if (
!(await waitUntilStopped({
pid: candidate.pid,
processAlive,
timeoutMs: stopGraceMs,
pollIntervalMs,
}))
) {
failed.push({ ...entry, detail: 'Process still running after SIGKILL' });
continue;
}
}
await removeStatePaths(candidate.statePaths);
stopped.push(entry);
} catch (error) {
failed.push({ ...entry, detail: error instanceof Error ? error.message : String(error) });
}
}
return {
runtimeRoot: discovery.runtimeRoot,
stopped,
stale,
failed,
scanErrors: discovery.scanErrors,
};
}

View file

@ -161,6 +161,14 @@ describe('verifyRuntimeAsset', () => {
await expect(verifyRuntimeAsset({ assetDir })).rejects.toThrow(/Unsafe runtime wheel filename/);
});
it('reports the source-checkout artifact command when the bundled manifest is missing', async () => {
const assetDir = join(tempDir, 'packages', 'cli', 'assets', 'python');
await expect(verifyRuntimeAsset({ assetDir })).rejects.toThrow(
/Missing bundled Python runtime manifest.*pnpm run artifacts:build/s,
);
});
});
describe('installManagedPythonRuntime', () => {
@ -210,6 +218,30 @@ describe('installManagedPythonRuntime', () => {
expect(manifest.python.daemonExecutable).toBe(result.layout.daemonPath);
});
it('disables repo uv config for managed runtime uv commands', async () => {
const { assetDir } = await writeAsset(tempDir, 'core-wheel');
const commands: Array<{ command: string; args: string[]; env?: NodeJS.ProcessEnv }> = [];
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args, options) => {
commands.push({ command, args, env: options?.env });
return { stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.11.13\n' : '', stderr: '' };
});
await installManagedPythonRuntime({
cliVersion: '0.2.0',
runtimeRoot: join(tempDir, 'runtime'),
assetDir,
env: { PATH: '/opt/homebrew/bin', UV_NO_CONFIG: '0' },
features: ['core'],
exec,
});
expect(commands.map((call) => [call.command, call.args[0], call.env?.UV_NO_CONFIG, call.env?.PATH])).toEqual([
['uv', '--version', '1', '/opt/homebrew/bin'],
['uv', 'venv', '1', '/opt/homebrew/bin'],
['uv', 'pip', '1', '/opt/homebrew/bin'],
]);
});
it('installs the local-embeddings extra when requested', async () => {
const { assetDir } = await writeAsset(tempDir, 'embedding-wheel');
const commands: Array<{ command: string; args: string[] }> = [];

View file

@ -186,9 +186,28 @@ async function readJsonFile(path: string): Promise<unknown> {
return JSON.parse(await readFile(path, 'utf8')) as unknown;
}
function isErrnoException(error: unknown, code: string): boolean {
return typeof error === 'object' && error !== null && 'code' in error && error.code === code;
}
export async function verifyRuntimeAsset(input: { assetDir: string }): Promise<ManagedRuntimeAsset> {
const manifestPath = join(input.assetDir, 'manifest.json');
const manifest = runtimeAssetManifestSchema.parse(await readJsonFile(manifestPath));
let manifestData: unknown;
try {
manifestData = await readJsonFile(manifestPath);
} catch (error) {
if (isErrnoException(error, 'ENOENT')) {
throw new Error(
[
`Missing bundled Python runtime manifest: ${manifestPath}`,
'In a source checkout, build the local runtime assets with: pnpm run artifacts:build',
'Then retry the runtime-backed KTX command.',
].join('\n'),
);
}
throw error;
}
const manifest = runtimeAssetManifestSchema.parse(manifestData);
assertSafeWheelFilename(manifest.wheel.file);
const wheelPath = join(input.assetDir, manifest.wheel.file);
const wheel = await readFile(wheelPath);
@ -243,10 +262,11 @@ async function runLogged(input: {
command: string;
args: string[];
cwd?: string;
env?: NodeJS.ProcessEnv;
}): Promise<{ stdout: string; stderr: string }> {
await appendFile(input.logPath, `$ ${input.command} ${input.args.join(' ')}\n`);
try {
const result = await input.exec(input.command, input.args, { cwd: input.cwd });
const result = await input.exec(input.command, input.args, { cwd: input.cwd, env: input.env });
if (result.stdout) {
await appendFile(input.logPath, result.stdout.endsWith('\n') ? result.stdout : `${result.stdout}\n`);
}
@ -266,9 +286,13 @@ async function runLogged(input: {
}
}
async function ensureUv(exec: ManagedPythonRuntimeExec): Promise<string> {
function managedRuntimeUvEnv(baseEnv: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
return { ...baseEnv, UV_NO_CONFIG: '1' };
}
async function ensureUv(exec: ManagedPythonRuntimeExec, env?: NodeJS.ProcessEnv): Promise<string> {
try {
const result = await exec('uv', ['--version']);
const result = await exec('uv', ['--version'], { env });
return result.stdout.trim() || 'uv available';
} catch {
throw new Error(MISSING_UV_RUNTIME_INSTALL_MESSAGE);
@ -282,6 +306,7 @@ export async function installManagedPythonRuntime(
const exec = options.exec ?? defaultExec;
const features = normalizeFeatures(options.features);
const asset = await verifyRuntimeAsset({ assetDir: layout.assetDir });
const uvEnv = managedRuntimeUvEnv(options.env ?? process.env);
const existing = await readInstalledManifest(layout.manifestPath);
if (
options.force !== true &&
@ -298,14 +323,21 @@ export async function installManagedPythonRuntime(
await rm(layout.versionDir, { recursive: true, force: true });
await mkdir(layout.versionDir, { recursive: true });
await writeFile(layout.installLogPath, '');
await ensureUv(exec);
await runLogged({ exec, logPath: layout.installLogPath, command: 'uv', args: ['venv', layout.venvDir] });
await ensureUv(exec, uvEnv);
await runLogged({
exec,
logPath: layout.installLogPath,
command: 'uv',
args: ['venv', layout.venvDir],
env: uvEnv,
});
const wheelSpec = features.includes('local-embeddings') ? `${asset.wheelPath}[local-embeddings]` : asset.wheelPath;
await runLogged({
exec,
logPath: layout.installLogPath,
command: 'uv',
args: ['pip', 'install', '--python', layout.pythonPath, wheelSpec],
env: uvEnv,
});
const manifest: InstalledKtxRuntimeManifest = {
@ -371,7 +403,7 @@ export async function doctorManagedPythonRuntime(
const exec = options.exec ?? defaultExec;
const checks: ManagedPythonRuntimeDoctorCheck[] = [];
try {
const version = await ensureUv(exec);
const version = await ensureUv(exec, managedRuntimeUvEnv(options.env ?? process.env));
checks.push(check('pass', { id: 'uv', label: 'uv', detail: version }));
} catch (error) {
checks.push(

View file

@ -1,5 +1,6 @@
import { describe, expect, it, vi } from 'vitest';
import type {
ManagedPythonDaemonStopAllResult,
ManagedPythonDaemonStartResult,
ManagedPythonDaemonStopResult,
} from './managed-python-daemon.js';
@ -199,13 +200,63 @@ describe('runKtxRuntime', () => {
})),
};
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0' }, io.io, deps)).resolves.toBe(0);
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0', all: false }, io.io, deps)).resolves.toBe(0);
expect(deps.stopDaemon).toHaveBeenCalledWith({ cliVersion: '0.2.0' });
expect(io.stdout()).toContain('Stopped KTX Python daemon');
expect(io.stdout()).toContain('pid: 4242');
});
it('stops all discovered Python daemons and reports the summary', async () => {
const io = makeIo();
const deps: KtxRuntimeDeps = {
stopAllDaemons: vi.fn(async (): Promise<ManagedPythonDaemonStopAllResult> => ({
runtimeRoot: '/runtime',
stopped: [
{ pid: 4242, source: 'state', url: 'http://127.0.0.1:61234', statePaths: ['/runtime/0.2.0/daemon.json'] },
{ pid: 5252, source: 'process', url: 'http://127.0.0.1:8765', statePaths: [] },
],
stale: [],
failed: [],
scanErrors: [],
})),
};
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0', all: true }, io.io, deps)).resolves.toBe(0);
expect(deps.stopAllDaemons).toHaveBeenCalledWith({ cliVersion: '0.2.0' });
expect(io.stdout()).toContain('Stopped 2 KTX Python daemons');
expect(io.stdout()).toContain('pid: 4242 source: state url: http://127.0.0.1:61234');
expect(io.stdout()).toContain('pid: 5252 source: process url: http://127.0.0.1:8765');
});
it('returns failure when stop all cannot stop every daemon', async () => {
const io = makeIo();
const deps: KtxRuntimeDeps = {
stopAllDaemons: vi.fn(async (): Promise<ManagedPythonDaemonStopAllResult> => ({
runtimeRoot: '/runtime',
stopped: [],
stale: [],
failed: [
{
pid: 4242,
source: 'state',
url: 'http://127.0.0.1:61234',
statePaths: ['/runtime/0.2.0/daemon.json'],
detail: 'Process still running after SIGKILL',
},
],
scanErrors: ['ps failed'],
})),
};
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0', all: true }, io.io, deps)).resolves.toBe(1);
expect(io.stderr()).toContain('Stopped 0 KTX Python daemons; failed 1');
expect(io.stderr()).toContain('pid: 4242 source: state url: http://127.0.0.1:61234');
expect(io.stderr()).toContain('process scan: ps failed');
});
it('prints runtime status as JSON', async () => {
const io = makeIo();
const deps: KtxRuntimeDeps = {

View file

@ -1,7 +1,9 @@
import type { KtxCliIo } from './cli-runtime.js';
import {
stopAllManagedPythonDaemons,
startManagedPythonDaemon,
stopManagedPythonDaemon,
type ManagedPythonDaemonStopAllResult,
type ManagedPythonDaemonStartResult,
type ManagedPythonDaemonStopResult,
} from './managed-python-daemon.js';
@ -22,7 +24,7 @@ import {
export type KtxRuntimeArgs =
| { command: 'install'; cliVersion: string; feature: KtxRuntimeFeature; force: boolean }
| { command: 'start'; cliVersion: string; feature: KtxRuntimeFeature; force: boolean }
| { command: 'stop'; cliVersion: string }
| { command: 'stop'; cliVersion: string; all: boolean }
| { command: 'status'; cliVersion: string; json: boolean }
| { command: 'doctor'; cliVersion: string; json: boolean }
| { command: 'prune'; cliVersion: string; dryRun: boolean; yes: boolean };
@ -35,6 +37,7 @@ export interface KtxRuntimeDeps {
force?: boolean;
}) => Promise<ManagedPythonDaemonStartResult>;
stopDaemon?: (options: { cliVersion: string }) => Promise<ManagedPythonDaemonStopResult>;
stopAllDaemons?: (options: { cliVersion: string }) => Promise<ManagedPythonDaemonStopAllResult>;
readStatus?: (options: ManagedPythonRuntimeLayoutOptions) => Promise<ManagedPythonRuntimeStatus>;
doctorRuntime?: (options: ManagedPythonRuntimeLayoutOptions) => Promise<ManagedPythonRuntimeDoctorCheck[]>;
pruneRuntime?: (options: {
@ -81,6 +84,58 @@ function writeDaemonStop(io: KtxCliIo, result: ManagedPythonDaemonStopResult): v
io.stdout.write(`state: ${result.layout.daemonStatePath}\n`);
}
function writeStopAllEntry(io: KtxCliIo, entry: { pid: number; source: string; url?: string; health?: string; detail?: string }): void {
io.stdout.write(
`pid: ${entry.pid} source: ${entry.source}${entry.url ? ` url: ${entry.url}` : ''}${
entry.health ? ` health: ${entry.health}` : ''
}${
entry.detail ? ` detail: ${entry.detail}` : ''
}\n`,
);
}
function writeDaemonStopAll(io: KtxCliIo, result: ManagedPythonDaemonStopAllResult): number {
const failed = result.failed.length + result.scanErrors.length;
if (
result.stopped.length === 0 &&
result.stale.length === 0 &&
result.failed.length === 0 &&
result.scanErrors.length === 0
) {
io.stdout.write('No KTX Python daemons found\n');
return 0;
}
if (failed === 0) {
io.stdout.write(`Stopped ${result.stopped.length} KTX Python daemons\n`);
if (result.stale.length > 0) {
io.stdout.write(`Cleaned ${result.stale.length} stale daemon states\n`);
}
for (const entry of result.stopped) {
writeStopAllEntry(io, entry);
}
for (const entry of result.stale) {
writeStopAllEntry(io, entry);
}
return 0;
}
io.stderr.write(
`Stopped ${result.stopped.length} KTX Python daemons; failed ${result.failed.length}${
result.stale.length > 0 ? `; cleaned stale ${result.stale.length}` : ''
}\n`,
);
for (const entry of result.failed) {
io.stderr.write(
`pid: ${entry.pid} source: ${entry.source}${entry.url ? ` url: ${entry.url}` : ''}${
entry.health ? ` health: ${entry.health}` : ''
} detail: ${entry.detail}\n`,
);
}
for (const error of result.scanErrors) {
io.stderr.write(`process scan: ${error}\n`);
}
return 1;
}
function writeStatus(io: KtxCliIo, status: ManagedPythonRuntimeStatus): void {
io.stdout.write('KTX Python runtime\n');
io.stdout.write(`status: ${status.kind}\n`);
@ -142,10 +197,16 @@ export async function runKtxRuntime(
return 0;
}
if (args.command === 'stop') {
const stopDaemon = deps.stopDaemon ?? stopManagedPythonDaemon;
const result = await stopDaemon({ cliVersion: args.cliVersion });
writeDaemonStop(io, result);
return 0;
if (args.all) {
const stopAllDaemons = deps.stopAllDaemons ?? stopAllManagedPythonDaemons;
const result = await stopAllDaemons({ cliVersion: args.cliVersion });
return writeDaemonStopAll(io, result);
} else {
const stopDaemon = deps.stopDaemon ?? stopManagedPythonDaemon;
const result = await stopDaemon({ cliVersion: args.cliVersion });
writeDaemonStop(io, result);
return 0;
}
}
if (args.command === 'status') {
const readStatus = deps.readStatus ?? readManagedPythonRuntimeStatus;

View file

@ -573,6 +573,32 @@ describe('runKtxScan', () => {
expect(io.stdout()).toContain('\n[90%] Building embeddings 1/4 batches\n');
});
it('scales nested progress phases by the parent phase weight', async () => {
const io = makeIo({ isTTY: true });
const previousCi = process.env.CI;
delete process.env.CI;
try {
const progress = createCliScanProgress(io.io);
await progress.update(0.82, 'Enriching schema metadata');
const enrichmentProgress = progress.startPhase(0.18);
await enrichmentProgress.update(0.05, 'Loaded schema snapshot with 56 tables');
const descriptionProgress = enrichmentProgress.startPhase(0.45);
await descriptionProgress.update(37 / 56, 'Generating descriptions 37/56 tables', { transient: true });
await descriptionProgress.update(1, 'Generated descriptions for 56 tables');
} finally {
if (previousCi === undefined) {
delete process.env.CI;
} else {
process.env.CI = previousCi;
}
}
expect(io.stdout()).toContain('\r[88%] Generating descriptions 37/56 tables');
expect(io.stdout()).toContain('\n[91%] Generated descriptions for 56 tables\n');
expect(io.stdout()).not.toContain('[100%] Generating descriptions 37/56 tables');
});
it('flushes transient TTY progress messages before printing scan failures', async () => {
await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
const runLocalScan = vi.fn(async (input: RunLocalScanOptions): Promise<LocalScanRunResult> => {

View file

@ -527,7 +527,7 @@ export function createCliScanProgress(
io.stdout.write(`${line}\n`);
},
startPhase(phaseWeight: number) {
return createCliScanProgress(io, state, state.progress, phaseWeight);
return createCliScanProgress(io, state, state.progress, weight * phaseWeight);
},
flush() {
if (!shouldWrite || !state.hasPendingTransient) {

View file

@ -1305,6 +1305,7 @@ describe('setup databases step', () => {
expect(config.connections.warehouse.historicSql).not.toHaveProperty('redactionPatterns');
expect(config.connections.warehouse.historicSql).not.toHaveProperty(legacyHistoricSqlServiceAccountPatternsKey);
expect(config.ingest.adapters).toContain('historic-sql');
expect(config.ingest.workUnits.maxConcurrency).toBe(6);
expect(io.stdout()).toContain('Historic SQL probe...');
expect(io.stdout()).toContain('pg_stat_statements ready');
});

View file

@ -14,6 +14,8 @@ import { runKtxScan } from './scan.js';
import { withSetupInterruptConfirmation } from './setup-interrupt.js';
import { writeProjectLocalSecretReference } from './setup-secrets.js';
const HISTORIC_SQL_WORK_UNIT_MAX_CONCURRENCY = 6;
export type KtxSetupDatabaseDriver =
| 'sqlite'
| 'postgres'
@ -930,7 +932,7 @@ async function writeConnectionConfig(input: {
? (input.connection.historicSql as Record<string, unknown>)
: null;
if (historicSql?.enabled === true) {
await ensureHistoricSqlAdapterEnabled(input.projectDir);
await ensureHistoricSqlIngestDefaults(input.projectDir);
}
}
@ -1057,9 +1059,19 @@ async function maybeConfigureSchemaScope(input: {
return true;
}
async function ensureHistoricSqlAdapterEnabled(projectDir: string): Promise<void> {
async function ensureHistoricSqlIngestDefaults(projectDir: string): Promise<void> {
const project = await loadKtxProject({ projectDir });
if (project.config.ingest.adapters.includes('historic-sql')) {
const adapters = project.config.ingest.adapters.includes('historic-sql')
? project.config.ingest.adapters
: [...project.config.ingest.adapters, 'historic-sql'];
const maxConcurrency = Math.max(
project.config.ingest.workUnits.maxConcurrency,
HISTORIC_SQL_WORK_UNIT_MAX_CONCURRENCY,
);
if (
adapters === project.config.ingest.adapters &&
maxConcurrency === project.config.ingest.workUnits.maxConcurrency
) {
return;
}
await writeFile(
@ -1068,7 +1080,11 @@ async function ensureHistoricSqlAdapterEnabled(projectDir: string): Promise<void
...project.config,
ingest: {
...project.config.ingest,
adapters: [...project.config.ingest.adapters, 'historic-sql'],
adapters,
workUnits: {
...project.config.ingest.workUnits,
maxConcurrency,
},
},
}),
'utf-8',

View file

@ -676,4 +676,53 @@ describe('setup Anthropic model step', () => {
).resolves.toMatchObject({ status: 'ready' });
expect(healthCheck).not.toHaveBeenCalled();
});
it.each([
{
backend: 'vertex',
providerLines: [' backend: vertex', ' vertex:', ' project: kaelio-dev', ' location: us-east5'],
model: 'claude-sonnet-4-6',
},
{
backend: 'gateway',
providerLines: [' backend: gateway', ' gateway:', ' api_key: env:AI_GATEWAY_API_KEY'],
model: 'anthropic/claude-sonnet-4-6',
},
])('preserves already configured $backend llm setup without asking for Anthropic credentials', async (fixture) => {
await writeFile(
join(tempDir, 'ktx.yaml'),
[
'project: warehouse',
'setup:',
' database_connection_ids: []',
' completed_steps:',
' - project',
' - llm',
'connections: {}',
'llm:',
' provider:',
...fixture.providerLines,
' models:',
` default: ${fixture.model}`,
'ingest:',
' embeddings:',
' backend: deterministic',
' model: deterministic',
' dimensions: 8',
].join('\n'),
'utf-8',
);
const healthCheck = vi.fn(async () => ({ ok: true as const }));
const io = makeIo();
await expect(
runKtxSetupAnthropicModelStep({ projectDir: tempDir, inputMode: 'disabled', skipLlm: false }, io.io, {
healthCheck,
}),
).resolves.toMatchObject({ status: 'ready' });
expect(healthCheck).not.toHaveBeenCalled();
expect(io.stdout()).toContain(`LLM ready: yes (${fixture.model})`);
expect(io.stderr()).not.toContain('Anthropic');
});
});

View file

@ -1,5 +1,6 @@
import { writeFile } from 'node:fs/promises';
import { cancel, isCancel, password, select, text } from '@clack/prompts';
import { resolveLocalKtxLlmConfig } from '@ktx/context';
import { resolveKtxConfigReference } from '@ktx/context/core';
import {
type KtxProjectConfig,
@ -170,13 +171,26 @@ export async function fetchAnthropicModels(
return models.map((item, index) => ({ ...item, recommended: index === Math.max(recommendedIndex, 0) }));
}
function hasCompletedLlm(config: KtxProjectConfig): boolean {
return (
config.setup?.completed_steps.includes('llm') === true &&
config.llm.provider.backend === 'anthropic' &&
typeof config.llm.models.default === 'string' &&
config.llm.models.default.length > 0
);
export function isKtxSetupLlmConfigReady(config: KtxProjectLlmConfig): boolean {
let resolved: KtxLlmConfig | null;
try {
resolved = resolveLocalKtxLlmConfig(config, process.env);
} catch {
return false;
}
if (!resolved) {
return false;
}
if (resolved.backend === 'vertex') {
return typeof resolved.vertex?.location === 'string' && resolved.vertex.location.trim().length > 0;
}
return resolved.backend === 'anthropic' || resolved.backend === 'gateway';
}
function hasUsableConfiguredLlm(config: KtxProjectConfig): boolean {
return isKtxSetupLlmConfigReady(config.llm);
}
function buildProjectLlmConfig(
@ -386,7 +400,7 @@ export async function runKtxSetupAnthropicModelStep(
const project = await loadKtxProject({ projectDir: args.projectDir });
if (
args.forcePrompt !== true &&
hasCompletedLlm(project.config) &&
hasUsableConfiguredLlm(project.config) &&
!args.anthropicApiKeyEnv &&
!args.anthropicApiKeyFile &&
!args.anthropicModel

View file

@ -5,6 +5,7 @@ import { join } from 'node:path';
import { promisify } from 'node:util';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { localFakeBundleReport, persistLocalBundleReport } from './ingest.test-utils.js';
import { contextBuildCommands, writeKtxSetupContextState } from './setup-context.js';
import { runDemoTour } from './setup-demo-tour.js';
import { readKtxSetupStatus, runKtxSetup } from './setup.js';
@ -91,6 +92,38 @@ describe('setup status', () => {
});
});
it.each([
{
backend: 'vertex',
providerLines: [' backend: vertex', ' vertex:', ' project: kaelio-dev', ' location: us-east5'],
model: 'claude-sonnet-4-6',
},
{
backend: 'gateway',
providerLines: [' backend: gateway', ' gateway:', ' api_key: env:AI_GATEWAY_API_KEY'],
model: 'anthropic/claude-sonnet-4-6',
},
])('reports configured $backend llm backends as setup-ready', async (fixture) => {
await mkdir(tempDir, { recursive: true });
await writeFile(
join(tempDir, 'ktx.yaml'),
[
'project: revenue',
'llm:',
' provider:',
...fixture.providerLines,
' models:',
` default: ${fixture.model}`,
'connections: {}',
].join('\n'),
'utf-8',
);
await expect(readKtxSetupStatus(tempDir)).resolves.toMatchObject({
llm: { backend: fixture.backend, ready: true, model: fixture.model },
});
});
it('uses setup database connection ids when present', async () => {
await writeFile(
join(tempDir, 'ktx.yaml'),
@ -283,6 +316,62 @@ describe('setup status', () => {
});
});
it('reports Vertex LLM and context ready after a successful Metabase ingest report', async () => {
await writeFile(
join(tempDir, 'ktx.yaml'),
[
'project: revenue',
'setup:',
' database_connection_ids:',
' - warehouse',
' completed_steps:',
' - project',
' - databases',
' - sources',
'connections:',
' warehouse:',
' driver: postgres',
' url: env:DATABASE_URL',
' metabase:',
' driver: metabase',
' url: env:METABASE_URL',
' api_key_ref: env:METABASE_API_KEY',
' warehouse_connection_id: warehouse',
'llm:',
' provider:',
' backend: vertex',
' vertex:',
' project: kaelio-dev',
' location: us-east5',
' models:',
' default: claude-sonnet-4-6',
'ingest:',
' embeddings:',
' backend: deterministic',
' model: deterministic',
' dimensions: 8',
'',
].join('\n'),
'utf-8',
);
await persistLocalBundleReport(
tempDir,
localFakeBundleReport('metabase-job-1', {
connectionId: 'warehouse',
sourceKey: 'metabase',
}),
);
const status = await readKtxSetupStatus(tempDir);
const io = makeIo();
await expect(runKtxSetup({ command: 'status', projectDir: tempDir, json: false }, io.io)).resolves.toBe(0);
expect(status.llm).toMatchObject({ backend: 'vertex', ready: true, model: 'claude-sonnet-4-6' });
expect(status.context).toMatchObject({ ready: true, status: 'completed' });
expect(io.stdout()).toContain('LLM ready: yes (claude-sonnet-4-6)');
expect(io.stdout()).toContain('KTX context built: yes');
});
it('prints plain and JSON setup status', async () => {
const plainIo = makeIo();
const jsonIo = makeIo();
@ -1178,6 +1267,77 @@ describe('setup status', () => {
expect(calls).toEqual(['model', 'embeddings', 'databases', 'sources']);
});
it.each([
{
backend: 'vertex',
providerLines: [' backend: vertex', ' vertex:', ' project: kaelio-dev', ' location: us-east5'],
model: 'claude-sonnet-4-6',
},
{
backend: 'gateway',
providerLines: [' backend: gateway', ' gateway:', ' api_key: env:AI_GATEWAY_API_KEY'],
model: 'anthropic/claude-sonnet-4-6',
},
])('adds a dbt source in non-interactive setup with existing $backend llm config', async (fixture) => {
const io = makeIo();
await writeFile(
join(tempDir, 'ktx.yaml'),
[
'project: revenue',
'setup:',
' database_connection_ids:',
' - warehouse',
' completed_steps:',
' - project',
' - databases',
'connections:',
' warehouse:',
' driver: postgres',
' url: env:WAREHOUSE_URL',
'llm:',
' provider:',
...fixture.providerLines,
' models:',
` default: ${fixture.model}`,
].join('\n'),
'utf-8',
);
await expect(
runKtxSetup(
{
command: 'run',
projectDir: tempDir,
mode: 'existing',
agents: false,
skipAgents: true,
inputMode: 'disabled',
yes: true,
cliVersion: '0.2.0',
skipLlm: false,
skipEmbeddings: true,
skipDatabases: true,
source: 'dbt',
sourceConnectionId: 'dbt-main',
sourceGitUrl: 'https://github.com/Kaelio/klo-dbt-demo',
sourceBranch: 'main',
sourceProjectName: 'orbit_analytics',
sourceWarehouseConnectionId: 'warehouse',
skipSources: false,
databaseSchemas: [],
},
io.io,
{
sourcesDeps: { validateDbt: vi.fn(async () => ({ ok: true as const, detail: 'dbt project valid' })) },
context: vi.fn(async () => ({ status: 'ready' as const, projectDir: tempDir, runId: 'setup-context-test' })),
},
),
).resolves.toBe(0);
expect(io.stderr()).not.toContain('Anthropic');
expect(await readFile(join(tempDir, 'ktx.yaml'), 'utf-8')).toContain('dbt-main:');
});
it('does not fail context build when prerequisites were explicitly skipped and agents are skipped', async () => {
const calls: string[] = [];
const io = makeIo();

View file

@ -1,7 +1,8 @@
import { existsSync } from 'node:fs';
import { join, resolve } from 'node:path';
import { cancel, isCancel, select } from '@clack/prompts';
import { loadKtxProject } from '@ktx/context/project';
import { getLatestLocalIngestStatus, savedMemoryCountsForReport } from '@ktx/context/ingest';
import { ktxLocalStateDbPath, loadKtxProject, type KtxLocalProject } from '@ktx/context/project';
import type { KtxCliIo } from './cli-runtime.js';
import { formatSetupNextStepLines } from './next-steps.js';
import { isKtxSetupExitError, withSetupInterruptConfirmation } from './setup-interrupt.js';
@ -20,7 +21,7 @@ import {
runKtxSetupDatabasesStep,
} from './setup-databases.js';
import { type KtxSetupEmbeddingsDeps, runKtxSetupEmbeddingsStep } from './setup-embeddings.js';
import { type KtxSetupModelDeps, runKtxSetupAnthropicModelStep } from './setup-models.js';
import { type KtxSetupModelDeps, isKtxSetupLlmConfigReady, runKtxSetupAnthropicModelStep } from './setup-models.js';
import { type KtxSetupProjectDeps, runKtxSetupProjectStep } from './setup-project.js';
import {
isKtxPreAgentSetupReady,
@ -226,10 +227,6 @@ async function runKtxSetupDemoFromEntryMenu(
);
}
function llmReady(status: KtxSetupStatus['llm']): boolean {
return status.backend === 'anthropic' && typeof status.model === 'string' && status.model.length > 0;
}
function embeddingsReady(status: KtxSetupStatus['embeddings']): boolean {
return (
status.backend !== undefined &&
@ -252,6 +249,31 @@ function sourceConnections(config: Awaited<ReturnType<typeof loadKtxProject>>['c
.sort((left, right) => left.connectionId.localeCompare(right.connectionId));
}
type LocalIngestStatusReport = NonNullable<Awaited<ReturnType<typeof getLatestLocalIngestStatus>>>;
function reportHasSavedContext(report: LocalIngestStatusReport): boolean {
if (report.body.failedWorkUnits.length > 0) {
return false;
}
const counts = savedMemoryCountsForReport(report);
return counts.wikiCount > 0 || counts.slCount > 0;
}
async function readIngestContextStatus(project: KtxLocalProject): Promise<KtxSetupContextStatusSummary | null> {
if (!existsSync(ktxLocalStateDbPath(project))) {
return null;
}
const report = await getLatestLocalIngestStatus(project);
if (!report || !reportHasSavedContext(report)) {
return null;
}
return {
ready: true,
status: 'completed',
runId: report.runId,
};
}
export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupStatus> {
const resolvedProjectDir = resolve(projectDir);
if (!existsSync(join(resolvedProjectDir, 'ktx.yaml'))) {
@ -269,10 +291,9 @@ export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupSt
const project = await loadKtxProject({ projectDir: resolvedProjectDir });
const llm = {
backend: project.config.llm.provider.backend,
ready: false,
ready: isKtxSetupLlmConfigReady(project.config.llm),
model: project.config.llm.models.default,
};
llm.ready = llmReady(llm);
const embeddings = {
backend: project.config.ingest.embeddings.backend,
@ -284,6 +305,10 @@ export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupSt
const completedSteps = project.config.setup?.completed_steps ?? [];
const contextState = await readKtxSetupContextState(resolvedProjectDir);
const setupContextStatus = setupContextStatusFromState(contextState, {
completedStep: completedSteps.includes('context'),
});
const ingestContextStatus = setupContextStatus.ready ? null : await readIngestContextStatus(project);
const databaseIds = project.config.setup?.database_connection_ids ?? Object.keys(project.config.connections);
const databasesComplete = completedSteps.includes('databases');
const manifest = await readKtxAgentInstallManifest(resolvedProjectDir);
@ -306,7 +331,7 @@ export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupSt
...source,
ready: completedSteps.includes('sources'),
})),
context: setupContextStatusFromState(contextState, { completedStep: completedSteps.includes('context') }),
context: ingestContextStatus ?? setupContextStatus,
agents,
};
}
@ -376,7 +401,7 @@ function setupStatusReady(status: KtxSetupStatus): boolean {
return true;
}
return (
llmReady(status.llm) &&
status.llm.ready &&
embeddingsReady(status.embeddings) &&
status.databases.every((database) => database.ready) &&
status.sources.every((source) => source.ready)

View file

@ -448,10 +448,18 @@ joins: []
listIo.io,
);
expect(code).toBe(0);
expect(listIo.stderr()).toBe('');
const parsed = JSON.parse(listIo.stdout());
expect(parsed.kind).toBe('list');
expect(parsed.meta).toEqual({ command: 'sl list' });
expect(parsed).toMatchObject({
kind: 'list',
data: {
items: expect.any(Array),
},
meta: {
command: 'sl list',
},
});
expect(parsed.data.items).toHaveLength(1);
expect(parsed.data.items[0]).toMatchObject({
connectionId: 'warehouse',

View file

@ -368,9 +368,9 @@ describe('standalone built ktx CLI smoke', () => {
const knowledgeSearch = structuredContent<{
results: Array<{ key: string; summary: string; score: number }>;
totalFound: number;
}>(await client.callTool({ name: 'knowledge_search', arguments: { query: 'ARR contract', limit: 5 } }));
}>(await client.callTool({ name: 'knowledge_search', arguments: { query: 'ARR contract-first definition', limit: 10 } }));
expect(knowledgeSearch.totalFound).toBeGreaterThan(0);
expect(knowledgeSearch.results.map((result) => result.key)).toContain('arr-contract-first');
expect(knowledgeSearch.results.map((result) => result.key)).toContain('orbit-arr-contract-first-definition');
const knowledgeRead = structuredContent<{
key: string;
@ -378,26 +378,26 @@ describe('standalone built ktx CLI smoke', () => {
content: string;
tags: string[];
slRefs: string[];
}>(await client.callTool({ name: 'knowledge_read', arguments: { key: 'arr-contract-first' } }));
expect(knowledgeRead.key).toBe('arr-contract-first');
}>(await client.callTool({ name: 'knowledge_read', arguments: { key: 'orbit-arr-contract-first-definition' } }));
expect(knowledgeRead.key).toBe('orbit-arr-contract-first-definition');
expect(knowledgeRead.summary).toContain('ARR');
expect(knowledgeRead.content).toContain('contract');
expect(knowledgeRead.slRefs).toContain('orbit_demo.contracts');
expect(knowledgeRead.slRefs).toContain('mart_arr_daily');
const slRead = structuredContent<{ sourceName: string; yaml: string }>(
await client.callTool({
name: 'sl_read_source',
arguments: { connectionId: 'orbit_demo', sourceName: 'accounts' },
arguments: { connectionId: 'dbt-main', sourceName: 'mart_arr_daily' },
}),
);
expect(slRead.sourceName).toBe('accounts');
expect(slRead.yaml).toContain('name: accounts');
expect(slRead.sourceName).toBe('mart_arr_daily');
expect(slRead.yaml).toContain('name: mart_arr_daily');
expect(slRead.yaml).toContain('measures:');
const slValidate = structuredContent<{ success: boolean; errors: string[]; warnings: string[] }>(
await client.callTool({
name: 'sl_validate',
arguments: { connectionId: 'orbit_demo', names: ['accounts', 'contracts'] },
arguments: { connectionId: 'dbt-main', names: ['mart_arr_daily', 'stg_contracts'] },
}),
);
expect(slValidate.success).toBe(true);
@ -716,7 +716,7 @@ describe('standalone built ktx CLI smoke', () => {
'--project-dir',
projectDir,
'--token-env',
'NOTION_AUTH_TOKEN',
'NOTION_TOKEN',
'--crawl-mode',
'all_accessible',
'--max-pages',
@ -729,7 +729,7 @@ describe('standalone built ktx CLI smoke', () => {
const yaml = await readFile(join(projectDir, 'ktx.yaml'), 'utf-8');
expect(yaml).toContain('driver: notion');
expect(yaml).toContain('auth_token_ref: env:NOTION_AUTH_TOKEN');
expect(yaml).toContain('auth_token_ref: env:NOTION_TOKEN');
expect(yaml).toContain('crawl_mode: all_accessible');
expect(yaml).toContain('max_pages_per_run: 5');
expect(yaml).not.toContain('ntn_');
@ -737,7 +737,7 @@ describe('standalone built ktx CLI smoke', () => {
const parsed = parseKtxProjectConfig(yaml);
expect(parsed.connections['notion-main']).toMatchObject({
driver: 'notion',
auth_token_ref: 'env:NOTION_AUTH_TOKEN',
auth_token_ref: 'env:NOTION_TOKEN',
crawl_mode: 'all_accessible',
});
});