mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-25 08:48:08 +02:00
Merge remote-tracking branch 'origin/main' into luca-martial/schema-select-ux-text
# Conflicts: # packages/cli/src/demo.test.ts # packages/context/src/ingest/local-adapters.ts
This commit is contained in:
commit
d0f650f44a
123 changed files with 3739 additions and 933 deletions
|
|
@ -12,7 +12,7 @@ refs:
|
|||
|
||||
## New Hire Week-One Onboarding Policy
|
||||
|
||||
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
|
||||
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
|
||||
**Owner:** Manager (not People Ops)
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ tables:
|
|||
|
||||
# Activation KPI Glossary
|
||||
|
||||
**Owner team:** Growth
|
||||
**Owner team:** Growth
|
||||
**Source:** Notion — Orbit Demo Home / Data Team - Onboarding / Activation KPI Glossary, last edited 2026-05-07
|
||||
|
||||
Use this when a question is about signup-to-habit behavior. Orbit uses activation language across Growth, Product, and CS conversations.
|
||||
|
|
@ -62,4 +62,3 @@ Growth conversations typically use D7 and D14 Activation Rate. Product and CS ma
|
|||
## Relationship to Account-Level Activation
|
||||
|
||||
This glossary defines **customer-level** activation (signup-to-habit). The **account-level** activation workflow (requester login → first approved purchase request → account activated) is a separate concept tracked in `mart_account_activity` and governed by the January 2026 policy change. See `orbit-activation-policy-change-jan-2026` for that definition.
|
||||
|
||||
|
|
|
|||
|
|
@ -14,9 +14,9 @@ sl_refs:
|
|||
|
||||
# Activation Policy Change — January 2026
|
||||
|
||||
**Governed metric key:** `activated_accounts`
|
||||
**Owner team:** growth
|
||||
**Notion:** `notion://notion_page_activation_policy_decision#policy-change`
|
||||
**Governed metric key:** `activated_accounts`
|
||||
**Owner team:** growth
|
||||
**Notion:** `notion://notion_page_activation_policy_decision#policy-change`
|
||||
**Sources:** `mart_account_activity`, `int_activation_policy_windows`, `stg_activation_events`
|
||||
|
||||
## Policy Boundary
|
||||
|
|
|
|||
|
|
@ -15,9 +15,9 @@ sl_refs:
|
|||
|
||||
# ARR — Contract-First Definition
|
||||
|
||||
**Governed metric key:** `arr`
|
||||
**Owner team:** finance
|
||||
**Notion:** `notion://notion_page_arr_contract_reporting#arr-contract-first`
|
||||
**Governed metric key:** `arr`
|
||||
**Owner team:** finance
|
||||
**Notion:** `notion://notion_page_arr_contract_reporting#arr-contract-first`
|
||||
**Source:** `mart_arr_daily` (grain: `metric_date`)
|
||||
|
||||
## Rule
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ refs:
|
|||
|
||||
Orbit sells procurement workflow and spend-control software. The core value proposition: route purchase requests, collect approvals, onboard suppliers, and issue purchase orders without turning every exception into a status hunt.
|
||||
|
||||
**Primary buyers:** Finance, Procurement, Business Operations.
|
||||
**Primary buyers:** Finance, Procurement, Business Operations.
|
||||
**Daily users:** department admins, office managers, IT leads, legal ops partners — anyone who has to get a vendor through the building.
|
||||
|
||||
## Product Workflow
|
||||
|
|
@ -69,4 +69,3 @@ Orbit sells procurement workflow and spend-control software. The core value prop
|
|||
- "Supplier onboarding is split across three teams."
|
||||
- "Renewals are visible too late."
|
||||
- "People keep asking Finance for status because there is nowhere better to look."
|
||||
|
||||
|
|
|
|||
|
|
@ -14,9 +14,9 @@ sl_refs:
|
|||
|
||||
# Customer Health Risk Definition
|
||||
|
||||
**Governed metric key:** `active_customers`
|
||||
**Owner team:** customer_success
|
||||
**Notion:** `notion://notion_page_customer_health_playbook#risk-definition`
|
||||
**Governed metric key:** `active_customers`
|
||||
**Owner team:** customer_success
|
||||
**Notion:** `notion://notion_page_customer_health_playbook#risk-definition`
|
||||
**Sources:** `mart_customer_health`, `int_customer_health_signals`
|
||||
|
||||
## Risk Levels
|
||||
|
|
|
|||
|
|
@ -18,8 +18,8 @@ tables:
|
|||
|
||||
# Orbit Customers Source
|
||||
|
||||
**Table:** `orbit_analytics.customer`
|
||||
**Grain:** one row per signed-up customer
|
||||
**Table:** `orbit_analytics.customer`
|
||||
**Grain:** one row per signed-up customer
|
||||
**Source:** Notion — Orbit Demo Home / Data Team - Onboarding / Orbit Customers Source, last edited 2026-05-07
|
||||
|
||||
Use this when a question needs customer identity, plan tier, signup timing, recent activity, or the standard customer joins.
|
||||
|
|
@ -58,4 +58,3 @@ Always join through `customer.id`. Do not join on `email`.
|
|||
- **Timezone:** `created_at` and `last_seen_at` are UTC. Confirm whether a question expects UTC or a local business day before filtering.
|
||||
- **Paying vs. all:** `free` customers must be excluded from paying-customer follow-ups. Use `paying_customer_count`, not `customer_count`.
|
||||
- **plan_tier values:** `free`, `pro`, `enterprise`. Note: `pro_plus` is a legacy alias for `growth` in the account/contract layer (see `orbit-plan-segment-normalization`), but `plan_tier` on this table uses `pro` not `pro_plus`.
|
||||
|
||||
|
|
|
|||
|
|
@ -42,4 +42,3 @@ Declared in `models/exposures.yml`. All exposures are type `dashboard` with matu
|
|||
- **Owner:** Growth (growth@orbit-demo.example.com)
|
||||
- **Depends on:** `mart_account_activity`
|
||||
- **Description:** Activation policy comparison around the January 2026 workflow update.
|
||||
|
||||
|
|
|
|||
|
|
@ -22,10 +22,10 @@ sl_refs:
|
|||
|
||||
# Orbit dbt Project Overview
|
||||
|
||||
**Project name:** `kaelio_demo`
|
||||
**dbt version:** 1.0.0
|
||||
**Profile target:** Postgres (`orbit_analytics` schema, `kaelio_demo` database)
|
||||
**Raw source schema:** `orbit_raw`
|
||||
**Project name:** `kaelio_demo`
|
||||
**dbt version:** 1.0.0
|
||||
**Profile target:** Postgres (`orbit_analytics` schema, `kaelio_demo` database)
|
||||
**Raw source schema:** `orbit_raw`
|
||||
**Analytics schema:** `orbit_analytics` (all models materialised as views by default)
|
||||
|
||||
## Model Layers
|
||||
|
|
@ -52,4 +52,3 @@ sl_refs:
|
|||
## Raw Source Tables (`orbit_raw` schema)
|
||||
|
||||
accounts, account_hierarchy, plans, contracts, subscriptions, contract_discount_terms, arr_movements, invoices, invoice_line_items, refunds, plan_segment_mapping, users, activation_events, sessions, purchase_requests, approval_events, suppliers, supplier_onboarding_events, purchase_orders, support_tickets, account_owners.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ tables:
|
|||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/106.json -->
|
||||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/107.json -->
|
||||
|
||||
**Table:** `orbit_analytics.mart_account_activity`
|
||||
**Table:** `orbit_analytics.mart_account_activity`
|
||||
**Grain:** one row per `policy_change_date`
|
||||
|
||||
## Columns
|
||||
|
|
@ -47,4 +47,3 @@ tables:
|
|||
- The January 2026 activation policy change (`policy_change_date = 2026-01-15`) is the primary boundary. `policy_version` in upstream events splits into `pre_2026_01_15` and `post_2026_01_15` cohorts.
|
||||
- Rates are ratios (0–1); multiply by 100 for percentage display.
|
||||
- See [orbit-activation-policy-change-jan-2026](orbit-activation-policy-change-jan-2026) for full policy context.
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ tables:
|
|||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/69.json -->
|
||||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/100.json -->
|
||||
|
||||
**Table:** `orbit_analytics.mart_account_segments`
|
||||
**Table:** `orbit_analytics.mart_account_segments`
|
||||
**Grain:** one row per `account_id`
|
||||
|
||||
## Columns
|
||||
|
|
@ -53,4 +53,3 @@ tables:
|
|||
- `normalized_plan_code` maps `pro_plus` → `growth`. Always use `normalized_plan_code` for plan-based reporting. See [orbit-plan-segment-normalization](orbit-plan-segment-normalization).
|
||||
- `segment` is derived from `canonical_plan_code × size_band` via `stg_plan_segment_mapping`.
|
||||
- `contract_arr_cents` is the contract-first ARR value. See [orbit-arr-contract-first-definition](orbit-arr-contract-first-definition).
|
||||
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ tables:
|
|||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/56.json -->
|
||||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/96.json -->
|
||||
|
||||
**Table:** `orbit_analytics.mart_arr_daily`
|
||||
**Table:** `orbit_analytics.mart_arr_daily`
|
||||
**Grain:** one row per `metric_date`
|
||||
|
||||
## Columns
|
||||
|
|
@ -44,4 +44,3 @@ tables:
|
|||
|
||||
- ARR is calculated contract-first: active contract ARR takes precedence over subscription ARR for any covered period. See [orbit-arr-contract-first-definition](orbit-arr-contract-first-definition).
|
||||
- `display` is a formatted label for UI rendering; use `arr_cents` for all arithmetic.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ tables:
|
|||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/98.json -->
|
||||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/103.json -->
|
||||
|
||||
**Table:** `orbit_analytics.mart_nrr_quarterly`
|
||||
**Table:** `orbit_analytics.mart_nrr_quarterly`
|
||||
**Grain:** one row per `quarter_label` × `segment`
|
||||
|
||||
## Columns
|
||||
|
|
@ -53,4 +53,3 @@ tables:
|
|||
- `net_revenue_retention` is a ratio, not a percentage. Multiply by 100 for display.
|
||||
- Contraction includes discount expirations (classified as contraction, not churn). See [orbit-nrr-discount-expiration-treatment](orbit-nrr-discount-expiration-treatment).
|
||||
- Enterprise is the primary executive reporting segment.
|
||||
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ tables:
|
|||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/88.json -->
|
||||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/108.json -->
|
||||
|
||||
**Table:** `orbit_analytics.mart_procurement_activity`
|
||||
**Table:** `orbit_analytics.mart_procurement_activity`
|
||||
**Grain:** one row per `week_start_date` × `contract_arr_threshold_cents`
|
||||
|
||||
## Columns
|
||||
|
|
@ -45,4 +45,3 @@ tables:
|
|||
- `active_requesters` counts non-internal, non-test requesters on large active contracts. See [orbit-procurement-qualifying-actions](orbit-procurement-qualifying-actions).
|
||||
- The standard threshold is `contract_arr_threshold_cents = 20000000` ($200k ARR).
|
||||
- Always filter by `contract_arr_threshold_cents` — the table contains rows for multiple threshold values.
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ tables:
|
|||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/105.json -->
|
||||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/115.json -->
|
||||
|
||||
**Table:** `orbit_analytics.mart_retention_movement_breakout`
|
||||
**Table:** `orbit_analytics.mart_retention_movement_breakout`
|
||||
**Grain:** one row per `quarter_label` × `segment` × `movement_type` × `movement_reason`
|
||||
|
||||
## Columns
|
||||
|
|
@ -53,4 +53,3 @@ tables:
|
|||
- Contraction includes discount expirations, classified as contraction (not churn), tracked via `movement_reason`. See [orbit-nrr-discount-expiration-treatment](orbit-nrr-discount-expiration-treatment).
|
||||
- This table is the row-level source for `mart_nrr_quarterly` aggregations.
|
||||
- Only one of `expansion_arr_cents`, `contraction_arr_cents`, `churned_arr_cents` is non-zero per row.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ tables:
|
|||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/102.json -->
|
||||
<!-- from: raw-sources/postgres-warehouse/metabase/2026-05-12-035303-local-metabase-3-114d957b-f564-4f46-8d4c-2770720a95be/cards/104.json -->
|
||||
|
||||
**Table:** `orbit_analytics.mart_revenue_daily`
|
||||
**Table:** `orbit_analytics.mart_revenue_daily`
|
||||
**Grain:** one row per `revenue_date`
|
||||
|
||||
## Columns
|
||||
|
|
@ -54,4 +54,3 @@ tables:
|
|||
- `reconciliation_check` must be `true` on every row. Any `false` row indicates a data quality issue.
|
||||
- Gross-to-net reconciliation: gross revenue − credits − refunds = net revenue. See [orbit-revenue-gross-to-net-reconciliation](orbit-revenue-gross-to-net-reconciliation).
|
||||
- All amounts are in cents; divide by 100 for USD, by 100,000,000 for $M.
|
||||
|
||||
|
|
|
|||
|
|
@ -69,4 +69,3 @@ Card 48 is the canonical reference; card 55 is a filtered variant for large-cont
|
|||
| 53 | Enterprise NRR quarter breakout | mart_nrr_quarterly | 0 |
|
||||
| 54 | February credits drilldown | mart_revenue_daily | 0 |
|
||||
| 55 | Large contract requesters | mart_account_segments | 0 |
|
||||
|
||||
|
|
|
|||
|
|
@ -15,9 +15,9 @@ sl_refs:
|
|||
|
||||
# NRR — Discount Expiration Treatment
|
||||
|
||||
**Governed metric key:** `net_revenue_retention`
|
||||
**Owner team:** analytics
|
||||
**Notion:** `notion://notion_page_retention_policy_current#nrr-definition` and `#discount-expiration-treatment`
|
||||
**Governed metric key:** `net_revenue_retention`
|
||||
**Owner team:** analytics
|
||||
**Notion:** `notion://notion_page_retention_policy_current#nrr-definition` and `#discount-expiration-treatment`
|
||||
**Sources:** `mart_nrr_quarterly`, `mart_retention_movement_breakout`
|
||||
|
||||
## NRR Definition
|
||||
|
|
|
|||
|
|
@ -14,9 +14,9 @@ sl_refs:
|
|||
|
||||
# Plan & Segment Normalization
|
||||
|
||||
**Governed metric key:** `segment`
|
||||
**Owner team:** sales_ops
|
||||
**Notion:** `notion://notion_page_sales_ops_segmentation#growth-plan-normalization`
|
||||
**Governed metric key:** `segment`
|
||||
**Owner team:** sales_ops
|
||||
**Notion:** `notion://notion_page_sales_ops_segmentation#growth-plan-normalization`
|
||||
**Sources:** `mart_account_segments`, `stg_plan_segment_mapping`, `stg_plans`
|
||||
|
||||
## Canonical Plan Codes
|
||||
|
|
|
|||
|
|
@ -14,9 +14,9 @@ sl_refs:
|
|||
|
||||
# Procurement — Qualifying Actions & Weekly Active Requesters
|
||||
|
||||
**Governed metric key:** `weekly_active_requesters`
|
||||
**Owner team:** product
|
||||
**Notion:** `notion://notion_page_procurement_instrumentation#qualifying-procurement-actions`
|
||||
**Governed metric key:** `weekly_active_requesters`
|
||||
**Owner team:** product
|
||||
**Notion:** `notion://notion_page_procurement_instrumentation#qualifying-procurement-actions`
|
||||
**Sources:** `mart_procurement_activity`, `int_procurement_qualifying_actions`
|
||||
|
||||
## Qualifying Action Definition
|
||||
|
|
|
|||
|
|
@ -14,9 +14,9 @@ sl_refs:
|
|||
|
||||
# Revenue — Gross-to-Net Reconciliation
|
||||
|
||||
**Governed metric key:** `net_revenue`
|
||||
**Owner team:** finance
|
||||
**Notion:** `notion://notion_page_revenue_reporting_policy#gross-to-net-reconciliation`
|
||||
**Governed metric key:** `net_revenue`
|
||||
**Owner team:** finance
|
||||
**Notion:** `notion://notion_page_revenue_reporting_policy#gross-to-net-reconciliation`
|
||||
**Source:** `mart_revenue_daily` (grain: `revenue_date`)
|
||||
|
||||
## Formula
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ refs:
|
|||
|
||||
## Sales Ops → Customer Success Implementation Handoff
|
||||
|
||||
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
|
||||
**Source:** Notion — People & Operating Norms, last edited 2026-05-07
|
||||
**Owner:** Sales Ops (sender), Customer Success (receiver)
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { spinner } from '@clack/prompts';
|
||||
import { cancel, confirm, isCancel, log, spinner } from '@clack/prompts';
|
||||
|
||||
export interface KtxCliSpinner {
|
||||
start(message: string): void;
|
||||
|
|
@ -6,6 +6,62 @@ export interface KtxCliSpinner {
|
|||
error(message: string): void;
|
||||
}
|
||||
|
||||
export interface KtxCliPromptAdapter {
|
||||
confirm(options: { message: string; initialValue?: boolean }): Promise<boolean>;
|
||||
cancel(message: string): void;
|
||||
log: {
|
||||
info(message: string): void;
|
||||
warn(message: string): void;
|
||||
error(message: string): void;
|
||||
success(message: string): void;
|
||||
step(message: string): void;
|
||||
};
|
||||
spinner(): KtxCliSpinner;
|
||||
}
|
||||
|
||||
export class KtxCliPromptCancelledError extends Error {
|
||||
constructor(message = 'Operation cancelled.') {
|
||||
super(message);
|
||||
this.name = 'KtxCliPromptCancelledError';
|
||||
}
|
||||
}
|
||||
|
||||
export function createClackSpinner(): KtxCliSpinner {
|
||||
return spinner();
|
||||
}
|
||||
|
||||
export function createClackPromptAdapter(): KtxCliPromptAdapter {
|
||||
return {
|
||||
async confirm(options) {
|
||||
const value = await confirm(options);
|
||||
if (isCancel(value)) {
|
||||
cancel('Operation cancelled.');
|
||||
throw new KtxCliPromptCancelledError();
|
||||
}
|
||||
return value;
|
||||
},
|
||||
cancel(message) {
|
||||
cancel(message);
|
||||
},
|
||||
log: {
|
||||
info(message) {
|
||||
log.info(message);
|
||||
},
|
||||
warn(message) {
|
||||
log.warn(message);
|
||||
},
|
||||
error(message) {
|
||||
log.error(message);
|
||||
},
|
||||
success(message) {
|
||||
log.success(message);
|
||||
},
|
||||
step(message) {
|
||||
log.step(message);
|
||||
},
|
||||
},
|
||||
spinner() {
|
||||
return createClackSpinner();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -138,7 +138,7 @@ function makeIo(options: { isTTY?: boolean; stdinIsTTY?: boolean } = {}) {
|
|||
describe('runKtxConnectionMetabaseSetup', () => {
|
||||
const fakeMetabaseCredential = 'mb_example';
|
||||
const existingMetabaseCredential = 'mb_existing';
|
||||
const fakeAdminCredential = 'pw';
|
||||
const fakeAdminCredential = 'admin-secret-value-123';
|
||||
|
||||
let tempDir: string;
|
||||
let projectDir: string;
|
||||
|
|
|
|||
|
|
@ -53,10 +53,12 @@ export function registerRuntimeCommands(program: Command, context: KtxCliCommand
|
|||
runtime
|
||||
.command('stop')
|
||||
.description('Stop the KTX-managed Python HTTP daemon')
|
||||
.action(async () => {
|
||||
.option('--all', 'Stop all KTX daemon processes recorded or discoverable on this machine', false)
|
||||
.action(async (options: { all?: boolean }) => {
|
||||
await runRuntimeArgs(context, {
|
||||
command: 'stop',
|
||||
cliVersion: context.packageInfo.version,
|
||||
all: options.all === true,
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { initKtxProject, parseKtxProjectConfig } from '@ktx/context/project';
|
||||
import type { MetabaseRuntimeClient } from '@ktx/context/ingest';
|
||||
import { initKtxProject, parseKtxProjectConfig, serializeKtxProjectConfig } from '@ktx/context/project';
|
||||
import type { KtxConnectionDriver, KtxScanConnector, KtxSchemaSnapshot } from '@ktx/context/scan';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { runKtxConnection } from './connection.js';
|
||||
|
|
@ -476,7 +477,7 @@ describe('runKtxConnection', () => {
|
|||
force: false,
|
||||
allowLiteralCredentials: false,
|
||||
notion: {
|
||||
authTokenRef: 'env:NOTION_AUTH_TOKEN',
|
||||
authTokenRef: 'env:NOTION_TOKEN',
|
||||
crawlMode: 'all_accessible',
|
||||
rootPageIds: [],
|
||||
rootDatabaseIds: [],
|
||||
|
|
@ -492,7 +493,7 @@ describe('runKtxConnection', () => {
|
|||
|
||||
const yaml = await readFile(join(projectDir, 'ktx.yaml'), 'utf-8');
|
||||
expect(yaml).toContain('driver: notion');
|
||||
expect(yaml).toContain('auth_token_ref: env:NOTION_AUTH_TOKEN');
|
||||
expect(yaml).toContain('auth_token_ref: env:NOTION_TOKEN');
|
||||
expect(yaml).toContain('crawl_mode: all_accessible');
|
||||
expect(yaml).toContain('max_pages_per_run: 50');
|
||||
expect(yaml).not.toContain('ntn_');
|
||||
|
|
@ -515,7 +516,7 @@ describe('runKtxConnection', () => {
|
|||
force: false,
|
||||
allowLiteralCredentials: false,
|
||||
notion: {
|
||||
authTokenRef: 'env:NOTION_AUTH_TOKEN',
|
||||
authTokenRef: 'env:NOTION_TOKEN',
|
||||
crawlMode: 'all_accessible',
|
||||
rootPageIds: [],
|
||||
rootDatabaseIds: ['database-1'],
|
||||
|
|
@ -598,6 +599,61 @@ describe('runKtxConnection', () => {
|
|||
expect(io.stdout()).toContain('Tables: 2');
|
||||
});
|
||||
|
||||
it('tests a configured Metabase connection through the Metabase runtime client', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await initKtxProject({ projectDir, projectName: 'warehouse' });
|
||||
const projectConfig = parseKtxProjectConfig(await readFile(join(projectDir, 'ktx.yaml'), 'utf-8'));
|
||||
await writeFile(
|
||||
join(projectDir, 'ktx.yaml'),
|
||||
serializeKtxProjectConfig({
|
||||
...projectConfig,
|
||||
connections: {
|
||||
...projectConfig.connections,
|
||||
prod_metabase: {
|
||||
driver: 'metabase',
|
||||
api_url: 'http://metabase.example.test',
|
||||
api_key: 'mb_test',
|
||||
},
|
||||
},
|
||||
}),
|
||||
'utf-8',
|
||||
);
|
||||
const testConnection = vi.fn(async () => ({ success: true as const }));
|
||||
const getDatabases = vi.fn(async () => [
|
||||
{ id: 1, name: 'Analytics', engine: 'postgres', details: {}, is_sample: false },
|
||||
{ id: 2, name: 'Sample Database', engine: 'h2', details: {}, is_sample: true },
|
||||
]);
|
||||
const cleanup = vi.fn(async () => undefined);
|
||||
const createMetabaseClient = vi.fn(
|
||||
async (): Promise<Pick<MetabaseRuntimeClient, 'testConnection' | 'getDatabases' | 'cleanup'>> => ({
|
||||
testConnection,
|
||||
getDatabases,
|
||||
cleanup,
|
||||
}),
|
||||
);
|
||||
const createScanConnector = vi.fn(async () => {
|
||||
throw new Error('native scanner should not be used for Metabase');
|
||||
});
|
||||
const io = makeIo();
|
||||
|
||||
await expect(
|
||||
runKtxConnection({ command: 'test', projectDir, connectionId: 'prod_metabase' }, io.io, {
|
||||
createScanConnector,
|
||||
createMetabaseClient,
|
||||
}),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(createScanConnector).not.toHaveBeenCalled();
|
||||
expect(createMetabaseClient).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), 'prod_metabase');
|
||||
expect(testConnection).toHaveBeenCalledTimes(1);
|
||||
expect(getDatabases).toHaveBeenCalledTimes(1);
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
expect(io.stdout()).toContain('Connection test passed: prod_metabase');
|
||||
expect(io.stdout()).toContain('Driver: metabase');
|
||||
expect(io.stdout()).toContain('Databases: 1');
|
||||
expect(io.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('cleans up the native scan connector when connection testing fails', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await initKtxProject({ projectDir, projectName: 'warehouse' });
|
||||
|
|
|
|||
|
|
@ -1,4 +1,10 @@
|
|||
import { cancel, confirm, isCancel } from '@clack/prompts';
|
||||
import {
|
||||
DEFAULT_METABASE_CLIENT_CONFIG,
|
||||
DefaultMetabaseConnectionClientFactory,
|
||||
type MetabaseRuntimeClient,
|
||||
metabaseRuntimeConfigFromLocalConnection,
|
||||
} from '@ktx/context/ingest';
|
||||
import { type KtxLocalProject, loadKtxProject, serializeKtxProjectConfig } from '@ktx/context/project';
|
||||
import type { KtxScanConnector } from '@ktx/context/scan';
|
||||
import type { KtxConnectionMappingArgs } from './commands/connection-mapping.js';
|
||||
|
|
@ -61,6 +67,7 @@ interface KtxConnectionIo extends KtxCliIo {
|
|||
|
||||
interface KtxConnectionDeps {
|
||||
createScanConnector?: typeof createKtxCliScanConnector;
|
||||
createMetabaseClient?: typeof createDefaultMetabaseClient;
|
||||
runMapping?: (argv: string[], io: KtxCliIo) => Promise<number>;
|
||||
prompts?: KtxConnectionPromptAdapter;
|
||||
}
|
||||
|
|
@ -104,6 +111,12 @@ async function cleanupConnector(connector: KtxScanConnector | null): Promise<voi
|
|||
}
|
||||
}
|
||||
|
||||
function normalizedConnectionDriver(project: KtxLocalProject, connectionId: string): string {
|
||||
return String(project.config.connections[connectionId]?.driver ?? '')
|
||||
.trim()
|
||||
.toLowerCase();
|
||||
}
|
||||
|
||||
async function testNativeConnection(
|
||||
project: KtxLocalProject,
|
||||
connectionId: string,
|
||||
|
|
@ -131,6 +144,48 @@ async function testNativeConnection(
|
|||
}
|
||||
}
|
||||
|
||||
async function createDefaultMetabaseClient(
|
||||
project: KtxLocalProject,
|
||||
connectionId: string,
|
||||
): Promise<Pick<MetabaseRuntimeClient, 'testConnection' | 'getDatabases' | 'cleanup'>> {
|
||||
const factory = new DefaultMetabaseConnectionClientFactory(
|
||||
(metabaseConnectionId) =>
|
||||
metabaseRuntimeConfigFromLocalConnection(
|
||||
metabaseConnectionId,
|
||||
project.config.connections[metabaseConnectionId],
|
||||
),
|
||||
DEFAULT_METABASE_CLIENT_CONFIG,
|
||||
);
|
||||
return factory.createClient(connectionId);
|
||||
}
|
||||
|
||||
async function testMetabaseConnection(
|
||||
project: KtxLocalProject,
|
||||
connectionId: string,
|
||||
createMetabaseClient: typeof createDefaultMetabaseClient,
|
||||
): Promise<{ driver: 'metabase'; databaseCount: number }> {
|
||||
let client: Pick<MetabaseRuntimeClient, 'testConnection' | 'getDatabases' | 'cleanup'> | null = null;
|
||||
try {
|
||||
client = await createMetabaseClient(project, connectionId);
|
||||
const testResult = await client.testConnection();
|
||||
if (!testResult.success) {
|
||||
throw new Error(
|
||||
`Metabase connection test failed: ${testResult.error ?? testResult.message ?? 'unknown error'}`,
|
||||
);
|
||||
}
|
||||
|
||||
const databases = await client.getDatabases();
|
||||
const databaseCount = databases.filter((database) => database.is_sample !== true).length;
|
||||
if (databaseCount === 0) {
|
||||
throw new Error('Metabase auth worked but no usable databases were returned');
|
||||
}
|
||||
|
||||
return { driver: 'metabase', databaseCount };
|
||||
} finally {
|
||||
await client?.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
interface BufferedIo extends KtxCliIo {
|
||||
stdoutText(): string;
|
||||
stderrText(): string;
|
||||
|
|
@ -399,6 +454,18 @@ export async function runKtxConnection(
|
|||
return 0;
|
||||
}
|
||||
|
||||
if (normalizedConnectionDriver(project, args.connectionId) === 'metabase') {
|
||||
const result = await testMetabaseConnection(
|
||||
project,
|
||||
args.connectionId,
|
||||
deps.createMetabaseClient ?? createDefaultMetabaseClient,
|
||||
);
|
||||
io.stdout.write(`Connection test passed: ${args.connectionId}\n`);
|
||||
io.stdout.write(`Driver: ${result.driver}\n`);
|
||||
io.stdout.write(`Databases: ${result.databaseCount}\n`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
const result = await testNativeConnection(
|
||||
project,
|
||||
args.connectionId,
|
||||
|
|
|
|||
|
|
@ -11,6 +11,9 @@ import type { renderMemoryFlowTui } from './memory-flow-tui.js';
|
|||
import { KTX_NEXT_STEP_DIRECT_COMMANDS } from './next-steps.js';
|
||||
import { resetVizFallbackWarningsForTest } from './viz-fallback.js';
|
||||
|
||||
const SEEDED_DEMO_SEMANTIC_SOURCE_COUNT = 46;
|
||||
const SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT = 28;
|
||||
|
||||
function makeIo(options: { isTTY?: boolean; columns?: number; rawMode?: boolean } = {}) {
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
|
|
@ -336,8 +339,14 @@ describe('runKtxDemo', () => {
|
|||
notion: { pageCount: 8 },
|
||||
},
|
||||
generatedOutputs: {
|
||||
semanticLayer: { manifestSourceCount: 46, fileCount: 46 },
|
||||
knowledge: { manifestPageCount: 28, fileCount: 28 },
|
||||
semanticLayer: {
|
||||
manifestSourceCount: SEEDED_DEMO_SEMANTIC_SOURCE_COUNT,
|
||||
fileCount: SEEDED_DEMO_SEMANTIC_SOURCE_COUNT,
|
||||
},
|
||||
knowledge: {
|
||||
manifestPageCount: SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT,
|
||||
fileCount: SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT,
|
||||
},
|
||||
links: { manifestLinkCount: 23, linkCount: 23 },
|
||||
reports: { primaryPath: 'reports/seeded-demo-report.json', fileCount: 1 },
|
||||
},
|
||||
|
|
@ -636,10 +645,16 @@ describe('runKtxDemo', () => {
|
|||
).resolves.toBe(0);
|
||||
|
||||
expect(seededIo.stdout()).toContain('Status: ready');
|
||||
expect(seededIo.stdout()).toContain('Semantic-layer sources: 46 manifest, 46 files');
|
||||
expect(seededIo.stdout()).toContain('Knowledge pages: 28 manifest, 28 files');
|
||||
expect(seededIo.stdout()).toContain(
|
||||
`Semantic-layer sources: ${SEEDED_DEMO_SEMANTIC_SOURCE_COUNT} manifest, ${SEEDED_DEMO_SEMANTIC_SOURCE_COUNT} files`,
|
||||
);
|
||||
expect(seededIo.stdout()).toContain(
|
||||
`Knowledge pages: ${SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT} manifest, ${SEEDED_DEMO_KNOWLEDGE_PAGE_COUNT} files`,
|
||||
);
|
||||
expect(seededIo.stdout()).not.toContain('Status: corrupt');
|
||||
expect(seededIo.stdout()).not.toContain('Semantic-layer sources: 6 manifest, 0 files');
|
||||
expect(seededIo.stdout()).not.toContain(
|
||||
`Semantic-layer sources: ${SEEDED_DEMO_SEMANTIC_SOURCE_COUNT} manifest, 0 files`,
|
||||
);
|
||||
});
|
||||
|
||||
it('fails corrupted demo projects in no-input mode with reset guidance', async () => {
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
|||
import { createRequire } from 'node:module';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { initKtxProject } from '@ktx/context/project';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import {
|
||||
|
|
@ -143,6 +144,7 @@ describe('runKtxCli', () => {
|
|||
const installIo = makeIo();
|
||||
const startIo = makeIo();
|
||||
const stopIo = makeIo();
|
||||
const stopAllIo = makeIo();
|
||||
const statusIo = makeIo();
|
||||
const doctorIo = makeIo();
|
||||
const pruneIo = makeIo();
|
||||
|
|
@ -156,6 +158,7 @@ describe('runKtxCli', () => {
|
|||
runKtxCli(['runtime', 'start', '--feature', 'local-embeddings', '--force'], startIo.io, { runtime }),
|
||||
).resolves.toBe(0);
|
||||
await expect(runKtxCli(['runtime', 'stop'], stopIo.io, { runtime })).resolves.toBe(0);
|
||||
await expect(runKtxCli(['runtime', 'stop', '--all'], stopAllIo.io, { runtime })).resolves.toBe(0);
|
||||
await expect(runKtxCli(['runtime', 'status', '--json'], statusIo.io, { runtime })).resolves.toBe(0);
|
||||
await expect(runKtxCli(['runtime', 'doctor'], doctorIo.io, { runtime })).resolves.toBe(0);
|
||||
await expect(runKtxCli(['runtime', 'prune', '--dry-run'], pruneIo.io, { runtime })).resolves.toBe(0);
|
||||
|
|
@ -185,11 +188,21 @@ describe('runKtxCli', () => {
|
|||
{
|
||||
command: 'stop',
|
||||
cliVersion: '0.0.0-private',
|
||||
all: false,
|
||||
},
|
||||
stopIo.io,
|
||||
);
|
||||
expect(runtime).toHaveBeenNthCalledWith(
|
||||
4,
|
||||
{
|
||||
command: 'stop',
|
||||
cliVersion: '0.0.0-private',
|
||||
all: true,
|
||||
},
|
||||
stopAllIo.io,
|
||||
);
|
||||
expect(runtime).toHaveBeenNthCalledWith(
|
||||
5,
|
||||
{
|
||||
command: 'status',
|
||||
cliVersion: '0.0.0-private',
|
||||
|
|
@ -198,7 +211,7 @@ describe('runKtxCli', () => {
|
|||
statusIo.io,
|
||||
);
|
||||
expect(runtime).toHaveBeenNthCalledWith(
|
||||
5,
|
||||
6,
|
||||
{
|
||||
command: 'doctor',
|
||||
cliVersion: '0.0.0-private',
|
||||
|
|
@ -207,7 +220,7 @@ describe('runKtxCli', () => {
|
|||
doctorIo.io,
|
||||
);
|
||||
expect(runtime).toHaveBeenNthCalledWith(
|
||||
6,
|
||||
7,
|
||||
{
|
||||
command: 'prune',
|
||||
cliVersion: '0.0.0-private',
|
||||
|
|
@ -218,6 +231,17 @@ describe('runKtxCli', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('documents runtime stop all in command help', async () => {
|
||||
const testIo = makeIo();
|
||||
|
||||
await expect(runKtxCli(['runtime', 'stop', '--help'], testIo.io)).resolves.toBe(0);
|
||||
|
||||
expect(testIo.stdout()).toContain('--all');
|
||||
expect(testIo.stdout()).toContain('Stop all KTX daemon processes recorded or discoverable');
|
||||
expect(testIo.stdout()).toContain('on this machine');
|
||||
expect(testIo.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('routes sl query managed runtime install policies', async () => {
|
||||
const sl = vi.fn(async () => 0);
|
||||
|
||||
|
|
@ -310,6 +334,23 @@ describe('runKtxCli', () => {
|
|||
expect(testIo.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('keeps representative JSON command stdout parseable', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await initKtxProject({ projectDir, projectName: 'warehouse' });
|
||||
const commands = [
|
||||
['--project-dir', projectDir, 'setup', 'status', '--json'],
|
||||
['--project-dir', projectDir, 'sl', 'list', '--json'],
|
||||
];
|
||||
|
||||
for (const argv of commands) {
|
||||
const testIo = makeIo();
|
||||
await expect(runKtxCli(argv, testIo.io)).resolves.toBe(0);
|
||||
|
||||
expect(() => JSON.parse(testIo.stdout())).not.toThrow();
|
||||
expect(testIo.stderr()).toBe('');
|
||||
}
|
||||
});
|
||||
|
||||
it('starts setup for bare ktx in a TTY when no project is discoverable', async () => {
|
||||
const { mkdtemp, realpath, rm } = await import('node:fs/promises');
|
||||
const { tmpdir } = await import('node:os');
|
||||
|
|
@ -1964,7 +2005,7 @@ describe('runKtxCli', () => {
|
|||
'--project-dir',
|
||||
tempDir,
|
||||
'--token-env',
|
||||
'NOTION_AUTH_TOKEN',
|
||||
'NOTION_TOKEN',
|
||||
'--crawl-mode',
|
||||
'selected_roots',
|
||||
'--root-page-id',
|
||||
|
|
@ -1991,7 +2032,7 @@ describe('runKtxCli', () => {
|
|||
force: false,
|
||||
allowLiteralCredentials: false,
|
||||
notion: {
|
||||
authTokenRef: 'env:NOTION_AUTH_TOKEN',
|
||||
authTokenRef: 'env:NOTION_TOKEN',
|
||||
crawlMode: 'selected_roots',
|
||||
rootPageIds: ['page-1'],
|
||||
rootDatabaseIds: ['database-1'],
|
||||
|
|
|
|||
|
|
@ -47,13 +47,18 @@ export { runKtxRuntime, type KtxRuntimeArgs, type KtxRuntimeDeps } from './runti
|
|||
export {
|
||||
allocateDaemonPort,
|
||||
readManagedPythonDaemonStatus,
|
||||
stopAllManagedPythonDaemons,
|
||||
startManagedPythonDaemon,
|
||||
stopManagedPythonDaemon,
|
||||
} from './managed-python-daemon.js';
|
||||
export type {
|
||||
ManagedPythonDaemonProcessInfo,
|
||||
ManagedPythonDaemonStartResult,
|
||||
ManagedPythonDaemonState,
|
||||
ManagedPythonDaemonStatus,
|
||||
ManagedPythonDaemonStopAllEntry,
|
||||
ManagedPythonDaemonStopAllFailure,
|
||||
ManagedPythonDaemonStopAllResult,
|
||||
ManagedPythonDaemonStopResult,
|
||||
} from './managed-python-daemon.js';
|
||||
export {
|
||||
|
|
|
|||
|
|
@ -331,8 +331,9 @@ describe('runKtxIngest viz and replay', () => {
|
|||
).resolves.toBe(0);
|
||||
|
||||
expect(runLocal).toHaveBeenCalledWith(expect.objectContaining({ memoryFlow: expect.anything() }));
|
||||
expect(io.stdout()).toContain('[5%] Fetching source files for warehouse/fake');
|
||||
expect(io.stderr()).toContain('[5%] Fetching source files for warehouse/fake');
|
||||
expect(io.stdout()).toContain('Job: plain-run');
|
||||
expect(io.stdout()).not.toContain('[5%]');
|
||||
expect(io.stdout()).not.toContain('KTX memory flow');
|
||||
});
|
||||
|
||||
|
|
@ -407,8 +408,9 @@ describe('runKtxIngest viz and replay', () => {
|
|||
|
||||
expect(startLiveMemoryFlow).not.toHaveBeenCalled();
|
||||
expect(runLocal).toHaveBeenCalledWith(expect.objectContaining({ memoryFlow: expect.anything() }));
|
||||
expect(io.stdout()).toContain('[5%] Fetching source files for warehouse/fake');
|
||||
expect(io.stderr()).toContain('[5%] Fetching source files for warehouse/fake');
|
||||
expect(io.stdout()).toContain('Job: raw-missing-viz-run');
|
||||
expect(io.stdout()).not.toContain('[5%]');
|
||||
expect(io.stdout()).not.toContain('KTX memory flow');
|
||||
expect(io.stderr()).toContain(
|
||||
'Visualization requested but stdin raw mode is unavailable; printing plain output.',
|
||||
|
|
|
|||
|
|
@ -546,7 +546,7 @@ export async function runPublicMetabaseSyncModeCase(tempDir: string, input: Sync
|
|||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(io.stderr()).toBe('');
|
||||
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
|
||||
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
|
||||
expect(io.stdout()).toContain(`target=warehouse_a database=1 status=done job=${jobId}`);
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import {
|
|||
import { initKtxProject, ktxLocalStateDbPath, loadKtxProject } from '@ktx/context/project';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { type KtxIngestArgs, runKtxIngest } from './ingest.js';
|
||||
import type { KtxCliLocalIngestAdaptersOptions } from './local-adapters.js';
|
||||
import {
|
||||
CliLookerSlWritingAgentRunner,
|
||||
CliMetabaseAgentRunner,
|
||||
|
|
@ -32,6 +33,7 @@ import {
|
|||
writeWarehouseConfig,
|
||||
} from './ingest.test-utils.js';
|
||||
import { resetVizFallbackWarningsForTest } from './viz-fallback.js';
|
||||
import { runKtxSetup } from './setup.js';
|
||||
|
||||
describe('runKtxIngest', () => {
|
||||
let tempDir: string;
|
||||
|
|
@ -105,6 +107,75 @@ describe('runKtxIngest', () => {
|
|||
expect(statusIo.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('prints provider setup guidance when a skip-llm setup project runs dev ingest', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
const setupIo = makeIo();
|
||||
await expect(
|
||||
runKtxSetup(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir,
|
||||
mode: 'new',
|
||||
agents: false,
|
||||
agentScope: 'project',
|
||||
agentInstallMode: 'cli',
|
||||
skipAgents: true,
|
||||
inputMode: 'disabled',
|
||||
yes: true,
|
||||
cliVersion: '0.0.0-test',
|
||||
skipLlm: true,
|
||||
skipEmbeddings: true,
|
||||
databaseDrivers: ['postgres'],
|
||||
databaseConnectionId: 'warehouse',
|
||||
databaseUrl: 'env:WAREHOUSE_URL',
|
||||
databaseSchemas: [],
|
||||
enableHistoricSql: true,
|
||||
skipDatabases: false,
|
||||
skipSources: true,
|
||||
},
|
||||
setupIo.io,
|
||||
{
|
||||
databasesDeps: {
|
||||
testConnection: async (_projectDir, _connectionId, io) => {
|
||||
io.stdout.write('Driver: postgres\nTables: 1\n');
|
||||
return 0;
|
||||
},
|
||||
scanConnection: async () => 0,
|
||||
historicSqlProbe: async () => ({ ok: true, lines: ['PASS Historic SQL probe skipped in test'] }),
|
||||
},
|
||||
context: async () => ({ status: 'skipped', projectDir }),
|
||||
},
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
const sourceDir = join(tempDir, 'source');
|
||||
await mkdir(join(sourceDir, 'orders'), { recursive: true });
|
||||
await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\n', 'utf-8');
|
||||
|
||||
const runIo = makeIo();
|
||||
await expect(
|
||||
runKtxIngest(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir,
|
||||
connectionId: 'warehouse',
|
||||
adapter: 'historic-sql',
|
||||
sourceDir,
|
||||
outputMode: 'plain',
|
||||
},
|
||||
runIo.io,
|
||||
),
|
||||
).resolves.toBe(1);
|
||||
|
||||
expect(runIo.stdout()).toBe('');
|
||||
expect(runIo.stderr()).toContain(
|
||||
'ktx dev ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.',
|
||||
);
|
||||
expect(runIo.stderr()).toContain(
|
||||
`ktx setup --project-dir ${projectDir} --anthropic-api-key-env ANTHROPIC_API_KEY --anthropic-model claude-sonnet-4-6 --no-input`,
|
||||
);
|
||||
});
|
||||
|
||||
it('routes metabase scheduled pulls to the fan-out runner and prints child summaries', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await writeMetabaseConfig(projectDir);
|
||||
|
|
@ -159,7 +230,7 @@ describe('runKtxIngest', () => {
|
|||
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
|
||||
expect(io.stdout()).toContain('warehouse_a');
|
||||
expect(io.stdout()).toContain('metabase-child-1');
|
||||
expect(io.stderr()).toBe('');
|
||||
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
|
||||
});
|
||||
|
||||
it('returns a non-zero code when Metabase fan-out has failed children', async () => {
|
||||
|
|
@ -229,7 +300,7 @@ describe('runKtxIngest', () => {
|
|||
expect(io.stdout()).toContain('Metabase fan-out: partial_failure');
|
||||
expect(io.stdout()).toContain('Failed work units: 1');
|
||||
expect(io.stdout()).toContain('status=error');
|
||||
expect(io.stderr()).toBe('');
|
||||
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
|
||||
});
|
||||
|
||||
it('prints Metabase fan-out progress before the final summary', async () => {
|
||||
|
|
@ -303,12 +374,56 @@ describe('runKtxIngest', () => {
|
|||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(io.stdout()).toContain('Metabase ingest: prod-metabase');
|
||||
expect(io.stdout()).toContain('Targets: 1 mapped database');
|
||||
expect(io.stdout()).toContain('- database=1 target=warehouse_a status=running job=metabase-child-1');
|
||||
expect(io.stdout()).toContain('- database=1 target=warehouse_a status=done job=metabase-child-1');
|
||||
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
|
||||
expect(io.stderr()).toContain('Targets: 1 mapped database');
|
||||
expect(io.stderr()).toContain('- database=1 target=warehouse_a status=running job=metabase-child-1');
|
||||
expect(io.stderr()).toContain('- database=1 target=warehouse_a status=done job=metabase-child-1');
|
||||
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
|
||||
expect(io.stderr()).toBe('');
|
||||
expect(io.stdout()).not.toContain('status=running job=metabase-child-1');
|
||||
});
|
||||
|
||||
it('writes metabase fan-out progress to stderr and final result to stdout', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await writeMetabaseConfig(projectDir);
|
||||
const io = makeIo({ isTTY: true });
|
||||
|
||||
await expect(
|
||||
runKtxIngest(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir,
|
||||
connectionId: 'prod-metabase',
|
||||
adapter: 'metabase',
|
||||
outputMode: 'plain',
|
||||
},
|
||||
io.io,
|
||||
{
|
||||
runLocalMetabaseIngest: async (input) => {
|
||||
input.progress?.onMetabaseFanoutPlanned?.({
|
||||
metabaseConnectionId: 'prod-metabase',
|
||||
children: [{ metabaseDatabaseId: 1, targetConnectionId: 'warehouse_a' }],
|
||||
});
|
||||
input.progress?.onMetabaseChildStarted?.({
|
||||
metabaseConnectionId: 'prod-metabase',
|
||||
metabaseDatabaseId: 1,
|
||||
targetConnectionId: 'warehouse_a',
|
||||
jobId: 'metabase-child-1',
|
||||
});
|
||||
return {
|
||||
metabaseConnectionId: 'prod-metabase',
|
||||
status: 'all_succeeded',
|
||||
totals: { workUnits: 0, failedWorkUnits: 0 },
|
||||
children: [],
|
||||
};
|
||||
},
|
||||
},
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
|
||||
expect(io.stderr()).toContain('status=running job=metabase-child-1');
|
||||
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
|
||||
expect(io.stdout()).not.toContain('status=running job=metabase-child-1');
|
||||
});
|
||||
|
||||
it('runs Metabase scheduled ingest through the public CLI command path with real fan-out', async () => {
|
||||
|
|
@ -393,7 +508,8 @@ describe('runKtxIngest', () => {
|
|||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(io.stderr()).toBe('');
|
||||
expect(io.stderr()).toContain('Metabase ingest: prod-metabase');
|
||||
expect(io.stderr()).toContain('Targets: 2 mapped databases');
|
||||
expect(io.stdout()).toContain('Metabase fan-out: all_succeeded');
|
||||
expect(io.stdout()).toContain('Source: prod-metabase');
|
||||
expect(io.stdout()).toContain('Children: 2');
|
||||
|
|
@ -483,6 +599,46 @@ describe('runKtxIngest', () => {
|
|||
expect(io.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('keeps metabase JSON stdout free of operational adapter logs', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await writeMetabaseConfig(projectDir);
|
||||
const io = makeIo();
|
||||
let adapterOptions: KtxCliLocalIngestAdaptersOptions | undefined;
|
||||
|
||||
await expect(
|
||||
runKtxIngest(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir,
|
||||
connectionId: 'prod-metabase',
|
||||
adapter: 'metabase',
|
||||
outputMode: 'json',
|
||||
},
|
||||
io.io,
|
||||
{
|
||||
createAdapters: (_project, options) => {
|
||||
adapterOptions = options;
|
||||
options?.logger?.warn('adapter warning');
|
||||
return [];
|
||||
},
|
||||
runLocalMetabaseIngest: async (input) => {
|
||||
input.adapters.find((adapter) => adapter.source === 'metabase');
|
||||
return {
|
||||
metabaseConnectionId: 'prod-metabase',
|
||||
status: 'all_succeeded',
|
||||
totals: { workUnits: 0, failedWorkUnits: 0 },
|
||||
children: [],
|
||||
};
|
||||
},
|
||||
},
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(adapterOptions?.logger).toEqual(expect.objectContaining({ warn: expect.any(Function) }));
|
||||
expect(() => JSON.parse(io.stdout())).not.toThrow();
|
||||
expect(io.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('rejects source-dir uploads through the metabase fan-out route', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await writeMetabaseConfig(projectDir);
|
||||
|
|
@ -694,17 +850,22 @@ describe('runKtxIngest', () => {
|
|||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
|
||||
databaseIntrospectionUrl: 'http://127.0.0.1:8765',
|
||||
});
|
||||
expect(createAdapters).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ projectDir }),
|
||||
expect.objectContaining({
|
||||
databaseIntrospectionUrl: 'http://127.0.0.1:8765',
|
||||
logger: expect.any(Object),
|
||||
}),
|
||||
);
|
||||
expect(runLocal).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
adapters: createdAdapters,
|
||||
adapter: 'fake',
|
||||
connectionId: 'warehouse',
|
||||
pullConfigOptions: {
|
||||
pullConfigOptions: expect.objectContaining({
|
||||
databaseIntrospectionUrl: 'http://127.0.0.1:8765',
|
||||
},
|
||||
logger: expect.any(Object),
|
||||
}),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
|
@ -747,14 +908,19 @@ describe('runKtxIngest', () => {
|
|||
installPolicy: 'auto',
|
||||
io: io.io,
|
||||
};
|
||||
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
|
||||
managedDaemon: expectedManagedDaemon,
|
||||
});
|
||||
expect(createAdapters).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ projectDir }),
|
||||
expect.objectContaining({
|
||||
managedDaemon: expectedManagedDaemon,
|
||||
logger: expect.any(Object),
|
||||
}),
|
||||
);
|
||||
expect(runLocal).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
pullConfigOptions: {
|
||||
pullConfigOptions: expect.objectContaining({
|
||||
managedDaemon: expectedManagedDaemon,
|
||||
},
|
||||
logger: expect.any(Object),
|
||||
}),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
|
@ -808,9 +974,13 @@ describe('runKtxIngest', () => {
|
|||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
|
||||
historicSqlConnectionId: 'warehouse',
|
||||
});
|
||||
expect(createAdapters).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ projectDir }),
|
||||
expect.objectContaining({
|
||||
historicSqlConnectionId: 'warehouse',
|
||||
logger: expect.any(Object),
|
||||
}),
|
||||
);
|
||||
expect(runLocal).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
adapters: createdAdapters,
|
||||
|
|
@ -912,12 +1082,228 @@ describe('runKtxIngest', () => {
|
|||
expect(stdout).toContain('[45%] Planned 1 work unit');
|
||||
expect(stdout).toContain('[80%] Processed 1/1 work units');
|
||||
expect(stdout).toContain('[100%] Ingest completed');
|
||||
expect(stdout.indexOf('[5%] Fetching source files for warehouse/historic-sql')).toBeLessThan(
|
||||
stdout.indexOf('Report: report-live-1'),
|
||||
);
|
||||
expect(stdout).toContain('Report: report-live-1');
|
||||
expect(io.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('writes plain TTY ingest progress and final report to stdout', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await writeWarehouseConfig(projectDir);
|
||||
const sourceDir = join(tempDir, 'source');
|
||||
await mkdir(join(sourceDir, 'orders'), { recursive: true });
|
||||
await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\n', 'utf-8');
|
||||
const runLocal = vi.fn(async (input: RunLocalIngestOptions) => completedLocalBundleRun(input, 'local-job-1'));
|
||||
const io = makeIo({ isTTY: true });
|
||||
|
||||
await expect(
|
||||
runKtxIngest(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir,
|
||||
connectionId: 'warehouse',
|
||||
adapter: 'fake',
|
||||
sourceDir,
|
||||
outputMode: 'plain',
|
||||
},
|
||||
io.io,
|
||||
{
|
||||
env: interactiveEnv(),
|
||||
runLocalIngest: runLocal,
|
||||
},
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(io.stdout()).toContain('[5%] Fetching source files for warehouse/fake');
|
||||
expect(io.stdout()).toContain('Report: report-live-1');
|
||||
expect(io.stderr()).toBe('');
|
||||
});
|
||||
|
||||
it('prints plain WorkUnit step progress during long-running local ingest', async () => {
|
||||
const projectDir = join(tempDir, 'historic-sql-step-progress-project');
|
||||
await mkdir(projectDir, { recursive: true });
|
||||
await writeFile(
|
||||
join(projectDir, 'ktx.yaml'),
|
||||
[
|
||||
'project: historic-sql-step-progress-project',
|
||||
'connections:',
|
||||
' warehouse:',
|
||||
' driver: postgres',
|
||||
' url: env:WAREHOUSE_DATABASE_URL',
|
||||
' historicSql:',
|
||||
' enabled: true',
|
||||
' dialect: postgres',
|
||||
' minExecutions: 2',
|
||||
'ingest:',
|
||||
' adapters:',
|
||||
' - historic-sql',
|
||||
'',
|
||||
].join('\n'),
|
||||
'utf-8',
|
||||
);
|
||||
const createdAdapters: SourceAdapter[] = [
|
||||
{ source: 'historic-sql', skillNames: [], detect: async () => true, chunk: async () => ({ workUnits: [] }) },
|
||||
];
|
||||
const runLocal = vi.fn(async (input: RunLocalIngestOptions) => {
|
||||
input.memoryFlow?.update({
|
||||
plannedWorkUnits: [
|
||||
{
|
||||
unitKey: 'historic-sql-table-public-orders',
|
||||
rawFiles: ['tables/public/orders.json'],
|
||||
peerFileCount: 0,
|
||||
dependencyCount: 0,
|
||||
},
|
||||
{
|
||||
unitKey: 'historic-sql-table-public-customers',
|
||||
rawFiles: ['tables/public/customers.json'],
|
||||
peerFileCount: 0,
|
||||
dependencyCount: 0,
|
||||
},
|
||||
],
|
||||
});
|
||||
input.memoryFlow?.emit({ type: 'chunks_planned', chunkCount: 2, workUnitCount: 2, evictionCount: 0 });
|
||||
input.memoryFlow?.emit({
|
||||
type: 'work_unit_started',
|
||||
unitKey: 'historic-sql-table-public-orders',
|
||||
skills: ['historic_sql_table_digest'],
|
||||
stepBudget: 40,
|
||||
});
|
||||
input.memoryFlow?.emit({
|
||||
type: 'work_unit_step',
|
||||
unitKey: 'historic-sql-table-public-orders',
|
||||
stepIndex: 7,
|
||||
stepBudget: 40,
|
||||
});
|
||||
input.memoryFlow?.emit({
|
||||
type: 'work_unit_finished',
|
||||
unitKey: 'historic-sql-table-public-orders',
|
||||
status: 'success',
|
||||
});
|
||||
input.memoryFlow?.finish('done');
|
||||
return completedLocalBundleRun(input, input.jobId ?? 'historic-step-progress-job');
|
||||
});
|
||||
const io = makeIo({ isTTY: true });
|
||||
|
||||
await expect(
|
||||
runKtxIngest(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir,
|
||||
connectionId: 'warehouse',
|
||||
adapter: 'historic-sql',
|
||||
outputMode: 'plain',
|
||||
},
|
||||
io.io,
|
||||
{
|
||||
env: interactiveEnv(),
|
||||
createAdapters: vi.fn(() => createdAdapters as never),
|
||||
runLocalIngest: runLocal,
|
||||
jobIdFactory: () => 'historic-step-progress-job',
|
||||
},
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
const stdout = io.stdout();
|
||||
expect(stdout).toContain('[45%] Planned 2 work units');
|
||||
expect(stdout).toContain('[55%] Processing 1/2 work units: historic-sql-table-public-orders');
|
||||
expect(stdout).toContain(
|
||||
'\r[58%] Processing work units: 0/2 complete, 1 active; latest historic-sql-table-public-orders step 7/40\u001b[K',
|
||||
);
|
||||
expect(stdout).toContain('[68%] Processed 1/2 work units');
|
||||
});
|
||||
|
||||
it('renders concurrent WorkUnit step progress as transient aggregate status', async () => {
|
||||
const projectDir = join(tempDir, 'historic-sql-concurrent-progress-project');
|
||||
await mkdir(projectDir, { recursive: true });
|
||||
await writeFile(
|
||||
join(projectDir, 'ktx.yaml'),
|
||||
[
|
||||
'project: historic-sql-concurrent-progress-project',
|
||||
'connections:',
|
||||
' warehouse:',
|
||||
' driver: postgres',
|
||||
' url: env:WAREHOUSE_DATABASE_URL',
|
||||
' historicSql:',
|
||||
' enabled: true',
|
||||
' dialect: postgres',
|
||||
' minExecutions: 2',
|
||||
'ingest:',
|
||||
' adapters:',
|
||||
' - historic-sql',
|
||||
'',
|
||||
].join('\n'),
|
||||
'utf-8',
|
||||
);
|
||||
const createdAdapters: SourceAdapter[] = [
|
||||
{ source: 'historic-sql', skillNames: [], detect: async () => true, chunk: async () => ({ workUnits: [] }) },
|
||||
];
|
||||
const workUnitKeys = [
|
||||
'historic-sql-table-public-orders',
|
||||
'historic-sql-table-public-customers',
|
||||
'historic-sql-table-public-line-items',
|
||||
'historic-sql-table-public-payments',
|
||||
'historic-sql-table-public-products',
|
||||
'historic-sql-table-public-suppliers',
|
||||
];
|
||||
const runLocal = vi.fn(async (input: RunLocalIngestOptions) => {
|
||||
input.memoryFlow?.update({
|
||||
plannedWorkUnits: workUnitKeys.map((unitKey) => ({
|
||||
unitKey,
|
||||
rawFiles: [`tables/${unitKey}.json`],
|
||||
peerFileCount: 0,
|
||||
dependencyCount: 0,
|
||||
})),
|
||||
});
|
||||
input.memoryFlow?.emit({
|
||||
type: 'chunks_planned',
|
||||
chunkCount: workUnitKeys.length,
|
||||
workUnitCount: workUnitKeys.length,
|
||||
evictionCount: 0,
|
||||
});
|
||||
for (const unitKey of workUnitKeys) {
|
||||
input.memoryFlow?.emit({
|
||||
type: 'work_unit_started',
|
||||
unitKey,
|
||||
skills: ['historic_sql_table_digest'],
|
||||
stepBudget: 40,
|
||||
});
|
||||
}
|
||||
for (const unitKey of workUnitKeys) {
|
||||
input.memoryFlow?.emit({ type: 'work_unit_step', unitKey, stepIndex: 1, stepBudget: 40 });
|
||||
}
|
||||
input.memoryFlow?.finish('done');
|
||||
return completedLocalBundleRun(input, input.jobId ?? 'historic-concurrent-progress-job');
|
||||
});
|
||||
const io = makeIo({ isTTY: true });
|
||||
|
||||
await expect(
|
||||
runKtxIngest(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir,
|
||||
connectionId: 'warehouse',
|
||||
adapter: 'historic-sql',
|
||||
outputMode: 'plain',
|
||||
},
|
||||
io.io,
|
||||
{
|
||||
env: interactiveEnv(),
|
||||
createAdapters: vi.fn(() => createdAdapters as never),
|
||||
runLocalIngest: runLocal,
|
||||
jobIdFactory: () => 'historic-concurrent-progress-job',
|
||||
},
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
const stdout = io.stdout();
|
||||
expect(stdout).toContain(
|
||||
'\r[56%] Processing work units: 0/6 complete, 6 active; latest historic-sql-table-public-suppliers step 1/40\u001b[K',
|
||||
);
|
||||
expect(stdout).not.toContain(
|
||||
'\n[56%] Processing 6/6 work units: historic-sql-table-public-suppliers step 1/40\n',
|
||||
);
|
||||
expect(stdout).toContain('\n[100%] Ingest completed\n');
|
||||
});
|
||||
|
||||
it('passes local Looker pull-config options and agent runner into scheduled ingest for Looker scheduled ingest', async () => {
|
||||
const projectDir = join(tempDir, 'project');
|
||||
await writeWarehouseConfig(projectDir);
|
||||
|
|
@ -958,15 +1344,19 @@ describe('runKtxIngest', () => {
|
|||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(createAdapters).toHaveBeenCalledWith(expect.objectContaining({ projectDir }), {
|
||||
looker: {
|
||||
parser: pullConfigOptions.looker.parser,
|
||||
},
|
||||
});
|
||||
expect(createAdapters).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ projectDir }),
|
||||
expect.objectContaining({
|
||||
logger: expect.any(Object),
|
||||
looker: {
|
||||
parser: pullConfigOptions.looker.parser,
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(runLocal).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
agentRunner,
|
||||
pullConfigOptions,
|
||||
pullConfigOptions: expect.objectContaining(pullConfigOptions),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import {
|
|||
} from '@ktx/context/ingest';
|
||||
import { loadKtxProject } from '@ktx/context/project';
|
||||
import { readIngestReportSnapshotFile } from './ingest-report-file.js';
|
||||
import { createCliOperationalLogger } from './io/logger.js';
|
||||
import { createKtxCliLocalIngestAdapters } from './local-adapters.js';
|
||||
import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js';
|
||||
import { type KtxMemoryFlowStdin, renderMemoryFlowInteractively } from './memory-flow-interactive.js';
|
||||
|
|
@ -142,22 +143,22 @@ function createMetabaseFanoutProgress(
|
|||
connectionId: string,
|
||||
io: KtxIngestIo,
|
||||
): LocalMetabaseFanoutProgress {
|
||||
io.stdout.write(`Metabase ingest: ${connectionId}\n`);
|
||||
io.stdout.write('Checking mappings and scheduled-pull targets...\n');
|
||||
io.stderr.write(`Metabase ingest: ${connectionId}\n`);
|
||||
io.stderr.write('Checking mappings and scheduled-pull targets...\n');
|
||||
return {
|
||||
onMetabaseFanoutPlanned(event) {
|
||||
io.stdout.write(`Targets: ${pluralize(event.children.length, 'mapped database')}\n`);
|
||||
io.stderr.write(`Targets: ${pluralize(event.children.length, 'mapped database')}\n`);
|
||||
for (const child of event.children) {
|
||||
io.stdout.write(`- database=${child.metabaseDatabaseId} target=${child.targetConnectionId} status=queued\n`);
|
||||
io.stderr.write(`- database=${child.metabaseDatabaseId} target=${child.targetConnectionId} status=queued\n`);
|
||||
}
|
||||
},
|
||||
onMetabaseChildStarted(event) {
|
||||
io.stdout.write(
|
||||
io.stderr.write(
|
||||
`- database=${event.metabaseDatabaseId} target=${event.targetConnectionId} status=running job=${event.jobId}\n`,
|
||||
);
|
||||
},
|
||||
onMetabaseChildCompleted(event) {
|
||||
io.stdout.write(
|
||||
io.stderr.write(
|
||||
`- database=${event.metabaseDatabaseId} target=${event.targetConnectionId} status=${event.status} job=${event.jobId}\n`,
|
||||
);
|
||||
},
|
||||
|
|
@ -168,14 +169,51 @@ function formatDiffProgress(event: Extract<MemoryFlowEvent, { type: 'diff_comput
|
|||
return `+${event.added}/~${event.modified}/-${event.deleted}/=${event.unchanged}`;
|
||||
}
|
||||
|
||||
function completedWorkUnitCount(snapshot: MemoryFlowReplayInput): number {
|
||||
return snapshot.events.filter((event) => event.type === 'work_unit_finished').length;
|
||||
function workUnitEventsThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): MemoryFlowEvent[] {
|
||||
return snapshot.events.slice(0, eventIndex + 1);
|
||||
}
|
||||
|
||||
function completedWorkUnitCountThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): number {
|
||||
return workUnitEventsThrough(snapshot, eventIndex).filter((event) => event.type === 'work_unit_finished').length;
|
||||
}
|
||||
|
||||
function activeWorkUnitCountThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): number {
|
||||
const active = new Set<string>();
|
||||
for (const event of workUnitEventsThrough(snapshot, eventIndex)) {
|
||||
if (event.type === 'work_unit_started') {
|
||||
active.add(event.unitKey);
|
||||
}
|
||||
if (event.type === 'work_unit_finished') {
|
||||
active.delete(event.unitKey);
|
||||
}
|
||||
}
|
||||
return active.size;
|
||||
}
|
||||
|
||||
function plannedWorkUnitCountThrough(snapshot: MemoryFlowReplayInput, eventIndex: number): number {
|
||||
if (snapshot.plannedWorkUnits.length > 0) {
|
||||
return snapshot.plannedWorkUnits.length;
|
||||
}
|
||||
const planEvent = workUnitEventsThrough(snapshot, eventIndex)
|
||||
.filter((event) => event.type === 'chunks_planned')
|
||||
.at(-1);
|
||||
return planEvent?.workUnitCount ?? completedWorkUnitCountThrough(snapshot, eventIndex);
|
||||
}
|
||||
|
||||
function workUnitOrdinalThrough(snapshot: MemoryFlowReplayInput, eventIndex: number, unitKey: string): number {
|
||||
const events = workUnitEventsThrough(snapshot, eventIndex);
|
||||
const startedIndex = events.findIndex((event) => event.type === 'work_unit_started' && event.unitKey === unitKey);
|
||||
if (startedIndex === -1) {
|
||||
return completedWorkUnitCountThrough(snapshot, eventIndex) + 1;
|
||||
}
|
||||
return events.slice(0, startedIndex + 1).filter((event) => event.type === 'work_unit_started').length;
|
||||
}
|
||||
|
||||
function plainIngestEventProgress(
|
||||
event: MemoryFlowEvent,
|
||||
snapshot: MemoryFlowReplayInput,
|
||||
): { percent: number; message: string } | null {
|
||||
eventIndex: number,
|
||||
): { percent: number; message: string; transient?: boolean } | null {
|
||||
switch (event.type) {
|
||||
case 'source_acquired':
|
||||
return {
|
||||
|
|
@ -196,11 +234,28 @@ function plainIngestEventProgress(
|
|||
};
|
||||
case 'stage_skipped':
|
||||
return { percent: 45, message: `Skipped ${event.stage}: ${event.reason}` };
|
||||
case 'work_unit_started':
|
||||
return { percent: 55, message: `Processing ${event.unitKey}` };
|
||||
case 'work_unit_started': {
|
||||
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
||||
const ordinal = workUnitOrdinalThrough(snapshot, eventIndex, event.unitKey);
|
||||
const progress = total > 0 ? `${ordinal}/${total} work units: ` : '';
|
||||
return { percent: 55, message: `Processing ${progress}${event.unitKey}` };
|
||||
}
|
||||
case 'work_unit_step': {
|
||||
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
||||
const completed = completedWorkUnitCountThrough(snapshot, eventIndex);
|
||||
const active = activeWorkUnitCountThrough(snapshot, eventIndex);
|
||||
const stepFraction = event.stepBudget > 0 ? Math.min(1, event.stepIndex / event.stepBudget) : 0;
|
||||
const percent = total > 0 ? 55 + Math.ceil(((completed + stepFraction) / total) * 25) : 55;
|
||||
const latest = `${event.unitKey} step ${event.stepIndex}/${event.stepBudget}`;
|
||||
return {
|
||||
percent,
|
||||
message: `Processing work units: ${completed}/${total} complete, ${active} active; latest ${latest}`,
|
||||
transient: true,
|
||||
};
|
||||
}
|
||||
case 'work_unit_finished': {
|
||||
const total = snapshot.plannedWorkUnits.length || completedWorkUnitCount(snapshot);
|
||||
const completed = completedWorkUnitCount(snapshot);
|
||||
const total = plannedWorkUnitCountThrough(snapshot, eventIndex);
|
||||
const completed = completedWorkUnitCountThrough(snapshot, eventIndex);
|
||||
const percent = total > 0 ? 55 + Math.round((completed / total) * 25) : 80;
|
||||
return {
|
||||
percent,
|
||||
|
|
@ -225,7 +280,6 @@ function plainIngestEventProgress(
|
|||
case 'report_created':
|
||||
return { percent: 98, message: `Created ingest report ${event.reportPath ?? event.runId}` };
|
||||
case 'scope_detected':
|
||||
case 'work_unit_step':
|
||||
case 'candidate_action':
|
||||
return null;
|
||||
}
|
||||
|
|
@ -242,15 +296,31 @@ function shouldWritePlainIngestProgress(
|
|||
function createPlainIngestProgressRenderer(
|
||||
args: Extract<KtxIngestArgs, { command: 'run' }>,
|
||||
io: KtxIngestIo,
|
||||
): { start(): void; update(snapshot: MemoryFlowReplayInput): void } {
|
||||
): { start(): void; update(snapshot: MemoryFlowReplayInput): void; flush(): void } {
|
||||
let printedEvents = 0;
|
||||
let lastPercent = 0;
|
||||
let printedCompletion = false;
|
||||
let hasPendingTransient = false;
|
||||
|
||||
const write = (percent: number, message: string) => {
|
||||
const flush = () => {
|
||||
if (!hasPendingTransient) {
|
||||
return;
|
||||
}
|
||||
io.stdout.write('\n');
|
||||
hasPendingTransient = false;
|
||||
};
|
||||
|
||||
const write = (percent: number, message: string, options?: { transient?: boolean }) => {
|
||||
const nextPercent = Math.max(lastPercent, Math.max(0, Math.min(100, percent)));
|
||||
lastPercent = nextPercent;
|
||||
io.stdout.write(`[${nextPercent}%] ${message}\n`);
|
||||
const line = `[${nextPercent}%] ${message}`;
|
||||
if (options?.transient === true) {
|
||||
io.stdout.write(`\r${line}\u001b[K`);
|
||||
hasPendingTransient = true;
|
||||
return;
|
||||
}
|
||||
flush();
|
||||
io.stdout.write(`${line}\n`);
|
||||
};
|
||||
|
||||
return {
|
||||
|
|
@ -259,13 +329,14 @@ function createPlainIngestProgressRenderer(
|
|||
},
|
||||
update(snapshot) {
|
||||
while (printedEvents < snapshot.events.length) {
|
||||
const eventIndex = printedEvents;
|
||||
const event = snapshot.events[printedEvents++];
|
||||
if (!event) {
|
||||
continue;
|
||||
}
|
||||
const progress = plainIngestEventProgress(event, snapshot);
|
||||
const progress = plainIngestEventProgress(event, snapshot, eventIndex);
|
||||
if (progress) {
|
||||
write(progress.percent, progress.message);
|
||||
write(progress.percent, progress.message, progress.transient === true ? { transient: true } : undefined);
|
||||
}
|
||||
}
|
||||
if (!printedCompletion && snapshot.status !== 'running') {
|
||||
|
|
@ -273,6 +344,7 @@ function createPlainIngestProgressRenderer(
|
|||
write(100, snapshot.status === 'done' ? 'Ingest completed' : 'Ingest failed');
|
||||
}
|
||||
},
|
||||
flush,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -435,11 +507,13 @@ export async function runKtxIngest(
|
|||
const executeLocalIngest = deps.runLocalIngest ?? runLocalIngest;
|
||||
const localIngestOptions = deps.localIngestOptions ?? {};
|
||||
const managedDaemon = managedDaemonOptionsForIngestRun(args, io);
|
||||
const operationalLogger = createCliOperationalLogger(io, args.outputMode);
|
||||
const adapterOptions = {
|
||||
...(localIngestOptions.pullConfigOptions ?? {}),
|
||||
...(args.databaseIntrospectionUrl ? { databaseIntrospectionUrl: args.databaseIntrospectionUrl } : {}),
|
||||
...(managedDaemon ? { managedDaemon } : {}),
|
||||
...(args.adapter === 'historic-sql' ? { historicSqlConnectionId: args.connectionId } : {}),
|
||||
logger: operationalLogger,
|
||||
};
|
||||
if (args.adapter === 'metabase' && args.sourceDir) {
|
||||
throw new Error('source-dir uploads are not supported for the Metabase fan-out adapter');
|
||||
|
|
@ -524,6 +598,7 @@ export async function runKtxIngest(
|
|||
io.stdout.write(formatMemoryFlowFinalSummary(latestMemoryFlowSnapshot));
|
||||
return reportStatus(result.report) === 'done' ? 0 : 1;
|
||||
}
|
||||
plainProgress?.flush();
|
||||
await writeReportRecord(result.report, runOutputMode, io, {
|
||||
interactive: (args.inputMode ?? 'auto') === 'auto',
|
||||
renderStoredMemoryFlow: deps.renderStoredMemoryFlow,
|
||||
|
|
@ -531,6 +606,7 @@ export async function runKtxIngest(
|
|||
});
|
||||
return reportStatus(result.report) === 'done' ? 0 : 1;
|
||||
} finally {
|
||||
plainProgress?.flush();
|
||||
liveTui?.close();
|
||||
}
|
||||
}
|
||||
|
|
|
|||
65
packages/cli/src/io/logger.test.ts
Normal file
65
packages/cli/src/io/logger.test.ts
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { createCliOperationalLogger, createNoopOperationalLogger } from './logger.js';
|
||||
|
||||
function makeIo() {
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
return {
|
||||
io: {
|
||||
stdout: {
|
||||
write: (chunk: string) => {
|
||||
stdout += chunk;
|
||||
},
|
||||
},
|
||||
stderr: {
|
||||
write: (chunk: string) => {
|
||||
stderr += chunk;
|
||||
},
|
||||
},
|
||||
},
|
||||
stdout: () => stdout,
|
||||
stderr: () => stderr,
|
||||
};
|
||||
}
|
||||
|
||||
describe('createCliOperationalLogger', () => {
|
||||
it('routes operational messages to stderr outside JSON mode', () => {
|
||||
const io = makeIo();
|
||||
const logger = createCliOperationalLogger(io.io, 'plain');
|
||||
|
||||
logger.log('progress');
|
||||
logger.warn('warning');
|
||||
logger.error('failure');
|
||||
logger.debug?.('debug');
|
||||
|
||||
expect(io.stdout()).toBe('');
|
||||
expect(io.stderr()).toBe('progress\nwarning\nfailure\ndebug\n');
|
||||
});
|
||||
|
||||
it('suppresses operational messages in JSON mode by default', () => {
|
||||
const io = makeIo();
|
||||
const logger = createCliOperationalLogger(io.io, 'json');
|
||||
|
||||
logger.log('progress');
|
||||
logger.warn('warning');
|
||||
logger.error('failure');
|
||||
logger.debug?.('debug');
|
||||
|
||||
expect(io.stdout()).toBe('');
|
||||
expect(io.stderr()).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('createNoopOperationalLogger', () => {
|
||||
it('never writes', () => {
|
||||
const logger = createNoopOperationalLogger();
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined);
|
||||
|
||||
logger.log('progress');
|
||||
logger.warn('warning');
|
||||
logger.error('failure');
|
||||
logger.debug?.('debug');
|
||||
|
||||
expect(warn).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
40
packages/cli/src/io/logger.ts
Normal file
40
packages/cli/src/io/logger.ts
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
import type { KtxCliIo } from '../cli-runtime.js';
|
||||
import type { KtxOutputMode } from './mode.js';
|
||||
|
||||
export interface KtxOperationalLogger {
|
||||
log(message: string): void;
|
||||
warn(message: string): void;
|
||||
error(message: string): void;
|
||||
debug?(message: string): void;
|
||||
}
|
||||
|
||||
export type KtxOperationalOutputMode = KtxOutputMode | 'viz';
|
||||
|
||||
function writeLine(io: KtxCliIo, message: string): void {
|
||||
io.stderr.write(message.endsWith('\n') ? message : `${message}\n`);
|
||||
}
|
||||
|
||||
export function createNoopOperationalLogger(): KtxOperationalLogger {
|
||||
return {
|
||||
log: () => undefined,
|
||||
warn: () => undefined,
|
||||
error: () => undefined,
|
||||
debug: () => undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export function createCliOperationalLogger(
|
||||
io: KtxCliIo,
|
||||
mode: KtxOperationalOutputMode,
|
||||
): KtxOperationalLogger {
|
||||
if (mode === 'json') {
|
||||
return createNoopOperationalLogger();
|
||||
}
|
||||
|
||||
return {
|
||||
log: (message) => writeLine(io, message),
|
||||
warn: (message) => writeLine(io, message),
|
||||
error: (message) => writeLine(io, message),
|
||||
debug: (message) => writeLine(io, message),
|
||||
};
|
||||
}
|
||||
|
|
@ -28,6 +28,16 @@ export interface PrintListArgs<Row> {
|
|||
io: KtxCliIo;
|
||||
}
|
||||
|
||||
export interface KtxJsonResultEnvelope<T> {
|
||||
kind: string;
|
||||
data: T;
|
||||
meta?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export function writeJsonResult<T>(io: KtxCliIo, envelope: KtxJsonResultEnvelope<T>): void {
|
||||
io.stdout.write(`${JSON.stringify(envelope, null, 2)}\n`);
|
||||
}
|
||||
|
||||
export function printList<Row extends object>(args: PrintListArgs<Row>): void {
|
||||
switch (args.mode) {
|
||||
case 'json':
|
||||
|
|
@ -61,12 +71,11 @@ function printListPlain<Row extends object>(args: PrintListArgs<Row>): void {
|
|||
}
|
||||
|
||||
function printListJson<Row extends object>(args: PrintListArgs<Row>): void {
|
||||
const envelope = {
|
||||
writeJsonResult(args.io, {
|
||||
kind: 'list',
|
||||
data: { items: args.rows },
|
||||
meta: { command: args.command },
|
||||
};
|
||||
args.io.stdout.write(`${JSON.stringify(envelope, null, 2)}\n`);
|
||||
});
|
||||
}
|
||||
|
||||
function pluralize(count: number, singular: string): string {
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ import {
|
|||
managedDaemonDatabaseIntrospectionOptions,
|
||||
type ManagedPythonCoreDaemonOptions,
|
||||
} from './managed-python-http.js';
|
||||
import type { KtxOperationalLogger } from './io/logger.js';
|
||||
|
||||
function hasSnowflakeDriver(connection: unknown): boolean {
|
||||
return (
|
||||
|
|
@ -162,6 +163,7 @@ export interface KtxCliLocalIngestAdaptersOptions extends DefaultLocalIngestAdap
|
|||
sqlAnalysis?: SqlAnalysisPort;
|
||||
sqlAnalysisUrl?: string;
|
||||
managedDaemon?: ManagedPythonCoreDaemonOptions;
|
||||
logger?: KtxOperationalLogger;
|
||||
}
|
||||
|
||||
function historicSqlRecord(connection: unknown): Record<string, unknown> | null {
|
||||
|
|
|
|||
|
|
@ -214,6 +214,7 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
|
|||
|
||||
expect(confirmInstall).toHaveBeenCalledWith(
|
||||
'KTX needs to install the core Python runtime. This downloads Python dependencies with uv. Continue?',
|
||||
io.io,
|
||||
);
|
||||
expect(installRuntime).toHaveBeenCalledWith({
|
||||
cliVersion: '0.2.0',
|
||||
|
|
@ -221,4 +222,45 @@ describe('createManagedPythonSemanticLayerComputePort', () => {
|
|||
force: false,
|
||||
});
|
||||
});
|
||||
|
||||
it('uses injected runtime confirmation instead of reading process TTY directly', async () => {
|
||||
const io = makeIo();
|
||||
const compute = { query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() };
|
||||
const installRuntime = vi.fn(async (): Promise<ManagedPythonRuntimeInstallResult> => installResult());
|
||||
const confirmInstall = vi.fn(async () => true);
|
||||
|
||||
await expect(
|
||||
createManagedPythonSemanticLayerComputePort({
|
||||
cliVersion: '0.2.0',
|
||||
installPolicy: 'prompt',
|
||||
io: io.io,
|
||||
readStatus: async () => missingStatus(),
|
||||
installRuntime,
|
||||
confirmInstall,
|
||||
createPythonCompute: () => compute,
|
||||
}),
|
||||
).resolves.toBe(compute);
|
||||
|
||||
expect(confirmInstall).toHaveBeenCalledWith(
|
||||
'KTX needs to install the core Python runtime. This downloads Python dependencies with uv. Continue?',
|
||||
io.io,
|
||||
);
|
||||
expect(io.stderr()).toContain('Installing KTX Python runtime (core) with uv...');
|
||||
});
|
||||
|
||||
it('can decide default runtime prompting from injected io capabilities', async () => {
|
||||
const io = makeIo();
|
||||
Object.assign(io.io.stdout, { isTTY: false });
|
||||
|
||||
await expect(
|
||||
createManagedPythonSemanticLayerComputePort({
|
||||
cliVersion: '0.2.0',
|
||||
installPolicy: 'prompt',
|
||||
io: io.io,
|
||||
readStatus: async () => missingStatus(),
|
||||
installRuntime: vi.fn(),
|
||||
createPythonCompute: () => ({ query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }),
|
||||
}),
|
||||
).rejects.toThrow('KTX Python runtime installation was cancelled');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { cancel, confirm, isCancel } from '@clack/prompts';
|
||||
import { createPythonSemanticLayerComputePort, type KtxSemanticLayerComputePort } from '@ktx/context/daemon';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { createClackPromptAdapter } from './clack.js';
|
||||
import {
|
||||
installManagedPythonRuntime,
|
||||
readManagedPythonRuntimeStatus,
|
||||
|
|
@ -36,7 +36,7 @@ export interface ManagedPythonCommandRuntime {
|
|||
export interface ManagedPythonCommandDeps {
|
||||
readStatus?: (options: ManagedPythonRuntimeLayoutOptions) => Promise<ManagedPythonRuntimeStatus>;
|
||||
installRuntime?: (options: ManagedPythonRuntimeInstallOptions) => Promise<ManagedPythonRuntimeInstallResult>;
|
||||
confirmInstall?: (message: string) => Promise<boolean>;
|
||||
confirmInstall?: (message: string, io: KtxCliIo) => Promise<boolean>;
|
||||
}
|
||||
|
||||
export interface ManagedPythonCommandOptions extends ManagedPythonCommandDeps {
|
||||
|
|
@ -69,16 +69,12 @@ function hasFeature(manifest: InstalledKtxRuntimeManifest, feature: KtxRuntimeFe
|
|||
return manifest.features.includes(feature);
|
||||
}
|
||||
|
||||
async function defaultConfirmInstall(message: string): Promise<boolean> {
|
||||
if (process.stdin.isTTY !== true || process.stdout.isTTY !== true) {
|
||||
async function defaultConfirmInstall(message: string, io: KtxCliIo): Promise<boolean> {
|
||||
if (io.stdout.isTTY !== true) {
|
||||
return false;
|
||||
}
|
||||
const response = await confirm({ message, initialValue: true });
|
||||
if (isCancel(response)) {
|
||||
cancel('Runtime installation cancelled.');
|
||||
return false;
|
||||
}
|
||||
return response === true;
|
||||
const prompts = createClackPromptAdapter();
|
||||
return await prompts.confirm({ message, initialValue: true });
|
||||
}
|
||||
|
||||
export async function ensureManagedPythonCommandRuntime(
|
||||
|
|
@ -99,7 +95,7 @@ export async function ensureManagedPythonCommandRuntime(
|
|||
|
||||
if (options.installPolicy === 'prompt') {
|
||||
const confirmInstall = options.confirmInstall ?? defaultConfirmInstall;
|
||||
const confirmed = await confirmInstall(installPrompt(feature));
|
||||
const confirmed = await confirmInstall(installPrompt(feature), options.io);
|
||||
if (!confirmed) {
|
||||
throw new Error(`KTX Python runtime installation was cancelled. Run: ${managedRuntimeInstallCommand(feature)}`);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,9 +5,11 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
|||
import {
|
||||
readManagedPythonDaemonStatus,
|
||||
startManagedPythonDaemon,
|
||||
stopAllManagedPythonDaemons,
|
||||
stopManagedPythonDaemon,
|
||||
type ManagedPythonDaemonChild,
|
||||
type ManagedPythonDaemonFetch,
|
||||
type ManagedPythonDaemonProcessInfo,
|
||||
type ManagedPythonDaemonSpawn,
|
||||
type ManagedPythonDaemonState,
|
||||
} from './managed-python-daemon.js';
|
||||
|
|
@ -105,6 +107,24 @@ function runningState(root: string, overrides: Partial<ManagedPythonDaemonState>
|
|||
};
|
||||
}
|
||||
|
||||
function daemonStatePath(root: string, version: string): string {
|
||||
return join(root, 'runtime', version, 'daemon.json');
|
||||
}
|
||||
|
||||
function runningStateForVersion(
|
||||
root: string,
|
||||
version: string,
|
||||
overrides: Partial<ManagedPythonDaemonState> = {},
|
||||
): ManagedPythonDaemonState {
|
||||
return {
|
||||
...runningState(root),
|
||||
version,
|
||||
stdoutLog: join(root, 'runtime', version, 'daemon.stdout.log'),
|
||||
stderrLog: join(root, 'runtime', version, 'daemon.stderr.log'),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('managed Python daemon lifecycle', () => {
|
||||
let tempDir: string;
|
||||
|
||||
|
|
@ -170,6 +190,41 @@ describe('managed Python daemon lifecycle', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it('makes a final health probe before reporting startup failure', async () => {
|
||||
const spawnDaemon = makeSpawn(5556);
|
||||
const installRuntime = vi.fn(async () => installResult(tempDir));
|
||||
const fetch = vi
|
||||
.fn<ManagedPythonDaemonFetch>()
|
||||
.mockRejectedValueOnce(new Error('fetch failed'))
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
status: 200,
|
||||
json: async () => ({ status: 'healthy', version: '0.2.0' }),
|
||||
text: async () => '',
|
||||
});
|
||||
|
||||
const result = await startManagedPythonDaemon({
|
||||
cliVersion: '0.2.0',
|
||||
runtimeRoot: join(tempDir, 'runtime'),
|
||||
features: ['core'],
|
||||
installRuntime,
|
||||
spawnDaemon,
|
||||
fetch,
|
||||
allocatePort: vi.fn(async () => 61234),
|
||||
now: () => new Date('2026-05-11T00:00:00.000Z'),
|
||||
startupTimeoutMs: 5,
|
||||
pollIntervalMs: 20,
|
||||
});
|
||||
|
||||
expect(result.status).toBe('started');
|
||||
expect(fetch).toHaveBeenCalledTimes(2);
|
||||
expect(JSON.parse(await readFile(layout(tempDir).daemonStatePath, 'utf8'))).toMatchObject({
|
||||
pid: 5556,
|
||||
port: 61234,
|
||||
version: '0.2.0',
|
||||
});
|
||||
});
|
||||
|
||||
it('reuses a healthy daemon with the requested feature set', async () => {
|
||||
await mkdir(layout(tempDir).versionDir, { recursive: true });
|
||||
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
|
||||
|
|
@ -236,4 +291,138 @@ describe('managed Python daemon lifecycle', () => {
|
|||
expect(killProcess).toHaveBeenCalledWith(4242);
|
||||
await expect(readFile(layout(tempDir).daemonStatePath, 'utf8')).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('stops all recorded daemon states across runtime versions and removes state files', async () => {
|
||||
await mkdir(join(tempDir, 'runtime', '0.1.0'), { recursive: true });
|
||||
await mkdir(join(tempDir, 'runtime', '0.2.0'), { recursive: true });
|
||||
await writeFile(
|
||||
daemonStatePath(tempDir, '0.1.0'),
|
||||
`${JSON.stringify(runningStateForVersion(tempDir, '0.1.0', { pid: 1111, port: 61111 }), null, 2)}\n`,
|
||||
);
|
||||
await writeFile(
|
||||
daemonStatePath(tempDir, '0.2.0'),
|
||||
`${JSON.stringify(runningStateForVersion(tempDir, '0.2.0', { pid: 2222, port: 62222 }), null, 2)}\n`,
|
||||
);
|
||||
const alive = new Set([1111, 2222]);
|
||||
const killProcess = vi.fn((pid: number) => {
|
||||
alive.delete(pid);
|
||||
});
|
||||
|
||||
const result = await stopAllManagedPythonDaemons({
|
||||
cliVersion: '0.2.0',
|
||||
runtimeRoot: join(tempDir, 'runtime'),
|
||||
listProcesses: vi.fn(async () => []),
|
||||
processAlive: vi.fn((pid) => alive.has(pid)),
|
||||
killProcess,
|
||||
stopGraceMs: 0,
|
||||
});
|
||||
|
||||
expect(result.failed).toHaveLength(0);
|
||||
expect(result.stopped.map((entry) => entry.pid).sort()).toEqual([1111, 2222]);
|
||||
expect(killProcess).toHaveBeenCalledWith(1111, 'SIGTERM');
|
||||
expect(killProcess).toHaveBeenCalledWith(2222, 'SIGTERM');
|
||||
await expect(readFile(daemonStatePath(tempDir, '0.1.0'), 'utf8')).rejects.toThrow();
|
||||
await expect(readFile(daemonStatePath(tempDir, '0.2.0'), 'utf8')).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('removes stale state when the recorded daemon process is no longer alive', async () => {
|
||||
await mkdir(layout(tempDir).versionDir, { recursive: true });
|
||||
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
|
||||
|
||||
const result = await stopAllManagedPythonDaemons({
|
||||
cliVersion: '0.2.0',
|
||||
runtimeRoot: join(tempDir, 'runtime'),
|
||||
listProcesses: vi.fn(async () => []),
|
||||
processAlive: vi.fn(() => false),
|
||||
killProcess: vi.fn(),
|
||||
stopGraceMs: 0,
|
||||
});
|
||||
|
||||
expect(result.stopped).toHaveLength(0);
|
||||
expect(result.stale.map((entry) => entry.pid)).toEqual([4242]);
|
||||
await expect(readFile(layout(tempDir).daemonStatePath, 'utf8')).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('deduplicates a daemon found by state and process scan, preferring state metadata', async () => {
|
||||
await mkdir(layout(tempDir).versionDir, { recursive: true });
|
||||
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
|
||||
const alive = new Set([4242]);
|
||||
const killProcess = vi.fn((pid: number) => {
|
||||
alive.delete(pid);
|
||||
});
|
||||
|
||||
const result = await stopAllManagedPythonDaemons({
|
||||
cliVersion: '0.2.0',
|
||||
runtimeRoot: join(tempDir, 'runtime'),
|
||||
listProcesses: vi.fn(async (): Promise<ManagedPythonDaemonProcessInfo[]> => [
|
||||
{ pid: 4242, command: 'uv run ktx-daemon serve-http --host 127.0.0.1 --port 61234' },
|
||||
]),
|
||||
processAlive: vi.fn((pid) => alive.has(pid)),
|
||||
killProcess,
|
||||
stopGraceMs: 0,
|
||||
});
|
||||
|
||||
expect(result.stopped).toHaveLength(1);
|
||||
expect(result.stopped[0]).toMatchObject({
|
||||
pid: 4242,
|
||||
source: 'state',
|
||||
url: 'http://127.0.0.1:58731',
|
||||
});
|
||||
expect(killProcess).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('stops unrecorded ktx-daemon serve-http processes from process scan results', async () => {
|
||||
const alive = new Set([3333, 5555]);
|
||||
const killProcess = vi.fn((pid: number) => {
|
||||
alive.delete(pid);
|
||||
});
|
||||
|
||||
const result = await stopAllManagedPythonDaemons({
|
||||
cliVersion: '0.2.0',
|
||||
runtimeRoot: join(tempDir, 'runtime'),
|
||||
listProcesses: vi.fn(async (): Promise<ManagedPythonDaemonProcessInfo[]> => [
|
||||
{ pid: 3333, command: 'uv run ktx-daemon serve-http --host 127.0.0.1 --port 8765' },
|
||||
{ pid: 4444, command: 'node server.js --port 8765' },
|
||||
{ pid: 5555, command: 'grep ktx-daemon serve-http --port 8765' },
|
||||
]),
|
||||
processAlive: vi.fn((pid) => alive.has(pid)),
|
||||
killProcess,
|
||||
stopGraceMs: 0,
|
||||
});
|
||||
|
||||
expect(result.failed).toHaveLength(0);
|
||||
expect(result.stopped).toEqual([
|
||||
expect.objectContaining({
|
||||
pid: 3333,
|
||||
source: 'process',
|
||||
url: 'http://127.0.0.1:8765',
|
||||
}),
|
||||
]);
|
||||
expect(killProcess).toHaveBeenCalledWith(3333, 'SIGTERM');
|
||||
expect(killProcess).not.toHaveBeenCalledWith(4444, expect.anything());
|
||||
expect(killProcess).not.toHaveBeenCalledWith(5555, expect.anything());
|
||||
});
|
||||
|
||||
it('reports a failed stop when TERM and KILL leave a daemon running', async () => {
|
||||
await mkdir(layout(tempDir).versionDir, { recursive: true });
|
||||
await writeFile(layout(tempDir).daemonStatePath, `${JSON.stringify(runningState(tempDir), null, 2)}\n`);
|
||||
|
||||
const result = await stopAllManagedPythonDaemons({
|
||||
cliVersion: '0.2.0',
|
||||
runtimeRoot: join(tempDir, 'runtime'),
|
||||
listProcesses: vi.fn(async () => []),
|
||||
processAlive: vi.fn(() => true),
|
||||
killProcess: vi.fn(),
|
||||
stopGraceMs: 0,
|
||||
});
|
||||
|
||||
expect(result.stopped).toHaveLength(0);
|
||||
expect(result.failed).toEqual([
|
||||
expect.objectContaining({
|
||||
pid: 4242,
|
||||
detail: 'Process still running after SIGKILL',
|
||||
}),
|
||||
]);
|
||||
expect(await readFile(layout(tempDir).daemonStatePath, 'utf8')).toContain('"pid": 4242');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
import { spawn } from 'node:child_process';
|
||||
import { mkdir, open, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import { execFile, spawn } from 'node:child_process';
|
||||
import { mkdir, open, readdir, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import { createServer } from 'node:net';
|
||||
import { join } from 'node:path';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { promisify } from 'node:util';
|
||||
import { z } from 'zod';
|
||||
import {
|
||||
installManagedPythonRuntime,
|
||||
|
|
@ -44,6 +46,35 @@ export interface ManagedPythonDaemonStopResult {
|
|||
state?: ManagedPythonDaemonState;
|
||||
}
|
||||
|
||||
export interface ManagedPythonDaemonProcessInfo {
|
||||
pid: number;
|
||||
command: string;
|
||||
}
|
||||
|
||||
export type ManagedPythonDaemonStopAllSource = 'state' | 'process';
|
||||
|
||||
export interface ManagedPythonDaemonStopAllEntry {
|
||||
pid: number;
|
||||
source: ManagedPythonDaemonStopAllSource;
|
||||
url?: string;
|
||||
health?: 'healthy' | 'unreachable';
|
||||
version?: string;
|
||||
command?: string;
|
||||
statePaths: string[];
|
||||
}
|
||||
|
||||
export interface ManagedPythonDaemonStopAllFailure extends ManagedPythonDaemonStopAllEntry {
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface ManagedPythonDaemonStopAllResult {
|
||||
runtimeRoot: string;
|
||||
stopped: ManagedPythonDaemonStopAllEntry[];
|
||||
stale: ManagedPythonDaemonStopAllEntry[];
|
||||
failed: ManagedPythonDaemonStopAllFailure[];
|
||||
scanErrors: string[];
|
||||
}
|
||||
|
||||
export interface ManagedPythonDaemonChild {
|
||||
pid?: number;
|
||||
unref(): void;
|
||||
|
|
@ -68,6 +99,8 @@ export type ManagedPythonDaemonFetch = (
|
|||
text(): Promise<string>;
|
||||
}>;
|
||||
|
||||
export type ManagedPythonDaemonKillProcess = (pid: number, signal?: NodeJS.Signals) => void;
|
||||
|
||||
export interface ManagedPythonDaemonStartOptions extends ManagedPythonRuntimeLayoutOptions {
|
||||
features: KtxRuntimeFeature[];
|
||||
force?: boolean;
|
||||
|
|
@ -76,7 +109,7 @@ export interface ManagedPythonDaemonStartOptions extends ManagedPythonRuntimeLay
|
|||
fetch?: ManagedPythonDaemonFetch;
|
||||
allocatePort?: () => Promise<number>;
|
||||
processAlive?: (pid: number) => boolean;
|
||||
killProcess?: (pid: number) => void;
|
||||
killProcess?: ManagedPythonDaemonKillProcess;
|
||||
now?: () => Date;
|
||||
startupTimeoutMs?: number;
|
||||
pollIntervalMs?: number;
|
||||
|
|
@ -89,9 +122,20 @@ export interface ManagedPythonDaemonStatusOptions extends ManagedPythonRuntimeLa
|
|||
|
||||
export interface ManagedPythonDaemonStopOptions extends ManagedPythonRuntimeLayoutOptions {
|
||||
processAlive?: (pid: number) => boolean;
|
||||
killProcess?: (pid: number) => void;
|
||||
killProcess?: ManagedPythonDaemonKillProcess;
|
||||
}
|
||||
|
||||
export interface ManagedPythonDaemonStopAllOptions extends ManagedPythonRuntimeLayoutOptions {
|
||||
listProcesses?: () => Promise<ManagedPythonDaemonProcessInfo[]>;
|
||||
processAlive?: (pid: number) => boolean;
|
||||
killProcess?: ManagedPythonDaemonKillProcess;
|
||||
stopGraceMs?: number;
|
||||
pollIntervalMs?: number;
|
||||
healthProbeMs?: number;
|
||||
}
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
|
||||
const daemonStateSchema = z.object({
|
||||
schemaVersion: z.literal(1),
|
||||
pid: z.number().int().positive(),
|
||||
|
|
@ -126,9 +170,9 @@ function defaultProcessAlive(pid: number): boolean {
|
|||
}
|
||||
}
|
||||
|
||||
function defaultKillProcess(pid: number): void {
|
||||
function defaultKillProcess(pid: number, signal: NodeJS.Signals = 'SIGTERM'): void {
|
||||
try {
|
||||
process.kill(pid, 'SIGTERM');
|
||||
process.kill(pid, signal);
|
||||
} catch (error) {
|
||||
const code = (error as { code?: unknown }).code;
|
||||
if (code !== 'ESRCH') {
|
||||
|
|
@ -273,6 +317,15 @@ async function waitForHealth(input: {
|
|||
lastDetail = health.detail;
|
||||
await delay(input.pollIntervalMs);
|
||||
}
|
||||
const finalHealth = await healthOk({
|
||||
state: input.state,
|
||||
cliVersion: input.cliVersion,
|
||||
fetch: input.fetch,
|
||||
});
|
||||
if (finalHealth.ok) {
|
||||
return;
|
||||
}
|
||||
lastDetail = finalHealth.detail;
|
||||
throw new Error(`KTX Python daemon failed to start: ${lastDetail}. stderr: ${input.state.stderrLog}`);
|
||||
}
|
||||
|
||||
|
|
@ -284,7 +337,7 @@ async function stopRecordedDaemon(input: {
|
|||
layout: ManagedPythonRuntimeLayout;
|
||||
state: ManagedPythonDaemonState;
|
||||
processAlive: (pid: number) => boolean;
|
||||
killProcess: (pid: number) => void;
|
||||
killProcess: ManagedPythonDaemonKillProcess;
|
||||
}): Promise<void> {
|
||||
if (input.processAlive(input.state.pid)) {
|
||||
input.killProcess(input.state.pid);
|
||||
|
|
@ -292,6 +345,323 @@ async function stopRecordedDaemon(input: {
|
|||
await removeState(input.layout);
|
||||
}
|
||||
|
||||
function runtimeRootForStopAll(options: ManagedPythonRuntimeLayoutOptions): string {
|
||||
return managedPythonRuntimeLayout(options).runtimeRoot;
|
||||
}
|
||||
|
||||
async function removeStatePaths(paths: string[]): Promise<void> {
|
||||
await Promise.all([...new Set(paths)].map((path) => rm(path, { force: true })));
|
||||
}
|
||||
|
||||
interface ManagedPythonDaemonStopCandidate {
|
||||
pid: number;
|
||||
source: ManagedPythonDaemonStopAllSource;
|
||||
host?: string;
|
||||
port?: number;
|
||||
version?: string;
|
||||
command?: string;
|
||||
statePaths: string[];
|
||||
}
|
||||
|
||||
function candidateUrl(candidate: ManagedPythonDaemonStopCandidate): string | undefined {
|
||||
if (!candidate.host || !candidate.port) {
|
||||
return undefined;
|
||||
}
|
||||
return `http://${candidate.host}:${candidate.port}`;
|
||||
}
|
||||
|
||||
function candidateEntry(candidate: ManagedPythonDaemonStopCandidate): ManagedPythonDaemonStopAllEntry {
|
||||
return {
|
||||
pid: candidate.pid,
|
||||
source: candidate.source,
|
||||
...(candidateUrl(candidate) ? { url: candidateUrl(candidate) } : {}),
|
||||
...(candidate.version ? { version: candidate.version } : {}),
|
||||
...(candidate.command ? { command: candidate.command } : {}),
|
||||
statePaths: [...candidate.statePaths],
|
||||
};
|
||||
}
|
||||
|
||||
async function probeCandidateHealth(
|
||||
candidate: ManagedPythonDaemonStopCandidate,
|
||||
timeoutMs: number,
|
||||
): Promise<'healthy' | 'unreachable' | undefined> {
|
||||
const url = candidateUrl(candidate);
|
||||
if (!url) {
|
||||
return undefined;
|
||||
}
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => {
|
||||
controller.abort();
|
||||
}, timeoutMs);
|
||||
try {
|
||||
const response = await fetch(`${url}/health`, { signal: controller.signal });
|
||||
if (!response.ok) {
|
||||
return 'unreachable';
|
||||
}
|
||||
const body = (await response.json()) as unknown;
|
||||
if (!body || typeof body !== 'object' || Array.isArray(body)) {
|
||||
return 'unreachable';
|
||||
}
|
||||
return (body as Record<string, unknown>).status === 'healthy' ? 'healthy' : 'unreachable';
|
||||
} catch {
|
||||
return 'unreachable';
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
async function readStateCandidates(runtimeRoot: string): Promise<ManagedPythonDaemonStopCandidate[]> {
|
||||
let entries;
|
||||
try {
|
||||
entries = await readdir(runtimeRoot, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
const code = (error as { code?: unknown }).code;
|
||||
if (code === 'ENOENT') {
|
||||
return [];
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const candidates: ManagedPythonDaemonStopCandidate[] = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory()) {
|
||||
continue;
|
||||
}
|
||||
const statePath = join(runtimeRoot, entry.name, 'daemon.json');
|
||||
let state: ManagedPythonDaemonState | undefined;
|
||||
try {
|
||||
state = await readState(statePath);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (!state) {
|
||||
continue;
|
||||
}
|
||||
candidates.push({
|
||||
pid: state.pid,
|
||||
source: 'state',
|
||||
host: state.host,
|
||||
port: state.port,
|
||||
version: state.version,
|
||||
statePaths: [statePath],
|
||||
});
|
||||
}
|
||||
return candidates;
|
||||
}
|
||||
|
||||
function tokenizeCommand(command: string): string[] {
|
||||
const tokens: string[] = [];
|
||||
for (const match of command.matchAll(/"([^"]*)"|'([^']*)'|(\S+)/g)) {
|
||||
tokens.push(match[1] ?? match[2] ?? match[3] ?? '');
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
|
||||
function executableName(token: string): string {
|
||||
return token.split(/[\\/]/).at(-1) ?? token;
|
||||
}
|
||||
|
||||
function isKtxDaemonExecutable(token: string): boolean {
|
||||
return executableName(token) === 'ktx-daemon' || executableName(token) === 'ktx-daemon.exe';
|
||||
}
|
||||
|
||||
function normalizedExecutableName(token: string): string {
|
||||
return executableName(token).replace(/\.exe$/i, '').toLowerCase();
|
||||
}
|
||||
|
||||
function hasUvRunPrefix(tokens: string[], daemonIndex: number): boolean {
|
||||
return normalizedExecutableName(tokens[0] ?? '') === 'uv' && tokens.slice(1, daemonIndex).includes('run');
|
||||
}
|
||||
|
||||
function isPythonExecutable(token: string): boolean {
|
||||
const name = normalizedExecutableName(token);
|
||||
return name === 'python' || name === 'python3';
|
||||
}
|
||||
|
||||
function hasPythonModulePrefix(tokens: string[], moduleFlagIndex: number): boolean {
|
||||
if (moduleFlagIndex === 1 && isPythonExecutable(tokens[0] ?? '')) {
|
||||
return true;
|
||||
}
|
||||
return (
|
||||
normalizedExecutableName(tokens[0] ?? '') === 'uv' &&
|
||||
tokens.slice(1, moduleFlagIndex).includes('run') &&
|
||||
tokens.some((token, index) => index < moduleFlagIndex && isPythonExecutable(token))
|
||||
);
|
||||
}
|
||||
|
||||
function isKtxDaemonServeHttp(tokens: string[]): boolean {
|
||||
for (let index = 0; index < tokens.length; index += 1) {
|
||||
if (
|
||||
isKtxDaemonExecutable(tokens[index] ?? '') &&
|
||||
tokens[index + 1] === 'serve-http' &&
|
||||
(index === 0 || hasUvRunPrefix(tokens, index))
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
tokens[index] === '-m' &&
|
||||
tokens[index + 1] === 'ktx_daemon' &&
|
||||
tokens[index + 2] === 'serve-http' &&
|
||||
hasPythonModulePrefix(tokens, index)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function parseCommandOption(tokens: string[], option: string): string | undefined {
|
||||
for (let index = 0; index < tokens.length; index += 1) {
|
||||
const token = tokens[index];
|
||||
if (token === option) {
|
||||
return tokens[index + 1];
|
||||
}
|
||||
if (token?.startsWith(`${option}=`)) {
|
||||
return token.slice(option.length + 1);
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function processCandidate(processInfo: ManagedPythonDaemonProcessInfo): ManagedPythonDaemonStopCandidate | undefined {
|
||||
const tokens = tokenizeCommand(processInfo.command);
|
||||
if (!isKtxDaemonServeHttp(tokens)) {
|
||||
return undefined;
|
||||
}
|
||||
const host = parseCommandOption(tokens, '--host') ?? '127.0.0.1';
|
||||
const rawPort = parseCommandOption(tokens, '--port');
|
||||
const parsedPort = rawPort ? Number.parseInt(rawPort, 10) : 8765;
|
||||
const port = Number.isInteger(parsedPort) && parsedPort >= 1 && parsedPort <= 65535 ? parsedPort : 8765;
|
||||
return {
|
||||
pid: processInfo.pid,
|
||||
source: 'process',
|
||||
host,
|
||||
port,
|
||||
command: processInfo.command,
|
||||
statePaths: [],
|
||||
};
|
||||
}
|
||||
|
||||
function mergeCandidates(candidates: ManagedPythonDaemonStopCandidate[]): ManagedPythonDaemonStopCandidate[] {
|
||||
const byPid = new Map<number, ManagedPythonDaemonStopCandidate>();
|
||||
for (const candidate of candidates) {
|
||||
const existing = byPid.get(candidate.pid);
|
||||
if (!existing) {
|
||||
byPid.set(candidate.pid, { ...candidate, statePaths: [...candidate.statePaths] });
|
||||
continue;
|
||||
}
|
||||
existing.statePaths.push(...candidate.statePaths);
|
||||
if (existing.source === 'process' && candidate.source === 'state') {
|
||||
byPid.set(candidate.pid, {
|
||||
...candidate,
|
||||
statePaths: [...new Set([...existing.statePaths, ...candidate.statePaths])],
|
||||
});
|
||||
} else {
|
||||
existing.statePaths = [...new Set(existing.statePaths)];
|
||||
}
|
||||
}
|
||||
return [...byPid.values()].sort((left, right) => left.pid - right.pid);
|
||||
}
|
||||
|
||||
function parsePosixProcessList(output: string): ManagedPythonDaemonProcessInfo[] {
|
||||
const processes: ManagedPythonDaemonProcessInfo[] = [];
|
||||
for (const line of output.split(/\r?\n/)) {
|
||||
const match = line.match(/^\s*(\d+)\s+(.+)$/);
|
||||
if (!match) {
|
||||
continue;
|
||||
}
|
||||
processes.push({ pid: Number.parseInt(match[1], 10), command: match[2] });
|
||||
}
|
||||
return processes;
|
||||
}
|
||||
|
||||
function parseWindowsProcessList(output: string): ManagedPythonDaemonProcessInfo[] {
|
||||
if (!output.trim()) {
|
||||
return [];
|
||||
}
|
||||
const parsed = JSON.parse(output) as unknown;
|
||||
const records = Array.isArray(parsed) ? parsed : [parsed];
|
||||
const processes: ManagedPythonDaemonProcessInfo[] = [];
|
||||
for (const record of records) {
|
||||
if (!record || typeof record !== 'object') {
|
||||
continue;
|
||||
}
|
||||
const value = record as Record<string, unknown>;
|
||||
const pid = value.ProcessId;
|
||||
const command = value.CommandLine;
|
||||
if (typeof pid === 'number' && typeof command === 'string' && command.length > 0) {
|
||||
processes.push({ pid, command });
|
||||
}
|
||||
}
|
||||
return processes;
|
||||
}
|
||||
|
||||
async function defaultListProcesses(platform: NodeJS.Platform = process.platform): Promise<ManagedPythonDaemonProcessInfo[]> {
|
||||
if (platform === 'win32') {
|
||||
const command = [
|
||||
'Get-CimInstance Win32_Process',
|
||||
'| Where-Object { $_.CommandLine -ne $null }',
|
||||
'| Select-Object ProcessId,CommandLine',
|
||||
'| ConvertTo-Json -Compress',
|
||||
].join(' ');
|
||||
const { stdout } = await execFileAsync('powershell.exe', ['-NoProfile', '-Command', command], {
|
||||
encoding: 'utf8',
|
||||
maxBuffer: 10 * 1024 * 1024,
|
||||
});
|
||||
return parseWindowsProcessList(stdout);
|
||||
}
|
||||
const { stdout } = await execFileAsync('ps', ['-axo', 'pid=,command='], {
|
||||
encoding: 'utf8',
|
||||
maxBuffer: 10 * 1024 * 1024,
|
||||
});
|
||||
return parsePosixProcessList(stdout);
|
||||
}
|
||||
|
||||
async function waitUntilStopped(input: {
|
||||
pid: number;
|
||||
processAlive: (pid: number) => boolean;
|
||||
timeoutMs: number;
|
||||
pollIntervalMs: number;
|
||||
}): Promise<boolean> {
|
||||
const deadline = Date.now() + input.timeoutMs;
|
||||
do {
|
||||
if (!input.processAlive(input.pid)) {
|
||||
return true;
|
||||
}
|
||||
if (Date.now() >= deadline) {
|
||||
break;
|
||||
}
|
||||
await delay(input.pollIntervalMs);
|
||||
} while (Date.now() <= deadline);
|
||||
return !input.processAlive(input.pid);
|
||||
}
|
||||
|
||||
async function discoverStopAllCandidates(
|
||||
options: ManagedPythonDaemonStopAllOptions,
|
||||
): Promise<{
|
||||
runtimeRoot: string;
|
||||
candidates: ManagedPythonDaemonStopCandidate[];
|
||||
scanErrors: string[];
|
||||
}> {
|
||||
const runtimeRoot = runtimeRootForStopAll(options);
|
||||
const stateCandidates = await readStateCandidates(runtimeRoot);
|
||||
const scanErrors: string[] = [];
|
||||
let processCandidates: ManagedPythonDaemonStopCandidate[] = [];
|
||||
try {
|
||||
const processes = await (options.listProcesses ?? defaultListProcesses)();
|
||||
processCandidates = processes.flatMap((processInfo) => {
|
||||
const candidate = processCandidate(processInfo);
|
||||
return candidate ? [candidate] : [];
|
||||
});
|
||||
} catch (error) {
|
||||
scanErrors.push(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
return {
|
||||
runtimeRoot,
|
||||
candidates: mergeCandidates([...stateCandidates, ...processCandidates]),
|
||||
scanErrors,
|
||||
};
|
||||
}
|
||||
|
||||
export async function startManagedPythonDaemon(
|
||||
options: ManagedPythonDaemonStartOptions,
|
||||
): Promise<ManagedPythonDaemonStartResult> {
|
||||
|
|
@ -395,3 +765,63 @@ export async function stopManagedPythonDaemon(
|
|||
});
|
||||
return { status: 'stopped', layout, state };
|
||||
}
|
||||
|
||||
export async function stopAllManagedPythonDaemons(
|
||||
options: ManagedPythonDaemonStopAllOptions,
|
||||
): Promise<ManagedPythonDaemonStopAllResult> {
|
||||
const processAlive = options.processAlive ?? defaultProcessAlive;
|
||||
const killProcess = options.killProcess ?? defaultKillProcess;
|
||||
const stopGraceMs = options.stopGraceMs ?? 500;
|
||||
const pollIntervalMs = options.pollIntervalMs ?? 50;
|
||||
const healthProbeMs = options.healthProbeMs ?? 100;
|
||||
const discovery = await discoverStopAllCandidates(options);
|
||||
const stopped: ManagedPythonDaemonStopAllEntry[] = [];
|
||||
const stale: ManagedPythonDaemonStopAllEntry[] = [];
|
||||
const failed: ManagedPythonDaemonStopAllFailure[] = [];
|
||||
|
||||
for (const candidate of discovery.candidates) {
|
||||
const health = await probeCandidateHealth(candidate, healthProbeMs);
|
||||
const entry = { ...candidateEntry(candidate), ...(health ? { health } : {}) };
|
||||
if (!processAlive(candidate.pid)) {
|
||||
await removeStatePaths(candidate.statePaths);
|
||||
stale.push(entry);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
killProcess(candidate.pid, 'SIGTERM');
|
||||
if (
|
||||
!(await waitUntilStopped({
|
||||
pid: candidate.pid,
|
||||
processAlive,
|
||||
timeoutMs: stopGraceMs,
|
||||
pollIntervalMs,
|
||||
}))
|
||||
) {
|
||||
killProcess(candidate.pid, 'SIGKILL');
|
||||
if (
|
||||
!(await waitUntilStopped({
|
||||
pid: candidate.pid,
|
||||
processAlive,
|
||||
timeoutMs: stopGraceMs,
|
||||
pollIntervalMs,
|
||||
}))
|
||||
) {
|
||||
failed.push({ ...entry, detail: 'Process still running after SIGKILL' });
|
||||
continue;
|
||||
}
|
||||
}
|
||||
await removeStatePaths(candidate.statePaths);
|
||||
stopped.push(entry);
|
||||
} catch (error) {
|
||||
failed.push({ ...entry, detail: error instanceof Error ? error.message : String(error) });
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
runtimeRoot: discovery.runtimeRoot,
|
||||
stopped,
|
||||
stale,
|
||||
failed,
|
||||
scanErrors: discovery.scanErrors,
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -161,6 +161,14 @@ describe('verifyRuntimeAsset', () => {
|
|||
|
||||
await expect(verifyRuntimeAsset({ assetDir })).rejects.toThrow(/Unsafe runtime wheel filename/);
|
||||
});
|
||||
|
||||
it('reports the source-checkout artifact command when the bundled manifest is missing', async () => {
|
||||
const assetDir = join(tempDir, 'packages', 'cli', 'assets', 'python');
|
||||
|
||||
await expect(verifyRuntimeAsset({ assetDir })).rejects.toThrow(
|
||||
/Missing bundled Python runtime manifest.*pnpm run artifacts:build/s,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('installManagedPythonRuntime', () => {
|
||||
|
|
@ -210,6 +218,30 @@ describe('installManagedPythonRuntime', () => {
|
|||
expect(manifest.python.daemonExecutable).toBe(result.layout.daemonPath);
|
||||
});
|
||||
|
||||
it('disables repo uv config for managed runtime uv commands', async () => {
|
||||
const { assetDir } = await writeAsset(tempDir, 'core-wheel');
|
||||
const commands: Array<{ command: string; args: string[]; env?: NodeJS.ProcessEnv }> = [];
|
||||
const exec: ManagedPythonRuntimeExec = vi.fn(async (command, args, options) => {
|
||||
commands.push({ command, args, env: options?.env });
|
||||
return { stdout: command === 'uv' && args[0] === '--version' ? 'uv 0.11.13\n' : '', stderr: '' };
|
||||
});
|
||||
|
||||
await installManagedPythonRuntime({
|
||||
cliVersion: '0.2.0',
|
||||
runtimeRoot: join(tempDir, 'runtime'),
|
||||
assetDir,
|
||||
env: { PATH: '/opt/homebrew/bin', UV_NO_CONFIG: '0' },
|
||||
features: ['core'],
|
||||
exec,
|
||||
});
|
||||
|
||||
expect(commands.map((call) => [call.command, call.args[0], call.env?.UV_NO_CONFIG, call.env?.PATH])).toEqual([
|
||||
['uv', '--version', '1', '/opt/homebrew/bin'],
|
||||
['uv', 'venv', '1', '/opt/homebrew/bin'],
|
||||
['uv', 'pip', '1', '/opt/homebrew/bin'],
|
||||
]);
|
||||
});
|
||||
|
||||
it('installs the local-embeddings extra when requested', async () => {
|
||||
const { assetDir } = await writeAsset(tempDir, 'embedding-wheel');
|
||||
const commands: Array<{ command: string; args: string[] }> = [];
|
||||
|
|
|
|||
|
|
@ -186,9 +186,28 @@ async function readJsonFile(path: string): Promise<unknown> {
|
|||
return JSON.parse(await readFile(path, 'utf8')) as unknown;
|
||||
}
|
||||
|
||||
function isErrnoException(error: unknown, code: string): boolean {
|
||||
return typeof error === 'object' && error !== null && 'code' in error && error.code === code;
|
||||
}
|
||||
|
||||
export async function verifyRuntimeAsset(input: { assetDir: string }): Promise<ManagedRuntimeAsset> {
|
||||
const manifestPath = join(input.assetDir, 'manifest.json');
|
||||
const manifest = runtimeAssetManifestSchema.parse(await readJsonFile(manifestPath));
|
||||
let manifestData: unknown;
|
||||
try {
|
||||
manifestData = await readJsonFile(manifestPath);
|
||||
} catch (error) {
|
||||
if (isErrnoException(error, 'ENOENT')) {
|
||||
throw new Error(
|
||||
[
|
||||
`Missing bundled Python runtime manifest: ${manifestPath}`,
|
||||
'In a source checkout, build the local runtime assets with: pnpm run artifacts:build',
|
||||
'Then retry the runtime-backed KTX command.',
|
||||
].join('\n'),
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const manifest = runtimeAssetManifestSchema.parse(manifestData);
|
||||
assertSafeWheelFilename(manifest.wheel.file);
|
||||
const wheelPath = join(input.assetDir, manifest.wheel.file);
|
||||
const wheel = await readFile(wheelPath);
|
||||
|
|
@ -243,10 +262,11 @@ async function runLogged(input: {
|
|||
command: string;
|
||||
args: string[];
|
||||
cwd?: string;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
}): Promise<{ stdout: string; stderr: string }> {
|
||||
await appendFile(input.logPath, `$ ${input.command} ${input.args.join(' ')}\n`);
|
||||
try {
|
||||
const result = await input.exec(input.command, input.args, { cwd: input.cwd });
|
||||
const result = await input.exec(input.command, input.args, { cwd: input.cwd, env: input.env });
|
||||
if (result.stdout) {
|
||||
await appendFile(input.logPath, result.stdout.endsWith('\n') ? result.stdout : `${result.stdout}\n`);
|
||||
}
|
||||
|
|
@ -266,9 +286,13 @@ async function runLogged(input: {
|
|||
}
|
||||
}
|
||||
|
||||
async function ensureUv(exec: ManagedPythonRuntimeExec): Promise<string> {
|
||||
function managedRuntimeUvEnv(baseEnv: NodeJS.ProcessEnv): NodeJS.ProcessEnv {
|
||||
return { ...baseEnv, UV_NO_CONFIG: '1' };
|
||||
}
|
||||
|
||||
async function ensureUv(exec: ManagedPythonRuntimeExec, env?: NodeJS.ProcessEnv): Promise<string> {
|
||||
try {
|
||||
const result = await exec('uv', ['--version']);
|
||||
const result = await exec('uv', ['--version'], { env });
|
||||
return result.stdout.trim() || 'uv available';
|
||||
} catch {
|
||||
throw new Error(MISSING_UV_RUNTIME_INSTALL_MESSAGE);
|
||||
|
|
@ -282,6 +306,7 @@ export async function installManagedPythonRuntime(
|
|||
const exec = options.exec ?? defaultExec;
|
||||
const features = normalizeFeatures(options.features);
|
||||
const asset = await verifyRuntimeAsset({ assetDir: layout.assetDir });
|
||||
const uvEnv = managedRuntimeUvEnv(options.env ?? process.env);
|
||||
const existing = await readInstalledManifest(layout.manifestPath);
|
||||
if (
|
||||
options.force !== true &&
|
||||
|
|
@ -298,14 +323,21 @@ export async function installManagedPythonRuntime(
|
|||
await rm(layout.versionDir, { recursive: true, force: true });
|
||||
await mkdir(layout.versionDir, { recursive: true });
|
||||
await writeFile(layout.installLogPath, '');
|
||||
await ensureUv(exec);
|
||||
await runLogged({ exec, logPath: layout.installLogPath, command: 'uv', args: ['venv', layout.venvDir] });
|
||||
await ensureUv(exec, uvEnv);
|
||||
await runLogged({
|
||||
exec,
|
||||
logPath: layout.installLogPath,
|
||||
command: 'uv',
|
||||
args: ['venv', layout.venvDir],
|
||||
env: uvEnv,
|
||||
});
|
||||
const wheelSpec = features.includes('local-embeddings') ? `${asset.wheelPath}[local-embeddings]` : asset.wheelPath;
|
||||
await runLogged({
|
||||
exec,
|
||||
logPath: layout.installLogPath,
|
||||
command: 'uv',
|
||||
args: ['pip', 'install', '--python', layout.pythonPath, wheelSpec],
|
||||
env: uvEnv,
|
||||
});
|
||||
|
||||
const manifest: InstalledKtxRuntimeManifest = {
|
||||
|
|
@ -371,7 +403,7 @@ export async function doctorManagedPythonRuntime(
|
|||
const exec = options.exec ?? defaultExec;
|
||||
const checks: ManagedPythonRuntimeDoctorCheck[] = [];
|
||||
try {
|
||||
const version = await ensureUv(exec);
|
||||
const version = await ensureUv(exec, managedRuntimeUvEnv(options.env ?? process.env));
|
||||
checks.push(check('pass', { id: 'uv', label: 'uv', detail: version }));
|
||||
} catch (error) {
|
||||
checks.push(
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import type {
|
||||
ManagedPythonDaemonStopAllResult,
|
||||
ManagedPythonDaemonStartResult,
|
||||
ManagedPythonDaemonStopResult,
|
||||
} from './managed-python-daemon.js';
|
||||
|
|
@ -199,13 +200,63 @@ describe('runKtxRuntime', () => {
|
|||
})),
|
||||
};
|
||||
|
||||
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0' }, io.io, deps)).resolves.toBe(0);
|
||||
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0', all: false }, io.io, deps)).resolves.toBe(0);
|
||||
|
||||
expect(deps.stopDaemon).toHaveBeenCalledWith({ cliVersion: '0.2.0' });
|
||||
expect(io.stdout()).toContain('Stopped KTX Python daemon');
|
||||
expect(io.stdout()).toContain('pid: 4242');
|
||||
});
|
||||
|
||||
it('stops all discovered Python daemons and reports the summary', async () => {
|
||||
const io = makeIo();
|
||||
const deps: KtxRuntimeDeps = {
|
||||
stopAllDaemons: vi.fn(async (): Promise<ManagedPythonDaemonStopAllResult> => ({
|
||||
runtimeRoot: '/runtime',
|
||||
stopped: [
|
||||
{ pid: 4242, source: 'state', url: 'http://127.0.0.1:61234', statePaths: ['/runtime/0.2.0/daemon.json'] },
|
||||
{ pid: 5252, source: 'process', url: 'http://127.0.0.1:8765', statePaths: [] },
|
||||
],
|
||||
stale: [],
|
||||
failed: [],
|
||||
scanErrors: [],
|
||||
})),
|
||||
};
|
||||
|
||||
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0', all: true }, io.io, deps)).resolves.toBe(0);
|
||||
|
||||
expect(deps.stopAllDaemons).toHaveBeenCalledWith({ cliVersion: '0.2.0' });
|
||||
expect(io.stdout()).toContain('Stopped 2 KTX Python daemons');
|
||||
expect(io.stdout()).toContain('pid: 4242 source: state url: http://127.0.0.1:61234');
|
||||
expect(io.stdout()).toContain('pid: 5252 source: process url: http://127.0.0.1:8765');
|
||||
});
|
||||
|
||||
it('returns failure when stop all cannot stop every daemon', async () => {
|
||||
const io = makeIo();
|
||||
const deps: KtxRuntimeDeps = {
|
||||
stopAllDaemons: vi.fn(async (): Promise<ManagedPythonDaemonStopAllResult> => ({
|
||||
runtimeRoot: '/runtime',
|
||||
stopped: [],
|
||||
stale: [],
|
||||
failed: [
|
||||
{
|
||||
pid: 4242,
|
||||
source: 'state',
|
||||
url: 'http://127.0.0.1:61234',
|
||||
statePaths: ['/runtime/0.2.0/daemon.json'],
|
||||
detail: 'Process still running after SIGKILL',
|
||||
},
|
||||
],
|
||||
scanErrors: ['ps failed'],
|
||||
})),
|
||||
};
|
||||
|
||||
await expect(runKtxRuntime({ command: 'stop', cliVersion: '0.2.0', all: true }, io.io, deps)).resolves.toBe(1);
|
||||
|
||||
expect(io.stderr()).toContain('Stopped 0 KTX Python daemons; failed 1');
|
||||
expect(io.stderr()).toContain('pid: 4242 source: state url: http://127.0.0.1:61234');
|
||||
expect(io.stderr()).toContain('process scan: ps failed');
|
||||
});
|
||||
|
||||
it('prints runtime status as JSON', async () => {
|
||||
const io = makeIo();
|
||||
const deps: KtxRuntimeDeps = {
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import {
|
||||
stopAllManagedPythonDaemons,
|
||||
startManagedPythonDaemon,
|
||||
stopManagedPythonDaemon,
|
||||
type ManagedPythonDaemonStopAllResult,
|
||||
type ManagedPythonDaemonStartResult,
|
||||
type ManagedPythonDaemonStopResult,
|
||||
} from './managed-python-daemon.js';
|
||||
|
|
@ -22,7 +24,7 @@ import {
|
|||
export type KtxRuntimeArgs =
|
||||
| { command: 'install'; cliVersion: string; feature: KtxRuntimeFeature; force: boolean }
|
||||
| { command: 'start'; cliVersion: string; feature: KtxRuntimeFeature; force: boolean }
|
||||
| { command: 'stop'; cliVersion: string }
|
||||
| { command: 'stop'; cliVersion: string; all: boolean }
|
||||
| { command: 'status'; cliVersion: string; json: boolean }
|
||||
| { command: 'doctor'; cliVersion: string; json: boolean }
|
||||
| { command: 'prune'; cliVersion: string; dryRun: boolean; yes: boolean };
|
||||
|
|
@ -35,6 +37,7 @@ export interface KtxRuntimeDeps {
|
|||
force?: boolean;
|
||||
}) => Promise<ManagedPythonDaemonStartResult>;
|
||||
stopDaemon?: (options: { cliVersion: string }) => Promise<ManagedPythonDaemonStopResult>;
|
||||
stopAllDaemons?: (options: { cliVersion: string }) => Promise<ManagedPythonDaemonStopAllResult>;
|
||||
readStatus?: (options: ManagedPythonRuntimeLayoutOptions) => Promise<ManagedPythonRuntimeStatus>;
|
||||
doctorRuntime?: (options: ManagedPythonRuntimeLayoutOptions) => Promise<ManagedPythonRuntimeDoctorCheck[]>;
|
||||
pruneRuntime?: (options: {
|
||||
|
|
@ -81,6 +84,58 @@ function writeDaemonStop(io: KtxCliIo, result: ManagedPythonDaemonStopResult): v
|
|||
io.stdout.write(`state: ${result.layout.daemonStatePath}\n`);
|
||||
}
|
||||
|
||||
function writeStopAllEntry(io: KtxCliIo, entry: { pid: number; source: string; url?: string; health?: string; detail?: string }): void {
|
||||
io.stdout.write(
|
||||
`pid: ${entry.pid} source: ${entry.source}${entry.url ? ` url: ${entry.url}` : ''}${
|
||||
entry.health ? ` health: ${entry.health}` : ''
|
||||
}${
|
||||
entry.detail ? ` detail: ${entry.detail}` : ''
|
||||
}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
function writeDaemonStopAll(io: KtxCliIo, result: ManagedPythonDaemonStopAllResult): number {
|
||||
const failed = result.failed.length + result.scanErrors.length;
|
||||
if (
|
||||
result.stopped.length === 0 &&
|
||||
result.stale.length === 0 &&
|
||||
result.failed.length === 0 &&
|
||||
result.scanErrors.length === 0
|
||||
) {
|
||||
io.stdout.write('No KTX Python daemons found\n');
|
||||
return 0;
|
||||
}
|
||||
if (failed === 0) {
|
||||
io.stdout.write(`Stopped ${result.stopped.length} KTX Python daemons\n`);
|
||||
if (result.stale.length > 0) {
|
||||
io.stdout.write(`Cleaned ${result.stale.length} stale daemon states\n`);
|
||||
}
|
||||
for (const entry of result.stopped) {
|
||||
writeStopAllEntry(io, entry);
|
||||
}
|
||||
for (const entry of result.stale) {
|
||||
writeStopAllEntry(io, entry);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
io.stderr.write(
|
||||
`Stopped ${result.stopped.length} KTX Python daemons; failed ${result.failed.length}${
|
||||
result.stale.length > 0 ? `; cleaned stale ${result.stale.length}` : ''
|
||||
}\n`,
|
||||
);
|
||||
for (const entry of result.failed) {
|
||||
io.stderr.write(
|
||||
`pid: ${entry.pid} source: ${entry.source}${entry.url ? ` url: ${entry.url}` : ''}${
|
||||
entry.health ? ` health: ${entry.health}` : ''
|
||||
} detail: ${entry.detail}\n`,
|
||||
);
|
||||
}
|
||||
for (const error of result.scanErrors) {
|
||||
io.stderr.write(`process scan: ${error}\n`);
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
function writeStatus(io: KtxCliIo, status: ManagedPythonRuntimeStatus): void {
|
||||
io.stdout.write('KTX Python runtime\n');
|
||||
io.stdout.write(`status: ${status.kind}\n`);
|
||||
|
|
@ -142,10 +197,16 @@ export async function runKtxRuntime(
|
|||
return 0;
|
||||
}
|
||||
if (args.command === 'stop') {
|
||||
const stopDaemon = deps.stopDaemon ?? stopManagedPythonDaemon;
|
||||
const result = await stopDaemon({ cliVersion: args.cliVersion });
|
||||
writeDaemonStop(io, result);
|
||||
return 0;
|
||||
if (args.all) {
|
||||
const stopAllDaemons = deps.stopAllDaemons ?? stopAllManagedPythonDaemons;
|
||||
const result = await stopAllDaemons({ cliVersion: args.cliVersion });
|
||||
return writeDaemonStopAll(io, result);
|
||||
} else {
|
||||
const stopDaemon = deps.stopDaemon ?? stopManagedPythonDaemon;
|
||||
const result = await stopDaemon({ cliVersion: args.cliVersion });
|
||||
writeDaemonStop(io, result);
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
if (args.command === 'status') {
|
||||
const readStatus = deps.readStatus ?? readManagedPythonRuntimeStatus;
|
||||
|
|
|
|||
|
|
@ -573,6 +573,32 @@ describe('runKtxScan', () => {
|
|||
expect(io.stdout()).toContain('\n[90%] Building embeddings 1/4 batches\n');
|
||||
});
|
||||
|
||||
it('scales nested progress phases by the parent phase weight', async () => {
|
||||
const io = makeIo({ isTTY: true });
|
||||
const previousCi = process.env.CI;
|
||||
delete process.env.CI;
|
||||
|
||||
try {
|
||||
const progress = createCliScanProgress(io.io);
|
||||
await progress.update(0.82, 'Enriching schema metadata');
|
||||
const enrichmentProgress = progress.startPhase(0.18);
|
||||
await enrichmentProgress.update(0.05, 'Loaded schema snapshot with 56 tables');
|
||||
const descriptionProgress = enrichmentProgress.startPhase(0.45);
|
||||
await descriptionProgress.update(37 / 56, 'Generating descriptions 37/56 tables', { transient: true });
|
||||
await descriptionProgress.update(1, 'Generated descriptions for 56 tables');
|
||||
} finally {
|
||||
if (previousCi === undefined) {
|
||||
delete process.env.CI;
|
||||
} else {
|
||||
process.env.CI = previousCi;
|
||||
}
|
||||
}
|
||||
|
||||
expect(io.stdout()).toContain('\r[88%] Generating descriptions 37/56 tables');
|
||||
expect(io.stdout()).toContain('\n[91%] Generated descriptions for 56 tables\n');
|
||||
expect(io.stdout()).not.toContain('[100%] Generating descriptions 37/56 tables');
|
||||
});
|
||||
|
||||
it('flushes transient TTY progress messages before printing scan failures', async () => {
|
||||
await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' });
|
||||
const runLocalScan = vi.fn(async (input: RunLocalScanOptions): Promise<LocalScanRunResult> => {
|
||||
|
|
|
|||
|
|
@ -527,7 +527,7 @@ export function createCliScanProgress(
|
|||
io.stdout.write(`${line}\n`);
|
||||
},
|
||||
startPhase(phaseWeight: number) {
|
||||
return createCliScanProgress(io, state, state.progress, phaseWeight);
|
||||
return createCliScanProgress(io, state, state.progress, weight * phaseWeight);
|
||||
},
|
||||
flush() {
|
||||
if (!shouldWrite || !state.hasPendingTransient) {
|
||||
|
|
|
|||
|
|
@ -1305,6 +1305,7 @@ describe('setup databases step', () => {
|
|||
expect(config.connections.warehouse.historicSql).not.toHaveProperty('redactionPatterns');
|
||||
expect(config.connections.warehouse.historicSql).not.toHaveProperty(legacyHistoricSqlServiceAccountPatternsKey);
|
||||
expect(config.ingest.adapters).toContain('historic-sql');
|
||||
expect(config.ingest.workUnits.maxConcurrency).toBe(6);
|
||||
expect(io.stdout()).toContain('Historic SQL probe...');
|
||||
expect(io.stdout()).toContain('pg_stat_statements ready');
|
||||
});
|
||||
|
|
|
|||
|
|
@ -14,6 +14,8 @@ import { runKtxScan } from './scan.js';
|
|||
import { withSetupInterruptConfirmation } from './setup-interrupt.js';
|
||||
import { writeProjectLocalSecretReference } from './setup-secrets.js';
|
||||
|
||||
const HISTORIC_SQL_WORK_UNIT_MAX_CONCURRENCY = 6;
|
||||
|
||||
export type KtxSetupDatabaseDriver =
|
||||
| 'sqlite'
|
||||
| 'postgres'
|
||||
|
|
@ -930,7 +932,7 @@ async function writeConnectionConfig(input: {
|
|||
? (input.connection.historicSql as Record<string, unknown>)
|
||||
: null;
|
||||
if (historicSql?.enabled === true) {
|
||||
await ensureHistoricSqlAdapterEnabled(input.projectDir);
|
||||
await ensureHistoricSqlIngestDefaults(input.projectDir);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1057,9 +1059,19 @@ async function maybeConfigureSchemaScope(input: {
|
|||
return true;
|
||||
}
|
||||
|
||||
async function ensureHistoricSqlAdapterEnabled(projectDir: string): Promise<void> {
|
||||
async function ensureHistoricSqlIngestDefaults(projectDir: string): Promise<void> {
|
||||
const project = await loadKtxProject({ projectDir });
|
||||
if (project.config.ingest.adapters.includes('historic-sql')) {
|
||||
const adapters = project.config.ingest.adapters.includes('historic-sql')
|
||||
? project.config.ingest.adapters
|
||||
: [...project.config.ingest.adapters, 'historic-sql'];
|
||||
const maxConcurrency = Math.max(
|
||||
project.config.ingest.workUnits.maxConcurrency,
|
||||
HISTORIC_SQL_WORK_UNIT_MAX_CONCURRENCY,
|
||||
);
|
||||
if (
|
||||
adapters === project.config.ingest.adapters &&
|
||||
maxConcurrency === project.config.ingest.workUnits.maxConcurrency
|
||||
) {
|
||||
return;
|
||||
}
|
||||
await writeFile(
|
||||
|
|
@ -1068,7 +1080,11 @@ async function ensureHistoricSqlAdapterEnabled(projectDir: string): Promise<void
|
|||
...project.config,
|
||||
ingest: {
|
||||
...project.config.ingest,
|
||||
adapters: [...project.config.ingest.adapters, 'historic-sql'],
|
||||
adapters,
|
||||
workUnits: {
|
||||
...project.config.ingest.workUnits,
|
||||
maxConcurrency,
|
||||
},
|
||||
},
|
||||
}),
|
||||
'utf-8',
|
||||
|
|
|
|||
|
|
@ -676,4 +676,53 @@ describe('setup Anthropic model step', () => {
|
|||
).resolves.toMatchObject({ status: 'ready' });
|
||||
expect(healthCheck).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{
|
||||
backend: 'vertex',
|
||||
providerLines: [' backend: vertex', ' vertex:', ' project: kaelio-dev', ' location: us-east5'],
|
||||
model: 'claude-sonnet-4-6',
|
||||
},
|
||||
{
|
||||
backend: 'gateway',
|
||||
providerLines: [' backend: gateway', ' gateway:', ' api_key: env:AI_GATEWAY_API_KEY'],
|
||||
model: 'anthropic/claude-sonnet-4-6',
|
||||
},
|
||||
])('preserves already configured $backend llm setup without asking for Anthropic credentials', async (fixture) => {
|
||||
await writeFile(
|
||||
join(tempDir, 'ktx.yaml'),
|
||||
[
|
||||
'project: warehouse',
|
||||
'setup:',
|
||||
' database_connection_ids: []',
|
||||
' completed_steps:',
|
||||
' - project',
|
||||
' - llm',
|
||||
'connections: {}',
|
||||
'llm:',
|
||||
' provider:',
|
||||
...fixture.providerLines,
|
||||
' models:',
|
||||
` default: ${fixture.model}`,
|
||||
'ingest:',
|
||||
' embeddings:',
|
||||
' backend: deterministic',
|
||||
' model: deterministic',
|
||||
' dimensions: 8',
|
||||
].join('\n'),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
const healthCheck = vi.fn(async () => ({ ok: true as const }));
|
||||
const io = makeIo();
|
||||
await expect(
|
||||
runKtxSetupAnthropicModelStep({ projectDir: tempDir, inputMode: 'disabled', skipLlm: false }, io.io, {
|
||||
healthCheck,
|
||||
}),
|
||||
).resolves.toMatchObject({ status: 'ready' });
|
||||
|
||||
expect(healthCheck).not.toHaveBeenCalled();
|
||||
expect(io.stdout()).toContain(`LLM ready: yes (${fixture.model})`);
|
||||
expect(io.stderr()).not.toContain('Anthropic');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import { writeFile } from 'node:fs/promises';
|
||||
import { cancel, isCancel, password, select, text } from '@clack/prompts';
|
||||
import { resolveLocalKtxLlmConfig } from '@ktx/context';
|
||||
import { resolveKtxConfigReference } from '@ktx/context/core';
|
||||
import {
|
||||
type KtxProjectConfig,
|
||||
|
|
@ -170,13 +171,26 @@ export async function fetchAnthropicModels(
|
|||
return models.map((item, index) => ({ ...item, recommended: index === Math.max(recommendedIndex, 0) }));
|
||||
}
|
||||
|
||||
function hasCompletedLlm(config: KtxProjectConfig): boolean {
|
||||
return (
|
||||
config.setup?.completed_steps.includes('llm') === true &&
|
||||
config.llm.provider.backend === 'anthropic' &&
|
||||
typeof config.llm.models.default === 'string' &&
|
||||
config.llm.models.default.length > 0
|
||||
);
|
||||
export function isKtxSetupLlmConfigReady(config: KtxProjectLlmConfig): boolean {
|
||||
let resolved: KtxLlmConfig | null;
|
||||
try {
|
||||
resolved = resolveLocalKtxLlmConfig(config, process.env);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (!resolved) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (resolved.backend === 'vertex') {
|
||||
return typeof resolved.vertex?.location === 'string' && resolved.vertex.location.trim().length > 0;
|
||||
}
|
||||
|
||||
return resolved.backend === 'anthropic' || resolved.backend === 'gateway';
|
||||
}
|
||||
|
||||
function hasUsableConfiguredLlm(config: KtxProjectConfig): boolean {
|
||||
return isKtxSetupLlmConfigReady(config.llm);
|
||||
}
|
||||
|
||||
function buildProjectLlmConfig(
|
||||
|
|
@ -386,7 +400,7 @@ export async function runKtxSetupAnthropicModelStep(
|
|||
const project = await loadKtxProject({ projectDir: args.projectDir });
|
||||
if (
|
||||
args.forcePrompt !== true &&
|
||||
hasCompletedLlm(project.config) &&
|
||||
hasUsableConfiguredLlm(project.config) &&
|
||||
!args.anthropicApiKeyEnv &&
|
||||
!args.anthropicApiKeyFile &&
|
||||
!args.anthropicModel
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import { join } from 'node:path';
|
|||
import { promisify } from 'node:util';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { localFakeBundleReport, persistLocalBundleReport } from './ingest.test-utils.js';
|
||||
import { contextBuildCommands, writeKtxSetupContextState } from './setup-context.js';
|
||||
import { runDemoTour } from './setup-demo-tour.js';
|
||||
import { readKtxSetupStatus, runKtxSetup } from './setup.js';
|
||||
|
|
@ -91,6 +92,38 @@ describe('setup status', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
{
|
||||
backend: 'vertex',
|
||||
providerLines: [' backend: vertex', ' vertex:', ' project: kaelio-dev', ' location: us-east5'],
|
||||
model: 'claude-sonnet-4-6',
|
||||
},
|
||||
{
|
||||
backend: 'gateway',
|
||||
providerLines: [' backend: gateway', ' gateway:', ' api_key: env:AI_GATEWAY_API_KEY'],
|
||||
model: 'anthropic/claude-sonnet-4-6',
|
||||
},
|
||||
])('reports configured $backend llm backends as setup-ready', async (fixture) => {
|
||||
await mkdir(tempDir, { recursive: true });
|
||||
await writeFile(
|
||||
join(tempDir, 'ktx.yaml'),
|
||||
[
|
||||
'project: revenue',
|
||||
'llm:',
|
||||
' provider:',
|
||||
...fixture.providerLines,
|
||||
' models:',
|
||||
` default: ${fixture.model}`,
|
||||
'connections: {}',
|
||||
].join('\n'),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
await expect(readKtxSetupStatus(tempDir)).resolves.toMatchObject({
|
||||
llm: { backend: fixture.backend, ready: true, model: fixture.model },
|
||||
});
|
||||
});
|
||||
|
||||
it('uses setup database connection ids when present', async () => {
|
||||
await writeFile(
|
||||
join(tempDir, 'ktx.yaml'),
|
||||
|
|
@ -283,6 +316,62 @@ describe('setup status', () => {
|
|||
});
|
||||
});
|
||||
|
||||
it('reports Vertex LLM and context ready after a successful Metabase ingest report', async () => {
|
||||
await writeFile(
|
||||
join(tempDir, 'ktx.yaml'),
|
||||
[
|
||||
'project: revenue',
|
||||
'setup:',
|
||||
' database_connection_ids:',
|
||||
' - warehouse',
|
||||
' completed_steps:',
|
||||
' - project',
|
||||
' - databases',
|
||||
' - sources',
|
||||
'connections:',
|
||||
' warehouse:',
|
||||
' driver: postgres',
|
||||
' url: env:DATABASE_URL',
|
||||
' metabase:',
|
||||
' driver: metabase',
|
||||
' url: env:METABASE_URL',
|
||||
' api_key_ref: env:METABASE_API_KEY',
|
||||
' warehouse_connection_id: warehouse',
|
||||
'llm:',
|
||||
' provider:',
|
||||
' backend: vertex',
|
||||
' vertex:',
|
||||
' project: kaelio-dev',
|
||||
' location: us-east5',
|
||||
' models:',
|
||||
' default: claude-sonnet-4-6',
|
||||
'ingest:',
|
||||
' embeddings:',
|
||||
' backend: deterministic',
|
||||
' model: deterministic',
|
||||
' dimensions: 8',
|
||||
'',
|
||||
].join('\n'),
|
||||
'utf-8',
|
||||
);
|
||||
await persistLocalBundleReport(
|
||||
tempDir,
|
||||
localFakeBundleReport('metabase-job-1', {
|
||||
connectionId: 'warehouse',
|
||||
sourceKey: 'metabase',
|
||||
}),
|
||||
);
|
||||
|
||||
const status = await readKtxSetupStatus(tempDir);
|
||||
const io = makeIo();
|
||||
await expect(runKtxSetup({ command: 'status', projectDir: tempDir, json: false }, io.io)).resolves.toBe(0);
|
||||
|
||||
expect(status.llm).toMatchObject({ backend: 'vertex', ready: true, model: 'claude-sonnet-4-6' });
|
||||
expect(status.context).toMatchObject({ ready: true, status: 'completed' });
|
||||
expect(io.stdout()).toContain('LLM ready: yes (claude-sonnet-4-6)');
|
||||
expect(io.stdout()).toContain('KTX context built: yes');
|
||||
});
|
||||
|
||||
it('prints plain and JSON setup status', async () => {
|
||||
const plainIo = makeIo();
|
||||
const jsonIo = makeIo();
|
||||
|
|
@ -1178,6 +1267,77 @@ describe('setup status', () => {
|
|||
expect(calls).toEqual(['model', 'embeddings', 'databases', 'sources']);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{
|
||||
backend: 'vertex',
|
||||
providerLines: [' backend: vertex', ' vertex:', ' project: kaelio-dev', ' location: us-east5'],
|
||||
model: 'claude-sonnet-4-6',
|
||||
},
|
||||
{
|
||||
backend: 'gateway',
|
||||
providerLines: [' backend: gateway', ' gateway:', ' api_key: env:AI_GATEWAY_API_KEY'],
|
||||
model: 'anthropic/claude-sonnet-4-6',
|
||||
},
|
||||
])('adds a dbt source in non-interactive setup with existing $backend llm config', async (fixture) => {
|
||||
const io = makeIo();
|
||||
await writeFile(
|
||||
join(tempDir, 'ktx.yaml'),
|
||||
[
|
||||
'project: revenue',
|
||||
'setup:',
|
||||
' database_connection_ids:',
|
||||
' - warehouse',
|
||||
' completed_steps:',
|
||||
' - project',
|
||||
' - databases',
|
||||
'connections:',
|
||||
' warehouse:',
|
||||
' driver: postgres',
|
||||
' url: env:WAREHOUSE_URL',
|
||||
'llm:',
|
||||
' provider:',
|
||||
...fixture.providerLines,
|
||||
' models:',
|
||||
` default: ${fixture.model}`,
|
||||
].join('\n'),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
await expect(
|
||||
runKtxSetup(
|
||||
{
|
||||
command: 'run',
|
||||
projectDir: tempDir,
|
||||
mode: 'existing',
|
||||
agents: false,
|
||||
skipAgents: true,
|
||||
inputMode: 'disabled',
|
||||
yes: true,
|
||||
cliVersion: '0.2.0',
|
||||
skipLlm: false,
|
||||
skipEmbeddings: true,
|
||||
skipDatabases: true,
|
||||
source: 'dbt',
|
||||
sourceConnectionId: 'dbt-main',
|
||||
sourceGitUrl: 'https://github.com/Kaelio/klo-dbt-demo',
|
||||
sourceBranch: 'main',
|
||||
sourceProjectName: 'orbit_analytics',
|
||||
sourceWarehouseConnectionId: 'warehouse',
|
||||
skipSources: false,
|
||||
databaseSchemas: [],
|
||||
},
|
||||
io.io,
|
||||
{
|
||||
sourcesDeps: { validateDbt: vi.fn(async () => ({ ok: true as const, detail: 'dbt project valid' })) },
|
||||
context: vi.fn(async () => ({ status: 'ready' as const, projectDir: tempDir, runId: 'setup-context-test' })),
|
||||
},
|
||||
),
|
||||
).resolves.toBe(0);
|
||||
|
||||
expect(io.stderr()).not.toContain('Anthropic');
|
||||
expect(await readFile(join(tempDir, 'ktx.yaml'), 'utf-8')).toContain('dbt-main:');
|
||||
});
|
||||
|
||||
it('does not fail context build when prerequisites were explicitly skipped and agents are skipped', async () => {
|
||||
const calls: string[] = [];
|
||||
const io = makeIo();
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
import { existsSync } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { cancel, isCancel, select } from '@clack/prompts';
|
||||
import { loadKtxProject } from '@ktx/context/project';
|
||||
import { getLatestLocalIngestStatus, savedMemoryCountsForReport } from '@ktx/context/ingest';
|
||||
import { ktxLocalStateDbPath, loadKtxProject, type KtxLocalProject } from '@ktx/context/project';
|
||||
import type { KtxCliIo } from './cli-runtime.js';
|
||||
import { formatSetupNextStepLines } from './next-steps.js';
|
||||
import { isKtxSetupExitError, withSetupInterruptConfirmation } from './setup-interrupt.js';
|
||||
|
|
@ -20,7 +21,7 @@ import {
|
|||
runKtxSetupDatabasesStep,
|
||||
} from './setup-databases.js';
|
||||
import { type KtxSetupEmbeddingsDeps, runKtxSetupEmbeddingsStep } from './setup-embeddings.js';
|
||||
import { type KtxSetupModelDeps, runKtxSetupAnthropicModelStep } from './setup-models.js';
|
||||
import { type KtxSetupModelDeps, isKtxSetupLlmConfigReady, runKtxSetupAnthropicModelStep } from './setup-models.js';
|
||||
import { type KtxSetupProjectDeps, runKtxSetupProjectStep } from './setup-project.js';
|
||||
import {
|
||||
isKtxPreAgentSetupReady,
|
||||
|
|
@ -226,10 +227,6 @@ async function runKtxSetupDemoFromEntryMenu(
|
|||
);
|
||||
}
|
||||
|
||||
function llmReady(status: KtxSetupStatus['llm']): boolean {
|
||||
return status.backend === 'anthropic' && typeof status.model === 'string' && status.model.length > 0;
|
||||
}
|
||||
|
||||
function embeddingsReady(status: KtxSetupStatus['embeddings']): boolean {
|
||||
return (
|
||||
status.backend !== undefined &&
|
||||
|
|
@ -252,6 +249,31 @@ function sourceConnections(config: Awaited<ReturnType<typeof loadKtxProject>>['c
|
|||
.sort((left, right) => left.connectionId.localeCompare(right.connectionId));
|
||||
}
|
||||
|
||||
type LocalIngestStatusReport = NonNullable<Awaited<ReturnType<typeof getLatestLocalIngestStatus>>>;
|
||||
|
||||
function reportHasSavedContext(report: LocalIngestStatusReport): boolean {
|
||||
if (report.body.failedWorkUnits.length > 0) {
|
||||
return false;
|
||||
}
|
||||
const counts = savedMemoryCountsForReport(report);
|
||||
return counts.wikiCount > 0 || counts.slCount > 0;
|
||||
}
|
||||
|
||||
async function readIngestContextStatus(project: KtxLocalProject): Promise<KtxSetupContextStatusSummary | null> {
|
||||
if (!existsSync(ktxLocalStateDbPath(project))) {
|
||||
return null;
|
||||
}
|
||||
const report = await getLatestLocalIngestStatus(project);
|
||||
if (!report || !reportHasSavedContext(report)) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
ready: true,
|
||||
status: 'completed',
|
||||
runId: report.runId,
|
||||
};
|
||||
}
|
||||
|
||||
export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupStatus> {
|
||||
const resolvedProjectDir = resolve(projectDir);
|
||||
if (!existsSync(join(resolvedProjectDir, 'ktx.yaml'))) {
|
||||
|
|
@ -269,10 +291,9 @@ export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupSt
|
|||
const project = await loadKtxProject({ projectDir: resolvedProjectDir });
|
||||
const llm = {
|
||||
backend: project.config.llm.provider.backend,
|
||||
ready: false,
|
||||
ready: isKtxSetupLlmConfigReady(project.config.llm),
|
||||
model: project.config.llm.models.default,
|
||||
};
|
||||
llm.ready = llmReady(llm);
|
||||
|
||||
const embeddings = {
|
||||
backend: project.config.ingest.embeddings.backend,
|
||||
|
|
@ -284,6 +305,10 @@ export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupSt
|
|||
|
||||
const completedSteps = project.config.setup?.completed_steps ?? [];
|
||||
const contextState = await readKtxSetupContextState(resolvedProjectDir);
|
||||
const setupContextStatus = setupContextStatusFromState(contextState, {
|
||||
completedStep: completedSteps.includes('context'),
|
||||
});
|
||||
const ingestContextStatus = setupContextStatus.ready ? null : await readIngestContextStatus(project);
|
||||
const databaseIds = project.config.setup?.database_connection_ids ?? Object.keys(project.config.connections);
|
||||
const databasesComplete = completedSteps.includes('databases');
|
||||
const manifest = await readKtxAgentInstallManifest(resolvedProjectDir);
|
||||
|
|
@ -306,7 +331,7 @@ export async function readKtxSetupStatus(projectDir: string): Promise<KtxSetupSt
|
|||
...source,
|
||||
ready: completedSteps.includes('sources'),
|
||||
})),
|
||||
context: setupContextStatusFromState(contextState, { completedStep: completedSteps.includes('context') }),
|
||||
context: ingestContextStatus ?? setupContextStatus,
|
||||
agents,
|
||||
};
|
||||
}
|
||||
|
|
@ -376,7 +401,7 @@ function setupStatusReady(status: KtxSetupStatus): boolean {
|
|||
return true;
|
||||
}
|
||||
return (
|
||||
llmReady(status.llm) &&
|
||||
status.llm.ready &&
|
||||
embeddingsReady(status.embeddings) &&
|
||||
status.databases.every((database) => database.ready) &&
|
||||
status.sources.every((source) => source.ready)
|
||||
|
|
|
|||
|
|
@ -448,10 +448,18 @@ joins: []
|
|||
listIo.io,
|
||||
);
|
||||
expect(code).toBe(0);
|
||||
expect(listIo.stderr()).toBe('');
|
||||
|
||||
const parsed = JSON.parse(listIo.stdout());
|
||||
expect(parsed.kind).toBe('list');
|
||||
expect(parsed.meta).toEqual({ command: 'sl list' });
|
||||
expect(parsed).toMatchObject({
|
||||
kind: 'list',
|
||||
data: {
|
||||
items: expect.any(Array),
|
||||
},
|
||||
meta: {
|
||||
command: 'sl list',
|
||||
},
|
||||
});
|
||||
expect(parsed.data.items).toHaveLength(1);
|
||||
expect(parsed.data.items[0]).toMatchObject({
|
||||
connectionId: 'warehouse',
|
||||
|
|
|
|||
|
|
@ -368,9 +368,9 @@ describe('standalone built ktx CLI smoke', () => {
|
|||
const knowledgeSearch = structuredContent<{
|
||||
results: Array<{ key: string; summary: string; score: number }>;
|
||||
totalFound: number;
|
||||
}>(await client.callTool({ name: 'knowledge_search', arguments: { query: 'ARR contract', limit: 5 } }));
|
||||
}>(await client.callTool({ name: 'knowledge_search', arguments: { query: 'ARR contract-first definition', limit: 10 } }));
|
||||
expect(knowledgeSearch.totalFound).toBeGreaterThan(0);
|
||||
expect(knowledgeSearch.results.map((result) => result.key)).toContain('arr-contract-first');
|
||||
expect(knowledgeSearch.results.map((result) => result.key)).toContain('orbit-arr-contract-first-definition');
|
||||
|
||||
const knowledgeRead = structuredContent<{
|
||||
key: string;
|
||||
|
|
@ -378,26 +378,26 @@ describe('standalone built ktx CLI smoke', () => {
|
|||
content: string;
|
||||
tags: string[];
|
||||
slRefs: string[];
|
||||
}>(await client.callTool({ name: 'knowledge_read', arguments: { key: 'arr-contract-first' } }));
|
||||
expect(knowledgeRead.key).toBe('arr-contract-first');
|
||||
}>(await client.callTool({ name: 'knowledge_read', arguments: { key: 'orbit-arr-contract-first-definition' } }));
|
||||
expect(knowledgeRead.key).toBe('orbit-arr-contract-first-definition');
|
||||
expect(knowledgeRead.summary).toContain('ARR');
|
||||
expect(knowledgeRead.content).toContain('contract');
|
||||
expect(knowledgeRead.slRefs).toContain('orbit_demo.contracts');
|
||||
expect(knowledgeRead.slRefs).toContain('mart_arr_daily');
|
||||
|
||||
const slRead = structuredContent<{ sourceName: string; yaml: string }>(
|
||||
await client.callTool({
|
||||
name: 'sl_read_source',
|
||||
arguments: { connectionId: 'orbit_demo', sourceName: 'accounts' },
|
||||
arguments: { connectionId: 'dbt-main', sourceName: 'mart_arr_daily' },
|
||||
}),
|
||||
);
|
||||
expect(slRead.sourceName).toBe('accounts');
|
||||
expect(slRead.yaml).toContain('name: accounts');
|
||||
expect(slRead.sourceName).toBe('mart_arr_daily');
|
||||
expect(slRead.yaml).toContain('name: mart_arr_daily');
|
||||
expect(slRead.yaml).toContain('measures:');
|
||||
|
||||
const slValidate = structuredContent<{ success: boolean; errors: string[]; warnings: string[] }>(
|
||||
await client.callTool({
|
||||
name: 'sl_validate',
|
||||
arguments: { connectionId: 'orbit_demo', names: ['accounts', 'contracts'] },
|
||||
arguments: { connectionId: 'dbt-main', names: ['mart_arr_daily', 'stg_contracts'] },
|
||||
}),
|
||||
);
|
||||
expect(slValidate.success).toBe(true);
|
||||
|
|
@ -716,7 +716,7 @@ describe('standalone built ktx CLI smoke', () => {
|
|||
'--project-dir',
|
||||
projectDir,
|
||||
'--token-env',
|
||||
'NOTION_AUTH_TOKEN',
|
||||
'NOTION_TOKEN',
|
||||
'--crawl-mode',
|
||||
'all_accessible',
|
||||
'--max-pages',
|
||||
|
|
@ -729,7 +729,7 @@ describe('standalone built ktx CLI smoke', () => {
|
|||
|
||||
const yaml = await readFile(join(projectDir, 'ktx.yaml'), 'utf-8');
|
||||
expect(yaml).toContain('driver: notion');
|
||||
expect(yaml).toContain('auth_token_ref: env:NOTION_AUTH_TOKEN');
|
||||
expect(yaml).toContain('auth_token_ref: env:NOTION_TOKEN');
|
||||
expect(yaml).toContain('crawl_mode: all_accessible');
|
||||
expect(yaml).toContain('max_pages_per_run: 5');
|
||||
expect(yaml).not.toContain('ntn_');
|
||||
|
|
@ -737,7 +737,7 @@ describe('standalone built ktx CLI smoke', () => {
|
|||
const parsed = parseKtxProjectConfig(yaml);
|
||||
expect(parsed.connections['notion-main']).toMatchObject({
|
||||
driver: 'notion',
|
||||
auth_token_ref: 'env:NOTION_AUTH_TOKEN',
|
||||
auth_token_ref: 'env:NOTION_TOKEN',
|
||||
crawl_mode: 'all_accessible',
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue