diff --git a/README.md b/README.md index cfabfbcc..b52a31f6 100644 --- a/README.md +++ b/README.md @@ -130,9 +130,7 @@ Scan artifacts are written under ```bash SCAN_OUTPUT="$(ktx scan warehouse --project-dir "$PROJECT_DIR")" printf '%s\n' "$SCAN_OUTPUT" -SCAN_RUN_ID="$(printf '%s\n' "$SCAN_OUTPUT" | awk '/^Run: / { print $2 }')" -ktx scan status --project-dir "$PROJECT_DIR" "$SCAN_RUN_ID" -ktx scan report --project-dir "$PROJECT_DIR" "$SCAN_RUN_ID" +ktx status --project-dir "$PROJECT_DIR" ``` For non-SQLite drivers, prefer credential references such as `--url env:NAME` @@ -147,16 +145,13 @@ version, and is managed by `ktx dev runtime` commands. KTX requires `uv` on `PATH` to create the managed runtime. Install `uv` with your system package manager or the official installer before running Python- backed KTX commands. KTX doesn't download `uv` automatically; run -`ktx dev runtime doctor` if runtime installation fails: +`ktx dev runtime status` if runtime installation fails: ```bash ktx dev runtime install --yes ktx dev runtime status -ktx dev runtime doctor ktx dev runtime start ktx dev runtime stop -ktx dev runtime prune --dry-run -ktx dev runtime prune --yes ``` The release artifact manifest contains the public npm tarball and the bundled `kaelio-ktx` @@ -223,7 +218,7 @@ KTX provider. Enable it with an environment flag when running an LLM-backed command: ```bash -KTX_AI_DEVTOOLS_ENABLED=true ktx dev ingest run \ +KTX_AI_DEVTOOLS_ENABLED=true ktx ingest run \ --connection-id warehouse \ --adapter metabase ``` diff --git a/docs-site/components/terminal-preview.tsx b/docs-site/components/terminal-preview.tsx index a1f950c8..d430c4ac 100644 --- a/docs-site/components/terminal-preview.tsx +++ b/docs-site/components/terminal-preview.tsx @@ -47,7 +47,7 @@ export function TerminalPreview() {
${" "} - ktx agent context --json + ktx status --json
diff --git a/docs-site/content/docs/ai-resources/agent-quickstart.mdx b/docs-site/content/docs/ai-resources/agent-quickstart.mdx index 40983224..6fd6e5ac 100644 --- a/docs-site/content/docs/ai-resources/agent-quickstart.mdx +++ b/docs-site/content/docs/ai-resources/agent-quickstart.mdx @@ -22,7 +22,7 @@ Agents should start with the smallest source that answers the task: | How to check project readiness | [ktx status](/docs/cli-reference/ktx-status) | [Quickstart](/docs/getting-started/quickstart) | | How context gets built | [Building Context](/docs/guides/building-context) | [ktx ingest](/docs/cli-reference/ktx-ingest) | | How semantic YAML works | [Writing Context](/docs/guides/writing-context) | [ktx sl](/docs/cli-reference/ktx-sl) | -| How machine-readable CLI output is shaped | [ktx agent](/docs/cli-reference/ktx-agent) | [Markdown Access](/docs/ai-resources/markdown-access) | +| How machine-readable CLI output is shaped | [ktx sl](/docs/cli-reference/ktx-sl) | [ktx wiki](/docs/cli-reference/ktx-wiki) | ## Operating workflow diff --git a/docs-site/content/docs/ai-resources/markdown-access.mdx b/docs-site/content/docs/ai-resources/markdown-access.mdx index c363a215..12bb7456 100644 --- a/docs-site/content/docs/ai-resources/markdown-access.mdx +++ b/docs-site/content/docs/ai-resources/markdown-access.mdx @@ -31,7 +31,8 @@ Every docs page has a Markdown route: ```text https://docs.kaelio.com/ktx/docs/getting-started/quickstart.md -https://docs.kaelio.com/ktx/docs/cli-reference/ktx-agent.md +https://docs.kaelio.com/ktx/docs/cli-reference/ktx-sl.md +https://docs.kaelio.com/ktx/docs/cli-reference/ktx-wiki.md https://docs.kaelio.com/ktx/docs/guides/building-context.md ``` diff --git a/docs-site/content/docs/cli-reference/ktx-agent.mdx b/docs-site/content/docs/cli-reference/ktx-agent.mdx deleted file mode 100644 index cdc4ceac..00000000 --- a/docs-site/content/docs/cli-reference/ktx-agent.mdx +++ /dev/null @@ -1,148 +0,0 @@ ---- -title: "ktx agent" -description: "Machine-readable commands for coding agents." ---- - -Hidden commands that provide machine-readable JSON output for coding agents. These are the commands that agent integrations (Claude Code, Cursor, Codex, OpenCode) call under the hood — you typically won't use them directly. - -All `ktx agent` subcommands require `--json` and produce structured JSON output on stdout. - -## Command signature - -```bash -ktx agent --json [options] -``` - -## Subcommands - -| Subcommand | Description | -|-----------|-------------| -| `tools` | Print available agent-facing KTX tools | -| `context` | Print project context for agent planning | -| `sl list` | List semantic-layer sources | -| `sl read ` | Read one semantic-layer source | -| `sl query` | Run a semantic-layer query from a JSON file | -| `wiki search ` | Search KTX wiki pages | -| `wiki read ` | Read one KTX wiki page | -| `sql execute` | Execute read-only SQL with a row limit | - -## Options - -### `agent tools` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | - -### `agent context` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | - -### `agent sl list` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | -| `--connection-id ` | Filter by connection id | — | -| `--query ` | Search source names and descriptions | — | - -### `agent sl read` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | -| `--connection-id ` | Connection id containing the source | — | - -### `agent sl query` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | -| `--connection-id ` | Connection id for execution (required) | — | -| `--query-file ` | JSON semantic-layer query file (required) | — | -| `--execute` | Execute the compiled query against the connection | `false` | -| `--max-rows ` | Maximum rows to return when executing (1-1000) | — | - -### `agent wiki search` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | -| `--limit ` | Maximum search results | `10` | - -### `agent wiki read` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | - -### `agent sql execute` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output (required) | — | -| `--connection-id ` | Connection id for execution (required) | — | -| `--sql-file ` | SQL file to execute (required) | — | -| `--max-rows ` | Maximum rows to return, 1-1000 (required) | — | - -## Examples - -```bash -# List available tools -ktx agent tools --json - -# Get project context for planning -ktx agent context --json - -# List semantic sources -ktx agent sl list --json - -# Search semantic sources by name -ktx agent sl list --json --query "revenue" - -# Read a semantic source -ktx agent sl read orders --json --connection-id my-warehouse - -# Run a semantic-layer query from a file -ktx agent sl query --json \ - --connection-id my-warehouse \ - --query-file /tmp/query.json \ - --execute \ - --max-rows 100 - -# Search wiki pages -ktx agent wiki search "churn definition" --json - -# Read a specific wiki page -ktx agent wiki read page-abc123 --json - -# Execute read-only SQL -ktx agent sql execute --json \ - --connection-id my-warehouse \ - --sql-file /tmp/query.sql \ - --max-rows 500 -``` - -## Output - -Every `ktx agent` command writes JSON to stdout and diagnostic text to stderr. Agents should parse stdout as JSON and treat a non-zero exit code as a failed tool call. - -```json -{ - "ok": true, - "data": { - "type": "agent-response" - } -} -``` - -## Common errors - -| Error | Cause | Recovery | -|-------|-------|----------| -| Missing JSON output | `--json` was omitted | Re-run the same subcommand with `--json` | -| Unknown connection id | The requested connection is not configured in `ktx.yaml` | Call `ktx agent context --json` or `ktx connection list` to discover valid ids | -| Query file cannot be read | `--query-file` points to a missing or invalid JSON file | Write the query payload to a real file and pass its absolute path | -| SQL execution rejected | SQL is not read-only or `--max-rows` is missing | Use semantic-layer queries first; for direct SQL, pass read-only SQL and an explicit row limit | diff --git a/docs-site/content/docs/cli-reference/ktx-dev.mdx b/docs-site/content/docs/cli-reference/ktx-dev.mdx index 82ba9acb..e00a4585 100644 --- a/docs-site/content/docs/cli-reference/ktx-dev.mdx +++ b/docs-site/content/docs/cli-reference/ktx-dev.mdx @@ -1,9 +1,9 @@ --- title: "ktx dev" -description: "Low-level diagnostics, scans, adapter commands, and mapping tools." +description: "Low-level project initialization and runtime management." --- -Hidden commands for low-level project management, diagnostics, direct adapter control, and shell completion. Most users interact with these through higher-level commands like [`ktx ingest`](/docs/cli-reference/ktx-ingest) and [`ktx setup`](/docs/cli-reference/ktx-setup), but `ktx dev` provides direct access when you need fine-grained control. +`ktx dev` contains development-only project initialization and managed runtime commands. Scan and ingest commands live at the root as [`ktx scan`](/docs/cli-reference/ktx-scan) and [`ktx ingest`](/docs/cli-reference/ktx-ingest). ## Command signature @@ -16,145 +16,42 @@ ktx dev [options] | Subcommand | Description | |-----------|-------------| | `init [directory]` | Initialize a Git-backed KTX project directory | -| `runtime` | Install, inspect, and prune the KTX-managed Python runtime | -| `scan` | Run or inspect standalone connection scans | -| `ingest run` | Run local ingest for one configured connection and source adapter | -| `ingest status [runId]` | Print status for a stored local ingest run | -| `ingest watch [runId]` | Open a stored ingest visual report | -| `ingest replay ` | Replay a stored ingest run through memory-flow output | -| `mapping` | Manage Metabase warehouse mappings (same as `ktx connection mapping`) | -| `completion zsh` | Generate zsh completion script | +| `runtime` | Install, start, stop, and inspect the KTX-managed Python runtime | -## Options - -### `dev init` +## `dev init` | Flag | Description | Default | |------|-------------|---------| | `--name ` | Project name written to `ktx.yaml` | — | | `--force` | Rewrite `ktx.yaml` and scaffold files in an existing project | `false` | -### `dev runtime` +## `dev runtime` + +`ktx dev runtime` supports `install`, `start`, `stop`, and `status`. | Flag | Description | Default | |------|-------------|---------| | `--feature ` | Runtime feature level for `install` and `start` (`core` or `local-embeddings`) | `core` | -| `--json` | Print JSON output | `false` | -| `--yes` | Confirm runtime install or prune actions where supported | `false` | +| `--json` | Print JSON output for `status` | `false` | +| `--yes` | Confirm runtime install actions where supported | `false` | | `--force` | Reinstall or restart where supported | `false` | -### `dev scan` - -See [`ktx scan`](/docs/cli-reference/ktx-scan) for the full scan command reference. - -### `dev ingest run` - -| Flag | Description | Default | -|------|-------------|---------| -| `--connection-id ` | KTX connection id (required) | — | -| `--adapter ` | Ingest source adapter name (required) | — | -| `--source-dir ` | Directory containing source files | — | -| `--database-introspection-url ` | Daemon URL for live-database introspection | — | -| `--debug-llm-request-file ` | Write sanitized LLM request structure to a JSONL file | — | -| `--plain` | Print plain text output | `false` | -| `--json` | Print JSON output | `false` | -| `--viz` | Render memory-flow TUI output | `false` | -| `--no-input` | Disable interactive terminal input for visualization | — | - -### `dev ingest status` - -| Flag | Description | Default | -|------|-------------|---------| -| `--report-file ` | Bundle ingest report JSON file to render | — | -| `--plain` | Print plain text output | `false` | -| `--json` | Print JSON output | `false` | -| `--viz` | Render memory-flow TUI output | `false` | -| `--no-input` | Disable interactive terminal input for visualization | — | - -### `dev ingest watch` - -| Flag | Description | Default | -|------|-------------|---------| -| `--report-file ` | Bundle ingest report JSON file to render | — | -| `--plain` | Print plain text output | `false` | -| `--json` | Print JSON output | `false` | -| `--viz` | Render memory-flow TUI output (the default unless `--plain` or `--json` is set) | `true` | -| `--no-input` | Disable interactive terminal input for visualization | — | - -### `dev ingest replay` - -| Flag | Description | Default | -|------|-------------|---------| -| `--report-file ` | Bundle ingest report JSON file to render | — | -| `--plain` | Print plain text output | `false` | -| `--json` | Print JSON output | `false` | -| `--viz` | Render memory-flow TUI output | `false` | -| `--no-input` | Disable interactive terminal input for visualization | — | - -### `dev completion zsh` - -| Flag | Description | Default | -|------|-------------|---------| -| `--install` | Install zsh completion into `~/.zfunc` and update `~/.zshrc` | `false` | - ## Examples ```bash -# Initialize a new KTX project ktx dev init - -# Initialize in a specific directory with a project name ktx dev init ./my-project --name "Analytics Context" - -# Re-initialize an existing project ktx dev init --force -# Check managed Python runtime readiness -ktx dev runtime doctor - -# Start the managed Python daemon +ktx dev runtime install --yes +ktx dev runtime status ktx dev runtime start - -# Run a low-level ingest with a specific adapter -ktx dev ingest run --connection-id my-dbt --adapter dbt - -# Run ingest from a specific source directory -ktx dev ingest run \ - --connection-id my-dbt \ - --adapter dbt \ - --source-dir ./dbt-project - -# View ingest status with the visual TUI -ktx dev ingest watch run-abc123 - -# Replay a stored ingest session -ktx dev ingest replay run-abc123 - -# View ingest status from a report file -ktx dev ingest status --report-file /tmp/ingest-report.json - -# Generate zsh completions -ktx dev completion zsh - -# Install zsh completions -ktx dev completion zsh --install +ktx dev runtime stop ``` -## Output - -`ktx dev` commands are diagnostic and may print plain text, JSON, or visual reports depending on the selected flags. - -| Mode | How to request it | Use case | -|------|-------------------|----------| -| Plain text | `--plain` or default diagnostic output | Human-readable terminal inspection | -| JSON | `--json` | Agent parsing and automation | -| Visual report | `--viz` | Interactive memory-flow and ingest debugging | - ## Common errors | Error | Cause | Recovery | |-------|-------|----------| -| Doctor reports missing runtime pieces | Packages, Python environment, or linked CLI are not ready | Run `pnpm install`, `pnpm run setup:dev`, and `uv sync --all-groups` | -| Ingest run cannot find adapter | `--adapter` does not match a supported source adapter | Use configured source names from `ktx.yaml` or run higher-level `ktx ingest` | -| Replay/report file cannot be read | The report path is wrong or the run id is not stored locally | Run `ktx dev ingest status --json` to discover stored run ids and report files | -| Visual output fails in CI | TUI rendering requires an interactive terminal | Use `--plain --no-input` or `--json --no-input` | +| Runtime status reports missing pieces | Packages, Python environment, or linked CLI are not ready | Run `pnpm install`, `pnpm run setup:dev`, `uv sync --all-groups`, then `ktx dev runtime status` | +| Runtime daemon does not start | The managed Python runtime is missing or stale | Run `ktx dev runtime install --yes`, then `ktx dev runtime start` | diff --git a/docs-site/content/docs/cli-reference/ktx-ingest.mdx b/docs-site/content/docs/cli-reference/ktx-ingest.mdx index 8ce9d9a5..e1c0e339 100644 --- a/docs-site/content/docs/cli-reference/ktx-ingest.mdx +++ b/docs-site/content/docs/cli-reference/ktx-ingest.mdx @@ -1,14 +1,13 @@ --- title: "ktx ingest" -description: "Build and refresh context from configured sources." +description: "Run and inspect local ingest memory-flow output." --- -Ingest context from your configured sources — dbt, Looker, Metabase, MetricFlow, LookML, or Notion. The ingest process extracts metadata from your tools, then uses an LLM agent to reconcile it with existing context, writing semantic sources and knowledge pages to your project. +`ktx ingest` runs adapter-level local ingest and renders stored ingest reports. ## Command signature ```bash -ktx ingest [connectionId] [options] ktx ingest [options] ``` @@ -16,80 +15,59 @@ ktx ingest [options] | Subcommand | Description | |-----------|-------------| -| `status [runId]` | Print status for the latest or selected public ingest run | -| `watch [runId]` | Open the latest or selected public ingest visual report | +| `run` | Run local ingest for one configured connection and source adapter | +| `status [runId]` | Print status for the latest or selected stored local ingest run or report file | +| `watch [runId]` | Open the latest or selected stored ingest visual report | +| `replay ` | Replay a stored ingest run or bundle report through memory-flow output | -## Options - -### `ingest` (run) +## `ingest run` | Flag | Description | Default | |------|-------------|---------| -| `--all` | Ingest every eligible configured source | `false` | +| `--connection-id ` | KTX connection id | Required | +| `--adapter ` | Ingest source adapter name | Required | +| `--source-dir ` | Directory containing source files | — | +| `--database-introspection-url ` | Daemon URL for live-database introspection | — | +| `--debug-llm-request-file ` | Write sanitized LLM request structure to a JSONL file | — | +| `--plain` | Print plain text output | `true` | | `--json` | Print JSON output | `false` | -| `--no-input` | Disable interactive terminal input | — | +| `--viz` | Render memory-flow TUI output | `false` | +| `--yes` | Install the managed Python runtime without prompting when required | `false` | +| `--no-input` | Disable interactive terminal input for visualization and runtime installation | — | -### `ingest status` +## `ingest status`, `watch`, and `replay` | Flag | Description | Default | |------|-------------|---------| +| `--report-file ` | Bundle ingest report JSON file to render | — | +| `--plain` | Print plain text output | `true` for `status` and `replay` | | `--json` | Print JSON output | `false` | -| `--no-input` | Disable interactive terminal input | — | - -### `ingest watch` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print JSON output instead of the visual report | `false` | -| `--no-input` | Disable interactive terminal input | — | +| `--viz` | Render memory-flow TUI output | `true` for `watch` | +| `--no-input` | Disable interactive terminal input for visualization | — | ## Examples ```bash -# Ingest from a specific connection -ktx ingest my-dbt-source +ktx ingest run --connection-id my-dbt-source --adapter dbt +ktx ingest run --connection-id prod-metabase --adapter metabase --yes -# Ingest from all eligible sources -ktx ingest --all - -# Check the status of the latest ingest ktx ingest status - -# Check the status of a specific ingest run ktx ingest status run-abc123 - -# Watch the latest ingest report -ktx ingest watch - -# Get ingest status as JSON ktx ingest status --json -``` -## Low-level ingest commands +ktx ingest watch +ktx ingest watch run-abc123 -For adapter-level control, use `ktx dev ingest`. See [`ktx dev`](/docs/cli-reference/ktx-dev) for the full low-level ingest surface including `run`, `status`, `watch`, and `replay` with output mode options (`--plain`, `--json`, `--viz`). - -## Output - -Ingest run commands print progress and create a stored ingest report. `ktx ingest status --json` returns the run state, adapter, connection, and summary information. - -```json -{ - "runId": "ingest-local-abc123", - "status": "completed", - "connectionId": "dbt-main", - "summary": { - "semanticSourcesChanged": 4, - "knowledgePagesChanged": 2 - } -} +ktx ingest replay run-abc123 +ktx ingest replay run-abc123 --viz +ktx ingest replay run-abc123 --report-file /tmp/ingest-report.json ``` ## Common errors | Error | Cause | Recovery | |-------|-------|----------| -| No eligible sources | `ktx.yaml` has no configured context source for ingest | Add a source with `ktx setup` or `ktx connection add`, then rerun ingest | | Ingest needs credentials | The source adapter requires API or git access | Configure the referenced environment variable or secret file | -| Latest run not found | No ingest run has been started in this project | Run `ktx ingest ` or `ktx ingest --all` first | +| Ingest run cannot find adapter | `--adapter` does not match a supported source adapter | Use a configured adapter such as `dbt`, `metabase`, `looker`, `lookml`, `notion`, or `live-database` | +| Latest run not found | No ingest run has been started in this project | Run `ktx ingest run --connection-id --adapter ` first | | Report watch fails in a non-interactive shell | Visual report needs a terminal | Use `ktx ingest status --json` for agent and CI workflows | diff --git a/docs-site/content/docs/cli-reference/ktx-scan.mdx b/docs-site/content/docs/cli-reference/ktx-scan.mdx index 0c37eccb..2f73ed99 100644 --- a/docs-site/content/docs/cli-reference/ktx-scan.mdx +++ b/docs-site/content/docs/cli-reference/ktx-scan.mdx @@ -1,163 +1,39 @@ --- title: "ktx scan" -description: "Run or inspect database scans." +description: "Run standalone database scans." --- -Discover your database schema — tables, columns, types, constraints, and relationships. Scanning is the first step in building context: KTX needs to understand your warehouse structure before it can build semantic sources. - -Scan commands live under `ktx dev scan`. See also the [Building Context](/docs/guides/building-context) guide for a walkthrough. +Discover a configured database connection's schema, including tables, columns, types, constraints, and optional relationship signals. ## Command signature ```bash -ktx dev scan [options] -ktx dev scan [options] +ktx scan [options] ``` -## Subcommands - -| Subcommand | Description | -|-----------|-------------| -| `status ` | Print status for a local scan run | -| `report ` | Print a local scan report | -| `relationships ` | Print relationship artifacts for a local scan run | -| `relationship-apply ` | Apply accepted relationship review decisions as manual manifest joins | -| `relationship-feedback` | Export persisted relationship review decisions as calibration labels | -| `relationship-calibration` | Summarize relationship feedback labels against current score thresholds | -| `relationship-thresholds` | Evaluate relationship feedback labels for offline threshold advice | - ## Options -### `scan` (run) - | Flag | Description | Default | |------|-------------|---------| | `--mode ` | Scan mode: `structural`, `enriched`, or `relationships` | `structural` | | `--dry-run` | Run without writing scan results | `false` | | `--database-introspection-url ` | Daemon URL for live-database introspection | — | - -### `scan report` - -| Flag | Description | Default | -|------|-------------|---------| -| `--json` | Print the raw scan report JSON | `false` | - -### `scan relationships` - -| Flag | Description | Default | -|------|-------------|---------| -| `--status ` | Filter by status: `accepted`, `review`, `rejected`, `skipped`, or `all` | `review` | -| `--limit ` | Maximum relationships to print per status | `25` | -| `--accept ` | Record an accepted decision for a relationship candidate | — | -| `--reject ` | Record a rejected decision for a relationship candidate | — | -| `--note ` | Attach a note when recording a relationship review decision | — | -| `--reviewer ` | Reviewer name for a relationship review decision | — | -| `--json` | Print relationship artifacts as JSON | `false` | - -### `scan relationship-apply` - -| Flag | Description | Default | -|------|-------------|---------| -| `--all-accepted` | Apply all accepted relationship review decisions for the scan run | `false` | -| `--candidate ` | Apply one accepted relationship review decision; repeatable | — | -| `--dry-run` | Preview relationships that would be written without rewriting manifest shards | `false` | -| `--json` | Print the apply result as JSON | `false` | - -### `scan relationship-feedback` - -| Flag | Description | Default | -|------|-------------|---------| -| `--connection ` | Only export labels for one KTX connection | — | -| `--decision ` | Filter: `accepted`, `rejected`, or `all` | `all` | -| `--json` | Print the export as JSON | `false` | -| `--jsonl` | Print labels as newline-delimited JSON | `false` | - -### `scan relationship-calibration` - -| Flag | Description | Default | -|------|-------------|---------| -| `--connection ` | Only calibrate labels for one KTX connection | — | -| `--decision ` | Filter: `accepted`, `rejected`, or `all` | `all` | -| `--accept-threshold ` | Score threshold treated as predicted accepted (0–1) | `0.85` | -| `--review-threshold ` | Score threshold treated as predicted review (0–1) | `0.55` | -| `--json` | Print the calibration report as JSON | `false` | - -### `scan relationship-thresholds` - -| Flag | Description | Default | -|------|-------------|---------| -| `--connection ` | Only evaluate labels for one KTX connection | — | -| `--min-total-labels ` | Minimum scored labels before advice can be ready | `20` | -| `--min-accepted-labels ` | Minimum accepted labels before advice can be ready | `5` | -| `--min-rejected-labels ` | Minimum rejected labels before advice can be ready | `5` | -| `--json` | Print the threshold advice report as JSON | `false` | +| `--yes` | Install the managed Python runtime without prompting when required | `false` | +| `--no-input` | Disable interactive managed runtime installation | — | ## Examples ```bash -# Run a structural scan of a connection -ktx dev scan my-warehouse - -# Run a scan with LLM enrichment -ktx dev scan my-warehouse --mode enriched - -# Run a scan with relationship detection -ktx dev scan my-warehouse --mode relationships - -# Dry-run a scan (don't write results) -ktx dev scan my-warehouse --dry-run - -# Check the status of a scan run -ktx dev scan status run-abc123 - -# View the scan report -ktx dev scan report run-abc123 - -# View scan report as JSON -ktx dev scan report run-abc123 --json - -# List relationship candidates pending review -ktx dev scan relationships run-abc123 - -# List all relationships regardless of status -ktx dev scan relationships run-abc123 --status all - -# Accept a relationship candidate -ktx dev scan relationships run-abc123 --accept candidate-xyz - -# Reject a relationship candidate with a note -ktx dev scan relationships run-abc123 --reject candidate-xyz --note "false positive" - -# Apply all accepted relationships to the manifest -ktx dev scan relationship-apply run-abc123 --all-accepted - -# Preview what would be applied -ktx dev scan relationship-apply run-abc123 --all-accepted --dry-run - -# Export relationship feedback as calibration labels -ktx dev scan relationship-feedback --json - -# Calibrate relationship detection thresholds -ktx dev scan relationship-calibration --accept-threshold 0.9 --review-threshold 0.6 - -# Get threshold advice based on review decisions -ktx dev scan relationship-thresholds +ktx scan my-warehouse +ktx scan my-warehouse --mode enriched +ktx scan my-warehouse --mode relationships +ktx scan my-warehouse --dry-run +ktx scan my-warehouse --database-introspection-url http://127.0.0.1:8765 ``` ## Output -Scan commands write scan artifacts under the KTX project directory and print status or report summaries. Use `--json` on report and relationship commands when an agent needs structured output. - -```json -{ - "runId": "scan-local-abc123", - "status": "completed", - "mode": "structural", - "changes": { - "tablesAdded": 42 - } -} -``` +`ktx scan` prints a human summary and writes scan artifacts under the KTX project directory unless `--dry-run` is set. Use `ktx status` after a scan to inspect project readiness and next setup work. ## Common errors @@ -165,5 +41,4 @@ Scan commands write scan artifacts under the KTX project directory and print sta |-------|-------|----------| | Scan cannot connect | Connection credentials or network access are invalid | Run `ktx connection test ` and update the connection before scanning | | Enriched scan cannot describe columns | LLM credentials are missing or invalid | Complete LLM setup with `ktx setup` before enriched scans | -| Relationship apply writes nothing | No accepted candidates match the provided run id or candidate ids | Inspect `ktx dev scan relationships --status accepted` first | -| Calibration is not ready | Too few reviewed relationship labels exist | Review and accept/reject more candidates, then rerun calibration | +| Relationship scan has limited evidence | The connector cannot provide optional validation or statistics | Re-run with a connector that supports the missing capability, or treat relationship output as lower-confidence context | diff --git a/docs-site/content/docs/cli-reference/ktx-sl.mdx b/docs-site/content/docs/cli-reference/ktx-sl.mdx index 4ec7bdd1..f5a31b27 100644 --- a/docs-site/content/docs/cli-reference/ktx-sl.mdx +++ b/docs-site/content/docs/cli-reference/ktx-sl.mdx @@ -28,6 +28,7 @@ ktx sl [options] | Flag | Description | Default | |------|-------------|---------| | `--connection-id ` | Filter by KTX connection id | — | +| `--query ` | Search source names and descriptions | — | | `--output ` | Output mode: `pretty` (default in TTY), `plain` (TSV), or `json` | `pretty` | | `--json` | Shortcut for `--output=json` (overrides `--output`) | `false` | @@ -36,6 +37,7 @@ ktx sl [options] | Flag | Description | Default | |------|-------------|---------| | `--connection-id ` | KTX connection id (required) | — | +| `--json` | Print JSON output | `false` | ### `sl validate` @@ -55,6 +57,7 @@ ktx sl [options] | Flag | Description | Default | |------|-------------|---------| | `--connection-id ` | KTX connection id | — | +| `--query-file ` | JSON semantic-layer query file | — | | `--measure ` | Measure to query; repeatable (at least one required) | — | | `--dimension ` | Dimension to include; repeatable | — | | `--filter ` | Filter expression; repeatable | — | @@ -78,9 +81,15 @@ ktx sl list --connection-id my-warehouse # List sources as JSON ktx sl list --json +# Search sources as JSON +ktx sl list --json --query "revenue" + # Read a source definition ktx sl read orders --connection-id my-warehouse +# Read a source definition as JSON +ktx sl read orders --connection-id my-warehouse --json + # Validate a source against the live schema ktx sl validate orders --connection-id my-warehouse @@ -119,6 +128,13 @@ ktx sl query \ --dimension orders.created_date \ --execute \ --max-rows 1000 + +# Execute a query from a JSON file +ktx sl query \ + --connection-id my-warehouse \ + --query-file query.json \ + --execute \ + --max-rows 100 ``` ## Output diff --git a/docs-site/content/docs/cli-reference/ktx-wiki.mdx b/docs-site/content/docs/cli-reference/ktx-wiki.mdx index a709ac07..7e45420e 100644 --- a/docs-site/content/docs/cli-reference/ktx-wiki.mdx +++ b/docs-site/content/docs/cli-reference/ktx-wiki.mdx @@ -26,19 +26,23 @@ ktx wiki [options] | Flag | Description | Default | |------|-------------|---------| +| `--json` | Print JSON output | `false` | | `--user-id ` | Local user id | `local` | ### `wiki read` | Flag | Description | Default | |------|-------------|---------| +| `--json` | Print JSON output | `false` | | `--user-id ` | Local user id | `local` | ### `wiki search` | Flag | Description | Default | |------|-------------|---------| +| `--json` | Print JSON output | `false` | | `--user-id ` | Local user id | `local` | +| `--limit ` | Maximum search results | — | ### `wiki write` @@ -58,12 +62,21 @@ ktx wiki [options] # List all wiki pages ktx wiki list +# List all wiki pages as JSON +ktx wiki list --json + # Read a specific wiki page ktx wiki read revenue-definitions +# Read a specific wiki page as JSON +ktx wiki read revenue-definitions --json + # Search wiki pages ktx wiki search "monthly recurring revenue" +# Search wiki pages as JSON +ktx wiki search "monthly recurring revenue" --json --limit 10 + # Write a global knowledge page ktx wiki write revenue-definitions \ --summary "Canonical revenue metric definitions" \ @@ -97,13 +110,16 @@ Wiki commands print local knowledge pages and search results. Agents should sear ```json { - "results": [ - { - "key": "revenue-definitions", - "summary": "Canonical revenue metric definitions", - "score": 0.92 - } - ] + "kind": "list", + "data": { + "items": [ + { + "key": "revenue-definitions", + "summary": "Canonical revenue metric definitions", + "score": 0.92 + } + ] + } } ``` diff --git a/docs-site/content/docs/cli-reference/meta.json b/docs-site/content/docs/cli-reference/meta.json index a5d7a95f..bed3f98c 100644 --- a/docs-site/content/docs/cli-reference/meta.json +++ b/docs-site/content/docs/cli-reference/meta.json @@ -9,7 +9,6 @@ "ktx-sl", "ktx-wiki", "ktx-status", - "ktx-agent", "ktx-dev" ] } diff --git a/docs-site/content/docs/concepts/context-as-code.mdx b/docs-site/content/docs/concepts/context-as-code.mdx index e40665ec..3c43082e 100644 --- a/docs-site/content/docs/concepts/context-as-code.mdx +++ b/docs-site/content/docs/concepts/context-as-code.mdx @@ -59,7 +59,7 @@ dbt / Looker / Metabase / Notion A typical branch shows a semantic diff: "this ingest added 3 new sources from dbt, updated 2 join definitions based on schema changes, and created 1 knowledge page from a Notion doc." Analytics engineers review the diff, verify that the new sources look correct, and merge. -Teams usually run this on demand while setting up a source, then schedule it once the source is stable. A cron job or CI schedule can run `ktx ingest --all --no-input` overnight on an ingest branch so the latest dbt manifests, BI metadata, and documentation updates are ready for review each morning. +Teams usually run this on demand while setting up a source, then schedule it once the source is stable. A cron job or CI schedule can run `ktx ingest run --connection-id --adapter --no-input` overnight on an ingest branch so the latest dbt manifests, BI metadata, and documentation updates are ready for review each morning. Once merged, agents querying through the KTX CLI see the updated context immediately. No deployment step, no cache invalidation, no restart. The files are the source of truth, and agents read them on every request. diff --git a/docs-site/content/docs/getting-started/quickstart.mdx b/docs-site/content/docs/getting-started/quickstart.mdx index ece3ceac..6aef2b14 100644 --- a/docs-site/content/docs/getting-started/quickstart.mdx +++ b/docs-site/content/docs/getting-started/quickstart.mdx @@ -211,7 +211,7 @@ KTX writes project state as plain files so agents can inspect and edit changes i | `semantic-layer//*.yaml` | context build, ingestion, or `ktx sl write` | Semantic source definitions agents use for SQL generation | | `knowledge/global/*.md` | ingestion or `ktx wiki write --scope global` | Shared business context and metric definitions | | `knowledge/user//*.md` | `ktx wiki write --scope user` | User-scoped notes for one agent/user context | -| `.claude/skills/ktx/SKILL.md`, `.agents/skills/ktx/SKILL.md` | CLI-mode agent integration setup | Agent instructions for calling `ktx agent` commands | +| `.claude/skills/ktx/SKILL.md`, `.agents/skills/ktx/SKILL.md` | CLI-mode agent integration setup | Agent instructions for calling public `ktx` commands | ## Verify it worked @@ -239,7 +239,7 @@ Agent integration ready: yes (claude-code:project) | `ktx: command not found` | The KTX package is not installed globally, or the shell cannot find the global binary | Run `npm install -g @kaelio/ktx` and open a new shell | | LLM health check fails | Missing, invalid, or unauthorized Anthropic API key | Export `ANTHROPIC_API_KEY` or rerun `ktx setup` and choose the file-backed secret option | | OpenAI embedding check fails | `OPENAI_API_KEY` is missing when OpenAI embeddings are selected | Export `OPENAI_API_KEY`, or rerun setup and choose local sentence-transformers embeddings | -| Local embeddings hang or fail | The managed Python runtime cannot start or the local model runtime is unavailable | Install `uv`, run `ktx dev runtime doctor`, then run `ktx dev runtime install --feature local-embeddings --yes` and rerun setup | +| Local embeddings hang or fail | The managed Python runtime cannot start or the local model runtime is unavailable | Install `uv`, run `ktx dev runtime status`, then run `ktx dev runtime install --feature local-embeddings --yes` and rerun setup | | Database connection test fails | Credentials, network access, warehouse, database, or schema value is wrong | Test the same URL with the database's native client, then rerun `ktx connection add ... --force` or rerun setup | | `KTX context built: no` in `ktx status` | Setup saved configuration but did not build context | Run `ktx setup` and choose to build context now | | Agent integration is incomplete | Setup skipped the agents step or the target was not installed | Run `ktx setup --agents --target codex --project` using the target you need | diff --git a/docs-site/content/docs/guides/building-context.mdx b/docs-site/content/docs/guides/building-context.mdx index 31d55bac..25d873d9 100644 --- a/docs-site/content/docs/guides/building-context.mdx +++ b/docs-site/content/docs/guides/building-context.mdx @@ -12,7 +12,7 @@ Scanning connects to your database and extracts structural metadata. KTX stores ### Running a scan ```bash -ktx dev scan +ktx scan ``` This runs a structural scan by default. You can control what the scan does with the `--mode` flag: @@ -25,25 +25,18 @@ This runs a structural scan by default. You can control what the scan does with ```bash # Scan with relationship detection -ktx dev scan my-postgres --mode relationships +ktx scan my-postgres --mode relationships # Preview without writing results -ktx dev scan my-postgres --dry-run +ktx scan my-postgres --dry-run ``` -### Checking scan status +### Checking scan results -Every scan produces a run ID. Use it to check progress or review results: +Every scan prints a summary and writes local artifacts. Use `ktx status` after a scan to review project readiness and follow-up setup work: ```bash -# Check status of a scan run -ktx dev scan status - -# Print the full scan report -ktx dev scan report - -# Get the report as JSON for scripting -ktx dev scan report --json +ktx status ``` ### Relationship detection @@ -56,49 +49,7 @@ Many databases lack declared foreign keys. KTX infers relationships by scoring c | 0.55 – 0.84 | `review` | Plausible — needs human review | | < 0.55 | `rejected` | Low confidence — not applied | -After a relationship scan, review the candidates: - -```bash -# Show candidates pending review (default) -ktx dev scan relationships - -# Show all candidates regardless of status -ktx dev scan relationships --status all - -# Accept a specific candidate -ktx dev scan relationships --accept - -# Reject a candidate with a note -ktx dev scan relationships --reject --note "These columns share a name but are unrelated" -``` - -Once you've reviewed candidates, apply the accepted ones as joins in your semantic layer: - -```bash -# Apply all accepted relationships -ktx dev scan relationship-apply --all-accepted - -# Preview what would be applied -ktx dev scan relationship-apply --all-accepted --dry-run - -# Apply a specific candidate -ktx dev scan relationship-apply --candidate -``` - -### Calibrating thresholds - -As you review more relationships, KTX can evaluate whether the default thresholds (0.85 accept, 0.55 review) are optimal for your schema: - -```bash -# See how your feedback aligns with current thresholds -ktx dev scan relationship-calibration --connection my-postgres - -# Get threshold recommendations (needs 20+ labels, 5+ accepted, 5+ rejected) -ktx dev scan relationship-thresholds --connection my-postgres - -# Export your review decisions as calibration labels -ktx dev scan relationship-feedback --connection my-postgres -``` +Relationship scans run with `ktx scan --mode relationships`. This command only executes the scan; relationship review and calibration subcommands are not part of the current CLI surface. ## Ingestion @@ -115,19 +66,7 @@ Each ingest run follows this flow: ### Running an ingest ```bash -# Ingest one configured context source -ktx ingest my-dbt-source - -# Ingest every configured context source -ktx ingest --all -``` - -The public `ktx ingest` command uses the source configuration in `ktx.yaml`, including the source `driver` and any adapter-specific paths or credentials. - -For adapter-level debugging, use the low-level `ktx dev ingest run` command: - -```bash -ktx dev ingest run --connection-id my-dbt-source --adapter dbt +ktx ingest run --connection-id my-dbt-source --adapter dbt ``` Useful low-level flags: @@ -152,7 +91,7 @@ ktx ingest status ktx ingest watch # Replay a past ingest run -ktx dev ingest replay +ktx ingest replay ``` The `watch` command opens an interactive TUI that shows the memory-flow output — every tool call, LLM decision, and artifact written during the ingest. @@ -235,7 +174,7 @@ Orders in "pending" status for more than 48 hours are flagged for review. Every ingest session records a full transcript — tool calls, LLM responses, and write decisions. You can replay any session to debug why a source was written a certain way: ```bash -ktx dev ingest replay --viz +ktx ingest replay --viz ``` This opens the same TUI view as the original run, letting you step through the agent's reasoning. diff --git a/docs-site/content/docs/guides/serving-agents.mdx b/docs-site/content/docs/guides/serving-agents.mdx index 4285611b..b6f073b8 100644 --- a/docs-site/content/docs/guides/serving-agents.mdx +++ b/docs-site/content/docs/guides/serving-agents.mdx @@ -3,37 +3,36 @@ title: Serving Agents description: Expose your context to Claude Code, Cursor, Codex, and other coding agents. --- -Once you've built and refined your context, the final step is exposing it to -coding agents. KTX provides machine-readable CLI commands for direct terminal -access from Claude Code, Cursor, Codex, OpenCode, and custom agent workflows. +Once you've built and refined your context, expose it to coding agents through +the public KTX CLI. Claude Code, Cursor, Codex, OpenCode, and custom agent +workflows can call the same commands you use at a terminal. ## CLI Commands -KTX provides a set of machine-readable commands under `ktx agent`. These return -JSON output designed for programmatic consumption. +KTX public commands support JSON output for the context reads that agents use +most often. Use `--project-dir` when the agent is not already running inside the +KTX project directory. ### Available commands ```bash -# List available tools and their descriptions -ktx agent tools --json - -# Get project context for planning -ktx agent context --json +# Check setup and context readiness +ktx status --json ``` **Semantic layer:** ```bash # List sources -ktx agent sl list --json -ktx agent sl list --json --connection-id my-postgres +ktx sl list --json +ktx sl list --json --connection-id my-postgres +ktx sl list --json --query "revenue" # Read a source -ktx agent sl read orders --json --connection-id my-postgres +ktx sl read orders --json --connection-id my-postgres # Run a query from a JSON file -ktx agent sl query --json \ +ktx sl query --json \ --connection-id my-postgres \ --query-file query.json \ --execute \ @@ -44,20 +43,10 @@ ktx agent sl query --json \ ```bash # Search knowledge pages -ktx agent wiki search "revenue recognition" --json --limit 10 +ktx wiki search "revenue recognition" --json --limit 10 # Read a specific page -ktx agent wiki read order-status-definitions --json -``` - -**SQL execution:** - -```bash -# Execute read-only SQL with a row limit -ktx agent sql execute --json \ - --connection-id my-postgres \ - --sql-file query.sql \ - --max-rows 500 +ktx wiki read order-status-definitions --json ``` ## Setting Up Your Agent diff --git a/docs-site/content/docs/integrations/agent-clients.mdx b/docs-site/content/docs/integrations/agent-clients.mdx index 1c105e1f..8a055fda 100644 --- a/docs-site/content/docs/integrations/agent-clients.mdx +++ b/docs-site/content/docs/integrations/agent-clients.mdx @@ -3,7 +3,9 @@ title: Agent Clients description: Set up KTX with Claude Code, Cursor, Codex, and OpenCode. --- -KTX integrates with coding agents through CLI skills and command files. These files teach agents to call `ktx agent ...` commands directly from the terminal for semantic-layer context, wiki knowledge, and safe SQL execution. +KTX integrates with coding agents through CLI skills and command files. These +files teach agents to call public `ktx` commands directly from the terminal for +semantic-layer context and wiki knowledge. Run `ktx setup` and select your agent targets, or configure manually using the snippets below. @@ -26,17 +28,17 @@ Create `.claude/skills/ktx/SKILL.md`: ```markdown title=".claude/skills/ktx/SKILL.md" --- name: ktx -description: Use local KTX semantic context, wiki knowledge, and safe SQL execution for this project. +description: Use local KTX semantic context and wiki knowledge for this project. --- Available commands: -- `ktx agent context --json --project-dir /path/to/project` -- `ktx agent sl list --json --project-dir /path/to/project` -- `ktx agent sl read '' --json --project-dir /path/to/project` -- `ktx agent sl query --json --project-dir /path/to/project --connection-id '' --query-file '' --execute --max-rows 100` -- `ktx agent wiki search '' --json --project-dir /path/to/project` -- `ktx agent wiki read '' --json --project-dir /path/to/project` -- `ktx agent sql execute --json --project-dir /path/to/project --connection-id '' --sql-file '' --max-rows 100` +- `ktx status --json --project-dir /path/to/project` +- `ktx sl list --json --project-dir /path/to/project` +- `ktx sl list --json --project-dir /path/to/project --query ''` +- `ktx sl read '' --json --project-dir /path/to/project --connection-id ''` +- `ktx sl query --json --project-dir /path/to/project --connection-id '' --query-file '' --execute --max-rows 100` +- `ktx wiki search '' --json --project-dir /path/to/project --limit 10` +- `ktx wiki read '' --json --project-dir /path/to/project` ``` ### Workflow tips @@ -123,22 +125,19 @@ All supported agent clients call the same KTX CLI commands: | Command | Description | |---------|-------------| -| `ktx agent context --json` | Return a compact project context summary | -| `ktx agent tools --json` | List available agent-facing commands | -| `ktx agent wiki search --json` | Search knowledge pages | -| `ktx agent wiki read --json` | Read a knowledge page | -| `ktx agent wiki write --json` | Write or update a knowledge page | -| `ktx agent sl list --json` | List semantic layer sources | -| `ktx agent sl read --json` | Read a semantic source definition | -| `ktx agent sl write --json` | Write or update a semantic source | -| `ktx agent sl validate --json` | Validate semantic source definitions | -| `ktx agent sl query --json` | Execute a semantic layer query when semantic compute is configured | -| `ktx agent sql execute --json` | Execute read-only SQL with an explicit row limit | +| `ktx status --json` | Return project setup and context readiness | +| `ktx wiki search --json` | Search knowledge pages | +| `ktx wiki read --json` | Read a knowledge page | +| `ktx wiki write ` | Write or update a knowledge page | +| `ktx sl list --json` | List semantic-layer sources | +| `ktx sl list --query --json` | Search semantic-layer sources | +| `ktx sl read --json --connection-id ` | Read a semantic source definition | +| `ktx sl write --connection-id ` | Write or update a semantic source | +| `ktx sl validate --connection-id ` | Validate semantic source definitions | +| `ktx sl query --json` | Execute a semantic-layer query when semantic compute is configured | ### Security constraints -- SQL execution is always read-only. -- Agent SQL execution requires an explicit `--max-rows` limit from 1 to 1000. - Secrets and credentials are never exposed in command output. - Commands resolve the project from `--project-dir`, `KTX_PROJECT_DIR`, or the nearest `ktx.yaml`. diff --git a/docs-site/content/docs/integrations/context-sources.mdx b/docs-site/content/docs/integrations/context-sources.mdx index 02554e08..904e3f95 100644 --- a/docs-site/content/docs/integrations/context-sources.mdx +++ b/docs-site/content/docs/integrations/context-sources.mdx @@ -13,7 +13,7 @@ Agents should configure and ingest context sources in this order: 1. Add the context source connection in `ktx.yaml` or with `ktx setup`. 2. Store tokens as `env:NAME` or `file:/path/to/secret`. -3. Run `ktx ingest ` for one source or `ktx ingest --all`. +3. Run `ktx ingest run --connection-id --adapter ` for one source or `ktx ingest run --connection-id --adapter `. 4. Check progress with `ktx ingest status --json`. 5. Review generated `semantic-layer/` YAML and `knowledge/` Markdown files in git. 6. Validate changed semantic sources with `ktx sl validate`. diff --git a/docs-site/content/docs/integrations/primary-sources.mdx b/docs-site/content/docs/integrations/primary-sources.mdx index 49200d47..94dc4e44 100644 --- a/docs-site/content/docs/integrations/primary-sources.mdx +++ b/docs-site/content/docs/integrations/primary-sources.mdx @@ -511,4 +511,4 @@ No authentication required — SQLite is file-based. The file must be readable b | Scan returns no tables | Schema/database/project filter is wrong or the user lacks metadata permissions | Verify the schema list and grant metadata read permissions | | Historic SQL is empty | Query history extension or warehouse history view is unavailable | Enable the warehouse-specific history feature, then rerun scan or setup | | Column statistics are missing | Connector cannot access stats tables or the warehouse does not expose them | Grant stats permissions where supported; otherwise rely on structural scan output | -| SQL execution fails through agents | Connection is missing, unreachable, or query execution is disabled | Run `ktx connection test ` and check the agent command flags | +| Semantic query execution fails | Connection is missing, unreachable, or query execution is disabled | Run `ktx connection test ` and check the `ktx sl query` flags | diff --git a/docs-site/lib/llm-docs.ts b/docs-site/lib/llm-docs.ts index 9d9b5c74..69aac698 100644 --- a/docs-site/lib/llm-docs.ts +++ b/docs-site/lib/llm-docs.ts @@ -67,12 +67,12 @@ ${link("/docs/guides/writing-context", "Writing Context", "Write semantic source - [Full documentation](${absoluteUrl("/llms-full.txt")}): All docs pages in one plain-text markdown response - [Markdown access guide](${absoluteUrl("/docs/ai-resources/markdown-access.md")}): How to fetch llms.txt, llms-full.txt, and per-page Markdown - [Quickstart markdown](${absoluteUrl("/docs/getting-started/quickstart.md")}): Human setup walkthrough -- [Agent CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-agent.md")}): Machine-readable agent commands +- [Semantic-layer CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-sl.md")}): Semantic-layer commands and JSON output +- [Wiki CLI markdown](${absoluteUrl("/docs/cli-reference/ktx-wiki.md")}): Knowledge page commands and JSON output ## CLI Reference ${link("/docs/cli-reference/ktx-setup", "ktx setup", "Interactive project setup")} -${link("/docs/cli-reference/ktx-agent", "ktx agent", "Machine-readable commands for coding agents")} ${link("/docs/cli-reference/ktx-sl", "ktx sl", "Semantic-layer commands")} ${link("/docs/cli-reference/ktx-wiki", "ktx wiki", "Knowledge page commands")} ${link("/docs/cli-reference/ktx-connection", "ktx connection", "Connection management commands")} diff --git a/examples/package-artifacts/README.md b/examples/package-artifacts/README.md index be161d88..22ecaf92 100644 --- a/examples/package-artifacts/README.md +++ b/examples/package-artifacts/README.md @@ -13,10 +13,8 @@ generated local project. The managed Python runtime smoke requires `uv` on `PATH`, isolates `KTX_RUNTIME_ROOT`, verifies `ktx dev runtime status`, runs `ktx sl query --yes` to -install the core runtime from the bundled wheel, checks `ktx dev runtime doctor`, -starts and reuses the managed daemon, stops it, previews a stale runtime with -`ktx dev runtime prune --dry-run`, verifies confirmation is required, and removes -the stale runtime with `ktx dev runtime prune --yes`. +install the core runtime from the bundled wheel, checks `ktx dev runtime status`, +starts and reuses the managed daemon, and stops it. The artifact manifest contains the public `@kaelio/ktx` npm tarball and the bundled `kaelio-ktx` runtime wheel. The smoke does not install standalone diff --git a/examples/postgres-historic/README.md b/examples/postgres-historic/README.md index c8c85cdc..40ae1674 100644 --- a/examples/postgres-historic/README.md +++ b/examples/postgres-historic/README.md @@ -95,7 +95,7 @@ note, not a warning. Run local historic-SQL ingest: ```bash -pnpm run ktx -- dev ingest run --project-dir /tmp/ktx-postgres-historic \ +pnpm run ktx -- ingest run --project-dir /tmp/ktx-postgres-historic \ --connection-id warehouse \ --adapter historic-sql \ --plain \ @@ -103,7 +103,7 @@ pnpm run ktx -- dev ingest run --project-dir /tmp/ktx-postgres-historic \ --no-input ``` -The full `dev ingest run` path also runs curation WorkUnits, so it requires a +The full `ingest run` path also runs curation WorkUnits, so it requires a configured LLM provider. Inspect the latest manifest: @@ -127,6 +127,6 @@ table. - Missing grants: confirm `GRANT pg_read_all_stats TO ktx_reader;`. - Empty snapshot: rerun `scripts/generate-workload.sh base` and keep `--historic-sql-min-executions 2` for the smoke. -- SQL-analysis failures: run `pnpm run ktx -- dev runtime doctor` from the KTX +- SQL-analysis failures: run `pnpm run ktx -- dev runtime status` from the KTX repository root and confirm `uv`, the bundled Python wheel, and the managed runtime all pass. diff --git a/packages/cli/src/agent-runtime.test.ts b/packages/cli/src/agent-runtime.test.ts deleted file mode 100644 index 36258f5b..00000000 --- a/packages/cli/src/agent-runtime.test.ts +++ /dev/null @@ -1,152 +0,0 @@ -import { mkdtemp, rm, writeFile } from 'node:fs/promises'; -import { tmpdir } from 'node:os'; -import { join } from 'node:path'; -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import { - KTX_AGENT_MAX_ROWS_CAP, - createKtxAgentRuntime, - parseAgentMaxRows, - readAgentJsonFile, - writeAgentJson, - writeAgentJsonError, -} from './agent-runtime.js'; - -function makeIo() { - let stdout = ''; - let stderr = ''; - return { - io: { - stdout: { write: (chunk: string) => (stdout += chunk) }, - stderr: { write: (chunk: string) => (stderr += chunk) }, - }, - stdout: () => stdout, - stderr: () => stderr, - }; -} - -describe('agent runtime helpers', () => { - let tempDir: string; - - beforeEach(async () => { - tempDir = await mkdtemp(join(tmpdir(), 'ktx-agent-runtime-')); - }); - - afterEach(async () => { - await rm(tempDir, { recursive: true, force: true }); - }); - - it('writes JSON success and error envelopes without color or spinners', () => { - const successIo = makeIo(); - const errorIo = makeIo(); - - writeAgentJson(successIo.io, { ok: true }); - writeAgentJsonError(errorIo.io, 'missing source', { code: 'NOT_FOUND' }); - - expect(JSON.parse(successIo.stdout())).toEqual({ ok: true }); - expect(successIo.stderr()).toBe(''); - expect(JSON.parse(errorIo.stderr())).toEqual({ - ok: false, - error: { message: 'missing source', code: 'NOT_FOUND' }, - }); - expect(errorIo.stdout()).toBe(''); - }); - - it('reads JSON query files as objects', async () => { - const path = join(tempDir, 'query.json'); - await writeFile(path, '{"measures":["revenue"],"limit":50}', 'utf-8'); - - await expect(readAgentJsonFile(path)).resolves.toEqual({ measures: ['revenue'], limit: 50 }); - }); - - it('rejects non-object JSON query files', async () => { - const path = join(tempDir, 'query.json'); - await writeFile(path, '["revenue"]', 'utf-8'); - - await expect(readAgentJsonFile(path)).rejects.toThrow('must contain a JSON object'); - }); - - it('requires positive row limits and enforces the agent cap', () => { - expect(parseAgentMaxRows(100)).toBe(100); - expect(() => parseAgentMaxRows(undefined)).toThrow('maxRows is required'); - expect(() => parseAgentMaxRows(0)).toThrow('positive integer'); - expect(() => parseAgentMaxRows(KTX_AGENT_MAX_ROWS_CAP + 1)).toThrow(String(KTX_AGENT_MAX_ROWS_CAP)); - }); - - it('constructs local context ports with semantic compute and query executor', async () => { - const project = { - projectDir: tempDir, - configPath: join(tempDir, 'ktx.yaml'), - config: { project: 'revenue', connections: {} }, - coreConfig: {}, - git: {}, - fileStore: {}, - } as never; - const ports = { knowledge: {}, semanticLayer: {} } as never; - const semanticLayerCompute = { query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }; - const queryExecutor = { execute: vi.fn() }; - const loadProject = vi.fn(async () => project); - const createContextTools = vi.fn(() => ports); - - await expect( - createKtxAgentRuntime( - { projectDir: tempDir, enableSemanticCompute: true, enableQueryExecution: true }, - { - loadProject, - createContextTools, - createSemanticLayerCompute: () => semanticLayerCompute, - createQueryExecutor: () => queryExecutor, - }, - ), - ).resolves.toMatchObject({ project, ports, queryExecutor }); - - expect(loadProject).toHaveBeenCalledWith({ projectDir: tempDir }); - expect(createContextTools).toHaveBeenCalledWith(project, { - semanticLayerCompute, - queryExecutor, - }); - }); - - it('creates managed semantic compute when no test override is injected', async () => { - const project = { - projectDir: tempDir, - configPath: join(tempDir, 'ktx.yaml'), - config: { project: 'revenue', connections: {} }, - coreConfig: {}, - git: {}, - fileStore: {}, - } as never; - const ports = { semanticLayer: {} } as never; - const semanticLayerCompute = { query: vi.fn(), validateSources: vi.fn(), generateSources: vi.fn() }; - const loadProject = vi.fn(async () => project); - const createContextTools = vi.fn(() => ports); - const createManagedSemanticLayerCompute = vi.fn(async () => semanticLayerCompute); - const { io } = makeIo(); - - await expect( - createKtxAgentRuntime( - { - projectDir: tempDir, - enableSemanticCompute: true, - enableQueryExecution: false, - cliVersion: '0.2.0', - runtimeInstallPolicy: 'auto', - io, - }, - { - loadProject, - createContextTools, - createManagedSemanticLayerCompute, - }, - ), - ).resolves.toMatchObject({ project, ports, semanticLayerCompute }); - - expect(createManagedSemanticLayerCompute).toHaveBeenCalledWith({ - cliVersion: '0.2.0', - installPolicy: 'auto', - io, - }); - expect(createContextTools).toHaveBeenCalledWith(project, { - semanticLayerCompute, - }); - }); -}); diff --git a/packages/cli/src/agent-runtime.ts b/packages/cli/src/agent-runtime.ts deleted file mode 100644 index feccae7c..00000000 --- a/packages/cli/src/agent-runtime.ts +++ /dev/null @@ -1,109 +0,0 @@ -import { readFile } from 'node:fs/promises'; -import { createDefaultLocalQueryExecutor, type KtxSqlQueryExecutorPort } from '@ktx/context/connections'; -import type { KtxSemanticLayerComputePort } from '@ktx/context/daemon'; -import { createLocalProjectMcpContextPorts, type KtxMcpContextPorts } from '@ktx/context/mcp'; -import { type KtxLocalProject, loadKtxProject } from '@ktx/context/project'; -import type { KtxCliIo } from './cli-runtime.js'; -import { - createManagedPythonSemanticLayerComputePort, - type KtxManagedPythonInstallPolicy, -} from './managed-python-command.js'; - -export const KTX_AGENT_MAX_ROWS_CAP = 1000; - -export interface KtxAgentRuntimeOptions { - projectDir: string; - enableSemanticCompute: boolean; - enableQueryExecution: boolean; - cliVersion?: string; - runtimeInstallPolicy?: KtxManagedPythonInstallPolicy; - io?: KtxCliIo; -} - -export interface KtxAgentRuntime { - project: KtxLocalProject; - ports: KtxMcpContextPorts; - semanticLayerCompute?: KtxSemanticLayerComputePort; - queryExecutor?: KtxSqlQueryExecutorPort; -} - -export interface KtxAgentRuntimeDeps { - loadProject?: typeof loadKtxProject; - createContextTools?: typeof createLocalProjectMcpContextPorts; - createSemanticLayerCompute?: () => KtxSemanticLayerComputePort; - createManagedSemanticLayerCompute?: typeof createManagedPythonSemanticLayerComputePort; - createQueryExecutor?: () => KtxSqlQueryExecutorPort; -} - -export function writeAgentJson(io: KtxCliIo, value: unknown): void { - io.stdout.write(`${JSON.stringify(value, null, 2)}\n`); -} - -export function writeAgentJsonError( - io: KtxCliIo, - message: string, - detail: Record = {}, -): void { - io.stderr.write(`${JSON.stringify({ ok: false, error: { message, ...detail } }, null, 2)}\n`); -} - -export async function readAgentJsonFile(path: string): Promise> { - const parsed = JSON.parse(await readFile(path, 'utf-8')) as unknown; - if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { - throw new Error(`${path} must contain a JSON object.`); - } - return parsed as Record; -} - -export function parseAgentMaxRows(value: number | undefined): number { - if (!Number.isInteger(value) || value === undefined || value <= 0) { - throw new Error('maxRows is required and must be a positive integer.'); - } - if (value > KTX_AGENT_MAX_ROWS_CAP) { - throw new Error(`maxRows must be less than or equal to ${KTX_AGENT_MAX_ROWS_CAP}.`); - } - return value; -} - -async function createAgentSemanticLayerCompute( - options: KtxAgentRuntimeOptions, - deps: KtxAgentRuntimeDeps, -): Promise { - if (!options.enableSemanticCompute) { - return undefined; - } - if (deps.createSemanticLayerCompute) { - return deps.createSemanticLayerCompute(); - } - if (!options.cliVersion || !options.runtimeInstallPolicy || !options.io) { - throw new Error('Managed Python semantic compute requires cliVersion, runtimeInstallPolicy, and io.'); - } - const createManagedSemanticLayerCompute = - deps.createManagedSemanticLayerCompute ?? createManagedPythonSemanticLayerComputePort; - return createManagedSemanticLayerCompute({ - cliVersion: options.cliVersion, - installPolicy: options.runtimeInstallPolicy, - io: options.io, - }); -} - -export async function createKtxAgentRuntime( - options: KtxAgentRuntimeOptions, - deps: KtxAgentRuntimeDeps = {}, -): Promise { - const project = await (deps.loadProject ?? loadKtxProject)({ projectDir: options.projectDir }); - const semanticLayerCompute = await createAgentSemanticLayerCompute(options, deps); - const queryExecutor = options.enableQueryExecution - ? (deps.createQueryExecutor ?? createDefaultLocalQueryExecutor)() - : undefined; - const ports = (deps.createContextTools ?? createLocalProjectMcpContextPorts)(project, { - ...(semanticLayerCompute ? { semanticLayerCompute } : {}), - ...(queryExecutor ? { queryExecutor } : {}), - }); - return { - project, - ports, - ...(semanticLayerCompute ? { semanticLayerCompute } : {}), - ...(queryExecutor ? { queryExecutor } : {}), - }; -} diff --git a/packages/cli/src/agent-search-readiness.test.ts b/packages/cli/src/agent-search-readiness.test.ts deleted file mode 100644 index cfb2999e..00000000 --- a/packages/cli/src/agent-search-readiness.test.ts +++ /dev/null @@ -1,51 +0,0 @@ -import { describe, expect, it } from 'vitest'; -import { - isMissingProjectConfigError, - missingConnectionSlSearchReadiness, - missingProjectSlSearchReadiness, - noConnectionsSlSearchReadiness, - noIndexedSourcesSlSearchReadiness, -} from './agent-search-readiness.js'; - -describe('agent semantic-layer search readiness guidance', () => { - it('formats missing project guidance with exact recovery commands', () => { - expect(missingProjectSlSearchReadiness('/tmp/ktx-search', 'gross revenue')).toEqual({ - code: 'agent_sl_search_missing_project', - message: 'Semantic-layer search needs an initialized KTX project at /tmp/ktx-search.', - nextSteps: [ - 'ktx setup --project-dir /tmp/ktx-search', - 'ktx status --project-dir /tmp/ktx-search', - 'ktx ingest ', - 'ktx agent sl list --json --query "gross revenue" --project-dir /tmp/ktx-search', - ], - }); - }); - - it('formats no-connection and no-index guidance without hiding the project path', () => { - expect(noConnectionsSlSearchReadiness('/tmp/ktx-search', 'revenue')).toMatchObject({ - code: 'agent_sl_search_no_connections', - message: 'Semantic-layer search found no configured connections in /tmp/ktx-search.', - }); - expect(noIndexedSourcesSlSearchReadiness('/tmp/ktx-search', 'orders')).toMatchObject({ - code: 'agent_sl_search_no_indexed_sources', - message: 'Semantic-layer search found no indexed semantic-layer sources in /tmp/ktx-search.', - }); - }); - - it('formats unknown connection guidance', () => { - expect(missingConnectionSlSearchReadiness('/tmp/ktx-search', 'warehouse', 'revenue')).toMatchObject({ - code: 'agent_sl_search_unknown_connection', - message: 'Semantic-layer search connection "warehouse" is not configured in /tmp/ktx-search.', - }); - }); - - it('detects missing ktx.yaml read errors', () => { - const error = Object.assign(new Error('ENOENT: no such file or directory'), { - code: 'ENOENT', - path: '/tmp/ktx-search/ktx.yaml', - }); - - expect(isMissingProjectConfigError(error)).toBe(true); - expect(isMissingProjectConfigError(new Error('other'))).toBe(false); - }); -}); diff --git a/packages/cli/src/agent-search-readiness.ts b/packages/cli/src/agent-search-readiness.ts deleted file mode 100644 index c3927613..00000000 --- a/packages/cli/src/agent-search-readiness.ts +++ /dev/null @@ -1,94 +0,0 @@ -export type KtxAgentSlSearchReadinessCode = - | 'agent_sl_search_missing_project' - | 'agent_sl_search_no_connections' - | 'agent_sl_search_unknown_connection' - | 'agent_sl_search_no_indexed_sources'; - -export interface KtxAgentSlSearchReadinessDetail { - code: KtxAgentSlSearchReadinessCode; - message: string; - nextSteps: string[]; -} - -function queryForCommand(query: string | undefined): string { - const trimmed = query?.trim(); - return trimmed && trimmed.length > 0 ? trimmed : 'revenue'; -} - -function projectSearchCommand(projectDir: string, query: string | undefined): string { - return `ktx agent sl list --json --query ${JSON.stringify(queryForCommand(query))} --project-dir ${projectDir}`; -} - -function baseNextSteps(projectDir: string, query: string | undefined): string[] { - return [ - `ktx setup --project-dir ${projectDir}`, - `ktx status --project-dir ${projectDir}`, - 'ktx ingest ', - projectSearchCommand(projectDir, query), - ]; -} - -export function missingProjectSlSearchReadiness( - projectDir: string, - query: string | undefined, -): KtxAgentSlSearchReadinessDetail { - return { - code: 'agent_sl_search_missing_project', - message: `Semantic-layer search needs an initialized KTX project at ${projectDir}.`, - nextSteps: baseNextSteps(projectDir, query), - }; -} - -export function noConnectionsSlSearchReadiness( - projectDir: string, - query: string | undefined, -): KtxAgentSlSearchReadinessDetail { - return { - code: 'agent_sl_search_no_connections', - message: `Semantic-layer search found no configured connections in ${projectDir}.`, - nextSteps: baseNextSteps(projectDir, query), - }; -} - -export function missingConnectionSlSearchReadiness( - projectDir: string, - connectionId: string, - query: string | undefined, -): KtxAgentSlSearchReadinessDetail { - return { - code: 'agent_sl_search_unknown_connection', - message: `Semantic-layer search connection "${connectionId}" is not configured in ${projectDir}.`, - nextSteps: baseNextSteps(projectDir, query), - }; -} - -export function noIndexedSourcesSlSearchReadiness( - projectDir: string, - query: string | undefined, -): KtxAgentSlSearchReadinessDetail { - return { - code: 'agent_sl_search_no_indexed_sources', - message: `Semantic-layer search found no indexed semantic-layer sources in ${projectDir}.`, - nextSteps: baseNextSteps(projectDir, query), - }; -} - -function errorCode(error: unknown): string | undefined { - if (typeof error !== 'object' || error === null || !('code' in error)) { - return undefined; - } - const code = (error as { code?: unknown }).code; - return typeof code === 'string' ? code : undefined; -} - -function errorPath(error: unknown): string | undefined { - if (typeof error !== 'object' || error === null || !('path' in error)) { - return undefined; - } - const path = (error as { path?: unknown }).path; - return typeof path === 'string' ? path : undefined; -} - -export function isMissingProjectConfigError(error: unknown): boolean { - return errorCode(error) === 'ENOENT' && (errorPath(error)?.endsWith('ktx.yaml') ?? false); -} diff --git a/packages/cli/src/agent.test.ts b/packages/cli/src/agent.test.ts deleted file mode 100644 index 2c86598d..00000000 --- a/packages/cli/src/agent.test.ts +++ /dev/null @@ -1,428 +0,0 @@ -import { mkdtemp, rm, writeFile } from 'node:fs/promises'; -import { tmpdir } from 'node:os'; -import { join } from 'node:path'; -import { buildDefaultKtxProjectConfig } from '@ktx/context/project'; -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import { runKtxAgent } from './agent.js'; -import type { KtxAgentRuntime } from './agent-runtime.js'; - -function makeIo() { - let stdout = ''; - let stderr = ''; - return { - io: { - stdout: { write: (chunk: string) => (stdout += chunk) }, - stderr: { write: (chunk: string) => (stderr += chunk) }, - }, - stdout: () => stdout, - stderr: () => stderr, - }; -} - -function runtime(overrides: Record = {}): KtxAgentRuntime { - const config = buildDefaultKtxProjectConfig('revenue'); - return { - project: { - projectDir: '/tmp/revenue', - configPath: '/tmp/revenue/ktx.yaml', - config: { - ...config, - connections: { - warehouse: { driver: 'sqlite', path: 'warehouse.sqlite', readonly: true as const }, - }, - }, - coreConfig: {} as KtxAgentRuntime['project']['coreConfig'], - git: {} as KtxAgentRuntime['project']['git'], - fileStore: {} as KtxAgentRuntime['project']['fileStore'], - }, - ports: { - connections: { list: vi.fn(async () => [{ id: 'warehouse', name: 'warehouse', connectionType: 'sqlite' }]) }, - semanticLayer: { - listSources: vi.fn(async () => ({ - sources: [ - { - connectionId: 'warehouse', - connectionName: 'warehouse', - name: 'orders', - columnCount: 2, - measureCount: 1, - joinCount: 0, - }, - ], - totalSources: 1, - })), - readSource: vi.fn(async () => ({ sourceName: 'orders', yaml: 'name: orders\n' })), - writeSource: vi.fn(async () => ({ success: true, sourceName: 'orders' })), - validate: vi.fn(async () => ({ success: true, errors: [], warnings: [] })), - query: vi.fn(async () => ({ sql: 'select 1', headers: ['x'], rows: [[1]], totalRows: 1, plan: {} })), - }, - knowledge: { - search: vi.fn(async () => ({ - results: [ - { - key: 'page-1', - path: 'knowledge/global/page-1.md', - scope: 'GLOBAL' as const, - summary: 'Revenue logic', - score: 0.9, - matchReasons: ['lexical' as const], - }, - ], - totalFound: 1, - })), - read: vi.fn(async () => ({ - key: 'page-1', - scope: 'GLOBAL' as const, - summary: 'Revenue logic', - content: 'Use net revenue.', - })), - write: vi.fn(async () => ({ success: true, key: 'page-1', action: 'created' as const })), - }, - }, - queryExecutor: { - execute: vi.fn(async () => ({ headers: ['x'], rows: [[1]], totalRows: 1, command: 'SELECT', rowCount: 1 })), - }, - ...overrides, - }; -} - -function runtimeWithoutConnections(): KtxAgentRuntime { - const base = runtime(); - return { - ...base, - project: { - ...base.project, - config: { - ...base.project.config, - connections: {}, - }, - }, - ports: { - ...base.ports, - semanticLayer: { - ...base.ports.semanticLayer!, - listSources: vi.fn(async () => ({ sources: [], totalSources: 0 })), - }, - }, - }; -} - -describe('runKtxAgent', () => { - let tempDir: string; - - beforeEach(async () => { - tempDir = await mkdtemp(join(tmpdir(), 'ktx-agent-')); - }); - - afterEach(async () => { - await rm(tempDir, { recursive: true, force: true }); - }); - - it('prints tool discovery with every stable command', async () => { - const io = makeIo(); - - await expect(runKtxAgent({ command: 'tools', projectDir: tempDir, json: true }, io.io)).resolves.toBe(0); - - const body = JSON.parse(io.stdout()); - expect(body.projectDir).toBe(tempDir); - expect(body.tools.map((tool: { name: string }) => tool.name)).toEqual([ - 'context', - 'sl.list', - 'sl.read', - 'sl.query', - 'wiki.search', - 'wiki.read', - 'sql.execute', - ]); - expect(io.stderr()).toBe(''); - }); - - it('prints project context from setup status, connections, and SL summaries', async () => { - const io = makeIo(); - const createRuntime = vi.fn(async () => runtime()); - const readSetupStatus = vi.fn(async () => ({ project: { path: tempDir, ready: true }, agents: [] })); - - await expect( - runKtxAgent({ command: 'context', projectDir: tempDir, json: true }, io.io, { createRuntime, readSetupStatus }), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ - projectDir: tempDir, - status: { project: { ready: true } }, - connections: [{ id: 'warehouse' }], - semanticLayer: { totalSources: 1 }, - }); - }); - - it('dispatches SL list, SL read, wiki search, and wiki read through local ports', async () => { - for (const args of [ - { command: 'sl-list' as const, projectDir: tempDir, json: true as const, connectionId: 'warehouse' }, - { - command: 'sl-read' as const, - projectDir: tempDir, - json: true as const, - connectionId: 'warehouse', - sourceName: 'orders', - }, - { command: 'wiki-search' as const, projectDir: tempDir, json: true as const, query: 'revenue', limit: 10 }, - { command: 'wiki-read' as const, projectDir: tempDir, json: true as const, pageId: 'page-1' }, - ]) { - const io = makeIo(); - await expect(runKtxAgent(args, io.io, { createRuntime: async () => runtime() })).resolves.toBe(0); - expect(JSON.parse(io.stdout())).toBeTruthy(); - expect(io.stderr()).toBe(''); - } - }); - - it('prints wiki hybrid search metadata from the hidden agent wiki search command', async () => { - const fakeRuntime = runtime(); - const knowledge = fakeRuntime.ports.knowledge; - if (!knowledge) { - throw new Error('Expected runtime knowledge port'); - } - fakeRuntime.ports.knowledge = { - ...knowledge, - search: vi.fn(async () => ({ - results: [ - { - key: 'metrics-revenue', - path: 'knowledge/global/metrics-revenue.md', - scope: 'GLOBAL' as const, - summary: 'Revenue metric definition', - score: 0.02459016393442623, - matchReasons: ['lexical' as const, 'token' as const], - }, - ], - totalFound: 1, - })), - }; - const io = makeIo(); - - await expect( - runKtxAgent({ command: 'wiki-search', projectDir: tempDir, json: true, query: 'paid order', limit: 5 }, io.io, { - createRuntime: async () => fakeRuntime, - }), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toEqual({ - results: [ - expect.objectContaining({ - key: 'metrics-revenue', - path: 'knowledge/global/metrics-revenue.md', - matchReasons: ['lexical', 'token'], - }), - ], - totalFound: 1, - }); - }); - - it('executes SL queries from a JSON query file', async () => { - const queryFile = join(tempDir, 'sl-query.json'); - const io = makeIo(); - await writeFile(queryFile, '{"measures":["total_revenue"],"dimensions":[]}', 'utf-8'); - - await expect( - runKtxAgent( - { - command: 'sl-query', - projectDir: tempDir, - json: true, - connectionId: 'warehouse', - queryFile, - execute: true, - maxRows: 100, - cliVersion: '0.2.0', - runtimeInstallPolicy: 'never', - }, - io.io, - { createRuntime: async () => runtime() }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ sql: 'select 1', rows: [[1]] }); - }); - - it('passes managed runtime options into default SL query runtime creation', async () => { - const queryFile = join(tempDir, 'sl-query.json'); - const io = makeIo(); - const createRuntime = vi.fn(async () => runtime()); - await writeFile(queryFile, '{"measures":["total_revenue"],"dimensions":[]}', 'utf-8'); - - await expect( - runKtxAgent( - { - command: 'sl-query', - projectDir: tempDir, - json: true, - connectionId: 'warehouse', - queryFile, - execute: false, - cliVersion: '0.2.0', - runtimeInstallPolicy: 'auto', - }, - io.io, - { createRuntime }, - ), - ).resolves.toBe(0); - - expect(createRuntime).toHaveBeenCalledWith({ - projectDir: tempDir, - enableSemanticCompute: true, - enableQueryExecution: false, - cliVersion: '0.2.0', - runtimeInstallPolicy: 'auto', - io: io.io, - }); - }); - - it('executes read-only SQL from a SQL file with an explicit row limit', async () => { - const sqlFile = join(tempDir, 'query.sql'); - const fakeRuntime = runtime(); - const io = makeIo(); - await writeFile(sqlFile, 'select 1', 'utf-8'); - - await expect( - runKtxAgent( - { - command: 'sql-execute', - projectDir: tempDir, - json: true, - connectionId: 'warehouse', - sqlFile, - maxRows: 100, - }, - io.io, - { createRuntime: async () => fakeRuntime as never }, - ), - ).resolves.toBe(0); - - expect(fakeRuntime.queryExecutor?.execute).toHaveBeenCalledWith({ - connectionId: 'warehouse', - projectDir: '/tmp/revenue', - connection: { driver: 'sqlite', path: 'warehouse.sqlite', readonly: true }, - sql: 'select 1', - maxRows: 100, - }); - }); - - it('prints guided JSON when semantic-layer search runs outside a project', async () => { - const io = makeIo(); - const missingProjectError = Object.assign(new Error('ENOENT: no such file or directory'), { - code: 'ENOENT', - path: join(tempDir, 'ktx.yaml'), - }); - - await expect( - runKtxAgent( - { command: 'sl-list', projectDir: tempDir, json: true, query: 'gross revenue' }, - io.io, - { createRuntime: vi.fn(async () => Promise.reject(missingProjectError)) }, - ), - ).resolves.toBe(1); - - expect(JSON.parse(io.stderr())).toEqual({ - ok: false, - error: { - code: 'agent_sl_search_missing_project', - message: `Semantic-layer search needs an initialized KTX project at ${tempDir}.`, - nextSteps: [ - `ktx setup --project-dir ${tempDir}`, - `ktx status --project-dir ${tempDir}`, - 'ktx ingest ', - `ktx agent sl list --json --query "gross revenue" --project-dir ${tempDir}`, - ], - }, - }); - expect(io.stdout()).toBe(''); - }); - - it('prints guided JSON when semantic-layer search has no configured connections', async () => { - const io = makeIo(); - - await expect( - runKtxAgent( - { command: 'sl-list', projectDir: tempDir, json: true, query: 'revenue' }, - io.io, - { createRuntime: async () => runtimeWithoutConnections() }, - ), - ).resolves.toBe(1); - - expect(JSON.parse(io.stderr())).toMatchObject({ - ok: false, - error: { - code: 'agent_sl_search_no_connections', - message: `Semantic-layer search found no configured connections in ${tempDir}.`, - nextSteps: [ - `ktx setup --project-dir ${tempDir}`, - `ktx status --project-dir ${tempDir}`, - 'ktx ingest ', - `ktx agent sl list --json --query "revenue" --project-dir ${tempDir}`, - ], - }, - }); - }); - - it('prints guided JSON when semantic-layer search asks for an unknown connection', async () => { - const io = makeIo(); - - await expect( - runKtxAgent( - { command: 'sl-list', projectDir: tempDir, json: true, connectionId: 'missing', query: 'revenue' }, - io.io, - { createRuntime: async () => runtime() }, - ), - ).resolves.toBe(1); - - expect(JSON.parse(io.stderr())).toMatchObject({ - ok: false, - error: { - code: 'agent_sl_search_unknown_connection', - message: `Semantic-layer search connection "missing" is not configured in ${tempDir}.`, - }, - }); - }); - - it('prints guided JSON when semantic-layer search has no indexed sources', async () => { - const fakeRuntime = runtime(); - const semanticLayer = fakeRuntime.ports.semanticLayer!; - fakeRuntime.ports.semanticLayer = { - ...semanticLayer, - listSources: vi.fn(async () => ({ sources: [], totalSources: 0 })), - }; - const io = makeIo(); - - await expect( - runKtxAgent( - { command: 'sl-list', projectDir: tempDir, json: true, connectionId: 'warehouse', query: 'revenue' }, - io.io, - { createRuntime: async () => fakeRuntime }, - ), - ).resolves.toBe(1); - - expect(JSON.parse(io.stderr())).toMatchObject({ - ok: false, - error: { - code: 'agent_sl_search_no_indexed_sources', - message: `Semantic-layer search found no indexed semantic-layer sources in ${tempDir}.`, - }, - }); - }); - - it('returns JSON errors when required ports or records are missing', async () => { - const io = makeIo(); - - await expect( - runKtxAgent({ command: 'wiki-read', projectDir: tempDir, json: true, pageId: 'missing' }, io.io, { - createRuntime: async () => - runtime({ - ports: { knowledge: { read: vi.fn(async () => null) } }, - }) as never, - }), - ).resolves.toBe(1); - - expect(JSON.parse(io.stderr())).toMatchObject({ - ok: false, - error: { message: expect.stringContaining('missing') }, - }); - }); -}); diff --git a/packages/cli/src/agent.ts b/packages/cli/src/agent.ts deleted file mode 100644 index 61d85b8c..00000000 --- a/packages/cli/src/agent.ts +++ /dev/null @@ -1,219 +0,0 @@ -import { readFile } from 'node:fs/promises'; -import type { KtxCliIo } from './cli-runtime.js'; -import { - createKtxAgentRuntime, - parseAgentMaxRows, - readAgentJsonFile, - writeAgentJson, - writeAgentJsonError, - type KtxAgentRuntime, - type KtxAgentRuntimeDeps, -} from './agent-runtime.js'; -import { - isMissingProjectConfigError, - missingConnectionSlSearchReadiness, - missingProjectSlSearchReadiness, - noConnectionsSlSearchReadiness, - noIndexedSourcesSlSearchReadiness, - type KtxAgentSlSearchReadinessDetail, -} from './agent-search-readiness.js'; -import type { KtxManagedPythonInstallPolicy } from './managed-python-command.js'; -import { readKtxSetupStatus, type KtxSetupStatus } from './setup.js'; - -export type KtxAgentArgs = - | { command: 'tools'; projectDir: string; json: true } - | { command: 'context'; projectDir: string; json: true } - | { command: 'sl-list'; projectDir: string; json: true; connectionId?: string; query?: string } - | { command: 'sl-read'; projectDir: string; json: true; connectionId?: string; sourceName: string } - | { - command: 'sl-query'; - projectDir: string; - json: true; - connectionId: string; - queryFile: string; - execute: boolean; - maxRows?: number; - cliVersion: string; - runtimeInstallPolicy: KtxManagedPythonInstallPolicy; - } - | { command: 'wiki-search'; projectDir: string; json: true; query: string; limit: number } - | { command: 'wiki-read'; projectDir: string; json: true; pageId: string } - | { command: 'sql-execute'; projectDir: string; json: true; connectionId: string; sqlFile: string; maxRows?: number }; - -export interface KtxAgentDeps extends KtxAgentRuntimeDeps { - createRuntime?: (options: { - projectDir: string; - enableSemanticCompute: boolean; - enableQueryExecution: boolean; - cliVersion?: string; - runtimeInstallPolicy?: KtxManagedPythonInstallPolicy; - io?: KtxCliIo; - }) => Promise; - readSetupStatus?: ( - projectDir: string, - ) => Promise; -} - -const AGENT_TOOLS = [ - { name: 'context', command: 'ktx agent context --json' }, - { name: 'sl.list', command: 'ktx agent sl list --json [--connection-id ] [--query ]' }, - { name: 'sl.read', command: 'ktx agent sl read --json [--connection-id ]' }, - { - name: 'sl.query', - command: 'ktx agent sl query --json --connection-id --query-file --execute --max-rows 100', - }, - { name: 'wiki.search', command: 'ktx agent wiki search --json [--limit 10]' }, - { name: 'wiki.read', command: 'ktx agent wiki read --json' }, - { - name: 'sql.execute', - command: 'ktx agent sql execute --json --connection-id --sql-file --max-rows 100', - }, -] as const; - -function writeAgentSlSearchReadinessError(io: KtxCliIo, detail: KtxAgentSlSearchReadinessDetail): void { - writeAgentJsonError(io, detail.message, { code: detail.code, nextSteps: detail.nextSteps }); -} - -async function runtimeFor(args: KtxAgentArgs, deps: KtxAgentDeps, io: KtxCliIo): Promise { - const needsSemanticCompute = args.command === 'sl-query'; - const needsQueryExecution = args.command === 'sql-execute' || (args.command === 'sl-query' && args.execute); - const runtimeOptions = { - projectDir: args.projectDir, - enableSemanticCompute: needsSemanticCompute, - enableQueryExecution: needsQueryExecution, - ...(args.command === 'sl-query' - ? { - cliVersion: args.cliVersion, - runtimeInstallPolicy: args.runtimeInstallPolicy, - io, - } - : {}), - }; - return deps.createRuntime ? deps.createRuntime(runtimeOptions) : createKtxAgentRuntime(runtimeOptions, deps); -} - -function connectionIdForSource(runtime: KtxAgentRuntime, requested: string | undefined): string { - if (requested) return requested; - const ids = Object.keys(runtime.project.config.connections ?? {}); - if (ids.length === 1) return ids[0] as string; - throw new Error('Use --connection-id when the project has zero or multiple connections.'); -} - -export async function runKtxAgent(args: KtxAgentArgs, io: KtxCliIo, deps: KtxAgentDeps = {}): Promise { - try { - if (args.command === 'tools') { - writeAgentJson(io, { projectDir: args.projectDir, tools: AGENT_TOOLS }); - return 0; - } - - const runtime = await runtimeFor(args, deps, io); - - if (args.command === 'context') { - const [status, connections, semanticLayer] = await Promise.all([ - (deps.readSetupStatus ?? readKtxSetupStatus)(args.projectDir), - runtime.ports.connections?.list() ?? [], - runtime.ports.semanticLayer?.listSources({}) ?? { sources: [], totalSources: 0 }, - ]); - writeAgentJson(io, { projectDir: args.projectDir, status, connections, semanticLayer, tools: AGENT_TOOLS }); - return 0; - } - - if (args.command === 'sl-list') { - const semanticLayer = runtime.ports.semanticLayer; - if (!semanticLayer) throw new Error('Semantic-layer tools are not available for this project.'); - if (args.query) { - const connectionIds = Object.keys(runtime.project.config.connections ?? {}); - if (args.connectionId && !runtime.project.config.connections[args.connectionId]) { - writeAgentSlSearchReadinessError( - io, - missingConnectionSlSearchReadiness(args.projectDir, args.connectionId, args.query), - ); - return 1; - } - if (connectionIds.length === 0) { - writeAgentSlSearchReadinessError(io, noConnectionsSlSearchReadiness(args.projectDir, args.query)); - return 1; - } - } - - const listed = await semanticLayer.listSources({ connectionId: args.connectionId, query: args.query }); - if (args.query && listed.sources.length === 0) { - const allSources = await semanticLayer.listSources({ connectionId: args.connectionId }); - if (allSources.totalSources === 0) { - writeAgentSlSearchReadinessError(io, noIndexedSourcesSlSearchReadiness(args.projectDir, args.query)); - return 1; - } - } - - writeAgentJson(io, listed); - return 0; - } - - if (args.command === 'sl-read') { - const semanticLayer = runtime.ports.semanticLayer; - if (!semanticLayer) throw new Error('Semantic-layer tools are not available for this project.'); - const source = await semanticLayer.readSource({ - connectionId: connectionIdForSource(runtime, args.connectionId), - sourceName: args.sourceName, - }); - if (!source) throw new Error(`Semantic-layer source "${args.sourceName}" was not found.`); - writeAgentJson(io, source); - return 0; - } - - if (args.command === 'sl-query') { - const semanticLayer = runtime.ports.semanticLayer; - if (!semanticLayer) throw new Error('Semantic-layer tools are not available for this project.'); - const query = await readAgentJsonFile(args.queryFile); - const maxRows = args.execute ? parseAgentMaxRows(args.maxRows) : args.maxRows; - writeAgentJson( - io, - await semanticLayer.query({ - connectionId: args.connectionId, - query: { ...query, ...(maxRows !== undefined ? { limit: maxRows } : {}) } as never, - }), - ); - return 0; - } - - if (args.command === 'wiki-search') { - const knowledge = runtime.ports.knowledge; - if (!knowledge) throw new Error('Wiki tools are not available for this project.'); - writeAgentJson(io, await knowledge.search({ userId: 'agent', query: args.query, limit: args.limit })); - return 0; - } - - if (args.command === 'wiki-read') { - const knowledge = runtime.ports.knowledge; - if (!knowledge) throw new Error('Wiki tools are not available for this project.'); - const page = await knowledge.read({ userId: 'agent', key: args.pageId }); - if (!page) throw new Error(`Wiki page "${args.pageId}" was not found.`); - writeAgentJson(io, page); - return 0; - } - - const queryExecutor = runtime.queryExecutor; - if (!queryExecutor) throw new Error('SQL execution is not available for this project.'); - const connection = runtime.project.config.connections[args.connectionId]; - if (!connection) throw new Error(`Connection "${args.connectionId}" was not found.`); - const maxRows = parseAgentMaxRows(args.maxRows); - writeAgentJson( - io, - await queryExecutor.execute({ - connectionId: args.connectionId, - projectDir: runtime.project.projectDir, - connection, - sql: await readFile(args.sqlFile, 'utf-8'), - maxRows, - }), - ); - return 0; - } catch (error) { - if (args.command === 'sl-list' && args.query && isMissingProjectConfigError(error)) { - writeAgentSlSearchReadinessError(io, missingProjectSlSearchReadiness(args.projectDir, args.query)); - return 1; - } - writeAgentJsonError(io, error instanceof Error ? error.message : String(error)); - return 1; - } -} diff --git a/packages/cli/src/cli-program.ts b/packages/cli/src/cli-program.ts index e2091bef..7d6a98f3 100644 --- a/packages/cli/src/cli-program.ts +++ b/packages/cli/src/cli-program.ts @@ -1,9 +1,9 @@ import { Command, InvalidArgumentError } from '@commander-js/extra-typings'; import type { KtxCliDeps, KtxCliIo, KtxCliPackageInfo } from './cli-runtime.js'; -import { registerAgentCommands } from './commands/agent-commands.js'; import { registerConnectionCommands } from './commands/connection-commands.js'; +import { registerIngestCommands } from './commands/ingest-commands.js'; import { registerWikiCommands } from './commands/knowledge-commands.js'; -import { registerPublicIngestCommands } from './commands/public-ingest-commands.js'; +import { registerScanCommands } from './commands/scan-commands.js'; import { registerSetupCommands } from './commands/setup-commands.js'; import { registerSlCommands } from './commands/sl-commands.js'; import { registerStatusCommands } from './commands/status-commands.js'; @@ -53,7 +53,7 @@ type CommandPathNode = CommandWithGlobalOptions & { parent?: CommandPathNode | null; }; -const PROJECT_AWARE_ROOT_COMMANDS = new Set(['setup', 'connection', 'ingest', 'wiki', 'sl', 'status']); +const PROJECT_AWARE_ROOT_COMMANDS = new Set(['setup', 'connection', 'ingest', 'wiki', 'sl', 'status', 'scan']); export interface CommandWithGlobalOptions { opts: () => object; @@ -151,7 +151,7 @@ function isProjectAwareCommand(path: string[]): boolean { const rootCommand = path[1]; if (rootCommand === 'dev') { - return path[2] !== undefined && path[2] !== 'completion' && path[2] !== 'runtime'; + return path[2] !== undefined && path[2] !== 'runtime'; } return rootCommand !== undefined && PROJECT_AWARE_ROOT_COMMANDS.has(rootCommand); } @@ -176,9 +176,6 @@ function shouldSuppressProjectDirLine(path: string[], options: Record + await (ingestDeps.ingest ?? defaultRunIngest)(ingestArgs, ingestIo), + }); + registerScanCommands(program, context); registerWikiCommands(program, context); registerSlCommands(program, context); registerStatusCommands(program, context); - registerAgentCommands(program, context); registerDevCommands(program, context); return program; diff --git a/packages/cli/src/cli-runtime.ts b/packages/cli/src/cli-runtime.ts index 8b143373..f303309a 100644 --- a/packages/cli/src/cli-runtime.ts +++ b/packages/cli/src/cli-runtime.ts @@ -2,12 +2,10 @@ import { createRequire } from 'node:module'; import type { KtxConnectionMetabaseSetupArgs } from './commands/connection-metabase-setup.js'; import type { KtxConnectionNotionArgs } from './commands/connection-notion.js'; -import type { KtxAgentArgs } from './agent.js'; import type { KtxConnectionArgs } from './connection.js'; import type { KtxDoctorArgs } from './doctor.js'; import type { KtxIngestArgs } from './ingest.js'; import type { KtxKnowledgeArgs } from './knowledge.js'; -import type { KtxPublicIngestArgs } from './public-ingest.js'; import type { KtxRuntimeArgs } from './runtime.js'; import type { KtxScanArgs } from './scan.js'; import type { KtxSetupArgs } from './setup.js'; @@ -31,13 +29,11 @@ export interface KtxCliIo { export interface KtxCliDeps { setup?: (args: KtxSetupArgs, io: KtxCliIo) => Promise; - agent?: (args: KtxAgentArgs, io: KtxCliIo) => Promise; connection?: (args: KtxConnectionArgs, io: KtxCliIo) => Promise; connectionNotion?: (args: KtxConnectionNotionArgs, io: KtxCliIo) => Promise; connectionMetabaseSetup?: (args: KtxConnectionMetabaseSetupArgs, io: KtxCliIo) => Promise; doctor?: (args: KtxDoctorArgs, io: KtxCliIo) => Promise; ingest?: (args: KtxIngestArgs, io: KtxCliIo) => Promise; - publicIngest?: (args: KtxPublicIngestArgs, io: KtxCliIo) => Promise; runtime?: (args: KtxRuntimeArgs, io: KtxCliIo) => Promise; scan?: (args: KtxScanArgs, io: KtxCliIo) => Promise; knowledge?: (args: KtxKnowledgeArgs, io: KtxCliIo) => Promise; diff --git a/packages/cli/src/command-schemas.ts b/packages/cli/src/command-schemas.ts index 1a442af7..cb11f2eb 100644 --- a/packages/cli/src/command-schemas.ts +++ b/packages/cli/src/command-schemas.ts @@ -53,35 +53,21 @@ export const slQueryCommandSchema = z.object({ command: z.literal('query'), projectDir: projectDirSchema, connectionId: z.string().min(1).optional(), - query: z.object({ - measures: z.array(z.string().min(1)).min(1), - dimensions: stringArraySchema, - filters: stringArraySchema.optional(), - segments: stringArraySchema.optional(), - order_by: z.array(orderBySchema).optional(), - limit: z.number().int().positive().optional(), - include_empty: z.literal(true).optional(), - }), + query: z + .object({ + measures: z.array(z.string().min(1)).min(1), + dimensions: stringArraySchema, + filters: stringArraySchema.optional(), + segments: stringArraySchema.optional(), + order_by: z.array(orderBySchema).optional(), + limit: z.number().int().positive().optional(), + include_empty: z.literal(true).optional(), + }) + .optional(), + queryFile: z.string().min(1).optional(), format: z.enum(['json', 'sql']), execute: z.boolean(), cliVersion: z.string().min(1), runtimeInstallPolicy: z.enum(['prompt', 'auto', 'never']), maxRows: z.number().int().positive().optional(), }); - -export const publicIngestRunCommandSchema = z.object({ - command: z.literal('run'), - projectDir: projectDirSchema, - targetConnectionId: safeConnectionIdSchema.optional(), - all: z.boolean(), - json: z.boolean(), - inputMode: z.enum(['auto', 'disabled']), -}); - -export const publicIngestReadCommandSchema = z.object({ - command: z.enum(['status', 'watch']), - projectDir: projectDirSchema, - runId: z.string().min(1).optional(), - json: z.boolean(), - inputMode: z.enum(['auto', 'disabled']), -}); diff --git a/packages/cli/src/commands/agent-commands.ts b/packages/cli/src/commands/agent-commands.ts deleted file mode 100644 index 2593991a..00000000 --- a/packages/cli/src/commands/agent-commands.ts +++ /dev/null @@ -1,149 +0,0 @@ -import { Option, type Command } from '@commander-js/extra-typings'; -import type { KtxAgentArgs } from '../agent.js'; -import type { KtxCliCommandContext } from '../cli-program.js'; -import { parsePositiveIntegerOption, resolveCommandProjectDir } from '../cli-program.js'; -import { runtimeInstallPolicyFromFlags } from '../managed-python-command.js'; - -async function runAgent(context: KtxCliCommandContext, args: KtxAgentArgs): Promise { - const runner = context.deps.agent ?? (await import('../agent.js')).runKtxAgent; - context.setExitCode(await runner(args, context.io)); -} - -function jsonOption(): Option { - return new Option('--json', 'Print JSON output').makeOptionMandatory(); -} - -export function registerAgentCommands(program: Command, context: KtxCliCommandContext): void { - const agent = program - .command('agent', { hidden: true }) - .description('Machine-readable KTX commands for coding agents') - .showHelpAfterError(); - - agent.hook('preAction', (_thisCommand, actionCommand) => { - context.writeDebug?.('agent', actionCommand); - }); - - agent - .command('tools') - .description('Print available agent-facing KTX tools') - .addOption(jsonOption()) - .action(async (_options, command) => { - await runAgent(context, { command: 'tools', projectDir: resolveCommandProjectDir(command), json: true }); - }); - - agent - .command('context') - .description('Print project context for agent planning') - .addOption(jsonOption()) - .action(async (_options, command) => { - await runAgent(context, { command: 'context', projectDir: resolveCommandProjectDir(command), json: true }); - }); - - const sl = agent.command('sl').description('Semantic-layer agent commands'); - sl.command('list') - .description('List semantic-layer sources') - .addOption(jsonOption()) - .option('--connection-id ', 'Filter by connection id') - .option('--query ', 'Search source names and descriptions') - .action(async (options: { connectionId?: string; query?: string }, command) => { - await runAgent(context, { - command: 'sl-list', - projectDir: resolveCommandProjectDir(command), - json: true, - ...(options.connectionId ? { connectionId: options.connectionId } : {}), - ...(options.query ? { query: options.query } : {}), - }); - }); - sl.command('read') - .description('Read one semantic-layer source') - .argument('') - .addOption(jsonOption()) - .option('--connection-id ', 'Connection id containing the source') - .action(async (sourceName: string, options: { connectionId?: string }, command) => { - await runAgent(context, { - command: 'sl-read', - projectDir: resolveCommandProjectDir(command), - json: true, - sourceName, - ...(options.connectionId ? { connectionId: options.connectionId } : {}), - }); - }); - sl.command('query') - .description('Run a semantic-layer query JSON file') - .addOption(jsonOption()) - .requiredOption('--connection-id ', 'Connection id for execution') - .requiredOption('--query-file ', 'JSON semantic-layer query file') - .option('--execute', 'Execute the compiled query against the connection', false) - .option('--yes', 'Install the managed Python runtime without prompting when required', false) - .option('--no-input', 'Disable interactive managed runtime installation') - .option('--max-rows ', 'Maximum rows to return when executing', parsePositiveIntegerOption) - .action( - async ( - options: { - connectionId: string; - queryFile: string; - execute: boolean; - maxRows?: number; - yes?: boolean; - input?: boolean; - }, - command, - ) => { - await runAgent(context, { - command: 'sl-query', - projectDir: resolveCommandProjectDir(command), - json: true, - connectionId: options.connectionId, - queryFile: options.queryFile, - execute: options.execute, - cliVersion: context.packageInfo.version, - runtimeInstallPolicy: runtimeInstallPolicyFromFlags(options), - ...(options.maxRows !== undefined ? { maxRows: options.maxRows } : {}), - }); - }, - ); - - const wiki = agent.command('wiki').description('KTX wiki agent commands'); - wiki - .command('search') - .description('Search KTX wiki pages') - .argument('') - .addOption(jsonOption()) - .option('--limit ', 'Maximum search results', parsePositiveIntegerOption, 10) - .action(async (query: string, options: { limit: number }, command) => { - await runAgent(context, { - command: 'wiki-search', - projectDir: resolveCommandProjectDir(command), - json: true, - query, - limit: options.limit, - }); - }); - wiki - .command('read') - .description('Read one KTX wiki page') - .argument('') - .addOption(jsonOption()) - .action(async (pageId: string, _options, command) => { - await runAgent(context, { command: 'wiki-read', projectDir: resolveCommandProjectDir(command), json: true, pageId }); - }); - - const sql = agent.command('sql').description('Safe SQL execution commands'); - sql - .command('execute') - .description('Execute read-only SQL with a row limit') - .addOption(jsonOption()) - .requiredOption('--connection-id ', 'Connection id for execution') - .requiredOption('--sql-file ', 'SQL file to execute') - .requiredOption('--max-rows ', 'Maximum rows to return', parsePositiveIntegerOption) - .action(async (options: { connectionId: string; sqlFile: string; maxRows: number }, command) => { - await runAgent(context, { - command: 'sql-execute', - projectDir: resolveCommandProjectDir(command), - json: true, - connectionId: options.connectionId, - sqlFile: options.sqlFile, - maxRows: options.maxRows, - }); - }); -} diff --git a/packages/cli/src/commands/completion-commands.ts b/packages/cli/src/commands/completion-commands.ts deleted file mode 100644 index 23c45429..00000000 --- a/packages/cli/src/commands/completion-commands.ts +++ /dev/null @@ -1,47 +0,0 @@ -import type { CommandUnknownOpts } from '@commander-js/extra-typings'; -import type { KtxCliCommandContext } from '../cli-program.js'; -import { completeCommanderInput, installZshCompletion, zshCompletionScript } from '../completion.js'; - -export function registerCompletionCommands( - program: CommandUnknownOpts, - context: KtxCliCommandContext, - completionRoot: CommandUnknownOpts = program, -): void { - program - .command('completion') - .description('Generate shell completion scripts') - .command('zsh') - .description('Generate zsh completion script') - .option('--install', 'Install zsh completion into ~/.zfunc and update ~/.zshrc', false) - .action(async (options: { install?: boolean }) => { - if (options.install === true) { - const result = await installZshCompletion(); - context.io.stdout.write(`Installed zsh completion: ${result.completionPath}\n`); - context.io.stdout.write(`Updated zsh config: ${result.zshrcPath}\n`); - context.io.stdout.write('Restart your shell or run: source ~/.zshrc\n'); - context.setExitCode(0); - return; - } - context.io.stdout.write(zshCompletionScript()); - context.setExitCode(0); - }); - - program - .command('__complete', { hidden: true }) - .description('Internal shell completion endpoint') - .requiredOption('--shell ', 'Shell requesting completions') - .requiredOption('--position ', 'Current shell word position', (value) => Number(value)) - .argument('[words...]', 'Current shell words') - .allowUnknownOption() - .allowExcessArguments() - .action((words: string[], options: { shell: string; position: number }) => { - if (options.shell !== 'zsh') { - context.setExitCode(1); - return; - } - for (const completion of completeCommanderInput(completionRoot, { position: options.position, words })) { - context.io.stdout.write(`${completion}\n`); - } - context.setExitCode(0); - }); -} diff --git a/packages/cli/src/commands/connection-commands.ts b/packages/cli/src/commands/connection-commands.ts index f3c87709..4ce75057 100644 --- a/packages/cli/src/commands/connection-commands.ts +++ b/packages/cli/src/commands/connection-commands.ts @@ -188,7 +188,7 @@ export function registerConnectionCommands(program: Command, context: KtxCliComm registerConnectionNotionCommands(connection, context); } -export function registerConnectionMappingCommands(connection: Command, context: KtxCliCommandContext): void { +function registerConnectionMappingCommands(connection: Command, context: KtxCliCommandContext): void { const mapping = connection .command('mapping') .description('Manage Metabase warehouse mappings') diff --git a/packages/cli/src/commands/connection-metabase-commands.ts b/packages/cli/src/commands/connection-metabase-commands.ts index 1a07be3a..c20b8e86 100644 --- a/packages/cli/src/commands/connection-metabase-commands.ts +++ b/packages/cli/src/commands/connection-metabase-commands.ts @@ -88,7 +88,7 @@ export function registerConnectionMetabaseCommands(connection: Command, context: ' ktx connection mapping refresh --auto-accept\n' + ' ktx connection mapping set databaseMappings =\n' + ' ktx connection mapping set-sync-enabled --enabled true\n' + - ' ktx ingest \n', + ' ktx ingest run --connection-id --adapter metabase\n', ) .option( '--map ', diff --git a/packages/cli/src/commands/connection-metabase-setup.test.ts b/packages/cli/src/commands/connection-metabase-setup.test.ts index cf7308d7..9d462bbd 100644 --- a/packages/cli/src/commands/connection-metabase-setup.test.ts +++ b/packages/cli/src/commands/connection-metabase-setup.test.ts @@ -230,7 +230,7 @@ describe('runKtxConnectionMetabaseSetup', () => { expect(io.stdout()).toContain('Connection: metabase'); expect(io.stdout()).toContain('Discovered 1 database'); - expect(io.stdout()).toContain(`ktx ingest metabase --project-dir ${projectDir}`); + expect(io.stdout()).toContain(`ktx ingest run --connection-id metabase --adapter metabase --project-dir ${projectDir}`); expect(io.stdout()).not.toContain('mb_example'); expect(io.stderr()).not.toContain('mb_example'); @@ -784,7 +784,7 @@ describe('runKtxConnectionMetabaseSetup', () => { const config = await readFile(join(projectDir, 'ktx.yaml'), 'utf-8'); expect(config).toContain('driver: metabase'); - expect(io.stderr()).toContain(`ktx ingest metabase --project-dir ${projectDir}`); + expect(io.stderr()).toContain(`ktx ingest run --connection-id metabase --adapter metabase --project-dir ${projectDir}`); const updatedProject = await loadKtxProject({ projectDir }); const store = new LocalMetabaseSourceStateReader({ dbPath: ktxLocalStateDbPath(updatedProject) }); diff --git a/packages/cli/src/commands/connection-metabase-setup.ts b/packages/cli/src/commands/connection-metabase-setup.ts index 9b5e21d7..2321ea3d 100644 --- a/packages/cli/src/commands/connection-metabase-setup.ts +++ b/packages/cli/src/commands/connection-metabase-setup.ts @@ -743,7 +743,9 @@ export async function runKtxConnectionMetabaseSetup( io.stdout.write(`Connection: ${connectionId}\n`); io.stdout.write(`Discovered ${discovered.length} ${discovered.length === 1 ? 'database' : 'databases'}\n`); - io.stdout.write(`Next: ktx ingest ${connectionId} --project-dir ${args.projectDir}\n`); + io.stdout.write( + `Next: ktx ingest run --connection-id ${connectionId} --adapter metabase --project-dir ${args.projectDir}\n`, + ); if (args.runIngest) { const ingestRunner = deps.runPublicIngest ?? runKtxPublicIngest; @@ -759,7 +761,9 @@ export async function runKtxConnectionMetabaseSetup( io, ); if (exitCode !== 0) { - io.stderr.write(`Ingest failed; re-run: ktx ingest ${connectionId} --project-dir ${args.projectDir}\n`); + io.stderr.write( + `Ingest failed; re-run: ktx ingest run --connection-id ${connectionId} --adapter metabase --project-dir ${args.projectDir}\n`, + ); return 1; } } diff --git a/packages/cli/src/commands/knowledge-commands.ts b/packages/cli/src/commands/knowledge-commands.ts index c85a118c..f8d716f7 100644 --- a/packages/cli/src/commands/knowledge-commands.ts +++ b/packages/cli/src/commands/knowledge-commands.ts @@ -1,5 +1,10 @@ import { type Command, Option } from '@commander-js/extra-typings'; -import { collectOption, type KtxCliCommandContext, resolveCommandProjectDir } from '../cli-program.js'; +import { + collectOption, + type KtxCliCommandContext, + parsePositiveIntegerOption, + resolveCommandProjectDir, +} from '../cli-program.js'; import { wikiWriteCommandSchema } from '../command-schemas.js'; import type { KtxKnowledgeArgs } from '../knowledge.js'; import { profileMark } from '../startup-profile.js'; @@ -24,12 +29,14 @@ export function registerWikiCommands(program: Command, context: KtxCliCommandCon wiki .command('list') .description('List local wiki pages') + .option('--json', 'Print JSON output', false) .option('--user-id ', 'Local user id', 'local') - .action(async (options: { userId: string }, command) => { + .action(async (options: { userId: string; json?: boolean }, command) => { await runKnowledgeArgs(context, { command: 'list', projectDir: resolveCommandProjectDir(command), userId: options.userId, + json: options.json, }); }); @@ -37,13 +44,15 @@ export function registerWikiCommands(program: Command, context: KtxCliCommandCon .command('read') .description('Read one local wiki page') .argument('', 'Wiki page key') + .option('--json', 'Print JSON output', false) .option('--user-id ', 'Local user id', 'local') - .action(async (key: string, options: { userId: string }, command) => { + .action(async (key: string, options: { userId: string; json?: boolean }, command) => { await runKnowledgeArgs(context, { command: 'read', projectDir: resolveCommandProjectDir(command), key, userId: options.userId, + json: options.json, }); }); @@ -51,13 +60,17 @@ export function registerWikiCommands(program: Command, context: KtxCliCommandCon .command('search') .description('Search local wiki pages') .argument('', 'Search query') + .option('--json', 'Print JSON output', false) .option('--user-id ', 'Local user id', 'local') - .action(async (query: string, options: { userId: string }, command) => { + .option('--limit ', 'Maximum search results', parsePositiveIntegerOption) + .action(async (query: string, options: { userId: string; json?: boolean; limit?: number }, command) => { await runKnowledgeArgs(context, { command: 'search', projectDir: resolveCommandProjectDir(command), query, userId: options.userId, + json: options.json, + ...(options.limit !== undefined ? { limit: options.limit } : {}), }); }); diff --git a/packages/cli/src/commands/public-ingest-commands.ts b/packages/cli/src/commands/public-ingest-commands.ts deleted file mode 100644 index dfe63c42..00000000 --- a/packages/cli/src/commands/public-ingest-commands.ts +++ /dev/null @@ -1,109 +0,0 @@ -import { InvalidArgumentError, type Command } from '@commander-js/extra-typings'; -import { type KtxCliCommandContext, resolveCommandProjectDir } from '../cli-program.js'; -import { publicIngestReadCommandSchema, publicIngestRunCommandSchema } from '../command-schemas.js'; -import type { KtxPublicIngestArgs, KtxPublicIngestInputMode } from '../public-ingest.js'; -import { profileMark } from '../startup-profile.js'; - -profileMark('module:commands/public-ingest-commands'); - -interface PublicIngestOptions { - all?: boolean; - json?: boolean; - input?: boolean; -} - -function inputMode(options: { input?: boolean }): KtxPublicIngestInputMode { - return options.input === false ? 'disabled' : 'auto'; -} - -async function runPublicIngestArgs(context: KtxCliCommandContext, args: KtxPublicIngestArgs): Promise { - const runner = context.deps.publicIngest ?? (await import('../public-ingest.js')).runKtxPublicIngest; - context.setExitCode(await runner(args, context.io)); -} - -function parsePublicIngestConnectionId(value: string): string { - if (value === 'run') { - throw new InvalidArgumentError('run is reserved; use ktx dev ingest run for low-level adapter syntax'); - } - return value; -} - -export function registerPublicIngestCommands(program: Command, context: KtxCliCommandContext): void { - const ingest = program - .command('ingest') - .description('Build and refresh KTX context from configured sources') - .usage('[options] [connectionId]') - .argument('[connectionId]', 'Connection id to ingest', parsePublicIngestConnectionId) - .option('--all', 'Ingest every eligible configured source', false) - .option('--json', 'Print JSON output', false) - .option('--no-input', 'Disable interactive terminal input') - .addHelpText( - 'after', - [ - '', - 'Examples:', - ' ktx ingest [options]', - ' ktx ingest --all [options]', - ' ktx ingest status [runId] [options]', - ' ktx ingest watch [runId] [options]', - '', - 'Project directory defaults to KTX_PROJECT_DIR when set, otherwise the current working directory.', - '', - ].join('\n'), - ) - .showHelpAfterError() - .hook('preAction', (_thisCommand, actionCommand) => { - context.writeDebug?.('ingest', actionCommand); - }) - .action(async (connectionId: string | undefined, _options: PublicIngestOptions, command) => { - const options = command.opts(); - if (options.all === true && connectionId) { - throw new Error('ktx ingest accepts either --all or , not both'); - } - const args = publicIngestRunCommandSchema.parse({ - command: 'run', - projectDir: resolveCommandProjectDir(command), - ...(connectionId ? { targetConnectionId: connectionId } : {}), - all: options.all === true, - json: options.json === true, - inputMode: inputMode(options), - }); - await runPublicIngestArgs(context, args); - }); - - ingest - .command('status') - .description('Print status for the latest or selected public ingest run') - .argument('[runId]', 'Public ingest run id') - .option('--json', 'Print JSON output', false) - .option('--no-input', 'Disable interactive terminal input') - .action(async (runId: string | undefined, _options: PublicIngestOptions, command) => { - const options = (command.optsWithGlobals ? command.optsWithGlobals() : command.opts()) as PublicIngestOptions; - const args = publicIngestReadCommandSchema.parse({ - command: 'status', - projectDir: resolveCommandProjectDir(command), - ...(runId ? { runId } : {}), - json: options.json === true, - inputMode: inputMode(options), - }); - await runPublicIngestArgs(context, args); - }); - - ingest - .command('watch') - .description('Open the latest or selected public ingest visual report') - .argument('[runId]', 'Public ingest run id') - .option('--json', 'Print JSON output instead of the visual report', false) - .option('--no-input', 'Disable interactive terminal input') - .action(async (runId: string | undefined, _options: PublicIngestOptions, command) => { - const options = (command.optsWithGlobals ? command.optsWithGlobals() : command.opts()) as PublicIngestOptions; - const args = publicIngestReadCommandSchema.parse({ - command: 'watch', - projectDir: resolveCommandProjectDir(command), - ...(runId ? { runId } : {}), - json: options.json === true, - inputMode: inputMode(options), - }); - await runPublicIngestArgs(context, args); - }); -} diff --git a/packages/cli/src/commands/runtime-commands.ts b/packages/cli/src/commands/runtime-commands.ts index 3ce7d9ba..cf0abb42 100644 --- a/packages/cli/src/commands/runtime-commands.ts +++ b/packages/cli/src/commands/runtime-commands.ts @@ -18,7 +18,7 @@ async function runRuntimeArgs(context: KtxCliCommandContext, args: KtxRuntimeArg export function registerRuntimeCommands(program: Command, context: KtxCliCommandContext): void { const runtime = program .command('runtime') - .description('Install, inspect, and prune the KTX-managed Python runtime') + .description('Install, start, stop, and inspect the KTX-managed Python runtime') .showHelpAfterError(); runtime @@ -64,7 +64,7 @@ export function registerRuntimeCommands(program: Command, context: KtxCliCommand runtime .command('status') - .description('Show managed Python runtime status') + .description('Show managed Python runtime status and readiness checks') .option('--json', 'Print JSON output', false) .action(async (options: { json?: boolean }) => { await runRuntimeArgs(context, { @@ -73,30 +73,4 @@ export function registerRuntimeCommands(program: Command, context: KtxCliCommand json: options.json === true, }); }); - - runtime - .command('doctor') - .description('Check managed Python runtime prerequisites and installation') - .option('--json', 'Print JSON output', false) - .action(async (options: { json?: boolean }) => { - await runRuntimeArgs(context, { - command: 'doctor', - cliVersion: context.packageInfo.version, - json: options.json === true, - }); - }); - - runtime - .command('prune') - .description('Remove stale managed Python runtimes for older CLI versions') - .option('--dry-run', 'List stale runtimes without deleting them', false) - .option('--yes', 'Confirm deletion of stale runtime directories', false) - .action(async (options: { dryRun?: boolean; yes?: boolean }) => { - await runRuntimeArgs(context, { - command: 'prune', - cliVersion: context.packageInfo.version, - dryRun: options.dryRun === true, - yes: options.yes === true, - }); - }); } diff --git a/packages/cli/src/commands/scan-commands.ts b/packages/cli/src/commands/scan-commands.ts index fc30fafa..2c19bcdf 100644 --- a/packages/cli/src/commands/scan-commands.ts +++ b/packages/cli/src/commands/scan-commands.ts @@ -1,5 +1,5 @@ -import { type Command, InvalidArgumentError, Option } from '@commander-js/extra-typings'; -import { type KtxCliCommandContext, parsePositiveIntegerOption, resolveCommandProjectDir } from '../cli-program.js'; +import { type Command, InvalidArgumentError } from '@commander-js/extra-typings'; +import { type KtxCliCommandContext, resolveCommandProjectDir } from '../cli-program.js'; import { runtimeInstallPolicyFromFlags } from '../managed-python-command.js'; import type { KtxScanArgs } from '../scan.js'; import { profileMark } from '../startup-profile.js'; @@ -13,6 +13,16 @@ async function runScanArgs(context: KtxCliCommandContext, args: KtxScanArgs): Pr type KtxScanModeOption = Extract['mode']; +const REMOVED_SCAN_SUBCOMMAND_NAMES = new Set([ + 'status', + 'report', + 'relationships', + 'relationship-apply', + 'relationship-feedback', + 'relationship-calibration', + 'relationship-thresholds', +]); + function parseScanModeOption(value: string): KtxScanModeOption { if (value === 'structural' || value === 'enriched' || value === 'relationships') { return value; @@ -20,82 +30,18 @@ function parseScanModeOption(value: string): KtxScanModeOption { throw new InvalidArgumentError('Allowed choices are structural, enriched, relationships'); } -type KtxRelationshipStatusOption = Extract['status']; -type KtxRelationshipFeedbackDecisionOption = Extract['decision']; - -function parseRelationshipStatusOption(value: string): KtxRelationshipStatusOption { - if (value === 'accepted' || value === 'review' || value === 'rejected' || value === 'skipped' || value === 'all') { - return value; - } - throw new InvalidArgumentError('Allowed choices are accepted, review, rejected, skipped, all'); -} - -function parseRelationshipFeedbackDecisionOption(value: string): KtxRelationshipFeedbackDecisionOption { - if (value === 'accepted' || value === 'rejected' || value === 'all') { - return value; - } - throw new InvalidArgumentError('Allowed choices are accepted, rejected, all'); -} - -function parseNonEmptyOption(value: string): string { - if (value.trim().length === 0) { - throw new InvalidArgumentError('must not be empty'); +function parseConnectionId(value: string): string { + if (REMOVED_SCAN_SUBCOMMAND_NAMES.has(value)) { + throw new InvalidArgumentError(`"${value}" is not a scan connection id`); } return value; } -function parseRelationshipCalibrationThreshold(value: string): number { - const parsed = Number(value); - if (Number.isFinite(parsed) && parsed >= 0 && parsed <= 1) { - return parsed; - } - throw new InvalidArgumentError('Allowed range is 0 through 1'); -} - -function relationshipDecisionArgs(options: { - accept?: string; - reject?: string; - reviewer?: string; - note?: string; - json?: boolean; -}): Pick< - Extract, - 'candidateId' | 'decision' | 'reviewer' | 'note' | 'json' -> | null { - const decisionCount = [options.accept !== undefined, options.reject !== undefined].filter(Boolean).length; - if (decisionCount > 1) { - throw new Error('Only one relationship review decision option can be used: --accept and --reject conflict'); - } - if (options.accept !== undefined) { - return { - candidateId: options.accept, - decision: 'accepted', - reviewer: options.reviewer ?? 'ktx', - note: options.note ?? null, - json: options.json === true, - }; - } - if (options.reject !== undefined) { - return { - candidateId: options.reject, - decision: 'rejected', - reviewer: options.reviewer ?? 'ktx', - note: options.note ?? null, - json: options.json === true, - }; - } - return null; -} - -function collectRelationshipCandidateOption(value: string, previous: string[]): string[] { - return [...previous, parseNonEmptyOption(value)]; -} - export function registerScanCommands(program: Command, context: KtxCliCommandContext): void { - const scan = program + program .command('scan') - .description('Run or inspect standalone connection scans') - .argument('[connectionId]', 'KTX connection id to scan') + .description('Run a standalone connection scan') + .argument('', 'KTX connection id to scan', parseConnectionId) .option( '--mode ', 'Scan mode: structural, enriched, relationships (default: structural)', @@ -113,13 +59,7 @@ export function registerScanCommands(program: Command, context: KtxCliCommandCon .hook('preAction', (_thisCommand, actionCommand) => { context.writeDebug?.('scan', actionCommand); }) - .action(async (connectionId: string | undefined, options, command) => { - if (!connectionId) { - scan.outputHelp(); - context.io.stderr.write('ktx dev scan requires or a subcommand\n'); - context.setExitCode(1); - return; - } + .action(async (connectionId: string, options, command) => { const mode = options.mode ?? 'structural'; await runScanArgs(context, { command: 'run', @@ -133,226 +73,4 @@ export function registerScanCommands(program: Command, context: KtxCliCommandCon runtimeInstallPolicy: runtimeInstallPolicyFromFlags(options), }); }); - - scan - .command('status') - .description('Print status for a local scan run') - .argument('', 'Local scan run id') - .addHelpText( - 'after', - '\n--project-dir is inherited from `ktx dev scan` (default: KTX_PROJECT_DIR or current working directory).\n', - ) - .action(async (runId: string, _options: unknown, command) => { - await runScanArgs(context, { - command: 'status', - projectDir: resolveCommandProjectDir(command), - runId, - }); - }); - - scan - .command('report') - .description('Print a local scan report') - .argument('', 'Local scan run id') - .option('--json', 'Print the raw scan report JSON', false) - .addHelpText( - 'after', - '\n--project-dir is inherited from `ktx dev scan` (default: KTX_PROJECT_DIR or current working directory).\n', - ) - .action(async (runId: string, options, command) => { - await runScanArgs(context, { - command: 'report', - projectDir: resolveCommandProjectDir(command), - runId, - json: options.json === true, - }); - }); - - scan - .command('relationships') - .description('Print relationship artifacts for a local scan run') - .argument('', 'Local scan run id') - .option( - '--status ', - 'Relationship status: accepted, review, rejected, skipped, all', - parseRelationshipStatusOption, - 'review', - ) - .option('--limit ', 'Maximum relationships to print per status', parsePositiveIntegerOption, 25) - .addOption( - new Option('--accept ', 'Record a reviewer accepted decision for a relationship candidate') - .argParser(parseNonEmptyOption) - .conflicts('reject'), - ) - .addOption( - new Option('--reject ', 'Record a reviewer rejected decision for a relationship candidate') - .argParser(parseNonEmptyOption) - .conflicts('accept'), - ) - .option('--note ', 'Attach a note when recording a relationship review decision') - .option('--reviewer ', 'Reviewer name for a relationship review decision') - .option('--json', 'Print relationship artifacts as JSON', false) - .addHelpText( - 'after', - '\n--project-dir is inherited from `ktx dev scan` (default: KTX_PROJECT_DIR or current working directory).\n', - ) - .action(async (runId: string, options, command) => { - const decision = relationshipDecisionArgs(options); - if (decision) { - await runScanArgs(context, { - command: 'relationshipDecision', - projectDir: resolveCommandProjectDir(command), - runId, - candidateId: decision.candidateId, - decision: decision.decision, - reviewer: decision.reviewer, - note: decision.note, - json: decision.json, - }); - return; - } - await runScanArgs(context, { - command: 'relationships', - projectDir: resolveCommandProjectDir(command), - runId, - status: options.status, - json: options.json === true, - limit: options.limit, - }); - }); - - scan - .command('relationship-apply') - .description('Apply accepted relationship review decisions as manual manifest joins') - .argument('', 'Local scan run id') - .option('--all-accepted', 'Apply all accepted relationship review decisions for the scan run', false) - .option( - '--candidate ', - 'Apply one accepted relationship review decision', - collectRelationshipCandidateOption, - [], - ) - .option('--dry-run', 'Preview relationships that would be written without rewriting manifest shards', false) - .option('--json', 'Print the apply result as JSON', false) - .addHelpText( - 'after', - '\n--project-dir is inherited from `ktx dev scan` (default: KTX_PROJECT_DIR or current working directory).\n', - ) - .action(async (runId: string, options, command) => { - const parentOptions = command.parent?.opts() as { dryRun?: boolean } | undefined; - await runScanArgs(context, { - command: 'relationshipApply', - projectDir: resolveCommandProjectDir(command), - runId, - applyAllAccepted: options.allAccepted === true, - candidateIds: options.candidate, - dryRun: options.dryRun === true || parentOptions?.dryRun === true, - json: options.json === true, - }); - }); - - scan - .command('relationship-feedback') - .description('Export persisted relationship review decisions as calibration labels') - .option('--connection ', 'Only export labels for one KTX connection') - .option( - '--decision ', - 'Relationship feedback decision: accepted, rejected, all', - parseRelationshipFeedbackDecisionOption, - 'all', - ) - .addOption(new Option('--json', 'Print the export as JSON').default(false).conflicts('jsonl')) - .addOption(new Option('--jsonl', 'Print labels as newline-delimited JSON').default(false).conflicts('json')) - .addHelpText( - 'after', - '\n--project-dir is inherited from `ktx dev scan` (default: KTX_PROJECT_DIR or current working directory).\n', - ) - .action(async (options, command) => { - await runScanArgs(context, { - command: 'relationshipFeedback', - projectDir: resolveCommandProjectDir(command), - connectionId: options.connection ?? null, - decision: options.decision, - json: options.json === true, - jsonl: options.jsonl === true, - }); - }); - - scan - .command('relationship-calibration') - .description('Summarize relationship feedback labels against current score thresholds') - .option('--connection ', 'Only calibrate labels for one KTX connection') - .option( - '--decision ', - 'Relationship feedback decision: accepted, rejected, all', - parseRelationshipFeedbackDecisionOption, - 'all', - ) - .option( - '--accept-threshold ', - 'Score threshold treated as predicted accepted', - parseRelationshipCalibrationThreshold, - 0.85, - ) - .option( - '--review-threshold ', - 'Score threshold treated as predicted review', - parseRelationshipCalibrationThreshold, - 0.55, - ) - .option('--json', 'Print the calibration report as JSON', false) - .addHelpText( - 'after', - '\n--project-dir is inherited from `ktx dev scan` (default: KTX_PROJECT_DIR or current working directory).\n', - ) - .action(async (options, command) => { - await runScanArgs(context, { - command: 'relationshipCalibration', - projectDir: resolveCommandProjectDir(command), - connectionId: options.connection ?? null, - decision: options.decision, - acceptThreshold: options.acceptThreshold, - reviewThreshold: options.reviewThreshold, - json: options.json === true, - }); - }); - - scan - .command('relationship-thresholds') - .description('Evaluate relationship feedback labels for offline threshold advice') - .option('--connection ', 'Only evaluate labels for one KTX connection') - .option( - '--min-total-labels ', - 'Minimum scored labels before advice can be ready', - parsePositiveIntegerOption, - 20, - ) - .option( - '--min-accepted-labels ', - 'Minimum accepted labels before advice can be ready', - parsePositiveIntegerOption, - 5, - ) - .option( - '--min-rejected-labels ', - 'Minimum rejected labels before advice can be ready', - parsePositiveIntegerOption, - 5, - ) - .option('--json', 'Print the threshold advice report as JSON', false) - .addHelpText( - 'after', - '\n--project-dir is inherited from `ktx dev scan` (default: KTX_PROJECT_DIR or current working directory).\n', - ) - .action(async (options, command) => { - await runScanArgs(context, { - command: 'relationshipThresholds', - projectDir: resolveCommandProjectDir(command), - connectionId: options.connection ?? null, - minTotalLabels: options.minTotalLabels, - minAcceptedLabels: options.minAcceptedLabels, - minRejectedLabels: options.minRejectedLabels, - json: options.json === true, - }); - }); } diff --git a/packages/cli/src/commands/sl-commands.ts b/packages/cli/src/commands/sl-commands.ts index 36d75fac..e1b985a3 100644 --- a/packages/cli/src/commands/sl-commands.ts +++ b/packages/cli/src/commands/sl-commands.ts @@ -51,6 +51,7 @@ export function registerSlCommands(program: Command, context: KtxCliCommandConte sl.command('list') .description('List semantic-layer sources') .option('--connection-id ', 'KTX connection id') + .option('--query ', 'Search source names and descriptions') .addOption( new Option('--output ', 'Output mode: pretty (default in TTY), plain (TSV), or json').choices([ 'pretty', @@ -59,26 +60,34 @@ export function registerSlCommands(program: Command, context: KtxCliCommandConte ]), ) .option('--json', 'Shortcut for --output=json (overrides --output)', false) - .action(async (options: { connectionId?: string; output?: 'pretty' | 'plain' | 'json'; json?: boolean }, command) => { + .action( + async ( + options: { connectionId?: string; query?: string; output?: 'pretty' | 'plain' | 'json'; json?: boolean }, + command, + ) => { await runSlArgs(context, { command: 'list', projectDir: resolveCommandProjectDir(command), connectionId: options.connectionId, + query: options.query, output: options.output, json: options.json, }); - }); + }, + ); sl.command('read') .description('Read a semantic-layer source') .argument('', 'Semantic-layer source name') .requiredOption('--connection-id ', 'KTX connection id') - .action(async (sourceName: string, options: { connectionId: string }, command) => { + .option('--json', 'Print JSON output', false) + .action(async (sourceName: string, options: { connectionId: string; json?: boolean }, command) => { await runSlArgs(context, { command: 'read', projectDir: resolveCommandProjectDir(command), connectionId: options.connectionId, sourceName, + json: options.json, }); }); @@ -113,6 +122,7 @@ export function registerSlCommands(program: Command, context: KtxCliCommandConte sl.command('query') .description('Compile or execute a semantic-layer query') .option('--connection-id ', 'KTX connection id') + .option('--query-file ', 'JSON semantic-layer query file') .option('--measure ', 'Measure to query; repeatable', collectOption, []) .option('--dimension ', 'Dimension to include; repeatable', collectOption, []) .option('--filter ', 'Filter expression; repeatable', collectOption, []) @@ -126,22 +136,26 @@ export function registerSlCommands(program: Command, context: KtxCliCommandConte .option('--no-input', 'Disable interactive managed runtime installation') .option('--max-rows ', 'Maximum rows to return when executing', parsePositiveIntegerOption) .action(async (options, command) => { - if (options.measure.length === 0) { + if (options.measure.length === 0 && !options.queryFile) { throw new Error('sl query requires at least one --measure'); } const args = slQueryCommandSchema.parse({ command: 'query', projectDir: resolveCommandProjectDir(command), connectionId: options.connectionId, - query: { - measures: options.measure, - dimensions: options.dimension, - ...(options.filter.length > 0 ? { filters: options.filter } : {}), - ...(options.segment.length > 0 ? { segments: options.segment } : {}), - ...(options.orderBy.length > 0 ? { order_by: options.orderBy } : {}), - ...(options.limit !== undefined ? { limit: options.limit } : {}), - ...(options.includeEmpty === true ? { include_empty: true } : {}), - }, + ...(options.queryFile + ? { queryFile: options.queryFile } + : { + query: { + measures: options.measure, + dimensions: options.dimension, + ...(options.filter.length > 0 ? { filters: options.filter } : {}), + ...(options.segment.length > 0 ? { segments: options.segment } : {}), + ...(options.orderBy.length > 0 ? { order_by: options.orderBy } : {}), + ...(options.limit !== undefined ? { limit: options.limit } : {}), + ...(options.includeEmpty === true ? { include_empty: true } : {}), + }, + }), format: options.format, execute: options.execute === true, cliVersion: context.packageInfo.version, diff --git a/packages/cli/src/completion.ts b/packages/cli/src/completion.ts deleted file mode 100644 index 10e787f6..00000000 --- a/packages/cli/src/completion.ts +++ /dev/null @@ -1,353 +0,0 @@ -import { mkdir, readFile, writeFile } from 'node:fs/promises'; -import { homedir } from 'node:os'; -import { dirname, join } from 'node:path'; -import type { CommandUnknownOpts, Option } from '@commander-js/extra-typings'; - -export interface CompletionRequest { - position: number; - words: string[]; -} - -interface CompletionCandidate { - value: string; - description?: string; -} - -interface CommandWithHiddenFlag extends CommandUnknownOpts { - _hidden?: boolean; -} - -interface ResolveState { - command: CommandUnknownOpts; - pendingOption?: Option; - positionalIndex: number; -} - -export interface ZshCompletionInstallResult { - completionPath: string; - zshrcPath: string; -} - -const KTX_COMPLETION_BLOCK_START = '# >>> ktx completion >>>'; -const KTX_COMPLETION_BLOCK_END = '# <<< ktx completion <<<'; -const KTX_COMPLETION_BLOCK_PATTERN = new RegExp( - `\\n?${escapeRegExp(KTX_COMPLETION_BLOCK_START)}[\\s\\S]*?${escapeRegExp(KTX_COMPLETION_BLOCK_END)}\\n?`, - 'g', -); - -export function zshCompletionScript(): string { - const zshWords = '$' + '{words[@]}'; - const zshCompletionCapture = [ - '$', - `{(@f)$("${'$'}{ktx_completion_command[@]}" dev __complete --shell zsh --position "$CURRENT" -- "${zshWords}" 2>/dev/null)}`, - ].join(''); - const zshCompletionsCount = '$' + '{#completions[@]}'; - const zshCompletionCommand = '$' + '(eval "print -r -- $' + '{KTX_COMPLETION_COMMAND:-ktx}")'; - - return [ - '#compdef ktx', - '', - '_ktx() {', - ' local -a completions', - ' local -a ktx_completion_command', - ` ktx_completion_command=("\${(@z)${zshCompletionCommand}}")`, - ` completions=("${zshCompletionCapture}")`, - ` if (( ${zshCompletionsCount} )); then`, - " _describe 'ktx completions' completions", - ' else', - ' _files', - ' fi', - '}', - '', - 'compdef _ktx ktx', - '', - ].join('\n'); -} - -export async function installZshCompletion(): Promise { - const homeDir = process.env.HOME || homedir(); - const zshConfigDir = process.env.ZDOTDIR || homeDir; - const completionDir = join(homeDir, '.zfunc'); - const completionPath = join(completionDir, '_ktx'); - const zshrcPath = join(zshConfigDir, '.zshrc'); - - await mkdir(completionDir, { recursive: true }); - await mkdir(dirname(zshrcPath), { recursive: true }); - await writeFile(completionPath, zshCompletionScript(), 'utf-8'); - - const existingZshrc = await readOptionalTextFile(zshrcPath); - const nextZshrc = updateZshrcCompletionBlock(existingZshrc); - await writeFile(zshrcPath, nextZshrc, 'utf-8'); - - return { completionPath, zshrcPath }; -} - -export function completeCommanderInput(program: CommandUnknownOpts, request: CompletionRequest): string[] { - const words = completionWordsForPosition(request.words, request.position); - const tokens = stripProgramName(program, words); - const current = tokens.at(-1) ?? ''; - const previous = tokens.slice(0, -1); - const state = resolveCommandState(program, previous); - - return candidatesForState(state, current).map(formatZshCandidate); -} - -function completionWordsForPosition(words: string[], position: number): string[] { - if (!Number.isInteger(position) || position < 1) { - return words; - } - return words.slice(0, position); -} - -function stripProgramName(program: CommandUnknownOpts, words: string[]): string[] { - const [first, ...rest] = words; - if (!first) { - return []; - } - return first === program.name() || first.endsWith(`/${program.name()}`) ? rest : words; -} - -function resolveCommandState(program: CommandUnknownOpts, tokens: string[]): ResolveState { - let command = program; - let positionalIndex = 0; - let pendingOption: Option | undefined; - let positionalOnly = false; - - for (let index = 0; index < tokens.length; index += 1) { - const token = tokens[index]; - if (pendingOption) { - pendingOption = undefined; - continue; - } - - if (token === '--') { - positionalOnly = true; - continue; - } - - if (!positionalOnly && token.startsWith('-')) { - const option = findOption(command, optionNameFromToken(token)); - if (option && !token.includes('=') && optionTakesValue(option)) { - if (index === tokens.length - 1) { - pendingOption = option; - } else if (option.required || !tokens[index + 1]?.startsWith('-')) { - index += 1; - } - } - continue; - } - - const child = findVisibleSubcommand(command, token); - if (child) { - command = child; - positionalIndex = 0; - continue; - } - - positionalIndex += 1; - } - - return { command, pendingOption, positionalIndex }; -} - -function candidatesForState(state: ResolveState, current: string): CompletionCandidate[] { - const optionValue = splitOptionValueToken(current); - if (optionValue) { - const option = findOption(state.command, optionValue.optionName); - return choiceCandidates(option?.argChoices, optionValue.valuePrefix, optionValue.optionPrefix); - } - - if (state.pendingOption) { - return choiceCandidates(state.pendingOption.argChoices, current); - } - - if (current.startsWith('-')) { - return visibleOptions(state.command) - .map(optionCandidate) - .filter((candidate) => candidate.value.startsWith(current)); - } - - const commandCandidates = visibleSubcommands(state.command) - .map(commandCandidate) - .filter((candidate) => candidate.value.startsWith(current)); - const argument = state.command.registeredArguments[state.positionalIndex]; - return [...commandCandidates, ...choiceCandidates(argument?.argChoices, current)]; -} - -function visibleSubcommands(command: CommandUnknownOpts): CommandUnknownOpts[] { - return command.commands.filter((subcommand) => (subcommand as CommandWithHiddenFlag)._hidden !== true); -} - -function findVisibleSubcommand(command: CommandUnknownOpts, name: string): CommandUnknownOpts | undefined { - return visibleSubcommands(command).find( - (subcommand) => subcommand.name() === name || subcommand.aliases().includes(name), - ); -} - -function visibleOptions(command: CommandUnknownOpts): Option[] { - const options: Option[] = []; - const seen = new Set(); - for (const current of commandChain(command)) { - for (const option of current.options) { - if (option.hidden) { - continue; - } - const key = option.long ?? option.short ?? option.flags; - if (seen.has(key)) { - continue; - } - seen.add(key); - options.push(option); - } - } - return options; -} - -function commandChain(command: CommandUnknownOpts): CommandUnknownOpts[] { - const chain: CommandUnknownOpts[] = []; - let current: CommandUnknownOpts | null = command; - while (current) { - chain.unshift(current); - current = current.parent; - } - return chain; -} - -function findOption(command: CommandUnknownOpts, name: string): Option | undefined { - return visibleOptions(command).find((option) => option.long === name || option.short === name); -} - -function optionTakesValue(option: Option): boolean { - return option.required || option.optional; -} - -function optionNameFromToken(token: string): string { - return token.split('=', 1)[0] ?? token; -} - -function splitOptionValueToken( - token: string, -): { optionName: string; optionPrefix: string; valuePrefix: string } | null { - const separatorIndex = token.indexOf('='); - if (!token.startsWith('-') || separatorIndex < 0) { - return null; - } - return { - optionName: token.slice(0, separatorIndex), - optionPrefix: token.slice(0, separatorIndex + 1), - valuePrefix: token.slice(separatorIndex + 1), - }; -} - -function commandCandidate(command: CommandUnknownOpts): CompletionCandidate { - return { - value: command.name(), - description: command.summary() || command.description(), - }; -} - -function optionCandidate(option: Option): CompletionCandidate { - return { - value: option.long ?? option.short ?? option.flags, - description: option.description, - }; -} - -function choiceCandidates( - choices: readonly string[] | undefined, - prefix: string, - completionPrefix = '', -): CompletionCandidate[] { - return (choices ?? []) - .filter((choice) => choice.startsWith(prefix)) - .map((choice) => ({ value: `${completionPrefix}${choice}` })); -} - -function formatZshCandidate(candidate: CompletionCandidate): string { - if (!candidate.description) { - return escapeZshCompletion(candidate.value); - } - return `${escapeZshCompletion(candidate.value)}:${escapeZshDescription(candidate.description)}`; -} - -function escapeZshCompletion(value: string): string { - return value.replace(/\\/g, '\\\\').replace(/:/g, '\\:'); -} - -function escapeZshDescription(value: string): string { - return value.replace(/\s+/g, ' ').replace(/\\/g, '\\\\').replace(/:/g, '\\:').trim(); -} - -async function readOptionalTextFile(path: string): Promise { - try { - return await readFile(path, 'utf-8'); - } catch (error) { - if (isNodeError(error) && error.code === 'ENOENT') { - return ''; - } - throw error; - } -} - -function updateZshrcCompletionBlock(contents: string): string { - const withoutManagedBlock = contents.replace(KTX_COMPLETION_BLOCK_PATTERN, normalizeTrailingNewline); - const hasCompinit = /^.*\bcompinit\b.*$/m.test(withoutManagedBlock); - const block = zshrcCompletionBlock({ includeCompinit: !hasCompinit }); - - if (!hasCompinit) { - return appendBlock(withoutManagedBlock, block); - } - - const compinitMatch = /^.*\bcompinit\b.*$/m.exec(withoutManagedBlock); - if (!compinitMatch || compinitMatch.index === undefined) { - return appendBlock(withoutManagedBlock, block); - } - - return [ - withoutManagedBlock.slice(0, compinitMatch.index), - block, - '\n', - withoutManagedBlock.slice(compinitMatch.index), - ].join(''); -} - -function zshrcCompletionBlock(options: { includeCompinit: boolean }): string { - return [ - KTX_COMPLETION_BLOCK_START, - '_ktx_completion_command() {', - ' local dir="$PWD"', - ' while [[ "$dir" != "/" ]]; do', - ` if [[ -f "$dir/package.json" ]] && command grep -q '"name": "ktx-workspace"' "$dir/package.json" 2>/dev/null; then`, - ' print -r -- "node $dir/scripts/run-ktx.mjs --"', - ' return', - ' fi', - ' dir="' + '$' + '{dir:h}"', - ' done', - ' print -r -- "ktx"', - '}', - "export KTX_COMPLETION_COMMAND='$(_ktx_completion_command)'", - 'setopt complete_aliases', - 'fpath=("$HOME/.zfunc" $fpath)', - ...(options.includeCompinit ? ['autoload -Uz compinit', 'compinit'] : []), - KTX_COMPLETION_BLOCK_END, - ].join('\n'); -} - -function appendBlock(contents: string, block: string): string { - if (!contents.trim()) { - return `${block}\n`; - } - return `${contents.replace(/\s*$/, '\n\n')}${block}\n`; -} - -function normalizeTrailingNewline(match: string): string { - return match.startsWith('\n') || match.endsWith('\n') ? '\n' : ''; -} - -function escapeRegExp(value: string): string { - return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); -} - -function isNodeError(error: unknown): error is NodeJS.ErrnoException { - return error instanceof Error && 'code' in error; -} diff --git a/packages/cli/src/connection.test.ts b/packages/cli/src/connection.test.ts index 04c73cf1..4b6cacf1 100644 --- a/packages/cli/src/connection.test.ts +++ b/packages/cli/src/connection.test.ts @@ -310,8 +310,8 @@ describe('runKtxConnection', () => { expect(io.stdout()).toContain('Mappings:'); expect(io.stdout()).toContain('1 -> [unmapped]'); expect(io.stdout()).toContain('Next:'); - expect(io.stdout()).toContain('ktx ingest prod-metabase'); - expect(io.stdout()).toContain('ktx dev mapping'); + expect(io.stdout()).toContain('ktx ingest run --connection-id prod-metabase --adapter '); + expect(io.stdout()).toContain('ktx connection mapping'); expect(io.stderr()).toBe(''); }); diff --git a/packages/cli/src/connection.ts b/packages/cli/src/connection.ts index 1dde60ac..b199239a 100644 --- a/packages/cli/src/connection.ts +++ b/packages/cli/src/connection.ts @@ -326,8 +326,8 @@ async function runPublicConnectionMap( io.stdout.write('\nMappings:\n'); io.stdout.write(listIo.stdoutText().trim() ? listIo.stdoutText() : 'No mappings found.\n'); io.stdout.write('\nNext:\n'); - io.stdout.write(` ktx ingest ${args.sourceConnectionId}\n`); - io.stdout.write(` ktx dev mapping list ${args.sourceConnectionId}\n`); + io.stdout.write(` ktx ingest run --connection-id ${args.sourceConnectionId} --adapter \n`); + io.stdout.write(` ktx connection mapping list ${args.sourceConnectionId}\n`); return 0; } diff --git a/packages/cli/src/dev.test.ts b/packages/cli/src/dev.test.ts index c7894a9e..fe75d1af 100644 --- a/packages/cli/src/dev.test.ts +++ b/packages/cli/src/dev.test.ts @@ -29,11 +29,14 @@ describe('dev Commander tree', () => { await expect(runKtxCli(['dev', '--help'], testIo.io)).resolves.toBe(0); expect(testIo.stdout()).toContain('Usage: ktx dev [options] [command]'); - for (const command of ['init', 'runtime', 'scan', 'ingest', 'mapping']) { + for (const command of ['init', 'runtime']) { expect(testIo.stdout()).toContain(command); } for (const removed of [ 'doctor', + 'scan', + 'ingest', + 'mapping', 'knowledge', 'model', 'replay', @@ -102,6 +105,13 @@ describe('dev Commander tree', () => { it('rejects removed dev command groups', async () => { for (const argv of [ ['dev', 'doctor', 'setup'], + ['dev', 'runtime', 'doctor'], + ['dev', 'runtime', 'prune', '--dry-run'], + ['dev', 'scan', 'warehouse'], + ['dev', 'ingest', 'run'], + ['dev', 'mapping', 'list'], + ['dev', 'completion', 'zsh'], + ['dev', '__complete', '--shell', 'zsh', '--position', '2', '--', 'ktx', ''], ['dev', 'knowledge', 'list'], ['dev', 'model', 'list'], ['dev', 'artifacts'], @@ -117,90 +127,15 @@ describe('dev Commander tree', () => { it.each([ { argv: ['dev', 'runtime', '--help'], - expected: ['Usage: ktx dev runtime', 'install', 'start', 'stop', 'status', 'doctor', 'prune'], + expected: ['Usage: ktx dev runtime', 'install', 'start', 'stop', 'status'], }, { - argv: ['dev', 'scan', '--help'], - expected: [ - 'Usage: ktx dev scan', - '--mode ', - 'structural', - 'relationships', - '--dry-run', - 'status', - 'report', - 'relationships', - 'relationship-apply', - 'relationship-feedback', - 'relationship-calibration', - 'relationship-thresholds', - ], + argv: ['scan', '--help'], + expected: ['Usage: ktx scan [options] ', '--mode ', 'structural', 'relationships', '--dry-run'], }, { - argv: ['dev', 'scan', 'report', '--help'], - expected: ['Usage: ktx dev scan report [options] ', '', '--json'], - }, - { - argv: ['dev', 'scan', 'relationships', '--help'], - expected: [ - 'Usage: ktx dev scan relationships [options] ', - '--status ', - '--limit ', - '--accept ', - '--reject ', - '--note ', - '--reviewer ', - '--json', - ], - }, - { - argv: ['dev', 'scan', 'relationship-apply', '--help'], - expected: [ - 'Usage: ktx dev scan relationship-apply [options] ', - '--all-accepted', - '--candidate ', - '--dry-run', - ], - }, - { - argv: ['dev', 'scan', 'relationship-thresholds', '--help'], - expected: [ - 'Usage: ktx dev scan relationship-thresholds [options]', - '--connection ', - '--min-total-labels ', - '--min-accepted-labels ', - '--min-rejected-labels ', - '--json', - ], - }, - { - argv: ['dev', 'scan', 'relationship-feedback', '--help'], - expected: [ - 'Usage: ktx dev scan relationship-feedback [options]', - '--connection ', - '--decision ', - '--json', - '--jsonl', - ], - }, - { - argv: ['dev', 'scan', 'relationship-calibration', '--help'], - expected: [ - 'Usage: ktx dev scan relationship-calibration [options]', - '--connection ', - '--decision ', - '--accept-threshold ', - '--review-threshold ', - '--json', - ], - }, - { - argv: ['dev', 'ingest', 'run', '--help'], - expected: ['Usage: ktx dev ingest run [options]', '--connection-id ', '--adapter '], - }, - { - argv: ['dev', 'mapping', 'sync-state', 'set', '--help'], - expected: ['Usage: ktx dev mapping sync-state set [options] ', '--mode '], + argv: ['ingest', 'run', '--help'], + expected: ['Usage: ktx ingest run [options]', '--connection-id ', '--adapter '], }, ])('prints generated nested help for $argv', async ({ argv, expected }) => { const io = makeIo(); @@ -213,18 +148,22 @@ describe('dev Commander tree', () => { for (const text of expected) { expect(io.stdout()).toContain(text); } + if (argv.join(' ') === 'dev runtime --help') { + expect(io.stdout()).not.toContain('prune'); + expect(io.stdout()).not.toContain('doctor'); + } expect(io.stderr()).toBe(''); expect(doctor).not.toHaveBeenCalled(); expect(ingest).not.toHaveBeenCalled(); expect(scan).not.toHaveBeenCalled(); }); - it('dispatches dev scan through Commander with injected dependencies', async () => { + it('dispatches top-level scan through Commander with injected dependencies', async () => { const scanIo = makeIo(); const scan = vi.fn(async () => 0); await expect( - runKtxCli(['dev', 'scan', 'warehouse', '--project-dir', '/tmp/project', '--dry-run'], scanIo.io, { scan }), + runKtxCli(['scan', 'warehouse', '--project-dir', '/tmp/project', '--dry-run'], scanIo.io, { scan }), ).resolves.toBe(0); expect(scan).toHaveBeenCalledWith( @@ -244,12 +183,12 @@ describe('dev Commander tree', () => { expect(scanIo.stderr()).toBe('Project: /tmp/project\n'); }); - it('dispatches dev scan --mode relationships through Commander', async () => { + it('dispatches top-level scan --mode relationships through Commander', async () => { const io = makeIo(); const scan = vi.fn(async () => 0); await expect( - runKtxCli(['dev', 'scan', 'warehouse', '--project-dir', '/tmp/project', '--mode', 'relationships'], io.io, { + runKtxCli(['scan', 'warehouse', '--project-dir', '/tmp/project', '--mode', 'relationships'], io.io, { scan, }), ).resolves.toBe(0); @@ -275,375 +214,53 @@ describe('dev Commander tree', () => { const io = makeIo(); const scan = vi.fn(async () => 0); - await expect(runKtxCli(['dev', 'scan', 'warehouse', option], io.io, { scan })).resolves.toBe(1); + await expect(runKtxCli(['scan', 'warehouse', option], io.io, { scan })).resolves.toBe(1); expect(scan).not.toHaveBeenCalled(); expect(io.stderr()).toContain(`unknown option '${option}'`); }); - it('rejects dev scan without a connection id or subcommand', async () => { + it('rejects scan without a connection id', async () => { const io = makeIo(); const scan = vi.fn(async () => 0); - await expect(runKtxCli(['dev', 'scan', '--dry-run'], io.io, { scan })).resolves.toBe(1); + await expect(runKtxCli(['scan', '--dry-run'], io.io, { scan })).resolves.toBe(1); expect(scan).not.toHaveBeenCalled(); - expect(io.stdout()).toContain('Usage: ktx dev scan'); - expect(io.stderr()).toContain('ktx dev scan requires or a subcommand'); + expect(io.stderr()).toMatch(/missing required argument/i); }); it('rejects invalid scan modes before dispatch', async () => { const io = makeIo(); const scan = vi.fn(async () => 0); - await expect(runKtxCli(['dev', 'scan', 'warehouse', '--mode', 'deep'], io.io, { scan })).resolves.toBe(1); + await expect(runKtxCli(['scan', 'warehouse', '--mode', 'deep'], io.io, { scan })).resolves.toBe(1); expect(scan).not.toHaveBeenCalled(); expect(io.stderr()).toContain("argument 'deep' is invalid"); expect(io.stderr()).toContain('Allowed choices are structural, enriched, relationships'); }); - it('prints dev scan subcommand help with the canonical command name', async () => { + it.each([ + ['scan', 'report', 'scan-run-1'], + ['scan', 'relationships', 'scan-run-1'], + ])('rejects removed scan subcommand %s %s', async (command, subcommand, runId) => { const io = makeIo(); const scan = vi.fn(async () => 0); - await expect(runKtxCli(['dev', 'scan', 'report', '--help'], io.io, { scan })).resolves.toBe(0); - - expect(io.stdout()).toContain('--project-dir is inherited from `ktx dev scan`'); - expect(io.stdout()).not.toContain('--project-dir is inherited from `ktx scan`'); - expect(scan).not.toHaveBeenCalled(); - }); - - it('dispatches dev scan report in human and json modes', async () => { - const humanIo = makeIo(); - const jsonIo = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli(['dev', 'scan', 'report', 'scan-run-1', '--project-dir', '/tmp/project'], humanIo.io, { scan }), - ).resolves.toBe(0); - await expect( - runKtxCli(['dev', 'scan', 'report', 'scan-run-2', '--project-dir', '/tmp/project', '--json'], jsonIo.io, { - scan, - }), - ).resolves.toBe(0); - - expect(scan).toHaveBeenNthCalledWith( - 1, - { command: 'report', projectDir: '/tmp/project', runId: 'scan-run-1', json: false }, - humanIo.io, - ); - expect(scan).toHaveBeenNthCalledWith( - 2, - { command: 'report', projectDir: '/tmp/project', runId: 'scan-run-2', json: true }, - jsonIo.io, - ); - }); - - it('dispatches dev scan relationships with filters through Commander', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli( - [ - 'dev', - 'scan', - 'relationships', - 'scan-run-review', - '--project-dir', - '/tmp/project', - '--status', - 'rejected', - '--limit', - '5', - '--json', - ], - io.io, - { scan }, - ), - ).resolves.toBe(0); - - expect(scan).toHaveBeenCalledWith( - { - command: 'relationships', - projectDir: '/tmp/project', - runId: 'scan-run-review', - status: 'rejected', - json: true, - limit: 5, - }, - io.io, - ); - expect(io.stderr()).toBe(''); - }); - - it('dispatches dev scan relationship decision recording through Commander', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli( - [ - 'dev', - 'scan', - 'relationships', - 'scan-run-review', - '--project-dir', - '/tmp/project', - '--accept', - 'orders:orders.customer_id->customers:customers.id', - '--reviewer', - 'Andrey', - '--note', - 'Looks right', - '--json', - ], - io.io, - { scan }, - ), - ).resolves.toBe(0); - - expect(scan).toHaveBeenCalledWith( - { - command: 'relationshipDecision', - projectDir: '/tmp/project', - runId: 'scan-run-review', - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'accepted', - reviewer: 'Andrey', - note: 'Looks right', - json: true, - }, - io.io, - ); - expect(io.stderr()).toBe(''); - }); - - it.each(['--accept', '--reject'])('rejects empty relationship decision candidate ids for %s', async (option) => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli(['dev', 'scan', 'relationships', 'scan-run-review', option, ''], io.io, { scan }), - ).resolves.toBe(1); + await expect(runKtxCli([command, subcommand, runId], io.io, { scan })).resolves.toBe(1); expect(scan).not.toHaveBeenCalled(); - expect(io.stderr()).toContain('must not be empty'); + expect(io.stderr()).toMatch(/too many arguments|unknown command|error:/); }); - it('rejects relationship feedback JSON and JSONL output together', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli(['dev', 'scan', 'relationship-feedback', '--json', '--jsonl'], io.io, { scan }), - ).resolves.toBe(1); - - expect(scan).not.toHaveBeenCalled(); - expect(io.stderr()).toMatch(/conflict|cannot be used/i); - }); - - it('dispatches relationship apply command args', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli( - [ - 'dev', - 'scan', - 'relationship-apply', - 'scan-run-a', - '--project-dir', - '/tmp/project', - '--candidate', - 'orders:orders.customer_id->customers:customers.id', - '--dry-run', - '--json', - ], - io.io, - { scan }, - ), - ).resolves.toBe(0); - - expect(scan).toHaveBeenCalledWith( - { - command: 'relationshipApply', - projectDir: '/tmp/project', - runId: 'scan-run-a', - applyAllAccepted: false, - candidateIds: ['orders:orders.customer_id->customers:customers.id'], - dryRun: true, - json: true, - }, - io.io, - ); - }); - - it('dispatches scan relationship feedback command with filters and JSONL output', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli( - [ - 'dev', - 'scan', - 'relationship-feedback', - '--project-dir', - '/tmp/project', - '--connection', - 'warehouse', - '--decision', - 'accepted', - '--jsonl', - ], - io.io, - { scan }, - ), - ).resolves.toBe(0); - - expect(scan).toHaveBeenCalledWith( - { - command: 'relationshipFeedback', - projectDir: '/tmp/project', - connectionId: 'warehouse', - decision: 'accepted', - json: false, - jsonl: true, - }, - io.io, - ); - }); - - it('dispatches scan relationship calibration command with thresholds', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli( - [ - 'dev', - 'scan', - 'relationship-calibration', - '--project-dir', - '/tmp/project', - '--connection', - 'warehouse', - '--decision', - 'rejected', - '--accept-threshold', - '0.9', - '--review-threshold', - '0.5', - '--json', - ], - io.io, - { scan }, - ), - ).resolves.toBe(0); - - expect(scan).toHaveBeenCalledWith( - { - command: 'relationshipCalibration', - projectDir: '/tmp/project', - connectionId: 'warehouse', - decision: 'rejected', - acceptThreshold: 0.9, - reviewThreshold: 0.5, - json: true, - }, - io.io, - ); - }); - - it('dispatches relationship threshold advice command args', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli( - [ - 'dev', - 'scan', - 'relationship-thresholds', - '--project-dir', - '/tmp/project', - '--connection', - 'warehouse', - '--min-total-labels', - '12', - '--min-accepted-labels', - '4', - '--min-rejected-labels', - '3', - '--json', - ], - io.io, - { scan }, - ), - ).resolves.toBe(0); - - expect(scan).toHaveBeenCalledWith( - { - command: 'relationshipThresholds', - projectDir: '/tmp/project', - connectionId: 'warehouse', - minTotalLabels: 12, - minAcceptedLabels: 4, - minRejectedLabels: 3, - json: true, - }, - io.io, - ); - }); - - it('rejects invalid relationship calibration thresholds before dispatch', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli(['dev', 'scan', 'relationship-calibration', '--accept-threshold', '1.5'], io.io, { scan }), - ).resolves.toBe(1); - - expect(scan).not.toHaveBeenCalled(); - expect(io.stderr()).toContain('Allowed range is 0 through 1'); - }); - - it('rejects relationship accept and reject options together before dispatch', async () => { - const io = makeIo(); - const scan = vi.fn(async () => 0); - - await expect( - runKtxCli( - [ - 'dev', - 'scan', - 'relationships', - 'scan-run-review', - '--accept', - 'orders:orders.customer_id->customers:customers.id', - '--reject', - 'orders:orders.customer_id->customers:customers.id', - ], - io.io, - { scan }, - ), - ).resolves.toBe(1); - - expect(scan).not.toHaveBeenCalled(); - expect(io.stderr()).toMatch(/conflict|cannot be used/i); - }); - - it('dispatches dev ingest run through the low-level ingest Commander registration', async () => { + it('dispatches top-level ingest run through the low-level ingest Commander registration', async () => { const io = makeIo(); const ingest = vi.fn(async () => 0); await expect( runKtxCli( [ - 'dev', 'ingest', 'run', '--connection-id', diff --git a/packages/cli/src/dev.ts b/packages/cli/src/dev.ts index 06d73856..9391cc43 100644 --- a/packages/cli/src/dev.ts +++ b/packages/cli/src/dev.ts @@ -1,11 +1,7 @@ import { resolve } from 'node:path'; import type { Command } from '@commander-js/extra-typings'; import { type CommandWithGlobalOptions, type KtxCliCommandContext, resolveCommandProjectDir } from './cli-program.js'; -import { registerCompletionCommands } from './commands/completion-commands.js'; -import { registerConnectionMappingCommands } from './commands/connection-commands.js'; -import { registerIngestCommands } from './commands/ingest-commands.js'; import { registerRuntimeCommands } from './commands/runtime-commands.js'; -import { registerScanCommands } from './commands/scan-commands.js'; import { profileMark } from './startup-profile.js'; profileMark('module:dev'); @@ -13,7 +9,7 @@ profileMark('module:dev'); export function registerDevCommands(program: Command, context: KtxCliCommandContext): void { const dev = program .command('dev', { hidden: true }) - .description('Low-level diagnostics, scans, adapter commands, and mapping tools') + .description('Low-level project initialization and runtime management') .showHelpAfterError(); dev.hook('preAction', (_thisCommand, actionCommand) => { @@ -51,11 +47,4 @@ export function registerDevCommands(program: Command, context: KtxCliCommandCont ); registerRuntimeCommands(dev, context); - registerScanCommands(dev, context); - registerIngestCommands(dev, context, { - runIngestWithProgress: async (ingestArgs, ingestIo, ingestDeps, defaultRunIngest) => - await (ingestDeps.ingest ?? defaultRunIngest)(ingestArgs, ingestIo), - }); - registerConnectionMappingCommands(dev, context); - registerCompletionCommands(dev, context, program); } diff --git a/packages/cli/src/example-smoke.test.ts b/packages/cli/src/example-smoke.test.ts index b3bbc681..bb475a46 100644 --- a/packages/cli/src/example-smoke.test.ts +++ b/packages/cli/src/example-smoke.test.ts @@ -73,26 +73,27 @@ describe('standalone local warehouse example', () => { const projectDir = await copyExampleProject(tempDir); const sourceDir = join(projectDir, 'source'); - const knowledgeList = await runBuiltCli(['agent', 'wiki', 'search', 'revenue', '--json', '--project-dir', projectDir]); + const knowledgeList = await runBuiltCli(['wiki', 'search', 'revenue', '--json', '--project-dir', projectDir]); expect(knowledgeList).toMatchObject({ code: 0, stderr: '' }); - expect(parseJsonOutput<{ results: Array<{ key: string; summary: string }> }>(knowledgeList.stdout).results).toContainEqual( - expect.objectContaining({ key: 'revenue', summary: 'Paid order value after refunds' }), - ); + expect( + parseJsonOutput<{ data: { items: Array<{ key: string; summary: string }> } }>(knowledgeList.stdout).data.items, + ).toContainEqual(expect.objectContaining({ key: 'revenue', summary: 'Paid order value after refunds' })); - const knowledgeRead = await runBuiltCli(['agent', 'wiki', 'read', 'revenue', '--json', '--project-dir', projectDir]); + const knowledgeRead = await runBuiltCli(['wiki', 'read', 'revenue', '--json', '--project-dir', projectDir]); expect(knowledgeRead).toMatchObject({ code: 0, stderr: '' }); - expect(parseJsonOutput<{ content: string }>(knowledgeRead.stdout).content).toContain( + expect(parseJsonOutput<{ data: { content: string } }>(knowledgeRead.stdout).data.content).toContain( 'Revenue is paid order amount after refund adjustments.', ); - const slList = await runBuiltCli(['agent', 'sl', 'list', '--json', '--project-dir', projectDir, '--connection-id', 'warehouse']); + const slList = await runBuiltCli(['sl', 'list', '--json', '--project-dir', projectDir, '--connection-id', 'warehouse']); expect(slList).toMatchObject({ code: 0, stderr: '' }); - expect(parseJsonOutput<{ sources: Array<{ connectionId: string; name: string; columnCount: number }> }>(slList.stdout).sources).toContainEqual( - expect.objectContaining({ connectionId: 'warehouse', name: 'orders', columnCount: 3 }), - ); + expect( + parseJsonOutput<{ data: { items: Array<{ connectionId: string; name: string; columnCount: number }> } }>( + slList.stdout, + ).data.items, + ).toContainEqual(expect.objectContaining({ connectionId: 'warehouse', name: 'orders', columnCount: 3 })); const slRead = await runBuiltCli([ - 'agent', 'sl', 'read', 'orders', @@ -103,10 +104,9 @@ describe('standalone local warehouse example', () => { projectDir, ]); expect(slRead).toMatchObject({ code: 0, stderr: '' }); - expect(parseJsonOutput<{ yaml: string }>(slRead.stdout).yaml).toContain('name: orders'); + expect(parseJsonOutput<{ data: { yaml: string } }>(slRead.stdout).data.yaml).toContain('name: orders'); const ingest = await runBuiltCli([ - 'dev', 'ingest', 'run', '--project-dir', diff --git a/packages/cli/src/index.test.ts b/packages/cli/src/index.test.ts index 6a968fa9..1e69c590 100644 --- a/packages/cli/src/index.test.ts +++ b/packages/cli/src/index.test.ts @@ -1,4 +1,4 @@ -import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { mkdtemp, rm } from 'node:fs/promises'; import { createRequire } from 'node:module'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; @@ -123,10 +123,10 @@ describe('runKtxCli', () => { await expect(runKtxCli(['--help'], testIo.io)).resolves.toBe(0); expect(testIo.stdout()).toContain('Usage: ktx [options] [command]'); - for (const command of ['setup', 'connection', 'ingest', 'wiki', 'sl', 'status']) { + for (const command of ['setup', 'connection', 'ingest', 'wiki', 'sl', 'status', 'scan']) { expect(testIo.stdout()).toContain(`${command}`); } - for (const removed of ['demo', 'init', 'connect', 'scan', 'ask', 'knowledge', 'agent', 'completion', 'runtime', 'serve']) { + for (const removed of ['demo', 'init', 'connect', 'ask', 'knowledge', 'agent', 'completion', 'serve']) { expect(testIo.stdout()).not.toContain(`${removed} [`); expect(testIo.stdout()).not.toContain(`${removed} `); } @@ -146,7 +146,6 @@ describe('runKtxCli', () => { const stopIo = makeIo(); const stopAllIo = makeIo(); const statusIo = makeIo(); - const doctorIo = makeIo(); const pruneIo = makeIo(); await expect( @@ -160,8 +159,7 @@ describe('runKtxCli', () => { await expect(runKtxCli(['dev', 'runtime', 'stop'], stopIo.io, { runtime })).resolves.toBe(0); await expect(runKtxCli(['dev', 'runtime', 'stop', '--all'], stopAllIo.io, { runtime })).resolves.toBe(0); await expect(runKtxCli(['dev', 'runtime', 'status', '--json'], statusIo.io, { runtime })).resolves.toBe(0); - await expect(runKtxCli(['dev', 'runtime', 'doctor'], doctorIo.io, { runtime })).resolves.toBe(0); - await expect(runKtxCli(['dev', 'runtime', 'prune', '--dry-run'], pruneIo.io, { runtime })).resolves.toBe(0); + await expect(runKtxCli(['dev', 'runtime', 'prune', '--dry-run'], pruneIo.io, { runtime })).resolves.toBe(1); expect(runtime).toHaveBeenNthCalledWith( 1, @@ -210,28 +208,11 @@ describe('runKtxCli', () => { }, statusIo.io, ); - expect(runtime).toHaveBeenNthCalledWith( - 6, - { - command: 'doctor', - cliVersion: '0.0.0-private', - json: false, - }, - doctorIo.io, - ); - expect(runtime).toHaveBeenNthCalledWith( - 7, - { - command: 'prune', - cliVersion: '0.0.0-private', - dryRun: true, - yes: false, - }, - pruneIo.io, - ); - for (const io of [installIo, startIo, stopIo, stopAllIo, statusIo, doctorIo, pruneIo]) { + expect(runtime).toHaveBeenCalledTimes(5); + for (const io of [installIo, startIo, stopIo, stopAllIo, statusIo]) { expect(io.stderr()).toBe(''); } + expect(pruneIo.stderr()).toMatch(/unknown command|error:/); }); it('prints the resolved project directory for ordinary project commands', async () => { @@ -247,16 +228,15 @@ describe('runKtxCli', () => { }); it('skips the project directory line for JSON and TUI output modes', async () => { - const publicIngest = vi.fn(async () => 0); const ingest = vi.fn(async () => 0); const jsonIo = makeIo(); const vizIo = makeIo({ stdoutIsTty: true }); - await expect(runKtxCli(['--project-dir', tempDir, 'ingest', '--all', '--json'], jsonIo.io, { publicIngest })) + await expect(runKtxCli(['--project-dir', tempDir, 'ingest', 'status', 'run-1', '--json'], jsonIo.io, { ingest })) .resolves.toBe(0); await expect( runKtxCli( - ['--project-dir', tempDir, 'dev', 'ingest', 'status', 'run-1', '--viz'], + ['--project-dir', tempDir, 'ingest', 'status', 'run-1', '--viz'], vizIo.io, { ingest }, ), @@ -503,158 +483,17 @@ describe('runKtxCli', () => { expect(testIo.stdout()).toBe(''); }); - it('prints a zsh completion function', async () => { - const testIo = makeIo(); - const zshWords = '$' + '{words[@]}'; - - await expect(runKtxCli(['dev', 'completion', 'zsh'], testIo.io)).resolves.toBe(0); - - expect(testIo.stdout()).toContain('#compdef ktx'); - expect(testIo.stdout()).toContain('KTX_COMPLETION_COMMAND:-ktx'); - expect(testIo.stdout()).toContain(`dev __complete --shell zsh --position "$CURRENT" -- "${zshWords}"`); - expect(testIo.stdout()).toContain('compdef _ktx ktx'); - expect(testIo.stderr()).toBe(''); - }); - - it('installs zsh completions into the user zsh config directory', async () => { - const testIo = makeIo(); - const previousHome = process.env.HOME; - const previousZdotdir = process.env.ZDOTDIR; - const tempHome = await mkdtemp(join(tmpdir(), 'ktx-completion-home-')); - - try { - process.env.HOME = tempHome; - delete process.env.ZDOTDIR; - - await expect(runKtxCli(['dev', 'completion', 'zsh', '--install'], testIo.io)).resolves.toBe(0); - - const completionFile = await readFile(join(tempHome, '.zfunc', '_ktx'), 'utf-8'); - const zshrc = await readFile(join(tempHome, '.zshrc'), 'utf-8'); - expect(completionFile).toContain('#compdef ktx'); - expect(zshrc).toContain('# >>> ktx completion >>>'); - expect(zshrc).toContain('_ktx_completion_command()'); - expect(zshrc).toContain('"name": "ktx-workspace"'); - expect(zshrc).toContain('scripts/run-ktx.mjs'); - expect(zshrc).toContain("export KTX_COMPLETION_COMMAND='$(_ktx_completion_command)'"); - expect(zshrc).toContain('setopt complete_aliases'); - expect(zshrc).toContain('fpath=("$HOME/.zfunc" $fpath)'); - expect(zshrc).toContain('autoload -Uz compinit'); - expect(zshrc).toContain('compinit'); - expect(testIo.stdout()).toContain('Installed zsh completion:'); - expect(testIo.stdout()).toContain('Restart your shell or run: source ~/.zshrc'); - expect(testIo.stderr()).toBe(''); - } finally { - if (previousHome === undefined) { - delete process.env.HOME; - } else { - process.env.HOME = previousHome; - } - if (previousZdotdir === undefined) { - delete process.env.ZDOTDIR; - } else { - process.env.ZDOTDIR = previousZdotdir; - } - await rm(tempHome, { recursive: true, force: true }); - } - }); - - it('updates zsh completion install block idempotently before existing compinit', async () => { - const firstIo = makeIo(); - const secondIo = makeIo(); - const previousHome = process.env.HOME; - const previousZdotdir = process.env.ZDOTDIR; - const tempHome = await mkdtemp(join(tmpdir(), 'ktx-completion-home-')); - - try { - process.env.HOME = tempHome; - delete process.env.ZDOTDIR; - await writeFile(join(tempHome, '.zshrc'), 'export EDITOR=vim\nautoload -Uz compinit\ncompinit\n', 'utf-8'); - - await expect(runKtxCli(['dev', 'completion', 'zsh', '--install'], firstIo.io)).resolves.toBe(0); - await expect(runKtxCli(['dev', 'completion', 'zsh', '--install'], secondIo.io)).resolves.toBe(0); - - const zshrc = await readFile(join(tempHome, '.zshrc'), 'utf-8'); - expect(zshrc.match(/# >>> ktx completion >>>/g)).toHaveLength(1); - expect(zshrc.indexOf('fpath=("$HOME/.zfunc" $fpath)')).toBeLessThan(zshrc.indexOf('autoload -Uz compinit')); - expect(zshrc.match(/_ktx_completion_command\(\)/g)).toHaveLength(1); - expect(zshrc.match(/^compinit$/gm)).toHaveLength(1); - expect(secondIo.stdout()).toContain('Updated zsh config:'); - expect(firstIo.stderr()).toBe(''); - expect(secondIo.stderr()).toBe(''); - } finally { - if (previousHome === undefined) { - delete process.env.HOME; - } else { - process.env.HOME = previousHome; - } - if (previousZdotdir === undefined) { - delete process.env.ZDOTDIR; - } else { - process.env.ZDOTDIR = previousZdotdir; - } - await rm(tempHome, { recursive: true, force: true }); - } - }); - - it('completes root and nested Commander command names', async () => { - const rootIo = makeIo(); - const connectionIo = makeIo(); + it('rejects removed shell completion commands', async () => { + const completionIo = makeIo(); + const hiddenIo = makeIo(); + await expect(runKtxCli(['dev', 'completion', 'zsh'], completionIo.io)).resolves.toBe(1); await expect( - runKtxCli(['dev', '__complete', '--shell', 'zsh', '--position', '2', '--', 'ktx', 'co'], rootIo.io), - ).resolves.toBe(0); - await expect( - runKtxCli( - ['dev', '__complete', '--shell', 'zsh', '--position', '3', '--', 'ktx', 'connection', 'm'], - connectionIo.io, - ), - ).resolves.toBe(0); + runKtxCli(['dev', '__complete', '--shell', 'zsh', '--position', '2', '--', 'ktx', 'co'], hiddenIo.io), + ).resolves.toBe(1); - expect(rootIo.stdout()).toContain('connection:Add, list, test, and map data sources'); - expect(rootIo.stdout()).not.toContain('__complete'); - expect(connectionIo.stdout()).toContain('map:Refresh and validate BI-to-warehouse mappings'); - expect(connectionIo.stdout()).toContain('mapping:Manage Metabase warehouse mappings'); - expect(rootIo.stderr()).toBe(''); - expect(connectionIo.stderr()).toBe(''); - }); - - it('completes options and Commander choices', async () => { - const optionIo = makeIo(); - const choiceIo = makeIo(); - - await expect( - runKtxCli( - ['dev', '__complete', '--shell', 'zsh', '--position', '4', '--', 'ktx', 'connection', 'add', '--cr'], - optionIo.io, - ), - ).resolves.toBe(0); - await expect( - runKtxCli( - [ - 'dev', - '__complete', - '--shell', - 'zsh', - '--position', - '7', - '--', - 'ktx', - 'connection', - 'add', - 'notion', - 'docs', - '--crawl-mode', - '', - ], - choiceIo.io, - ), - ).resolves.toBe(0); - - expect(optionIo.stdout()).toContain('--crawl-mode:Notion crawl mode'); - expect(choiceIo.stdout()).toContain('all_accessible'); - expect(choiceIo.stdout()).toContain('selected_roots'); - expect(optionIo.stderr()).toBe(''); - expect(choiceIo.stderr()).toBe(''); + expect(completionIo.stderr()).toMatch(/unknown command|error:/); + expect(hiddenIo.stderr()).toMatch(/unknown command|error:/); }); it('rejects removed serve commands', async () => { @@ -666,35 +505,22 @@ describe('runKtxCli', () => { expect(testIo.stderr()).toMatch(/unknown command|error:/); }); - it('routes public ingest through the public ingest parser', async () => { + it('rejects removed public ingest shorthand', async () => { const testIo = makeIo(); const ingest = vi.fn().mockResolvedValue(0); - await expect( - runKtxCli(['--project-dir', '/tmp/project', 'ingest', 'warehouse'], testIo.io, { publicIngest: ingest }), - ).resolves.toBe(0); + await expect(runKtxCli(['--project-dir', '/tmp/project', 'ingest', 'warehouse'], testIo.io, { ingest })) + .resolves.toBe(1); - expect(ingest).toHaveBeenCalledWith( - { - command: 'run', - projectDir: '/tmp/project', - targetConnectionId: 'warehouse', - all: false, - json: false, - inputMode: 'auto', - }, - testIo.io, - ); + expect(ingest).not.toHaveBeenCalled(); + expect(testIo.stderr()).toMatch(/unknown command|error:/); }); - it('prints public ingest watch help from Commander', async () => { + it('prints ingest watch help from Commander', async () => { const testIo = makeIo(); - const publicIngest = vi.fn(async () => 0); - const lowLevelIngest = vi.fn(async () => 0); + const ingest = vi.fn(async () => 0); - await expect( - runKtxCli(['ingest', 'watch', '--help'], testIo.io, { publicIngest, ingest: lowLevelIngest }), - ).resolves.toBe(0); + await expect(runKtxCli(['ingest', 'watch', '--help'], testIo.io, { ingest })).resolves.toBe(0); expect(testIo.stdout()).toContain('Usage: ktx ingest watch [options] [runId]'); expect(testIo.stdout()).toContain('[runId]'); @@ -702,43 +528,42 @@ describe('runKtxCli', () => { expect(testIo.stdout()).toContain('--json'); expect(testIo.stdout()).toContain('--no-input'); expect(testIo.stderr()).toBe(''); - expect(publicIngest).not.toHaveBeenCalled(); - expect(lowLevelIngest).not.toHaveBeenCalled(); + expect(ingest).not.toHaveBeenCalled(); }); - it('dispatches public ingest status and watch through Commander', async () => { + it('dispatches ingest status and watch through Commander', async () => { const statusIo = makeIo(); const watchIo = makeIo(); - const publicIngest = vi.fn(async () => 0); + const ingest = vi.fn(async () => 0); await expect( runKtxCli(['--project-dir', tempDir, 'ingest', 'status', 'run-1', '--json', '--no-input'], statusIo.io, { - publicIngest, + ingest, }), ).resolves.toBe(0); await expect( runKtxCli(['--project-dir', tempDir, 'ingest', 'watch', '--no-input'], watchIo.io, { - publicIngest, + ingest, }), ).resolves.toBe(0); - expect(publicIngest).toHaveBeenNthCalledWith( + expect(ingest).toHaveBeenNthCalledWith( 1, { command: 'status', projectDir: tempDir, runId: 'run-1', - json: true, + outputMode: 'json', inputMode: 'disabled', }, statusIo.io, ); - expect(publicIngest).toHaveBeenNthCalledWith( + expect(ingest).toHaveBeenNthCalledWith( 2, { command: 'watch', projectDir: tempDir, - json: false, + outputMode: 'viz', inputMode: 'disabled', }, watchIo.io, @@ -778,60 +603,44 @@ describe('runKtxCli', () => { expect(setup).not.toHaveBeenCalled(); }); - it('prints public ingest help without invoking ingest execution', async () => { + it('prints ingest help without invoking ingest execution', async () => { const testIo = makeIo(); - const publicIngest = vi.fn(); - const lowLevelIngest = vi.fn(); + const ingest = vi.fn(); - await expect(runKtxCli(['ingest', '--help'], testIo.io, { publicIngest, ingest: lowLevelIngest })).resolves.toBe(0); + await expect(runKtxCli(['ingest', '--help'], testIo.io, { ingest })).resolves.toBe(0); - expect(testIo.stdout()).toContain('Usage: ktx ingest [options] [connectionId]'); - expect(testIo.stdout()).toContain('Build and refresh KTX context from configured sources'); + expect(testIo.stdout()).toContain('Usage: ktx ingest [options] [command]'); + expect(testIo.stdout()).toContain('Run or inspect local ingest memory-flow output'); + expect(testIo.stdout()).toContain('run'); expect(testIo.stdout()).toContain('status'); expect(testIo.stdout()).toContain('watch'); - expect(testIo.stdout()).toContain('ktx ingest --all [options]'); - expect(testIo.stdout()).toContain('ktx ingest status [runId] [options]'); - expect(testIo.stdout()).toContain('ktx ingest watch [runId] [options]'); - expect(testIo.stdout()).not.toContain('ktx ingest replay [options]'); - expect(testIo.stdout()).toContain('--no-input'); - expect(testIo.stdout()).not.toContain('--adapter'); + expect(testIo.stdout()).toContain('replay'); + expect(testIo.stdout()).not.toContain('--all'); expect(testIo.stderr()).toBe(''); - expect(publicIngest).not.toHaveBeenCalled(); - expect(lowLevelIngest).not.toHaveBeenCalled(); + expect(ingest).not.toHaveBeenCalled(); }); - it('reserves public ingest run while keeping dev ingest run available', async () => { - const publicRunIo = makeIo(); - const publicHelpIo = makeIo(); + it('routes ingest run at the top level and rejects removed dev ingest', async () => { + const runIo = makeIo(); const devRunIo = makeIo(); - const publicIngest = vi.fn(async () => 0); - const lowLevelIngest = vi.fn(async () => 0); - - await expect(runKtxCli(['ingest', 'run'], publicRunIo.io, { publicIngest, ingest: lowLevelIngest })).resolves.toBe( - 1, - ); - expect(publicRunIo.stderr()).toMatch(/invalid argument|reserved|run/i); - expect(publicIngest).not.toHaveBeenCalled(); + const ingest = vi.fn(async () => 0); await expect( - runKtxCli(['ingest', 'run', '--help'], publicHelpIo.io, { publicIngest, ingest: lowLevelIngest }), + runKtxCli(['ingest', 'run', '--connection-id', 'warehouse', '--adapter', 'metabase'], runIo.io, { ingest }), ).resolves.toBe(0); - expect(publicHelpIo.stdout()).toContain('Usage: ktx ingest [options] [connectionId]'); - expect(publicHelpIo.stdout()).not.toContain('Usage: ktx ingest ' + 'run'); - await expect( runKtxCli(['dev', 'ingest', 'run', '--connection-id', 'warehouse', '--adapter', 'metabase'], devRunIo.io, { - publicIngest, - ingest: lowLevelIngest, + ingest, }), - ).resolves.toBe(0); - expect(lowLevelIngest).toHaveBeenCalledWith( + ).resolves.toBe(1); + expect(ingest).toHaveBeenCalledWith( expect.objectContaining({ command: 'run', connectionId: 'warehouse', adapter: 'metabase' }), expect.anything(), ); + expect(devRunIo.stderr()).toMatch(/unknown command|error:/); }); - it('rejects removed dev doctor while keeping ingest parser cases under dev', async () => { + it('rejects removed dev doctor while keeping ingest parser cases at the root', async () => { const doctor = vi.fn(async () => 0); const ingest = vi.fn(async () => 0); const doctorIo = makeIo(); @@ -842,7 +651,6 @@ describe('runKtxCli', () => { await expect( runKtxCli( [ - 'dev', 'ingest', 'run', '--project-dir', @@ -862,7 +670,7 @@ describe('runKtxCli', () => { { ingest }, ), ).resolves.toBe(0); - await expect(runKtxCli(['dev', 'ingest', 'replay', '--help'], ingestReplayHelpIo.io, { ingest })).resolves.toBe(0); + await expect(runKtxCli(['ingest', 'replay', '--help'], ingestReplayHelpIo.io, { ingest })).resolves.toBe(0); expect(doctor).not.toHaveBeenCalled(); expect(ingest).toHaveBeenCalledWith( @@ -881,7 +689,7 @@ describe('runKtxCli', () => { }, ingestRunIo.io, ); - expect(ingestReplayHelpIo.stdout()).toContain('Usage: ktx dev ingest replay [options] '); + expect(ingestReplayHelpIo.stdout()).toContain('Usage: ktx ingest replay [options] '); expect(ingestReplayHelpIo.stdout()).toContain(''); expect(doctorIo.stderr()).toMatch(/unknown command|error:/); expect(ingestRunIo.stderr()).toBe(''); @@ -896,7 +704,6 @@ describe('runKtxCli', () => { await expect( runKtxCli( [ - 'dev', 'ingest', 'run', '--project-dir', @@ -914,7 +721,6 @@ describe('runKtxCli', () => { await expect( runKtxCli( [ - 'dev', 'ingest', 'run', '--project-dir', @@ -1335,136 +1141,28 @@ describe('runKtxCli', () => { expect(setupIo.stderr()).toContain('Choose only one Historic SQL action'); }); - it('registers hidden agent help and tools discovery without showing agent in root help', async () => { - const helpIo = makeIo(); - const toolsIo = makeIo(); - const agent = vi.fn(async () => 0); + it('rejects the removed hidden agent command', async () => { + const io = makeIo(); - await expect(runKtxCli(['agent', '--help'], helpIo.io, { agent })).resolves.toBe(0); - await expect( - runKtxCli(['--project-dir', tempDir, 'agent', 'tools', '--json'], toolsIo.io, { agent }), - ).resolves.toBe(0); + await expect(runKtxCli(['agent'], io.io)).resolves.toBe(1); - expect(helpIo.stdout()).toContain('Usage: ktx agent'); - expect(toolsIo.stderr()).toBe(''); - expect(agent).toHaveBeenCalledWith({ command: 'tools', projectDir: tempDir, json: true }, toolsIo.io); + expect(io.stderr()).toContain("unknown command 'agent'"); + expect(io.stdout()).toBe(''); }); - it('dispatches full hidden agent commands without exposing agent in root help', async () => { - const agent = vi.fn(async () => 0); - const cases = [ - { - argv: ['--project-dir', tempDir, 'agent', 'context', '--json'], - args: { command: 'context', projectDir: tempDir, json: true }, - }, - { - argv: [ - '--project-dir', - tempDir, - 'agent', - 'sl', - 'list', - '--json', - '--connection-id', - 'warehouse', - '--query', - 'orders', - ], - args: { command: 'sl-list', projectDir: tempDir, json: true, connectionId: 'warehouse', query: 'orders' }, - }, - { - argv: ['--project-dir', tempDir, 'agent', 'sl', 'read', 'orders', '--json', '--connection-id', 'warehouse'], - args: { command: 'sl-read', projectDir: tempDir, json: true, sourceName: 'orders', connectionId: 'warehouse' }, - }, - { - argv: [ - '--project-dir', - tempDir, - 'agent', - 'sl', - 'query', - '--json', - '--connection-id', - 'warehouse', - '--query-file', - '/tmp/query.json', - '--execute', - '--max-rows', - '100', - ], - args: { - command: 'sl-query', - projectDir: tempDir, - json: true, - connectionId: 'warehouse', - queryFile: '/tmp/query.json', - execute: true, - maxRows: 100, - cliVersion: '0.0.0-private', - runtimeInstallPolicy: 'prompt', - }, - }, - { - argv: ['--project-dir', tempDir, 'agent', 'wiki', 'search', 'revenue', '--json', '--limit', '5'], - args: { command: 'wiki-search', projectDir: tempDir, json: true, query: 'revenue', limit: 5 }, - }, - { - argv: ['--project-dir', tempDir, 'agent', 'wiki', 'read', 'page-1', '--json'], - args: { command: 'wiki-read', projectDir: tempDir, json: true, pageId: 'page-1' }, - }, - { - argv: [ - '--project-dir', - tempDir, - 'agent', - 'sql', - 'execute', - '--json', - '--connection-id', - 'warehouse', - '--sql-file', - '/tmp/query.sql', - '--max-rows', - '100', - ], - args: { - command: 'sql-execute', - projectDir: tempDir, - json: true, - connectionId: 'warehouse', - sqlFile: '/tmp/query.sql', - maxRows: 100, - }, - }, - ]; - - for (const entry of cases) { - const io = makeIo(); - await expect(runKtxCli(entry.argv, io.io, { agent })).resolves.toBe(0); - expect(agent).toHaveBeenLastCalledWith(entry.args, io.io); - expect(io.stderr()).toBe(''); - } - - const helpIo = makeIo(); - await expect(runKtxCli(['--help'], helpIo.io, { agent })).resolves.toBe(0); - expect(helpIo.stdout()).not.toContain('agent '); - }); - - it('routes hidden agent SL query managed runtime policies', async () => { + it('routes public SL query files with managed runtime policies', async () => { const autoIo = makeIo(); const neverIo = makeIo(); const conflictIo = makeIo(); - const agent = vi.fn(async () => 0); + const sl = vi.fn(async () => 0); await expect( runKtxCli( [ '--project-dir', tempDir, - 'agent', 'sl', 'query', - '--json', '--connection-id', 'warehouse', '--query-file', @@ -1472,7 +1170,7 @@ describe('runKtxCli', () => { '--yes', ], autoIo.io, - { agent }, + { sl }, ), ).resolves.toBe(0); @@ -1481,10 +1179,8 @@ describe('runKtxCli', () => { [ '--project-dir', tempDir, - 'agent', 'sl', 'query', - '--json', '--connection-id', 'warehouse', '--query-file', @@ -1492,7 +1188,7 @@ describe('runKtxCli', () => { '--no-input', ], neverIo.io, - { agent }, + { sl }, ), ).resolves.toBe(0); @@ -1501,10 +1197,8 @@ describe('runKtxCli', () => { [ '--project-dir', tempDir, - 'agent', 'sl', 'query', - '--json', '--connection-id', 'warehouse', '--query-file', @@ -1513,33 +1207,33 @@ describe('runKtxCli', () => { '--no-input', ], conflictIo.io, - { agent }, + { sl }, ), ).resolves.toBe(1); - expect(agent).toHaveBeenNthCalledWith( + expect(sl).toHaveBeenNthCalledWith( 1, { - command: 'sl-query', + command: 'query', projectDir: tempDir, - json: true, connectionId: 'warehouse', queryFile: '/tmp/query.json', execute: false, + format: 'json', cliVersion: '0.0.0-private', runtimeInstallPolicy: 'auto', }, autoIo.io, ); - expect(agent).toHaveBeenNthCalledWith( + expect(sl).toHaveBeenNthCalledWith( 2, { - command: 'sl-query', + command: 'query', projectDir: tempDir, - json: true, connectionId: 'warehouse', queryFile: '/tmp/query.json', execute: false, + format: 'json', cliVersion: '0.0.0-private', runtimeInstallPolicy: 'never', }, @@ -1548,112 +1242,6 @@ describe('runKtxCli', () => { expect(conflictIo.stderr()).toContain('Choose only one runtime install mode: --yes or --no-input'); }); - it('prints semantic-layer hybrid search metadata from the hidden agent sl list command', async () => { - const agent = vi.fn(async (args, io) => { - expect(args).toEqual({ - command: 'sl-list', - projectDir: tempDir, - json: true, - connectionId: 'warehouse', - query: 'paid', - }); - io.stdout.write( - `${JSON.stringify( - { - sources: [ - { - connectionId: 'warehouse', - connectionName: 'warehouse', - name: 'orders', - columnCount: 2, - measureCount: 1, - joinCount: 0, - score: 0.03278688524590164, - matchReasons: ['dictionary'], - dictionaryMatches: [{ column: 'status', values: ['paid'] }], - }, - ], - totalSources: 1, - }, - null, - 2, - )}\n`, - ); - return 0; - }); - const io = makeIo(); - - await expect( - runKtxCli( - ['--project-dir', tempDir, 'agent', 'sl', 'list', '--json', '--connection-id', 'warehouse', '--query', 'paid'], - io.io, - { agent }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toEqual({ - sources: [ - expect.objectContaining({ - connectionId: 'warehouse', - name: 'orders', - matchReasons: ['dictionary'], - dictionaryMatches: [{ column: 'status', values: ['paid'] }], - }), - ], - totalSources: 1, - }); - }); - - it('prints wiki hybrid search metadata from the hidden agent wiki search command', async () => { - const agent = vi.fn(async (args, io) => { - expect(args).toEqual({ - command: 'wiki-search', - projectDir: tempDir, - json: true, - query: 'paid order', - limit: 5, - }); - io.stdout.write( - `${JSON.stringify( - { - results: [ - { - key: 'metrics-revenue', - path: 'knowledge/global/metrics-revenue.md', - scope: 'GLOBAL', - summary: 'Revenue metric definition', - score: 0.02459016393442623, - matchReasons: ['lexical', 'token'], - }, - ], - totalFound: 1, - }, - null, - 2, - )}\n`, - ); - return 0; - }); - const io = makeIo(); - - await expect( - runKtxCli(['--project-dir', tempDir, 'agent', 'wiki', 'search', 'paid order', '--json', '--limit', '5'], io.io, { - agent, - }), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toEqual({ - results: [ - expect.objectContaining({ - key: 'metrics-revenue', - path: 'knowledge/global/metrics-revenue.md', - matchReasons: ['lexical', 'token'], - }), - ], - totalFound: 1, - }); - }); - it('dispatches public connection subcommands through the existing connection implementation', async () => { const tempDir = await mkdtemp(join(tmpdir(), 'ktx-connection-dispatch-')); const connection = vi.fn(async () => 0); @@ -1729,7 +1317,7 @@ describe('runKtxCli', () => { 'ktx connection mapping refresh --auto-accept', 'ktx connection mapping set databaseMappings =', 'ktx connection mapping set-sync-enabled --enabled true', - 'ktx ingest ', + 'ktx ingest run --connection-id --adapter metabase', ]) { expect(helpIo.stdout()).toContain(line); } @@ -1870,7 +1458,6 @@ describe('runKtxCli', () => { for (const argv of [ ['init'], ['connect', 'list'], - ['scan', 'warehouse'], ['knowledge', 'list'], ['ask', 'What sources are connected?'], ]) { @@ -2041,11 +1628,11 @@ describe('runKtxCli', () => { expect(testIo.stderr()).toContain('[debug] dispatch=connection'); }); - it('routes low-level scan through ktx dev with top-level project-dir', async () => { + it('routes scan through the top-level command with top-level project-dir', async () => { const testIo = makeIo(); const scan = vi.fn().mockResolvedValue(0); - await expect(runKtxCli(['--project-dir', tempDir, 'dev', 'scan', 'warehouse'], testIo.io, { scan })).resolves.toBe( + await expect(runKtxCli(['--project-dir', tempDir, 'scan', 'warehouse'], testIo.io, { scan })).resolves.toBe( 0, ); @@ -2071,12 +1658,12 @@ describe('runKtxCli', () => { const conflictIo = makeIo(); const scan = vi.fn().mockResolvedValue(0); - await expect(runKtxCli(['--project-dir', tempDir, 'dev', 'scan', 'warehouse', '--yes'], autoIo.io, { scan })) + await expect(runKtxCli(['--project-dir', tempDir, 'scan', 'warehouse', '--yes'], autoIo.io, { scan })) .resolves.toBe(0); - await expect(runKtxCli(['--project-dir', tempDir, 'dev', 'scan', 'warehouse', '--no-input'], neverIo.io, { scan })) + await expect(runKtxCli(['--project-dir', tempDir, 'scan', 'warehouse', '--no-input'], neverIo.io, { scan })) .resolves.toBe(0); await expect( - runKtxCli(['--project-dir', tempDir, 'dev', 'scan', 'warehouse', '--yes', '--no-input'], conflictIo.io, { + runKtxCli(['--project-dir', tempDir, 'scan', 'warehouse', '--yes', '--no-input'], conflictIo.io, { scan, }), ).resolves.toBe(1); @@ -2131,44 +1718,38 @@ describe('runKtxCli', () => { await expect(runKtxCli(['dev'], testIo.io)).resolves.toBe(0); expect(testIo.stdout()).toContain('Usage: ktx dev [options] [command]'); - expect(testIo.stdout()).toContain('Low-level diagnostics'); - expect(testIo.stdout()).toContain('scan'); - expect(testIo.stdout()).toContain('ingest'); - expect(testIo.stdout()).toContain('mapping'); + expect(testIo.stdout()).toContain('Low-level project initialization'); + expect(testIo.stdout()).toContain('init'); + expect(testIo.stdout()).toContain('runtime'); + expect(testIo.stdout()).not.toContain('scan'); + expect(testIo.stdout()).not.toContain('ingest'); + expect(testIo.stdout()).not.toContain('mapping'); expect(testIo.stdout()).not.toContain('model'); expect(testIo.stdout()).not.toContain('knowledge'); expect(testIo.stderr()).toBe(''); }); - it('prints dev command help without invoking low-level execution', async () => { - for (const [command, expected] of [ - ['scan', ['Usage: ktx dev scan', '--dry-run', 'status', 'report']], - ['ingest', ['Usage: ktx dev ingest', 'run', 'replay']], - ['mapping', ['Usage: ktx dev mapping', 'sync-state', 'validate']], - ] as const) { + it('rejects removed dev command groups without invoking execution', async () => { + for (const command of ['scan', 'ingest', 'mapping']) { const testIo = makeIo(); const scan = vi.fn().mockResolvedValue(0); const sl = vi.fn().mockResolvedValue(0); - await expect(runKtxCli(['dev', command, '--help'], testIo.io, { scan, sl })).resolves.toBe(0); + await expect(runKtxCli(['dev', command], testIo.io, { scan, sl })).resolves.toBe(1); - for (const text of expected) { - expect(testIo.stdout()).toContain(text); - } - expect(testIo.stderr()).toBe(''); + expect(testIo.stderr()).toMatch(/unknown command|error:/); expect(scan).not.toHaveBeenCalled(); expect(sl).not.toHaveBeenCalled(); } }); - it('prints dev scan subcommand help without invoking scan execution', async () => { + it('rejects removed scan subcommands without invoking scan execution', async () => { const testIo = makeIo(); const scan = vi.fn().mockResolvedValue(0); - await expect(runKtxCli(['dev', 'scan', 'report', '--help'], testIo.io, { scan })).resolves.toBe(0); + await expect(runKtxCli(['scan', 'report'], testIo.io, { scan })).resolves.toBe(1); - expect(testIo.stdout()).toContain('Usage: ktx dev scan report [options] '); - expect(testIo.stderr()).toBe(''); + expect(testIo.stderr()).toMatch(/too many arguments|unknown command|error:/); expect(scan).not.toHaveBeenCalled(); }); @@ -2184,8 +1765,8 @@ describe('runKtxCli', () => { const ingest = vi.fn(async () => 0); for (const argv of [ - ['dev', 'ingest', 'run', '--connection-id', 'warehouse', '--adapter', 'fake', '--json', '--plain'], - ['dev', 'ingest', 'status', 'run-1', '--json', '--viz'], + ['ingest', 'run', '--connection-id', 'warehouse', '--adapter', 'fake', '--json', '--plain'], + ['ingest', 'status', 'run-1', '--json', '--viz'], ]) { const testIo = makeIo(); await expect(runKtxCli(argv, testIo.io, { ingest })).resolves.toBe(1); diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index de906ece..2cf9d5b2 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -9,17 +9,6 @@ export { type KtxCliIo, type KtxCliPackageInfo, } from './cli-runtime.js'; -export { runKtxAgent, type KtxAgentArgs } from './agent.js'; -export { - KTX_AGENT_MAX_ROWS_CAP, - createKtxAgentRuntime, - parseAgentMaxRows, - readAgentJsonFile, - writeAgentJson, - writeAgentJsonError, - type KtxAgentRuntime, - type KtxAgentRuntimeDeps, -} from './agent-runtime.js'; export { runKtxSetup, type KtxSetupArgs, type KtxSetupStatus } from './setup.js'; export type { KtxSetupDatabaseDriver, diff --git a/packages/cli/src/ingest.test.ts b/packages/cli/src/ingest.test.ts index cb91788a..410312f5 100644 --- a/packages/cli/src/ingest.test.ts +++ b/packages/cli/src/ingest.test.ts @@ -103,7 +103,7 @@ describe('runKtxIngest', () => { expect(statusIo.stderr()).toBe(''); }); - it('prints provider setup guidance when a skip-llm setup project runs dev ingest', async () => { + it('prints provider setup guidance when a skip-llm setup project runs ingest', async () => { const projectDir = join(tempDir, 'project'); const setupIo = makeIo(); await expect( @@ -164,7 +164,7 @@ describe('runKtxIngest', () => { expect(runIo.stdout()).toBe(''); expect(runIo.stderr()).toContain( - 'ktx dev ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.', + 'ktx ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.', ); expect(runIo.stderr()).toContain( `ktx setup --project-dir ${projectDir} --anthropic-api-key-env ANTHROPIC_API_KEY --anthropic-model claude-sonnet-4-6 --no-input`, @@ -659,7 +659,7 @@ describe('runKtxIngest', () => { ).resolves.toBe(1); expect(io.stderr()).toContain('source-dir uploads are not supported for the Metabase fan-out adapter'); - expect(io.stderr()).not.toContain('ktx dev ingest run requires llm.provider.backend'); + expect(io.stderr()).not.toContain('ktx ingest run requires llm.provider.backend'); expect(io.stdout()).toBe(''); }); diff --git a/packages/cli/src/ingest.ts b/packages/cli/src/ingest.ts index cf7a7aff..4a68edf3 100644 --- a/packages/cli/src/ingest.ts +++ b/packages/cli/src/ingest.ts @@ -518,7 +518,9 @@ export async function runKtxIngest( const project = await loadKtxProject({ projectDir: args.projectDir }); const env = deps.env ?? process.env; if (args.command === 'run') { - const createAdapters = deps.createAdapters ?? createKtxCliLocalIngestAdapters; + const createAdapters = + deps.createAdapters ?? + (deps.runLocalIngest || deps.runLocalMetabaseIngest ? () => [] : createKtxCliLocalIngestAdapters); const executeLocalIngest = deps.runLocalIngest ?? runLocalIngest; const localIngestOptions = deps.localIngestOptions ?? {}; const managedDaemon = managedDaemonOptionsForIngestRun(args, io); @@ -645,7 +647,7 @@ export async function runKtxIngest( throw new Error( args.runId ? `Local ingest run or report "${args.runId}" was not found` - : 'No local ingest reports were found. Run `ktx ingest --all` first.', + : 'No local ingest reports were found. Run `ktx ingest run --connection-id --adapter ` first.', ); } await writeReportRecord(report, args.outputMode, io, { diff --git a/packages/cli/src/knowledge.test.ts b/packages/cli/src/knowledge.test.ts index d7e17605..db794289 100644 --- a/packages/cli/src/knowledge.test.ts +++ b/packages/cli/src/knowledge.test.ts @@ -93,6 +93,65 @@ describe('runKtxKnowledge', () => { expect(searchIo.stdout()).toContain('metrics-revenue'); }); + it('prints wiki list, search, and read as public JSON envelopes', async () => { + const projectDir = join(tempDir, 'project'); + await initKtxProject({ projectDir, projectName: 'warehouse' }); + + await expect( + runKtxKnowledge( + { + command: 'write', + projectDir, + key: 'metrics-revenue', + scope: 'GLOBAL', + userId: 'local', + summary: 'Revenue', + content: 'Revenue is paid order value.', + tags: ['finance'], + refs: [], + slRefs: ['orders'], + }, + makeIo().io, + ), + ).resolves.toBe(0); + + const listIo = makeIo(); + await expect(runKtxKnowledge({ command: 'list', projectDir, userId: 'local', json: true }, listIo.io)).resolves.toBe( + 0, + ); + expect(JSON.parse(listIo.stdout())).toMatchObject({ + kind: 'list', + data: { items: [expect.objectContaining({ key: 'metrics-revenue', summary: 'Revenue' })] }, + meta: { command: 'wiki list' }, + }); + + const searchIo = makeIo(); + await expect( + runKtxKnowledge( + { command: 'search', projectDir, query: 'paid order', userId: 'local', json: true, limit: 5 }, + searchIo.io, + ), + ).resolves.toBe(0); + expect(JSON.parse(searchIo.stdout())).toMatchObject({ + kind: 'list', + data: { items: [expect.objectContaining({ key: 'metrics-revenue', summary: 'Revenue' })] }, + meta: { command: 'wiki search' }, + }); + + const readIo = makeIo(); + await expect( + runKtxKnowledge({ command: 'read', projectDir, key: 'metrics-revenue', userId: 'local', json: true }, readIo.io), + ).resolves.toBe(0); + expect(JSON.parse(readIo.stdout())).toMatchObject({ + kind: 'wiki.page', + data: { + key: 'metrics-revenue', + summary: 'Revenue', + content: 'Revenue is paid order value.', + }, + }); + }); + it('rejects slash-delimited write keys with a flat-key suggestion', async () => { const projectDir = join(tempDir, 'project'); await initKtxProject({ projectDir, projectName: 'warehouse' }); diff --git a/packages/cli/src/knowledge.ts b/packages/cli/src/knowledge.ts index 40cc5372..5c5df1ea 100644 --- a/packages/cli/src/knowledge.ts +++ b/packages/cli/src/knowledge.ts @@ -11,11 +11,12 @@ import { searchLocalKnowledgePages, writeLocalKnowledgePage, } from '@ktx/context/wiki'; +import { writeJsonResult } from './io/print-list.js'; export type KtxKnowledgeArgs = - | { command: 'list'; projectDir: string; userId: string } - | { command: 'read'; projectDir: string; key: string; userId: string } - | { command: 'search'; projectDir: string; query: string; userId: string } + | { command: 'list'; projectDir: string; userId: string; json?: boolean } + | { command: 'read'; projectDir: string; key: string; userId: string; json?: boolean } + | { command: 'search'; projectDir: string; query: string; userId: string; json?: boolean; limit?: number } | { command: 'write'; projectDir: string; @@ -61,6 +62,14 @@ export async function runKtxKnowledge( const project = await loadKtxProject({ projectDir: args.projectDir }); if (args.command === 'list') { const pages = await listLocalKnowledgePages(project, { userId: args.userId }); + if (args.json) { + writeJsonResult(io, { + kind: 'list', + data: { items: pages }, + meta: { command: 'wiki list' }, + }); + return 0; + } for (const page of pages) { io.stdout.write(`${page.scope}\t${page.key}\t${page.summary}\n`); } @@ -71,6 +80,14 @@ export async function runKtxKnowledge( if (!page) { throw new Error(`Knowledge page "${args.key}" was not found`); } + if (args.json) { + writeJsonResult(io, { + kind: 'wiki.page', + data: page, + meta: { command: 'wiki read' }, + }); + return 0; + } io.stdout.write(`# ${page.key}\n\n`); io.stdout.write(`Scope: ${page.scope}\n`); io.stdout.write(`Summary: ${page.summary}\n\n`); @@ -82,7 +99,16 @@ export async function runKtxKnowledge( query: args.query, userId: args.userId, embeddingService: wikiSearchEmbeddingService(project, deps), + limit: args.limit, }); + if (args.json) { + writeJsonResult(io, { + kind: 'list', + data: { items: results }, + meta: { command: 'wiki search' }, + }); + return 0; + } if (results.length === 0) { const pages = await listLocalKnowledgePages(project, { userId: args.userId }); if (pages.length === 0) { diff --git a/packages/cli/src/managed-python-runtime.test.ts b/packages/cli/src/managed-python-runtime.test.ts index f2d9e784..63755ad1 100644 --- a/packages/cli/src/managed-python-runtime.test.ts +++ b/packages/cli/src/managed-python-runtime.test.ts @@ -1,5 +1,5 @@ import { createHash } from 'node:crypto'; -import { mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; @@ -8,7 +8,6 @@ import { doctorManagedPythonRuntime, installManagedPythonRuntime, managedPythonRuntimeLayout, - pruneManagedPythonRuntimes, readManagedPythonRuntimeStatus, verifyRuntimeAsset, type ManagedPythonRuntimeExec, @@ -471,41 +470,3 @@ describe('doctorManagedPythonRuntime', () => { }); }); }); - -describe('pruneManagedPythonRuntimes', () => { - let tempDir: string; - - beforeEach(async () => { - tempDir = await mkdtemp(join(tmpdir(), 'ktx-runtime-prune-')); - }); - - afterEach(async () => { - await rm(tempDir, { recursive: true, force: true }); - }); - - it('removes stale version directories and keeps the current version', async () => { - const runtimeRoot = join(tempDir, 'runtime'); - await mkdir(join(runtimeRoot, '0.1.0'), { recursive: true }); - await mkdir(join(runtimeRoot, '0.2.0'), { recursive: true }); - await writeFile(join(runtimeRoot, 'README.txt'), 'not a runtime directory\n'); - - const result = await pruneManagedPythonRuntimes({ cliVersion: '0.2.0', runtimeRoot }); - - expect(result.removed).toEqual([join(runtimeRoot, '0.1.0')]); - expect(result.kept).toEqual([join(runtimeRoot, '0.2.0')]); - await expect(stat(join(runtimeRoot, '0.1.0'))).rejects.toThrow(); - expect(await readdir(runtimeRoot)).toEqual(['0.2.0', 'README.txt']); - }); - - it('supports dry-run without deleting stale directories', async () => { - const runtimeRoot = join(tempDir, 'runtime'); - await mkdir(join(runtimeRoot, '0.1.0'), { recursive: true }); - await mkdir(join(runtimeRoot, '0.2.0'), { recursive: true }); - - const result = await pruneManagedPythonRuntimes({ cliVersion: '0.2.0', runtimeRoot, dryRun: true }); - - expect(result.removed).toEqual([]); - expect(result.stale).toEqual([join(runtimeRoot, '0.1.0')]); - expect(await readdir(runtimeRoot)).toEqual(['0.1.0', '0.2.0']); - }); -}); diff --git a/packages/cli/src/managed-python-runtime.ts b/packages/cli/src/managed-python-runtime.ts index 251ac7da..563b62f7 100644 --- a/packages/cli/src/managed-python-runtime.ts +++ b/packages/cli/src/managed-python-runtime.ts @@ -1,6 +1,6 @@ import { execFile } from 'node:child_process'; import { createHash } from 'node:crypto'; -import { access, appendFile, mkdir, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises'; +import { access, appendFile, mkdir, readFile, rm, writeFile } from 'node:fs/promises'; import { homedir } from 'node:os'; import { basename, join } from 'node:path'; import { fileURLToPath } from 'node:url'; @@ -107,13 +107,6 @@ export interface ManagedPythonRuntimeDoctorCheck { fix?: string; } -export interface ManagedPythonRuntimePruneResult { - runtimeRoot: string; - stale: string[]; - kept: string[]; - removed: string[]; -} - export const MISSING_UV_RUNTIME_INSTALL_MESSAGE = 'uv is required to install the KTX Python runtime. KTX does not download uv automatically. Install uv, make sure it is on PATH, and retry: ktx dev runtime install --yes'; @@ -441,36 +434,3 @@ export async function doctorManagedPythonRuntime( ); return checks; } - -export async function pruneManagedPythonRuntimes(options: { - cliVersion: string; - runtimeRoot: string; - dryRun?: boolean; -}): Promise { - if (!(await pathExists(options.runtimeRoot))) { - return { runtimeRoot: options.runtimeRoot, stale: [], kept: [], removed: [] }; - } - const entries = await readdir(options.runtimeRoot); - const stale: string[] = []; - const kept: string[] = []; - for (const entry of entries) { - const path = join(options.runtimeRoot, entry); - const info = await stat(path); - if (!info.isDirectory()) { - continue; - } - if (entry === options.cliVersion) { - kept.push(path); - } else { - stale.push(path); - } - } - const removed: string[] = []; - if (options.dryRun !== true) { - for (const path of stale) { - await rm(path, { recursive: true, force: true }); - removed.push(path); - } - } - return { runtimeRoot: options.runtimeRoot, stale, kept, removed }; -} diff --git a/packages/cli/src/next-steps.test.ts b/packages/cli/src/next-steps.test.ts index fd8d8216..b4706d72 100644 --- a/packages/cli/src/next-steps.test.ts +++ b/packages/cli/src/next-steps.test.ts @@ -25,12 +25,8 @@ describe('KTX demo next steps', () => { it('uses supported final public commands', () => { expect(KTX_NEXT_STEP_COMMANDS).toEqual([ { - command: 'ktx agent context --json', - description: 'Verify the project context your agent can read', - }, - { - command: 'ktx agent tools --json', - description: 'List direct CLI tools available to agents', + command: 'ktx status --json', + description: 'Verify project setup and context readiness', }, { command: 'ktx sl list', @@ -46,8 +42,8 @@ describe('KTX demo next steps', () => { it('uses only the direct CLI route for agent verification', () => { const commands = KTX_NEXT_STEP_COMMANDS.map((step) => step.command); - expect(commands).toContain('ktx agent context --json'); - expect(commands).toContain('ktx agent tools --json'); + expect(commands).not.toContain('ktx agent context --json'); + expect(commands).toContain('ktx status --json'); expect(commands).not.toContain('ktx serve --mcp stdio --user-id local'); }); @@ -64,8 +60,8 @@ describe('KTX demo next steps', () => { it('does not advertise removed Commander migration commands', () => { const rendered = formatNextStepLines().join('\n'); - expect(rendered).toContain('ktx agent tools --json'); - expect(rendered).toContain('ktx agent context --json'); + expect(rendered).toContain('ktx status --json'); + expect(rendered).not.toContain('ktx agent'); expect(rendered).toContain('ktx sl list'); expect(rendered).toContain('ktx wiki list'); @@ -109,7 +105,8 @@ describe('KTX demo next steps', () => { }).join('\n'); expect(rendered).toContain('KTX context is ready for agents.'); - expect(rendered).toContain('ktx agent context --json'); + expect(rendered).toContain('ktx status --json'); + expect(rendered).not.toContain('ktx agent'); expect(rendered).not.toContain('ktx serve --mcp stdio --user-id local'); expect(rendered).not.toContain('Build KTX context next.'); }); diff --git a/packages/cli/src/next-steps.ts b/packages/cli/src/next-steps.ts index db85da66..ee7535d7 100644 --- a/packages/cli/src/next-steps.ts +++ b/packages/cli/src/next-steps.ts @@ -11,12 +11,8 @@ export const KTX_CONTEXT_BUILD_COMMANDS = [ export const KTX_NEXT_STEP_DIRECT_COMMANDS = [ { - command: 'ktx agent context --json', - description: 'Verify the project context your agent can read', - }, - { - command: 'ktx agent tools --json', - description: 'List direct CLI tools available to agents', + command: 'ktx status --json', + description: 'Verify project setup and context readiness', }, { command: 'ktx sl list', diff --git a/packages/cli/src/project-dir.test.ts b/packages/cli/src/project-dir.test.ts index 6e3ca901..c0022d4d 100644 --- a/packages/cli/src/project-dir.test.ts +++ b/packages/cli/src/project-dir.test.ts @@ -33,11 +33,9 @@ describe('project directory defaults', () => { const connection = vi.fn(async () => 0); const doctor = vi.fn(async () => 0); const ingest = vi.fn(async () => 0); - const publicIngest = vi.fn(async () => 0); const scan = vi.fn(async () => 0); const setup = vi.fn(async () => 0); - const agent = vi.fn(async () => 0); - const deps: KtxCliDeps = { agent, connection, doctor, ingest, publicIngest, scan, setup }; + const deps: KtxCliDeps = { connection, doctor, ingest, scan, setup }; const cases: Array<{ argv: string[]; @@ -59,8 +57,8 @@ describe('project directory defaults', () => { }, { argv: ['ingest', 'status', 'run-1'], - spy: publicIngest, - expected: { command: 'status', projectDir: '/tmp/ktx-env-project', runId: 'run-1' }, + spy: ingest, + expected: { command: 'status', projectDir: '/tmp/ktx-env-project', runId: 'run-1', outputMode: 'plain' }, expectedStderr: 'Project: /tmp/ktx-env-project\n', }, { @@ -70,17 +68,11 @@ describe('project directory defaults', () => { expectedStderr: 'Project: /tmp/ktx-env-project\n', }, { - argv: ['dev', 'scan', 'warehouse'], + argv: ['scan', 'warehouse'], spy: scan, expected: { command: 'run', projectDir: '/tmp/ktx-env-project', connectionId: 'warehouse' }, expectedStderr: 'Project: /tmp/ktx-env-project\n', }, - { - argv: ['agent', 'tools', '--json'], - spy: agent, - expected: { command: 'tools', projectDir: '/tmp/ktx-env-project' }, - expectedStderr: '', - }, ]; for (const item of cases) { @@ -95,16 +87,16 @@ describe('project directory defaults', () => { process.env.KTX_PROJECT_DIR = '/tmp/ktx-env-project'; const scan = vi.fn(async () => 0); - const publicIngest = vi.fn(async () => 0); + const ingest = vi.fn(async () => 0); const scanIo = makeIo(); const ingestIo = makeIo(); await expect( - runKtxCli(['--project-dir', '/tmp/ktx-explicit-project', 'dev', 'scan', 'warehouse'], scanIo.io, { scan }), + runKtxCli(['--project-dir', '/tmp/ktx-explicit-project', 'scan', 'warehouse'], scanIo.io, { scan }), ).resolves.toBe(0); await expect( runKtxCli(['ingest', 'status', 'run-1', '--project-dir=/tmp/ktx-explicit-project'], ingestIo.io, { - publicIngest, + ingest, }), ).resolves.toBe(0); @@ -112,7 +104,7 @@ describe('project directory defaults', () => { expect.objectContaining({ command: 'run', projectDir: '/tmp/ktx-explicit-project' }), scanIo.io, ); - expect(publicIngest).toHaveBeenCalledWith( + expect(ingest).toHaveBeenCalledWith( expect.objectContaining({ command: 'status', projectDir: '/tmp/ktx-explicit-project' }), ingestIo.io, ); @@ -139,7 +131,7 @@ describe('project directory defaults', () => { try { process.chdir(nestedDir); - await expect(runKtxCli(['dev', 'scan', 'warehouse'], testIo.io, { scan })).resolves.toBe(0); + await expect(runKtxCli(['scan', 'warehouse'], testIo.io, { scan })).resolves.toBe(0); } finally { process.chdir(originalCwd); await rm(root, { recursive: true, force: true }); diff --git a/packages/cli/src/public-ingest.test.ts b/packages/cli/src/public-ingest.test.ts index 13d8f364..1c133a19 100644 --- a/packages/cli/src/public-ingest.test.ts +++ b/packages/cli/src/public-ingest.test.ts @@ -57,7 +57,7 @@ describe('buildPublicIngestPlan', () => { driver: 'notion', operation: 'source-ingest', adapter: 'notion', - debugCommand: 'ktx dev ingest run --connection-id docs --adapter notion --debug', + debugCommand: 'ktx ingest run --connection-id docs --adapter notion --debug', steps: ['source-ingest', 'memory-update'], }, { @@ -65,7 +65,7 @@ describe('buildPublicIngestPlan', () => { driver: 'metabase', operation: 'source-ingest', adapter: 'metabase', - debugCommand: 'ktx dev ingest run --connection-id prod_metabase --adapter metabase --debug', + debugCommand: 'ktx ingest run --connection-id prod_metabase --adapter metabase --debug', steps: ['source-ingest', 'memory-update'], }, ], @@ -76,7 +76,7 @@ describe('buildPublicIngestPlan', () => { const project = projectWithConnections({ warehouse: { driver: 'postgres' } }); expect(() => buildPublicIngestPlan(project, { projectDir: '/tmp/project', all: false })).toThrow( - 'ktx ingest requires or --all in this release', + 'Context build requires a connection id or all targets', ); }); diff --git a/packages/cli/src/public-ingest.ts b/packages/cli/src/public-ingest.ts index 3c0ede75..f8296177 100644 --- a/packages/cli/src/public-ingest.ts +++ b/packages/cli/src/public-ingest.ts @@ -6,9 +6,9 @@ import { profileMark } from './startup-profile.js'; profileMark('module:public-ingest'); -export type KtxPublicIngestStepName = 'scan' | 'source-ingest' | 'enrich' | 'memory-update'; -export type KtxPublicIngestStepStatus = 'done' | 'skipped' | 'failed' | 'not-run'; -export type KtxPublicIngestInputMode = 'auto' | 'disabled'; +type KtxPublicIngestStepName = 'scan' | 'source-ingest' | 'enrich' | 'memory-update'; +type KtxPublicIngestStepStatus = 'done' | 'skipped' | 'failed' | 'not-run'; +type KtxPublicIngestInputMode = 'auto' | 'disabled'; export type KtxPublicIngestArgs = | { @@ -107,7 +107,7 @@ function targetForConnection(connectionId: string, connection: KtxProjectConnect operation: 'source-ingest', adapter, ...(sourceDir ? { sourceDir } : {}), - debugCommand: `ktx dev ingest run --connection-id ${connectionId} --adapter ${adapter} --debug`, + debugCommand: `ktx ingest run --connection-id ${connectionId} --adapter ${adapter} --debug`, steps: ['source-ingest', 'memory-update'], }; } @@ -130,7 +130,7 @@ export function buildPublicIngestPlan( args: { projectDir: string; targetConnectionId?: string; all: boolean }, ): KtxPublicIngestPlan { if (!args.all && !args.targetConnectionId) { - throw new Error('ktx ingest requires or --all in this release'); + throw new Error('Context build requires a connection id or all targets'); } const entries = Object.entries(project.config.connections).sort(([a], [b]) => a.localeCompare(b)); diff --git a/packages/cli/src/runtime.test.ts b/packages/cli/src/runtime.test.ts index a10b5b36..8151a4b3 100644 --- a/packages/cli/src/runtime.test.ts +++ b/packages/cli/src/runtime.test.ts @@ -257,7 +257,7 @@ describe('runKtxRuntime', () => { expect(io.stderr()).toContain('process scan: ps failed'); }); - it('prints runtime status as JSON', async () => { + it('prints runtime status and doctor checks as JSON with doctor-style exit status', async () => { const io = makeIo(); const deps: KtxRuntimeDeps = { readStatus: vi.fn(async (): Promise => ({ @@ -279,60 +279,41 @@ describe('runKtxRuntime', () => { daemonStderrPath: '/runtime/0.2.0/daemon.stderr.log', }, })), - }; - - await expect(runKtxRuntime({ command: 'status', cliVersion: '0.2.0', json: true }, io.io, deps)).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ - kind: 'missing', - detail: 'No runtime manifest at /runtime/0.2.0/manifest.json', - layout: { runtimeRoot: '/runtime' }, - }); - }); - - it('returns failure for doctor when any check fails', async () => { - const io = makeIo(); - const deps: KtxRuntimeDeps = { doctorRuntime: vi.fn(async (): Promise => [ { id: 'uv', label: 'uv', status: 'pass', detail: 'uv 0.9.5' }, + { id: 'asset', label: 'Bundled Python wheel', status: 'pass', detail: '/assets/python/runtime.whl' }, { id: 'runtime', label: 'Managed Python runtime', status: 'fail', - detail: 'No runtime manifest', + detail: 'No runtime manifest at /runtime/0.2.0/manifest.json', fix: 'Run: ktx dev runtime install --yes', }, ]), }; - await expect(runKtxRuntime({ command: 'doctor', cliVersion: '0.2.0', json: false }, io.io, deps)).resolves.toBe(1); + await expect(runKtxRuntime({ command: 'status', cliVersion: '0.2.0', json: true }, io.io, deps)).resolves.toBe(1); - expect(io.stdout()).toContain('PASS uv: uv 0.9.5'); - expect(io.stdout()).toContain('FAIL Managed Python runtime: No runtime manifest'); - expect(io.stdout()).toContain('Fix: Run: ktx dev runtime install --yes'); + expect(JSON.parse(io.stdout())).toMatchObject({ + kind: 'missing', + detail: 'No runtime manifest at /runtime/0.2.0/manifest.json', + layout: { runtimeRoot: '/runtime' }, + checks: [ + { id: 'uv', status: 'pass' }, + { id: 'asset', status: 'pass' }, + { id: 'runtime', status: 'fail' }, + ], + }); + expect(deps.readStatus).toHaveBeenCalledWith({ cliVersion: '0.2.0' }); + expect(deps.doctorRuntime).toHaveBeenCalledWith({ cliVersion: '0.2.0' }); }); - it('requires --yes before pruning stale runtime directories', async () => { - const io = makeIo(); - const deps: KtxRuntimeDeps = { - pruneRuntime: vi.fn(async () => { - throw new Error('should not prune without --yes'); - }), - }; - - await expect(runKtxRuntime({ command: 'prune', cliVersion: '0.2.0', dryRun: false, yes: false }, io.io, deps)) - .resolves.toBe(1); - - expect(io.stderr()).toContain('Refusing to prune without --yes'); - expect(deps.pruneRuntime).not.toHaveBeenCalled(); - }); - - it('prints stale directories during prune dry-run', async () => { + it('prints runtime status and doctor checks in plain output', async () => { const io = makeIo(); const deps: KtxRuntimeDeps = { readStatus: vi.fn(async (): Promise => ({ - kind: 'missing', - detail: 'No runtime manifest at /runtime/0.2.0/manifest.json', + kind: 'ready', + detail: 'Runtime ready at /runtime/0.2.0', layout: { cliVersion: '0.2.0', runtimeRoot: '/runtime', @@ -348,19 +329,43 @@ describe('runKtxRuntime', () => { daemonStdoutPath: '/runtime/0.2.0/daemon.stdout.log', daemonStderrPath: '/runtime/0.2.0/daemon.stderr.log', }, + manifest: { + schemaVersion: 1, + cliVersion: '0.2.0', + installedAt: '2026-05-11T00:00:00.000Z', + asset: { + schemaVersion: 1, + distributionName: 'kaelio-ktx', + normalizedName: 'kaelio_ktx', + version: '0.1.0', + wheel: { + file: 'kaelio_ktx-0.1.0-py3-none-any.whl', + sha256: 'a'.repeat(64), + bytes: 10, + }, + }, + features: ['core'], + python: { + executable: '/runtime/0.2.0/.venv/bin/python', + daemonExecutable: '/runtime/0.2.0/.venv/bin/ktx-daemon', + }, + installLog: '/runtime/0.2.0/install.log', + }, })), - pruneRuntime: vi.fn(async () => ({ - runtimeRoot: '/runtime', - stale: ['/runtime/0.1.0'], - kept: ['/runtime/0.2.0'], - removed: [], - })), + doctorRuntime: vi.fn(async (): Promise => [ + { id: 'uv', label: 'uv', status: 'pass', detail: 'uv 0.9.5' }, + { id: 'asset', label: 'Bundled Python wheel', status: 'pass', detail: '/assets/python/runtime.whl' }, + { id: 'runtime', label: 'Managed Python runtime', status: 'pass', detail: 'Runtime ready at /runtime/0.2.0' }, + ]), }; - await expect(runKtxRuntime({ command: 'prune', cliVersion: '0.2.0', dryRun: true, yes: false }, io.io, deps)) - .resolves.toBe(0); + await expect(runKtxRuntime({ command: 'status', cliVersion: '0.2.0', json: false }, io.io, deps)).resolves.toBe(0); - expect(io.stdout()).toContain('Stale KTX Python runtimes'); - expect(io.stdout()).toContain('/runtime/0.1.0'); + expect(io.stdout()).toContain('KTX Python runtime'); + expect(io.stdout()).toContain('status: ready'); + expect(io.stdout()).toContain('KTX Python runtime checks'); + expect(io.stdout()).toContain('PASS uv: uv 0.9.5'); + expect(io.stdout()).toContain('PASS Managed Python runtime: Runtime ready at /runtime/0.2.0'); + expect(io.stderr()).toBe(''); }); }); diff --git a/packages/cli/src/runtime.ts b/packages/cli/src/runtime.ts index ed8cca22..8bb3fc7c 100644 --- a/packages/cli/src/runtime.ts +++ b/packages/cli/src/runtime.ts @@ -10,14 +10,12 @@ import { import { doctorManagedPythonRuntime, installManagedPythonRuntime, - pruneManagedPythonRuntimes, readManagedPythonRuntimeStatus, type KtxRuntimeFeature, type ManagedPythonRuntimeDoctorCheck, type ManagedPythonRuntimeInstallOptions, type ManagedPythonRuntimeInstallResult, type ManagedPythonRuntimeLayoutOptions, - type ManagedPythonRuntimePruneResult, type ManagedPythonRuntimeStatus, } from './managed-python-runtime.js'; @@ -25,9 +23,7 @@ export type KtxRuntimeArgs = | { command: 'install'; cliVersion: string; feature: KtxRuntimeFeature; force: boolean } | { command: 'start'; cliVersion: string; feature: KtxRuntimeFeature; force: boolean } | { command: 'stop'; cliVersion: string; all: boolean } - | { command: 'status'; cliVersion: string; json: boolean } - | { command: 'doctor'; cliVersion: string; json: boolean } - | { command: 'prune'; cliVersion: string; dryRun: boolean; yes: boolean }; + | { command: 'status'; cliVersion: string; json: boolean }; export interface KtxRuntimeDeps { installRuntime?: (options: ManagedPythonRuntimeInstallOptions) => Promise; @@ -40,11 +36,6 @@ export interface KtxRuntimeDeps { stopAllDaemons?: (options: { cliVersion: string }) => Promise; readStatus?: (options: ManagedPythonRuntimeLayoutOptions) => Promise; doctorRuntime?: (options: ManagedPythonRuntimeLayoutOptions) => Promise; - pruneRuntime?: (options: { - cliVersion: string; - runtimeRoot: string; - dryRun?: boolean; - }) => Promise; } function writeJson(io: KtxCliIo, value: unknown): void { @@ -149,8 +140,8 @@ function writeStatus(io: KtxCliIo, status: ManagedPythonRuntimeStatus): void { } } -function writeDoctor(io: KtxCliIo, checks: ManagedPythonRuntimeDoctorCheck[]): void { - io.stdout.write('KTX Python runtime doctor\n'); +function writeRuntimeChecks(io: KtxCliIo, checks: ManagedPythonRuntimeDoctorCheck[]): void { + io.stdout.write('KTX Python runtime checks\n'); for (const check of checks) { io.stdout.write(`${check.status.toUpperCase()} ${check.label}: ${check.detail}\n`); if (check.fix) { @@ -159,15 +150,8 @@ function writeDoctor(io: KtxCliIo, checks: ManagedPythonRuntimeDoctorCheck[]): v } } -function writePrune(io: KtxCliIo, result: ManagedPythonRuntimePruneResult, dryRun: boolean): void { - if (result.stale.length === 0) { - io.stdout.write(`No stale KTX Python runtimes found under ${result.runtimeRoot}\n`); - return; - } - io.stdout.write(dryRun ? 'Stale KTX Python runtimes\n' : 'Removed stale KTX Python runtimes\n'); - for (const path of dryRun ? result.stale : result.removed) { - io.stdout.write(`${path}\n`); - } +function hasRuntimeCheckFailures(checks: ManagedPythonRuntimeDoctorCheck[]): boolean { + return checks.some((check) => check.status === 'fail'); } export async function runKtxRuntime( @@ -210,37 +194,19 @@ export async function runKtxRuntime( } if (args.command === 'status') { const readStatus = deps.readStatus ?? readManagedPythonRuntimeStatus; - const status = await readStatus({ cliVersion: args.cliVersion }); - if (args.json) { - writeJson(io, status); - } else { - writeStatus(io, status); - } - return 0; - } - if (args.command === 'doctor') { const doctorRuntime = deps.doctorRuntime ?? doctorManagedPythonRuntime; + const status = await readStatus({ cliVersion: args.cliVersion }); const checks = await doctorRuntime({ cliVersion: args.cliVersion }); if (args.json) { - writeJson(io, { checks }); + writeJson(io, { ...status, checks }); } else { - writeDoctor(io, checks); + writeStatus(io, status); + writeRuntimeChecks(io, checks); } - return checks.some((check) => check.status === 'fail') ? 1 : 0; + return hasRuntimeCheckFailures(checks) ? 1 : 0; } - if (!args.dryRun && !args.yes) { - io.stderr.write('Refusing to prune without --yes. Preview with: ktx dev runtime prune --dry-run\n'); - return 1; - } - const status = await (deps.readStatus ?? readManagedPythonRuntimeStatus)({ cliVersion: args.cliVersion }); - const pruneRuntime = deps.pruneRuntime ?? pruneManagedPythonRuntimes; - const result = await pruneRuntime({ - cliVersion: args.cliVersion, - runtimeRoot: status.layout.runtimeRoot, - dryRun: args.dryRun, - }); - writePrune(io, result, args.dryRun); - return 0; + const _exhaustive: never = args; + return _exhaustive; } catch (error) { io.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`); return 1; diff --git a/packages/cli/src/scan.test.ts b/packages/cli/src/scan.test.ts index 152c8b3a..74d52f35 100644 --- a/packages/cli/src/scan.test.ts +++ b/packages/cli/src/scan.test.ts @@ -1,18 +1,12 @@ -import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; +import type { SourceAdapter } from '@ktx/context/ingest'; import { initKtxProject } from '@ktx/context/project'; import type { - ApplyLocalScanRelationshipReviewDecisionsResult, - ExportLocalRelationshipFeedbackLabelsResult, - KtxRelationshipFeedbackCalibrationReport, - KtxRelationshipThresholdAdviceReport, KtxScanReport, LocalScanRunResult, - LocalScanStatusResponse, - ReadLocalScanRelationshipArtifactsResult, RunLocalScanOptions, - WriteLocalScanRelationshipReviewDecisionResult, } from '@ktx/context/scan'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { createCliScanProgress, runKtxScan } from './scan.js'; @@ -190,6 +184,32 @@ function makeIo(options: { isTTY?: boolean } = {}) { }; } +function fakeLiveDatabaseAdapter( + createIntrospection: (options: { connections: unknown }) => { + extractSchema: (connectionId: string) => Promise; + }, +): SourceAdapter { + return { + source: 'live-database', + skillNames: [], + async detect() { + return true; + }, + async fetch(_pullConfig: unknown, stagedDir: string, ctx: { connectionId: string }) { + await mkdir(stagedDir, { recursive: true }); + const schema = await createIntrospection({ connections: {} }).extractSchema(ctx.connectionId); + await writeFile( + join(stagedDir, 'connection.json'), + JSON.stringify({ connectionId: ctx.connectionId, schema }, null, 2), + 'utf-8', + ); + }, + async chunk() { + return { workUnits: [] }; + }, + }; +} + const report: KtxScanReport = { connectionId: 'warehouse', driver: 'postgres', @@ -285,6 +305,7 @@ const reportWithAttention: KtxScanReport = { describe('runKtxScan', () => { let tempDir: string; + const noLocalIngestAdapters = () => []; beforeEach(async () => { tempDir = await mkdtemp(join(tmpdir(), 'ktx-cli-scan-')); @@ -322,7 +343,7 @@ describe('runKtxScan', () => { databaseIntrospectionUrl: 'http://127.0.0.1:8765', }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); @@ -346,10 +367,9 @@ describe('runKtxScan', () => { expect(io.stdout()).toContain('Artifacts\n'); expect(io.stdout()).toContain('Report: raw-sources/warehouse/live-database/sync-1/scan-report.json'); expect(io.stdout()).toContain('Next:\n'); - expect(io.stdout()).toContain('ktx dev scan status --project-dir '); - expect(io.stdout()).toContain(' scan-run-1\n'); - expect(io.stdout()).toContain('ktx dev scan report --project-dir '); - expect(io.stdout()).toContain(' scan-run-1\n'); + expect(io.stdout()).toContain('ktx status --project-dir '); + expect(io.stdout()).not.toContain('ktx dev scan status'); + expect(io.stdout()).not.toContain('ktx dev scan report'); expect(io.stdout()).not.toContain('\u001b['); expect(io.stdout()).not.toContain('✓'); expect(io.stdout()).not.toContain('+1'); @@ -426,7 +446,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); @@ -490,7 +510,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); @@ -534,7 +554,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ); expect({ exitCode, stderr: io.stderr() }).toEqual({ exitCode: 0, stderr: '' }); } finally { @@ -664,7 +684,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); @@ -706,7 +726,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); } finally { @@ -762,7 +782,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); } finally { @@ -777,1034 +797,6 @@ describe('runKtxScan', () => { expect(io.stdout()).not.toContain('\u001b['); }); - it('prints status and human report output by default', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const status: LocalScanStatusResponse = { - runId: 'scan-run-1', - status: 'done', - done: true, - connectionId: 'warehouse', - mode: 'structural', - dryRun: false, - syncId: 'sync-1', - progress: 1, - startedAt: '2026-04-29T09:00:00.000Z', - completedAt: '2026-04-29T09:00:01.000Z', - reportPath: 'raw-sources/warehouse/live-database/sync-1/scan-report.json', - warnings: [], - }; - const io = makeIo(); - - await expect( - runKtxScan({ command: 'status', projectDir: tempDir, runId: 'scan-run-1' }, io.io, { - getLocalScanStatus: vi.fn().mockResolvedValue(status), - }), - ).resolves.toBe(0); - expect(io.stdout()).toContain('Run: scan-run-1'); - expect(io.stdout()).toContain('Status: done'); - - const reportIo = makeIo(); - await expect( - runKtxScan({ command: 'report', projectDir: tempDir, runId: 'scan-run-1', json: false }, reportIo.io, { - getLocalScanReport: vi.fn().mockResolvedValue(report), - }), - ).resolves.toBe(0); - expect(reportIo.stdout()).toContain('KTX scan report\n'); - expect(reportIo.stdout()).toContain('Run: scan-run-1'); - expect(reportIo.stdout()).toContain('What changed\n'); - expect(() => JSON.parse(reportIo.stdout())).toThrow(); - }); - - it('prints raw report JSON when requested', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const reportIo = makeIo(); - - await expect( - runKtxScan({ command: 'report', projectDir: tempDir, runId: 'scan-run-1', json: true }, reportIo.io, { - getLocalScanReport: vi.fn().mockResolvedValue(report), - }), - ).resolves.toBe(0); - - expect(JSON.parse(reportIo.stdout())).toMatchObject({ runId: 'scan-run-1', connectionId: 'warehouse' }); - }); - - it('prints review relationship artifacts in human form', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const reviewReport: KtxScanReport = { - ...reportWithAttention, - runId: 'scan-run-review', - syncId: 'sync-review', - relationships: { accepted: 0, review: 1, rejected: 1, skipped: 0 }, - artifactPaths: { - ...reportWithAttention.artifactPaths, - reportPath: 'raw-sources/warehouse/live-database/sync-review/scan-report.json', - enrichmentArtifacts: [ - 'raw-sources/warehouse/live-database/sync-review/enrichment/relationships.json', - 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-diagnostics.json', - 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-profile.json', - ], - }, - }; - const relationshipArtifacts: ReadLocalScanRelationshipArtifactsResult = { - runId: 'scan-run-review', - connectionId: 'warehouse', - syncId: 'sync-review', - report: reviewReport, - relationships: { - connectionId: 'warehouse', - accepted: [], - review: [ - { - id: 'orders:orders.customer_id->customers:customers.id', - status: 'review', - source: 'deterministic_name', - from: { - tableId: 'orders', - columnIds: ['orders.customer_id'], - table: { catalog: null, db: 'public', name: 'orders' }, - columns: ['customer_id'], - }, - to: { - tableId: 'customers', - columnIds: ['customers.id'], - table: { catalog: null, db: 'public', name: 'customers' }, - columns: ['id'], - }, - relationshipType: 'many_to_one', - confidence: 0.62, - pkScore: 0.91, - fkScore: 0.62, - score: 0.62, - evidence: { sources: ['table_suffix'] }, - validation: { status: 'unavailable' }, - graph: { reasons: ['validation_unavailable_review_only'] }, - reasons: ['validation_unavailable_review_only', 'fk_score_review'], - }, - ], - rejected: [ - { - id: 'orders:orders.note_id->notes:notes.id', - status: 'rejected', - source: 'deterministic_name', - from: { - tableId: 'orders', - columnIds: ['orders.note_id'], - table: { catalog: null, db: 'public', name: 'orders' }, - columns: ['note_id'], - }, - to: { - tableId: 'notes', - columnIds: ['notes.id'], - table: { catalog: null, db: 'public', name: 'notes' }, - columns: ['id'], - }, - relationshipType: 'many_to_one', - confidence: 0.2, - pkScore: 0.4, - fkScore: 0.2, - score: 0.2, - evidence: { sources: ['exact_column_match'] }, - validation: { status: 'failed' }, - graph: { reasons: ['low_source_coverage'] }, - reasons: ['low_source_coverage'], - }, - ], - skipped: [], - }, - diagnostics: { - connectionId: 'warehouse', - generatedAt: '2026-05-07T10:00:00.000Z', - summary: { accepted: 0, review: 1, rejected: 1, skipped: 0 }, - noAcceptedReason: 'relationship candidates require review before manifest writes', - candidateCountsBySource: { deterministic_name: 2 }, - validation: { available: false, sqlAvailable: false, queryCount: 0 }, - thresholds: { acceptThreshold: 0.85, reviewThreshold: 0.55 }, - policy: { - validationRequiredForManifest: true, - maxCandidatesPerColumn: 25, - profileSampleRows: 10000, - validationConcurrency: 4, - }, - warnings: [], - profileWarnings: [], - }, - profile: { - connectionId: 'warehouse', - driver: 'sqlite', - sqlAvailable: false, - tables: [], - columns: {}, - queryCount: 0, - warnings: ['KTX scan connector cannot run read-only SQL relationship validation'], - }, - paths: { - relationships: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationships.json', - diagnostics: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-diagnostics.json', - profile: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-profile.json', - }, - }; - const readLocalScanRelationshipArtifacts = vi.fn(async () => relationshipArtifacts); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationships', - projectDir: tempDir, - runId: 'scan-run-review', - status: 'review', - json: false, - limit: 10, - }, - io.io, - { readLocalScanRelationshipArtifacts }, - ), - ).resolves.toBe(0); - - expect(readLocalScanRelationshipArtifacts).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - 'scan-run-review', - ); - - expect(io.stdout()).toContain('KTX relationship artifacts'); - expect(io.stdout()).toContain('Run: scan-run-review'); - expect(io.stdout()).toContain('Summary: accepted=0 review=1 rejected=1 skipped=0'); - expect(io.stdout()).toContain('Reason: relationship candidates require review before manifest writes'); - expect(io.stdout()).toContain('Review relationships (1)'); - expect(io.stdout()).toContain('orders.customer_id -> customers.id'); - expect(io.stdout()).toContain( - 'type=many_to_one source=deterministic_name confidence=0.62 pkScore=0.91 fkScore=0.62', - ); - expect(io.stdout()).toContain('reasons=validation_unavailable_review_only, fk_score_review'); - expect(io.stdout()).toContain('relationships.json'); - }); - - it('prints filtered relationship artifacts as JSON', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const jsonReport: KtxScanReport = { - ...reportWithAttention, - runId: 'scan-run-json', - syncId: 'sync-json', - artifactPaths: { - ...reportWithAttention.artifactPaths, - reportPath: 'raw-sources/warehouse/live-database/sync-json/scan-report.json', - enrichmentArtifacts: ['raw-sources/warehouse/live-database/sync-json/enrichment/relationships.json'], - }, - }; - const relationshipArtifacts: ReadLocalScanRelationshipArtifactsResult = { - runId: 'scan-run-json', - connectionId: 'warehouse', - syncId: 'sync-json', - report: jsonReport, - relationships: { - connectionId: 'warehouse', - accepted: [], - review: [], - rejected: [], - skipped: [{ relationshipId: 'composite:orders', reason: 'composite_key_width_limit' }], - }, - diagnostics: null, - profile: null, - paths: { - relationships: 'raw-sources/warehouse/live-database/sync-json/enrichment/relationships.json', - diagnostics: null, - profile: null, - }, - }; - const readLocalScanRelationshipArtifacts = vi.fn(async () => relationshipArtifacts); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationships', - projectDir: tempDir, - runId: 'scan-run-json', - status: 'skipped', - json: true, - limit: 25, - }, - io.io, - { readLocalScanRelationshipArtifacts }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ - runId: 'scan-run-json', - connectionId: 'warehouse', - status: 'skipped', - relationships: { - accepted: [], - review: [], - rejected: [], - skipped: [{ relationshipId: 'composite:orders', reason: 'composite_key_width_limit' }], - }, - }); - }); - - it('records an accepted relationship review decision in human form', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const decisionResult: WriteLocalScanRelationshipReviewDecisionResult = { - path: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-review-decisions.json', - decision: { - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'accepted', - previousStatus: 'review', - connectionId: 'warehouse', - runId: 'scan-run-review', - syncId: 'sync-review', - decidedAt: '2026-05-07T12:00:00.000Z', - reviewer: 'Andrey', - note: 'Looks right', - from: { - tableId: 'orders', - columnIds: ['orders.customer_id'], - table: { catalog: null, db: 'public', name: 'orders' }, - columns: ['customer_id'], - }, - to: { - tableId: 'customers', - columnIds: ['customers.id'], - table: { catalog: null, db: 'public', name: 'customers' }, - columns: ['id'], - }, - relationshipType: 'many_to_one', - source: 'deterministic_name', - score: 0.62, - confidence: 0.62, - pkScore: 0.91, - fkScore: 0.62, - reasons: ['fk_score_review'], - }, - artifact: { - connectionId: 'warehouse', - runId: 'scan-run-review', - syncId: 'sync-review', - generatedAt: '2026-05-07T12:00:00.000Z', - decisions: [], - }, - }; - const writeLocalScanRelationshipReviewDecision = vi.fn(async () => decisionResult); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipDecision', - projectDir: tempDir, - runId: 'scan-run-review', - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'accepted', - reviewer: 'Andrey', - note: 'Looks right', - json: false, - }, - io.io, - { writeLocalScanRelationshipReviewDecision }, - ), - ).resolves.toBe(0); - - expect(writeLocalScanRelationshipReviewDecision).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - { - runId: 'scan-run-review', - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'accepted', - reviewer: 'Andrey', - note: 'Looks right', - }, - ); - expect(io.stdout()).toContain('Recorded relationship decision'); - expect(io.stdout()).toContain('Decision: accepted'); - expect(io.stdout()).toContain('Candidate: orders:orders.customer_id->customers:customers.id'); - expect(io.stdout()).toContain('Previous status: review'); - expect(io.stdout()).toContain( - 'Path: raw-sources/warehouse/live-database/sync-review/enrichment/relationship-review-decisions.json', - ); - }); - - it('records a rejected relationship review decision as JSON', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const decisionResult: WriteLocalScanRelationshipReviewDecisionResult = { - path: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-review-decisions.json', - decision: { - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'rejected', - previousStatus: 'review', - connectionId: 'warehouse', - runId: 'scan-run-review', - syncId: 'sync-review', - decidedAt: '2026-05-07T12:00:00.000Z', - reviewer: 'Andrey', - note: null, - from: { - tableId: 'orders', - columnIds: ['orders.customer_id'], - table: { catalog: null, db: 'public', name: 'orders' }, - columns: ['customer_id'], - }, - to: { - tableId: 'customers', - columnIds: ['customers.id'], - table: { catalog: null, db: 'public', name: 'customers' }, - columns: ['id'], - }, - relationshipType: 'many_to_one', - source: 'deterministic_name', - score: 0.62, - confidence: 0.62, - pkScore: 0.91, - fkScore: 0.62, - reasons: ['fk_score_review'], - }, - artifact: { - connectionId: 'warehouse', - runId: 'scan-run-review', - syncId: 'sync-review', - generatedAt: '2026-05-07T12:00:00.000Z', - decisions: [], - }, - }; - const writeLocalScanRelationshipReviewDecision = vi.fn(async () => decisionResult); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipDecision', - projectDir: tempDir, - runId: 'scan-run-review', - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'rejected', - reviewer: 'ktx', - note: null, - json: true, - }, - io.io, - { writeLocalScanRelationshipReviewDecision }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ - path: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-review-decisions.json', - decision: { - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'rejected', - previousStatus: 'review', - }, - }); - }); - - it('reports missing scan runs when recording relationship decisions', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const writeLocalScanRelationshipReviewDecision = vi.fn(async () => null); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipDecision', - projectDir: tempDir, - runId: 'missing-run', - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'accepted', - reviewer: 'ktx', - note: null, - json: false, - }, - io.io, - { writeLocalScanRelationshipReviewDecision }, - ), - ).resolves.toBe(1); - - expect(io.stderr()).toContain('Scan run "missing-run" was not found'); - }); - - it('applies accepted relationship review decisions with human output', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const applyLocalScanRelationshipReviewDecisions = vi.fn( - async (): Promise => ({ - runId: 'scan-run-a', - connectionId: 'warehouse', - syncId: 'sync-a', - dryRun: true, - decisionsPath: 'raw-sources/warehouse/live-database/sync-a/enrichment/relationship-review-decisions.json', - selectedDecisions: 1, - appliedRelationships: 1, - relationships: [ - { - id: 'orders:orders.customer_id->customers:customers.id', - source: 'manual', - from: { - tableId: 'public.orders', - columnIds: ['public.orders.customer_id'], - table: { catalog: null, db: 'public', name: 'orders' }, - columns: ['customer_id'], - }, - to: { - tableId: 'public.customers', - columnIds: ['public.customers.id'], - table: { catalog: null, db: 'public', name: 'customers' }, - columns: ['id'], - }, - relationshipType: 'many_to_one', - confidence: 1, - isPrimaryKeyReference: true, - }, - ], - manifestShards: [], - manifestShardsWritten: 0, - }), - ); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipApply', - projectDir: tempDir, - runId: 'scan-run-a', - applyAllAccepted: true, - candidateIds: [], - dryRun: true, - json: false, - }, - io.io, - { applyLocalScanRelationshipReviewDecisions }, - ), - ).resolves.toBe(0); - - expect(applyLocalScanRelationshipReviewDecisions).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - { - runId: 'scan-run-a', - applyAllAccepted: true, - candidateIds: [], - dryRun: true, - }, - ); - expect(io.stdout()).toContain('Relationship review apply'); - expect(io.stdout()).toContain('Run: scan-run-a'); - expect(io.stdout()).toContain('Mode: dry-run'); - expect(io.stdout()).toContain('Applied: 1 manual relationship'); - expect(io.stdout()).toContain('Schema shards written: 0'); - }); - - it('prints relationship review apply JSON', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const applyResult: ApplyLocalScanRelationshipReviewDecisionsResult = { - runId: 'scan-run-a', - connectionId: 'warehouse', - syncId: 'sync-a', - dryRun: false, - decisionsPath: 'raw-sources/warehouse/live-database/sync-a/enrichment/relationship-review-decisions.json', - selectedDecisions: 1, - appliedRelationships: 1, - relationships: [], - manifestShards: ['semantic-layer/warehouse/_schema/public.yaml'], - manifestShardsWritten: 1, - }; - const applyLocalScanRelationshipReviewDecisions = vi.fn(async () => applyResult); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipApply', - projectDir: tempDir, - runId: 'scan-run-a', - applyAllAccepted: false, - candidateIds: ['orders:orders.customer_id->customers:customers.id'], - dryRun: false, - json: true, - }, - io.io, - { applyLocalScanRelationshipReviewDecisions }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toEqual(applyResult); - expect(applyLocalScanRelationshipReviewDecisions).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - { - runId: 'scan-run-a', - applyAllAccepted: false, - candidateIds: ['orders:orders.customer_id->customers:customers.id'], - dryRun: false, - }, - ); - }); - - it('prints relationship feedback export summary in human form', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const feedback: ExportLocalRelationshipFeedbackLabelsResult = { - generatedAt: '2026-05-07T13:00:00.000Z', - filters: { connectionId: null, decision: 'all' }, - summary: { total: 2, accepted: 1, rejected: 1, connections: 1, runs: 1 }, - labels: [ - { - schemaVersion: 1, - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'accepted', - previousStatus: 'review', - connectionId: 'warehouse', - runId: 'scan-run-review', - syncId: 'sync-review', - decidedAt: '2026-05-07T12:00:00.000Z', - reviewer: 'Andrey', - note: 'Confirmed in warehouse docs', - relationshipType: 'many_to_one', - source: 'deterministic_name', - score: 0.62, - confidence: 0.62, - pkScore: 0.91, - fkScore: 0.62, - fromTable: 'public.orders', - fromColumns: ['customer_id'], - toTable: 'public.customers', - toColumns: ['id'], - reasons: ['fk_score_review'], - artifactPath: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-review-decisions.json', - }, - { - schemaVersion: 1, - candidateId: 'orders:orders.note_id->notes:notes.id', - decision: 'rejected', - previousStatus: 'rejected', - connectionId: 'warehouse', - runId: 'scan-run-review', - syncId: 'sync-review', - decidedAt: '2026-05-07T12:05:00.000Z', - reviewer: 'Andrey', - note: null, - relationshipType: 'many_to_one', - source: 'deterministic_name', - score: 0.2, - confidence: 0.2, - pkScore: 0.4, - fkScore: 0.2, - fromTable: 'public.orders', - fromColumns: ['note_id'], - toTable: 'public.notes', - toColumns: ['id'], - reasons: ['low_source_coverage'], - artifactPath: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-review-decisions.json', - }, - ], - warnings: [], - }; - const exportLocalRelationshipFeedbackLabels = vi.fn(async () => feedback); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipFeedback', - projectDir: tempDir, - connectionId: null, - decision: 'all', - json: false, - jsonl: false, - }, - io.io, - { exportLocalRelationshipFeedbackLabels }, - ), - ).resolves.toBe(0); - - expect(exportLocalRelationshipFeedbackLabels).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - { - connectionId: null, - decision: 'all', - }, - ); - expect(io.stdout()).toContain('KTX relationship feedback labels'); - expect(io.stdout()).toContain('Total: 2'); - expect(io.stdout()).toContain('Accepted: 1'); - expect(io.stdout()).toContain('Rejected: 1'); - expect(io.stdout()).toContain('orders.customer_id -> customers.id'); - expect(io.stdout()).toContain('decision=accepted previous=review score=0.62 reviewer=Andrey'); - }); - - it('prints relationship feedback labels as JSONL', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const feedback: ExportLocalRelationshipFeedbackLabelsResult = { - generatedAt: '2026-05-07T13:00:00.000Z', - filters: { connectionId: 'warehouse', decision: 'accepted' }, - summary: { total: 1, accepted: 1, rejected: 0, connections: 1, runs: 1 }, - labels: [ - { - schemaVersion: 1, - candidateId: 'orders:orders.customer_id->customers:customers.id', - decision: 'accepted', - previousStatus: 'review', - connectionId: 'warehouse', - runId: 'scan-run-review', - syncId: 'sync-review', - decidedAt: '2026-05-07T12:00:00.000Z', - reviewer: 'ktx', - note: null, - relationshipType: 'many_to_one', - source: 'deterministic_name', - score: 0.62, - confidence: 0.62, - pkScore: 0.91, - fkScore: 0.62, - fromTable: 'public.orders', - fromColumns: ['customer_id'], - toTable: 'public.customers', - toColumns: ['id'], - reasons: ['fk_score_review'], - artifactPath: 'raw-sources/warehouse/live-database/sync-review/enrichment/relationship-review-decisions.json', - }, - ], - warnings: [], - }; - const exportLocalRelationshipFeedbackLabels = vi.fn(async () => feedback); - const formatKtxRelationshipFeedbackLabelsJsonl = vi.fn( - () => '{"candidateId":"orders:orders.customer_id->customers:customers.id"}\n', - ); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipFeedback', - projectDir: tempDir, - connectionId: 'warehouse', - decision: 'accepted', - json: false, - jsonl: true, - }, - io.io, - { exportLocalRelationshipFeedbackLabels, formatKtxRelationshipFeedbackLabelsJsonl }, - ), - ).resolves.toBe(0); - - expect(exportLocalRelationshipFeedbackLabels).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - { - connectionId: 'warehouse', - decision: 'accepted', - }, - ); - expect(formatKtxRelationshipFeedbackLabelsJsonl).toHaveBeenCalledWith(feedback); - expect(JSON.parse(io.stdout())).toEqual({ candidateId: 'orders:orders.customer_id->customers:customers.id' }); - }); - - it('prints relationship feedback export as JSON', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const feedback: ExportLocalRelationshipFeedbackLabelsResult = { - generatedAt: '2026-05-07T13:00:00.000Z', - filters: { connectionId: null, decision: 'rejected' }, - summary: { total: 0, accepted: 0, rejected: 0, connections: 0, runs: 0 }, - labels: [], - warnings: [], - }; - const exportLocalRelationshipFeedbackLabels = vi.fn(async () => feedback); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipFeedback', - projectDir: tempDir, - connectionId: null, - decision: 'rejected', - json: true, - jsonl: false, - }, - io.io, - { exportLocalRelationshipFeedbackLabels }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ - filters: { connectionId: null, decision: 'rejected' }, - summary: { total: 0, accepted: 0, rejected: 0 }, - labels: [], - }); - }); - - it('prints relationship feedback calibration as human output', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const calibration: KtxRelationshipFeedbackCalibrationReport = { - generatedAt: '2026-05-07T13:00:00.000Z', - filters: { connectionId: null, decision: 'all' }, - thresholds: { accept: 0.85, review: 0.55 }, - summary: { - total: 2, - scored: 2, - unscored: 0, - acceptedLabels: 1, - rejectedLabels: 1, - predictedAccepted: 1, - predictedReview: 0, - predictedRejected: 1, - acceptedBandPrecision: 1, - rejectedBandPrecision: 1, - reviewBandAcceptedRate: null, - meanAcceptedScore: 0.91, - meanRejectedScore: 0.21, - }, - buckets: [ - { - label: '0.00-0.24', - minInclusive: 0, - maxInclusive: 0.24, - total: 1, - accepted: 0, - rejected: 1, - acceptanceRate: 0, - }, - { - label: '0.25-0.49', - minInclusive: 0.25, - maxInclusive: 0.49, - total: 0, - accepted: 0, - rejected: 0, - acceptanceRate: null, - }, - { - label: '0.50-0.74', - minInclusive: 0.5, - maxInclusive: 0.74, - total: 0, - accepted: 0, - rejected: 0, - acceptanceRate: null, - }, - { - label: '0.75-1.00', - minInclusive: 0.75, - maxInclusive: 1, - total: 1, - accepted: 1, - rejected: 0, - acceptanceRate: 1, - }, - ], - labels: [], - warnings: [], - }; - const calibrateLocalRelationshipFeedbackLabels = vi.fn(async () => calibration); - const formatKtxRelationshipFeedbackCalibrationMarkdown = vi.fn( - () => 'KTX relationship feedback calibration\nTotal labels: 2\n', - ); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipCalibration', - projectDir: tempDir, - connectionId: null, - decision: 'all', - acceptThreshold: 0.85, - reviewThreshold: 0.55, - json: false, - }, - io.io, - { calibrateLocalRelationshipFeedbackLabels, formatKtxRelationshipFeedbackCalibrationMarkdown }, - ), - ).resolves.toBe(0); - - expect(calibrateLocalRelationshipFeedbackLabels).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - { - connectionId: null, - decision: 'all', - acceptThreshold: 0.85, - reviewThreshold: 0.55, - }, - ); - expect(formatKtxRelationshipFeedbackCalibrationMarkdown).toHaveBeenCalledWith(calibration); - expect(io.stdout()).toBe('KTX relationship feedback calibration\nTotal labels: 2\n'); - }); - - it('prints relationship feedback calibration as JSON', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const calibration: KtxRelationshipFeedbackCalibrationReport = { - generatedAt: '2026-05-07T13:00:00.000Z', - filters: { connectionId: 'warehouse', decision: 'rejected' }, - thresholds: { accept: 0.9, review: 0.5 }, - summary: { - total: 0, - scored: 0, - unscored: 0, - acceptedLabels: 0, - rejectedLabels: 0, - predictedAccepted: 0, - predictedReview: 0, - predictedRejected: 0, - acceptedBandPrecision: null, - rejectedBandPrecision: null, - reviewBandAcceptedRate: null, - meanAcceptedScore: null, - meanRejectedScore: null, - }, - buckets: [], - labels: [], - warnings: [], - }; - const calibrateLocalRelationshipFeedbackLabels = vi.fn(async () => calibration); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipCalibration', - projectDir: tempDir, - connectionId: 'warehouse', - decision: 'rejected', - acceptThreshold: 0.9, - reviewThreshold: 0.5, - json: true, - }, - io.io, - { calibrateLocalRelationshipFeedbackLabels }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ - filters: { connectionId: 'warehouse', decision: 'rejected' }, - thresholds: { accept: 0.9, review: 0.5 }, - summary: { total: 0, scored: 0 }, - }); - }); - - it('prints relationship threshold advice as human output', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const advice: KtxRelationshipThresholdAdviceReport = { - generatedAt: '2026-05-07T14:00:00.000Z', - filters: { connectionId: null, decision: 'all' }, - status: 'ready', - gates: { - minTotalLabels: 4, - minAcceptedLabels: 2, - minRejectedLabels: 2, - minAcceptedBandPrecision: 0.9, - minAcceptedOrReviewRecall: 0.8, - minRejectedBandPrecision: 0.8, - }, - summary: { - totalLabels: 4, - scoredLabels: 4, - unscoredLabels: 0, - acceptedLabels: 2, - rejectedLabels: 2, - evaluatedCandidates: 2, - eligibleCandidates: 1, - }, - recommended: { - acceptThreshold: 0.9, - reviewThreshold: 0.55, - eligible: true, - predictedAccepted: 1, - predictedReview: 1, - predictedRejected: 2, - acceptedBandPrecision: 1, - acceptedRecall: 0.5, - acceptedOrReviewRecall: 1, - rejectedBandPrecision: 1, - rejectedRecall: 1, - falseAcceptedRejectedLabels: 0, - falseRejectedAcceptedLabels: 0, - }, - candidates: [], - reasons: [], - warnings: [], - }; - const adviseLocalRelationshipFeedbackThresholds = vi.fn(async () => advice); - const formatKtxRelationshipThresholdAdviceMarkdown = vi.fn( - () => 'KTX relationship threshold advice\nRecommended: accept=0.90 review=0.55\n', - ); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipThresholds', - projectDir: tempDir, - connectionId: null, - minTotalLabels: 4, - minAcceptedLabels: 2, - minRejectedLabels: 2, - json: false, - }, - io.io, - { adviseLocalRelationshipFeedbackThresholds, formatKtxRelationshipThresholdAdviceMarkdown }, - ), - ).resolves.toBe(0); - - expect(adviseLocalRelationshipFeedbackThresholds).toHaveBeenCalledWith( - expect.objectContaining({ projectDir: tempDir }), - { - connectionId: null, - minTotalLabels: 4, - minAcceptedLabels: 2, - minRejectedLabels: 2, - }, - ); - expect(formatKtxRelationshipThresholdAdviceMarkdown).toHaveBeenCalledWith(advice); - expect(io.stdout()).toBe('KTX relationship threshold advice\nRecommended: accept=0.90 review=0.55\n'); - }); - - it('prints relationship threshold advice as JSON', async () => { - await initKtxProject({ projectDir: tempDir, projectName: 'warehouse' }); - const advice: KtxRelationshipThresholdAdviceReport = { - generatedAt: '2026-05-07T14:00:00.000Z', - filters: { connectionId: 'warehouse', decision: 'all' }, - status: 'insufficient_labels', - gates: { - minTotalLabels: 20, - minAcceptedLabels: 5, - minRejectedLabels: 5, - minAcceptedBandPrecision: 0.9, - minAcceptedOrReviewRecall: 0.8, - minRejectedBandPrecision: 0.8, - }, - summary: { - totalLabels: 0, - scoredLabels: 0, - unscoredLabels: 0, - acceptedLabels: 0, - rejectedLabels: 0, - evaluatedCandidates: 0, - eligibleCandidates: 0, - }, - recommended: null, - candidates: [], - reasons: ['Need at least 20 scored labels; found 0.'], - warnings: [], - }; - const adviseLocalRelationshipFeedbackThresholds = vi.fn(async () => advice); - - const io = makeIo(); - await expect( - runKtxScan( - { - command: 'relationshipThresholds', - projectDir: tempDir, - connectionId: 'warehouse', - minTotalLabels: 20, - minAcceptedLabels: 5, - minRejectedLabels: 5, - json: true, - }, - io.io, - { adviseLocalRelationshipFeedbackThresholds }, - ), - ).resolves.toBe(0); - - expect(JSON.parse(io.stdout())).toMatchObject({ - filters: { connectionId: 'warehouse', decision: 'all' }, - status: 'insufficient_labels', - recommended: null, - }); - }); - it('passes native CLI adapters into local scan runs for mysql configs', async () => { const tempProject = await mkdtemp(join(tmpdir(), 'ktx-scan-cli-native-')); await initKtxProject({ projectDir: tempProject, projectName: 'warehouse' }); @@ -1846,7 +838,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); @@ -1895,7 +887,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); @@ -1953,7 +945,10 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { + runLocalScan, + createLocalIngestAdapters: () => [fakeLiveDatabaseAdapter(createPostgresLiveDatabaseIntrospection)], + }, ), ).resolves.toBe(0); @@ -2017,7 +1012,7 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { runLocalScan, createLocalIngestAdapters: noLocalIngestAdapters }, ), ).resolves.toBe(0); @@ -2069,7 +1064,10 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { + runLocalScan, + createLocalIngestAdapters: () => [fakeLiveDatabaseAdapter(createSqlServerLiveDatabaseIntrospection)], + }, ), ).resolves.toBe(0); @@ -2132,7 +1130,10 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { + runLocalScan, + createLocalIngestAdapters: () => [fakeLiveDatabaseAdapter(createBigQueryLiveDatabaseIntrospection)], + }, ), ).resolves.toBe(0); @@ -2198,7 +1199,10 @@ describe('runKtxScan', () => { dryRun: false, }, io.io, - { runLocalScan }, + { + runLocalScan, + createLocalIngestAdapters: () => [fakeLiveDatabaseAdapter(createSnowflakeLiveDatabaseIntrospection)], + }, ), ).resolves.toBe(0); diff --git a/packages/cli/src/scan.ts b/packages/cli/src/scan.ts index e3bda577..bca6057d 100644 --- a/packages/cli/src/scan.ts +++ b/packages/cli/src/scan.ts @@ -1,34 +1,10 @@ import { loadKtxProject } from '@ktx/context/project'; import { - type ApplyLocalScanRelationshipReviewDecisionsResult, - adviseLocalRelationshipFeedbackThresholds, - applyLocalScanRelationshipReviewDecisions, - calibrateLocalRelationshipFeedbackLabels, - type ExportLocalRelationshipFeedbackLabelsResult, - exportLocalRelationshipFeedbackLabels, - formatKtxRelationshipFeedbackCalibrationMarkdown, - formatKtxRelationshipFeedbackLabelsJsonl, - formatKtxRelationshipThresholdAdviceMarkdown, - getLocalScanReport, - getLocalScanStatus, type KtxProgressPort, - type KtxRelationshipArtifact, - type KtxRelationshipArtifactEdge, - type KtxRelationshipArtifactStatus, - type KtxRelationshipDiagnosticsArtifact, - type KtxRelationshipFeedbackCalibrationReport, - type KtxRelationshipFeedbackDecisionFilter, - type KtxRelationshipFeedbackLabel, - type KtxRelationshipReviewDecisionValue, - type KtxRelationshipThresholdAdviceReport, type KtxScanMode, type KtxScanReport, type KtxScanWarning, - type LocalScanStatusResponse, - readLocalScanRelationshipArtifacts, runLocalScan, - type WriteLocalScanRelationshipReviewDecisionResult, - writeLocalScanRelationshipReviewDecision, } from '@ktx/context/scan'; import type { KtxCliIo } from './index.js'; import { createKtxCliLocalIngestAdapters } from './local-adapters.js'; @@ -38,88 +14,21 @@ import { profileMark } from './startup-profile.js'; profileMark('module:scan'); -export type KtxScanArgs = - | { - command: 'run'; - projectDir: string; - connectionId: string; - mode: KtxScanMode; - detectRelationships: boolean; - dryRun: boolean; - databaseIntrospectionUrl?: string; - cliVersion?: string; - runtimeInstallPolicy?: KtxManagedPythonInstallPolicy; - } - | { command: 'status'; projectDir: string; runId: string } - | { command: 'report'; projectDir: string; runId: string; json: boolean } - | { - command: 'relationships'; - projectDir: string; - runId: string; - status: KtxRelationshipArtifactStatus; - json: boolean; - limit: number; - } - | { - command: 'relationshipDecision'; - projectDir: string; - runId: string; - candidateId: string; - decision: KtxRelationshipReviewDecisionValue; - reviewer: string; - note: string | null; - json: boolean; - } - | { - command: 'relationshipApply'; - projectDir: string; - runId: string; - applyAllAccepted: boolean; - candidateIds: string[]; - dryRun: boolean; - json: boolean; - } - | { - command: 'relationshipFeedback'; - projectDir: string; - connectionId: string | null; - decision: KtxRelationshipFeedbackDecisionFilter; - json: boolean; - jsonl: boolean; - } - | { - command: 'relationshipCalibration'; - projectDir: string; - connectionId: string | null; - decision: KtxRelationshipFeedbackDecisionFilter; - acceptThreshold: number; - reviewThreshold: number; - json: boolean; - } - | { - command: 'relationshipThresholds'; - projectDir: string; - connectionId: string | null; - minTotalLabels: number; - minAcceptedLabels: number; - minRejectedLabels: number; - json: boolean; - }; +export interface KtxScanArgs { + command: 'run'; + projectDir: string; + connectionId: string; + mode: KtxScanMode; + detectRelationships: boolean; + dryRun: boolean; + databaseIntrospectionUrl?: string; + cliVersion?: string; + runtimeInstallPolicy?: KtxManagedPythonInstallPolicy; +} interface KtxScanDeps { runLocalScan?: typeof runLocalScan; createLocalIngestAdapters?: typeof createKtxCliLocalIngestAdapters; - getLocalScanStatus?: typeof getLocalScanStatus; - getLocalScanReport?: typeof getLocalScanReport; - readLocalScanRelationshipArtifacts?: typeof readLocalScanRelationshipArtifacts; - writeLocalScanRelationshipReviewDecision?: typeof writeLocalScanRelationshipReviewDecision; - applyLocalScanRelationshipReviewDecisions?: typeof applyLocalScanRelationshipReviewDecisions; - exportLocalRelationshipFeedbackLabels?: typeof exportLocalRelationshipFeedbackLabels; - formatKtxRelationshipFeedbackLabelsJsonl?: typeof formatKtxRelationshipFeedbackLabelsJsonl; - calibrateLocalRelationshipFeedbackLabels?: typeof calibrateLocalRelationshipFeedbackLabels; - formatKtxRelationshipFeedbackCalibrationMarkdown?: typeof formatKtxRelationshipFeedbackCalibrationMarkdown; - adviseLocalRelationshipFeedbackThresholds?: typeof adviseLocalRelationshipFeedbackThresholds; - formatKtxRelationshipThresholdAdviceMarkdown?: typeof formatKtxRelationshipThresholdAdviceMarkdown; } function shouldUseStyledOutput(io: KtxCliIo): boolean { @@ -284,208 +193,8 @@ function writeRunSummary(report: KtxScanReport, projectDir: string, io: KtxCliIo writeHumanReportBody(report, io); const projectDirArg = quoteCliArg(projectDir); io.stdout.write('\nNext:\n'); - const statusCommand = styled ? dim('ktx dev scan status') : 'ktx dev scan status'; - const reportCommand = styled ? dim('ktx dev scan report') : 'ktx dev scan report'; - io.stdout.write(` ${statusCommand} --project-dir ${projectDirArg} ${report.runId}\n`); - io.stdout.write(` ${reportCommand} --project-dir ${projectDirArg} ${report.runId}\n`); -} - -function writeReport(report: KtxScanReport, io: KtxCliIo): void { - io.stdout.write('KTX scan report\n'); - writeHumanReportBody(report, io); -} - -function formatRelationshipEndpoint(edge: KtxRelationshipArtifactEdge, side: 'from' | 'to'): string { - const endpoint = edge[side]; - if (endpoint.columns.length === 1) { - return `${endpoint.table.name}.${endpoint.columns[0]}`; - } - return `${endpoint.table.name}.(${endpoint.columns.join(',')})`; -} - -function formatRelationshipScore(value: number | null): string { - return value === null ? 'n/a' : value.toFixed(2); -} - -function relationshipStatusTitle(status: Exclude): string { - if (status === 'accepted') { - return 'Accepted relationships'; - } - if (status === 'review') { - return 'Review relationships'; - } - if (status === 'rejected') { - return 'Rejected relationships'; - } - return 'Skipped relationships'; -} - -function filteredRelationshipArtifact( - relationships: KtxRelationshipArtifact, - status: KtxRelationshipArtifactStatus, -): KtxRelationshipArtifact { - if (status === 'all') { - return relationships; - } - return { - connectionId: relationships.connectionId, - accepted: status === 'accepted' ? relationships.accepted : [], - review: status === 'review' ? relationships.review : [], - rejected: status === 'rejected' ? relationships.rejected : [], - skipped: status === 'skipped' ? relationships.skipped : [], - }; -} - -function writeRelationshipEdge(edge: KtxRelationshipArtifactEdge, index: number, io: KtxCliIo): void { - io.stdout.write( - ` ${index + 1}. ${formatRelationshipEndpoint(edge, 'from')} -> ${formatRelationshipEndpoint(edge, 'to')}\n`, - ); - io.stdout.write( - ` type=${edge.relationshipType} source=${edge.source} confidence=${edge.confidence.toFixed(2)} pkScore=${formatRelationshipScore(edge.pkScore)} fkScore=${formatRelationshipScore(edge.fkScore)}\n`, - ); - io.stdout.write(` reasons=${edge.reasons.length > 0 ? edge.reasons.join(', ') : 'none'}\n`); -} - -function writeRelationshipGroup( - status: Exclude, - relationships: KtxRelationshipArtifact, - limit: number, - io: KtxCliIo, -): void { - if (status === 'skipped') { - io.stdout.write(`\n${relationshipStatusTitle(status)} (${relationships.skipped.length})\n`); - relationships.skipped.slice(0, limit).forEach((item, index) => { - io.stdout.write(` ${index + 1}. ${item.relationshipId}\n`); - io.stdout.write(` reason=${item.reason}\n`); - }); - return; - } - - const edges = - status === 'accepted' - ? relationships.accepted - : status === 'review' - ? relationships.review - : relationships.rejected; - io.stdout.write(`\n${relationshipStatusTitle(status)} (${edges.length})\n`); - edges.slice(0, limit).forEach((edge, index) => { - writeRelationshipEdge(edge, index, io); - }); - if (edges.length > limit) { - io.stdout.write(` ${edges.length - limit} more not shown; rerun with --limit ${edges.length}\n`); - } -} - -function writeRelationshipArtifactSummary(input: { - runId: string; - connectionId: string; - syncId: string; - status: KtxRelationshipArtifactStatus; - limit: number; - summary: KtxRelationshipArtifact; - relationships: KtxRelationshipArtifact; - diagnostics: KtxRelationshipDiagnosticsArtifact | null; - relationshipsPath: string; - io: KtxCliIo; -}): void { - input.io.stdout.write('KTX relationship artifacts\n'); - input.io.stdout.write(`Run: ${input.runId}\n`); - input.io.stdout.write(`Connection: ${input.connectionId}\n`); - input.io.stdout.write(`Sync: ${input.syncId}\n`); - input.io.stdout.write( - `Summary: accepted=${input.summary.accepted.length} review=${input.summary.review.length} rejected=${input.summary.rejected.length} skipped=${input.summary.skipped.length}\n`, - ); - if (input.diagnostics?.noAcceptedReason) { - input.io.stdout.write(`Reason: ${input.diagnostics.noAcceptedReason}\n`); - } - input.io.stdout.write(`Artifacts: ${input.relationshipsPath}\n`); - - const statuses: Array> = - input.status === 'all' ? ['accepted', 'review', 'rejected', 'skipped'] : [input.status]; - for (const status of statuses) { - writeRelationshipGroup(status, input.relationships, input.limit, input.io); - } -} - -function writeRelationshipDecisionResult(result: WriteLocalScanRelationshipReviewDecisionResult, io: KtxCliIo): void { - io.stdout.write('Recorded relationship decision\n'); - io.stdout.write(`Decision: ${result.decision.decision}\n`); - io.stdout.write(`Candidate: ${result.decision.candidateId}\n`); - io.stdout.write(`Previous status: ${result.decision.previousStatus}\n`); - io.stdout.write(`Reviewer: ${result.decision.reviewer}\n`); - if (result.decision.note) { - io.stdout.write(`Note: ${result.decision.note}\n`); - } - io.stdout.write(`Path: ${result.path}\n`); -} - -function writeRelationshipApplyResult(result: ApplyLocalScanRelationshipReviewDecisionsResult, io: KtxCliIo): void { - io.stdout.write('Relationship review apply\n'); - io.stdout.write(`Run: ${result.runId}\n`); - io.stdout.write(`Connection: ${result.connectionId}\n`); - io.stdout.write(`Sync: ${result.syncId}\n`); - io.stdout.write(`Mode: ${result.dryRun ? 'dry-run' : 'write'}\n`); - io.stdout.write(`Decisions: ${result.selectedDecisions} ${plural(result.selectedDecisions, 'accepted decision')}\n`); - io.stdout.write( - `Applied: ${result.appliedRelationships} manual ${plural(result.appliedRelationships, 'relationship')}\n`, - ); - io.stdout.write(`Schema shards written: ${result.manifestShardsWritten}\n`); - if (result.manifestShards.length > 0) { - io.stdout.write('Schema shards:\n'); - for (const shard of result.manifestShards) { - io.stdout.write(` - ${shard}\n`); - } - } - io.stdout.write(`Decisions: ${result.decisionsPath}\n`); -} - -function formatFeedbackColumns(columns: readonly string[]): string { - return columns.length === 1 ? (columns[0] ?? 'unknown') : `(${columns.join(',')})`; -} - -function feedbackTableShortName(value: string): string { - return value.split('.').at(-1) ?? value; -} - -function feedbackEndpoint(label: KtxRelationshipFeedbackLabel, side: 'from' | 'to'): string { - if (side === 'from') { - return `${feedbackTableShortName(label.fromTable)}.${formatFeedbackColumns(label.fromColumns)}`; - } - return `${feedbackTableShortName(label.toTable)}.${formatFeedbackColumns(label.toColumns)}`; -} - -function writeRelationshipFeedbackSummary(result: ExportLocalRelationshipFeedbackLabelsResult, io: KtxCliIo): void { - io.stdout.write('KTX relationship feedback labels\n'); - io.stdout.write(`Generated: ${result.generatedAt}\n`); - io.stdout.write(`Filter connection: ${result.filters.connectionId ?? 'all'}\n`); - io.stdout.write(`Filter decision: ${result.filters.decision}\n`); - io.stdout.write(`Total: ${result.summary.total}\n`); - io.stdout.write(`Accepted: ${result.summary.accepted}\n`); - io.stdout.write(`Rejected: ${result.summary.rejected}\n`); - io.stdout.write(`Connections: ${result.summary.connections}\n`); - io.stdout.write(`Runs: ${result.summary.runs}\n`); - - if (result.warnings.length > 0) { - io.stdout.write('\nWarnings\n'); - for (const warning of result.warnings.slice(0, 5)) { - io.stdout.write(` - ${warning.path}: ${warning.message}\n`); - } - } - - if (result.labels.length === 0) { - return; - } - - io.stdout.write('\nLabels\n'); - for (const label of result.labels.slice(0, 25)) { - io.stdout.write(` - ${feedbackEndpoint(label, 'from')} -> ${feedbackEndpoint(label, 'to')}\n`); - io.stdout.write( - ` decision=${label.decision} previous=${label.previousStatus} score=${formatRelationshipScore(label.score)} reviewer=${label.reviewer}\n`, - ); - } - if (result.labels.length > 25) { - io.stdout.write(` ${result.labels.length - 25} more labels not shown; rerun with --jsonl for the full dataset\n`); - } + const statusCommand = styled ? dim('ktx status') : 'ktx status'; + io.stdout.write(` ${statusCommand} --project-dir ${projectDirArg}\n`); } interface KtxCliScanProgressState { @@ -540,184 +249,9 @@ export function createCliScanProgress( return progress; } -function writeStatus(status: LocalScanStatusResponse, io: KtxCliIo): void { - io.stdout.write(`Run: ${status.runId}\n`); - io.stdout.write(`Status: ${status.status}\n`); - io.stdout.write(`Connection: ${status.connectionId}\n`); - io.stdout.write(`Mode: ${status.mode}\n`); - io.stdout.write(`Sync: ${status.syncId}\n`); - io.stdout.write(`Progress: ${status.progress}\n`); - io.stdout.write(`Report: ${status.reportPath ?? 'none'}\n`); -} - export async function runKtxScan(args: KtxScanArgs, io: KtxCliIo = process, deps: KtxScanDeps = {}): Promise { try { const project = await loadKtxProject({ projectDir: args.projectDir }); - if (args.command === 'status') { - const status = await (deps.getLocalScanStatus ?? getLocalScanStatus)(project, args.runId); - if (!status) { - throw new Error(`Scan run "${args.runId}" was not found`); - } - writeStatus(status, io); - return 0; - } - if (args.command === 'report') { - const report = await (deps.getLocalScanReport ?? getLocalScanReport)(project, args.runId); - if (!report) { - throw new Error(`Scan report "${args.runId}" was not found`); - } - if (args.json) { - io.stdout.write(`${JSON.stringify(report, null, 2)}\n`); - } else { - writeReport(report, io); - } - return 0; - } - if (args.command === 'relationships') { - const result = await (deps.readLocalScanRelationshipArtifacts ?? readLocalScanRelationshipArtifacts)( - project, - args.runId, - ); - if (!result) { - throw new Error(`Scan run "${args.runId}" was not found`); - } - const filtered = filteredRelationshipArtifact(result.relationships, args.status); - if (args.json) { - io.stdout.write( - `${JSON.stringify( - { - runId: result.runId, - connectionId: result.connectionId, - syncId: result.syncId, - status: args.status, - paths: result.paths, - diagnostics: result.diagnostics, - summary: { - accepted: result.relationships.accepted.length, - review: result.relationships.review.length, - rejected: result.relationships.rejected.length, - skipped: result.relationships.skipped.length, - }, - relationships: filtered, - }, - null, - 2, - )}\n`, - ); - } else { - writeRelationshipArtifactSummary({ - runId: result.runId, - connectionId: result.connectionId, - syncId: result.syncId, - status: args.status, - limit: args.limit, - summary: result.relationships, - relationships: filtered, - diagnostics: result.diagnostics, - relationshipsPath: result.paths.relationships, - io, - }); - } - return 0; - } - if (args.command === 'relationshipDecision') { - const result = await (deps.writeLocalScanRelationshipReviewDecision ?? writeLocalScanRelationshipReviewDecision)( - project, - { - runId: args.runId, - candidateId: args.candidateId, - decision: args.decision, - reviewer: args.reviewer, - note: args.note, - }, - ); - if (!result) { - throw new Error(`Scan run "${args.runId}" was not found`); - } - if (args.json) { - io.stdout.write(`${JSON.stringify(result, null, 2)}\n`); - } else { - writeRelationshipDecisionResult(result, io); - } - return 0; - } - if (args.command === 'relationshipApply') { - const result = await ( - deps.applyLocalScanRelationshipReviewDecisions ?? applyLocalScanRelationshipReviewDecisions - )(project, { - runId: args.runId, - applyAllAccepted: args.applyAllAccepted, - candidateIds: args.candidateIds, - dryRun: args.dryRun, - }); - if (args.json) { - io.stdout.write( - `${JSON.stringify(result satisfies ApplyLocalScanRelationshipReviewDecisionsResult, null, 2)}\n`, - ); - } else { - writeRelationshipApplyResult(result, io); - } - return 0; - } - if (args.command === 'relationshipFeedback') { - const result = await (deps.exportLocalRelationshipFeedbackLabels ?? exportLocalRelationshipFeedbackLabels)( - project, - { - connectionId: args.connectionId, - decision: args.decision, - }, - ); - if (args.jsonl) { - io.stdout.write( - (deps.formatKtxRelationshipFeedbackLabelsJsonl ?? formatKtxRelationshipFeedbackLabelsJsonl)(result), - ); - } else if (args.json) { - io.stdout.write(`${JSON.stringify(result, null, 2)}\n`); - } else { - writeRelationshipFeedbackSummary(result, io); - } - return 0; - } - if (args.command === 'relationshipCalibration') { - const result = await (deps.calibrateLocalRelationshipFeedbackLabels ?? calibrateLocalRelationshipFeedbackLabels)( - project, - { - connectionId: args.connectionId, - decision: args.decision, - acceptThreshold: args.acceptThreshold, - reviewThreshold: args.reviewThreshold, - }, - ); - if (args.json) { - io.stdout.write(`${JSON.stringify(result satisfies KtxRelationshipFeedbackCalibrationReport, null, 2)}\n`); - } else { - io.stdout.write( - (deps.formatKtxRelationshipFeedbackCalibrationMarkdown ?? formatKtxRelationshipFeedbackCalibrationMarkdown)( - result, - ), - ); - } - return 0; - } - if (args.command === 'relationshipThresholds') { - const result = await ( - deps.adviseLocalRelationshipFeedbackThresholds ?? adviseLocalRelationshipFeedbackThresholds - )(project, { - connectionId: args.connectionId, - minTotalLabels: args.minTotalLabels, - minAcceptedLabels: args.minAcceptedLabels, - minRejectedLabels: args.minRejectedLabels, - }); - if (args.json) { - io.stdout.write(`${JSON.stringify(result satisfies KtxRelationshipThresholdAdviceReport, null, 2)}\n`); - } else { - io.stdout.write( - (deps.formatKtxRelationshipThresholdAdviceMarkdown ?? formatKtxRelationshipThresholdAdviceMarkdown)(result), - ); - } - return 0; - } - const managedDaemon = managedDaemonOptionsForScanRun(args, io); const connector = args.mode !== 'structural' || args.detectRelationships diff --git a/packages/cli/src/setup-agents.test.ts b/packages/cli/src/setup-agents.test.ts index 322db2aa..19647a3f 100644 --- a/packages/cli/src/setup-agents.test.ts +++ b/packages/cli/src/setup-agents.test.ts @@ -84,7 +84,10 @@ describe('setup agents', () => { const skill = await readFile(join(tempDir, '.agents/skills/ktx/SKILL.md'), 'utf-8'); expect(skill).toContain(`--project-dir ${tempDir}`); expect(skill).toContain('must not print secrets'); - expect(skill).toContain('agent sql execute'); + expect(skill).toContain('status --json'); + expect(skill).toContain('sl list --json'); + expect(skill).not.toContain('agent '); + expect(skill).not.toContain('sql execute'); expect(await readKtxAgentInstallManifest(tempDir)).toMatchObject({ version: 1, projectDir: tempDir, @@ -115,8 +118,9 @@ describe('setup agents', () => { const skill = await readFile(join(tempDir, '.agents/skills/ktx/SKILL.md'), 'utf-8'); expect(skill).not.toContain('`ktx agent'); - expect(skill).toContain('agent context --json'); - expect(skill).toContain('agent sql execute'); + expect(skill).toContain('status --json'); + expect(skill).toContain('sl query'); + expect(skill).not.toContain('sql execute'); }); it('removes only manifest-listed files', async () => { diff --git a/packages/cli/src/setup-agents.ts b/packages/cli/src/setup-agents.ts index 151967aa..b4202ed6 100644 --- a/packages/cli/src/setup-agents.ts +++ b/packages/cli/src/setup-agents.ts @@ -124,7 +124,7 @@ function cliInstructionContent(input: { projectDir: string; launcher: KtxCliLaun return [ '---', 'name: ktx', - 'description: Use local KTX semantic context, wiki knowledge, and safe SQL execution for this project.', + 'description: Use local KTX semantic context and wiki knowledge for this project.', '---', '', '# KTX Local Context', @@ -137,11 +137,11 @@ function cliInstructionContent(input: { projectDir: string; launcher: KtxCliLaun '', 'Available commands:', '', - `- \`${ktxCommandLine(input.launcher, ['agent', 'context', ...projectDirArgs])}\``, - `- \`${ktxCommandLine(input.launcher, ['agent', 'sl', 'list', ...projectDirArgs])}\``, - `- \`${ktxCommandLine(input.launcher, ['agent', 'sl', 'read', '', ...projectDirArgs])}\``, + `- \`${ktxCommandLine(input.launcher, ['status', ...projectDirArgs])}\``, + `- \`${ktxCommandLine(input.launcher, ['sl', 'list', ...projectDirArgs])}\``, + `- \`${ktxCommandLine(input.launcher, ['sl', 'list', ...projectDirArgs, '--query', ''])}\``, + `- \`${ktxCommandLine(input.launcher, ['sl', 'read', '', ...projectDirArgs, '--connection-id', ''])}\``, `- \`${ktxCommandLine(input.launcher, [ - 'agent', 'sl', 'query', ...projectDirArgs, @@ -153,29 +153,17 @@ function cliInstructionContent(input: { projectDir: string; launcher: KtxCliLaun '--max-rows', '100', ])}\``, - `- \`${ktxCommandLine(input.launcher, ['agent', 'wiki', 'search', '', ...projectDirArgs])}\``, - `- \`${ktxCommandLine(input.launcher, ['agent', 'wiki', 'read', '', ...projectDirArgs])}\``, - `- \`${ktxCommandLine(input.launcher, [ - 'agent', - 'sql', - 'execute', - ...projectDirArgs, - '--connection-id', - '', - '--sql-file', - '', - '--max-rows', - '100', - ])}\``, + `- \`${ktxCommandLine(input.launcher, ['wiki', 'search', '', ...projectDirArgs, '--limit', '10'])}\``, + `- \`${ktxCommandLine(input.launcher, ['wiki', 'read', '', ...projectDirArgs])}\``, '', - 'SQL execution is read-only, requires an explicit row limit, and should use the smallest useful limit.', + 'Use semantic-layer queries before direct database access. Do not print secrets or credential references.', '', ].join('\n'); } function ruleInstructionContent(input: { projectDir: string }): string { return [ - `Use the \`ktx\` CLI to query local semantic context, wiki knowledge, and execute safe SQL for this project (\`--project-dir ${input.projectDir}\`).`, + `Use the \`ktx\` CLI to query local semantic context and wiki knowledge for this project (\`--project-dir ${input.projectDir}\`).`, '', 'Use when the user asks about data schemas, metrics, dimensions, database structure, or wants to run SQL queries.', '', diff --git a/packages/cli/src/setup-context.test.ts b/packages/cli/src/setup-context.test.ts index 3fd4dd10..1ef044ae 100644 --- a/packages/cli/src/setup-context.test.ts +++ b/packages/cli/src/setup-context.test.ts @@ -1,4 +1,4 @@ -import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { readKtxSetupState } from '@ktx/context/project'; @@ -204,9 +204,8 @@ describe('setup context build state', () => { expect.objectContaining({ onDetach: expect.any(Function) }), ); expect(verifyContextReady).toHaveBeenCalledWith(tempDir); - await expect(readKtxSetupState(tempDir)).resolves.toEqual({ - completed_steps: ['context'], - }); + expect(await readFile(join(tempDir, 'ktx.yaml'), 'utf-8')).not.toContain('completed_steps:'); + expect((await readKtxSetupState(tempDir)).completed_steps).toContain('context'); await expect(readKtxSetupContextState(tempDir)).resolves.toMatchObject({ runId: 'setup-context-local-abc123', status: 'completed', @@ -287,9 +286,8 @@ describe('setup context build state', () => { ).resolves.toEqual({ status: 'ready', projectDir: tempDir, runId: 'setup-context-local-existing' }); expect(runContextBuildMock).not.toHaveBeenCalled(); - await expect(readKtxSetupState(tempDir)).resolves.toEqual({ - completed_steps: ['context'], - }); + expect(await readFile(join(tempDir, 'ktx.yaml'), 'utf-8')).not.toContain('completed_steps:'); + expect((await readKtxSetupState(tempDir)).completed_steps).toContain('context'); await expect(readKtxSetupContextState(tempDir)).resolves.toMatchObject({ runId: 'setup-context-local-existing', status: 'completed', diff --git a/packages/cli/src/setup-databases.test.ts b/packages/cli/src/setup-databases.test.ts index fe480b13..46506ae7 100644 --- a/packages/cli/src/setup-databases.test.ts +++ b/packages/cli/src/setup-databases.test.ts @@ -929,7 +929,7 @@ describe('setup databases step', () => { commandIo.stdout.write(' Raw sources: raw-sources/postgres-warehouse/live-database/2026-05-09-221301-local-moywh3ky\n'); commandIo.stdout.write(' Schema shards: 1\n\n'); commandIo.stdout.write('Next:\n'); - commandIo.stdout.write(` ktx dev scan status --project-dir ${tempDir} local-moywh3ky\n`); + commandIo.stdout.write(` ktx status --project-dir ${tempDir} local-moywh3ky\n`); return 0; }); diff --git a/packages/cli/src/setup-databases.ts b/packages/cli/src/setup-databases.ts index d71b7225..f770c5c4 100644 --- a/packages/cli/src/setup-databases.ts +++ b/packages/cli/src/setup-databases.ts @@ -1448,7 +1448,7 @@ async function validateAndScanConnection(input: { if (scanCode !== 0) { flushBufferedCommandOutput(input.io, scanIo); input.io.stderr.write(`Structural scan failed for ${input.connectionId}.\n`); - input.io.stderr.write(`Debug command: ktx dev scan --project-dir ${input.projectDir} ${input.connectionId}\n`); + input.io.stderr.write(`Debug command: ktx scan --project-dir ${input.projectDir} ${input.connectionId}\n`); return false; } const scanOutput = scanIo.stdoutText(); diff --git a/packages/cli/src/setup-sources.test.ts b/packages/cli/src/setup-sources.test.ts index edadc17e..76ba5d0f 100644 --- a/packages/cli/src/setup-sources.test.ts +++ b/packages/cli/src/setup-sources.test.ts @@ -664,7 +664,7 @@ describe('setup sources step', () => { expect(runInitialIngest).toHaveBeenCalledTimes(1); expect((await readConfig()).connections['dbt-main']).toMatchObject({ driver: 'dbt', source_dir: '/repo/dbt' }); expect(io.stdout()).toContain('Context source saved without a completed context build for dbt-main.'); - expect(io.stdout()).toContain('Run later: ktx ingest dbt-main'); + expect(io.stdout()).toContain('Run later: ktx ingest run --connection-id dbt-main --adapter '); }); it('retries initial source ingest from the failure menu', async () => { diff --git a/packages/cli/src/setup-sources.ts b/packages/cli/src/setup-sources.ts index 695fc1c1..6674ef75 100644 --- a/packages/cli/src/setup-sources.ts +++ b/packages/cli/src/setup-sources.ts @@ -739,7 +739,7 @@ async function runInitialSourceIngestWithRecovery(input: { } if (action === 'continue') { input.io.stdout.write(`│ Context source saved without a completed context build for ${input.connectionId}.\n`); - input.io.stdout.write(`│ Run later: ktx ingest ${input.connectionId}\n`); + input.io.stdout.write(`│ Run later: ktx ingest run --connection-id ${input.connectionId} --adapter \n`); return 'continue'; } return 'back'; diff --git a/packages/cli/src/setup.test.ts b/packages/cli/src/setup.test.ts index e74dca5d..bf9c381f 100644 --- a/packages/cli/src/setup.test.ts +++ b/packages/cli/src/setup.test.ts @@ -584,13 +584,13 @@ describe('setup status', () => { expect(projectPrompts.select).toHaveBeenCalledWith( expect.objectContaining({ - message: 'Which KTX project should setup use?', + message: 'Where should KTX create the project?', options: expect.arrayContaining([expect.objectContaining({ value: 'back', label: 'Back' })]), }), ); expect(projectPrompts.select).toHaveBeenCalledWith( expect.objectContaining({ - message: 'Which KTX project should setup use?', + message: 'Where should KTX create the project?', options: expect.not.arrayContaining([expect.objectContaining({ value: 'exit', label: 'Exit' })]), }), ); diff --git a/packages/cli/src/sl.test.ts b/packages/cli/src/sl.test.ts index c04ec15d..8d360c58 100644 --- a/packages/cli/src/sl.test.ts +++ b/packages/cli/src/sl.test.ts @@ -84,6 +84,71 @@ describe('runKtxSl', () => { expect(listIo.stdout()).toContain('warehouse\torders\tcolumns=1\tmeasures=0\tjoins=0'); }); + it('prints semantic-layer reads and searched lists as public JSON envelopes', async () => { + const projectDir = join(tempDir, 'project'); + await initKtxProject({ projectDir, projectName: 'warehouse' }); + + await expect( + runKtxSl( + { + command: 'write', + projectDir, + connectionId: 'warehouse', + sourceName: 'orders', + yaml: [ + 'name: orders', + 'table: public.orders', + 'description: Paid order facts', + 'grain: [order_id]', + 'columns:', + ' - name: order_id', + ' type: string', + '', + ].join('\n'), + }, + makeIo().io, + ), + ).resolves.toBe(0); + + const readIo = makeIo(); + await expect( + runKtxSl( + { command: 'read', projectDir, connectionId: 'warehouse', sourceName: 'orders', json: true }, + readIo.io, + ), + ).resolves.toBe(0); + expect(JSON.parse(readIo.stdout())).toMatchObject({ + kind: 'sl.source', + data: { + connectionId: 'warehouse', + name: 'orders', + yaml: expect.stringContaining('name: orders'), + }, + }); + + const listIo = makeIo(); + await expect( + runKtxSl( + { command: 'list', projectDir, connectionId: 'warehouse', query: 'paid', json: true }, + listIo.io, + ), + ).resolves.toBe(0); + expect(JSON.parse(listIo.stdout())).toMatchObject({ + kind: 'list', + data: { + items: [ + expect.objectContaining({ + connectionId: 'warehouse', + name: 'orders', + score: expect.any(Number), + matchReasons: expect.arrayContaining(['token']), + }), + ], + }, + meta: { command: 'sl list' }, + }); + }); + it('fails validation when a table-backed source declares columns absent from a matching warehouse manifest', async () => { const projectDir = join(tempDir, 'project'); const project = await initKtxProject({ projectDir, projectName: 'warehouse' }); @@ -191,6 +256,73 @@ joins: [] expect(stderr.write).not.toHaveBeenCalled(); }); + it('runs sl query from a JSON query file', async () => { + const projectDir = join(tempDir, 'project'); + const project = await initKtxProject({ projectDir, projectName: 'warehouse' }); + project.config.connections.warehouse = { driver: 'postgres', readonly: true }; + await project.fileStore.writeFile( + 'semantic-layer/warehouse/orders.yaml', + `name: orders +table: public.orders +grain: [id] +columns: + - name: id + type: number +measures: + - name: order_count + expr: count(*) +joins: [] +`, + 'ktx', + 'ktx@example.com', + 'Add orders source', + ); + const queryFile = join(tempDir, 'query.json'); + await writeFile(queryFile, '{"measures":["orders.order_count"],"dimensions":[]}', 'utf-8'); + + const stdout = { write: vi.fn() }; + const stderr = { write: vi.fn() }; + const query = vi.fn(async () => ({ + sql: 'select count(*) as order_count from public.orders', + dialect: 'postgres', + columns: [{ name: 'orders.order_count' }], + plan: {}, + })); + const createSemanticLayerCompute = vi.fn(() => ({ + query, + validateSources: vi.fn(), + generateSources: vi.fn(), + })); + + await expect( + runKtxSl( + { + command: 'query', + projectDir, + connectionId: 'warehouse', + queryFile, + format: 'json', + execute: false, + cliVersion: '0.2.0', + runtimeInstallPolicy: 'auto', + }, + { stdout, stderr }, + { createSemanticLayerCompute }, + ), + ).resolves.toBe(0); + + expect(query).toHaveBeenCalledWith( + expect.objectContaining({ + query: { measures: ['orders.order_count'], dimensions: [] }, + }), + ); + expect(JSON.parse(String(stdout.write.mock.calls[0][0]))).toMatchObject({ + sql: 'select count(*) as order_count from public.orders', + plan: { execution: { mode: 'compile_only' } }, + }); + expect(stderr.write).not.toHaveBeenCalled(); + }); + it('creates default sl query compute through the managed runtime helper', async () => { const projectDir = join(tempDir, 'project'); const project = await initKtxProject({ projectDir, projectName: 'warehouse' }); diff --git a/packages/cli/src/sl.ts b/packages/cli/src/sl.ts index fb0f129e..ebf3eca7 100644 --- a/packages/cli/src/sl.ts +++ b/packages/cli/src/sl.ts @@ -1,14 +1,22 @@ +import { readFile } from 'node:fs/promises'; import { createDefaultLocalQueryExecutor, type KtxSqlQueryExecutorPort } from '@ktx/context/connections'; +import { + createLocalKtxEmbeddingProviderFromConfig, + KtxIngestEmbeddingPortAdapter, + type KtxEmbeddingPort, +} from '@ktx/context'; import type { KtxSemanticLayerComputePort } from '@ktx/context/daemon'; import { loadKtxProject, type KtxLocalProject } from '@ktx/context/project'; import { compileLocalSlQuery, listLocalSlSources, readLocalSlSource, + searchLocalSlSources, validateLocalSlSource, writeLocalSlSource, type SemanticLayerQueryInput, } from '@ktx/context/sl'; +import { writeJsonResult } from './io/print-list.js'; import { createManagedPythonSemanticLayerComputePort, type KtxManagedPythonInstallPolicy, @@ -20,15 +28,16 @@ profileMark('module:sl'); type SlQueryFormat = 'json' | 'sql'; export type KtxSlArgs = - | { command: 'list'; projectDir: string; connectionId?: string; output?: string; json?: boolean } - | { command: 'read'; projectDir: string; connectionId: string; sourceName: string } + | { command: 'list'; projectDir: string; connectionId?: string; query?: string; output?: string; json?: boolean } + | { command: 'read'; projectDir: string; connectionId: string; sourceName: string; json?: boolean } | { command: 'validate'; projectDir: string; connectionId: string; sourceName: string } | { command: 'write'; projectDir: string; connectionId: string; sourceName: string; yaml: string } | { command: 'query'; projectDir: string; connectionId?: string; - query: SemanticLayerQueryInput; + query?: SemanticLayerQueryInput; + queryFile?: string; format: SlQueryFormat; execute: boolean; maxRows?: number; @@ -43,6 +52,8 @@ interface KtxSlIo { interface KtxSlDeps { loadProject?: typeof loadKtxProject; + embeddingService?: KtxEmbeddingPort | null; + createEmbeddingProvider?: typeof createLocalKtxEmbeddingProviderFromConfig; createSemanticLayerCompute?: () => KtxSemanticLayerComputePort; createManagedSemanticLayerCompute?: (options: { cliVersion: string; @@ -52,11 +63,35 @@ interface KtxSlDeps { createQueryExecutor?: () => KtxSqlQueryExecutorPort; } +function slSearchEmbeddingService(project: KtxLocalProject, deps: KtxSlDeps): KtxEmbeddingPort | null { + if ('embeddingService' in deps) { + return deps.embeddingService ?? null; + } + const provider = (deps.createEmbeddingProvider ?? createLocalKtxEmbeddingProviderFromConfig)( + project.config.ingest.embeddings, + ); + return provider ? new KtxIngestEmbeddingPortAdapter(provider) : null; +} + +async function readSlQueryFile(path: string): Promise { + const parsed = JSON.parse(await readFile(path, 'utf-8')) as unknown; + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + throw new Error(`${path} must contain a JSON object.`); + } + return parsed as SemanticLayerQueryInput; +} + export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: KtxSlDeps = {}): Promise { try { const project = await (deps.loadProject ?? loadKtxProject)({ projectDir: args.projectDir }); if (args.command === 'list') { - const sources = await listLocalSlSources(project, { connectionId: args.connectionId }); + const sources = args.query + ? await searchLocalSlSources(project, { + connectionId: args.connectionId, + query: args.query, + embeddingService: slSearchEmbeddingService(project, deps), + }) + : await listLocalSlSources(project, { connectionId: args.connectionId }); const { resolveOutputMode } = await import('./io/mode.js'); const { printList } = await import('./io/print-list.js'); const mode = resolveOutputMode({ explicit: args.output, json: args.json, io }); @@ -86,6 +121,14 @@ export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: Ktx if (!source) { throw new Error(`Semantic-layer source "${args.connectionId}/${args.sourceName}" was not found`); } + if (args.json) { + writeJsonResult(io, { + kind: 'sl.source', + data: source, + meta: { command: 'sl read' }, + }); + return 0; + } io.stdout.write(source.yaml); return 0; } @@ -108,6 +151,10 @@ export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: Ktx return 0; } if (args.command === 'query') { + const query = args.query ?? (args.queryFile ? await readSlQueryFile(args.queryFile) : undefined); + if (!query) { + throw new Error('sl query requires query input from --query-file or at least one --measure'); + } const compute = deps.createSemanticLayerCompute ? deps.createSemanticLayerCompute() : await (deps.createManagedSemanticLayerCompute ?? createManagedPythonSemanticLayerComputePort)({ @@ -118,7 +165,7 @@ export async function runKtxSl(args: KtxSlArgs, io: KtxSlIo = process, deps: Ktx const queryExecutor = args.execute ? (deps.createQueryExecutor ?? createDefaultLocalQueryExecutor)() : undefined; const result = await compileLocalSlQuery(project as KtxLocalProject, { connectionId: args.connectionId, - query: args.query, + query, compute, execute: args.execute, maxRows: args.maxRows, diff --git a/packages/cli/src/standalone-smoke.test.ts b/packages/cli/src/standalone-smoke.test.ts index a712e373..a6eb42d7 100644 --- a/packages/cli/src/standalone-smoke.test.ts +++ b/packages/cli/src/standalone-smoke.test.ts @@ -50,14 +50,6 @@ async function runBuiltCli(args: string[], options: { env?: NodeJS.ProcessEnv } } } -function getRunId(stdout: string): string { - const match = stdout.match(/^Run: (.+)$/m); - if (!match) { - throw new Error(`Could not find run id in output:\n${stdout}`); - } - return match[1]; -} - async function writeWarehouseConfig(projectDir: string): Promise { await writeFile( join(projectDir, 'ktx.yaml'), @@ -134,10 +126,6 @@ async function writeSqliteScanConfig(projectDir: string, dbPath: string, enrich ); } -function parseJsonOutput(stdout: string): T { - return JSON.parse(stdout) as T; -} - function expectProjectStderr(result: CliResult, projectDir: string): void { expect(result).toMatchObject({ code: 0, stderr: `Project: ${projectDir}\n` }); } @@ -181,7 +169,6 @@ describe('standalone built ktx CLI smoke', () => { await writeSourceFixture(sourceDir); const run = await runBuiltCli([ - 'dev', 'ingest', 'run', '--project-dir', @@ -199,49 +186,21 @@ describe('standalone built ktx CLI smoke', () => { ); }); - it('prints guided JSON for agent semantic-layer search outside a project through the built binary', async () => { - const projectDir = join(tempDir, 'missing-search-project'); - await mkdir(projectDir, { recursive: true }); - - const result = await runBuiltCli([ - 'agent', - 'sl', - 'list', - '--json', - '--query', - 'revenue', - '--project-dir', - projectDir, - ]); + it('rejects the removed agent command through the built binary', async () => { + const result = await runBuiltCli(['agent']); expect(result.code).toBe(1); expect(result.stdout).toBe(''); - const errorJson = parseJsonOutput<{ - ok: false; - error: { code: string; message: string; nextSteps: string[] }; - }>(result.stderr); - expect(errorJson).toEqual({ - ok: false, - error: { - code: 'agent_sl_search_missing_project', - message: `Semantic-layer search needs an initialized KTX project at ${projectDir}.`, - nextSteps: [ - `ktx setup --project-dir ${projectDir}`, - `ktx status --project-dir ${projectDir}`, - 'ktx ingest ', - `ktx agent sl list --json --query "revenue" --project-dir ${projectDir}`, - ], - }, - }); + expect(result.stderr).toContain("unknown command 'agent'"); }); it('runs doctor setup through the built binary', async () => { const result = await runBuiltCli(['status', '--no-input']); - expect(result.stdout).toContain('KTX setup doctor'); + expect(result.stdout).toMatch(/KTX (setup|project) doctor/); expect(result.stdout).toContain('Node 22+'); expect(result.stdout).toContain('Workspace-local CLI'); - expect(result.stderr).toBe(''); + expect(result.stderr === '' || result.stderr.startsWith('Project: ')).toBe(true); expect([0, 1]).toContain(result.code); }); @@ -260,31 +219,11 @@ describe('standalone built ktx CLI smoke', () => { expect(connectionTest.stdout).toContain('Driver: sqlite'); expect(connectionTest.stdout).toContain('Tables: 2'); - const structural = await runBuiltCli(['dev', 'scan', 'warehouse', '--project-dir', projectDir]); + const structural = await runBuiltCli(['scan', 'warehouse', '--project-dir', projectDir]); expectProjectStderr(structural, projectDir); expect(structural.stdout).toContain('Status: done'); expect(structural.stdout).toContain('Mode: structural'); - const structuralRunId = getRunId(structural.stdout); - - const structuralReportResult = await runBuiltCli([ - 'dev', - 'scan', - 'report', - '--json', - '--project-dir', - projectDir, - structuralRunId, - ]); - expect(structuralReportResult).toMatchObject({ code: 0, stderr: '' }); - const structuralReport = parseJsonOutput<{ - mode: string; - artifactPaths: { manifestShards: string[]; enrichmentArtifacts: string[] }; - manifestShardsWritten: number; - }>(structuralReportResult.stdout); - expect(structuralReport.mode).toBe('structural'); - expect(structuralReport.artifactPaths.manifestShards).toEqual(['semantic-layer/warehouse/_schema/public.yaml']); - expect(structuralReport.artifactPaths.enrichmentArtifacts).toEqual([]); - expect(structuralReport.manifestShardsWritten).toBe(1); + expect(structural.stdout).toContain('Schema shards: 1'); const structuralManifest = await readFile( join(projectDir, 'semantic-layer/warehouse/_schema/public.yaml'), @@ -296,7 +235,6 @@ describe('standalone built ktx CLI smoke', () => { expect(structuralManifest).not.toContain('ai:'); const providerlessEnriched = await runBuiltCli([ - 'dev', 'scan', 'warehouse', '--project-dir', @@ -310,89 +248,11 @@ describe('standalone built ktx CLI smoke', () => { expect(providerlessEnriched.stdout).toContain('Accepted: 1'); expect(providerlessEnriched.stdout).toContain('scan_enrichment_backend_not_configured'); expect(providerlessEnriched.stdout).toContain('Enrichment artifacts: 3'); - const providerlessRunId = getRunId(providerlessEnriched.stdout); - - const providerlessReportResult = await runBuiltCli([ - 'dev', - 'scan', - 'report', - '--json', - '--project-dir', - projectDir, - providerlessRunId, - ]); - expect(providerlessReportResult).toMatchObject({ code: 0, stderr: '' }); - const providerlessReport = parseJsonOutput<{ - mode: string; - enrichment: { - tableDescriptions: string; - columnDescriptions: string; - embeddings: string; - deterministicRelationships: string; - statisticalValidation: string; - }; - relationships: { accepted: number; review: number; rejected: number; skipped: number }; - warnings: Array<{ code: string }>; - artifactPaths: { enrichmentArtifacts: string[]; manifestShards: string[] }; - }>(providerlessReportResult.stdout); - expect(providerlessReport.mode).toBe('enriched'); - expect(providerlessReport.enrichment).toMatchObject({ - tableDescriptions: 'skipped', - columnDescriptions: 'skipped', - embeddings: 'skipped', - deterministicRelationships: 'completed', - statisticalValidation: 'completed', - }); - expect(providerlessReport.relationships).toEqual({ accepted: 1, review: 0, rejected: 0, skipped: 0 }); - expect(providerlessReport.warnings).toEqual( - expect.arrayContaining([expect.objectContaining({ code: 'scan_enrichment_backend_not_configured' })]), - ); - expect(providerlessReport.artifactPaths.enrichmentArtifacts).toEqual( - expect.arrayContaining([ - expect.stringContaining('/enrichment/relationships.json'), - expect.stringContaining('/enrichment/relationship-profile.json'), - expect.stringContaining('/enrichment/relationship-diagnostics.json'), - ]), - ); - expect(providerlessReport.artifactPaths.manifestShards).toEqual(['semantic-layer/warehouse/_schema/public.yaml']); - await writeSqliteScanConfig(projectDir, dbPath, true); - const enriched = await runBuiltCli(['dev', 'scan', 'warehouse', '--project-dir', projectDir, '--mode', 'enriched']); + const enriched = await runBuiltCli(['scan', 'warehouse', '--project-dir', projectDir, '--mode', 'enriched']); expectProjectStderr(enriched, projectDir); expect(enriched.stdout).toContain('Mode: enriched'); - const enrichedRunId = getRunId(enriched.stdout); - - const enrichedReportResult = await runBuiltCli([ - 'dev', - 'scan', - 'report', - '--json', - '--project-dir', - projectDir, - enrichedRunId, - ]); - expect(enrichedReportResult).toMatchObject({ code: 0, stderr: '' }); - const enrichedReport = parseJsonOutput<{ - mode: string; - enrichment: { tableDescriptions: string; columnDescriptions: string; embeddings: string }; - artifactPaths: { enrichmentArtifacts: string[]; manifestShards: string[] }; - }>(enrichedReportResult.stdout); - expect(enrichedReport.mode).toBe('enriched'); - expect(enrichedReport.enrichment).toMatchObject({ - tableDescriptions: 'completed', - columnDescriptions: 'completed', - embeddings: 'completed', - }); - expect(enrichedReport.artifactPaths.enrichmentArtifacts).toEqual( - expect.arrayContaining([ - expect.stringContaining('/enrichment/descriptions.json'), - expect.stringContaining('/enrichment/embeddings.json'), - expect.stringContaining('/enrichment/relationships.json'), - expect.stringContaining('/enrichment/relationship-profile.json'), - expect.stringContaining('/enrichment/relationship-diagnostics.json'), - ]), - ); - expect(enrichedReport.artifactPaths.manifestShards).toEqual(['semantic-layer/warehouse/_schema/public.yaml']); + expect(enriched.stdout).toContain('Enrichment artifacts:'); const enrichedManifest = await readFile(join(projectDir, 'semantic-layer/warehouse/_schema/public.yaml'), 'utf-8'); expect(enrichedManifest).toContain('Deterministic description'); diff --git a/packages/context/src/ingest/local-bundle-runtime.test.ts b/packages/context/src/ingest/local-bundle-runtime.test.ts index d8cd3907..af40d44f 100644 --- a/packages/context/src/ingest/local-bundle-runtime.test.ts +++ b/packages/context/src/ingest/local-bundle-runtime.test.ts @@ -55,7 +55,7 @@ describe('createLocalBundleIngestRuntime', () => { }), ).toThrow( [ - 'ktx dev ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.', + 'ktx ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.', `Configure an Anthropic provider, then rerun ingest:`, ` ktx setup --project-dir ${project.projectDir} --anthropic-api-key-env ANTHROPIC_API_KEY --anthropic-model claude-sonnet-4-6 --no-input`, ].join('\n'), diff --git a/packages/context/src/ingest/local-bundle-runtime.ts b/packages/context/src/ingest/local-bundle-runtime.ts index 38e37b1d..b453cae6 100644 --- a/packages/context/src/ingest/local-bundle-runtime.ts +++ b/packages/context/src/ingest/local-bundle-runtime.ts @@ -559,7 +559,7 @@ function nextLocalJobId(): string { function localIngestLlmProviderGuardMessage(projectDir: string): string { return [ - 'ktx dev ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.', + 'ktx ingest run requires llm.provider.backend: anthropic, vertex, or gateway, or an injected agentRunner.', 'Configure an Anthropic provider, then rerun ingest:', ` ktx setup --project-dir ${projectDir} --anthropic-api-key-env ANTHROPIC_API_KEY --anthropic-model claude-sonnet-4-6 --no-input`, ].join('\n'); diff --git a/packages/context/src/search/pglite-runtime-boundary.test.ts b/packages/context/src/search/pglite-runtime-boundary.test.ts index ce2f5b7a..2db3209b 100644 --- a/packages/context/src/search/pglite-runtime-boundary.test.ts +++ b/packages/context/src/search/pglite-runtime-boundary.test.ts @@ -46,7 +46,8 @@ describe('PGlite hybrid search runtime boundary', () => { } const productionRoutingFiles = [ - 'packages/cli/src/agent.ts', + 'packages/cli/src/sl.ts', + 'packages/cli/src/knowledge.ts', 'packages/context/src/mcp/local-project-ports.ts', 'packages/context/src/wiki/local-knowledge.ts', 'packages/context/src/ingest/context-evidence/sqlite-context-evidence-store.ts', diff --git a/packages/llm/src/model-provider.test.ts b/packages/llm/src/model-provider.test.ts index ff65a12a..e89c7c17 100644 --- a/packages/llm/src/model-provider.test.ts +++ b/packages/llm/src/model-provider.test.ts @@ -55,6 +55,7 @@ describe('createKtxLlmProvider', () => { }, { createAnthropic: vi.fn(() => vi.fn(() => anthropicModel)), + devtoolsEnabled: false, wrapLanguageModel, devToolsMiddleware, } satisfies KtxLlmProviderFactoryDeps, @@ -145,7 +146,7 @@ describe('createKtxLlmProvider', () => { modelSlots: { default: 'claude-sonnet-4-6' }, promptCaching: { enabled: false }, }, - { createAnthropic }, + { createAnthropic, devtoolsEnabled: false }, ); expect(provider.getModel('default')).toBe(anthropicModel); @@ -171,7 +172,7 @@ describe('createKtxLlmProvider', () => { modelSlots: { default: 'claude-sonnet-4-6' }, promptCaching: { enabled: false }, }, - { createVertexAnthropic }, + { createVertexAnthropic, devtoolsEnabled: false }, ); expect(provider.getModel('default')).toBe(vertexModel); @@ -191,7 +192,7 @@ describe('createKtxLlmProvider', () => { modelSlots: { default: 'anthropic/claude-sonnet-4-6' }, promptCaching: { enabled: false }, }, - { createGateway }, + { createGateway, devtoolsEnabled: false }, ); expect(provider.getModel('curator')).toBe(gatewayModel); diff --git a/scripts/examples-docs.test.mjs b/scripts/examples-docs.test.mjs index 661d66ac..793566ed 100644 --- a/scripts/examples-docs.test.mjs +++ b/scripts/examples-docs.test.mjs @@ -152,14 +152,12 @@ describe('standalone example docs', () => { const servingAgents = await readText('docs-site/content/docs/guides/serving-agents.mdx'); for (const command of [ - 'ktx agent tools --json', - 'ktx agent context --json', - 'ktx agent sl list --json', - 'ktx agent sl read orders --json', - 'ktx agent sl query --json', - 'ktx agent wiki search "revenue recognition" --json', - 'ktx agent wiki read order-status-definitions --json', - 'ktx agent sql execute --json', + 'ktx status --json', + 'ktx sl list --json', + 'ktx sl read orders --json', + 'ktx sl query --json', + 'ktx wiki search "revenue recognition" --json', + 'ktx wiki read order-status-definitions --json', ]) { assert.match(servingAgents, new RegExp(command.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'))); } @@ -184,12 +182,11 @@ describe('standalone example docs', () => { assert.match(quickstart, publicPackagePattern('npm install -g {package}')); assert.match(quickstart, /ktx dev runtime install --feature local-embeddings --yes/); assert.match(quickstart, /ktx dev runtime start --feature local-embeddings/); - assert.match(quickstart, /Install `uv`, run `ktx dev runtime doctor`/); + assert.match(quickstart, /Install `uv`, run `ktx dev runtime status`/); assert.match(packageArtifacts, /requires `uv` on `PATH`/); assert.match(packageArtifacts, /ktx dev runtime status/); - assert.match(packageArtifacts, /ktx dev runtime doctor/); - assert.match(packageArtifacts, /ktx dev runtime prune --dry-run/); - assert.match(packageArtifacts, /ktx dev runtime prune --yes/); + assert.match(packageArtifacts, /ktx dev runtime status/); + assert.doesNotMatch(packageArtifacts, /ktx dev runtime prune/); assert.match( packageArtifacts, new RegExp( @@ -221,9 +218,8 @@ describe('standalone example docs', () => { assert.doesNotMatch(readme, /installs the Python artifacts directly/); assert.match(readme, /requires `uv` on `PATH`/); assert.match(readme, /ktx dev runtime status/); - assert.match(readme, /ktx dev runtime doctor/); - assert.match(readme, /ktx dev runtime prune --dry-run/); - assert.match(readme, /ktx dev runtime prune --yes/); + assert.match(readme, /ktx dev runtime status/); + assert.doesNotMatch(readme, /ktx dev runtime prune/); assert.doesNotMatch(readme, /@ktx\/context/); assert.doesNotMatch(readme, /@ktx\/cli/); assert.doesNotMatch(readme, /python -m ktx_daemon semantic-validate/); @@ -234,14 +230,15 @@ describe('standalone example docs', () => { const buildingContext = await readText('docs-site/content/docs/guides/building-context.mdx'); const scanReference = await readText('docs-site/content/docs/cli-reference/ktx-scan.mdx'); - assert.match(buildingContext, /ktx dev scan /); - assert.match(buildingContext, /ktx dev scan status /); - assert.match(buildingContext, /ktx dev scan report /); - assert.match(scanReference, /ktx dev scan \[options\]/); + assert.match(buildingContext, /ktx scan /); + assert.match(buildingContext, /ktx status/); + assert.doesNotMatch(buildingContext, /ktx scan status /); + assert.doesNotMatch(buildingContext, /ktx scan report /); + assert.match(scanReference, /ktx scan \[options\]/); assert.match(rootReadme, /raw-sources\//); assert.match(rootReadme, /live-database\//); assert.doesNotMatch(rootReadme, /Run a local ingest smoke test/); - assert.doesNotMatch(rootReadme, /ktx dev ingest run --project-dir/); + assert.doesNotMatch(rootReadme, /ktx ingest run --project-dir/); assert.doesNotMatch(rootReadme, /ktx ingest status --project-dir/); }); diff --git a/scripts/installed-live-database-smoke.mjs b/scripts/installed-live-database-smoke.mjs index 75c3cc61..653f53b9 100644 --- a/scripts/installed-live-database-smoke.mjs +++ b/scripts/installed-live-database-smoke.mjs @@ -107,7 +107,6 @@ export function buildLiveDatabaseIngestArgs(projectDir, databaseIntrospectionUrl return [ 'exec', 'ktx', - 'dev', 'ingest', 'run', '--project-dir', @@ -325,12 +324,12 @@ async function main() { env: managedRuntimeEnv(cleanInstallDir), timeout: 120_000, }); - requireSuccess('ktx dev ingest run live-database', ingestRun); - requireOutput('ktx dev ingest run live-database', ingestRun, /Status: done/); - requireOutput('ktx dev ingest run live-database', ingestRun, /Adapter: live-database/); - requireOutput('ktx dev ingest run live-database', ingestRun, /Diff: \+4\/~0\/-0\/=0/); - requireOutput('ktx dev ingest run live-database', ingestRun, /Raw files: 4/); - requireOutput('ktx dev ingest run live-database', ingestRun, /Work units: 2/); + requireSuccess('ktx ingest run live-database', ingestRun); + requireOutput('ktx ingest run live-database', ingestRun, /Status: done/); + requireOutput('ktx ingest run live-database', ingestRun, /Adapter: live-database/); + requireOutput('ktx ingest run live-database', ingestRun, /Diff: \+4\/~0\/-0\/=0/); + requireOutput('ktx ingest run live-database', ingestRun, /Raw files: 4/); + requireOutput('ktx ingest run live-database', ingestRun, /Work units: 2/); const runId = getRunId(ingestRun.stdout); const ingestStatus = await run('pnpm', buildLiveDatabaseStatusArgs(projectDir, runId), { diff --git a/scripts/installed-live-database-smoke.test.mjs b/scripts/installed-live-database-smoke.test.mjs index a3d6be9e..c62e98b0 100644 --- a/scripts/installed-live-database-smoke.test.mjs +++ b/scripts/installed-live-database-smoke.test.mjs @@ -102,7 +102,6 @@ describe('installed live-database artifact smoke helpers', () => { assert.deepEqual(buildLiveDatabaseIngestArgs('/tmp/project', 'http://127.0.0.1:8765'), [ 'exec', 'ktx', - 'dev', 'ingest', 'run', '--project-dir', diff --git a/scripts/local-embeddings-runtime-smoke.mjs b/scripts/local-embeddings-runtime-smoke.mjs index 064cd070..8b0243a0 100644 --- a/scripts/local-embeddings-runtime-smoke.mjs +++ b/scripts/local-embeddings-runtime-smoke.mjs @@ -205,6 +205,17 @@ function parseJsonStdout(label, result) { } } +function parseJsonStdoutWithExitCode(label, result, expectedCode) { + if (result.code !== expectedCode) { + throw new Error(`${label} failed with code ${result.code}\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`); + } + try { + return JSON.parse(result.stdout); + } catch (error) { + throw new Error(`${label} did not write JSON stdout: ${error.message}\nstdout:\n${result.stdout}`); + } +} + function requireOutput(label, result, pattern) { if (!pattern.test(result.stdout)) { throw new Error(`${label} stdout did not match ${pattern}\nstdout:\n${result.stdout}`); @@ -283,13 +294,14 @@ export async function runLocalEmbeddingsRuntimeSmoke(options = {}) { requireSuccess(commands[0].label, version); requireOutput(commands[0].label, version, expectedPublicKtxVersionPattern()); - const missingStatus = parseJsonStdout( + const missingStatus = parseJsonStdoutWithExitCode( commands[1].label, await run(commands[1].command, commands[1].args, { cwd: installDir, env: smokeEnv, timeoutMs: commands[1].timeoutMs, }), + 1, ); if (missingStatus.kind !== 'missing') { throw new Error(`Expected missing runtime before install, got ${JSON.stringify(missingStatus)}`); diff --git a/scripts/package-artifacts.mjs b/scripts/package-artifacts.mjs index 07b9aaa7..2428dd2e 100644 --- a/scripts/package-artifacts.mjs +++ b/scripts/package-artifacts.mjs @@ -548,10 +548,13 @@ function parseJsonResult(label, result) { return JSON.parse(result.stdout); } -function parseJsonFailure(label, result) { - assert.equal(result.code, 1, label + ' should fail with exit code 1'); - assert.equal(result.stdout, '', label + ' should not write stdout when failing'); - return JSON.parse(result.stderr); +function parseJsonResultWithExitCode(label, result, expectedCode) { + assert.equal( + result.code, + expectedCode, + label + ' failed with code ' + result.code + '\\nstdout:\\n' + result.stdout + '\\nstderr:\\n' + result.stderr, + ); + return JSON.parse(result.stdout); } function requireIncludes(values, expected, label) { @@ -594,38 +597,15 @@ try { requireSuccess('ktx public package version', version); requireOutput('ktx public package version', version, /@kaelio\\/ktx 0\\.1\\.0/); - const runtimeStatusBefore = parseJsonResult( + const runtimeStatusBefore = parseJsonResultWithExitCode( 'ktx dev runtime status missing', await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'status', '--json']), + 1, ); assert.equal(runtimeStatusBefore.kind, 'missing'); assert.equal(runtimeStatusBefore.layout.runtimeRoot, process.env.KTX_RUNTIME_ROOT); process.stdout.write('ktx managed runtime starts missing in isolated root\\n'); - const missingProjectDir = join(root, 'missing-project'); - await mkdir(missingProjectDir, { recursive: true }); - const missingProjectSearch = await run('pnpm', [ - 'exec', - 'ktx', - 'agent', - 'sl', - 'list', - '--json', - '--query', - 'revenue', - '--project-dir', - missingProjectDir, - ]); - const missingProjectError = parseJsonFailure('ktx agent sl list missing project', missingProjectSearch); - assert.equal(missingProjectError.error.code, 'agent_sl_search_missing_project'); - assert.deepEqual(missingProjectError.error.nextSteps, [ - 'ktx setup --project-dir ' + missingProjectDir, - 'ktx status --project-dir ' + missingProjectDir, - 'ktx ingest ', - 'ktx agent sl list --json --query "revenue" --project-dir ' + missingProjectDir, - ]); - process.stdout.write('ktx agent sl list missing project guidance verified\\n'); - const init = await run('pnpm', [ 'exec', 'ktx', @@ -661,28 +641,6 @@ try { '--skip-agents', ]); requireProjectStderr('ktx setup empty project', emptyInit, emptyProjectDir); - const emptySearch = await run('pnpm', [ - 'exec', - 'ktx', - 'agent', - 'sl', - 'list', - '--json', - '--query', - 'revenue', - '--project-dir', - emptyProjectDir, - ]); - const emptySearchError = parseJsonFailure('ktx agent sl list no connections', emptySearch); - assert.equal(emptySearchError.error.code, 'agent_sl_search_no_connections'); - assert.deepEqual(emptySearchError.error.nextSteps, [ - 'ktx setup --project-dir ' + emptyProjectDir, - 'ktx status --project-dir ' + emptyProjectDir, - 'ktx ingest ', - 'ktx agent sl list --json --query "revenue" --project-dir ' + emptyProjectDir, - ]); - process.stdout.write('ktx agent sl list no connections guidance verified\\n'); - await writeFile( join(projectDir, 'ktx.yaml'), [ @@ -727,10 +685,9 @@ try { 'utf-8', ); - const agentWikiSearch = await run('pnpm', [ + const wikiSearch = await run('pnpm', [ 'exec', 'ktx', - 'agent', 'wiki', 'search', 'revenue', @@ -740,40 +697,17 @@ try { '--project-dir', projectDir, ]); - const agentWikiSearchJson = parseJsonResult('ktx agent wiki search', agentWikiSearch); - assert.equal(agentWikiSearchJson.totalFound, 1); - assert.equal(agentWikiSearchJson.results[0].key, 'revenue'); - assert.equal(agentWikiSearchJson.results[0].path, 'knowledge/global/revenue.md'); - assert.equal(typeof agentWikiSearchJson.results[0].score, 'number'); - requireIncludes(agentWikiSearchJson.results[0].matchReasons, 'lexical', 'agent wiki search match reasons'); - process.stdout.write('ktx agent wiki search hybrid metadata verified\\n'); + const wikiSearchJson = parseJsonResult('ktx wiki search', wikiSearch); + assert.equal(wikiSearchJson.kind, 'list'); + assert.equal(wikiSearchJson.data.items.length, 1); + assert.equal(wikiSearchJson.data.items[0].key, 'revenue'); + assert.equal(wikiSearchJson.data.items[0].path, 'knowledge/global/revenue.md'); + assert.equal(typeof wikiSearchJson.data.items[0].score, 'number'); + requireIncludes(wikiSearchJson.data.items[0].matchReasons, 'lexical', 'wiki search match reasons'); + process.stdout.write('ktx wiki search hybrid metadata verified\\n'); await access(join(projectDir, '.ktx', 'db.sqlite')); process.stdout.write('SQLite knowledge index: ' + join(projectDir, '.ktx', 'db.sqlite') + '\\n'); - const noSourceSearch = await run('pnpm', [ - 'exec', - 'ktx', - 'agent', - 'sl', - 'list', - '--json', - '--connection-id', - 'warehouse', - '--query', - 'revenue', - '--project-dir', - projectDir, - ]); - const noSourceSearchError = parseJsonFailure('ktx agent sl list no indexed sources', noSourceSearch); - assert.equal(noSourceSearchError.error.code, 'agent_sl_search_no_indexed_sources'); - assert.deepEqual(noSourceSearchError.error.nextSteps, [ - 'ktx setup --project-dir ' + projectDir, - 'ktx status --project-dir ' + projectDir, - 'ktx ingest ', - 'ktx agent sl list --json --query "revenue" --project-dir ' + projectDir, - ]); - process.stdout.write('ktx agent sl list no indexed sources guidance verified\\n'); - const slYaml = [ 'name: orders', 'table: orders', @@ -794,10 +728,9 @@ try { await mkdir(join(projectDir, 'semantic-layer', 'warehouse'), { recursive: true }); await writeFile(join(projectDir, 'semantic-layer', 'warehouse', 'orders.yaml'), slYaml, 'utf-8'); - const agentSlSearch = await run('pnpm', [ + const slSearch = await run('pnpm', [ 'exec', 'ktx', - 'agent', 'sl', 'list', '--json', @@ -808,13 +741,14 @@ try { '--project-dir', projectDir, ]); - const agentSlSearchJson = parseJsonResult('ktx agent sl list', agentSlSearch); - assert.equal(agentSlSearchJson.totalSources, 1); - assert.equal(agentSlSearchJson.sources[0].connectionId, 'warehouse'); - assert.equal(agentSlSearchJson.sources[0].name, 'orders'); - assert.equal(typeof agentSlSearchJson.sources[0].score, 'number'); - requireIncludes(agentSlSearchJson.sources[0].matchReasons, 'lexical', 'agent sl search match reasons'); - process.stdout.write('ktx agent sl list hybrid metadata verified\\n'); + const slSearchJson = parseJsonResult('ktx sl list', slSearch); + assert.equal(slSearchJson.kind, 'list'); + assert.equal(slSearchJson.data.items.length, 1); + assert.equal(slSearchJson.data.items[0].connectionId, 'warehouse'); + assert.equal(slSearchJson.data.items[0].name, 'orders'); + assert.equal(typeof slSearchJson.data.items[0].score, 'number'); + requireIncludes(slSearchJson.data.items[0].matchReasons, 'lexical', 'sl search match reasons'); + process.stdout.write('ktx sl list hybrid metadata verified\\n'); const slQuery = await run('pnpm', ['exec', 'ktx', 'sl', 'query', '--connection-id', @@ -865,12 +799,11 @@ try { requireOutput('ktx sl query sqlite execute', sqliteSlQuery, /"rows": \\[\\s*\\[\\s*3\\s*\\]\\s*\\]/); process.stdout.write('ktx sl query sqlite execute verified\\n'); - const runtimeDoctor = await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'doctor']); - requireSuccess('ktx dev runtime doctor', runtimeDoctor); - requireOutput('ktx dev runtime doctor', runtimeDoctor, /PASS uv/); - requireOutput('ktx dev runtime doctor', runtimeDoctor, /PASS Bundled Python wheel/); - requireOutput('ktx dev runtime doctor', runtimeDoctor, /PASS Managed Python runtime/); - process.stdout.write('ktx dev runtime doctor verified\\n'); + const runtimeDoctor = await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'status']); + requireSuccess('ktx dev runtime status', runtimeDoctor); + requireOutput('ktx dev runtime status', runtimeDoctor, /KTX Python runtime/); + requireOutput('ktx dev runtime status', runtimeDoctor, /status: ready/); + process.stdout.write('ktx dev runtime status verified\\n'); const runtimeStart = await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'start']); requireSuccess('ktx dev runtime start', runtimeStart); @@ -890,28 +823,7 @@ try { requireOutput('ktx dev runtime stop', runtimeStop, /Stopped KTX Python daemon/); process.stdout.write('ktx dev runtime daemon lifecycle verified\\n'); - const staleRuntimeDir = join(process.env.KTX_RUNTIME_ROOT, '0.0.0'); - await mkdir(staleRuntimeDir, { recursive: true }); - - const runtimePruneDryRun = await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'prune', '--dry-run']); - requireSuccess('ktx dev runtime prune dry run', runtimePruneDryRun); - requireOutput('ktx dev runtime prune dry run', runtimePruneDryRun, /Stale KTX Python runtimes/); - requireOutput('ktx dev runtime prune dry run', runtimePruneDryRun, /0\\.0\\.0/); - await access(staleRuntimeDir); - - const runtimePruneNeedsConfirmation = await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'prune']); - assert.equal(runtimePruneNeedsConfirmation.code, 1, 'ktx dev runtime prune needs confirmation'); - assert.equal(runtimePruneNeedsConfirmation.stdout, '', 'ktx dev runtime prune needs confirmation wrote stdout'); - assert.match(runtimePruneNeedsConfirmation.stderr, /Refusing to prune without --yes/); - - const runtimePruneConfirmed = await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'prune', '--yes']); - requireSuccess('ktx dev runtime prune confirmed', runtimePruneConfirmed); - requireOutput('ktx dev runtime prune confirmed', runtimePruneConfirmed, /Removed stale KTX Python runtimes/); - requireOutput('ktx dev runtime prune confirmed', runtimePruneConfirmed, /0\\.0\\.0/); - await assert.rejects(() => access(staleRuntimeDir)); - process.stdout.write('ktx dev runtime prune verified\\n'); - - const structuralScan = await run('pnpm', ['exec', 'ktx', 'dev', 'scan', 'warehouse', + const structuralScan = await run('pnpm', ['exec', 'ktx', 'scan', 'warehouse', '--project-dir', projectDir, ]); @@ -920,34 +832,10 @@ try { requireOutput('ktx scan structural', structuralScan, /Mode: structural/); requireOutput('ktx scan structural', structuralScan, /Needs attention\\s+None/); const structuralScanRunId = getRunId(structuralScan.stdout); - - const scanStatus = await run('pnpm', ['exec', 'ktx', 'dev', 'scan', 'status', - '--project-dir', - projectDir, - structuralScanRunId, - ]); - requireProjectStderr('ktx scan status', scanStatus, projectDir); - requireOutput('ktx scan status', scanStatus, new RegExp('Run: ' + structuralScanRunId)); - requireOutput('ktx scan status', scanStatus, /Status: done/); - requireOutput('ktx scan status', scanStatus, /Mode: structural/); - - const scanReport = await run('pnpm', ['exec', 'ktx', 'dev', 'scan', 'report', - '--project-dir', - projectDir, - '--json', - structuralScanRunId, - ]); - requireSuccess('ktx scan report', scanReport); - const scanReportJson = JSON.parse(scanReport.stdout); - assert.equal(scanReportJson.mode, 'structural'); - assert.equal(scanReportJson.connectionId, 'warehouse'); - assert.equal(scanReportJson.manifestShardsWritten, 1); - assert.deepEqual(scanReportJson.artifactPaths.enrichmentArtifacts, []); - assert.deepEqual(scanReportJson.artifactPaths.manifestShards, ['semantic-layer/warehouse/_schema/public.yaml']); await access(join(projectDir, 'semantic-layer', 'warehouse', '_schema', 'public.yaml')); process.stdout.write('ktx scan structural verified: ' + structuralScanRunId + '\\n'); - const enrichedScan = await run('pnpm', ['exec', 'ktx', 'dev', 'scan', 'warehouse', + const enrichedScan = await run('pnpm', ['exec', 'ktx', 'scan', 'warehouse', '--project-dir', projectDir, '--mode', @@ -956,24 +844,14 @@ try { requireProjectStderr('ktx scan enriched', enrichedScan, projectDir); requireOutput('ktx scan enriched', enrichedScan, /Status: done/); requireOutput('ktx scan enriched', enrichedScan, /Mode: enriched/); + requireOutput('ktx scan enriched', enrichedScan, /Enrichment artifacts:/); const enrichedScanRunId = getRunId(enrichedScan.stdout); - const enrichedScanReport = await run('pnpm', ['exec', 'ktx', 'dev', 'scan', 'report', - '--project-dir', - projectDir, - '--json', - enrichedScanRunId, - ]); - requireSuccess('ktx scan enriched report', enrichedScanReport); - const enrichedScanReportJson = JSON.parse(enrichedScanReport.stdout); - assert.equal(enrichedScanReportJson.mode, 'enriched'); - assert.ok(enrichedScanReportJson.artifactPaths.enrichmentArtifacts.length > 0); - assert.deepEqual(enrichedScanReportJson.artifactPaths.manifestShards, ['semantic-layer/warehouse/_schema/public.yaml']); process.stdout.write('ktx scan enriched verified: ' + enrichedScanRunId + '\\n'); await mkdir(join(sourceDir, 'orders'), { recursive: true }); await writeFile(join(sourceDir, 'orders', 'orders.json'), '{"name":"orders"}\\n', 'utf-8'); - const ingestRun = await run('pnpm', ['exec', 'ktx', 'dev', 'ingest', 'run', + const ingestRun = await run('pnpm', ['exec', 'ktx', 'ingest', 'run', '--project-dir', projectDir, '--connection-id', @@ -983,14 +861,14 @@ try { '--source-dir', sourceDir, ]); - assert.equal(ingestRun.code, 1, 'ktx dev ingest run without an LLM provider must fail'); + assert.equal(ingestRun.code, 1, 'ktx ingest run without an LLM provider must fail'); assert.match( ingestRun.stderr, - /ktx dev ingest run requires llm\\.provider\\.backend: anthropic, vertex, or gateway, or an injected agentRunner/, + /ktx ingest run requires llm\\.provider\\.backend: anthropic, vertex, or gateway, or an injected agentRunner/, ); await access(join(projectDir, '.ktx', 'db.sqlite')); - process.stdout.write('ktx dev ingest provider guard verified\\n'); + process.stdout.write('ktx ingest provider guard verified\\n'); } finally { if (daemonStarted) { await run('pnpm', ['exec', 'ktx', 'dev', 'runtime', 'stop']); diff --git a/scripts/package-artifacts.test.mjs b/scripts/package-artifacts.test.mjs index 704f1a6f..112c3468 100644 --- a/scripts/package-artifacts.test.mjs +++ b/scripts/package-artifacts.test.mjs @@ -459,9 +459,9 @@ describe('verification snippets', () => { assert.doesNotMatch(source, /startSemanticDaemon/); assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'setup'/); assert.match(source, /knowledge', 'global', 'revenue\.md'/); - assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'agent',\s*'wiki',\s*'search'/); + assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'wiki',\s*'search'/); assert.match(source, /semantic-layer', 'warehouse', 'orders\.yaml'/); - assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'agent',\s*'sl',\s*'list'/); + assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'sl',\s*'list'/); assert.match(source, /orders\.order_count/); assert.match(source, /sqlite3/); assert.match(source, /driver: sqlite/); @@ -482,33 +482,26 @@ describe('verification snippets', () => { assert.match(source, /ktx dev runtime status ready/); assert.match(source, /runtimeStatusAfter\.kind, 'ready'/); assert.match(source, /runtimeStatusAfter\.manifest\.features/); - assert.match(source, /ktx dev runtime doctor/); - assert.match(source, /PASS Managed Python runtime/); + assert.match(source, /ktx dev runtime status/); + assert.match(source, /status: ready/); assert.match(source, /ktx dev runtime start/); assert.match(source, /ktx dev runtime start reuse/); assert.match(source, /Using existing KTX Python daemon/); assert.match(source, /ktx dev runtime stop/); - assert.match(source, /ktx dev runtime prune dry run/); - assert.match(source, /0\.0\.0/); - assert.match(source, /ktx dev runtime prune needs confirmation/); - assert.match(source, /Refusing to prune without --yes/); - assert.match(source, /ktx dev runtime prune confirmed/); - assert.match(source, /Removed stale KTX Python runtimes/); - assert.match(source, /assert\.rejects\(\(\) => access\(staleRuntimeDir\)\)/); - assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'dev',\s*'scan',\s*'warehouse'/); + assert.doesNotMatch(source, /ktx dev runtime prune/); + assert.doesNotMatch(source, /staleRuntimeDir/); + assert.match(source, /run\('pnpm', \[\s*'exec',\s*'ktx',\s*'scan',\s*'warehouse'/); assert.match(source, /'--mode',\s*'enriched'/); assert.doesNotMatch(source, /'--enrich'/); assert.match(source, /ktx scan structural verified/); assert.match(source, /ktx scan enriched verified/); - assert.match(source, /scanReportJson\.artifactPaths\.manifestShards/); - assert.match(source, /scanReportJson\.artifactPaths\.enrichmentArtifacts/); assert.match(source, /enrichment:/); assert.match(source, /mode: deterministic/); - assert.match(source, /run\('pnpm', \['exec', 'ktx', 'dev', 'ingest', 'run'/); + assert.match(source, /run\('pnpm', \['exec', 'ktx', 'ingest', 'run'/); assert.match(source, /access\(join\(projectDir, '\.ktx', 'db\.sqlite'\)\)/); assert.match(source, /SQLite knowledge index/); - assert.match(source, /ktx dev ingest run requires llm\\.provider\\.backend: anthropic, vertex, or gateway/); - assert.match(source, /ktx dev ingest provider guard verified/); + assert.match(source, /ktx ingest run requires llm\\.provider\\.backend: anthropic, vertex, or gateway/); + assert.match(source, /ktx ingest provider guard verified/); }); describe('npmCliSmokeSource', () => { diff --git a/scripts/relationship-orbit-verification.mjs b/scripts/relationship-orbit-verification.mjs index f480a404..81f81187 100644 --- a/scripts/relationship-orbit-verification.mjs +++ b/scripts/relationship-orbit-verification.mjs @@ -1,6 +1,6 @@ #!/usr/bin/env node -import { mkdir as fsMkdir, writeFile as fsWriteFile } from 'node:fs/promises'; +import { mkdir as fsMkdir, readFile as fsReadFile, writeFile as fsWriteFile } from 'node:fs/promises'; import { execFile as childExecFile } from 'node:child_process'; import { dirname, resolve } from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; @@ -89,11 +89,7 @@ function parseArgs(argv) { } export function buildOrbitScanArgv(input) { - return ['dev', 'scan', input.connectionId, '--enrich', '--project-dir', input.projectDir]; -} - -export function buildOrbitReportArgv(input) { - return ['dev', 'scan', 'report', '--json', '--project-dir', input.projectDir, input.runId]; + return ['scan', input.connectionId, '--mode', 'relationships', '--project-dir', input.projectDir]; } export function extractRunId(stdout) { @@ -101,6 +97,11 @@ export function extractRunId(stdout) { return match?.[1] ?? null; } +export function extractReportPath(stdout) { + const match = stdout.match(/^\s*Report:\s*(\S+)/m); + return match?.[1] ?? null; +} + function listLines(values) { if (!values || values.length === 0) { return ['- none']; @@ -203,11 +204,9 @@ export function formatOrbitVerificationMarkdown(result) { if (result.status === 'success') { lines.push( - '## JSON Report Command', + '## Scan Report Artifact', '', - '```bash', - result.reportCommand, - '```', + `- ${result.reportPath}`, '', ...formatSuccess(result), ); @@ -249,6 +248,7 @@ export async function runOrbitVerification(options = {}) { const now = options.now ?? (() => new Date()); const mkdir = options.mkdir ?? fsMkdir; const writeFile = options.writeFile ?? fsWriteFile; + const readFile = options.readFile ?? fsReadFile; const date = dateOnly(now()); const env = options.env ?? orbitVerificationEnv(projectDir); const runWithEnv = (argv, runnerOptions) => runner(argv, { ...runnerOptions, env }); @@ -284,33 +284,32 @@ export async function runOrbitVerification(options = {}) { scanStderr: scan.stderr, }; } else { - const reportArgv = buildOrbitReportArgv({ projectDir, runId }); - const reportOutput = await runBufferedWorkspaceKtx(runWithEnv, reportArgv, rootDir, execFile); - if (reportOutput.exitCode !== 0) { + const scanReportPath = extractReportPath(scan.stdout); + if (!scanReportPath) { result = { status: 'blocked', date, connectionId, projectDir, scanCommand: shellCommand(scanArgv), - scanExitCode: reportOutput.exitCode, - blocker: firstNonEmptyLine(reportOutput.stderr, reportOutput.stdout), - scanStdout: `${scan.stdout}\n${reportOutput.stdout}`.trim(), - scanStderr: `${scan.stderr}\n${reportOutput.stderr}`.trim(), + scanExitCode: scan.exitCode, + blocker: 'KTX scan completed without printing a report artifact path', + scanStdout: scan.stdout, + scanStderr: scan.stderr, }; } else { + const fullScanReportPath = resolve(projectDir, scanReportPath); result = { status: 'success', date, connectionId, projectDir, scanCommand: shellCommand(scanArgv), - reportCommand: shellCommand(reportArgv), + reportPath: fullScanReportPath, scanExitCode: scan.exitCode, - reportExitCode: reportOutput.exitCode, scanStdout: scan.stdout, scanStderr: scan.stderr, - report: JSON.parse(reportOutput.stdout), + report: JSON.parse(await readFile(fullScanReportPath, 'utf8')), }; } } diff --git a/scripts/relationship-orbit-verification.test.mjs b/scripts/relationship-orbit-verification.test.mjs index 017b2518..a6dc3607 100644 --- a/scripts/relationship-orbit-verification.test.mjs +++ b/scripts/relationship-orbit-verification.test.mjs @@ -3,9 +3,9 @@ import { readFile } from 'node:fs/promises'; import { dirname } from 'node:path'; import { describe, it } from 'node:test'; import { - buildOrbitReportArgv, buildOrbitScanArgv, defaultOrbitVerificationProjectDir, + extractReportPath, extractRunId, formatOrbitVerificationMarkdown, runOrbitVerification, @@ -59,24 +59,15 @@ describe('relationship Orbit verification helper', () => { ); }); - it('builds the current KTX launcher arguments for scan and JSON report commands', () => { + it('builds the current KTX launcher arguments for scan commands', () => { assert.deepEqual(buildOrbitScanArgv({ connectionId: 'orbit', projectDir: '/tmp/orbit-project' }), [ - 'dev', 'scan', 'orbit', - '--enrich', + '--mode', + 'relationships', '--project-dir', '/tmp/orbit-project', ]); - assert.deepEqual(buildOrbitReportArgv({ projectDir: '/tmp/orbit-project', runId: 'scan-orbit-1' }), [ - 'dev', - 'scan', - 'report', - '--json', - '--project-dir', - '/tmp/orbit-project', - 'scan-orbit-1', - ]); }); it('uses the checked-in Orbit verification project by default', async () => { @@ -95,22 +86,17 @@ describe('relationship Orbit verification helper', () => { runWorkspaceKtx: async (argv, options) => { calls.push(argv); envs.push(options.env); - if (argv[2] === 'report') { - options.stdout.write(successReportJson()); - return 0; - } - options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n'); + options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n Report: reports/scan-report.json\n'); return 0; }, + readFile: async () => successReportJson(), }); assert.equal(result.status, 'success'); assert.deepEqual(calls, [ - ['dev', 'scan', 'orbit', '--enrich', '--project-dir', defaultProjectDir], - ['dev', 'scan', 'report', '--json', '--project-dir', defaultProjectDir, 'scan-orbit-1'], + ['scan', 'orbit', '--mode', 'relationships', '--project-dir', defaultProjectDir], ]); assert.equal(envs[0].GIT_CEILING_DIRECTORIES, dirname(defaultProjectDir)); - assert.equal(envs[1].GIT_CEILING_DIRECTORIES, dirname(defaultProjectDir)); assert.equal(writes.length, 1); assert.match(writes[0].content, new RegExp(defaultProjectDir.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'))); }); @@ -129,19 +115,15 @@ describe('relationship Orbit verification helper', () => { writeFile: async () => {}, runWorkspaceKtx: async (argv, options) => { calls.push(argv); - if (argv[2] === 'report') { - options.stdout.write(successReportJson()); - return 0; - } - options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n'); + options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n Report: reports/scan-report.json\n'); return 0; }, + readFile: async () => successReportJson(), }); assert.equal(result.projectDir, '/tmp/orbit-project-from-env'); assert.deepEqual(calls, [ - ['dev', 'scan', 'orbit', '--enrich', '--project-dir', '/tmp/orbit-project-from-env'], - ['dev', 'scan', 'report', '--json', '--project-dir', '/tmp/orbit-project-from-env', 'scan-orbit-1'], + ['scan', 'orbit', '--mode', 'relationships', '--project-dir', '/tmp/orbit-project-from-env'], ]); } finally { if (previousProjectDir === undefined) { @@ -155,6 +137,7 @@ describe('relationship Orbit verification helper', () => { it('extracts the run id from human scan output', () => { assert.equal(extractRunId(`KTX scan completed\nStatus: done\nRun: scan-orbit-1\nConnection: orbit\n`), 'scan-orbit-1'); assert.equal(extractRunId('KTX scan completed without a run line\n'), null); + assert.equal(extractReportPath('Artifacts\n Report: reports/scan-report.json\n'), 'reports/scan-report.json'); }); it('formats successful Orbit verification evidence from the JSON report', () => { @@ -163,10 +146,9 @@ describe('relationship Orbit verification helper', () => { date: '2026-05-07', connectionId: 'orbit', projectDir: '/tmp/orbit-project', - scanCommand: 'pnpm run ktx -- dev scan orbit --enrich --project-dir /tmp/orbit-project', - reportCommand: 'pnpm run ktx -- dev scan report --json --project-dir /tmp/orbit-project scan-orbit-1', + scanCommand: 'pnpm run ktx -- scan orbit --mode relationships --project-dir /tmp/orbit-project', + reportPath: '/tmp/orbit-project/reports/scan-report.json', scanExitCode: 0, - reportExitCode: 0, scanStdout: 'KTX scan completed\nRun: scan-orbit-1\n', scanStderr: '', report: JSON.parse(successReportJson()), @@ -189,7 +171,7 @@ describe('relationship Orbit verification helper', () => { date: '2026-05-07', connectionId: 'orbit', projectDir: '/tmp/orbit-project', - scanCommand: 'pnpm run ktx -- dev scan orbit --enrich --project-dir /tmp/orbit-project', + scanCommand: 'pnpm run ktx -- scan orbit --mode relationships --project-dir /tmp/orbit-project', scanExitCode: 1, blocker: 'Connection "orbit" was not found', scanStdout: '', @@ -202,7 +184,7 @@ describe('relationship Orbit verification helper', () => { assert.doesNotMatch(markdown, /scan\.enrichment\.mode is required/); }); - it('runs scan then JSON report and writes success Markdown', async () => { + it('runs scan then reads the report artifact and writes success Markdown', async () => { const calls = []; const writes = []; const result = await runOrbitVerification({ @@ -216,19 +198,15 @@ describe('relationship Orbit verification helper', () => { }, runWorkspaceKtx: async (argv, options) => { calls.push(argv); - if (argv[2] === 'report') { - options.stdout.write(successReportJson()); - return 0; - } - options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n'); + options.stdout.write('KTX scan completed\nRun: scan-orbit-1\nConnection: orbit\n Report: reports/scan-report.json\n'); return 0; }, + readFile: async () => successReportJson(), }); assert.equal(result.status, 'success'); assert.deepEqual(calls, [ - ['dev', 'scan', 'orbit', '--enrich', '--project-dir', '/tmp/orbit-project'], - ['dev', 'scan', 'report', '--json', '--project-dir', '/tmp/orbit-project', 'scan-orbit-1'], + ['scan', 'orbit', '--mode', 'relationships', '--project-dir', '/tmp/orbit-project'], ]); assert.equal(writes.length, 1); assert.equal(writes[0].path, '/tmp/orbit-report.md'); diff --git a/scripts/run-ktx.mjs b/scripts/run-ktx.mjs index e595376f..a283dcae 100644 --- a/scripts/run-ktx.mjs +++ b/scripts/run-ktx.mjs @@ -83,10 +83,6 @@ async function isBuildStale(rootDir, binPath, fs) { return false; } -function isShellCompletionRequest(argv) { - return argv[0] === '__complete' || (argv[0] === 'dev' && argv[1] === '__complete'); -} - async function runBuffered(execFile, stdout, stderr, command, args, options) { try { const result = await execFile(command, args, { cwd: options.cwd, env: options.env, maxBuffer: 1024 * 1024 * 16 }); @@ -150,8 +146,7 @@ export async function runWorkspaceKtx(argv, options = {}) { const commandEnv = options.env; const binExists = await fileExists(binPath, access); - const skipStaleBuildCheck = binExists && isShellCompletionRequest(cliArgv); - const needsBuild = !binExists || (!skipStaleBuildCheck && (await isBuildStale(rootDir, binPath, fs))); + const needsBuild = !binExists || (await isBuildStale(rootDir, binPath, fs)); if (needsBuild) { stderr.write( binExists diff --git a/scripts/run-ktx.test.mjs b/scripts/run-ktx.test.mjs index 76315a36..3263ef30 100644 --- a/scripts/run-ktx.test.mjs +++ b/scripts/run-ktx.test.mjs @@ -105,53 +105,6 @@ test('runWorkspaceKtx drops a leading npm argument separator', async () => { ]); }); -test('runWorkspaceKtx skips stale-build checks for shell completion when dist exists', async () => { - const calls = []; - let statCalls = 0; - - const exitCode = await runWorkspaceKtx(['dev', '__complete', '--shell', 'zsh', '--position', '2', '--', 'ktx', ''], { - rootDir: '/workspace/ktx', - access: async () => undefined, - stat: async (path) => { - statCalls += 1; - return { - mtimeMs: path.endsWith('/packages/cli/dist/bin.js') ? 2000 : 3000, - isDirectory: () => path.endsWith('/src') || path.endsWith('/packages'), - }; - }, - readdir: async () => { - throw new Error('completion should not scan source directories'); - }, - execFile: async (command, args, options) => { - calls.push({ command, args, cwd: options.cwd }); - return { stdout: 'connect:Add, list, test, and map data sources\n', stderr: '' }; - }, - stdout: { write: () => undefined }, - stderr: { write: () => undefined }, - }); - - assert.equal(exitCode, 0); - assert.equal(statCalls, 0); - assert.deepEqual(calls, [ - { - command: process.execPath, - args: [ - '/workspace/ktx/packages/cli/dist/bin.js', - 'dev', - '__complete', - '--shell', - 'zsh', - '--position', - '2', - '--', - 'ktx', - '', - ], - cwd: '/workspace/ktx', - }, - ]); -}); - test('runWorkspaceKtx builds the workspace CLI before running it when dist is missing', async () => { const calls = []; const logs = []; @@ -199,7 +152,7 @@ test('runWorkspaceKtx rebuilds before running when workspace sources are newer t const logs = []; let sourceMtimeMs = 3000; - const exitCode = await runWorkspaceKtx(['dev', 'scan', 'orbit', '--enrich'], { + const exitCode = await runWorkspaceKtx(['scan', 'orbit', '--mode', 'relationships'], { rootDir: '/workspace/ktx', access: async () => undefined, stat: async (path) => ({ @@ -232,7 +185,7 @@ test('runWorkspaceKtx rebuilds before running when workspace sources are newer t calls.map((call) => [call.command, call.args]), [ ['pnpm', ['run', 'build']], - [process.execPath, ['/workspace/ktx/packages/cli/dist/bin.js', 'dev', 'scan', 'orbit', '--enrich']], + [process.execPath, ['/workspace/ktx/packages/cli/dist/bin.js', 'scan', 'orbit', '--mode', 'relationships']], ], ); assert.deepEqual(logs, [