mirror of
https://github.com/Kaelio/ktx.git
synced 2026-06-28 08:49:38 +02:00
Merge remote-tracking branch 'origin/main' into ktx-agent-command-history
# Conflicts: # packages/cli/src/agent-search-readiness.test.ts # packages/cli/src/agent-search-readiness.ts # packages/cli/src/agent.test.ts # packages/cli/src/project-dir.test.ts # packages/cli/src/setup-context.test.ts # packages/cli/src/standalone-smoke.test.ts # scripts/package-artifacts.mjs
This commit is contained in:
Andrey Avtomonov
commit
a8bf0470ec
56 changed files with 411 additions and 3910 deletions
|
|
@ -1,9 +1,9 @@
|
|||
---
|
||||
title: "ktx dev"
|
||||
description: "Low-level diagnostics, scans, adapter commands, and mapping tools."
|
||||
description: "Low-level project initialization and runtime management."
|
||||
---
|
||||
|
||||
Hidden commands for low-level project management, diagnostics, direct adapter control, and shell completion. Most users interact with these through higher-level commands like [`ktx ingest`](/docs/cli-reference/ktx-ingest) and [`ktx setup`](/docs/cli-reference/ktx-setup), but `ktx dev` provides direct access when you need fine-grained control.
|
||||
`ktx dev` contains development-only project initialization and managed runtime commands. Scan and ingest commands live at the root as [`ktx scan`](/docs/cli-reference/ktx-scan) and [`ktx ingest`](/docs/cli-reference/ktx-ingest).
|
||||
|
||||
## Command signature
|
||||
|
||||
|
|
@ -16,145 +16,45 @@ ktx dev <subcommand> [options]
|
|||
| Subcommand | Description |
|
||||
|-----------|-------------|
|
||||
| `init [directory]` | Initialize a Git-backed KTX project directory |
|
||||
| `runtime` | Install, inspect, and prune the KTX-managed Python runtime |
|
||||
| `scan` | Run or inspect standalone connection scans |
|
||||
| `ingest run` | Run local ingest for one configured connection and source adapter |
|
||||
| `ingest status [runId]` | Print status for a stored local ingest run |
|
||||
| `ingest watch [runId]` | Open a stored ingest visual report |
|
||||
| `ingest replay <runId>` | Replay a stored ingest run through memory-flow output |
|
||||
| `mapping` | Manage Metabase warehouse mappings (same as `ktx connection mapping`) |
|
||||
| `completion zsh` | Generate zsh completion script |
|
||||
| `runtime` | Install, start, stop, inspect, and prune the KTX-managed Python runtime |
|
||||
|
||||
## Options
|
||||
|
||||
### `dev init`
|
||||
## `dev init`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--name <name>` | Project name written to `ktx.yaml` | — |
|
||||
| `--force` | Rewrite `ktx.yaml` and scaffold files in an existing project | `false` |
|
||||
|
||||
### `dev runtime`
|
||||
## `dev runtime`
|
||||
|
||||
`ktx dev runtime` supports `install`, `start`, `stop`, `status`, and `prune`.
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--feature <feature>` | Runtime feature level for `install` and `start` (`core` or `local-embeddings`) | `core` |
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--feature <feature>` | Runtime feature level for `install`, `start`, and `status` (`core` or `local-embeddings`) | `core` |
|
||||
| `--json` | Print JSON output for `status` | `false` |
|
||||
| `--yes` | Confirm runtime install or prune actions where supported | `false` |
|
||||
| `--force` | Reinstall or restart where supported | `false` |
|
||||
|
||||
### `dev scan`
|
||||
|
||||
See [`ktx scan`](/docs/cli-reference/ktx-scan) for the full scan command reference.
|
||||
|
||||
### `dev ingest run`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection-id <connectionId>` | KTX connection id (required) | — |
|
||||
| `--adapter <adapter>` | Ingest source adapter name (required) | — |
|
||||
| `--source-dir <path>` | Directory containing source files | — |
|
||||
| `--database-introspection-url <url>` | Daemon URL for live-database introspection | — |
|
||||
| `--debug-llm-request-file <path>` | Write sanitized LLM request structure to a JSONL file | — |
|
||||
| `--plain` | Print plain text output | `false` |
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--viz` | Render memory-flow TUI output | `false` |
|
||||
| `--no-input` | Disable interactive terminal input for visualization | — |
|
||||
|
||||
### `dev ingest status`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--report-file <path>` | Bundle ingest report JSON file to render | — |
|
||||
| `--plain` | Print plain text output | `false` |
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--viz` | Render memory-flow TUI output | `false` |
|
||||
| `--no-input` | Disable interactive terminal input for visualization | — |
|
||||
|
||||
### `dev ingest watch`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--report-file <path>` | Bundle ingest report JSON file to render | — |
|
||||
| `--plain` | Print plain text output | `false` |
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--viz` | Render memory-flow TUI output (the default unless `--plain` or `--json` is set) | `true` |
|
||||
| `--no-input` | Disable interactive terminal input for visualization | — |
|
||||
|
||||
### `dev ingest replay`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--report-file <path>` | Bundle ingest report JSON file to render | — |
|
||||
| `--plain` | Print plain text output | `false` |
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--viz` | Render memory-flow TUI output | `false` |
|
||||
| `--no-input` | Disable interactive terminal input for visualization | — |
|
||||
|
||||
### `dev completion zsh`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--install` | Install zsh completion into `~/.zfunc` and update `~/.zshrc` | `false` |
|
||||
| `--dry-run` | Preview runtime pruning without removing files | `false` |
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Initialize a new KTX project
|
||||
ktx dev init
|
||||
|
||||
# Initialize in a specific directory with a project name
|
||||
ktx dev init ./my-project --name "Analytics Context"
|
||||
|
||||
# Re-initialize an existing project
|
||||
ktx dev init --force
|
||||
|
||||
# Check managed Python runtime readiness
|
||||
ktx dev runtime doctor
|
||||
|
||||
# Start the managed Python daemon
|
||||
ktx dev runtime install --yes
|
||||
ktx dev runtime status
|
||||
ktx dev runtime start
|
||||
|
||||
# Run a low-level ingest with a specific adapter
|
||||
ktx dev ingest run --connection-id my-dbt --adapter dbt
|
||||
|
||||
# Run ingest from a specific source directory
|
||||
ktx dev ingest run \
|
||||
--connection-id my-dbt \
|
||||
--adapter dbt \
|
||||
--source-dir ./dbt-project
|
||||
|
||||
# View ingest status with the visual TUI
|
||||
ktx dev ingest watch run-abc123
|
||||
|
||||
# Replay a stored ingest session
|
||||
ktx dev ingest replay run-abc123
|
||||
|
||||
# View ingest status from a report file
|
||||
ktx dev ingest status --report-file /tmp/ingest-report.json
|
||||
|
||||
# Generate zsh completions
|
||||
ktx dev completion zsh
|
||||
|
||||
# Install zsh completions
|
||||
ktx dev completion zsh --install
|
||||
ktx dev runtime stop
|
||||
ktx dev runtime prune --dry-run
|
||||
ktx dev runtime prune --yes
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
`ktx dev` commands are diagnostic and may print plain text, JSON, or visual reports depending on the selected flags.
|
||||
|
||||
| Mode | How to request it | Use case |
|
||||
|------|-------------------|----------|
|
||||
| Plain text | `--plain` or default diagnostic output | Human-readable terminal inspection |
|
||||
| JSON | `--json` | Agent parsing and automation |
|
||||
| Visual report | `--viz` | Interactive memory-flow and ingest debugging |
|
||||
|
||||
## Common errors
|
||||
|
||||
| Error | Cause | Recovery |
|
||||
|-------|-------|----------|
|
||||
| Doctor reports missing runtime pieces | Packages, Python environment, or linked CLI are not ready | Run `pnpm install`, `pnpm run setup:dev`, and `uv sync --all-groups` |
|
||||
| Ingest run cannot find adapter | `--adapter` does not match a supported source adapter | Use configured source names from `ktx.yaml` or run higher-level `ktx ingest` |
|
||||
| Replay/report file cannot be read | The report path is wrong or the run id is not stored locally | Run `ktx dev ingest status --json` to discover stored run ids and report files |
|
||||
| Visual output fails in CI | TUI rendering requires an interactive terminal | Use `--plain --no-input` or `--json --no-input` |
|
||||
| Runtime status reports missing pieces | Packages, Python environment, or linked CLI are not ready | Run `pnpm install`, `pnpm run setup:dev`, `uv sync --all-groups`, then `ktx dev runtime status` |
|
||||
| Runtime daemon does not start | The managed Python runtime is missing or stale | Run `ktx dev runtime install --yes`, then `ktx dev runtime start` |
|
||||
|
|
|
|||
|
|
@ -1,14 +1,13 @@
|
|||
---
|
||||
title: "ktx ingest"
|
||||
description: "Build and refresh context from configured sources."
|
||||
description: "Run and inspect local ingest memory-flow output."
|
||||
---
|
||||
|
||||
Ingest context from your configured sources — dbt, Looker, Metabase, MetricFlow, LookML, or Notion. The ingest process extracts metadata from your tools, then uses an LLM agent to reconcile it with existing context, writing semantic sources and knowledge pages to your project.
|
||||
`ktx ingest` runs adapter-level local ingest and renders stored ingest reports.
|
||||
|
||||
## Command signature
|
||||
|
||||
```bash
|
||||
ktx ingest [connectionId] [options]
|
||||
ktx ingest <subcommand> [options]
|
||||
```
|
||||
|
||||
|
|
@ -16,80 +15,59 @@ ktx ingest <subcommand> [options]
|
|||
|
||||
| Subcommand | Description |
|
||||
|-----------|-------------|
|
||||
| `status [runId]` | Print status for the latest or selected public ingest run |
|
||||
| `watch [runId]` | Open the latest or selected public ingest visual report |
|
||||
| `run` | Run local ingest for one configured connection and source adapter |
|
||||
| `status [runId]` | Print status for the latest or selected stored local ingest run or report file |
|
||||
| `watch [runId]` | Open the latest or selected stored ingest visual report |
|
||||
| `replay <runId>` | Replay a stored ingest run or bundle report through memory-flow output |
|
||||
|
||||
## Options
|
||||
|
||||
### `ingest` (run)
|
||||
## `ingest run`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--all` | Ingest every eligible configured source | `false` |
|
||||
| `--connection-id <connectionId>` | KTX connection id | Required |
|
||||
| `--adapter <adapter>` | Ingest source adapter name | Required |
|
||||
| `--source-dir <path>` | Directory containing source files | — |
|
||||
| `--database-introspection-url <url>` | Daemon URL for live-database introspection | — |
|
||||
| `--debug-llm-request-file <path>` | Write sanitized LLM request structure to a JSONL file | — |
|
||||
| `--plain` | Print plain text output | `true` |
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--no-input` | Disable interactive terminal input | — |
|
||||
| `--viz` | Render memory-flow TUI output | `false` |
|
||||
| `--yes` | Install the managed Python runtime without prompting when required | `false` |
|
||||
| `--no-input` | Disable interactive terminal input for visualization and runtime installation | — |
|
||||
|
||||
### `ingest status`
|
||||
## `ingest status`, `watch`, and `replay`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--report-file <path>` | Bundle ingest report JSON file to render | — |
|
||||
| `--plain` | Print plain text output | `true` for `status` and `replay` |
|
||||
| `--json` | Print JSON output | `false` |
|
||||
| `--no-input` | Disable interactive terminal input | — |
|
||||
|
||||
### `ingest watch`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--json` | Print JSON output instead of the visual report | `false` |
|
||||
| `--no-input` | Disable interactive terminal input | — |
|
||||
| `--viz` | Render memory-flow TUI output | `true` for `watch` |
|
||||
| `--no-input` | Disable interactive terminal input for visualization | — |
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Ingest from a specific connection
|
||||
ktx ingest my-dbt-source
|
||||
ktx ingest run --connection-id my-dbt-source --adapter dbt
|
||||
ktx ingest run --connection-id prod-metabase --adapter metabase --yes
|
||||
|
||||
# Ingest from all eligible sources
|
||||
ktx ingest --all
|
||||
|
||||
# Check the status of the latest ingest
|
||||
ktx ingest status
|
||||
|
||||
# Check the status of a specific ingest run
|
||||
ktx ingest status run-abc123
|
||||
|
||||
# Watch the latest ingest report
|
||||
ktx ingest watch
|
||||
|
||||
# Get ingest status as JSON
|
||||
ktx ingest status --json
|
||||
```
|
||||
|
||||
## Low-level ingest commands
|
||||
ktx ingest watch
|
||||
ktx ingest watch run-abc123
|
||||
|
||||
For adapter-level control, use `ktx dev ingest`. See [`ktx dev`](/docs/cli-reference/ktx-dev) for the full low-level ingest surface including `run`, `status`, `watch`, and `replay` with output mode options (`--plain`, `--json`, `--viz`).
|
||||
|
||||
## Output
|
||||
|
||||
Ingest run commands print progress and create a stored ingest report. `ktx ingest status --json` returns the run state, adapter, connection, and summary information.
|
||||
|
||||
```json
|
||||
{
|
||||
"runId": "ingest-local-abc123",
|
||||
"status": "completed",
|
||||
"connectionId": "dbt-main",
|
||||
"summary": {
|
||||
"semanticSourcesChanged": 4,
|
||||
"knowledgePagesChanged": 2
|
||||
}
|
||||
}
|
||||
ktx ingest replay run-abc123
|
||||
ktx ingest replay run-abc123 --viz
|
||||
ktx ingest replay run-abc123 --report-file /tmp/ingest-report.json
|
||||
```
|
||||
|
||||
## Common errors
|
||||
|
||||
| Error | Cause | Recovery |
|
||||
|-------|-------|----------|
|
||||
| No eligible sources | `ktx.yaml` has no configured context source for ingest | Add a source with `ktx setup` or `ktx connection add`, then rerun ingest |
|
||||
| Ingest needs credentials | The source adapter requires API or git access | Configure the referenced environment variable or secret file |
|
||||
| Latest run not found | No ingest run has been started in this project | Run `ktx ingest <connectionId>` or `ktx ingest --all` first |
|
||||
| Ingest run cannot find adapter | `--adapter` does not match a supported source adapter | Use a configured adapter such as `dbt`, `metabase`, `looker`, `lookml`, `notion`, or `live-database` |
|
||||
| Latest run not found | No ingest run has been started in this project | Run `ktx ingest run --connection-id <id> --adapter <adapter>` first |
|
||||
| Report watch fails in a non-interactive shell | Visual report needs a terminal | Use `ktx ingest status --json` for agent and CI workflows |
|
||||
|
|
|
|||
|
|
@ -1,163 +1,39 @@
|
|||
---
|
||||
title: "ktx scan"
|
||||
description: "Run or inspect database scans."
|
||||
description: "Run standalone database scans."
|
||||
---
|
||||
|
||||
Discover your database schema — tables, columns, types, constraints, and relationships. Scanning is the first step in building context: KTX needs to understand your warehouse structure before it can build semantic sources.
|
||||
|
||||
Scan commands live under `ktx dev scan`. See also the [Building Context](/docs/guides/building-context) guide for a walkthrough.
|
||||
Discover a configured database connection's schema, including tables, columns, types, constraints, and optional relationship signals.
|
||||
|
||||
## Command signature
|
||||
|
||||
```bash
|
||||
ktx dev scan <connectionId> [options]
|
||||
ktx dev scan <subcommand> [options]
|
||||
ktx scan <connectionId> [options]
|
||||
```
|
||||
|
||||
## Subcommands
|
||||
|
||||
| Subcommand | Description |
|
||||
|-----------|-------------|
|
||||
| `status <runId>` | Print status for a local scan run |
|
||||
| `report <runId>` | Print a local scan report |
|
||||
| `relationships <runId>` | Print relationship artifacts for a local scan run |
|
||||
| `relationship-apply <runId>` | Apply accepted relationship review decisions as manual manifest joins |
|
||||
| `relationship-feedback` | Export persisted relationship review decisions as calibration labels |
|
||||
| `relationship-calibration` | Summarize relationship feedback labels against current score thresholds |
|
||||
| `relationship-thresholds` | Evaluate relationship feedback labels for offline threshold advice |
|
||||
|
||||
## Options
|
||||
|
||||
### `scan` (run)
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--mode <mode>` | Scan mode: `structural`, `enriched`, or `relationships` | `structural` |
|
||||
| `--dry-run` | Run without writing scan results | `false` |
|
||||
| `--database-introspection-url <url>` | Daemon URL for live-database introspection | — |
|
||||
|
||||
### `scan report`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--json` | Print the raw scan report JSON | `false` |
|
||||
|
||||
### `scan relationships`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--status <status>` | Filter by status: `accepted`, `review`, `rejected`, `skipped`, or `all` | `review` |
|
||||
| `--limit <count>` | Maximum relationships to print per status | `25` |
|
||||
| `--accept <candidateId>` | Record an accepted decision for a relationship candidate | — |
|
||||
| `--reject <candidateId>` | Record a rejected decision for a relationship candidate | — |
|
||||
| `--note <text>` | Attach a note when recording a relationship review decision | — |
|
||||
| `--reviewer <name>` | Reviewer name for a relationship review decision | — |
|
||||
| `--json` | Print relationship artifacts as JSON | `false` |
|
||||
|
||||
### `scan relationship-apply`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--all-accepted` | Apply all accepted relationship review decisions for the scan run | `false` |
|
||||
| `--candidate <candidateId>` | Apply one accepted relationship review decision; repeatable | — |
|
||||
| `--dry-run` | Preview relationships that would be written without rewriting manifest shards | `false` |
|
||||
| `--json` | Print the apply result as JSON | `false` |
|
||||
|
||||
### `scan relationship-feedback`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection <connectionId>` | Only export labels for one KTX connection | — |
|
||||
| `--decision <decision>` | Filter: `accepted`, `rejected`, or `all` | `all` |
|
||||
| `--json` | Print the export as JSON | `false` |
|
||||
| `--jsonl` | Print labels as newline-delimited JSON | `false` |
|
||||
|
||||
### `scan relationship-calibration`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection <connectionId>` | Only calibrate labels for one KTX connection | — |
|
||||
| `--decision <decision>` | Filter: `accepted`, `rejected`, or `all` | `all` |
|
||||
| `--accept-threshold <value>` | Score threshold treated as predicted accepted (0–1) | `0.85` |
|
||||
| `--review-threshold <value>` | Score threshold treated as predicted review (0–1) | `0.55` |
|
||||
| `--json` | Print the calibration report as JSON | `false` |
|
||||
|
||||
### `scan relationship-thresholds`
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `--connection <connectionId>` | Only evaluate labels for one KTX connection | — |
|
||||
| `--min-total-labels <count>` | Minimum scored labels before advice can be ready | `20` |
|
||||
| `--min-accepted-labels <count>` | Minimum accepted labels before advice can be ready | `5` |
|
||||
| `--min-rejected-labels <count>` | Minimum rejected labels before advice can be ready | `5` |
|
||||
| `--json` | Print the threshold advice report as JSON | `false` |
|
||||
| `--yes` | Install the managed Python runtime without prompting when required | `false` |
|
||||
| `--no-input` | Disable interactive managed runtime installation | — |
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Run a structural scan of a connection
|
||||
ktx dev scan my-warehouse
|
||||
|
||||
# Run a scan with LLM enrichment
|
||||
ktx dev scan my-warehouse --mode enriched
|
||||
|
||||
# Run a scan with relationship detection
|
||||
ktx dev scan my-warehouse --mode relationships
|
||||
|
||||
# Dry-run a scan (don't write results)
|
||||
ktx dev scan my-warehouse --dry-run
|
||||
|
||||
# Check the status of a scan run
|
||||
ktx dev scan status run-abc123
|
||||
|
||||
# View the scan report
|
||||
ktx dev scan report run-abc123
|
||||
|
||||
# View scan report as JSON
|
||||
ktx dev scan report run-abc123 --json
|
||||
|
||||
# List relationship candidates pending review
|
||||
ktx dev scan relationships run-abc123
|
||||
|
||||
# List all relationships regardless of status
|
||||
ktx dev scan relationships run-abc123 --status all
|
||||
|
||||
# Accept a relationship candidate
|
||||
ktx dev scan relationships run-abc123 --accept candidate-xyz
|
||||
|
||||
# Reject a relationship candidate with a note
|
||||
ktx dev scan relationships run-abc123 --reject candidate-xyz --note "false positive"
|
||||
|
||||
# Apply all accepted relationships to the manifest
|
||||
ktx dev scan relationship-apply run-abc123 --all-accepted
|
||||
|
||||
# Preview what would be applied
|
||||
ktx dev scan relationship-apply run-abc123 --all-accepted --dry-run
|
||||
|
||||
# Export relationship feedback as calibration labels
|
||||
ktx dev scan relationship-feedback --json
|
||||
|
||||
# Calibrate relationship detection thresholds
|
||||
ktx dev scan relationship-calibration --accept-threshold 0.9 --review-threshold 0.6
|
||||
|
||||
# Get threshold advice based on review decisions
|
||||
ktx dev scan relationship-thresholds
|
||||
ktx scan my-warehouse
|
||||
ktx scan my-warehouse --mode enriched
|
||||
ktx scan my-warehouse --mode relationships
|
||||
ktx scan my-warehouse --dry-run
|
||||
ktx scan my-warehouse --database-introspection-url http://127.0.0.1:8765
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
Scan commands write scan artifacts under the KTX project directory and print status or report summaries. Use `--json` on report and relationship commands when an agent needs structured output.
|
||||
|
||||
```json
|
||||
{
|
||||
"runId": "scan-local-abc123",
|
||||
"status": "completed",
|
||||
"mode": "structural",
|
||||
"changes": {
|
||||
"tablesAdded": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
`ktx scan` prints a human summary and writes scan artifacts under the KTX project directory unless `--dry-run` is set. Use `ktx status` after a scan to inspect project readiness and next setup work.
|
||||
|
||||
## Common errors
|
||||
|
||||
|
|
@ -165,5 +41,4 @@ Scan commands write scan artifacts under the KTX project directory and print sta
|
|||
|-------|-------|----------|
|
||||
| Scan cannot connect | Connection credentials or network access are invalid | Run `ktx connection test <connectionId>` and update the connection before scanning |
|
||||
| Enriched scan cannot describe columns | LLM credentials are missing or invalid | Complete LLM setup with `ktx setup` before enriched scans |
|
||||
| Relationship apply writes nothing | No accepted candidates match the provided run id or candidate ids | Inspect `ktx dev scan relationships <runId> --status accepted` first |
|
||||
| Calibration is not ready | Too few reviewed relationship labels exist | Review and accept/reject more candidates, then rerun calibration |
|
||||
| Relationship scan has limited evidence | The connector cannot provide optional validation or statistics | Re-run with a connector that supports the missing capability, or treat relationship output as lower-confidence context |
|
||||
|
|
|
|||
|
|
@ -59,7 +59,7 @@ dbt / Looker / Metabase / Notion
|
|||
|
||||
A typical branch shows a semantic diff: "this ingest added 3 new sources from dbt, updated 2 join definitions based on schema changes, and created 1 knowledge page from a Notion doc." Analytics engineers review the diff, verify that the new sources look correct, and merge.
|
||||
|
||||
Teams usually run this on demand while setting up a source, then schedule it once the source is stable. A cron job or CI schedule can run `ktx ingest --all --no-input` overnight on an ingest branch so the latest dbt manifests, BI metadata, and documentation updates are ready for review each morning.
|
||||
Teams usually run this on demand while setting up a source, then schedule it once the source is stable. A cron job or CI schedule can run `ktx ingest run --connection-id <id> --adapter <adapter> --no-input` overnight on an ingest branch so the latest dbt manifests, BI metadata, and documentation updates are ready for review each morning.
|
||||
|
||||
Once merged, agents querying through the KTX CLI see the updated context immediately. No deployment step, no cache invalidation, no restart. The files are the source of truth, and agents read them on every request.
|
||||
|
||||
|
|
|
|||
|
|
@ -239,7 +239,7 @@ Agent integration ready: yes (claude-code:project)
|
|||
| `ktx: command not found` | The KTX package is not installed globally, or the shell cannot find the global binary | Run `npm install -g @kaelio/ktx` and open a new shell |
|
||||
| LLM health check fails | Missing, invalid, or unauthorized Anthropic API key | Export `ANTHROPIC_API_KEY` or rerun `ktx setup` and choose the file-backed secret option |
|
||||
| OpenAI embedding check fails | `OPENAI_API_KEY` is missing when OpenAI embeddings are selected | Export `OPENAI_API_KEY`, or rerun setup and choose local sentence-transformers embeddings |
|
||||
| Local embeddings hang or fail | The managed Python runtime cannot start or the local model runtime is unavailable | Install `uv`, run `ktx dev runtime doctor`, then run `ktx dev runtime install --feature local-embeddings --yes` and rerun setup |
|
||||
| Local embeddings hang or fail | The managed Python runtime cannot start or the local model runtime is unavailable | Install `uv`, run `ktx dev runtime status`, then run `ktx dev runtime install --feature local-embeddings --yes` and rerun setup |
|
||||
| Database connection test fails | Credentials, network access, warehouse, database, or schema value is wrong | Test the same URL with the database's native client, then rerun `ktx connection add ... --force` or rerun setup |
|
||||
| `KTX context built: no` in `ktx status` | Setup saved configuration but did not build context | Run `ktx setup` and choose to build context now |
|
||||
| Agent integration is incomplete | Setup skipped the agents step or the target was not installed | Run `ktx setup --agents --target codex --project` using the target you need |
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ Scanning connects to your database and extracts structural metadata. KTX stores
|
|||
### Running a scan
|
||||
|
||||
```bash
|
||||
ktx dev scan <connection-id>
|
||||
ktx scan <connection-id>
|
||||
```
|
||||
|
||||
This runs a structural scan by default. You can control what the scan does with the `--mode` flag:
|
||||
|
|
@ -25,25 +25,18 @@ This runs a structural scan by default. You can control what the scan does with
|
|||
|
||||
```bash
|
||||
# Scan with relationship detection
|
||||
ktx dev scan my-postgres --mode relationships
|
||||
ktx scan my-postgres --mode relationships
|
||||
|
||||
# Preview without writing results
|
||||
ktx dev scan my-postgres --dry-run
|
||||
ktx scan my-postgres --dry-run
|
||||
```
|
||||
|
||||
### Checking scan status
|
||||
### Checking scan results
|
||||
|
||||
Every scan produces a run ID. Use it to check progress or review results:
|
||||
Every scan prints a summary and writes local artifacts. Use `ktx status` after a scan to review project readiness and follow-up setup work:
|
||||
|
||||
```bash
|
||||
# Check status of a scan run
|
||||
ktx dev scan status <run-id>
|
||||
|
||||
# Print the full scan report
|
||||
ktx dev scan report <run-id>
|
||||
|
||||
# Get the report as JSON for scripting
|
||||
ktx dev scan report <run-id> --json
|
||||
ktx status
|
||||
```
|
||||
|
||||
### Relationship detection
|
||||
|
|
@ -56,49 +49,7 @@ Many databases lack declared foreign keys. KTX infers relationships by scoring c
|
|||
| 0.55 – 0.84 | `review` | Plausible — needs human review |
|
||||
| < 0.55 | `rejected` | Low confidence — not applied |
|
||||
|
||||
After a relationship scan, review the candidates:
|
||||
|
||||
```bash
|
||||
# Show candidates pending review (default)
|
||||
ktx dev scan relationships <run-id>
|
||||
|
||||
# Show all candidates regardless of status
|
||||
ktx dev scan relationships <run-id> --status all
|
||||
|
||||
# Accept a specific candidate
|
||||
ktx dev scan relationships <run-id> --accept <candidate-id>
|
||||
|
||||
# Reject a candidate with a note
|
||||
ktx dev scan relationships <run-id> --reject <candidate-id> --note "These columns share a name but are unrelated"
|
||||
```
|
||||
|
||||
Once you've reviewed candidates, apply the accepted ones as joins in your semantic layer:
|
||||
|
||||
```bash
|
||||
# Apply all accepted relationships
|
||||
ktx dev scan relationship-apply <run-id> --all-accepted
|
||||
|
||||
# Preview what would be applied
|
||||
ktx dev scan relationship-apply <run-id> --all-accepted --dry-run
|
||||
|
||||
# Apply a specific candidate
|
||||
ktx dev scan relationship-apply <run-id> --candidate <candidate-id>
|
||||
```
|
||||
|
||||
### Calibrating thresholds
|
||||
|
||||
As you review more relationships, KTX can evaluate whether the default thresholds (0.85 accept, 0.55 review) are optimal for your schema:
|
||||
|
||||
```bash
|
||||
# See how your feedback aligns with current thresholds
|
||||
ktx dev scan relationship-calibration --connection my-postgres
|
||||
|
||||
# Get threshold recommendations (needs 20+ labels, 5+ accepted, 5+ rejected)
|
||||
ktx dev scan relationship-thresholds --connection my-postgres
|
||||
|
||||
# Export your review decisions as calibration labels
|
||||
ktx dev scan relationship-feedback --connection my-postgres
|
||||
```
|
||||
Relationship scans run with `ktx scan <connection-id> --mode relationships`. This command only executes the scan; relationship review and calibration subcommands are not part of the current CLI surface.
|
||||
|
||||
## Ingestion
|
||||
|
||||
|
|
@ -115,19 +66,7 @@ Each ingest run follows this flow:
|
|||
### Running an ingest
|
||||
|
||||
```bash
|
||||
# Ingest one configured context source
|
||||
ktx ingest my-dbt-source
|
||||
|
||||
# Ingest every configured context source
|
||||
ktx ingest --all
|
||||
```
|
||||
|
||||
The public `ktx ingest` command uses the source configuration in `ktx.yaml`, including the source `driver` and any adapter-specific paths or credentials.
|
||||
|
||||
For adapter-level debugging, use the low-level `ktx dev ingest run` command:
|
||||
|
||||
```bash
|
||||
ktx dev ingest run --connection-id my-dbt-source --adapter dbt
|
||||
ktx ingest run --connection-id my-dbt-source --adapter dbt
|
||||
```
|
||||
|
||||
Useful low-level flags:
|
||||
|
|
@ -152,7 +91,7 @@ ktx ingest status <run-id>
|
|||
ktx ingest watch
|
||||
|
||||
# Replay a past ingest run
|
||||
ktx dev ingest replay <run-id>
|
||||
ktx ingest replay <run-id>
|
||||
```
|
||||
|
||||
The `watch` command opens an interactive TUI that shows the memory-flow output — every tool call, LLM decision, and artifact written during the ingest.
|
||||
|
|
@ -235,7 +174,7 @@ Orders in "pending" status for more than 48 hours are flagged for review.
|
|||
Every ingest session records a full transcript — tool calls, LLM responses, and write decisions. You can replay any session to debug why a source was written a certain way:
|
||||
|
||||
```bash
|
||||
ktx dev ingest replay <run-id> --viz
|
||||
ktx ingest replay <run-id> --viz
|
||||
```
|
||||
|
||||
This opens the same TUI view as the original run, letting you step through the agent's reasoning.
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ Agents should configure and ingest context sources in this order:
|
|||
|
||||
1. Add the context source connection in `ktx.yaml` or with `ktx setup`.
|
||||
2. Store tokens as `env:NAME` or `file:/path/to/secret`.
|
||||
3. Run `ktx ingest <connectionId>` for one source or `ktx ingest --all`.
|
||||
3. Run `ktx ingest run --connection-id <connectionId> --adapter <adapter>` for one source or `ktx ingest run --connection-id <id> --adapter <adapter>`.
|
||||
4. Check progress with `ktx ingest status --json`.
|
||||
5. Review generated `semantic-layer/` YAML and `knowledge/` Markdown files in git.
|
||||
6. Validate changed semantic sources with `ktx sl validate`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue