ktx/docs/superpowers/specs/2026-05-11-npm-managed-python-runtime-design.md

npm-managed Python runtime design

This spec defines how KTX ships as one visible npm package while still using Python for sqlglot, semantic-layer planning, database-agent compute, and local embeddings. The goal is a user experience where users install or run only @kaelio/ktx, and KTX manages its Python runtime automatically when a command needs it.

Goals

KTX must be usable through the npm package @kaelio/ktx with a ktx binary. Users can run KTX without learning about the Python packages that power parts of the system.

The first release must support these invocation modes:

• npx @kaelio/ktx setup demo
• npx @kaelio/ktx sl query ...
• npm install @kaelio/ktx, followed by npx ktx ...
• npm install -g @kaelio/ktx, followed by ktx ...

KTX-owned Python code must ship inside the npm package as a bundled wheel. KTX doesn't need to publish its own Python code to PyPI for this release.

Non-goals

This release does not need to provide a public TypeScript SDK split across multiple npm packages. The internal workspace package layout can remain useful for development, but the public npm surface is a single package.

This release does not need a fully offline install. KTX's own Python wheel is bundled, but third-party Python dependencies can come from PyPI through uv.

This release does not install local embedding dependencies by default. Local embeddings remain lazy because sentence-transformers, torch, and model downloads are large.

Package model

KTX publishes one public npm package:

@kaelio/ktx

That package exposes one binary:

{
  "bin": {
    "ktx": "./dist/bin.js"
  }
}

The npm package includes these assets:

• Bundled JavaScript CLI output.
• Packaged demo assets.
• One KTX-owned Python wheel, for example python/kaelio_ktx-0.1.0-py3-none-any.whl.
• A wheel checksum or runtime manifest that lets the CLI verify the bundled Python payload before installation.

The Python wheel contains the current semantic_layer and ktx_daemon modules. It exposes at least the ktx-daemon console script.

Runtime installation

KTX creates a managed Python runtime only when a command needs Python-backed behavior. The runtime lives outside the npm cache so it survives npx runs.

The runtime root is platform-specific:

• macOS: ~/Library/Application Support/kaelio/ktx/runtime
• Linux: ${XDG_DATA_HOME:-~/.local/share}/kaelio/ktx/runtime
• Windows: %LOCALAPPDATA%/Kaelio/KTX/runtime

The runtime is versioned by the npm package version. A versioned runtime avoids mixing JavaScript and Python code from incompatible releases.

The installer performs these steps:

1. Locate uv.
2. Create a virtual environment under the versioned runtime directory.
3. Install the bundled KTX wheel into that environment.
4. Write a runtime manifest with the CLI version, wheel checksum, Python executable, daemon executable, and installed feature set.

For lightweight Python support, the install command uses the bundled wheel's default dependency set. For local embeddings, the installer adds the embeddings extra only when selected:

uv pip install "/path/to/kaelio_ktx-0.1.0-py3-none-any.whl"
uv pip install "/path/to/kaelio_ktx-0.1.0-py3-none-any.whl[local-embeddings]"

Feature installation levels

KTX manages Python runtime features in levels so first use stays fast.

core includes:

• sqlglot
• pydantic
• pyyaml
• fastapi
• uvicorn
• lightweight daemon dependencies

local-embeddings adds:

• sentence-transformers
• torch
• model download support for all-MiniLM-L6-v2

Commands that only need semantic-layer SQL generation require core. Commands that need local embeddings require local-embeddings.

Command behavior

Pure TypeScript commands run without the managed Python runtime.

Python-backed one-shot operations use the managed ktx-daemon executable directly. Examples include semantic query compilation, semantic validation, semantic source generation, and sqlglot-backed table identifier parsing.

Repeated or expensive operations use a managed HTTP daemon. Local embeddings use the daemon because loading the model for every one-shot process is too slow.

KTX provides runtime management commands:

ktx runtime install
ktx runtime status
ktx runtime start
ktx runtime stop
ktx runtime doctor
ktx runtime prune

Normal commands can install the runtime lazily. Runtime commands make that behavior inspectable and debuggable.

Daemon lifecycle

The daemon binds to 127.0.0.1 on an available random port. KTX writes daemon state to the runtime manifest or an adjacent state file:

{
  "pid": 12345,
  "port": 58731,
  "version": "0.1.0",
  "features": ["core", "local-embeddings"],
  "startedAt": "2026-05-11T00:00:00Z"
}

Before reusing a daemon, KTX checks that the process is alive, the port responds to /health, and the daemon version matches the CLI version. If any check fails, KTX treats the daemon as stale and starts a new one.

KTX uses one-shot Python for short operations by default. It starts the daemon only when a command benefits from process reuse.

Interactive and CI behavior

In an interactive terminal, KTX prompts before installing the managed runtime for the first time. The prompt states that Python dependencies will be downloaded.

With --yes, KTX installs the required runtime features without prompting.

With --no-input, KTX fails if a required runtime feature is missing and no explicit auto-install flag is present. The error prints the exact command to prepare the runtime.

For local embeddings, KTX prompts separately because the dependency and model downloads are larger than the core runtime.

Error handling

If uv is missing, KTX prints a focused error that explains how to install it and how to retry. A later release can add a bundled or downloaded uv strategy.

If Python runtime installation fails, KTX preserves install logs in the runtime directory and prints the log path.

If the daemon fails to start, KTX prints the captured daemon stdout and stderr path. It falls back to one-shot mode only when the requested operation supports one-shot execution.

If JavaScript and Python versions don't match, KTX reinstalls the managed runtime for the current npm package version.

Release flow

The release builds the Python wheel before packing npm artifacts. The npm pack step includes the wheel as an asset.

Release checks must cover:

1. Clean install of the packed npm package.
2. npx execution of the packed package.
3. First-run managed runtime install from the bundled wheel.
4. One-shot semantic-layer query through the managed runtime.
5. Runtime status and doctor output.
6. Daemon start, health check, reuse, and stop.
7. Optional local embeddings smoke in a separate job or opt-in check.

Open decisions

KTX still needs a final decision on whether uv is a hard prerequisite or a bootstrap dependency that KTX downloads automatically.

KTX also needs the final Python distribution name. This spec uses kaelio-ktx as the distribution name and kaelio_ktx in wheel filenames.

Success criteria

Users can run npx @kaelio/ktx ... and complete Python-backed KTX operations without manually installing a KTX Python package.

Users who install @kaelio/ktx locally can run npx ktx ... through the local project's npm binary resolution.

The first Python-backed command installs only the core runtime. Local embedding dependencies install only after the user selects local embeddings or explicitly requests the local-embeddings runtime feature.

KTX can diagnose and repair stale or mismatched managed runtimes without asking users to delete directories manually.